Relational assets builder¶
The relational assets builder package allows you to create (and update) net-new relational assets: connections, databases, schemas, tables, views, materialized views and columns.
Use cases¶
- Ingest metadata for relational assets where there is not (yet) an out-of-the-box crawler
Configuration¶
-
Import assets from: select how you want to provide the input file to be processed (the format must match the expected CSV file format)
Directly upload a CSV file containing the assets. Note that this is generally limited to ~10-20MB.
- Assets file: the CSV file containing details to load, for assets.
Specify the details of an S3 object, for which there is no size limit.
- S3 region: the S3 region where the input file is hosted. If you are using the S3 storage back-end for the Atlan tenant itself, you can leave this blank.
- S3 bucket: the S3 bucket where the input file is hosted. If you are using the S3 storage back-end for the Atlan tenant itself, you can leave this blank.
- S3 object key: the complete object key for the S3 object — including any prefix ("directory" structure) and the filename itself.
When not using Atlan's S3 back-end storage
When using your own (rather than Atlan's) S3 storage for the input, you must first set up a cross-acount bucket policy for Atlan to have access to your S3 bucket and the object within it.
-
Remove attributes, if empty: by default, any value that is blank in the CSV file will simply be ignored. If you would prefer the value to instead be overwritten on the asset in Atlan (i.e. that value removed from the asset), you can select the field here. Any fields selected here whose values are empty in the file will result in the value being removed from the asset in Atlan.
Atlan tags are always overwritten
Note that the Atlan tags (
atlanTags
) column will always be replaced wholesale. Therefore, if the Atlan tags column is empty in the CSV input, but the asset in Atlan has tags assigned, after running the import that asset will no longer have any tags assigned. -
Input handling: how to handle assets in the CSV file that do not exist in Atlan
Create a full-fledged asset that can be discovered and maintained like other assets in Atlan. (Any existing assets will be updated.)
Only update assets that already exist in Atlan, and do not create any asset, of any kind.
Does not apply to related READMEs and links
READMEs and links in Atlan are technically separate assets — but these will still be created, even in Update only mode.
-
Fail on errors: whether an invalid value in a field should cause the import to fail (
Yes
) or log a warning, skip that value, and proceed (No
).
What it does¶
For each row in the CSV file, will apply the values from the CSV file to that asset in Atlan.
- Any values that are empty in the CSV will be ignored, unless they were specified in the
Remove attributes, if empty
configuration — in which case they will be removed from the asset in Atlan. - Any attributes for which there can be multiple values (
ownerUsers
,ownerGroups
,assignedTerms
,atlanTags
,links
,starredDetails
) will be replaced on the asset in Atlan. For example, if the asset in Atlan had 3ownerUsers
and the CSV file only lists 1, then after importing the asset in Atlan will have only 1 user inownerUsers
(the one from the CSV input file).
Each typeName requires different minimal set of details
The type of asset loaded for each row in the CSV depends on the value in the typeName
column. Different types require different information: for example, a Connection
only requires a connectionName
and connectorType
, while a column requires: connectionName
, connectorType
, databaseName
, schemaName
, entityName
, and columnName
.
CSV file¶
The following describes the expected format of the CSV file. The ordering of the columns in the file does not matter, but they must have exactly the same name.
typeName
¶
Required. Type of the asset. This determines what kind of asset will be created or updated by the row of the CSV.
connectionName
¶
Required. Name of the connection. This is any arbitrary name you want to use to group assets for a particular source, for example a system or application name.
connectorType
¶
Required. Type of connector used to represent the source. View the available options and associated icons, specifically the all-lowercase value under the Raw REST API
tab.
databaseName
¶
Technical name of the database, as you would crawl it from source. This is required for database assets and any assets contained within a database.
schemaName
¶
Technical name of the schema, as you would crawl it from source. This is required for schema assets and any assets contained within a schema.
entityName
¶
Technical name of the table, view, or materialized view, as you would crawl it from source. This is required for table, view and materialized view assets, and any assets contained within them.
columnName
¶
Technical name of the column, as you would crawl it from source. This is required for columns.
dataType
¶
Data type of a column. You can specify this as a conceptual type (like string
) or a full SQL type (like NVARCHAR(255)
or DECIMAL(5,2)
).
displayName
¶
An optional name you can give to the asset to override how it is displayed in the Atlan UI. If present, this will be shown in the UI instead of name
.
description
¶
Explanation of the asset, possibly crawled from a source system.
userDescription
¶
Explanation of the asset, as entered or confirmed by a user through the Atlan UI. If present, this will be shown in the UI instead of description
.
ownerUsers
¶
Individual users who are owners of the asset. Each user should be separated by a newline within the cell.
ownerGroups
¶
Groups of users who are owners of the asset. Each group should be separated by a newline within the cell.
certificateStatus
¶
Certificate on the asset. Must either be empty or one of:
VERIFIED
DRAFT
DEPRECATED
certificateStatusMessage
¶
An optional message that can be associated with the certificate (only used if certificateStatus
is non-empty).
announcementType
¶
Type of announcement on the asset. Must either be empty or one of:
information
warning
issue
announcementTitle
¶
Heading line for the announcement on the asset (only used if announcementType
is non-empty).
announcementMessage
¶
An optional detailed message that can be associated with the announcement (only used if announcementType
is non-empty).
assignedTerms
¶
Business terms that are assigned to the asset. Each term should be separated by a newline within the cell, and formatted as:
Term Name@@@Glossary Name
atlanTags
¶
Atlan tags that are assigned to the asset. Each tag should be separated by a newline within the cell, and formatted as one of:
Tag Name
, for tags that should be directly assigned and should not be propagatedTag Name>>FULL
for tags that should be directly assigned to the asset and propagated down their hierarchy and through lineageTag Name>>HIERARCHY_ONLY
for tags that should be directly assigned to the asset and only be propagated down their hierarchy (not through lineage)-
Tag Name<<PROPAGATED
for tags that have been propagated to the asset.Propagated tags will be ignored on import
Any tag marked propagated (
Tag Name<<PROPAGATED
) will be ignored by an import. Only those tags that are directly applied will be imported, though of course any tags applied up-hierarchy or upstream that are marked to propagate will still propagate accordingly.
links
¶
List of resources (links) assigned to the asset. Each link should be separated by a newline within the cell, and formatted as embedded JSON:
{"name":"linkName","link":"https://www.example.com"}
readme
¶
Richly-formatted, detailed documentation for the asset. This should be an HTML-formatted string containing everything that would be inside <body></body>
, without the <body></body>
wrapping.
starredDetailsList
¶
Details about users who have starred the asset. Each starred asset detail entry should be separated by a newline within the cell, and formatted as embedded JSON:
{"assetStarredBy":"someone","assetStarredAt":1698769268966}
{{CM}}::{{Attribute}}
¶
Values for any custom metadata. You simply need to name the column after the custom metadata and attribute name. For example, if the custom metadata is RACI
and the attribute is Responsible
, the column name must be RACI::Responsible
. The values for the column follow the same rules as above: for multiple values use an in-cell newline to delimit them.
How it works
Reads from the CSV file and creates a number of parallel batches for submitting the updates in several passes:
- The first pass preprocesses certain information in the CSV file, for example: the relative position of columns (to determine their ordering), counts of children objects (columns per table, tables per schema, etc), and so on.
- The second pass will create and update assets themselves, noting any related assets (like links and READMEs) that may also need to be updated, created, or deleted.
- The third pass will load the related assets' and process any deletions.