Lineage builder

The lineage builder package allows you to create lineage between any source and any target asset.

Use cases

• Add your own arbitrary lineage connections, to render in lineage like any others
• Inject lineage for transformations from a tool Atlan does not mine out-of-the-box

Configuration

• Import lineage from: select how you want to provide the input file to be processed (see below for the required format)

  Direct uploadS3 object

  Directly upload a CSV file containing the lineage. Note that this is generally limited to ~10-20MB.

  • Lineage file: the CSV file containing lineage details to load.

  Specify the details of an S3 object, for which there is no size limit.

  • S3 region: the S3 region where the input file is hosted. If you are using the S3 storage back-end for the Atlan tenant itself, you can leave this blank.
  • S3 bucket: the S3 bucket where the input file is hosted. If you are using the S3 storage back-end for the Atlan tenant itself, you can leave this blank.
  • S3 object key: the complete object key for the S3 object — including any prefix ("directory" structure) and the filename itself.

  When not using Atlan's S3 back-end storage

  When using your own (rather than Atlan's) S3 storage for the input, you must first set up a cross-account bucket policy for Atlan to have access to your S3 bucket and the object within it.

• Unknown asset handling: how you want to handle source and target assets in the input file that do not match any assets in Atlan

  Skip themCreate partial assetsCreate full assets

  Any source or target assets in the input file that do not match any asset in Atlan will be skipped (those lineage rows will not be loaded).

  For any source or target asset in the input file that does not match any asset in Atlan, create a "partial" asset. These assets only appear in lineage, and cannot be searched or detailed through the sidebar. They can be later "resolved" into full assets which are then discoverable and visible in the asset sidebar.

  For any source or target asset in the input file that does not match any asset in Atlan, create a full asset. These assets will behave like any other: they will be discoverable through search and appear in the asset sidebar.

• Fail on errors: whether an invalid value in a field should cause the import to fail (Yes) or log a warning, skip that value, and proceed (No).

• Case-sensitive match for assets: whether attempts to match assets should be done case-sensitively (Yes) or case-insensitively (No).

What it does

Lineage CSV file

For each row of the CSV file supplied represents a single lineage process to be created (or updated).

Connections must pre-exist

This package will not create any connections — the connections referenced must already exist before running the package.

Detailed information on the columns in the CSV file:

Source Type

Required. Type of the source (input, upstream) asset in the lineage.

Source Connector

Required. Type of connector for that asset (e.g. the kind of source system).

Source Connection

Required. Name of the connection in Atlan that should contain the asset. (If the asset does not exist and creation is permitted, this is the connection in which the source asset will be created.)

Source Identity

Required. Unique identity of the source asset in Atlan within the connection. This is essentially its qualifiedName, excluding the connection portion of the qualifiedName (which will instead be calculated for you from the preceding fields).

Source Name

Required. Simple name of the source asset.

Target Type

Required. Type of the target (output, downstream) asset in the lineage.

Target Connector

Required. Type of connector for that asset (e.g. the kind of target system).

Target Connection

Required. Name of the connection in Atlan that should contain the asset. (If the asset does not exist and creation is permitted, this is the connection in which the target asset will be created.)

Target Identity

Required. Unique identity of the target asset in Atlan within the connection. This is essentially its qualifiedName, excluding the connection portion of the qualifiedName (which will instead be calculated for you from the preceding fields).

Target Name

Required. Simple name of the target asset.

Transformation Connector

Required. Type of connector for the transformation or data movement process itself (e.g. the kind of system).

Transformation Connection

Required. Name of the connection in Atlan that should contain the lineage process. (Where the lineage process will be created or updated.)

Transformation Identity

Required. Unique identity of the transformation (lineage) process in Atlan within the connection. This is essentially its qualifiedName, excluding the connection portion of the qualifiedName (which will instead be calculated for you from the preceding fields).

Transformation Name

Required. Simple name to use when displaying the lineage process.

.. (remaining columns) ..

You can also supply any number of additional columns. These will be loaded as attributes on the lineage process itself, and must follow the format described in the asset export (basic) package or use a column name that matches one of the lineage process properties or more general asset properties .

How it works

If asset creation is permitted, runs a first pass through the supplied CSV file to create any assets within their respective connections.

Then proceeds to create a single lineage process per row of the supplied CSV file.

---