Skip to content

Data model ingestion

Excel template for data model ingestion

A data model defines how data is structured, organized, and related within a system. You can ingest your data model into Atlan using the Data Model Ingestion package. This guide provides step-by-step instructions for ingesting your data model into Atlan.

Prerequisites

Before you begin, complete the following steps:

  • Enable the data model ingestion package: Raise a ticket with Atlan Support or contact your Customer Success Manager (CSM) to enable the Data Model Ingestion package for your tenant.

  • Prepare your data: Download the Excel template for data model ingestion and populate the following sheets with your data:

    • Objects
    • Relations
    • Mappings
    • Implementations

Permissions required

To successfully ingest your data model, confirm that your user role has the necessary permissions:

  • Access to the Atlan workspace: Required to create and manage connections.
  • Workflow execution permissions: Needed to run and schedule ingestion workflows.

Create and run workflow

Use the following steps to create and run a workflow for ingesting your data model into Atlan.

  1. In Atlan, click + New, then select New Workflow from the menu.
  2. Navigate to the Marketplace tab, search for Data Model Ingestion, and select it.
  3. Click Setup Workflow to configure the data model ingestion process.

  4. Set up data model ingestion workflow with following details:

    • Workflow Name - Enter a meaningful name, such as Production Finance Data Export.

    • Excel Input - Upload the Excel file you prepared in the Prerequisites section. Select how you want to provide the input file to be processed:

      Directly upload a CSV file containing the assets.

      • Assets file: the CSV file containing details to load, for assets.

      Limited file sizes

      This option is generally limited to ~10MB for each file. For anything larger, use object storage.

      Retrieve the relational assets file from cloud object storage.

      • Prefix (path) the directory (path) within the object store from which to fetch the file containing relational assets metadata.
      • Object key (filename) the object key (filename), including its extension, within the object store and prefix.

      • Cloud object store the object store from which to fetch the file conaining cube assets.

        • AWS access key: your AWS access key.
        • AWS secret key: your AWS secret key.
        • Region: your AWS region.
        • Bucket: your AWS bucket.

        Reusing Atlan's backing S3 store

        When your Atlan tenant is deployed in AWS, you can leave all of these blank to reuse the backing store of Atlan itself. You can also set up a cross-account bucket policy for Atlan to have access to your S3 bucket and the objects within it, and leave these blank.

        • Project ID: the ID of your GCP project.
        • Servive account JSON: your service account credentials, as JSON.
        • Bucket your GCS bucket.

        Reusing Atlan's backing GCS store

        When your Atlan tenant is deployed in GCP, you can leave all of these blank to reuse the backing store of Atlan itself.

        • Azure client ID: the unique application (client) ID assigned to your app by Azure AD when the app was registered.
        • Azure client secret: your Azure client secret (it's actual value, not its identifier).
        • Azure tenant ID: the unique identifier of the Azure Active Directory instance.
        • Storage account name: name of your storage account.
        • Container: your ADLS container.

        Reusing Atlan's backing ADLS store

        When your Atlan tenant is deployed in Azure, you can leave all of these blank to reuse the backing store of Atlan itself.

    • Connection - Select a connection type based on your requirements:

      Choose Create to set up a new connection for data model ingestion. Provide the following details:

      • Connection name – Enter a unique and descriptive name.
      • Connection admins (Optional) add or remove users or groups who can manage this connection.
      • Include all admins – Select this option to allow all admins to manage the connection.

      Careful!

      If you do not specify any users or groups, no one will be able to manage the connection—not even admins.

      Choose Reuse to use an existing connection:

      • Connection Name – Select the existing connection you want to use from the available connections.

  5. Choose Run to execute the workflow immediately or Schedule and Run to set up a scheduled execution.

Once the workflow runs successfully, your data model will be ingested into Atlan and available for further use.

See also