Skip to content

Metadata propagator

The metadata propagator package allows to propagate a set of selected metadata through lineage. The package gives the flexibility to select which metadata to propagate and where to propagate them.

Warning

By default metadata are propagated only if the target asset has only one dependency (upstream if the propagation is downstream, downstream if the propagation is upstream). The only exception are tags that are propagated upstream even if the asset has more than one dependency. To relax this constraint the flag Check direct dependency can be used.

Configuration

Credentials

  • Atlan API Token: this is required to generate the dynamic dropdowns (see the Filters and Impacted Asset Filters screens)

Configurations

  • Attributes to propagate:

    • Description

      Description: priority order for different attributes

      The userDescription takes priority over description when both are present. For more information, see How does version control work for description changes in source tools vs. Atlan?

    • Owner Users

    • Owner Groups
    • Announcement: type, title and message are propagated.
    • Assigned Terms
    • Certificate
    • Tags: tags are propagated only upstream. Dowstream tags propagation is already covered by the out of the box tags propagation feature.
    • Readme
    • Domain

  • Custom metadata to propagate: list of custom metadata to be propagated. Format: <custom_metadata_1_name>.<attribute_1_name>@@@<custom_metadata_2_name>.<attribute_2_name>.

    Custom metadata format: example

    Separate multiple custom metadata with @@@ and each custom metadata is composed by the name of the custom metadata and the name of the attribute to propagate, separated by a dot. See the example below. 

        |-- Scores
    |   |-- Missing Metadata
    |   |-- Score Contributors
    |-- SLA
        |-- Update Frequency
    For the given custom metadata, the value to enter is: Scores.Missing Metadata@@@SLA.Update Frequency

  • Propagation direction: whether to propagate metadata across the lineage or to related assets.

    • Lineage direction: whether the propagate the metadata downstream or upstream through lineage.
    • Lineage Depth: propagation depth. If empty metadata are propagated through the full lineage.
    • Check direct dependency: Propagate metadata only if the target asset has a single dependency—upstream for downstream propagation, downstream for upstream propagation. If you select No, propagation may result in conflicts during propagation.
    • Hierarchy Type: specify the type of hierarchy to be used for metadata propagation. Options include:
      • Relational (Any Connector): propagate metadata for any relational model asset based on qualified name of the source asset.
    • Hierarchy Direction: select the direction of the hierarchy to be followed from source asset for metadata propagation.
    • Relationship Names: comma delimited relationship names to find assets related to the source asset to propagate metadata to. For more information, see relationship names.
    • The relationship map from developer portal can be used to identify and manage all relationships in atlan.
    • See the example snippet below for AtlasGlossaryTerm and its relationships.
    flowchart TB
    subgraph Data-Assets
        direction TB
            Asset_1[Asset]
    end
    subgraph Glossary
        direction TB

            AtlasGlossary[Glossary] -->|categories| Categories
            Categories -->|anchor| AtlasGlossary

            subgraph Categories
                direction TB
                AtlasGlossaryCategory_1[Category] -->|childrenCategories| AtlasGlossaryCategory_2[Category]
                AtlasGlossaryCategory_2[Category] -->|parentCategory| AtlasGlossaryCategory_1[Category]
            end

            AtlasGlossary -->|terms| AtlasGlossaryTerm_1[Term]
            AtlasGlossaryTerm_1 -->|anchor| AtlasGlossary

            Categories -->|terms| AtlasGlossaryTerm_2[Term]
            AtlasGlossaryTerm_2 -->|categories| Categories

            AtlasGlossaryTerm_1 -->|seeAlso| AtlasGlossaryTerm_2
            AtlasGlossaryTerm_2 -->|seeAlso| AtlasGlossaryTerm_1
    end

    Asset_1 -->|meanings| AtlasGlossaryTerm_1
    AtlasGlossaryTerm_1 -->|assignedEntities| Asset_1

    The diagram illustrates the relationship between glossary assets and their associated data assets:

    • An AtlasGlossaryTerm is related to a data asset through the assignedEntities relationship.
    • The AtlasGlossaryTerm is related to the glossary asset through the meanings relationship.
    • Any relationship to an AtlasGlossary is represented by the anchor relationship.
    • The other end of the relationship to AtlasGlossary can either be categories or terms, depending on the asset.
    • Relationship Names string is based on the relationship names present in the source assets’ metamodel. For example:
      • To propagate metadata from glossary term to data assets that have the term assigned to them: assignedEntities
      • To propagate metadata from data assets to glossary terms assigned to the assets: meanings
      • To propagate metadata from glossary terms to it's parent category and glossary along with related terms that fall under seeAlso relationship: categories,anchor,seeAlso

    Navigate to desired asset's relationship documentation in developer portal to find all relationships for that asset.

  • Propagation mode:

    • Append: whether to append the propagated metadata to the ones already defined in the target asset. This option is valid only for Owner Users, Owner Groups, Assigned Terms and Tags.
    • Replace: whether to fully replace the metadata of the target asset with the one propagated.
    • Propagate only if empty: controls whether metadata is propagated only when the corresponding field on the target asset is empty. If the target already has a value, propagation is skipped for that attribute.

  • Propagate empty announcement: whether the propagate the empty/no announcement. Please note that if selected, it will remove all announcements from the propagated assets.

    Warning

    Only announcements propagated by this package will be removed.

  • Operation type:

    • Preview: the package generates a csv file with the list of metadata that will be propagated.
    • Propagate: the package propagates the metadata according to the configuration defined.

    Warning

    It's recommended to start with the Preview operation type to double check that the workflow is propagating metadata according to your expectations. Reverting back propagation requires ad-hoc scripts.

  • Mode: whether the propagation is executed for all assets (full) or just for the ones updated since the last time the workflow was executed (incremental). The incremental option is valid only if opearation type is set to 'Propagate'.

  • Asset name match: whether to propagate the metadata only to assets that match the name of the source asset.

  • Case sensitive match: whether to propagate the metadata only to assets that match the name of the source asset in a case sensitive way. This is only valid if the Asset name match is set to Yes.

Filters

This screen allows to apply filters on the asset population. Only the metadata of the assets selected by this set of filters will be propagated.

Each filterable attribute has an operator and a value. The operator is always a dropdown menu while the value can be either a dropdown or a free text field. The package considers the filter only if both the operator and the value are filled. The only exceptions are the operators Is Empty and Is Not Empty where the value isn't required (and ignored if filled). The available filters are:

  • Connector
  • Connection
  • Asset Type
  • Name
  • Certificate
  • Qualified Name
  • Tag
  • Description
  • Announcement
  • Owner User
  • Owner Group
  • Has Terms

Impacted asset filters

This screen allows to apply filters on the upstream/downstream assets. Metadata are propagated to assets that are upstream/dowstream to the one selcted in the above Filter section and that match the filters defined in this section. Leave it empty to propagate the metadata to all upstream/dowstream assets.

Each filterable attribute has an operator and a value. The operator is always a dropdown menu while the value can be either a dropdown or a free text field. The package considers the filter only if both the operator and the value are filled. The only exceptions are the operators Is Empty and Is Not Empty where the value isn't required (and ignored if filled). The available filters are:

  • Connector
  • Connection
  • Asset Type
  • Name
  • Certificate
  • Qualified Name
  • Tag
  • Description
  • Announcement
  • Owner User
  • Owner Group

Warning

Process, BIProcess, ColumnProcess, DbtColumnProcess, DbtProcess, Query assets are filtered out by default.

What it does

The package performs the following steps:

  • Translate the UI input into an ElasticSearch query.
  • Retrieve assets from Atlan.
  • Retrieve the dowstream/upstream/related assets of the one selected.
  • Filter them according to the rules defined in the Impacted Asset Filters section.
  • Propagate the metadata.