Skip to content

Orchestration Tasks Attribute Transfer

Prerequisite

The package requires the users to create a custom metadata with same number of properties as selected in the package config. All the property types should be of type text and correct names for custom metadata and properties should be provided in the config.

The Orchestration Tasks Attribute Transfer package provides orchestration tasks/jobs attribute migration capabilities for all downstream/upstream assets (only input/output assets to the task/job) at level-1 depth. The package propagates selected attributes to the provided custom metadata from all the tasks/jobs that fulfill the selected connector and source filter criteria to the downstream/upstream (direction selection in config) assets which come under the target asset filter provided in the config.

This package will ignore asset types like connection, process and DAGs when traversing the lineage. Inputs to a task/job are considered upstream assets and outputs from a task/job are considered downstream assets. Attributes supported by the package :

  • Run Status
  • Run End Timestamp
  • Task Atlan Link
  • Task Schedule
  • DAG Atlan Link
Create Custom Metadata
  • Go to Governance and select Custom Metadata.
  • Create a new custom metadata and note the name(this will be the custom metadata name input for the workflow,e.g.: OL Task).
  • Create a property in this custom metadata (with type Text for Run Status, Run End Timestamp, Task Schedule and type URL for Task Atlan Link, DAG Atlan Link) and optionally limit it to the target assets to which you want to propagate the selected attributes into the custom metadata(the name of the property is input for the workflow for the attribute it is created for,e.g.: Task Status).
  • Create a custom metadata property for each selected attribute.

Configuration

Configuration

  • Connector Names : Select the connectors for Tasks/Jobs, from which you want to propagate attributes to assets. Supported connecotrs are Airflow, MWAA Airflow, Astronomer Airflow, Google Cloud Composer Airflow and Spark.
  • Custom Metadata Name : Name of the custom metadata created to store the attributes.

  • Transfer Direction : The direction in which lineage traversal is to be done for attribute transfer.

    For depth=1, pick the outputs to a task/job as the downstream assets.

    For depth=1, pick the inputs to a task/job as the downstream assets.

  • Attribute Name : The task/job attribute to transfer to the custom metadata. Select one from the dropdown and map it to the corresponding custom metadata property.

  • Custom Metadata Property Name : Property to be mapped with the selected attribute name on the left. Repeat this mapping for all the properties that you desire to transfer.

Source Filters

This screen allows to apply filters on the tasks/jobs present in the selected connectors. Only the attributes of the tasks/jobs selected by this set of filters will be propagated.

Each filterable attribute has an operator and a value. The operator is always a dropdown menu while the value can be either a dropdown or a free text field. The package considers the filter only if both the operator and the value are filled. The only exceptions are the operators “Is Empty” and “Is Not Empty” where the value is not required (and ignored if filled). The available filters are:

  • Name
  • Certificate
  • Qualified Name
  • Description
  • Announcement

Target Filters

This screen allows to apply filters on the upstream/downstream assets. Attributes are transfered to assets that are upstream/dowstream to the tasks/jobs selcted in the above Source Filters section and that match the filters defined in this section. Leave it empty to propagate the metadata to all upstream/dowstream assets.

Each filterable attribute has an operator and a value. The operator is always a dropdown menu while the value can be either a dropdown or a free text field. The package considers the filter only if both the operator and the value are filled. The only exceptions are the operators “Is Empty” and “Is Not Empty” where the value is not required (and ignored if filled). The available filters are:

  • Connection
  • Name
  • Certificate
  • Qualified Name
  • Description
  • Announcement

What it does

Searches for all the Tasks/Jobs present in the selected connectors while considering the source filters for these tasks to limit the lookup. Then picks either inputs or outputs to these assets depending on the direction selected and transfers attributes to the custom metadata property after applying the target filters to these inputs/outputs.

How it works
  • Search for Orchestration assets in the selected connectors while also adding the source filters. This process adds ignore condition for process, connection and DAGs.
  • Include the inputs/outputs in the source search depending on the transfer direction selected (upstream/downstream).
  • Iterates over the retrieved assets and picks each input/output (depending on the direction) and checks if any asset is partial. Skips if partial.
  • Applys target filter on these picked asset and skips if any one of these filters return False.
  • If all target filters are true for the asset, replaces the custom metadata properties with the new attribute values.