Skip to content

Alert Propagation

The alert propagation package detects issue states on assets due to data quality or pipeline issues and propagates alerts as metadata to downstream assets.

It currently supports the following types of issues that are built on top of asset types for which Atlan has native connectors.

Data Quality Issues

  • Monte Carlo incidents (if they are related to a monitor)

Pipeline Errors

  • dbt jobs

Additionally, it can support custom integration with a pipeline or data quality process. In addition to supporting non-native pipeline systems/solutions, it also allows alert states to be set and propagated in Atlan in nearer-real-time. If interested in the options for this, please ask your Atlan Success Manager or Solutions Architect.

Configuration

  • Scope / Qualified Name Prefix: Qualified name prefix of the assets for which alerts should be propagated from/to. For all, enter 'default/'.
  • Incident Types: Types of incidents to propagate alerts for. Must be quoted and comma-separated. Eg. 'CUSTOM_RULE_ANOMALIES','METRIC_ANOMALIES'
  • Alerts Custom Metadata Set Name: Name of the Custom Metadata set to be used/created which will house the flags for the Custom Metadata Alerts.
  • Include Annoucements: Whether or not the alert propagation should create Announcements in addition to Custom Metadata. WARNING: Should not be used if announcments are used (or will be used) for other purposes. THEY COULD BE OVERWRITTEN OR DELETED. If in doubt, leave set to 'No'.
  • Include Announcement Confirmation: If 'Yes' is selected for 'Include Announcements', this is a required input to confirm that the risks of "including announcements" are understood.
  • Seconds to Sleep for Propagation: Number of seconds to sleep between applying tags and custom metadata (to allow for native Atlan tag propagation to occur).

What it does

  1. Searches for assets that have either a Monte Carlo incident OR a dbt job "Error" state.
  2. For any assets found, adds a tag to the assets (either 'Alert: DQ' or 'Alert: Pipeline', according to the type of issue). These tags are configured to propagate over lineage only.
  3. After tag propagation, update the asset with custom metadata + badge (specifc to 'DQ' or 'Pipeline' in order to better promote the fact that there is an upstream issue).
  4. Optionally, add an announcement so that visibility about the alert is present on the lineage graph for all assets potentially affected.
  5. If an asset previously had an issue, but is no longer in an issue state, the tag is removed, and downstream custom metadata/announcements are cleared.