SAP Datasphere crawler¶

The SAP Datasphere crawler package crawls SAP Datasphere system and brings in the views and analytical models(optional) into Atlan

Warning

Analytical models(if chosen) will be catalogued as a view in Atlan
Whitelisting of Atlan IPs at SAP DS (if required) must be done prior to running the package
Only Views and analytical models are supported with respect to SAP DS
Package supports only creation of full assets

Authentication¶

Authentication

The package supports OAuth based client id-secret authentication with SAP DS.

The authentication flow begins with manually obtaining an authorization code from the authorization endpoint (). This process requires user intervention and must be done initially to generate the necessary access token and refresh token. These tokens must then be provided during the first execution of the package. For subsequent runs within the validity period of the refresh token, the package automatically handles the refresh of the access token, ensuring seamless operation. The refresh token has a specific validity period, and once it expires, the authorization code must be obtained again, and the package must be re-run.

Data type mapping between SAP DS and Atlan¶

SAP DS Data Type	Atlan Data Type	Condition
Edm.String	NVARCHAR	If @MaxLength exists: NVARCHAR(@MaxLength)
Edm.Decimal	DECIMAL	DECIMAL(@Precision, @Scale) Defaults: DECIMAL(18,0)
Edm.Int32	INT	Always INT
Edm.Int16	SMALLINT	Always SMALLINT
Edm.Date	DATE	Always DATE
Edm.Byte	BYTE	Always BYTE
Edm.Binary	BINARY	Always BINARY
Edm.DateTimeOffset	DATETIMEOFFSET	If @Precision exists: DATETIMEOFFSET(@Precision)
Edm.Double	DOUBLE	Always DOUBLE
Other	UNKNOWN	Any other unrecognized type

Configuration¶

Workflow name¶

(required) Provide a unique name to identify your workflow instance

SAP Datasphere connection name¶

(required) Mandatory connection name under which asset will be catalogued in Atlan

SAP Datasphere Database name¶

(required) Provide a DB name under which the assets are present in SAP DS

SAP Dataspehere token issuer host name¶

(required) Provide the hostname in the token issuer endpoint

SAP Datasphere Assets host name¶

(required) Provide the hostname of the assets endpoint

SAP Datasphere Assets URI¶

(required) Provide the URI of the assets endpoint(e.g: /dwaas-core/odata/v4/catalog/assets)

SAP Datasphere Token URI¶

(required) Provide the URI of the token endpoint(e.g: /oauth/token)

Asset Endpoint¶

(required) Provide the endpoint from where assets can be scrapped

Refresh token¶

(required) Refresh token must be provided for subsequent refreshes

Client id¶

(required) OAuth client Id for Atlan package created in SAP DS

Client secret¶

(required) OAuth client secret for Atlan package created in SAP DS

Analytical models¶

(optional) Default: false Choose if you want to catalog analytical models

Delta handling¶

(required) how changes should be detected and handled (if at all)

=== "Full replacement"

    Will delete any assets that were in a previous input file, but are no longer in the most recent input file provided. (Will also update any existing assets with the details found in the input file, and create any net-new assets found in the input file.)

    !!! warning "Delta calculated from files"
        Be aware that the delta is calculated by comparing the input files, not the assets that currently exist in the connection within Atlan. This will only work when all management of the connection's assets is done through this package. (This also restricts the package to only allowing an input file with all of its assets in the same connection — you will receive an error if you attempt to load an input file with assets in multiple connections with this mode of delta handling.)

    - **Removal type**: how assets not found in the input file should be removed:

        - **Archive (recoverable)**: will mark each asset as soft-deleted. They will no longer appear in the UI, but can be recovered if needed.
        - **Purge (cannot be recovered)**: will permanently delete each such asset. They will no longer appear in the UI, and there is no way to recover them.

=== "Incremental"

    Will only create and update any assets that appear in the input file. Any assets that exist within the cube in Atlan, but are no longer in the input file provided, will be left as-is in Atlan.

Fail on errors¶

(optional) Default: false whether an invalid value in a field should cause the import to fail (Yes) or log a warning, skip that value, and proceed (No).

What it does¶

The package performs the following steps:

The crawler uses the client-id, client-secret and refresh token and obtains a token from the token endpoint URL
Next the crawler gets all "exposed" assets from the asset URL of SAP datasphere
Every object returned from above step contains an asset along with the metadata URL. The crawler gets the metadata of each asset and stores it localy for further processing
After asset extraction is completed, the crawler transforms and creates a csv
The crawler invokes the Relational asset builder with the csv created in the previous step