Datasources

Datasources are a reference to a certain cluster and catalog, wherein one or multiple schema’s can be chosen (if none are given, all schema’s of the catalog will be loaded). Furthermore, a schedule can be given to sync the objects to ADQ (to start creating Profifling Criteria).

YAML Example

# Example of a Datasource in YAML format
name: IT-Bronze
enabled: true
schedule: 0 0 0 0 0
type: databricks
config:
    host: 123
    cluster_id: 123
    token: somethingsomething
    catalog: sdp_bronze_dev
    schemas:
        - schema1 
        - schema2
        - schema3

YAML Reference

name: string # name of the datasource
enabled: bool # if datasource enabled or not
schedule: string # schedule based on CRON
type: databricks # only supported type at this time is databricks
config:
    host: string # name of the host
    cluster_id: string # clusterid of the 
    token: string # token used to make connection to databricks
    catalog: string # name of the catalog
    schemas: # list of strings
        - string 

End Points

At the following location the API references can be found: https://app-dq-dev-01.azurewebsites.net/docs#/. For Datasources, the following end points can be used:

Manage:
/datasources #POST for adding a new datasource
/datasources/{id} #PUT for changing an existing datasource

Find (GET):
/datasources #For returning all datasources
/datasources/{id} #For finding a specific datasource
/datasources/find_one #To find specific datasources without pagination
/datasources/find  #To find specific datasource with pagination