Integration
...
Collectors
Azure Data Factory
Register Azure Data Factory Connection
7 min
setting up azure data factory (adf) for data observability in this tutorial, we will guide you through the necessary steps to effectively set up and configure azure data factory for data observability the tutorial is structured as follows setup adf connection using adf sdk credentials (estimated task time 5 minutes) here, we'll walk you through the process of setting up a connection with azure data factory using a client/secret token configuration of the adf collector (estimated task time 2 minutes) the final step focuses on configuring the adf collector for data observability this involves setting up the necessary parameters and options within adf to collect, monitor, and manage your data effectively before you begin configuring the adf collector, ensure you have the following a kensu hub instance with the preinstalled adf component an adf client/secret login and password with the api access (see create and collect credentials docid\ y vdqh8iqv5yxj1zib8cy ) configuration navigate to the adf collector 1\ navigate to your kensu hub url https //xxxxx hub kensuapp com/ https //demo hub kensuapp com/kensu hub components plugin/#/connections 2\ under the data application collectors section, find the adf collector click on configure to proceed setting up the connection in the dependencies, you need to create the connection to your adf instance within the dependencies section, locate the adf option and click "configure" to start setting up the connection 2\ click the + icon to add a new connection 3\ fill in the connection details as follows \ conn id choose a unique identifier for your connection \ subscription id \ tenant id client id \ secret use the secret value, and not the secret id ensure to input only the necessary information as specified above, there is no need to modify the other fields 4\ click submit to save the new connection 5\ click on back to come back to the collector home page configure the collector 1\ navigate to the configuration panel within the kensu hub components collector azure data factory cell, click on configure 2\ click on the + icon (1) this action will generate a new configuration file 3\ you can now edit the configuration, by adding project names enter the names of the adf pipelines you're interested in observing in the specified section, choose the appropriate kensu and adf connections for this configuration you can select the default kensu connection or create your own 4\ click on submit all congratulations! you've successfully set up the collector it will be activated shortly, and you will begin monitoring your adf pipelines the collector will allow the retrieval of lineage, data sources, and activities information to allow full data observability, it is important to create the data source connections, which will also retrieve the schema and metrics of the tables used in the jobs to do so, please follow snowflake docid\ ojharjtq6htyg8q24uf9o