Register Azure Data Factory Connection
In this tutorial, we will guide you through the necessary steps to effectively set up and configure Azure Data Factory for data observability. The tutorial is structured as follows:
- Setup ADF Connection using ADF SDK credentials (Estimated Task Time: 5 minutes) Here, we'll walk you through the process of setting up a connection with Azure Data Factory using a client/secret token.
- Configuration of the ADF Collector (Estimated Task Time: 2 minutes) The final step focuses on configuring the ADF collector for data observability. This involves setting up the necessary parameters and options within ADF to collect, monitor, and manage your data effectively.
Before you begin configuring the ADF collector, ensure you have the following:
- A Kensu Hub instance with the preinstalled ADF component
1. Navigate to your Kensu Hub URL https://xxxxx-hub.kensuapp.com/
2. Under the Data Application Collectors section, find the ADF collector. Click on Configure to proceed.
In the dependencies, you need to create the connection to your ADF instance.
- Within the dependencies section, locate the ADF option and click "Configure" to start setting up the connection.
2. Click the + icon to add a new connection.
3. Fill in the connection details as follows: - conn_id: Choose a unique identifier for your connection. - subscription id - tenant id - client id - secret: use the secret value, and not the secret id
Ensure to input only the necessary information as specified above, there is no need to modify the other fields.
4. Click Submit to save the new connection.
5. Click on Back to come back to the Collector home page.
1. Navigate to the Configuration panel. Within the kensu-hub-components-collector-azure-data-factory cell, click on Configure
2. Click on the + icon (1). This action will generate a new configuration file.
3. You can now edit the configuration, by adding:
- Project Names: Enter the names of the ADF pipelines you're interested in observing.
- In the specified section, choose the appropriate Kensu and ADF connections for this configuration. You can select the default Kensu connection or create your own.
4. Click on Submit All
Congratulations! You've successfully set up the collector. It will be activated shortly, and you will begin monitoring your ADF pipelines.
The collector will allow the retrieval of lineage, data sources, and activities information. To allow full data observability, it is important to create the Data Source connections, which will also retrieve the schema and metrics of the tables used in the jobs. To do so, please follow this tutorial.
