Configure Matillion Collector
In this tutorial, we will guide you through the necessary steps to effectively set up and configure Matillion for data observability. The tutorial is structured as follows:
- Setup Matillion Connection using Matillion Token (Estimated Task Time: 5 minutes) Here, we'll walk you through the process of setting up a connection with Matillion using a Matillion token.
- Configuration of the Matillion Collector (Estimated Task Time: 2 minutes) The final step focuses on configuring the Matillion collector for data observability. This involves setting up the necessary parameters and options within Matillion to collect, monitor, and manage your data effectively.
Before you begin configuring the Matillion collector, ensure you have the following:
- A Kensu Hub instance with the preinstalled Matillion component
1. Navigate to your Kensu Hub URL https://xxxxx-hub.kensuapp.com/
2. Under the Data Application Collectors section, find the Matillion collector. Click on Configure to proceed.
In the dependencies, you need to create the connection to your Matillion instance.
- Within the dependencies section, locate the Matillion option and click "Configure" to start setting up the connection.
2. Click the + icon to add a new connection.
3. Fill in the connection details as follows: - conn_id: Choose a unique identifier for your connection. - host: Enter the Matillion host URL, formatted as https://your-matillion-instance-url/rest/v1/. - login and password: Use credentials for a Matillion user account with at least read and API access rights. Ensure to input only the necessary information as specified above, there is no need to modify the other fields.
4. Click Submit to save the new connection.
5. Click on Back to come back to the Collector home page.
1. Navigate to the Configuration panel. Within the kensu-hub-components-collector-matillion cell, click on Configure
2. Click on the + icon. This action will generate a new configuration file.
3. You can now edit the configuration, by adding:
- Environment Names: Specify the names of the Matillion environments you wish to monitor.
- Project Names: Enter the names of the Matillion projects you're interested in observing.
- In the specified section, choose the appropriate Kensu and Matillion connections for this configuration. You can select the default Kensu connection or create your own.
4. Click on Submit All
Congratulations! You've successfully set up the collector. It will be activated shortly, and you will begin monitoring your Matillion jobs.
The collector will allow the retrieval of lineage, data sources, and job information. To allow full data observability, it is important to create the Data Source connections, which will also retrieve the schema and metrics of the tables used in the jobs. To do so, please follow this tutorial.
