Import Matillion Circuit Breaker
The Circuit Breaker feature in Matillion allows you to control the execution of a pipeline's job based on predefined pre or post-conditions. This feature adds an extra layer of control and flexibility to your data processing workflows.
Once the Circuit Breaker detects a failure, it immediately halts the execution of the job and marks it as failed. This prevents further processing, ensuring that potentially problematic data is not propagated downstream. By making the Circuit Breaker a critical condition for the pipeline's success, you can proactively identify and respond to issues, thereby maintaining the integrity and reliability of your data processing workflows.
To install the Circuit Breaker feature in Matillion and configure the necessary environment variables, follow these steps:
- Import the Matillion job : Note: This will be updated to follow the new 'Shared job' component
- Log in to your Matillion instance.
- Navigate to the "Project" tab.
- Click on the "Manage Shared Jobs" button.
- Click on Import
- Select the .melt file located in this folder: https://public.usnek.com/n/repository/kensu-public/releases/matillion_agent/kensu_circuit_breaker%20(rev.7).melt
- Confirm by clicking on OK
- Configure Matillion environment variables:
- In the Matillion user interface, go to the "Project" tab.
- Click on "Manage Environment Variables" from the sidebar menu.
- Locate the following environment variables:
- KENSU_HUB_USER: the username for your Kensu Hub.
- KENSU_HUB_PASSWORD: the password for your Kensu Hub.
- KENSU_HUB_URL: the URL for your Kensu Hub instance, with /api/v1/
- Click on the "Add New" button to create a new environment variable.
- Enter the name of the environment variable (e.g., KENSU_HUB_USER) and its corresponding value.
- Repeat the above steps for the remaining environment variables (KENSU_HUB_PASSWORD and KENSU_HUB_URL), entering their respective values.
- Save the environment variable configurations.
In Matillion Jobs
To incorporate the Circuit Breaker functionality, you need to add a specific Matillion activity within your pipeline. This activity acts as a container for the logic that determines whether the job should proceed or be halted.
Evaluating Pre and Post Conditions
The Circuit Breaker evaluates both pre and post conditions to determine the job's execution status.
- Pre Conditions: These conditions check for any issues or problems in the upstream application or data sources. If any pre conditions fail, indicating a problem in the data supply chain, the Circuit Breaker will fail.
- Post Conditions: These conditions monitor the application and detect any reported issues or tickets. If a post condition fails, indicating an issue in the application's state, the Circuit Breaker will fail.
