Configure dbt core collector with Elementary for BigQuery
In this tutorial, we'll guide you through the essential steps to effectively set up and configure the dbt collector with Elementary data for BigQuery. This involves establishing connections to the BigQuery dataset used by Elementary to log the dbt artifacts, and configuring the collector, to ensure you can monitor, collect, and analyze the operations and impacts of your dbt models.
You will need to go through 2 steps: 1. Set Up BigQuery Connection to the Elementary table (Estimated Task Time: 3 minutes) 2. Configuration of the dbt core Collector (Estimated Task Time: 1 minute)
Before you begin configuring the collector for dbt core collector with Elementary for BigQuery, ensure you have the following:
- A Google Cloud service account (for instance a JSON keyfile) with specific roles granted the necessary read permissions to access Elementary artifact table in the BigQuery Project. The minimum roles needed are:
- BigQuery Data Viewer
- BigQuery Job User
The first step consist in creating a Google Cloud connection with the account credentials mentioned in the prerequistes.
1. Navigate to your Kensu Hub URL https://xxxxx-hub.kensuapp.com/
2. Within the Data Applications Collectors, locate the dbt core one and click on configure.
3. Under the Dependencies section, click on the Configure button in the dbt_core_elementary_bigquery cell
4. Look for the + icon. Clicking on this icon will open a form where you can enter the details for your new BigQuery connection.
5. Now, you'll need to fill out the form with specific information about your BigQuery account. Here's what you'll need:
- conn_id: A unique ID for your connection.
- Keyfile JSON: The credentials JSON to connect the service account.
- Project Id: Your Google cloud project containing the elementary dataset.
6. After double-checking the information you've entered, click on the Submit button to finalize the connection. Kensu Hub will process your details and establish a connection to your Google Cloud account.
1. Come back to the collector page.
2. Navigate to the Configuration panel. Within the kensu-hub-components-collector-dbt-core cell, click on Configure
3. Click on the + icon. This action will generate a new configuration file.
4. You can now edit the configuration, by adding:
- In the specified section, choose the appropriate Kensu and dbt_core_elementary_bigquery connections for this configuration. You can select the default Kensu connection or create your own. Do not select any other connection.
- You can identify the collector by adding a specific identifier in the id field.
- schema must refer to the Google BigQuery dataset_id which contains the elementary data.
- All the other fields can be left blank.
5. Click on Submit All
Congratulations! You've successfully set up the collector. It will be activated shortly, and you will begin monitoring your dbt models.
