Kensu Documentation

⌘K
Getting started with the Kensu Community Edition
Marketing campaign
Financial data report
Getting credentials
Recipe: Observe Your First Pipeline
Agents: getting started
Python
PySpark
Scala Spark
Databricks Notebook
Agent Listing
Docs powered by archbee 
8min

Configure the PySpark agent

Modify the Conf.ini

Modify ./kensu-spark-example/conf.ini. Add your Ingestion Token where it says ingestion_token. Put the ingestion_url. You can get those tokens from Getting Credentials.

ini
|

Code before modification

This is what the code looks like before we add Kensu Spark. It creates a regular Spark DataFrame by merging 3 csv files and saves it in a Parquet file.

Python
|

👨‍💻 Modify the program to use Kensu-Spark

To include Kensu in your program, follow these steps:

1️⃣ Modify the libraries imports to import Kensu modules

Note that we import the regular Spark SQL libraries. We also import kensu.pyspark.

Python
|

2️⃣ Init Kensu in your code

Here we create an instance of Kensu. We include the jar file that we passed to spark-submit.

Python
|



3️⃣ Send Metadata to Kensu

As with regular Spark, the DataFrames in the code are Spark DataFrames. The code uses regular Spark functions to read the .csv files and joins them.

The program saves the created DataFrame as a Parquet File with df.write.mode() . This sends the metadata to Kensu, like the DataSources, Schemas, and observability metrics.

Python
|



The complete code

Here is the complete code after the modifications.

Python
|
Updated 07 Nov 2022
Did this page help you?
Yes
No
UP NEXT
Running your first PySpark Kensu program
Docs powered by archbee 
TABLE OF CONTENTS
Modify the Conf.ini
Code before modification
👨‍💻 Modify the program to use Kensu-Spark
The complete code