To include Kensu in your program, follow these steps:
1️⃣ Modify the libraries imports to import Kensu modules
Note that we import the regular Spark SQL libraries. We also import kensu.pyspark.
from pyspark.sql import SparkSession
from kensu.pyspark import init_kensu_spark
2️⃣ Init Kensu in your code
Here we create an instance of Kensu. We include the jar file that we passed to spark-submit.
#Add the path to the .jar to the SparkSession
spark = SparkSession.builder.appName("Example")\
3️⃣ Send Metadata to Kensu
As with regular Spark, the DataFrames in the codeare Spark DataFrames. The code uses regular Spark functions to read the .csv files and joins them.
The program saves the created DataFrame as a Parquet File with df.write.mode() . This sends the metadata to Kensu, like the DataSources, Schemas, and observability metrics.