Kensu Documentation

⌘K
Getting started with the Kensu Community Edition
Marketing campaign
Financial data report
Getting credentials
Recipe: Observe Your First Pipeline
Agents: getting started
Python
PySpark
Scala Spark
Databricks Notebook
Agent Listing
Docs powered by archbee 
5min

Financial data report

Context

Based on some sample data of an asset management company, we have to execute a data pipeline to prepare financial reports.

The financial reports include an inventory of key assets, including stocks. Those reports are used by Risk Officers to make decisions on the portfolio.

Description of the pipeline

The pipeline uses Python programs.

Every month, the pipeline runs:

  1. data_ingestion.py reads data on all the stocks in the portfolio, joins the data frames, and stores the results in the master data. It uses pandas and Kensu-py.
  2. reporting.py prepared by the Reporting Officer, extracts stock data and creates a new column, Intraday_delta. That computes the daily return for a single stock. This program runs with pandas, and you can also try it with pyspark.

Here is the flow:

Representation of the Financial data report pipeline including data observability
Representation of the Financial data report pipeline including data observability


Program Steps

In this example, after having proceeded with the Installation and configuration , you will:

  • Create a monitoring rule in Kensuembedded in a Python pipeline. This includes:
    • Data sources metadata
    • Technical data lineage
    • Data profiling (metrics)
  • Add data monitoring rules:
    • Create a monitoring rule in Kensu
    • Create a monitoring rule programmatically
  • Use the technical data lineage to Find the root cause with Kensuof data issues flagged by a validation rule.

Start with the Installation and configuration



Updated 10 May 2022
Did this page help you?
Yes
No
UP NEXT
Installation and configuration
Docs powered by archbee 
TABLE OF CONTENTS
Context
Description of the pipeline
Program Steps