Collectors
A collector runs outside of the data tools and therefore, to create data observations, it will apply several strategies:
- Retrieve prepared information from data tools' interfaces (e.g., API)
- Consume and interprete raw information generated by the data tools (e.g., logs)
- Trigger processes in the data tools (e.g., sql metrics queries)
Here is a high-level view of those strategies:
Consequently, a collector generates data observations automatically for all applications running in the data tools. This removes the responsibility from the users developing the data application or using the data tool to generate data observations.
The information leveraged by collectors is generally produced after the data tools have fully executed the data applications. Hence, the data observations generated by collectors are often desynchronized from the data application runtime, and circuit breakers are not available
Ref: See also Fundamentals of Data Observability, chapter 6
Available collectors are:
