Data Observability

Data Observability

Find the run behind a failed or slow data job. Connect execution results, inputs and outputs with data lineage to investigate what changed, trace the source and identify downstream dependencies to check.

Job · Run · Dataset · Lineage

Move from a failing job to the evidence behind it

Start with a change in runtime or success rate, inspect a specific run, then follow the datasets it read and produced. Keep execution evidence and dependencies together as you investigate.

Find failing and slow jobs in a shared operational view

Review job counts, completed and unsuccessful runs, success rates and execution trends. Use run frequency, duration and job rankings to decide which task to investigate first.

Explore the distribution of data platforms and integrations so data engineers and platform teams start from the same execution records.
View documentation

Open the run that failed, not just the job definition

Inspect a run’s status, start and end times, duration, errors and execution environment. View the inputs it read and the outputs it produced to understand what happened during that execution.

Compare engine, version and parameter changes across runs, then return to the associated execution record to check whether a configuration change coincided with the problem.
View documentation

Trace the source and see which downstream jobs to check

Start from a dataset or job and follow collected upstream and downstream relationships. Adjust the depth, group by platform, database or schema, and switch between the lineage map and list.

Use dependencies to narrow the investigation, then inspect related run records to determine whether downstream objects were actually affected.
View documentation

Put each dataset’s structure and execution history in context

Search collected data assets by platform and type. Inspect schemas, last-observed times, related jobs and run history, then continue into the dependencies around that asset.

Bring metadata from Airflow, Spark, dbt or the Python Client through OpenLineage to connect Dataset, Job and Run records. Available fields and relationships reflect the information supplied by each producer.
View documentation

Frequently asked questions

How is Data Observability different from Database Monitoring?

Data Observability follows data-processing jobs, their runs, input and output datasets, and lineage. Database Monitoring investigates database instances, SQL queries, sessions and resource performance. They provide context for different parts of the data stack.

How do I connect an existing data-processing system?

Send RunEvent, JobEvent and DatasetEvent metadata through OpenLineage. The documentation includes setup for Airflow, Spark, dbt and the Python Client. The attributes and lineage available depend on the producer, integration and reported facets.

Does a downstream dependency mean the data is affected?

A dependency identifies an object to investigate. Check the related job’s execution status, inputs, outputs and run records to confirm whether it has actually been affected.

How is Data Observability billed?

The Commercial edition counts new valid job runs each day, deduplicated by runId within the same workspace. The Enterprise edition includes Data Observability in its existing total write volume and engine data storage charges. See the pricing page for retention options and rates.

Learn more

Related capabilities and documentation

OpenLineage setup guideConnect Airflow, Spark, dbt and the Python Client. Database MonitoringInvestigate database instances, queries and resource performance. Data Observability pricingReview run-based metering and retention options.

Follow the run. Understand the data path.

Next steps

Get started > View documentation > View pricing >