DataKit - Data collector

DataKit is Guance’s open-source data collection Agent. Deploy it on hosts or in Kubernetes to collect metrics, logs, and traces, then parse, filter, and enrich them with tags before sending them to Guance.
Collect from hosts, containers, and databases Process logs at the source with Pipeline Connect OpenTelemetry and Prometheus

Data Collection Agent

What is DataKit?

DataKit runs in your environment. It reads host and database metrics, collects container logs, and receives telemetry from applications using OpenTelemetry. In the Guance workspace, investigate by service, environment, or instance.

DataKit features Features
Infrastructure and database collection

Infrastructure and database collection

Collect host CPU, memory, and disk metrics. Connect databases such as MySQL and PostgreSQL to observe instances, queries, and resource metrics.

Kubernetes cluster collection

Kubernetes cluster collection

Collect node, pod, and container data with a DaemonSet. Use DataKit Operator to inject tracing or log collection configuration into newly created pods.

Process data before sending

Process data before sending

Use Pipeline to extract fields, convert types, drop records, or mask sensitive values so logs can be searched and aggregated by fields such as status code and duration.

Reuse existing collection pipelines

Reuse existing collection pipelines

Receive OpenTelemetry metrics, logs, and traces over OTLP or scrape Prometheus exporters. Keep using your existing instrumentation and exporters.

Deploy to suit your environment

Deploy to suit your environment

Install on hosts, Docker, or Kubernetes, or deploy offline. Host installations enable basic collectors by default. Add the integrations your stack needs.

Core Capabilities

Collect, process, and correlate your data

When a production service slows down, compare resource metrics by service and environment, then look at logs and traces from the same time window. Prepare fields and tags at collection time so the same filters work during an investigation.

Correlate telemetry

Use consistent tags such as service, env, and version. When application logs include trace_id, configure log-to-trace correlation.

Pipeline data processing

Use Pipeline to parse logs and text, extract fields, convert types, filter records, mask sensitive data, and enrich tags for more consistent search, alerting, and aggregate analysis.

Troubleshoot collection

If data does not reach the workspace, check input status in DataKit Monitor. Use collector debugging or logs to investigate collection and delivery.

Manage collection across hosts

Use DCA to check DataKit status. For hosts that support bulk configuration, update collector settings centrally and review results.

Deployment environments

Workflow

Recommended setup workflow

01

Deploy DataKit first

Choose 2.x for new installations. Verify that hosts or nodes are reporting in Guance, then enable the collectors your applications and databases need.

02

Set tags and processing rules

Use consistent service and environment tags to identify production and test data. Extract commonly queried fields and filter or mask values as needed.

03

Verify data in your workspace

Query metrics, logs, and traces by service, environment, and time. Confirm tags are consistent and fields are searchable, then save frequently used queries to a dashboard.

DataKit ecosystem support Ecosystem integrations
OpenTelemetry

Receive OpenTelemetry metrics, logs, and traces over OTLP

Supports receiving log data collected by Fluentd

Supports receiving log data collected by Fluentd

Supports receiving metrics collected by Telegraf

Supports receiving metrics collected by Telegraf

Scheck cloud-native programmable security inspection plugin

Scheck cloud-native programmable security inspection plugin

Collect and receive Prometheus metrics

Collect and receive Prometheus metrics

Supports receiving Statsd metrics

Supports receiving Statsd metrics

FAQ

Frequently Asked Questions

How do I choose between DataKit 2.x and 1.x?

2.x is the current release line, recommended for new installations and upgrades. 1.x is primarily for legacy systems. The lines use separate installation sources, so check system requirements and your installed version before upgrading.

Can DataKit be used together with OpenTelemetry, Prometheus, and Telegraf?

Yes. Use OTLP for OpenTelemetry, exporter scraping or remote write for Prometheus, and the HTTP output for Telegraf metrics. Enable the DataKit input appropriate for each source.

How does DataKit help logs and metrics perform correlation analysis?

During collection, DataKit adds consistent tags for service, environment, host, container, Pod, and version. Pipeline processes unstructured logs so metrics, logs, traces, and alerts can be correlated around the same resource.

If you already have custom systems or internal metrics, can you access them via DataKit?

Yes, you can. DataKit supports custom data collection, PythonD, open APIs, and pipeline processing, making it suitable for integrating internal business metrics, operations script results, and specialized technology stack data into Guance.