DataKit - Data collector

DataKit is an open-source, all-in-one data integration Agent for Linux, Windows, and macOS. It collects data from hosts, containers, middleware, traces, logs, and security inspections.
Collect data from hosts, containers, Kubernetes, middleware, databases, and networks, including logs, traces, and custom metrics Extract fields, convert types, filter records, and mask sensitive data with Pipeline before storage Compatible with ecosystem data sources such as OpenTelemetry, Prometheus, Telegraf, Fluentd, and StatsD

Data Collection Agent

What is DataKit?

DataKit is Guance’s data collection Agent. It collects metrics, logs, traces, objects, events, and custom data from hosts, containers, Kubernetes, cloud instances, and edge environments. Pipeline, consistent tags, and open protocols bring this data into your Guance workspace.

DataKit features Features
Multi-dimensional data collection

Multi-dimensional data collection

Supports collecting data such as Metrics, Logs, and Traces from various infrastructures and technology stacks, and processing these data in a structured manner.

Kubernetes cloud-native technology support

Kubernetes cloud-native technology support

Based on a cloud-native microservices architecture, it covers comprehensive data collection within the Kubernetes ecosystem.

Pipeline for observability data processing

Pipeline for observability data processing

Built-in easy-to-use data extraction and processing engine, Pipeline, is used to extract unstructured data, making querying and statistics convenient.

Broader technology stack support than Telegraf

Broader technology stack support than Telegraf

Compared to Telegraf, which can only collect time-series data, DataKit covers a much broader range of data acquisition types, with simpler data collection configurations and better data quality.

Flexible deployment, simple and easy to use

Flexible deployment, simple and easy to use

Deploy DataKit in one step with hundreds of built-in integrations. Basic data collection can start automatically during installation.

Core Capabilities

Collect, process, and correlate your data

DataKit handles collection, preprocessing, tag standardization, ecosystem compatibility, and deployment at scale, bringing observability data into a shared analysis context.

Comprehensive data collection

Deploy DataKit across hosts, containers, Kubernetes, databases, middleware, and cloud resources to collect metrics, logs, traces, objects, and events consistently, reducing discrepancies between separate collectors.

Pipeline data processing

Use Pipeline to parse logs and text, extract fields, convert types, filter records, mask sensitive data, and enrich tags for more consistent search, alerting, and aggregate analysis.

Open ecosystem compatibility

Receive data from OpenTelemetry, Prometheus, Telegraf, Fluentd, and StatsD to bring existing collection workflows into Guance for unified analysis.

Cloud-native deployment and governance

Deploy on Linux, Windows, macOS, Docker, Kubernetes, and cloud instances. Use collector election, GitOps, DataKit API, and command-line tools to reduce maintenance effort at scale.

Platforms supported by DataKit Available platforms
Guance
M3
InfluxDB
Data platform icon
Nightingale
Elastic
TAOS Data

Workflow

Recommended setup workflow

01

Deploy DataKit first

Deploy DataKit on hosts, containers, Kubernetes, or cloud instances. Start with infrastructure, critical applications, logs, and alert-related data.

02

Standardize tags and pipelines

Standardize service, environment, version, region, team, and business tags. Use Pipeline to process log fields, mask sensitive data, and improve data quality.

03

Analyze your data together in Guance

Correlate metrics, logs, traces, RUM, events, and alerts around the same resource to investigate an issue from its symptoms to its root cause.

DataKit ecosystem support Ecosystem integrations
DataFlux Func programmable data processing development platform

DataFlux Func programmable data processing development platform

Supports receiving log data collected by Fluentd

Supports receiving log data collected by Fluentd

Supports receiving metrics collected by Telegraf

Supports receiving metrics collected by Telegraf

Scheck cloud-native programmable security inspection plugin

Scheck cloud-native programmable security inspection plugin

Collect and receive Prometheus metrics

Collect and receive Prometheus metrics

Supports receiving Statsd metrics

Supports receiving Statsd metrics

FAQ

Frequently Asked Questions

What is DataKit?

DataKit is Guance’s open-source, all-in-one data collection Agent for hosts, containers, Kubernetes, logs, traces, databases, middleware, networks, cloud resources, and custom business data.

Can DataKit be used together with OpenTelemetry, Prometheus, and Telegraf?

Yes. DataKit collects data directly and can also receive data from OpenTelemetry, Prometheus, Telegraf, Fluentd, and StatsD, bringing existing collection workflows into Guance for unified analysis.

How does DataKit help logs and metrics perform correlation analysis?

During collection, DataKit adds consistent tags for service, environment, host, container, Pod, and version. Pipeline processes unstructured logs so metrics, logs, traces, and alerts can be correlated around the same resource.

If you already have custom systems or internal metrics, can you access them via DataKit?

Yes, you can. DataKit supports custom data collection, PythonD, open APIs, and pipeline processing, making it suitable for integrating internal business metrics, operations script results, and specialized technology stack data into Guance.