Pipeline data processing—log parsing, cleaning, and field extraction

What problems does Pipeline data processing solve?

Use Pipeline to process raw data into queryable and associable fields

When raw log fields are messy, formatting is inconsistent, or contains sensitive information, Pipeline completes parsing, field extraction, desensitization, and standardization before data enters the platform, making subsequent queries, alerts, and association analysis more reliable.

01Is the original log hard to find? First, parse the key fieldsPipeline supports parsing, extracting, modifying, filtering, and aggregating data locally or centrally within DataKit.
02Once fields are unified, logs can be associated with metrics, links, and networksThe processed data structure is clearer, allowing for filtering, grouping, and aggregation by unified fields.
03Sensitive information cannot be stored in the database? Desensitization is applied before processing the chainPipeline can replace phone numbers, ID numbers, tokens, accounts, IPs, or business-sensitive fields before data is stored.

Is the original log hard to find? First, parse the key fields

Pipeline supports parsing, extracting, modifying, filtering, and aggregating data locally or centrally within DataKit. Teams can convert host, service, status, duration, trace_id, and other information in messages into fields or tags, making subsequent retrieval, alerting, and aggregation analysis more stable.

Book a demo View documentation

Is the original log hard to find? First, parse the key fields

Once fields are unified, logs can be associated with metrics, links, and networks

The processed data structure is clearer, allowing for filtering, grouping, and aggregation by unified fields. For example, after standardizing service names, hosts, status codes, and request paths in logs, you can continue to associate APM Traces, infrastructure metrics, BPF network logs, and alert events.

Book a demo View documentation

Sensitive information cannot be stored in the database? Desensitization is applied before processing the chain

Pipeline can replace, delete, or encrypt mobile phone numbers, ID numbers, tokens, accounts, IPs, or business-sensitive fields before data is stored. For security, compliance, and multi-team collaboration scenarios, the earlier desensitization is completed, the more controllable subsequent retrieval and sharing will be.

Book a demo View documentation

Sensitive information cannot be stored in the database? Desensitization is applied before processing the chain

Frequently asked questions

What problems does Pipeline data processing mainly solve?

Pipeline is used to parse raw logs and observation data before data is stored, completing field extraction, format conversion, filtering, desensitization, and standardization, making subsequent retrieval, alerting, aggregation, and association analysis more accurate.

What data can the Pipeline process?

Pipeline can handle various types of observation data such as logs, metrics, objects, links, and networks. Common scenarios include parsing fields such as service, host, status, duration, trace_id, and standardizing field naming and types.

Why is data anxious before warehouse entry?

Immunodaction before entry can prevent sensitive information such as phone numbers, ID numbers, tokens, and accounts from entering subsequent queries, sharing, and archiving processes, reducing data leakage and compliance risks at the source.

Related reading

A powerful log viewer to support data linkage analysis

Guance viewer's powerful query filtering and search functions help users quickly and accurately retrieve data and locate faults. The log viewer not only helps us collect and manage various log files, but also enables correlation analysis with multi-party data, providing users with more comprehensive and accurate data information to support efficient decision-making.

Product capabilities

Leverage Guance to build a unified enterprise-level log center

Guance adheres to the philosophy of "unified collection, unified processing, and unified analysis," building an efficient observation data analysis platform. This article Guance how to help enterprises quickly and effectively build unified log centers through the various processes of log collection, processing, storage, and analysis.

Best practices

Pipeline helps you easily manage massive amounts of data!

Data governance has become crucial in today's information age. As the volume of data continues to grow and diversify, organizations need to effectively manage and utilize this data to support business decisions and innovative development. Whether for data analysis or parsing, using Pipeline helps organizations automate and standardize data governance; Providing organizations with clearer and more actionable data views to strongly support data governance.

Product capabilities

Guance VS ELK: Who is the king of log monitoring?

This article provides a detailed analysis of Guance and ELK from multiple dimensions including data collection, data storage, data query, data visualization, and usage costs, providing a reference for users when making a choice.

Industry insights