eBPF observability explained: how collection works

Learn how Linux eBPF probes attach, what privileges and kernel features they need, and how DataKit signals and OpenTelemetry instrumentation fit together.

Best practices
Illustration of system events and execution paths

An API can get slower while its application logs look normal. TCP retransmissions, connection failures, and round-trip times give you another way to narrow the search. A span from the request handler answers a different question. It can show which part of the application spent the time.

On Linux, eBPF lets a collector run programs that the kernel checks before loading. The programs observe events at selected attachment points, often with the application code and binary left as they are. eBPF grew out of classic BPF, the Berkeley Packet Filter. This guide explains how to choose a probe, interpret its signals, and use them alongside application telemetry. Linux BPF system call

eBPF observability at a glance

  • Programs run when a chosen hook fires. The kernel verifier checks constraints on control flow and memory access before loading.
  • System and network probes can collect common signals across application languages. Userspace probes still depend on the target binary, runtime, and hook. Every probe has a resource cost.
  • Use application telemetry to add operation names and business context. OpenTelemetry also includes OBI, an automatic instrumentation tool built on eBPF.

How eBPF attaches to Linux events

Linux already exposes events through facilities such as tracepoints. eBPF lets a collector filter, aggregate, or emit events as they occur. The verifier checks the program's paths, register use, and memory access. Kernels with bounded-loop support can accept loops the verifier can prove will terminate. Whether the program is JIT-compiled depends on kernel configuration and architecture. Linux verifier · Kernel change for bounded loops · Linux BPF

Hooks include system call entry and exit, kernel functions through kprobes, userspace functions through uprobes, static tracepoints, and network paths through XDP or TC. Each exposes a different context. A container relies on its node's Linux kernel, and its permissions and security policy affect what it can attach. libbpf overview · kprobes · uprobes

Where kernel signals help

  1. Keep the application unchanged where possible. For a service you cannot rebuild, choose a system or network hook if it exposes the signal you need.
  2. Compare common signals across languages. Java, Go, C++, and Rust services can use the same kernel interfaces. A userspace function probe or protocol parser still needs to match the runtime, binary symbols, or protocol.
  3. Aggregate before exporting. A program can count bytes per connection in a BPF map and let userspace read the totals periodically. Hook frequency, program work, and exported data volume all affect cost.
  4. See behavior outside the request handler. A suitable probe can expose retransmits or file I/O. When the hook sees TLS ciphertext, the connection is still visible. HTTP methods and paths need access to compatible application-layer data.

Start with one probe

  1. Check the toolchain. On an authorized Linux test host, verify kernel support and permissions. bpftool feature probe kernel reports support for program and map types and helpers. BCC provides a Python interface for developing probes. Loading and attaching may require tracing, networking, or other privileges, depending on the program and environment. bpftool feature probe
  2. Try a kprobe from the BCC tutorial. Its hello_fields.py example calls get_syscall_fnname("clone") to resolve the syscall symbol for the architecture, then attaches a kprobe. Check the symbol and observe a controlled test workload. Each record shows that execution reached the hook. Counting successful process creation takes a suitable event and accounting for threads, failed calls, and other creation paths. BCC Python tutorial
  3. Move counters into a map. Both the kernel program and userspace can access BPF maps through their respective interfaces. Start with per-connection traffic totals and periodic reads. Check capacity and lost events before expanding coverage.

The examples follow the documented behavior. For a longer exercise, try BCC, libbpf-bootstrap, or cilium/ebpf for Go.

In Guance, start by checking TCP connection statistics, RTT, or retransmits from DataKit's ebpf-net. Enable UDP and protocol collection as needed. Check the Linux node's architecture, DataKit version, eBPF, kprobe and perf support, and the collector's documented root requirement. For Kubernetes, check host mounts and permissions against the supported deployment manifest. Network statistics and tracing have separate setup. Experimental ebpf-trace must be enabled separately and send eBPF spans to DataKit or DataKit ELinker with the ebpftrace receiver enabled. Send spans that need to be joined across nodes to the same receiver. Coverage depends on the plugin, protocol, and environment. DataKit eBPF · eBPF tracing · DataKit deployment security

Use eBPF with OpenTelemetry

eBPF is a mechanism for collecting and processing events. OpenTelemetry supplies instrumentation, telemetry conventions, and transport. They overlap in tools such as OpenTelemetry OBI, which uses eBPF to collect application signals from supported Linux workloads.

Dimension eBPF collection OpenTelemetry application instrumentation
Deployment Often leaves application code unchanged. Depends on available Linux hooks and privileges. Choose SDK instrumentation or a zero-code option supported by the language and tool.
Signals System and network behavior, with protocol parsing where supported. Supported framework operations and custom business attributes, according to instrumentation and configuration.
Language coverage Kernel signals can span languages. Userspace probes may depend on runtime and symbols. SDKs and automatic instrumentation have their own language and framework coverage.
Typical uses Connection statistics, network diagnosis, runtime security. Request traces, business context, custom metrics.

Use OpenTelemetry to record operation names and business context, then check eBPF RTT, retransmits, and connection failures over the same window. Timing can point you toward a suspect. Matching individual requests takes propagated trace context or another reliable identifier. OpenTelemetry zero-code instrumentation · Context propagation

Frequently asked questions

What kernel version does eBPF require? Check the collector's program types, helpers, and hooks against the actual kernel and distribution backports. BTF describes types. CO-RE uses type information for the target kernel to adapt to layout differences. The target still needs the features the program uses. BTF · libbpf and CO-RE

How much overhead does it add? It depends on hook frequency, program logic, and output volume. Filter or aggregate high-frequency events when practical. Test with a representative load and compare collector CPU, memory, lost events, and application latency.

Can it replace an APM probe? Some eBPF tools produce application spans for supported protocols and runtimes. Add SDK, Agent, or other instrumentation where you need business attributes, method timing, or reliable cross-service context beyond that coverage.

Can eBPF support security monitoring? Tetragon uses it for runtime visibility and enforcement. Falco can collect kernel events with its modern eBPF probe and also supports a kernel-module driver. Deployment requirements depend on the selected tool and driver. Tetragon · Falco kernel events

Further reading

Documentation checked November 6, 2026.

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting