eBPF for Production Observability: Deep System Tracing Without Code Changes

Most observability stories follow the same arc: instrument your code, ship a tracing library, add spans everywhere, pipe it into a collector, and eventually you have a dashboard that shows you what happened after the incident was over. The instrumentation cost is real. You pay it in lines of code, in library dependencies, in the coordination required to add a span to a service owned by a different team.

eBPF cuts that dependency. It lets you attach programs directly to the Linux kernel and observe system behavior without touching application code. No redeployments. No library coordination. No sidecar injection required. The tradeoff is that you are writing (or consuming) programs that run in kernel space, which means kernel version requirements, verification overhead, and a different failure mode than userland tooling.

This article covers how eBPF actually works, practical use cases for production observability, the tooling ecosystem, and the production considerations that determine whether eBPF is the right tool for a given problem.

---

What eBPF Is and How It Works

eBPF (extended Berkeley Packet Filter) is a subsystem in the Linux kernel that allows sandboxed programs to run in kernel space in response to events. The key constraint is the verifier: before any eBPF program runs, the kernel statically analyzes it to confirm it cannot crash the system, loop forever, or access memory it should not. Programs that pass verification are JIT-compiled to native instructions and attached to hooks.

The hook types that matter for observability:

• kprobes / kretprobes: Attach to any kernel function entry or return. Used for syscall tracing, network stack inspection, file system monitoring.
• uprobes / uretprobes: Attach to user-space function entry or return in any running process. Used for application-layer tracing without modifying the binary.
• tracepoints: Stable kernel instrumentation points that survive kernel version changes. More reliable than kprobes for long-term programs.
• XDP (eXpress Data Path): Attach to the network driver layer before the kernel networking stack. Used for extremely low-latency packet processing and filtering.
• perf events: Attach to hardware performance counters and software events. Used for CPU profiling, cache miss analysis, branch misprediction counting.

Data flows from eBPF programs to user space through maps: shared data structures (hash maps, arrays, ring buffers, per-CPU maps) that both the kernel program and user-space consumer can read and write. A typical observability pipeline looks like this:

Kernel Event (syscall, kprobe, tracepoint)
         |
         v
   eBPF Program (verifier-checked, JIT-compiled)
         |
         v
    eBPF Map (ring buffer, hash map, perf event array)
         |
         v
  User-Space Daemon (Go, Rust, C) reads + processes
         |
         v
  Metrics / Traces / Logs (Prometheus, OTLP, stdout)

The verifier is what makes this safe. It enforces bounded loops (since kernel 5.3, with loop limits), no unbounded memory access, and stack size limits (512 bytes per frame). Programs that violate these constraints are rejected at load time, not at runtime.

---

HTTP Request Tracing Without Sidecar Proxies

Service meshes like Istio solve HTTP tracing by injecting an Envoy sidecar into every pod. The sidecar intercepts traffic, adds trace headers, and forwards to a collector. That works, but it adds 10-50ms per request in latency, burns CPU on every pod, and requires mTLS to be worth the deployment cost.

eBPF can trace HTTP requests at the socket level in the kernel, reading request/response data from the read() and write() syscalls without any proxy. Tools like Pixie and Grafana Beyla use this approach.

Here is the shape of a bpftrace program that captures HTTP request paths at the syscall level:

# Trace write() calls on port 80/8080 and print the first 200 bytes
bpftrace -e '
tracepoint:syscalls:sys_enter_write
/pid == $1/
{
  printf("PID %d wrote: %s\n", pid, str(args->buf, 200));
}
' -- <target-pid>

That is a toy example. Production tools go further: Pixie’s PEM (Pixie Edge Module) deploys eBPF programs that reconstruct HTTP/1.1 and HTTP/2 request/response pairs, including headers and body, with latency measurements, without touching application code or requiring any changes to TLS termination (because it hooks after TLS decryption in the process, using uprobes against OpenSSL/BoringSSL’s SSL_write and SSL_read).

The TLS case is the credibility detail here. Most sidecar-based approaches cannot inspect encrypted traffic unless they terminate TLS themselves. eBPF uprobes on the TLS library functions run in the process’s address space, after decryption and before encryption, so they see plaintext.

---

TCP Connection Monitoring

Understanding TCP connection state at scale is hard with traditional tooling. netstat and ss give you a point-in-time snapshot. Getting historical data, retransmit rates, or connection duration distributions requires either kernel module development or sampling with perf.

eBPF makes this straightforward. The tcp_v4_connect, tcp_close, tcp_retransmit_skb, and inet_csk_accept kernel functions cover the TCP lifecycle. A bpftrace one-liner to watch new TCP connections:

# Watch all new outbound TCP connections with PID, comm, source, and destination
bpftrace -e '
kprobe:tcp_connect
{
  $sk = (struct sock *)arg0;
  printf("%-6d %-16s %-16s -> %s:%d\n",
    pid,
    comm,
    ntop(AF_INET, $sk->__sk_common.skc_rcv_saddr),
    ntop(AF_INET, $sk->__sk_common.skc_daddr),
    $sk->__sk_common.skc_dport >> 8 | ($sk->__sk_common.skc_dport & 0xff) << 8
  );
}
'

For retransmit monitoring, tcp_retransmit_skb is the hook point. Cilium’s Hubble uses these hooks to provide network flow visibility at the pod level in Kubernetes, including drop reasons, policy verdicts, and TCP state transitions, without any application-level instrumentation.

---

DNS Query Logging

DNS is the first place to look when services suddenly cannot reach their dependencies. The problem is that DNS queries are cheap and ephemeral: standard logging misses them, packet capture is expensive, and most service mesh tools only trace application-layer traffic, not DNS resolution.

eBPF can hook into the kernel’s DNS resolver at the socket level. A bpftrace approach that hooks sendmsg for UDP packets on port 53:

# Log DNS queries from any process (hooks sendmsg on UDP port 53)
bpftrace -e '
kprobe:udp_sendmsg
{
  $sk = (struct sock *)arg0;
  $dport = $sk->__sk_common.skc_dport;
  $dport = $dport >> 8 | ($dport & 0xff) << 8;
  if ($dport == 53) {
    printf("pid=%-6d comm=%-16s querying DNS\n", pid, comm);
  }
}
'

That only captures the fact of a query, not the query content. Capturing the actual DNS payload requires reading the iov buffer from the msghdr struct, which is possible but requires more careful memory access patterns in the eBPF program.

Pixie handles this transparently: its table dns_events surfaces query name, response code, latency, and the PID of the originating process, collected via eBPF without any DNS library hooks.

---

CPU and Memory Profiling

Continuous profiling without code changes is where eBPF offers its clearest advantage over traditional APM. Traditional CPU profiling requires either language-specific agents (Java agents, Python decorators) or binary instrumentation. eBPF uses perf_event with PERF_TYPE_SOFTWARE / PERF_COUNT_SW_CPU_CLOCK to sample the call stack at a fixed frequency across all running processes.

A bpftrace program that samples CPU call stacks at 99 Hz:

# Sample call stacks across all processes at 99Hz for 10 seconds
bpftrace -e '
profile:hz:99
{
  @[comm, ustack, kstack] = count();
}
END
{
  print(@);
}
' -c 10

The output can be fed to flamegraph generators to produce per-process flame graphs without any application modification. Pixie and tools like Parca and Polar Signals use this approach to provide always-on, low-overhead CPU profiling across all workloads in a cluster.

The overhead is measurable but acceptable: sampling at 99 Hz adds roughly 1-2% CPU overhead across the fleet. Sampling at 1000 Hz pushes that to 5-10%. The 99 Hz default is not arbitrary; it avoids harmonic interference with system timers that run at multiples of 100 Hz.

---

The Tooling Ecosystem

bpftrace: Ad-Hoc Debugging

bpftrace is the scripting layer for eBPF. It compiles a high-level language (similar to awk) into eBPF programs and handles map management and output formatting. It is the right tool for one-off investigations: “what is this process writing to disk?” or “which kernel functions are slowest during this incident?”

It requires kernel 4.9+ for most features, 5.x for some tracepoints and ring buffer support. It reads BTF (BPF Type Format) debug information if available (kernel 5.2+ or via vmlinux), which enables type-aware field access in kernel structs without hardcoding offsets.

The one-liner format is where bpftrace shines:

# File opens by process name and filename
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# Syscall latency histogram for a specific PID
bpftrace -e '
tracepoint:raw_syscalls:sys_enter /pid == $1/ { @start[tid] = nsecs; }
tracepoint:raw_syscalls:sys_exit  /pid == $1 && @start[tid]/ {
  @latency = hist(nsecs - @start[tid]);
  delete(@start[tid]);
}
' -- <pid>

# Top kernel functions by on-CPU time
bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

Cilium: Kubernetes Networking and Observability

Cilium replaces kube-proxy and the standard CNI with an eBPF data plane. Every packet routing decision, policy enforcement, and load balancing operation happens in eBPF programs attached to network interfaces, bypassing the iptables chain entirely.

Hubble is the observability layer on top of Cilium. It provides:

• Per-flow network visibility at the pod level (source pod, destination pod, protocol, verdict, drop reason)
• HTTP/gRPC/Kafka protocol-aware metrics without application instrumentation
• DNS query tracking per pod
• TCP metrics (retransmits, connection state transitions) per service pair

Hubble exposes a gRPC API and a UI. The hubble observe CLI gives you a real-time stream of network flows across the cluster:

# Watch all flows to a specific pod
hubble observe --pod frontend/frontend-abc123 --follow

# Watch dropped packets cluster-wide
hubble observe --verdict DROPPED --follow

# HTTP requests to a service with status code filtering
hubble observe --http-status-code 5xx --follow

Cilium requires Linux kernel 4.9+ for the basic data path, 5.2+ for full feature parity including socket-level load balancing and bandwidth management.

Pixie: Auto-Instrumented Application Monitoring

Pixie deploys a per-node agent (PEM) that loads eBPF programs automatically based on detected workloads. It captures HTTP/1.1, HTTP/2, gRPC, MySQL, PostgreSQL, Redis, Kafka, and DNS traffic by hooking into the kernel and into TLS library functions in running processes.

The differentiator is that Pixie stores data locally on the node (in a memory-mapped ring buffer, default 2GB per node) and processes queries in-cluster using a columnar query engine. Data never leaves the cluster unless you explicitly export it. Query latency is low because you are querying data that is already local.

PxL (Pixie Language) is a pandas-like query language for this data:

import px

# HTTP request latency by service, p50/p95/p99
df = px.DataFrame(table='http_events', start_time='-5m')
df.latency_ms = df.latency / 1e6
df = df.groupby(['service']).agg(
  p50=('latency_ms', px.quantiles(0.5)),
  p95=('latency_ms', px.quantiles(0.95)),
  p99=('latency_ms', px.quantiles(0.99)),
  error_rate=('response_status', lambda r: px.mean(r >= 400)),
)
px.display(df)

Pixie requires Linux kernel 4.14+ and kernel headers or BTF debug info.

Grafana Beyla: Auto-Instrumentation as a Sidecar

Beyla takes a different packaging approach: it runs as a sidecar container (or DaemonSet) and automatically instruments processes using eBPF uprobes on Go, Java, Python, Node.js, Ruby, and Rust runtimes, plus kernel-level HTTP tracing as a fallback.

It emits OpenTelemetry traces and Prometheus metrics, so it slots into existing observability pipelines without requiring a new data store. Configuration is minimal:

# beyla-config.yaml
open_port: 8080          # instrument any process listening on this port
otel_traces_export:
  endpoint: http://otel-collector:4318
prometheus:
  port: 9090
  path: /metrics

Beyla does not require modifying application manifests beyond adding the sidecar. It is the lowest-friction path to getting distributed traces for services where you cannot or do not want to add an SDK.

---

Tradeoffs

Dimension	eBPF-Based	Sidecar Proxy (Envoy/Istio)	SDK Instrumentation
Code changes required	None	None (injection)	Yes
Latency overhead	<1ms	10-50ms per hop	Negligible
CPU overhead	1-3% continuous	5-15% continuous	2-5% per traced request
Kernel version requirement	4.9+ (basic) to 5.10+ (full)	None	None
TLS visibility	Yes (uprobe on TLS lib)	Only if terminating	Full
Language coverage	Runtime-agnostic	Protocol-agnostic	Language-specific
Deployment complexity	DaemonSet or node agent	Sidecar per pod	Library integration
Security surface	Kernel-level access required	Network-level only	User-space only
Managed K8s support	Varies by provider	Broad	Universal

---

Security Implications of Running Kernel Probes

eBPF programs run in kernel space. Loading them requires CAP_BPF (kernel 5.8+) or CAP_SYS_ADMIN on older kernels. In a Kubernetes context, this typically means the node agent or DaemonSet needs privileged access or specific capabilities.

The attack surface is real:

• A compromised eBPF program loader can read arbitrary kernel memory (subject to verifier limits, but the verifier is complex software with a history of bugs).
• eBPF programs can intercept any system call, including reads of sensitive files or environment variables.
• A bug in the verifier or JIT compiler could allow privilege escalation.

Mitigations worth applying in production:

• Run eBPF agents as dedicated service accounts with the minimum required capabilities (prefer CAP_BPF + CAP_PERFMON over CAP_SYS_ADMIN).
• Use seccomp profiles to restrict the agent’s own syscalls.
• Prefer pre-compiled, signed eBPF programs (CO-RE programs compiled against BTF) over programs that load raw bytecode at runtime.
• Enable kernel lockdown mode where it does not conflict with your observability requirements (lockdown restricts some eBPF attach points).
• Audit the eBPF programs loaded by any third-party tool before deploying to production.

The security tradeoff is why eBPF-based tools often require more approval cycles than SDK-based instrumentation. That overhead is appropriate: kernel access is a meaningful privilege boundary.

---

Kernel Version Requirements

This is the hard gate. Many articles gloss over it, but the kernel version requirement eliminates eBPF for a non-trivial fraction of production environments:

• Amazon Linux 2 ships with kernel 5.10 (patched with BPF backports). Most eBPF tools work.
• Amazon Linux 2023 ships with kernel 6.1. Full support.
• GKE supports kernel 5.10+ on Container-Optimized OS. Cilium is a supported option.
• EKS with AL2 or AL2023 supports eBPF workloads. CAP_BPF available from EKS 1.26+.
• AKS with Ubuntu node pools supports eBPF on kernels 5.15+. Cilium is a supported CNI.
• Older on-prem deployments running RHEL 7 or Ubuntu 18.04 ship kernels in the 4.x range and will have limited eBPF support. Several features (ring buffers, BTF, CO-RE) require 5.2+.

BTF (BPF Type Format) availability matters separately from kernel version. Distributions that do not ship vmlinux with BTF embedded require separate kernel headers or a vmlinux.h header generated from the target kernel. CO-RE (Compile Once, Run Everywhere) programs depend on BTF to work across kernel versions without recompilation.

Run bpftool btf list on a target node to verify BTF availability before committing to an eBPF-heavy tool.

---

Production Considerations for Kubernetes Deployments

DaemonSet or node agent, not namespace-scoped. eBPF programs attach at the kernel level, which means per-node deployment is the correct model. Namespace-scoped sidecars do not work for kernel tracing because the kernel is shared across all namespaces on a node. Tools like Pixie and Beyla deploy as DaemonSets for exactly this reason.

Monitor eBPF program health explicitly. The bpftool prog list command shows all loaded eBPF programs on a node, their attach points, run counts, and run times. Expose this as a metric in your node agent. A program that loads but never fires (zero run count) indicates an incorrect attach point or a version mismatch. A program with very high run time per call indicates a map contention or processing bottleneck.

# List all loaded eBPF programs with stats
bpftool prog list --pretty

# Show detailed stats for a specific program
bpftool prog show id <prog-id> --pretty

Map memory is per-node, not unlimited. eBPF maps are kernel memory. A hash map with 1,000,000 entries at 64 bytes per entry consumes 64MB of kernel memory. Multiply by the number of eBPF programs running simultaneously and by the number of nodes. Pixie’s default 2GB per-node memory budget is a hard limit on a node with 8GB RAM.

Tail calls and program chaining add complexity. When eBPF programs need to do more work than fits in a single program (the 1 million instruction limit is generous but not infinite), they use tail calls. Tail call chains have a maximum depth of 33. Debug these separately from the application-level data they collect.

Upgrade sequencing matters. When upgrading Cilium or any eBPF-based CNI, the data plane changes cannot be applied rolling-update style the way normal Kubernetes workloads are. Cilium requires a specific upgrade procedure: upgrade the operator first, then drain and upgrade nodes one at a time, or use in-place upgrade if the version supports it. Read the upgrade guide for the specific tool; do not assume Helm upgrade handles it correctly.

---

When eBPF Is Overkill vs When It Is Essential

eBPF is overkill when:

You have three services and a monolith. The instrumentation cost of adding OpenTelemetry SDK is one afternoon of work. The operational overhead of deploying and maintaining a DaemonSet with privileged node access exceeds the instrumentation cost by an order of magnitude.

You are running on managed Kubernetes with an older node OS that does not reliably support BTF. Debugging kernel version incompatibilities is a significant time sink that delivers no user-facing value.

You are tracing application logic (business transactions, feature flag evaluations, database query patterns). eBPF sees system calls and kernel events; it does not understand your domain model. SDK instrumentation is still the right tool for application-layer traces.

eBPF is essential when:

You need to understand what a third-party black-box service is doing at the network or syscall level without access to its source code. eBPF is the only option that does not require source access or binary rewriting.

You are debugging a latency problem that spans kernel and user space: disk I/O scheduling, TCP buffer sizing, interrupt coalescing. Traditional APM tools have no visibility here.

You need protocol-level observability across polyglot services without coordinating library upgrades across teams. Pixie seeing HTTP/2 traffic from a Go service, a Java service, and a Node service with identical metadata granularity, with no SDK changes, is a genuine operational advantage.

You are running a service mesh and paying the 10-50ms sidecar proxy overhead on every request. eBPF-based networking (Cilium) eliminates that overhead while providing equivalent or better observability.

---

Closing Insight

eBPF does not replace instrumentation. It replaces the assumption that instrumentation must be built into the application. The kernel sees everything: system calls, network packets, scheduler decisions, memory allocation paths. eBPF programs give you structured access to that data stream with low overhead and without touching the applications generating it.

The practical adoption path is not “replace everything with eBPF.” It is: deploy a node-level eBPF tool (Pixie for in-cluster debugging, Beyla for traces without SDK overhead, Cilium if you are already evaluating CNI options) alongside existing instrumentation. Use it to fill the gaps where SDK instrumentation cannot reach. Reserve bpftrace for ad-hoc kernel-level debugging during incidents.

The kernel version requirement is the real constraint to evaluate first. If your nodes are running 5.10+, most of the ecosystem works. If you are on 4.x, the subset of features available narrows significantly and the operational cost of working around limitations often exceeds the benefit.