eBPF Continuous Profiling & Zero-Overhead L7 Telemetry Invariants

--- description: eBPF Continuous Profiling & Zero-Overhead L7 Telemetry Invariants - Enforces architectural and binary compilation invariants required for non-invasive, out-of-band eBPF continuous profiling and L7 kernel socket tracing. Guarantees frame pointer preservation, dwarf debug symbol availability, and socket buffer visibility to eliminate reliance on userspace sidecars or bytecode re-instrumentation. globs: **/Dockerfile*, **/*.mk, **/Makefile, **/*.bpf.c, **/*.bpf.h, **/CMakeLists.txt, **/*.yaml, **/*.toml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-continuous-profiling-telemetry" --- # eBPF Continuous Profiling & Zero-Overhead L7 Telemetry Invariants > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Stripping Frame Pointers and Relying on Heavyweight Userspace Envoy/Proxy Sidecars ## System Constraints You are a Principal Systems & Observability Architect enforcing eBPF continuous profiling and kernel-level zero-overhead telemetry standards. Mandate that target binaries retain architectural observability primitives (such as uncompromised frame pointers `-fno-omit-frame-pointer`) and ensure eBPF programs adhere strictly to verifier safety, BPF ring buffer throughput constraints, and CO-RE (Compile Once – Run Everywhere) portability. ## Strict Guidelines - Engineers MUST ALWAYS compile native binaries (C, C++, Go, Rust) with frame pointers retained (`-fno-omit-frame-pointer` or equivalent) to allow the eBPF kernel stack walker to traverse call stacks without dwarf unwinding overhead in the hot path. - Engineers MUST NOT strip debug symbol tables (`.symtab`) completely from production container images; separate debug symbols into detached debuginfo servers or preserve minimal symbol tables for accurate out-of-band symbolication by eBPF telemetry agents. - eBPF telemetry probes MUST use `bpf_ringbuf_output` or `bpf_ringbuf_reserve`/`bpf_ringbuf_submit` instead of legacy `bpf_perf_event_output` to minimize per-CPU memory footprints and prevent ring buffer contention under high-throughput packet processing. - Services MUST leverage socket filter (`BPF_PROG_TYPE_SOCKET_FILTER`) or sockops/sk_msg programs for L7 zero-overhead protocol parsing rather than deploying CPU-intensive userspace sidecar proxies for basic metrics collection. - Components NEVER execute unbounded loops in eBPF kernel probes; all loops MUST use constant upper bounds evaluated at compile-time to pass the eBPF kernel verifier (`#pragma unroll` or verified `bpf_loop`). ## Bad vs Good Implementations ### Compilation Flag & eBPF Telemetry Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Compiling with '-s -w' strips symbol tables and debugging info, and omitting frame pointer preservation breaks eBPF kernel-level unwinding (perf_event_open), creating broken stack traces and blind spots in continuous profiling engines like Parca or Pyroscope. ```dockerfile # Stripping frame pointers and debug symbols renders eBPF stack unwinders useless, # forcing reliance on heavyweight userspace proxy sidecars. FROM golang:1.22-alpine AS builder WORKDIR /app COPY . . RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /service . FROM alpine:3.19 COPY --from=builder /service /service ENTRYPOINT ["/service"] ``` #### Production Standard (Correct) > **Standard**: Preserves frame pointers and uncompressed DWARF sections, allowing zero-overhead eBPF programs to traverse user-space stack traces safely in the kernel without triggering page faults or sidecar proxy overhead. ```dockerfile # Retaining frame pointers and exporting symbols for eBPF continuous profiling FROM golang:1.22-alpine AS builder WORKDIR /app COPY . . # -toolexec or -gcflags preserving frame pointers; preserve minimal symbols for eBPF maps RUN CGO_ENABLED=0 go build \ -gcflags="all=-N -l -d=ssa/check_bce=on" \ -ldflags="-compressdwarf=false" \ -o /service . FROM alpine:3.19 # Ensure libc/libunwind compatibility if dynamic, retain unstripped executable COPY --from=builder /service /service ENTRYPOINT ["/service"] ``` ### CO-RE Safe eBPF L7 Ingress Packet/Socket Tracepoint #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Uses legacy perf event arrays which consume per-CPU memory, unportable memory offsets vulnerable to kernel version mismatches, and deprecated probe_read helpers failing CO-RE validation. ```c // Non-CO-RE legacy eBPF program with unbounded loop and deprecated perf event buffer #include <linux/bpf.h> #include <bpf/bpf_helpers.h> struct { __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY); } events SEC(".maps"); SEC("tracepoint/syscalls/sys_enter_write") int trace_write(struct trace_event_raw_sys_enter *ctx) { // Anti-pattern: High-overhead allocations and unsafe memory dereferences char buf[256]; bpf_probe_read(buf, sizeof(buf), (void *)ctx->args[1]); bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, buf, sizeof(buf)); return 0; } ``` #### Production Standard (Correct) > **Standard**: Adheres to modern BPF standards: uses CO-RE via vmlinux.h, employs BPF ring buffer for efficient multi-core memory sharing, and performs zero-overhead L7 payload accounting. ```c // Production BPF CO-RE implementation with modern ringbuf and bounded checks #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_core_read.h> struct event { __u32 pid; __u64 bytes; }; struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024); } telemetry_rb SEC(".maps"); SEC("tp_btf/sys_enter_write") int BPF_PROG(trace_write_enter, struct pt_regs *regs, long id) { __u64 pid_tgid = bpf_get_current_pid_tgid(); __u32 pid = pid_tgid >> 32; struct event *e = bpf_ringbuf_reserve(&telemetry_rb, sizeof(*e), 0); if (!e) { return 0; // Drop sample under backpressure without kernel panic } e->pid = pid; e->bytes = (size_t)PT_REGS_PARM3_CORE(regs); bpf_ringbuf_submit(e, 0); return 0; } char LICENSE[] SEC("license") = "Dual BSD/GPL"; ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF Continuous Profiling & Zero-Overhead L7 Telemetry Invariants

eBPF Continuous Profiling & Zero-Overhead L7 Telemetry Invariants defines an industrial-grade behavioral standard for devops engineering. Enforces architectural and binary compilation invariants required for non-invasive, out-of-band eBPF continuous profiling and L7 kernel socket tracing. Guarantees frame pointer preservation, dwarf debug symbol availability, and socket buffer visibility to eliminate reliance on userspace sidecars or bytecode re-instrumentation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Stripping Frame Pointers and Relying on Heavyweight Userspace Envoy/Proxy Sidecars Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Stripping Frame Pointers and Relying on Heavyweight Userspace Envoy/Proxy Sidecars	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS compile native binaries (C, C++, Go, Rust) with frame pointers retained (`-fno-omit-frame-pointer` or equivalent) to allow the eBPF kernel stack walker to traverse call stacks without dwarf unwinding overhead in the hot path.

Engineers MUST ALWAYS compile native binaries (C, C++, Go, Rust) with frame pointers retained (`-fno-omit-frame-pointer` or equivalent) to allow the eBPF kernel stack walker to traverse call stacks without dwarf unwinding overhead in the hot path.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.