OpenTelemetry eBPF Continuous Profiling & Zero-Overhead W3C Span Correlation Standard

--- description: OpenTelemetry eBPF Continuous Profiling & Zero-Overhead W3C Span Correlation Standard - Mandates deterministic kernel-level CPU and memory allocation profiling via eBPF correlated directly to active W3C distributed trace contexts (trace_id, span_id). Enforces preservation of runtime frame pointers, external DWARF symbol retention, and precise Linux kernel capabilities (CAP_PERFMON, CAP_BPF, CAP_SYS_PTRACE) to eliminate blind spots without invasive bytecode runtime overhead. globs: k8s/**/*.{yaml,yml}, deploy/**/*.{yaml,yml}, Dockerfile*, **/otel-collector*.{yaml,yml}, **/beyla*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-span-correlation-standard" --- # OpenTelemetry eBPF Continuous Profiling & Zero-Overhead W3C Span Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Span-Blind eBPF Profiling Caused by Stripped Frame Pointers and Missing Kernel Correlation Capabilities ## System Constraints You are a Principal Software Architect enforcing the OpenTelemetry eBPF Continuous Profiling & Distributed Span Correlation Standard. You must mandate zero-overhead kernel profiling tied directly to active W3C trace contexts without invasive agent runtime injection. Enforce un-stripped frame pointers, symbol accessibility, and strictly scoped kernel capabilities across all deployment manifests and build configurations. ## Strict Guidelines - Engineers MUST ALWAYS compile production binaries with frame pointers preserved (e.g., `-fno-omit-frame-pointer` in C/C++, Go defaults, Node.js `--perf-basic-prof`) to enable zero-overhead eBPF stack walking. - Engineers MUST NOT run eBPF profiler daemons with unconstrained `privileged: true`; agents MUST specify granular Linux capabilities: `CAP_PERFMON`, `CAP_BPF`, and `CAP_SYS_PTRACE`. - Deployments MUST mount `/sys/kernel/debug` and `/sys/fs/bpf` with read-only/appropriate access and share host PID namespaces (`hostPID: true`) strictly on designated profiling daemon pods. - Services MUST emit W3C distributed trace context into thread-local storage or process context readable by eBPF uprobes/kprobes to guarantee automatic trace_id and span_id correlation. - Build pipelines MUST separate and publish DWARF symbols or source maps to a centralized symbol server rather than stripping symbols completely without an unwind mechanism. - Components NEVER use invasive, heavy monkey-patching runtimes when non-invasive eBPF auto-instrumentation (e.g., OpenTelemetry Beyla or Parca) can capture golden signals and stack traces at the kernel boundary. ## Bad vs Good Implementations ### Continuous Profiling DaemonSet & Workload Context Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The workload strips frame pointers making eBPF stack walking impossible or inaccurate. The profiler DaemonSet uses blunt `privileged: true` instead of least-privilege Linux capabilities, omits `hostPID: true`, and lacks configuration to inspect active W3C trace contexts from application runtimes. ```yaml # Anti-Pattern: Unprivileged/Over-privileged daemon without trace correlation or frame pointers apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: template: spec: containers: - name: app image: order-service:latest # Binaries compiled with -fomit-frame-pointer and stripped DWARF env: - name: NODE_OPTIONS value: "--max-old-space-size=2048" --- apiVersion: apps/v1 kind: DaemonSet metadata: name: profiler-agent spec: template: spec: # Insecure brute-force privilege without host PID access for stack unwinding hostPID: false containers: - name: profiler image: otel-ebpf-profiler:latest securityContext: privileged: true ``` #### Production Standard (Correct) > **Standard**: Workloads retain stack frames enabling reliable kernel-level unwinding. The eBPF profiler operates with least-privilege capabilities (SYS_PTRACE, PERFMON, BPF) and hostPID access, binding kernel profiling stacks directly to W3C trace spans without invasive runtime modification. ```yaml # Hardened: Least-privilege eBPF profiler with deterministic W3C span correlation apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: template: spec: containers: - name: app image: order-service:latest env: # Retain runtime frame pointer and enable trace context exports - name: NODE_OPTIONS value: "--perf-basic-prof --interpreted-frames-native-stack" - name: OTEL_PROPAGATORS value: "tracecontext,baggage" --- apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: hostPID: true containers: - name: beyla-profiler image: grafana/beyla:latest securityContext: runAsUser: 0 readOnlyRootFilesystem: true capabilities: add: - SYS_PTRACE - PERFMON - BPF drop: - ALL volumeMounts: - name: bpf-fs mountPath: /sys/fs/bpf env: - name: BEYLA_PROMETHEUS_FEATURES value: "application,application_service_graph" - name: BEYLA_BPF_ENABLE_TRACE_CORRELATION value: "true" volumes: - name: bpf-fs hostPath: path: /sys/fs/bpf ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling & Zero-Overhead W3C Span Correlation Standard

OpenTelemetry eBPF Continuous Profiling & Zero-Overhead W3C Span Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Mandates deterministic kernel-level CPU and memory allocation profiling via eBPF correlated directly to active W3C distributed trace contexts (trace_id, span_id). Enforces preservation of runtime frame pointers, external DWARF symbol retention, and precise Linux kernel capabilities (CAP_PERFMON, CAP_BPF, CAP_SYS_PTRACE) to eliminate blind spots without invasive bytecode runtime overhead. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Span-Blind eBPF Profiling Caused by Stripped Frame Pointers and Missing Kernel Correlation Capabilities Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Span-Blind eBPF Profiling Caused by Stripped Frame Pointers and Missing Kernel Correlation Capabilities	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS compile production binaries with frame pointers preserved (e.g., `-fno-omit-frame-pointer` in C/C++, Go defaults, Node.js `--perf-basic-prof`) to enable zero-overhead eBPF stack walking.

Engineers MUST ALWAYS compile production binaries with frame pointers preserved (e.g., `-fno-omit-frame-pointer` in C/C++, Go defaults, Node.js `--perf-basic-prof`) to enable zero-overhead eBPF stack walking.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.