OpenTelemetry eBPF Continuous Profiling & Trace Context Correlation

--- description: OpenTelemetry eBPF Continuous Profiling & Trace Context Correlation - Enforces deterministic linkage between OpenTelemetry distributed trace contexts and continuous eBPF CPU/memory profiling agents (such as OTel Profiling Agent / Beyla). Prevents blind spots in production microbenchmarks by ensuring compiled native runtimes preserve frame pointers, avoid stripped symbol tables, and inject W3C trace context into thread/task-local storage accessible by eBPF ring buffers without incurring sampling overhead. globs: Dockerfile*, **/infra/**, **/deployments/**/*.yaml, **/k8s/**/*.yaml, src/**/*.{ts,js,go,rs,cpp,c} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-continuous-profiling-standard" --- # OpenTelemetry eBPF Continuous Profiling & Trace Context Correlation > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Uncorrelated Profiling Samples and Stripped Native Binaries Blocking eBPF Stack Unwinding ## System Constraints You are a Principal Observability & Systems Engineer enforcing the OpenTelemetry eBPF Continuous CPU & Memory Profiling Standard. Mandate that all production service build configurations and deployment manifests retain debug unwinding capabilities (-fno-omit-frame-pointer, non-stripped DWARF/symbol tables or split debuginfod services) and strictly align W3C trace IDs with eBPF thread/coroutine state. Forbid ad-hoc high-overhead user-space profilers, stripped release targets without symbol servers, and unlinked span/profile pipelines. ## Strict Guidelines - Services MUST compile native binaries and runtimes with -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer to ensure low-overhead, zero-crash eBPF stack unwinding via BPF_PROG_TYPE_PERF_EVENT. - Containers running under eBPF profiling MUST preserve host pid namespace visibility (hostPID: false, but shareProcessNamespace: true in PodSpec if multi-container) or mount debugfs/tracefs with standard bounded security contexts (CAP_BPF, CAP_PERFMON). - Engineers MUST configure the OpenTelemetry collector profiling receiver (otelcol-contrib) to extract trace_id and span_id labels from eBPF uprobe/kprobe data feeds (via OTel Beyla or Grafana Pyroscope/Phlare eBPF agent). - Production deployment manifests MUST NOT omit debug symbol handling; split symbols into separate debuginfod services or symbol servers rather than using 'strip --strip-all' on deployed binaries. - Node.js, JVM, or Go runtimes running under eBPF profiling MUST enable native stack exposure (e.g., Node.js '--perf-prof', Go debug build symbols intact, JVM '-XX:+PreserveFramePointer'). - Engineers MUST NOT introduce synchronous userland wall-clock profiling libraries into critical path request handlers that compete with eBPF sampling intervals. ## Bad vs Good Implementations ### Configuring Container Build and Deployment for eBPF Trace-Profile Correlation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Compiling with '-ldflags="-s -w"' eliminates the symbol table and DWARF metadata, while omitting frame pointers prevents eBPF kernel probes (perf_event) from traversing runtime call frames. The profiler receives raw addresses that cannot be symbolicated or correlated to active W3C spans. ```dockerfile # Stripping symbols and omitting frame pointers breaks eBPF stack unwinding FROM golang:1.22-alpine AS builder WORKDIR /app COPY . . RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o server ./cmd/server FROM alpine:3.19 WORKDIR /app COPY --from=builder /app/server . # Runtime has no frame pointer retention, symbols are stripped, and no tracing integration ENTRYPOINT ["/app/server"] ``` #### Production Standard (Correct) > **Standard**: The binary maintains proper stack frames and debugging symbols. When sampled by the host eBPF continuous profiling daemon (e.g., OpenTelemetry Beyla or Parca Agent), the unwinder accurately maps CPU cycles and memory allocations directly to OpenTelemetry span IDs without runtime overhead. ```dockerfile # Retain frame pointers and compile with symbols for eBPF unwinder consumption FROM golang:1.22-bookworm AS builder WORKDIR /app COPY . . # Compile with frame pointers explicitly preserved and dwarf enabled RUN go build -gcflags="all=-N -l" -o server ./cmd/server FROM debian:bookworm-slim WORKDIR /app COPY --from=builder /app/server . # Expose process symbols and ensure eBPF agent can read W3C context injected in thread context ENV OTEL_SERVICE_NAME="checkout-api" ENV OTEL_EBPF_PROFILING_ENABLED="true" ENTRYPOINT ["/app/server"] ``` ### Correlating Trace Span Context with Continuous Profiler Sample Attributes #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Taking synchronous heap snapshots inside user-space application code introduces massive latency spikes (stop-the-world garbage collection) and decouples runtime metrics from the asynchronous eBPF sampling engine. ```typescript // Manual, high-overhead synchronous memory snapshotting within request handler import { trace } from '@opentelemetry/api'; import * as v8 from 'v8'; export async function handleTransaction(req: Request) { const tracer = trace.getTracer('checkout'); return tracer.startActiveSpan('processPayment', async (span) => { // ANTI-PATTERN: Synchronous user-space heap snapshot causes 100ms+ stop-the-world pauses const snapshot = v8.writeHeapSnapshot(); console.log(`Heap snapshot written: ${snapshot}`); const result = await executePayment(req); span.end(); return result; }); } ``` #### Production Standard (Correct) > **Standard**: Execution flow retains trace identity in async-context without invoking intrusive profilers. The eBPF kernel agent samples CPU execution and native allocations in background perf buffers, reading trace correlation IDs out-of-band with near-zero latency impact. ```typescript // Passive context propagation allowing continuous eBPF agent to sample zero-overhead import { trace } from '@opentelemetry/api'; import { AsyncLocalStorage } from 'async_hooks'; export const profileContext = new AsyncLocalStorage<{ traceId: string; spanId: string }>(); export async function handleTransaction(req: Request) { const tracer = trace.getTracer('checkout'); return tracer.startActiveSpan('processPayment', async (span) => { const spanContext = span.spanContext(); // Propagate trace IDs down execution context where eBPF ustack unwinder can correlate them return profileContext.run({ traceId: spanContext.traceId, spanId: spanContext.spanId }, async () => { try { return await executePayment(req); } finally { span.end(); } }); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling & Trace Context Correlation

OpenTelemetry eBPF Continuous Profiling & Trace Context Correlation defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic linkage between OpenTelemetry distributed trace contexts and continuous eBPF CPU/memory profiling agents (such as OTel Profiling Agent / Beyla). Prevents blind spots in production microbenchmarks by ensuring compiled native runtimes preserve frame pointers, avoid stripped symbol tables, and inject W3C trace context into thread/task-local storage accessible by eBPF ring buffers without incurring sampling overhead. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Uncorrelated Profiling Samples and Stripped Native Binaries Blocking eBPF Stack Unwinding Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Uncorrelated Profiling Samples and Stripped Native Binaries Blocking eBPF Stack Unwinding	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST compile native binaries and runtimes with -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer to ensure low-overhead, zero-crash eBPF stack unwinding via BPF_PROG_TYPE_PERF_EVENT.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.