OpenTelemetry eBPF Continuous Profiling Span Correlation Standard

--- description: OpenTelemetry eBPF Continuous Profiling Span Correlation Standard - Enforces low-overhead kernel and user-space continuous CPU profiling samples to deterministically stitch to W3C distributed trace contexts. Mandates frame pointer preservation, non-blocking thread-local trace context publication for eBPF maps, and standardized OpenTelemetry pprof span attributes. globs: **/Dockerfile*, **/*.dockerfile, **/daemonset*.{yaml,yml}, **/otel-*.{yaml,yml}, src/**/*.{go,c,cpp,rs,ts} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-ebpf-profiling-span-correlation-standard" --- # OpenTelemetry eBPF Continuous Profiling Span Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Stochastic Wall-Clock Profiling Correlation with Stripped Frame Pointers and Uncorrelated eBPF Samples ## System Constraints You are a Principal Observability and Systems Architect enforcing OpenTelemetry eBPF Continuous Profiling standards. You must mandate frame pointer preservation, explicit thread-local context sharing for eBPF probes, and strict adherence to the OTel Profiling Data Model (OTel Pprof / OEP 238). Stop agents from using stochastic timestamp heuristic matching, stripping debug symbols without debuginfod integration, or attaching high-overhead uprobes to application hot loops. ## Strict Guidelines - Engineers MUST compile all production native binaries and runtimes with frame pointers preserved (`-fno-omit-frame-pointer` and `-mno-omit-leaf-frame-pointer`) to enable zero-overhead eBPF FP stack unwinding. - Engineers MUST NOT rely on synchronous wall-clock timestamp heuristic matching to bind eBPF samples to spans; trace context correlation MUST be established via deterministic thread-local memory registers, task-local storage (bpf_task_storage), or USDT markers. - Engineers MUST export continuous profiling data using the OpenTelemetry Profiling Data Model, strictly populating `trace_id` and `span_id` fields directly inside stack sample locations or sample labels. - Services MUST run eBPF host agents (e.g., OTel Profiling Agent, Grafana Beyla, Parca) with host PID namespace access (`hostPID: true`) and bounded Linux capabilities (`CAP_BPF`, `CAP_PERFMON`, `CAP_SYS_PTRACE`), avoiding unconstrained `privileged: true`. - Components NEVER strip debugging symbols (`strip -s`) in container image stages without offloading unstripped binaries to an authenticated debuginfod server or symbol cache accessible by the profiling pipeline. - Code SHOULD leverage async-signal-safe thread-local storage (TLS) writes to update active `trace_id` and `span_id` during span activation, avoiding blocking system calls or synchronous uprobes on high-frequency execution paths. ## Bad vs Good Implementations ### Continuous Profiling Build Flags and Thread-Local Trace Context Exposure #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Stripping symbols (-s -w) and omitting frame pointers prevents eBPF agents from walking user-space call stacks without heavy DWARF unwinding. The binary also fails to export active span context to thread storage, rendering CPU samples completely uncorrelated to traces. ```dockerfile # Stripping frame pointers and debug information eliminates eBPF stack walking capability FROM golang:1.23-alpine AS builder WORKDIR /app COPY . . # Anti-pattern: -s -w strips all DWARF/symbol tables, default flags omit frame pointers RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app/server ./cmd/server FROM alpine:3.20 WORKDIR /app COPY --from=builder /app/server . USER nonroot ENTRYPOINT ["/app/server"] ``` #### Production Standard (Correct) > **Standard**: Compiles with frame pointers preserved and debugging symbols retained. This allows the eBPF profiler to collect stack samples with minimal CPU overhead (<1%) while resolving user-space function offsets and binding active spans. ```dockerfile # Retain frame pointers and compile-time symbols for low-overhead eBPF kernel stack walking FROM golang:1.23-alpine AS builder WORKDIR /app COPY . . # Preserve frame pointers (-toolexec not needed in modern Go; CGO flags ensure C dependencies retain FPs) ENV CGO_ENABLED=1 ENV CGO_CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g" # In Go 1.x, preserve symbols and register unwind info; do not pass -s or -w RUN go build -tags="ebpf_profiling" -o /app/server ./cmd/server FROM alpine:3.20 WORKDIR /app COPY --from=builder /app/server . # Run with non-root user while relying on Pod-level CAP_PERFMON/CAP_BPF daemonset USER 10001 ENTRYPOINT ["/app/server"] ``` ### Deterministic OTel Span Correlation via Thread-Local Context Registration (Go/CGO/USDT) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Timestamps alone cannot disambiguate multiple concurrent tasks or high-concurrency event loops on multi-core systems, resulting in unstitched profiles or incorrect trace attribution. ```go // Anti-pattern: Profile samples and spans are collected in isolation, // relying on collector-side timestamp range estimation to associate traces. func ProcessRequest(ctx context.Context, tr trace.Tracer) { ctx, span := tr.Start(ctx, "ProcessRequest") defer span.End() // CPU intensive task runs here; // The eBPF profiler sampling perf events on this OS thread has zero // visibility into span.SpanContext().TraceID or SpanID. ExecuteComputation() } ``` #### Production Standard (Correct) > **Standard**: Locks the execution unit to the OS thread and synchronizes the W3C trace and span IDs directly to an addressable thread-local state or eBPF task storage map. When the eBPF sampling interrupt fires, it extracts the traceparent context directly from the current task struct. ```go // Invariant: Explicitly register the active trace_id and span_id into // OS thread-local storage or USDT marker readable by the eBPF sampling program. package profiling import ( "context" "runtime" "go.opentelemetry.io/otel/trace" ) // SetThreadTraceContext exposes active trace identifiers to the eBPF profiler map // via runtime thread-pinning and TLS synchronization. func InstrumentWithProfileSpan(ctx context.Context, tr trace.Tracer, name string, fn func(context.Context)) { ctx, span := tr.Start(ctx, name) defer span.End() sc := span.SpanContext() if sc.IsValid() { runtime.LockOSThread() defer runtime.UnlockOSThread() traceID := sc.TraceID() spanID := sc.SpanID() // Write trace_id and span_id into thread-accessible memory register or eBPF task map SyncTraceStateToThread(traceID[:], spanID[:]) defer ClearThreadTraceState() } fn(ctx) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Span Correlation Standard

OpenTelemetry eBPF Continuous Profiling Span Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces low-overhead kernel and user-space continuous CPU profiling samples to deterministically stitch to W3C distributed trace contexts. Mandates frame pointer preservation, non-blocking thread-local trace context publication for eBPF maps, and standardized OpenTelemetry pprof span attributes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Stochastic Wall-Clock Profiling Correlation with Stripped Frame Pointers and Uncorrelated eBPF Samples Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Stochastic Wall-Clock Profiling Correlation with Stripped Frame Pointers and Uncorrelated eBPF Samples	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST compile all production native binaries and runtimes with frame pointers preserved (`-fno-omit-frame-pointer` and `-mno-omit-leaf-frame-pointer`) to enable zero-overhead eBPF FP stack unwinding.

Engineers MUST compile all production native binaries and runtimes with frame pointers preserved (`-fno-omit-frame-pointer` and `-mno-omit-leaf-frame-pointer`) to enable zero-overhead eBPF FP stack unwinding.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.