OpenTelemetry eBPF Continuous Profiling and Async Symbolication Standard

--- description: OpenTelemetry eBPF Continuous Profiling and Async Symbolication Standard - Enforces low-overhead eBPF stack sampling, in-kernel W3C trace context extraction, and asynchronous deferred symbolication via Build-ID caches. Mandates sub-one-percent CPU overhead and zero-allocation ring-buffer streaming into OTLP profiling pipelines under enterprise loads. globs: collector/**/*.{yaml,yml,go}, ebpf/**/*.{c,go,rs}, deploy/**/*.{yaml,yml}, src/profiling/**/*.{go,rs,ts} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-ebpf-profiling-symbolication-standard" --- # OpenTelemetry eBPF Continuous Profiling and Async Symbolication Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Synchronous Stack Symbolization and Userspace Context Bouncing on Collection Paths ## System Constraints You are a Principal Observability and Systems Architect enforcing the OpenTelemetry eBPF Continuous Profiling Symbolication Pipeline Standard. Continuous profiling must operate with strictly less than 1% CPU and memory overhead, correlate kernel and user-space stacks to W3C trace/span identifiers in-kernel, and isolate symbolication to an asynchronous out-of-band pipeline. ## Strict Guidelines - Engineers MUST ALWAYS sample eBPF stack events at non-harmonic, prime-numbered frequencies (e.g., 19 Hz, 49 Hz, or 97 Hz) to prevent lockstep phase alignment with OS scheduler intervals and thread tick loops. - Engineers MUST extract W3C traceparent context (TraceId and SpanId) directly from thread-local storage (TLS) or task struct context within the eBPF hook, writing it atomically alongside stack identifiers into the BPF ring buffer. - Engineers MUST NOT perform symbol table resolution, DWARF evaluation, or network-backed debuginfod queries on the data collection hot path or within the host collector agent. - Services MUST emit raw, compact instruction pointers accompanied by executable Build-IDs (GNU build-id) to downstream asynchronous symbolication workers via OTLP Profiling format. - Components NEVER drop eBPF ring buffer events silently; all buffer saturation events MUST increment dedicated kernel-counter telemetry (`perf_event_lost` or `ringbuf_dropped_samples_total`) and report backpressure. ## Bad vs Good Implementations ### Asynchronous Symbolication Pipeline with W3C Context Mapping #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Resolving symbols synchronously using binary utilities during sample processing blocks ingestion threads, spikes collector CPU far beyond the 1% overhead limit, and risks pipeline starvation. Fetching trace context over userspace IPC loses synchronization with the exact moment the CPU stack was captured. ```typescript import { CollectorPayload, StackFrame } from '@opentelemetry/profiling-types'; // ANTI-PATTERN: Synchronous in-band symbolication and trace hydration on collection path export async function processProfileSample(rawSample: CollectorPayload): Promise<void> { for (const frame of rawSample.stackPointers) { // Synchronous filesystem or network debuginfod resolution causes latency spikes // and forces agent CPU usage above the 1% production ceiling under heavy load. const symbol = executeSyncAddr2Line(rawSample.binaryPath, frame.instructionPointer); frame.functionName = symbol.functionName; frame.sourceFile = symbol.file; frame.lineNumber = symbol.line; } // Polling userspace HTTP client for trace context instead of using eBPF metadata const activeSpan = await fetchTraceContextFromHttp(rawSample.threadId); rawSample.traceId = activeSpan.traceId; rawSample.spanId = activeSpan.spanId; await sendToOtlpEndpoint(rawSample); } ``` #### Production Standard (Correct) > **Standard**: The capture process retains sub-1% CPU overhead by keeping instruction pointers unresolved and attaching W3C trace identifiers extracted directly at the kernel probe boundary. Expensive symbol resolution is queued for an isolated out-of-band symbolication worker using GNU Build-IDs. ```typescript import { RawProfileSample, OtlpProfileRecord, SymbolicationBatch } from '@opentelemetry/profiling-types'; // PRODUCTION FIX: Preserve raw frame pointers + build-id with in-kernel trace context; defer symbolication export function ingestEbpfRingBufferSample( sample: RawProfileSample, batchQueue: SymbolicationBatch ): OtlpProfileRecord { // Invariant 1: W3C Trace and Span IDs are captured atomically in-kernel via eBPF TLS mapping const otlpRecord: OtlpProfileRecord = { timestampUnixNano: sample.timestampNs, traceId: sample.bpfTraceContext.traceId, // 16-byte hex from BPF event struct spanId: sample.bpfTraceContext.spanId, // 8-byte hex from BPF event struct mapping: { buildId: sample.gnuBuildId, fileOffset: sample.mappingOffset, }, locations: sample.instructionPointers.map((ip) => ({ address: ip, mappingIndex: 0, })), values: [sample.cpuTimeNanos], }; // Invariant 2: Symbolication is fully deferred to asynchronous, non-blocking background queue batchQueue.enqueueUnresolvedProfile(otlpRecord); return otlpRecord; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling and Async Symbolication Standard

OpenTelemetry eBPF Continuous Profiling and Async Symbolication Standard defines an industrial-grade behavioral standard for devops engineering. Enforces low-overhead eBPF stack sampling, in-kernel W3C trace context extraction, and asynchronous deferred symbolication via Build-ID caches. Mandates sub-one-percent CPU overhead and zero-allocation ring-buffer streaming into OTLP profiling pipelines under enterprise loads. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Synchronous Stack Symbolization and Userspace Context Bouncing on Collection Paths Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Synchronous Stack Symbolization and Userspace Context Bouncing on Collection Paths	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS sample eBPF stack events at non-harmonic, prime-numbered frequencies (e.g., 19 Hz, 49 Hz, or 97 Hz) to prevent lockstep phase alignment with OS scheduler intervals and thread tick loops.

Engineers MUST ALWAYS sample eBPF stack events at non-harmonic, prime-numbered frequencies (e.g., 19 Hz, 49 Hz, or 97 Hz) to prevent lockstep phase alignment with OS scheduler intervals and thread tick loops.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.