OpenTelemetry Continuous Profiling & Trace-Span Exemplar Fusion Standard

--- description: OpenTelemetry Continuous Profiling & Trace-Span Exemplar Fusion Standard - Enforces deterministic correlation between distributed W3C trace contexts and low-level kernel profiling signals (eBPF CPU cycles, off-CPU lock contention, L3 cache misses). Requires bidirectional linkage between continuous profiling frame-sets and OpenTelemetry span exemplars to eliminate latency tail diagnostic opacity. globs: src/telemetry/**/*.{ts,js,go,rs,py}, src/instrumentation/**/*.{ts,js,go,rs,py}, src/infrastructure/profiler/**/*.{ts,js,go,rs,py}, src/middleware/tracing/**/*.{ts,js,go,rs,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-span-exemplar-fusion-standard" --- # OpenTelemetry Continuous Profiling & Trace-Span Exemplar Fusion Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Uncorrelated Continuous Profiling and Siloed Distributed Tracing Without Kernel-Level Exemplar Fusion ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Continuous Profiling and Trace-Span Exemplar Fusion. You must mandate that every trace span context propagates thread-level execution states to continuous eBPF sampling profilers, and that all tail latency outliers (>P99) attach flamegraph profile exemplars and kernel event counters directly to W3C TraceContext spans. ## Strict Guidelines - Services MUST inject the active OpenTelemetry TraceId and SpanId into thread-local storage or eBPF BPF_MAP_TYPE_HASH maps upon entering any trace span boundary. - Engineers MUST configure continuous profiling agents (e.g., Pyroscope, Parca, or custom eBPF probes) to sample off-CPU lock waits, futex transitions, and hardware PMU cache-miss events alongside standard CPU sampling. - Tracing middleware MUST link continuous profiling sample IDs or pprof profiles as OpenTelemetry Span Exemplars or Span Links when execution durations exceed the P95/P99 latency threshold. - Engineers MUST NOT rely on unindexed wall-clock timestamps to manually correlate flamegraphs with span durations. - Components NEVER drop thread identity or native OS TID mappings when transitioning across async/await event-loop tasks in managed runtimes (Node.js, Go, Tokio). - Telemetry bridges MUST scrub sensitive kernel-level registers and stack parameters from symbolicated pprof exemplars before exporting to OTLP collector endpoints. ## Bad vs Good Implementations ### Correlating Trace Span Context with Kernel Profiler Context & Attaching Profile Exemplars #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The span runs asynchronously without exposing its TraceId/SpanId to the native OS thread or local continuous profiling agent. If a P99 tail occurs due to a kernel futex lock or CPU cache thrashing, engineers cannot isolate the precise flamegraph matching this span. ```typescript import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('payment-service'); export async function processCriticalTransaction(txId: string): Promise<void> { return tracer.startActiveSpan('processCriticalTransaction', async (span) => { try { // ANTI-PATTERN: Heavy computational or lock-heavy task executed with no // synchronization to thread/process-level eBPF continuous profilers. // Profiler captures CPU spikes/futex blocks globally, but traces cannot correlate // specific span tails to kernel mutex contention or CPU cache misses. await executeHeavyWork(txId); } finally { span.end(); } }); } ``` #### Production Standard (Correct) > **Standard**: Explicitly binds the W3C TraceId and SpanId to the eBPF kernel tracking map via nativeProfilerBridge. When execution latency exceeds the P99 budget, it captures kernel metrics (L3 misses, futex wait time) and attaches the profile exemplar ID directly into the span attributes. ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; import { nativeProfilerBridge } from '../infrastructure/profiler/ebpfBridge'; const tracer = trace.getTracer('payment-service'); const TAIL_LATENCY_THRESHOLD_MS = 250; export async function processCriticalTransaction(txId: string): Promise<void> { return tracer.startActiveSpan('processCriticalTransaction', async (span) => { const spanContext = span.spanContext(); const startTime = performance.now(); // 1. Synchronize W3C context to eBPF TLS / BPF_MAP for native symbol correlation nativeProfilerBridge.registerThreadTraceContext( spanContext.traceId, spanContext.spanId ); try { await executeHeavyWork(txId); } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR }); throw error; } finally { const durationMs = performance.now() - startTime; // 2. Extract profile snapshot & hardware counter exemplar if tail latency observed if (durationMs > TAIL_LATENCY_THRESHOLD_MS) { const profileSnapshot = nativeProfilerBridge.captureSpanProfileExemplar( spanContext.traceId, spanContext.spanId ); span.setAttribute('profiler.exemplar.id', profileSnapshot.profileId); span.setAttribute('profiler.cpu.cycles', profileSnapshot.hwCounters.cpuCycles); span.setAttribute('profiler.l3.misses', profileSnapshot.hwCounters.l3Misses); span.setAttribute('profiler.futex.wait_ns', profileSnapshot.offCpu.futexWaitNs); } nativeProfilerBridge.clearThreadTraceContext(); span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling & Trace-Span Exemplar Fusion Standard

OpenTelemetry Continuous Profiling & Trace-Span Exemplar Fusion Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic correlation between distributed W3C trace contexts and low-level kernel profiling signals (eBPF CPU cycles, off-CPU lock contention, L3 cache misses). Requires bidirectional linkage between continuous profiling frame-sets and OpenTelemetry span exemplars to eliminate latency tail diagnostic opacity. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Uncorrelated Continuous Profiling and Siloed Distributed Tracing Without Kernel-Level Exemplar Fusion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Uncorrelated Continuous Profiling and Siloed Distributed Tracing Without Kernel-Level Exemplar Fusion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST inject the active OpenTelemetry TraceId and SpanId into thread-local storage or eBPF BPF_MAP_TYPE_HASH maps upon entering any trace span boundary.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.