OpenTelemetry Continuous Profiling Span Correlation & Thread Registry Standard

Enforces zero-overhead binding between OpenTelemetry distributed trace spans and kernel-level continuous profiler samples (eBPF CPU and off-CPU/lock contention). Requires asynchronous OS thread/goroutine registration to expose active TraceID and SpanID via stable thread-local memory buffers readable by kernel probes without intercepting the fast-path execution. Eliminates runtime lock contention and synchronous user-space frame unwinding while preserving sub-millisecond trace-to-flamegraph lineage.

--- description: OpenTelemetry Continuous Profiling Span Correlation & Thread Registry Standard - Enforces zero-overhead binding between OpenTelemetry distributed trace spans and kernel-level continuous profiler samples (eBPF CPU and off-CPU/lock contention). Requires asynchronous OS thread/goroutine registration to expose active TraceID and SpanID via stable thread-local memory buffers readable by kernel probes without intercepting the fast-path execution. Eliminates runtime lock contention and synchronous user-space frame unwinding while preserving sub-millisecond trace-to-flamegraph lineage. globs: src/**/*.{ts,js,go,rs,cpp,c}, instrumentation/**/*.{ts,js,go,rs,cpp,c}, infra/profiling/**/*.{yaml,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-continuous-profiling-span-correlation-standard" --- # OpenTelemetry Continuous Profiling Span Correlation & Thread Registry Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Synchronous User-Space Profiler Interception or Disconnected High-Frequency Wall-Clock Sampling ## System Constraints You are a Principal Observability and Runtime Systems Architect enforcing the OpenTelemetry Continuous Profiling Span Correlation Standard. You must mandate that all profiling telemetry correlates deterministic SpanID and TraceID fields using out-of-band kernel probes (such as eBPF) and dedicated thread-local context registers. You must forbid synchronous CPU interrupt handling in user space, heavy allocation inside profiling hooks, and sampling routines that drop context correlation under high lock contention. ## Strict Guidelines - Engineers MUST ALWAYS expose TraceID and SpanID context via low-overhead, thread-local memory or predictable OS thread registers accessible by kernel eBPF unwinders. - Engineers MUST NOT invoke synchronous user-space stack unwinding, IPC calls, or dynamic heap allocations within the active profiling or span lifecycle path. - Instrumentation MUST maintain consistent 64-bit SpanID and 128-bit TraceID memory alignment matching OpenTelemetry Profiling Data Model (OTel Pprof / OEP-0005) specifications. - Off-CPU and lock contention metrics MUST record the blocking span context alongside trace labels to allow direct pivot from APM waterfall views to kernel wait-state flamegraphs. - Services MUST compile with frame pointers enabled (`-fno-omit-frame-pointer`) or produce validated DWARF/split-debug artifacts for asynchronous symbolization. - Components NEVER attach custom user-space signal handlers (such as SIGPROF) that collide with or supersede kernel-level BPF profiling programs. ## Bad vs Good Implementations ### Correlating Trace Spans to Kernel-Readable Thread State in Node.js/C++ Runtime Addon #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Synchronously signaling the process via SIGPROF and executing blocking file I/O inside span execution incurs massive overhead, perturbs application timings, and disrupts out-of-band eBPF continuous profilers. ```typescript import { trace, Context } from '@opentelemetry/api'; import * as fs from 'fs'; // ANTI-PATTERN: Synchronously invoking user-space profiling utilities // and performing blocking disk/IPC operations within active span boundaries. export function executeWorkload(context: Context, payload: () => void): void { const tracer = trace.getTracer('workload-tracer'); const span = tracer.startSpan('cpu-heavy-task'); try { // Intercepting execution with synchronous signal triggers causes severe latency process.kill(process.pid, 'SIGPROF'); payload(); } finally { // Writing profile artifacts directly during execution path const profileData = Buffer.from(span.spanContext().spanId); fs.appendFileSync('/tmp/profiler.log', profileData); span.end(); } } ``` #### Production Standard (Correct) > **Standard**: Updates an atomic, thread-local register with current TraceID and SpanID values. The kernel eBPF profiler samples CPU and off-CPU events asynchronously, reading this memory offset to tag flamegraphs with zero runtime overhead or heap allocations. ```typescript import { trace, Context } from '@opentelemetry/api'; // Zero-overhead native bindings interface providing a static memory slot // readable by BPF uprobes (e.g., OpenTelemetry eBPF Continuous Profiler / Parca / Pyroscope) interface NativeThreadRegistry { setSpanContext(traceIdHigh: bigint, traceIdLow: bigint, spanId: bigint): void; clearSpanContext(): void; } declare const nativeRegistry: NativeThreadRegistry; export function executeWorkload(context: Context, payload: () => void): void { const tracer = trace.getTracer('workload-tracer'); const span = tracer.startSpan('cpu-heavy-task'); const { traceId, spanId } = span.spanContext(); // Atomic write to pre-allocated thread-local memory; zero allocation, zero I/O const tId = BigInt('0x' + traceId); const sId = BigInt('0x' + spanId); nativeRegistry.setSpanContext(tId >> 64n, tId & 0xFFFFFFFFFFFFFFFFn, sId); try { payload(); } finally { nativeRegistry.clearSpanContext(); span.end(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling Span Correlation & Thread Registry Standard

OpenTelemetry Continuous Profiling Span Correlation & Thread Registry Standard defines an industrial-grade behavioral standard for devops engineering. Enforces zero-overhead binding between OpenTelemetry distributed trace spans and kernel-level continuous profiler samples (eBPF CPU and off-CPU/lock contention). Requires asynchronous OS thread/goroutine registration to expose active TraceID and SpanID via stable thread-local memory buffers readable by kernel probes without intercepting the fast-path execution. Eliminates runtime lock contention and synchronous user-space frame unwinding while preserving sub-millisecond trace-to-flamegraph lineage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Synchronous User-Space Profiler Interception or Disconnected High-Frequency Wall-Clock Sampling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Synchronous User-Space Profiler Interception or Disconnected High-Frequency Wall-Clock Sampling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS expose TraceID and SpanID context via low-overhead, thread-local memory or predictable OS thread registers accessible by kernel eBPF unwinders.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.