OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standard

--- description: OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standard - Enforces strict bi-directional correlation between continuous CPU/memory profile samples (pprof/OTel profiling signal) and active OpenTelemetry distributed trace spans using span-context labels. Guarantees zero-overhead trace exemplar binding and precise call-tree attribution down to microsecond-level latency regressions across distributed services. globs: src/**/*.ts, src/telemetry/**/*.ts, src/observability/**/*.ts, src/middleware/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-profiling-exemplar-correlation-standard" --- # OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Decoupled Wall-Clock Profilers and Uncorrelated Span Context Leading to Orphaned Call Stacks ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standards. You mandate that any distributed trace span entering latency-critical or resource-intensive blocks explicitly propagates its SpanContext (trace_id and span_id) into the runtime's profiling sample labels. Profiles collected without trace context or spans exported without exemplar bindings create diagnostic disconnects and are considered production failures. Ensure asynchronous context preservation, non-blocking label detachment, and strict conformance to the OpenTelemetry Profiling Data Model. ## Strict Guidelines - Engineers MUST ALWAYS attach current SpanContext (`trace_id` and `span_id`) as labels to CPU and memory profiler sample frames using async-safe scope guards. - Engineers MUST NOT invoke blocking thread-local or synchronous global profiler mutation APIs that pollute concurrent fiber or event-loop execution paths. - Services MUST configure OpenTelemetry metric exemplars to retain the exact span ID associated with profiling spikes exceeding defined SLO latency thresholds. - Engineers MUST ALWAYS clean up or detach profiler labels in a `finally` block or lifecycle hook to prevent context leakage across recycled asynchronous tasks. - Code SHOULD leverage standardized continuous profiling agents (e.g., Pyroscope, Parca, or native OTel Profiling SDKs) that natively consume `labels[span_id]` and `labels[trace_id]`. ## Bad vs Good Implementations ### Correlating Trace Span Context to CPU Profiling via Pyroscope/OTel Labels #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The profiler tag is set globally across the process without binding to the active OpenTelemetry span context. In concurrent execution, tags leak between requests, and trace exemplars cannot link CPU call trees directly to specific trace or span IDs. ```typescript import { trace } from '@opentelemetry/api'; import Pyroscope from '@pyroscope/nodejs'; const tracer = trace.getTracer('workload-tracer'); export async function processBatch(batchId: string, items: Buffer[]): Promise<void> { return tracer.startActiveSpan('processBatch', async (span) => { // ANTI-PATTERN: Profiler runs globally without trace context correlation. // Microsecond latency regressions cannot be linked back to trace_id or span_id. Pyroscope.setTag('batchId', batchId); for (const item of items) { // CPU intensive operation cryptoTransform(item); } span.end(); }); } function cryptoTransform(data: Buffer): void { // Processing } ``` #### Production Standard (Correct) > **Standard**: SpanContext is retrieved directly from the active span and bound via Pyroscope.wrapLabels scoped to the async execution tree. Profiler sample frames are tagged with both trace_id and span_id, enabling direct exemplar-to-profile navigation in APM dashboards. ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; import Pyroscope from '@pyroscope/nodejs'; const tracer = trace.getTracer('workload-tracer'); export async function processBatch(batchId: string, items: Buffer[]): Promise<void> { return tracer.startActiveSpan('processBatch', async (span) => { const spanContext = span.spanContext(); // Invariant: Bind trace_id and span_id as scoped labels for the continuous profiler const profileLabels = { trace_id: spanContext.traceId, span_id: spanContext.spanId, batch_id: batchId, }; try { // Scoped execution ensures labels are attached only during this async call stack await Pyroscope.wrapLabels(profileLabels, async () => { for (const item of items) { cryptoTransform(item); } }); span.setStatus({ code: SpanStatusCode.OK }); } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message }); throw error; } finally { span.end(); } }); } function cryptoTransform(data: Buffer): void { // Processing } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standard

OpenTelemetry Continuous Profiling & Trace Exemplar Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict bi-directional correlation between continuous CPU/memory profile samples (pprof/OTel profiling signal) and active OpenTelemetry distributed trace spans using span-context labels. Guarantees zero-overhead trace exemplar binding and precise call-tree attribution down to microsecond-level latency regressions across distributed services. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Decoupled Wall-Clock Profilers and Uncorrelated Span Context Leading to Orphaned Call Stacks Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Decoupled Wall-Clock Profilers and Uncorrelated Span Context Leading to Orphaned Call Stacks	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS attach current SpanContext (`trace_id` and `span_id`) as labels to CPU and memory profiler sample frames using async-safe scope guards.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.