Mandatory Bi-directional OTel Exemplar Sampling and Continuous Profiling Correlation

--- description: Mandatory Bi-directional OTel Exemplar Sampling and Continuous Profiling Correlation - Enforces deterministic, bi-directional linkage between high-percentile latency metric exemplars, distributed traces, and continuous CPU profiling frames (via eBPF or runtime engine sampling). Eliminates wall-clock guesswork by guaranteeing every P99 latency outlier metric directly links to a root trace span, which synchronously propagates trace and span IDs to the profiling agent stack frames. globs: src/**/*.{ts,js,go,rs,py,java}, infrastructure/**/*.{yaml,json,tf}, telemetry/**/*.{ts,js,go,rs} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-exemplar-continuous-profiling-correlation-standard" --- # Mandatory Bi-directional OTel Exemplar Sampling and Continuous Profiling Correlation > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind Latency Spikes: Uncorrelated Metric Exemplars and Decoupled Wall-Clock Profiling ## System Constraints You are a Principal Software Architect enforcing OpenTelemetry Correlated Exemplar Sampling and Continuous Profiling invariants. You mandate that high-percentile latency measurements (P95/P99/P99.9) are strictly bound to metric exemplars with active trace and span IDs, and that application spans actively inject context into runtime continuous profilers (e.g., Pyroscope, Parca, or OTel Profiling Data Model). Disallow wall-clock cross-referencing and ensure zero blind spots during microsecond-level latency regressions. ## Strict Guidelines - Engineers MUST ALWAYS configure OpenTelemetry Metric Readers with an explicit `ExemplarFilter` (e.g., `TraceBasedExemplarFilter`) and an `ExemplarReservoir` to capture active trace context on latency histograms. - Engineers MUST synchronously bind active `trace_id` and `span_id` labels to the continuous profiler context during span execution so kernel and user-space profiling samples retain trace-level granularity. - Engineers MUST NOT rely on coarse temporal correlation (wall-clock timestamp matching) between APM traces and CPU flame graphs. - Services MUST inject continuous profiling query metadata or uniform profile identifiers (`pyroscope.profile.id` or profiling URLs) directly into the root span attributes of high-latency requests. - Components NEVER execute long-running or compute-intensive synchronous tasks outside an active profiling scope that masks the calling execution lineage. ## Bad vs Good Implementations ### Correlated Metric Exemplars and Continuous Profiler Context Injection in Node.js/TypeScript #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The MeterProvider lacks an ExemplarFilter, causing latency metric buckets to drop trace and span associations. Additionally, the continuous CPU profiler is not tagged with the current trace context, forcing engineers to manually align wall-clock graphs to debug P99 latency spikes. ```typescript import { trace } from '@opentelemetry/api'; import { MeterProvider } from '@opentelemetry/sdk-metrics'; // ANTI-PATTERN: Default meter configuration drops exemplars. // Profiling runs asynchronously without active span context. const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const histogram = meter.createHistogram('request_duration_seconds'); export async function processPayment(paymentId: string) { const tracer = trace.getTracer('billing-service'); return tracer.startActiveSpan('processPayment', async (span) => { const start = performance.now(); try { // CPU intensive computation without profiler labels await executeCryptoWork(); return { status: 'ok' }; } finally { const duration = (performance.now() - start) / 1000; // Histogram recorded without exemplar linking; no span correlation in profiler histogram.record(duration); span.end(); } }); } async function executeCryptoWork() { return new Promise((resolve) => setTimeout(resolve, 100)); } ``` #### Production Standard (Correct) > **Standard**: The MeterProvider is initialized with an ExemplarFilter and receives context.active() during recording, binding traceId and spanId to the metric exemplar. Pyroscope.wrapWithLabels dynamically assigns trace and span identifiers to profiling samples, achieving microsecond-level trace-to-profile and metric-to-trace bi-directional correlation. ```typescript import { trace, context } from '@opentelemetry/api'; import { MeterProvider, PeriodicExportingMetricReader, AlwaysSampleExemplarFilter } from '@opentelemetry/sdk-metrics'; import Pyroscope from '@pyroscope/nodejs'; // PRODUCTION STANDARD: Explicit exemplar filter retention for metric-to-trace linkage const meterProvider = new MeterProvider({ exemplarFilter: new AlwaysSampleExemplarFilter(), }); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('request_duration_seconds', { unit: 's', description: 'Execution duration with correlated exemplars', }); export async function processPayment(paymentId: string) { const tracer = trace.getTracer('billing-service'); return tracer.startActiveSpan('processPayment', async (span) => { const spanContext = span.spanContext(); const traceId = spanContext.traceId; const spanId = spanContext.spanId; const startTime = performance.now(); // Bi-directional linkage: Inject trace IDs directly into the continuous profiler execution labels return Pyroscope.wrapWithLabels( { trace_id: traceId, span_id: spanId, operation: 'processPayment' }, async () => { try { await executeCryptoWork(); return { status: 'ok' }; } finally { const durationSec = (performance.now() - startTime) / 1000; // Context passed explicitly; metrics engine captures exemplar with trace_id & span_id latencyHistogram.record(durationSec, { 'payment.id': paymentId }, context.active()); // Bi-directional linkage: Span attributes reference profiler location span.setAttribute('profiler.query', `{trace_id="${traceId}"}`); span.end(); } } ); }); } async function executeCryptoWork() { return new Promise((resolve) => setTimeout(resolve, 100)); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Mandatory Bi-directional OTel Exemplar Sampling and Continuous Profiling Correlation

Mandatory Bi-directional OTel Exemplar Sampling and Continuous Profiling Correlation defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, bi-directional linkage between high-percentile latency metric exemplars, distributed traces, and continuous CPU profiling frames (via eBPF or runtime engine sampling). Eliminates wall-clock guesswork by guaranteeing every P99 latency outlier metric directly links to a root trace span, which synchronously propagates trace and span IDs to the profiling agent stack frames. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind Latency Spikes: Uncorrelated Metric Exemplars and Decoupled Wall-Clock Profiling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind Latency Spikes: Uncorrelated Metric Exemplars and Decoupled Wall-Clock Profiling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure OpenTelemetry Metric Readers with an explicit `ExemplarFilter` (e.g., `TraceBasedExemplarFilter`) and an `ExemplarReservoir` to capture active trace context on latency histograms.

Engineers MUST ALWAYS configure OpenTelemetry Metric Readers with an explicit `ExemplarFilter` (e.g., `TraceBasedExemplarFilter`) and an `ExemplarReservoir` to capture active trace context on latency histograms.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.