OpenTelemetry Metric Exemplar Trace Coupling Standard

--- description: OpenTelemetry Metric Exemplar Trace Coupling Standard - Guarantees deterministic, exact temporal alignment between high-percentile (p99/p99.9) latency histogram bucket anomalies and high-cardinality distributed trace contexts. Enforces continuous active span context propagation into metric recording invocations to eliminate decoupling between metric aggregations and root-cause trace exemplars. globs: src/**/*.ts, src/**/*.js, packages/**/*.ts, lib/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-metric-exemplar-trace-coupling-standard" --- # OpenTelemetry Metric Exemplar Trace Coupling Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Decoupled Metric Recording Without Active Context Leading to Missing Exemplars During Tail Latency Anomalies ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing the OpenTelemetry Metric Exemplar Trace Coupling Standard. You must mandate that all metric recordings—especially for latency histograms and execution durations—are executed strictly within the lexical or explicit scope of the active distributed trace context (`context.active()`). Prevent detached metric recording patterns, unlinked asynchronous completion callbacks, and missing exemplar filter configurations that break trace-to-metric correlation in Prometheus/OpenTelemetry Collector pipelines. ## Strict Guidelines - Engineers MUST ALWAYS pass the active Context (`context.active()`) explicitly or execute within `context.with()` when invoking metric instrument recordings (`histogram.record(value, attributes, ctx)`). - Engineers MUST NOT record histogram measurements in detached async contexts or deferred process ticks where trace context propagation is lost. - Histogram bucket boundaries SHOULD be explicitly configured to guarantee distinct statistical separation around p95, p99, and p99.9 operational thresholds. - Services MUST configure the OpenTelemetry MeterProvider with an `AlwaysSampleExemplarFilter` or `TraceBasedExemplarFilter` to bind active SpanContext (TraceId, SpanId) to bucket measurements. - Components NEVER drop trace attributes before metric emission when measuring cross-service network or transaction boundaries. ## Bad vs Good Implementations ### Histogram Recording in Asynchronous Handlers #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The histogram.record() call omits the explicit Context instance. If async local storage is lost or unlinked across promise chains, the OpenTelemetry ExemplarFilter fails to extract TraceId/SpanId, rendering high-latency bucket spikes unreproducible via traces. ```typescript import { metrics } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service'); const requestDuration = meter.createHistogram('http_request_duration_seconds'); export async function handlePaymentRequest(req: Request, handler: () => Promise<Response>): Promise<Response> { const start = performance.now(); try { return await handler(); } finally { const durationSeconds = (performance.now() - start) / 1000; // ANTI-PATTERN: Metric recorded without context; TraceContext is omitted, // causing the exemplar reservoir to drop TraceId and SpanId during p99 spikes. requestDuration.record(durationSeconds, { route: req.url, method: req.method }); } } ``` #### Production Standard (Correct) > **Standard**: Passing context.active() directly into histogram.record() explicitly couples the trace context to the metric exemplar reservoir, guaranteeing deterministic trace-to-metric correlation even during extreme p99.9 latency anomalies. ```typescript import { metrics, trace, context, Context } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service'); const tracer = trace.getTracer('billing-service'); const requestDuration = meter.createHistogram('http_request_duration_seconds', { description: 'Measures end-to-end request duration with exemplars', unit: 's', }); export async function handlePaymentRequest(req: Request, handler: () => Promise<Response>): Promise<Response> { return tracer.startActiveSpan('handlePaymentRequest', async (span) => { const start = performance.now(); const currentCtx: Context = context.active(); try { return await handler(); } catch (error) { span.recordException(error as Error); throw error; } finally { const durationSeconds = (performance.now() - start) / 1000; // PRODUCTION FIX: Explicitly pass currentCtx containing active span context. // Guarantees exemplar reservoir attaches TraceId/SpanId to anomalous bucket values. requestDuration.record( durationSeconds, { route: req.url, method: req.method }, currentCtx ); span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Exemplar Trace Coupling Standard

OpenTelemetry Metric Exemplar Trace Coupling Standard defines an industrial-grade behavioral standard for devops engineering. Guarantees deterministic, exact temporal alignment between high-percentile (p99/p99.9) latency histogram bucket anomalies and high-cardinality distributed trace contexts. Enforces continuous active span context propagation into metric recording invocations to eliminate decoupling between metric aggregations and root-cause trace exemplars. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Decoupled Metric Recording Without Active Context Leading to Missing Exemplars During Tail Latency Anomalies Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Decoupled Metric Recording Without Active Context Leading to Missing Exemplars During Tail Latency Anomalies	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS pass the active Context (`context.active()`) explicitly or execute within `context.with()` when invoking metric instrument recordings (`histogram.record(value, attributes, ctx)`).

Engineers MUST ALWAYS pass the active Context (`context.active()`) explicitly or execute within `context.with()` when invoking metric instrument recordings (`histogram.record(value, attributes, ctx)`).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.