OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard

--- description: OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard - Mandates deterministic inline exemplar linkage between high-cardinality distributed traces and latency bucket histograms in OpenTelemetry SDK 1.28+. Ensures explicit ActiveContext propagation during histogram observations with custom ExemplarFilter/ExemplarReservoir configuration for sub-second root-cause isolation. Eliminates unlinked metrics, context-blind recording, and metric-label cardinality explosion. globs: src/**/*.{ts,js,mts,cts}, lib/**/*.{ts,js,mts,cts} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-native-exemplar-trace-correlation-standard" --- # OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Recording Context-Free Metric Observations or Over-Indexing Span Context in Metric Attributes ## System Constraints You are a Principal Observability Architect enforcing the OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard. You must mandate deterministic, zero-allocation trace context association to histogram bucket observations. Forbid stuffing dynamic trace IDs, span IDs, or high-cardinality attributes into metric attributes. Enforce explicit Context passing to `histogram.record(value, attributes, context)` and verify that MeterProvider is initialized with TraceBasedExemplarFilter or an aligned exemplar reservoir. ## Strict Guidelines - Engineers MUST ALWAYS pass the active Context explicitly as the third argument to `histogram.record(value, attributes, context)` to bind exemplar traces deterministically. - Engineers MUST NOT inject trace_id, span_id, user_id, or request-scoped UUIDs into metric attributes/labels; exemplars MUST handle this high-cardinality linkage natively. - Services MUST configure the MeterProvider with an explicit ExemplarFilter (e.g., `AlwaysSampleFilter` or `TraceBasedExemplarFilter`) and aligned `ExemplarReservoir` factories. - Code SHOULD extract the active span context via `trace.getActiveSpan(context)` or `context.active()` immediately at measurement call sites to prevent asynchronous context drift. - Components NEVER record latency metrics without an active trace context when instrumenting synchronous or asynchronous HTTP/RPC request lifecycles. ## Bad vs Good Implementations ### Histogram Recording With Trace Exemplar Correlation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting the third `context` argument to `histogram.record()` prevents the SDK's ExemplarReservoir from sampling the active span for the bucket observation. Manually dumping `trace.id` into metric attributes causes severe time-series database (TSDB) cardinality explosion. ```typescript import { metrics } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service', '1.28.0'); const latencyHistogram = meter.createHistogram('http_request_duration_ms', { description: 'Measures HTTP request latency in milliseconds.', unit: 'ms', }); export async function handleRequest(req: Request): Promise<Response> { const startTime = performance.now(); try { return await processRequest(req); } finally { const duration = performance.now() - startTime; // ANTI-PATTERN: Omits active Context in record(), preventing exemplar attachment. // Injects high-cardinality traceId into metric labels, causing TSDB cardinality explosion. latencyHistogram.record(duration, { 'http.route': req.url, 'http.method': req.method, 'trace.id': req.headers.get('x-trace-id') ?? 'unknown' }); } } ``` #### Production Standard (Correct) > **Standard**: Passing `activeCtx` as the 3rd argument allows the OpenTelemetry ExemplarFilter to extract the SpanContext (`traceId`, `spanId`) and attach it as a native OpenMetrics/OTLP Exemplar to the histogram bucket without increasing metric series cardinality. ```typescript import { context, metrics, trace } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service', '1.28.0'); const latencyHistogram = meter.createHistogram('http_request_duration_ms', { description: 'Measures HTTP request latency in milliseconds.', unit: 'ms', }); export async function handleRequest(req: Request): Promise<Response> { const activeCtx = context.active(); const startTime = performance.now(); try { return await processRequest(req); } finally { const duration = performance.now() - startTime; // PRODUCTION FIX: Explicitly pass activeCtx to bind span/trace IDs as native exemplars. // Metric attributes are strictly low-cardinality enum-like keys. latencyHistogram.record( duration, { 'http.route': req.url, 'http.method': req.method, }, activeCtx ); } } ``` ### MeterProvider Initialization With Native Exemplar Filter #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Instantiating MeterProvider without specifying or verifying the ExemplarFilter and ExemplarReservoir can lead to silent drop of exemplar metadata, rendering histogram-to-trace navigation useless in Grafana and Datadog. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // ANTI-PATTERN: Default MeterProvider without explicit ExemplarFilter config // Often falls back to AlwaysOffFilter in certain runtimes, dropping all trace exemplars. export const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }), ], }); ``` #### Production Standard (Correct) > **Standard**: Explicitly declaring `exemplarFilter: ExemplarFilterType.TRACE_BASED` guarantees that any span sampled by the TracerProvider automatically qualifies its corresponding histogram bucket entry to retain exemplar trace references. ```typescript import { MeterProvider, PeriodicExportingMetricReader, ExemplarFilterType } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // PRODUCTION FIX: Explicitly configure ExemplarFilter for TraceBased sampling export const meterProvider = new MeterProvider({ exemplarFilter: ExemplarFilterType.TRACE_BASED, readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }), ], }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard

OpenTelemetry 1.28 Native Exemplar-Metric Trace Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Mandates deterministic inline exemplar linkage between high-cardinality distributed traces and latency bucket histograms in OpenTelemetry SDK 1.28+. Ensures explicit ActiveContext propagation during histogram observations with custom ExemplarFilter/ExemplarReservoir configuration for sub-second root-cause isolation. Eliminates unlinked metrics, context-blind recording, and metric-label cardinality explosion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Recording Context-Free Metric Observations or Over-Indexing Span Context in Metric Attributes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Recording Context-Free Metric Observations or Over-Indexing Span Context in Metric Attributes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS pass the active Context explicitly as the third argument to `histogram.record(value, attributes, context)` to bind exemplar traces deterministically.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.