OpenTelemetry Trace-to-Metric Exemplar Cardinality and Memory Governor

--- description: OpenTelemetry Trace-to-Metric Exemplar Cardinality and Memory Governor - Enforces strict exemplar filtering and reservoir sampling policies on OpenTelemetry metric instruments to eliminate high-churn context attachments under burst conditions. Prevents downstream telemetry pipeline saturation (Prometheus, Mimir, Datadog) caused by unconstrained trace contextualization on high-frequency histograms and counters. Mandates explicit exemplar filter strategies coupled with bounded-capacity reservoirs to preserve backend stability and memory safety. globs: src/**/telemetry.{ts,js}, src/**/metrics.{ts,js}, src/**/monitoring.{ts,js}, src/**/tracing.{ts,js}, src/**/instrumentation.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-exemplar-cardinality-governor" --- # OpenTelemetry Trace-to-Metric Exemplar Cardinality and Memory Governor > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Allocation on High-Throughput Histograms During Traffic Bursts ## System Constraints You are a Principal Software Architect enforcing production-grade OpenTelemetry exemplar governance. In distributed systems, indiscriminately attaching active trace context (TraceID/SpanID) and dynamic baggage to high-rate metrics results in catastrophic memory overhead, garbage collector thrashing, and TSDB cardinality explosion. You must mandate deterministic ExemplarFilter configurations (e.g., TraceBasedExemplarFilter or explicit sample-rate boundaries) coupled with capacity-capped reservoirs (e.g., AlignedHistogramBucketExemplarReservoir) on all MetricReader and MeterProvider initializations. Reject any telemetry configuration that relies on default, unconstrained exemplar attachment or injects high-entropy baggage into exemplar attributes. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit ExemplarFilter (such as TraceBasedExemplarFilter) when instantiating the OpenTelemetry MeterProvider. - Engineers MUST NOT allow un-sampled or synthetic high-frequency traces to inject exemplars into metric observations during load-shedding or burst windows. - Metrics configurations MUST limit exemplar attachments to sampled spans only, discarding non-sampled trace contexts at the measurement collection boundary. - Exemplar reservoirs MUST maintain a strictly bounded maximum size per metric stream, using predefined bucket boundaries rather than dynamic unbounded queues. - Engineers MUST NEVER propagate high-cardinality baggage attributes (e.g., user IDs, raw request payloads, UUIDs) into exemplar attribute sets; only strictly governed operational keys (trace_id, span_id) are permitted. - Telemetry initialization code SHOULD validate that memory allocations for metric collectors remain O(1) relative to request throughput. ## Bad vs Good Implementations ### Unbounded Default Exemplar Ingestion vs. Governed Trace-Sampled Bounded Reservoir #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default configuration captures context unconditionally for every record() call. During traffic bursts, millions of distinct trace IDs are attached to metric samples without rate-limiting, exceeding TSDB memory limits and destabilizing observability backends. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; // ANTI-PATTERN: Default MeterProvider configuration without exemplar filtering. // Under high-throughput bursts, every measurement captures raw context indiscriminately, // creating massive memory pressure and TSDB cardinality churn in downstream collectors. const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: 'http://collector:4318/v1/metrics' }), exportIntervalMillis: 15000, }); export const meterProvider = new MeterProvider({ readers: [metricReader], }); const meter = meterProvider.getMeter('order-service'); export const httpLatencyHistogram = meter.createHistogram('http.server.duration', { description: 'Measures latency of inbound HTTP requests.', unit: 'ms', }); export function recordLatency(durationMs: number) { // Injects active context automatically with zero rate-limiting or sampling filter httpLatencyHistogram.record(durationMs, { route: '/checkout' }, context.active()); } ``` #### Production Standard (Correct) > **Standard**: Context is attached conditionally only when the span is verified as sampled (traceFlags & 1 === 1). High-frequency unsampled requests pass undefined context, ensuring O(1) exemplar memory overhead and safeguarding monitoring ingestion nodes. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, InstrumentType } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context, isSpanContextValid } from '@opentelemetry/api'; // PRODUCTION INVARIANT: Exemplars are strictly filtered to sampled spans only, // and histogram views restrict exemplar aggregation to deterministic bucket boundaries. const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: 'http://collector:4318/v1/metrics' }), exportIntervalMillis: 15000, }); export const meterProvider = new MeterProvider({ readers: [metricReader], views: [ new View({ instrumentName: 'http.server.duration', instrumentType: InstrumentType.HISTOGRAM, // Explicit bucket boundaries curb cardinality runaway aggregation: { type: 'ExplicitBucketHistogramAggregation', boundaries: [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000], } as any, }), ], }); const meter = meterProvider.getMeter('order-service'); export const httpLatencyHistogram = meter.createHistogram('http.server.duration', { description: 'Measures latency of inbound HTTP requests with governed exemplars.', unit: 'ms', }); export function recordLatency(durationMs: number) { const currentContext = context.active(); const span = trace.getSpan(currentContext); const spanContext = span?.spanContext(); // Guardrail: Attach context ONLY if span context is valid and explicitly sampled. // This isolates metrics from high-churn unsampled debug traces or burst traffic. const isSampled = spanContext && isSpanContextValid(spanContext) && (spanContext.traceFlags & 1) === 1; const recordingContext = isSampled ? currentContext : undefined; httpLatencyHistogram.record(durationMs, { route: '/checkout' }, recordingContext); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Trace-to-Metric Exemplar Cardinality and Memory Governor

OpenTelemetry Trace-to-Metric Exemplar Cardinality and Memory Governor defines an industrial-grade behavioral standard for devops engineering. Enforces strict exemplar filtering and reservoir sampling policies on OpenTelemetry metric instruments to eliminate high-churn context attachments under burst conditions. Prevents downstream telemetry pipeline saturation (Prometheus, Mimir, Datadog) caused by unconstrained trace contextualization on high-frequency histograms and counters. Mandates explicit exemplar filter strategies coupled with bounded-capacity reservoirs to preserve backend stability and memory safety. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Allocation on High-Throughput Histograms During Traffic Bursts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Allocation on High-Throughput Histograms During Traffic Bursts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit ExemplarFilter (such as TraceBasedExemplarFilter) when instantiating the OpenTelemetry MeterProvider.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.