OpenTelemetry Trace-Correlated Exemplar Reservoir Sampling and Cardinality Bounds

--- description: OpenTelemetry Trace-Correlated Exemplar Reservoir Sampling and Cardinality Bounds - Enforces deterministic sliding-window reservoir sampling and strict retention filters on OpenTelemetry metric exemplars. Prevents trace ID cardinality explosions and Prometheus TSDB head-chunk memory exhaustion caused by unconstrained exemplar allocation during high-throughput metric updates. globs: src/telemetry/**/*.{ts,js}, src/monitoring/**/*.{ts,js}, **/otel*.{ts,js}, otel-collector-config*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exemplar-cardinality-firewall" --- # OpenTelemetry Trace-Correlated Exemplar Reservoir Sampling and Cardinality Bounds > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Attachment and Naive AlwaysOn Trace Sampling in High-Throughput Histograms ## System Constraints You are a Principal Engineer enforcing OpenTelemetry Trace-Correlated Metric Exemplar Cardinality Firewall standards. Mandate invariant reservoir sampling, explicit exemplar filter configuration, and bounded TSDB payload budgets across all metrics collection pipelines. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit ExemplarReservoir (such as AlignedHistogramBucketExemplarReservoir) with hard boundaries on exemplar count rather than relying on unmetered telemetry sinks. - Engineers MUST NOT bind raw or high-cardinality execution metadata (e.g., user IDs, tenant IDs, full URIs) into exemplar attribute sets alongside TraceId and SpanId. - Telemetry pipelines MUST enforce a TraceContext-aware ExemplarFilter (e.g., TraceBasedExemplarFilter) ensuring only sampled traces generate exemplars. - Exemplar storage engines MUST enforce sliding-window retention with an eviction ceiling (default maximum 1 exemplar per bucket per collection cycle). - Telemetry configurations NEVER allow AlwaysOn metric exemplar collection in production environments where write traffic exceeds 1,000 operations per second. ## Bad vs Good Implementations ### Configuring Bounded Sliding-Window Exemplar Reservoir in OpenTelemetry Metrics #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly injecting trace IDs, span IDs, and user IDs into metric labels converts TSDB series dimensions into infinite cardinality dimensions, triggering immediate TSDB write-path degradation and Prometheus OOM crash loops. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http_request_duration_ms'); // ANTI-PATTERN: Manually attaching high-cardinality attributes and trace context // to metric attributes without an exemplar reservoir, bloating TSDB label cardinality export function recordRequest(durationMs: number, userId: string, tenantId: string) { const activeSpan = trace.getSpan(context.active()); const spanContext = activeSpan?.spanContext(); latencyHistogram.record(durationMs, { userId, tenantId, traceId: spanContext?.traceId ?? 'unknown', spanId: spanContext?.spanId ?? 'unknown', }); } ``` #### Production Standard (Correct) > **Standard**: Removes high-cardinality dimensions from metric labels, binds traces exclusively via OpenTelemetry's native exemplar mechanism using TraceBasedExemplarFilter, and enforces fixed histogram bucket boundaries. ```typescript import { MeterProvider, PeriodicExportingMetricReader, ExplicitBucketHistogramAggregation, View } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { ExemplarDefaults, ExemplarFilter } from '@opentelemetry/sdk-metrics'; // Production Invariant: Restrict metric attributes to low-cardinality invariants // and isolate trace correlation to bounded OpenTelemetry exemplars via TraceBasedExemplarFilter. const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }), exportIntervalMillis: 15000, }); const meterProvider = new MeterProvider({ readers: [metricReader], exemplarFilter: ExemplarDefaults.traceBased(), views: [ new View({ instrumentName: 'http_request_duration_ms', aggregation: new ExplicitBucketHistogramAggregation([ 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000 ]), attributeKeys: ['http_route', 'http_status_code'], }), ], }); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http_request_duration_ms'); export function recordRequest(durationMs: number, route: string, statusCode: number) { // Invariant: TraceId/SpanId are captured implicitly into the bucket exemplar reservoir // only when the active trace is sampled, preserving strictly bounded TSDB series. latencyHistogram.record(durationMs, { http_route: route, http_status_code: statusCode, }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Trace-Correlated Exemplar Reservoir Sampling and Cardinality Bounds

OpenTelemetry Trace-Correlated Exemplar Reservoir Sampling and Cardinality Bounds defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic sliding-window reservoir sampling and strict retention filters on OpenTelemetry metric exemplars. Prevents trace ID cardinality explosions and Prometheus TSDB head-chunk memory exhaustion caused by unconstrained exemplar allocation during high-throughput metric updates. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Attachment and Naive AlwaysOn Trace Sampling in High-Throughput Histograms Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Attachment and Naive AlwaysOn Trace Sampling in High-Throughput Histograms	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit ExemplarReservoir (such as AlignedHistogramBucketExemplarReservoir) with hard boundaries on exemplar count rather than relying on unmetered telemetry sinks.

Engineers MUST ALWAYS configure an explicit ExemplarReservoir (such as AlignedHistogramBucketExemplarReservoir) with hard boundaries on exemplar count rather than relying on unmetered telemetry sinks.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.