OpenTelemetry Trace Exemplar Rolling Reservoir & Cardinality Quota Boundary

--- description: OpenTelemetry Trace Exemplar Rolling Reservoir & Cardinality Quota Boundary - Enforces deterministic rolling reservoir sampling and strict attribute whitelisting on OpenTelemetry metric exemplars. Prevents telemetry collector Out-Of-Memory (OOM) crashes and time-series database (TSDB) block churn caused by unbounded exemplar attachment across high-throughput endpoints. Mandates sample-rate quotas and trace-sampled filter guards prior to exemplar ingestion. globs: src/**/telemetry.{ts,js}, src/**/metrics.{ts,js}, src/**/instrumentation.{ts,js}, config/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-high-cardinality-exemplar-quota-boundary" --- # OpenTelemetry Trace Exemplar Rolling Reservoir & Cardinality Quota Boundary > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Attaching Unbounded Trace Exemplars with High-Cardinality Context Across High-Throughput Metrics ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing OpenTelemetry metric-to-trace exemplar safety. You must ensure all metrics configurations implement bounded rolling reservoir sampling, attach trace context only for sampled traces, enforce strict attribute filtering on exemplar payloads, and maintain hard memory/bandwidth boundaries. ## Strict Guidelines - Engineers MUST ALWAYS configure `ExemplarFilter.traceBased()` or equivalent sampled-only trace filters to prevent unsampled spans from allocating exemplar slots. - Engineers MUST ALWAYS enforce bounded reservoir sizes (e.g., fixed-window rolling reservoirs of 1-2 exemplars per bucket per metric collection cycle). - Engineers MUST NOT inject dynamic high-cardinality metadata (e.g., user IDs, session tokens, request payloads, raw URLs) into exemplar attribute maps; only `trace_id` and `span_id` are permitted. - Collector pipelines MUST configure memory-limiter processors before metric batching when processing metrics with exemplars. - Services MUST NOT emit exemplars on counters or histograms that exceed 10,000 measurements per second without an explicit upstream rate-limited exemplar reservoir. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoir in OpenTelemetry SDK #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Records dynamic high-cardinality attributes ('user.id') directly into metric dimensions/exemplars on every invocation without an exemplar filter or reservoir. In high-traffic environments, this triggers runaway metric cardinality and forces unbounded exemplar allocation, causing OOM kills in telemetry collectors and TSDB storage bloat. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const requestLatency = meter.createHistogram('http.server.duration'); // ANTI-PATTERN: Manually injecting high-cardinality attributes into context // and recording every measurement as an exemplar with user-level data without reservoir boundaries export function recordRequest(durationMs: number, userId: string, route: string) { const activeSpan = trace.getSpan(context.active()); requestLatency.record(durationMs, { 'http.route': route, // Leaking per-user high-cardinality context into exemplar allocations 'user.id': userId, 'trace.id': activeSpan?.spanContext().traceId }); } ``` #### Production Standard (Correct) > **Standard**: Configures an explicit ExemplarFilter.traceBased() on the MeterProvider so only sampled traces allocate exemplar slots. Strips dynamic attributes (like userId) from the metric stream and relies on the SDK's built-in bucket-aligned rolling reservoir to keep memory deterministic. ```typescript import { MeterProvider, PeriodicExportingMetricReader, ExemplarFilter } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context, TraceFlags } from '@opentelemetry/api'; // PRODUCTION FIX: Enforce TraceBased exemplar filter & restrict exemplar attributes const metricExporter = new OTLPMetricExporter({ url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }); const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: metricExporter, exportIntervalMillis: 15000 }) ], exemplarFilter: ExemplarFilter.traceBased() }); const meter = meterProvider.getMeter('billing-service'); const requestLatency = meter.createHistogram('http.server.duration', { unit: 'ms', description: 'HTTP server execution latency' }); export function recordRequest(durationMs: number, route: string) { const span = trace.getSpan(context.active()); const isSampled = span && (span.spanContext().traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED; // Low-cardinality standard attributes only; reservoir manages exemplar bounds automatically requestLatency.record(durationMs, { 'http.route': route }, isSampled ? context.active() : undefined); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Trace Exemplar Rolling Reservoir & Cardinality Quota Boundary

OpenTelemetry Trace Exemplar Rolling Reservoir & Cardinality Quota Boundary defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic rolling reservoir sampling and strict attribute whitelisting on OpenTelemetry metric exemplars. Prevents telemetry collector Out-Of-Memory (OOM) crashes and time-series database (TSDB) block churn caused by unbounded exemplar attachment across high-throughput endpoints. Mandates sample-rate quotas and trace-sampled filter guards prior to exemplar ingestion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Attaching Unbounded Trace Exemplars with High-Cardinality Context Across High-Throughput Metrics Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Attaching Unbounded Trace Exemplars with High-Cardinality Context Across High-Throughput Metrics	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure `ExemplarFilter.traceBased()` or equivalent sampled-only trace filters to prevent unsampled spans from allocating exemplar slots.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.