OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standard

--- description: OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standard - Enforces strict memory-bounded reservoir sampling and trace-context filtering for OpenTelemetry metric exemplars. Prevents collector and agent heap exhaustion, stop-the-world garbage collection pauses, and telemetry pipeline OOM kills during high-throughput p99 anomaly bursts. Mandates deterministic reservoir sizing and strict exemplar attribute key filtering. globs: src/**/telemetry/**/*.{ts,js,go,java}, src/**/instrumentation/**/*.{ts,js,go,java}, src/**/metrics/**/*.{ts,js,go,java} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-metric-exemplar-reservoir-sampling-standard" --- # OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Attachment and Naive Always-On Reservoir Retention Causing OOM Crashes ## System Constraints You are a Principal Software Architect and Site Reliability Engineer enforcing OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standards. You mandate deterministic bounds on span-to-metric exemplar allocations, ensure trace-sampled gating on all exemplar reservoirs, and prohibit dynamic un-whitelisted attribute pass-through from span contexts into metric storage buffers. ## Strict Guidelines - Engineers MUST ALWAYS pair metric aggregations with an explicit bounded ExemplarReservoir implementation (e.g., AlignedHistogramBucketExemplarReservoir or SimpleFixedSizeExemplarReservoir). - Engineers MUST NEVER use AlwaysOnExemplarFilter in production services; services MUST configure TraceBasedExemplarFilter (or sample-gated equivalent) to drop exemplars from unsampled traces. - Exemplar attribute attachments MUST NOT capture arbitrary high-cardinality attributes (such as user IDs, raw request payloads, UUIDs, or transient auth tokens); attachments MUST be restricted to trace_id, span_id, and an explicit allowlist of system dimensions. - Engineers MUST bound the reservoir capacity per histogram bucket to a deterministic constant (maximum 1 to 2 exemplars per bucket) to eliminate memory explosion on explicit bucket histograms. - Instrumentation code SHOULD pre-allocate exemplar slot buffers or leverage memory pools during SDK bootstrapping to eliminate object allocation churning in p99 execution critical paths. ## Bad vs Good Implementations ### Configuring Bounded Histogram Bucket Exemplar Reservoir with Trace-Filter Safeguards #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting explicit exemplar reservoir bounds and trace-based filters exposes the node runtime to arbitrary memory growth. When latency spikes occur, thousands of spans attach their context and baggage to histogram aggregations, causing massive GC pressure and OOM death. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // ANTI-PATTERN: Default or naive exemplar configuration // Missing explicit reservoir configuration uses unbounded or default memory retainers; // Metric reader will attempt to capture exemplars indiscriminately, retaining un-sampled spans // and raw span baggage, precipitating rapid heap exhaustion during traffic surges. const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], }); const meter = meterProvider.getMeter('order-service'); export const latencyHistogram = meter.createHistogram('http.server.duration', { description: 'Measures latency of inbound HTTP transactions', unit: 'ms', }); ``` #### Production Standard (Correct) > **Standard**: Enforces an explicit boundary configuration, ensures only trace-sampled spans are captured into the exemplar stream, and strips uncontrolled dynamic attributes, ensuring constant time and space complexity during p99 anomaly bursts. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation, InstrumentType } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { TraceFlags } from '@opentelemetry/api'; // PRODUCTION FIX: Deterministically bound exemplar reservoir and enforce trace sampling filter const BOUNDED_LATENCY_BOUNDARIES = [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000]; // Custom factory enforcing TraceBased filtering and deterministic bucket-to-exemplar mapping function createSafeReservoir() { return { offer: (value: number, timestamp: [number, number], attributes: Record<string, string>, ctx: any) => { // 1. Invariant: Only sampled spans can generate metric exemplars const spanContext = ctx.getSpan()?.spanContext(); if (!spanContext || (spanContext.traceFlags & TraceFlags.SAMPLED) !== TraceFlags.SAMPLED) { return; } // 2. Invariant: Enforce explicit attribute key whitelist (drop high-cardinality baggage) const sanitizedAttributes: Record<string, string> = { env: attributes['env'] ?? 'prod', region: attributes['region'] ?? 'us-east-1' }; // Storage bounded to single slot per bucket in AlignedHistogramBucketExemplarReservoir } }; } const meterProvider = new MeterProvider({ views: [ new View({ instrumentType: InstrumentType.HISTOGRAM, instrumentName: 'http.server.duration', aggregation: new ExplicitBucketHistogramAggregation(BOUNDED_LATENCY_BOUNDARIES), }), ], readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }), ], }); export const meter = meterProvider.getMeter('order-service'); export const latencyHistogram = meter.createHistogram('http.server.duration', { description: 'Measures latency with bounded exemplar reservoir memory footprint', unit: 'ms', }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standard

OpenTelemetry Metric Exemplar High-Cardinality Reservoir Sampling Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded reservoir sampling and trace-context filtering for OpenTelemetry metric exemplars. Prevents collector and agent heap exhaustion, stop-the-world garbage collection pauses, and telemetry pipeline OOM kills during high-throughput p99 anomaly bursts. Mandates deterministic reservoir sizing and strict exemplar attribute key filtering. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Attachment and Naive Always-On Reservoir Retention Causing OOM Crashes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Attachment and Naive Always-On Reservoir Retention Causing OOM Crashes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS pair metric aggregations with an explicit bounded ExemplarReservoir implementation (e.g., AlignedHistogramBucketExemplarReservoir or SimpleFixedSizeExemplarReservoir).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.