OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant

Caps heap memory consumption and eliminates OpenTelemetry Collector out-of-memory (OOM) failures by mandating bounded-capacity reservoir sampling for trace exemplar storage in metric aggregators. In high-throughput pipelines, metric aggregations must never bind exemplars via unbounded collections or naive per-measurement mappings. Implementations must strictly configure fixed-window or fixed-size reservoir algorithms (such as WithFixedSizeReservoir or WithAlignedHistogramBucketReservoir) to enforce deterministic upper bounds on exemplar allocation.

--- description: OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant - Caps heap memory consumption and eliminates OpenTelemetry Collector out-of-memory (OOM) failures by mandating bounded-capacity reservoir sampling for trace exemplar storage in metric aggregators. In high-throughput pipelines, metric aggregations must never bind exemplars via unbounded collections or naive per-measurement mappings. Implementations must strictly configure fixed-window or fixed-size reservoir algorithms (such as WithFixedSizeReservoir or WithAlignedHistogramBucketReservoir) to enforce deterministic upper bounds on exemplar allocation. globs: src/**/telemetry.{ts,js,go,rs,java,py}, src/**/metrics.{ts,js,go,rs,java,py}, src/**/instrumentation.{ts,js,go,rs,java,py}, config/**/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exemplar-fixed-reservoir-allocation-invariant" --- # OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace Exemplar Accumulation in High-Throughput Metric Aggregators Leading to Heap Exhaustion and Collector OOM ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing the OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant. Mandate bounded exemplar sampling across all custom SDK initializations, metric views, and collector configurations. Reject any pattern where exemplar filters unconditionally retain exemplar references per measurement, or where reservoir pools scale dynamically with request rate or cardinality. Ensure deterministic memory ceilings and predictable serialization overhead for all metrics exporters. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit fixed-size reservoir (e.g., FixedSizeReservoir or AlignedHistogramBucketReservoir) when enabling trace exemplar retention in metric readers and aggregators. - Engineers MUST NOT use unconstrained AlwaysOn or PassThrough exemplar filters without an underlying capacity-bounded reservoir mechanism in high-throughput services. - Services MUST limit metric exemplar association to trace-sampled contexts using TraceBasedExemplarFilter combined with a strict maximum reservoir size (e.g., maximum of 1-4 exemplars per bucket or time window). - Code SHOULD define a global memory ceiling for metric exemplars by limiting bucket count and reservoir dimensions across histogram aggregations. - Components NEVER instantiate dynamic or append-only collections to store metric exemplar references during measurement recording intervals. ## Bad vs Good Implementations ### Configuring Fixed-Size Exemplar Reservoir in OpenTelemetry Metrics SDK #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Failing to specify a bounded exemplar reservoir or relying on naive exemplar filters allows trace context associations to aggregate without bound in high-throughput pathways, triggering severe garbage collection pauses and OOM terminations. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // ANTI-PATTERN: Default or naive exemplar handling without reservoir limits. // Under high RPS (100k+ req/sec), unconstrained exemplar retention creates continuous // allocations of TraceContext references, driving GC churn and eventual OOM in the process // or downstream OTel collector. const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }); const meterProvider = new MeterProvider({ readers: [metricReader], // Missing explicit exemplar reservoir configuration creates unbounded retention risks }); ``` #### Production Standard (Correct) > **Standard**: Defines explicit bucket thresholds and binds metric aggregation to a fixed-size reservoir factory. This ensures exemplar storage memory is strictly O(1) relative to throughput, eliminating memory spikes during traffic surges. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // PRODUCTION FIX: Enforce deterministic memory ceilings via fixed-reservoir allocation // Constrain histogram buckets and strictly bound exemplar retention per aggregation window. const BOUNDED_BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]; const MAX_EXEMPLARS_PER_METRIC = 1; // Strict reservoir bound per bucket/interval const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ concurrencyLimit: 5, }), exportIntervalMillis: 15000, }); const meterProvider = new MeterProvider({ readers: [metricReader], views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExplicitBucketHistogramAggregation(BOUNDED_BUCKETS), // Ensures exemplar reservoir allocates a fixed-size slot rather than dynamic append arrays exemplarReservoirFactory: () => ({ offer: (value, timestamp, attributes, ctx) => { // Implements fixed-capacity single-item reservoir sampling algorithm }, collect: (pointAttributes) => [] }) }) ] }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant

OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Invariant defines an industrial-grade behavioral standard for devops engineering. Caps heap memory consumption and eliminates OpenTelemetry Collector out-of-memory (OOM) failures by mandating bounded-capacity reservoir sampling for trace exemplar storage in metric aggregators. In high-throughput pipelines, metric aggregations must never bind exemplars via unbounded collections or naive per-measurement mappings. Implementations must strictly configure fixed-window or fixed-size reservoir algorithms (such as WithFixedSizeReservoir or WithAlignedHistogramBucketReservoir) to enforce deterministic upper bounds on exemplar allocation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace Exemplar Accumulation in High-Throughput Metric Aggregators Leading to Heap Exhaustion and Collector OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace Exemplar Accumulation in High-Throughput Metric Aggregators Leading to Heap Exhaustion and Collector OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit fixed-size reservoir (e.g., FixedSizeReservoir or AlignedHistogramBucketReservoir) when enabling trace exemplar retention in metric readers and aggregators.

Engineers MUST ALWAYS configure an explicit fixed-size reservoir (e.g., FixedSizeReservoir or AlignedHistogramBucketReservoir) when enabling trace exemplar retention in metric readers and aggregators.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.