OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy

--- description: OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy - Enforces deterministic, upper-bounded memory consumption for metric exemplar storage across high-throughput telemetry pipelines. Prevents catastrophic garbage collection spikes and out-of-memory (OOM) failures under million-event-per-second ingestion by requiring fixed-capacity, trace-filtered reservoirs over default or unbounded exemplar sampling strategies. globs: src/**/telemetry/**/*.{ts,js}, src/**/metrics/**/*.{ts,js}, src/**/instrumentation/**/*.{ts,js}, src/**/observability/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-metric-exemplar-fixed-reservoir-allocation-policy" --- # OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded or Naive Metric Exemplar Retention Causing Severe GC Pressure and Heap Exhaustion ## System Constraints You are a Principal Software Architect enforcing the OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy. You must strictly mandate deterministic memory bounds for OpenTelemetry exemplars. Reject any unbounded exemplar storage, naive random sampling that bypasses trace context, or default memory configurations that destabilize services under million-event-per-second ingestion. Ensure all MetricReader and View declarations configure an explicit ExemplarFilter (TraceBased) and a strict Fixed-Size or Histogram-Bucket-Aligned Reservoir. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit `ExemplarFilter` that discards unsampled contexts (e.g., `TraceBasedExemplarFilter`) to avoid allocating exemplars for dead spans. - Engineers MUST NOT allow default or unconstrained memory allocation for exemplar reservoirs in high-throughput services (>10,000 req/sec). - Services MUST bound exemplar memory to O(1) per metric stream using `FixedSizeExemplarReservoir` or bucket-aligned reservoirs with hard capacity limits. - Components NEVER store raw high-cardinality payload attributes within exemplar metadata; only trace IDs, span IDs, and strictly whitelisted dimension keys are permitted. - Code SHOULD pre-allocate reservoir buffers during meter initialization to prevent heap fragmentation during production traffic bursts. ## Bad vs Good Implementations ### Configuring Fixed-Size Exemplar Reservoir with Trace-Based Filtering #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting the exemplar filter and reservoir factory causes the runtime to fall back to unbounded or unoptimized default exemplar strategies. Under million-event-per-second load, this creates millions of short-lived exemplar objects, leading to frequent GC cycles and inevitable heap exhaustion. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // ANTI-PATTERN: Default MeterProvider configuration retains exemplars // with default reservoir policies, creating runaway heap allocations // under 1M+ req/sec streams with active tracing contexts. export const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }), ], }); ``` #### Production Standard (Correct) > **Standard**: The fix enforces `TraceBasedExemplarFilter` so that only sampled traces trigger exemplar checks, and limits reservoir retention to a deterministic `FixedSizeExemplarReservoir(4)`. This caps memory overhead to O(1) per metric series and eliminates allocation churn. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; import { TraceBasedExemplarFilter, FixedSizeExemplarReservoir } from '@opentelemetry/sdk-metrics'; // PRODUCTION FIX: Explicitly bound exemplar allocations using TraceBasedExemplarFilter // and a deterministic fixed-capacity reservoir factory per time series. const MAX_EXEMPLARS_PER_SERIES = 4; export const meterProvider = new MeterProvider({ exemplarFilter: new TraceBasedExemplarFilter(), views: [ new View({ instrumentName: '*_duration_ms', aggregation: new ExplicitBucketHistogramAggregation([5, 10, 25, 50, 100, 250, 500, 1000]), exemplarReservoirFactory: () => new FixedSizeExemplarReservoir(MAX_EXEMPLARS_PER_SERIES), }), ], readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }), ], }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy

OpenTelemetry Metric Exemplar Fixed-Reservoir Allocation Policy defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, upper-bounded memory consumption for metric exemplar storage across high-throughput telemetry pipelines. Prevents catastrophic garbage collection spikes and out-of-memory (OOM) failures under million-event-per-second ingestion by requiring fixed-capacity, trace-filtered reservoirs over default or unbounded exemplar sampling strategies. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded or Naive Metric Exemplar Retention Causing Severe GC Pressure and Heap Exhaustion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded or Naive Metric Exemplar Retention Causing Severe GC Pressure and Heap Exhaustion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit `ExemplarFilter` that discards unsampled contexts (e.g., `TraceBasedExemplarFilter`) to avoid allocating exemplars for dead spans.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.