OpenTelemetry Exemplar Reservoir Memory Saturation and Fixed-Capacity Guardrails

--- description: OpenTelemetry Exemplar Reservoir Memory Saturation and Fixed-Capacity Guardrails - Enforces deterministic memory bounds and strictly capped reservoir capacity on OpenTelemetry exemplar samplers attached to metric histograms. Prevents heap exhaustion, Garbage Collection thrashing, and unmetered trace context retention across high-throughput microservice runtimes under heavy traffic. globs: src/**/telemetry/**/*.{ts,js}, src/**/metrics/**/*.{ts,js}, src/**/instrumentation/**/*.{ts,js}, src/**/observability/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exemplar-reservoir-memory-saturation-standard" --- # OpenTelemetry Exemplar Reservoir Memory Saturation and Fixed-Capacity Guardrails > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Collection and Naive Reservoir Allocation in High-Cardinality Histograms ## System Constraints You are a Principal Software Architect enforcing the OpenTelemetry Exemplar Reservoir Memory Saturation Standard. You must mandate strict memory bounding on metric exemplar collection, reject unbounded or naive exemplar accumulation, enforce trace-based filtering with fixed-size reservoirs, and eliminate high-cardinality exemplar attribute leakage across all metric pipelines. ## Strict Guidelines - Engineers MUST ALWAYS use TraceBasedExemplarFilter or explicit sampling rates rather than AlwaysSampleExemplarFilter in production environments. - Engineers MUST configure metric histograms with explicit, fixed-capacity exemplar reservoirs (e.g., maximum bucket capacity of 1-4 exemplars per bucket) to guarantee deterministic upper memory bounds. - Engineers MUST NOT attach raw user-generated strings, dynamic payloads, or high-cardinality metadata (such as UUIDs, email addresses, or unindexed query strings) to exemplar context attributes. - Code SHOULD sanitize and strictly whitelist exemplar attributes to TraceID, SpanID, and bounded operational tenant identifiers before storage in the reservoir. - Services MUST ensure exemplar reservoirs cleanly discard or recycle stale exemplar references upon metric collection cycles to prevent GC generational promotion and heap fragmentation. - Components NEVER instantiate unbounded custom exemplar accumulators or arrays that grow proportionally with request throughput. ## Bad vs Good Implementations ### Bounded vs Unbounded Exemplar Reservoir and Filter Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Using AlwaysSampleExemplarFilter unconditionally stores exemplars on every observation. Serializing unbounded dynamic request payloads into context attributes pins large objects to metric buckets indefinitely, triggering severe heap fragmentation and rapid OOM crashes under high load. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { AlwaysSampleExemplarFilter } from '@opentelemetry/sdk-metrics'; // ANTI-PATTERN: AlwaysSample captures exemplars on every single measurement. // When paired with high-throughput histogram recording, memory balloons unboundedly. const meterProvider = new MeterProvider({ exemplarFilter: new AlwaysSampleExemplarFilter(), readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], }); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http.server.duration'); // High-cardinality metadata leaking directly into metric exemplar context export function recordLatency(durationMs: number, traceCtx: any, requestPayload: Record<string, any>) { latencyHistogram.record(durationMs, { ...traceCtx, rawPayload: JSON.stringify(requestPayload), // Catastrophic memory leak }); } ``` #### Production Standard (Correct) > **Standard**: Employs TraceBasedExemplarFilter to ensure exemplars are only sampled when the trace is already sampled downstream. Constrains metric attributes to low-cardinality dimensions ('http.route', 'http.status_code') and configures explicit histogram buckets, guaranteeing a deterministic upper bound on memory consumption. ```typescript import { MeterProvider, PeriodicExportingMetricReader, TraceBasedExemplarFilter, ExplicitBucketHistogramAggregation, View } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; // PRODUCTION FIX: TraceBasedExemplarFilter only retains exemplars for actively sampled traces. // Combine with View boundaries and sanitized bounded attributes. const meterProvider = new MeterProvider({ exemplarFilter: new TraceBasedExemplarFilter(), readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 30000, }), ], views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExplicitBucketHistogramAggregation([ 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 ]), }), ], }); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http.server.duration', { unit: 's', description: 'Measures incoming HTTP request durations with deterministic exemplar bounds', }); export function recordLatency(durationSeconds: number, route: string, statusCode: number): void { const currentSpan = trace.getSpan(context.active()); // Only recorded if the active trace is sampled; zero allocation for dynamic baggage latencyHistogram.record(durationSeconds, { 'http.route': route, 'http.status_code': statusCode, }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Exemplar Reservoir Memory Saturation and Fixed-Capacity Guardrails

OpenTelemetry Exemplar Reservoir Memory Saturation and Fixed-Capacity Guardrails defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds and strictly capped reservoir capacity on OpenTelemetry exemplar samplers attached to metric histograms. Prevents heap exhaustion, Garbage Collection thrashing, and unmetered trace context retention across high-throughput microservice runtimes under heavy traffic. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Collection and Naive Reservoir Allocation in High-Cardinality Histograms Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Collection and Naive Reservoir Allocation in High-Cardinality Histograms	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS use TraceBasedExemplarFilter or explicit sampling rates rather than AlwaysSampleExemplarFilter in production environments.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.