OpenTelemetry OTLP Exemplar Reservoir Memory Bound Invariant

--- description: OpenTelemetry OTLP Exemplar Reservoir Memory Bound Invariant - Enforces deterministic memory bounds and fixed-size reservoir sampling for OpenTelemetry metric exemplars exported via OTLP and Prometheus. Prevents unbounded heap growth and catastrophic Out-Of-Memory (OOM) failures caused by high-cardinality trace exemplar attachments under high-throughput traffic. globs: src/**/telemetry.{ts,js}, src/**/instrumentation.{ts,js}, src/**/metrics.{ts,js}, src/**/otel.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exemplar-reservoir-memory-bounds" --- # OpenTelemetry OTLP Exemplar Reservoir Memory Bound Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Allocation and Dynamic Baggage Attachment Causing Runtime Heap Exhaustion ## System Constraints You are a Principal Observability Architect enforcing deterministic memory limits on OpenTelemetry metrics. You mandate strict reservoir bounds, attribute filtering, and deterministic sampling for metric exemplars across all OTLP and Prometheus exporters. ## Strict Guidelines - Engineers MUST ALWAYS configure fixed-capacity circular reservoirs (e.g., AlignedHistogramBucketExemplarReservoir or FixedSizeExemplarReservoir) when attaching trace exemplars to metric streams. - Engineers MUST NOT allow unbounded or default memory allocation strategies for metric exemplars on high-throughput counters and histograms. - Services MUST restrict exemplar metadata to only TraceId and SpanId, stripping dynamic baggage, arbitrary labels, or unhashed request payloads. - Components NEVER attach exemplars without an active TraceFlags.SAMPLED check to prevent leaking unsampled spans into exporter memory queues. - Code SHOULD enforce global View configurations that explicitly register histogram bucket boundaries alongside deterministic exemplar filter predicates. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoir vs Naive Memory Leak #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaches high-cardinality dynamic fields directly to metric attributes and lacks explicit reservoir sizing, allowing metric storage and exemplar queues to expand uncontrollably with traffic spikes. ```typescript import { MeterProvider } from '@opentelemetry/sdk-metrics'; import { trace, context } from '@opentelemetry/api'; // ANTI-PATTERN: Default or naive exemplar handling stores arbitrary spans & attributes. // In a high-throughput endpoint (e.g., 50k rps), capturing every span context and dynamic // attributes without bounds causes rapid GC degradation and Heap OOM. const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http_request_duration_ms'); export function recordRequest(durationMs: number, reqHeaders: Record<string, string>) { const activeSpan = trace.getSpan(context.active()); // DANGEROUS: Attaching unbounded dynamic baggage/headers directly to metrics latencyHistogram.record(durationMs, { userId: reqHeaders['x-user-id'], // High cardinality label traceId: activeSpan?.spanContext().traceId ?? 'none', rawPayload: reqHeaders['x-payload-debug'], // Unbounded payload leak }); } ``` #### Production Standard (Correct) > **Standard**: Constrains metric cardinality via explicit View attributes, defines deterministic bucket boundaries, and gates exemplar attachment on active sampled trace flags, ensuring memory consumption remains strictly O(1) relative to traffic volume. ```typescript import { MeterProvider, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { trace, context, TraceFlags } from '@opentelemetry/api'; // PRODUCTION INVARIANT: Enforce fixed-size reservoir with bounded bucket allocations // and deterministic exemplar filtering restricted to sampled traces. const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http_request_duration_ms', aggregation: new ExplicitBucketHistogramAggregation([ 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000 ]), // Exemplar reservoir bounds are managed at view/meter creation; // attributes are strictly filtered to low-cardinality static sets. attributeKeys: ['http_status_code', 'http_method'], }), ], }); const meter = meterProvider.getMeter('billing-service'); const latencyHistogram = meter.createHistogram('http_request_duration_ms', { unit: 'ms', description: 'Deterministic bounded request latency histogram', }); export function recordRequest(durationMs: number, statusCode: number, method: string) { const span = trace.getSpan(context.active()); const spanCtx = span?.spanContext(); // Enforce invariant: only sampled traces qualify for exemplar recording, // and attribute cardinality is strictly capped. if (spanCtx && (spanCtx.traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED) { latencyHistogram.record(durationMs, { http_status_code: statusCode.toString(), http_method: method, }, context.active()); } else { latencyHistogram.record(durationMs, { http_status_code: statusCode.toString(), http_method: method, }); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry OTLP Exemplar Reservoir Memory Bound Invariant

OpenTelemetry OTLP Exemplar Reservoir Memory Bound Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds and fixed-size reservoir sampling for OpenTelemetry metric exemplars exported via OTLP and Prometheus. Prevents unbounded heap growth and catastrophic Out-Of-Memory (OOM) failures caused by high-cardinality trace exemplar attachments under high-throughput traffic. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Allocation and Dynamic Baggage Attachment Causing Runtime Heap Exhaustion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Allocation and Dynamic Baggage Attachment Causing Runtime Heap Exhaustion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure fixed-capacity circular reservoirs (e.g., AlignedHistogramBucketExemplarReservoir or FixedSizeExemplarReservoir) when attaching trace exemplars to metric streams.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.