OpenTelemetry In-Process Trace-to-Metric Exemplar Reservoir and Memory Budget Guardrail

--- description: OpenTelemetry In-Process Trace-to-Metric Exemplar Reservoir and Memory Budget Guardrail - Enforces strict memory budgeting and bucket-bound exemplar sampling for OpenTelemetry metric instruments to eliminate TSDB cardinality explosions and in-process heap exhaustion. Mandates deterministic ExemplarReservoir allocation, explicit bucket boundaries, and strict exemplar attribute pruning under high concurrency. Prevents unbounded retention of context references, high-cardinality span baggage, and runaway exemplar buffers in metric readers. globs: src/telemetry/**/*.{ts,js}, src/metrics/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/instrumentation/**/*.{ts,js}, src/server/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-in-process-exemplar-budgeting-guardrail" --- # OpenTelemetry In-Process Trace-to-Metric Exemplar Reservoir and Memory Budget Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Sampling with Full Attribute Propagation Causing TSDB Saturation and OOM ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Semantic Conventions and in-process exemplar budgeting. Your mission is to prevent telemetry-induced Out-Of-Memory (OOM) failures and downstream TSDB cardinality explosions. Mandate explicit exemplar reservoirs, deterministic bucket-level allocation limits, strict attribute filtering on exemplars, and bounded sample retention intervals across all metric pipelines. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) paired with a bounded ExemplarReservoir factory instead of relying on default unbounded SDK accumulators. - Engineers MUST NOT attach raw Span baggage, dynamic request bodies, tenant IDs, or arbitrary span attributes to metric exemplars; exemplars MUST retain strictly `trace_id` and `span_id`. - Services MUST enforce explicit bucket histogram aggregations (`ExplicitBucketHistogramAggregation`) with deterministic bucket boundaries to establish an upper bound on exemplar memory slots. - Reservoirs MUST limit sampling to a maximum of one exemplar per histogram bucket per metric collection cycle to cap TSDB ingestion rates under high throughput. - Components NEVER hold strong object references to active Span contexts or parent contexts inside exemplar storage; references must be decoupled into primitive string identifiers immediately upon capture. - Metric views MUST discard exemplar capture on non-sampled spans (`SampledFlag.NOT_SAMPLED`) to protect memory reserves during high-concurrency traffic bursts. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoir and Filtered Metric Aggregation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Defaulting exemplar capture without explicit reservoir bounds causes memory retention of active context chains. Passing high-cardinality attributes like 'user.id' directly into the metric instrument while allowing exemplars to capture the full span context saturates TSDB storage and leaks memory in the in-process accumulator under heavy load. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; // ANTI-PATTERN: Default MeterProvider without View or Exemplar bounds. // In high-concurrency environments, every span attaches full baggage, // creating unbounded exemplar references and memory leaks in the TSDB collector. const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], }); const meter = meterProvider.getMeter('http-service'); const latencyHistogram = meter.createHistogram('http.server.duration'); // Bad record call: Captures active context with arbitrary unbounded attributes export function recordRequest(durationMs: number, userId: string, route: string) { latencyHistogram.record(durationMs, { 'http.route': route, 'user.id': userId, // Cardinality disaster }, context.active()); } ``` #### Production Standard (Correct) > **Standard**: The production configuration enforces an ExplicitBucketHistogramAggregation view, strictly budgeting exemplar reservoir memory to match predefined latency buckets. Low-cardinality semantic conventions are enforced on attributes ('http.route', 'http.response.status_code'), and exemplar linking is conditionally bypassed for non-sampled spans to prevent CPU and memory exhaustion. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation, InstrumentType } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context, isSpanContextValid } from '@opentelemetry/api'; // PRODUCTION FIX: Explicit bucket boundaries define strict upper-bound on reservoir slots. const LATENCY_BOUNDARIES = [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]; const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 30000, }), ], views: [ new View({ instrumentName: 'http.server.request.duration', instrumentType: InstrumentType.HISTOGRAM, aggregation: new ExplicitBucketHistogramAggregation(LATENCY_BOUNDARIES), // Downstream TSDB exemplar budget is deterministic: max 1 exemplar per bucket boundary }), ], }); const meter = meterProvider.getMeter('http-service', '1.0.0'); const latencyHistogram = meter.createHistogram('http.server.request.duration', { unit: 'ms', description: 'Measures incoming HTTP request duration with bounded exemplar allocation.', }); export function recordRequestDuration(durationMs: number, httpRoute: string, statusCode: number): void { const activeSpan = trace.getSpan(context.active()); const spanContext = activeSpan?.spanContext(); // Strictly enforce trace-sampled filter: only capture exemplar if span is sampled const isSampled = spanContext && isSpanContextValid(spanContext) && (spanContext.traceFlags & 1) === 1; latencyHistogram.record(durationMs, { 'http.route': httpRoute, 'http.response.status_code': statusCode, }, isSampled ? context.active() : undefined); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry In-Process Trace-to-Metric Exemplar Reservoir and Memory Budget Guardrail

OpenTelemetry In-Process Trace-to-Metric Exemplar Reservoir and Memory Budget Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory budgeting and bucket-bound exemplar sampling for OpenTelemetry metric instruments to eliminate TSDB cardinality explosions and in-process heap exhaustion. Mandates deterministic ExemplarReservoir allocation, explicit bucket boundaries, and strict exemplar attribute pruning under high concurrency. Prevents unbounded retention of context references, high-cardinality span baggage, and runaway exemplar buffers in metric readers. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Sampling with Full Attribute Propagation Causing TSDB Saturation and OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Sampling with Full Attribute Propagation Causing TSDB Saturation and OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) paired with a bounded ExemplarReservoir factory instead of relying on default unbounded SDK accumulators.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.