Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standard

--- description: Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standard - Enforces deterministic memory bounds for OpenTelemetry-to-Prometheus exemplar storage within high-throughput ingestion pipelines. Prevents out-of-memory (OOM) conditions by replacing unbounded exemplar attachments with strictly sized circular ring buffers and enforcing Prometheus 3.0 max_exemplars storage limits. Guarantees predictable trace-to-metric correlation without destabilizing TSDB head block allocations. globs: src/**/*.{ts,js,go,rs,py}, config/**/*.{yaml,yml}, deploy/**/*.{yaml,yml}, helm/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/prometheus-3-otel-exemplar-bounded-retention-standard" --- # Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded In-Memory Trace ID Accumulation in Metric Exemplar Buffers ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standards. You must mandate that every metric pipeline ingesting OpenTelemetry traces into Prometheus metrics strictly bounds exemplar memory allocations. Reject any implementation that allows unbounded exemplar collections, dynamic slice growth without hard ceilings, or misconfigured Prometheus TSDB exemplar storage limits. ## Strict Guidelines - Engineers MUST ALWAYS set a strict upper bound (e.g., storage.exemplars.max-exemplars in Prometheus 3.0, default 100000) on total retained TSDB exemplars. - Engineers MUST NOT attach raw OpenTelemetry span contexts directly to metrics without passing through a fixed-capacity circular ring buffer or deterministic reservoir sampler. - Code SHOULD enforce sample-rate filtering on exemplars at the SDK level to guarantee that trace ID correlation does not exceed 1 exemplar per series per scraping interval. - Services MUST ensure that combined exemplar label pairs (TraceID, SpanID) adhere to the OpenTelemetry and Prometheus combined limit of 128 runic characters. - Components NEVER dynamically allocate unconstrained maps or growing slices to store exemplar references awaiting scrape export. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoirs in OpenTelemetry TypeScript SDK #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching exemplar attributes directly to metric recordings without a bounded reservoir sampler causes the in-memory exemplar storage to grow proportional to request throughput rather than scraping frequency, quickly triggering V8 heap exhaustion and OOM kills. ```typescript import { Counter, MeterProvider } from '@opentelemetry/sdk-metrics'; import { trace } from '@opentelemetry/api'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const requestCounter = meter.createCounter('http_requests_total'); // Anti-pattern: Unbounded exemplar attachment on every incoming trace // In high-concurrency loops, this creates arbitrary memory growth per metric series export function recordRequest(route: string, durationMs: number) { const activeSpan = trace.getActiveSpan(); const spanContext = activeSpan?.spanContext(); // Storing trace metadata without a reservoir or bounded ring buffer requestCounter.add(1, { route }, { traceId: spanContext?.traceId, spanId: spanContext?.spanId } as any); } ``` #### Production Standard (Correct) > **Standard**: Implements a strict bounded reservoir that holds at most one exemplar candidate per metric series window, ensuring that memory usage remains O(1) regardless of request cardinality or pipeline throughput. ```typescript import { Counter, MeterProvider, ExemplarFilter } from '@opentelemetry/sdk-metrics'; import { trace } from '@opentelemetry/api'; // Fixed-capacity circular ring buffer preventing unbounded heap allocation class BoundedExemplarReservoir { private readonly capacity: number = 1; private storage: { traceId: string; spanId: string; timestamp: number } | null = null; public offer(traceId: string, spanId: string): void { // Circular replacement: overwrites the single slot per series window this.storage = { traceId, spanId, timestamp: Date.now() }; } public drain(): Record<string, string> | undefined { if (!this.storage) return undefined; const exemplar = { TraceID: this.storage.traceId, SpanID: this.storage.spanId }; this.storage = null; // Flush on harvest return exemplar; } } const reservoir = new BoundedExemplarReservoir(); const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('billing-service'); const requestCounter = meter.createCounter('http_requests_total'); export function recordRequest(route: string, durationMs: number) { const activeSpan = trace.getActiveSpan(); const spanContext = activeSpan?.spanContext(); if (spanContext && spanContext.traceId) { // Bound retention through reservoir to guarantee single-sample memory cap reservoir.offer(spanContext.traceId, spanContext.spanId); } const exemplar = reservoir.drain(); requestCounter.add(1, { route }, exemplar ? { ...exemplar } : undefined); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standard

Prometheus 3.0 Native OpenTelemetry Exemplar Bounded Retention Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds for OpenTelemetry-to-Prometheus exemplar storage within high-throughput ingestion pipelines. Prevents out-of-memory (OOM) conditions by replacing unbounded exemplar attachments with strictly sized circular ring buffers and enforcing Prometheus 3.0 max_exemplars storage limits. Guarantees predictable trace-to-metric correlation without destabilizing TSDB head block allocations. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded In-Memory Trace ID Accumulation in Metric Exemplar Buffers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded In-Memory Trace ID Accumulation in Metric Exemplar Buffers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS set a strict upper bound (e.g., storage.exemplars.max-exemplars in Prometheus 3.0, default 100000) on total retained TSDB exemplars.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.