OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Exemplar Collection and Static Trace-Binding Under Traffic Surges) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap

--- description: OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap - Enforces deterministic memory bounds and ingestion rate throttling on OpenTelemetry Prometheus exemplars via dynamic reservoir sampling. Prevents TSDB cardinality explosions, GC pressure, and Prometheus scrape timeouts during high-concurrency traffic spikes by capping in-memory exemplar allocation per metric family and scrape cycle. globs: src/telemetry/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/metrics/**/*.{ts,js}, src/infra/monitoring/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-prometheus-exemplar-dynamic-reservoir-cap" --- # OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Collection and Static Trace-Binding Under Traffic Surges ## System Constraints You are a Principal Observability Architect enforcing strict memory and cardinality invariants on OpenTelemetry Prometheus exemplars. You must guarantee that all Prometheus metric exporters and meter providers deploy dynamic reservoir sampling caps, prevent unbounded exemplar allocation under traffic spikes, filter non-sampled spans, and enforce hard limits on exemplar byte size and count per scrape interval. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit exemplar reservoir with strict upper bounds (e.g., AlignedHistogramBucketExemplarReservoir with fixed bucket counts or a rate-capped dynamic reservoir) instead of default unbounded or greedy collectors. - Engineers MUST NOT use 'AlwaysSampleExemplarFilter' in production pipelines; telemetry pipelines MUST enforce 'TraceBasedExemplarFilter' or rate-governed exemplar filters to discard unsampled trace contexts. - Reservoirs MUST dynamically drop or uniformly downsample incoming exemplars once the per-metric allocation cap is reached within a scrape cycle using reservoir sampling algorithms (e.g., Vitter's Algorithm R). - Telemetry configurations MUST enforce attribute truncation on exemplar label sets to prevent unbounded dimension explosion from high-cardinality metadata (e.g., raw user IDs, transaction IDs) attached to exemplar attachments. - Services MUST monitor and expose the drop-counter metric for discarded exemplars to preserve auditability of telemetry degradation during ingestion spikes. - Components NEVER store exemplar references across metric harvest intervals; reservoirs MUST flush or reset exemplar slots deterministically upon scrape reader collection to prevent heap fragmentation. ## Bad vs Good Implementations ### Configuring Dynamic Reservoir Sampling Cap for Prometheus Exemplars #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default exemplar handling attaches traces without strict reservoir constraints or rate limits. Under burst conditions, millions of exemplar references remain resident in heap between Prometheus scrape intervals, degrading process performance, triggering GC pauses, and causing scrape timeouts or TSDB crashes in Prometheus. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { PrometheusExporter } from '@opentelemetry/exporter-prometheus'; // ANTI-PATTERN: Default configuration allows unthrottled exemplar collection. // Under a 50,000 req/sec spike, millions of exemplars allocate heap objects, // creating severe GC pause spikes, scrape timeouts, and Prometheus TSDB explosion. const exporter = new PrometheusExporter({ preventServerStart: false, port: 9464, }); const meterProvider = new MeterProvider({ readers: [exporter], // Exemplar filter left to defaults or unbounded capture }); ``` #### Production Standard (Correct) > **Standard**: Implements an explicit reservoir cap with dynamic uniform sampling (Algorithm R), guarantees unsampled trace contexts are discarded before memory allocation, and provides deterministic state clearance upon reader collection. ```typescript import { MeterProvider, ExplicitBucketHistogramAggregation, View } from '@opentelemetry/sdk-metrics'; import { PrometheusExporter } from '@opentelemetry/exporter-prometheus'; import { TraceFlags } from '@opentelemetry/api'; // PRODUCTION FIX: Enforce deterministic reservoir sampling & trace-based filtering interface ReservoirSample { value: number; timestamp: [number, number]; traceId: string; spanId: string; } class CappedDynamicReservoir { private sample: ReservoirSample | null = null; private count = 0; constructor(private readonly maxCapacity: number = 1) {} public offer(value: number, traceId: string, spanId: string, traceFlags: number): void { // Invariant: Discard unsampled spans immediately if ((traceFlags & TraceFlags.SAMPLED) !== TraceFlags.SAMPLED) { return; } this.count++; // Reservoir sampling: replace slot with probability maxCapacity / count if (this.sample === null) { this.sample = { value, timestamp: process.hrtime(), traceId, spanId }; } else if (Math.random() < this.maxCapacity / this.count) { this.sample = { value, timestamp: process.hrtime(), traceId, spanId }; } } public collectAndReset(): ReservoirSample | null { const current = this.sample; this.sample = null; this.count = 0; return current; } } const exporter = new PrometheusExporter({ port: 9464, preventServerStart: false, }); const meterProvider = new MeterProvider({ readers: [exporter], views: [ new View({ instrumentName: 'http_request_duration_seconds', aggregation: new ExplicitBucketHistogramAggregation([0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]), }), ], }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap

OpenTelemetry Prometheus Exemplar Dynamic Reservoir Sampling Cap defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds and ingestion rate throttling on OpenTelemetry Prometheus exemplars via dynamic reservoir sampling. Prevents TSDB cardinality explosions, GC pressure, and Prometheus scrape timeouts during high-concurrency traffic spikes by capping in-memory exemplar allocation per metric family and scrape cycle. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Collection and Static Trace-Binding Under Traffic Surges Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Collection and Static Trace-Binding Under Traffic Surges	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit exemplar reservoir with strict upper bounds (e.g., AlignedHistogramBucketExemplarReservoir with fixed bucket counts or a rate-capped dynamic reservoir) instead of default unbounded or greedy collectors.

Engineers MUST ALWAYS configure an explicit exemplar reservoir with strict upper bounds (e.g., AlignedHistogramBucketExemplarReservoir with fixed bucket counts or a rate-capped dynamic reservoir) instead of default unbounded or greedy collectors.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.