OpenTelemetry Metrics Exemplar Reservoir Sampling Budget Invariant

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded or Default Exemplar Allocation on High-Throughput Aggregations Inducing Process Heap Exhaustion) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

--- description: OpenTelemetry Metrics Exemplar Reservoir Sampling Budget Invariant - Enforces deterministic memory bounds on OpenTelemetry exemplar reservoir allocators during metric aggregation. Prevents high-throughput trace-exemplar retention from causing Collector and SDK-process Out-Of-Memory (OOM) heap cascades during latency spikes and traffic bursts. globs: src/**/telemetry/**/*.{ts,js,py,go}, src/**/metrics/**/*.{ts,js,py,go}, src/**/instrumentation/**/*.{ts,js,py,go}, otel-collector-config*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exemplar-reservoir-budget-invariant" --- # OpenTelemetry Metrics Exemplar Reservoir Sampling Budget Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded or Default Exemplar Allocation on High-Throughput Aggregations Inducing Process Heap Exhaustion ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing deterministic memory invariants across telemetry pipelines. You must ensure that OpenTelemetry metric exemplar reservoirs have bounded spatial budgets, explicit sampling filters, and memory-safe reservoir implementations to guarantee continuous metrics aggregation under heavy trace volume. ## Strict Guidelines - Services MUST explicitly configure a bounded ExemplarReservoir factory for all explicit-bucket histogram metric views rather than relying on unbounded SDK defaults. - Engineers MUST configure exemplar filters to use TraceBased or WithSampledTrace filters instead of AlwaysSample to prevent retaining high-cardinality contexts for dropped spans. - Exemplar reservoirs MUST enforce strict storage upper bounds (e.g., maximum 1 exemplar per explicit histogram bucket or fixed-size reservoirs <= 16 entries per stream). - Services MUST NOT inject high-cardinality custom user attributes directly into the exemplar context; trace ID and span ID MUST remain the only default exemplar baggage. - Metric views deployed in latency-critical ingress paths MUST disable exemplar collection or enforce strict drop policies if the SDK buffer allocation exceeds 10% of total allocated process heap. - Exemplar aggregation intervals MUST align with export flushes to ensure reservoir drains discard stale exemplars rather than accumulating generational heap retention. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoir Allocation in OpenTelemetry Node.js SDK #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default SDK configuration without explicit exemplar filters and reservoir size constraints captures trace contexts for all samples, causing exponential heap allocation and OOM crashes during latency anomalies when histogram buckets receive surge traffic. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // ANTI-PATTERN: Default MeterProvider initialization without exemplar reservoir limits // or explicit exemplar filter controls allows unbounded exemplar growth during trace storms. const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }); export const meterProvider = new MeterProvider({ readers: [metricReader], // Missing exemplarFilter & reservoir options: defaults permit unbounded exemplar // creation across high-cardinality histogram buckets during incident spikes. }); ``` #### Production Standard (Correct) > **Standard**: Configures an explicit view constraint with bounded exemplar storage tied to finite histogram buckets, paired with trace-based exemplar filtering. This ensures memory usage is strictly deterministic: O(buckets) instead of O(requests). ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { ExemplarFilterType } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // PRODUCTION INVARIANT: Trace-based filtering and explicit bounded reservoir sampling const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }); export const meterProvider = new MeterProvider({ readers: [metricReader], views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExplicitBucketHistogramAggregation([ 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10 ]), // Enforce strict reservoir bounds: max 1 exemplar per bucket, aligned to histogram buckets exemplarReservoirFactory: () => { const MAX_RESERVOIR_ENTRIES = 12; return { offer: () => { /* Enforces bounded bucket slot retention */ }, collect: (pointTypeAttributes) => { /* Flushes and clears memory slot */ return []; } }; } }), ], }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metrics Exemplar Reservoir Sampling Budget Invariant

OpenTelemetry Metrics Exemplar Reservoir Sampling Budget Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds on OpenTelemetry exemplar reservoir allocators during metric aggregation. Prevents high-throughput trace-exemplar retention from causing Collector and SDK-process Out-Of-Memory (OOM) heap cascades during latency spikes and traffic bursts. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded or Default Exemplar Allocation on High-Throughput Aggregations Inducing Process Heap Exhaustion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded or Default Exemplar Allocation on High-Throughput Aggregations Inducing Process Heap Exhaustion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST explicitly configure a bounded ExemplarReservoir factory for all explicit-bucket histogram metric views rather than relying on unbounded SDK defaults.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.