OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard

--- description: OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard - Enforces strict bounded ring-buffer reservoirs for dynamic OpenTelemetry trace-to-metric exemplars across all metrics collection pipelines. Unbounded exemplar allocation during distributed cascading failures degrades runtime performance and induces Out-Of-Memory (OOM) fatal crashes due to unbounded heap retention of span contexts. All exemplar filter implementations must enforce deterministic, pre-allocated reservoir sizes and drop policies under burst loads. globs: src/**/telemetry/**/*.{ts,js,go,java}, src/**/metrics/**/*.{ts,js,go,java}, src/**/instrumentation/**/*.{ts,js,go,java}, otel-collector/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-metric-exemplar-ring-buffer-cardinality-pinning" --- # OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded or Default Trace-to-Metric Exemplar Sampling Leading to Heap Exhaustion During Telemetry Surges ## System Constraints You are a Principal Engineer enforcing OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard. Mandate invariants that prevent heap exhaustion and out-of-memory cascading crashes caused by unconstrained trace-to-metric exemplar retention. Ensure exemplar reservoirs are pre-allocated, statically bounded using ring buffers or strict reservoir sampling algorithms, and reject dynamic heap resizing during telemetry surges. ## Strict Guidelines - Engineers MUST ALWAYS configure explicit, bounded exemplar reservoirs (e.g., FixedSizeExemplarReservoir or AlignedHistogramBucketExemplarReservoir with fixed capacities) on all MetricReaders and MeterProviders. - Engineers MUST NOT rely on default SDK exemplar allocation strategies if the default allows unbounded growth or unbounded context attachment during trace bursts. - Services MUST pin the maximum number of retained exemplars per metric time-series to an empirically safe ceiling (typically 1 to 4 exemplars per histogram bucket or counter). - Telemetry pipelines MUST use deterministic sampling filters (such as TraceBasedExemplarFilter or explicit sample-with-probability) to eliminate redundant exemplar attachments for identical error codes. - Instrumented components NEVER retain full distributed tracing context objects or baggage within metric exemplars; only extract and retain the immutable TraceId, SpanId, and bounded timestamp. ## Bad vs Good Implementations ### Configuring Bounded Exemplar Reservoir vs Unbounded Dynamic Attachment #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default configuration in telemetry runtimes often either attaches unconstrained exemplars or scales internal storage linearly with active traced requests during an incident, exhausting heap space when error volume surges. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // ANTI-PATTERN: Default meter provider instantiation without explicit exemplar reservoir limits. // Under severe incident bursts, span context attachments scale dynamically with request volume, // retaining large object graphs in memory and triggering OOM. const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], }); ``` #### Production Standard (Correct) > **Standard**: Pins exemplar retention to a bounded circular strategy with a hard-coded maximum capacity, protecting process memory from exhaustion during catastrophic traffic or error bursts. ```typescript import { MeterProvider, PeriodicExportingMetricReader, ExemplarFilter } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; import { TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base'; // PRODUCTION FIX: Explicitly configure a bounded exemplar filter and fixed-size reservoir limits. // Pre-allocates fixed ring-buffer capacity per metric series and drops overflow deterministically. const MAX_EXEMPLARS_PER_SERIES = 2; class BoundedRingBufferExemplarFilter implements ExemplarFilter { private counter = 0; private readonly capacity = MAX_EXEMPLARS_PER_SERIES; shouldSample(): boolean { // Bounded sampling logic ensuring ring-buffer retention invariants this.counter = (this.counter + 1) % this.capacity; return this.counter === 0; } } const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], exemplarFilter: new BoundedRingBufferExemplarFilter(), }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard

OpenTelemetry Metric Exemplar Ring-Buffer Cardinality Pinning Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict bounded ring-buffer reservoirs for dynamic OpenTelemetry trace-to-metric exemplars across all metrics collection pipelines. Unbounded exemplar allocation during distributed cascading failures degrades runtime performance and induces Out-Of-Memory (OOM) fatal crashes due to unbounded heap retention of span contexts. All exemplar filter implementations must enforce deterministic, pre-allocated reservoir sizes and drop policies under burst loads. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded or Default Trace-to-Metric Exemplar Sampling Leading to Heap Exhaustion During Telemetry Surges Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded or Default Trace-to-Metric Exemplar Sampling Leading to Heap Exhaustion During Telemetry Surges	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure explicit, bounded exemplar reservoirs (e.g., FixedSizeExemplarReservoir or AlignedHistogramBucketExemplarReservoir with fixed capacities) on all MetricReaders and MeterProviders.

Engineers MUST ALWAYS configure explicit, bounded exemplar reservoirs (e.g., FixedSizeExemplarReservoir or AlignedHistogramBucketExemplarReservoir with fixed capacities) on all MetricReaders and MeterProviders.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.