OpenTelemetry High-Cardinality Dynamic Exemplar Retention and Adaptive Quota Standard

--- description: OpenTelemetry High-Cardinality Dynamic Exemplar Retention and Adaptive Quota Standard - Enforces adaptive exemplar sampling and dynamic rate-limiting on OpenTelemetry metric streams to prevent backend telemetry degradation during distributed incidents. Mandates deterministic exemplar reservation reservoirs bounded by leaky-bucket quotas that throttle high-cardinality trace associations during error spikes. Ensures metric collection pipelines maintain zero-allocation invariants and resist OOM crashes under downstream telemetry partition events. globs: src/observability/**/*.{ts,js}, src/telemetry/**/*.{ts,js}, src/infrastructure/monitoring/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-dynamic-exemplar-quota-standard" --- # OpenTelemetry High-Cardinality Dynamic Exemplar Retention and Adaptive Quota Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exemplar Sampling on Error Metrics Triggering Telemetry Backpressure and Collector OOM ## System Constraints You are a Principal Observability Architect enforcing strict OpenTelemetry Dynamic Exemplar Retention and Cardinality Guardrails. You must prevent telemetry storms, unbounded memory growth, and backend collector eviction during cascading microservice failures. Ensure all trace-metric associations adhere to adaptive exemplar filters, strictly bounded reservoir algorithms, and circuit-broken exporter pipelines. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) wrapped with an adaptive rate-limiting or token-bucket ceiling to prevent metric stream bloat during error storms. - Engineers MUST NOT attach raw, unbound high-cardinality metadata (such as tenant IDs, raw UUIDs, or unhashed query strings) directly to exemplar attributes; attributes MUST be restricted to standard trace/span IDs and pre-hashed cluster identifiers. - Reservoirs MUST use fixed-capacity, pre-allocated algorithms (such as an aligned circular buffer or fixed-size bucket reservoir) with bounded retention windows to guarantee predictable heap utilization. - Services MUST implement circuit-breaking drop-tail policies on exemplar retention when telemetry exporter queue saturation exceeds 80% capacity. - Instrumentation NEVER attaches exemplars synchronously on hot-path counters without checking the adaptive quota token balance first, avoiding lock contention and GC overhead. ## Bad vs Good Implementations ### Configuring Adaptive Exemplar Reservoir with Token-Bucket Ceiling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching raw user identifiers directly to metric attributes causes an unrecoverable cardinality explosion in the metrics store. Concurrently, using the default unconstrained context passing without an adaptive exemplar rate limiter floods the downstream collector with exemplars during cascading service failure spikes, causing memory pressure and dropped metrics. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; // ANTI-PATTERN: Default or naive exemplar handling with unbounded trace association on error paths const meterProvider = new MeterProvider({ readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], }); const meter = meterProvider.getMeter('billing-service'); const errorCounter = meter.createCounter('http_server_errors_total'); export function recordError(route: string, userId: string, err: Error) { const span = trace.getSpan(context.active()); // Unchecked exemplar attachment attaching high-cardinality attributes directly on error storms errorCounter.add(1, { route, userId, // Cardinality explosion in labels errorMessage: err.message, }, context.active()); } ``` #### Production Standard (Correct) > **Standard**: Enforces a strict Low-Cardinality metric dimension contract and implements an `AdaptiveExemplarFilter` backed by a token-bucket algorithm. It couples instrument views to a deterministic `FixedSizeExemplarReservoir`, guaranteeing that exemplar memory consumption remains bounded regardless of downstream microservice error cascades. ```typescript import { context, trace, Context } from '@opentelemetry/api'; import { MeterProvider, PeriodicExportingMetricReader, View, InstrumentType, ExemplarFilter, ExemplarReservoir, FixedSizeExemplarReservoir, } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // Token-bucket adaptive rate limiter for exemplar admission class AdaptiveRateLimiter { private tokens: number; private lastRefill: number = Date.now(); constructor( private readonly capacity: number = 50, private readonly refillRatePerSec: number = 10 ) { this.tokens = capacity; } public allow(): boolean { const now = Date.now(); const elapsed = (now - this.lastRefill) / 1000; this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRatePerSec); this.lastRefill = now; if (this.tokens >= 1) { this.tokens -= 1; return true; } return false; } } class AdaptiveExemplarFilter implements ExemplarFilter { private readonly limiter = new AdaptiveRateLimiter(50, 10); shouldSample(value: number, timestamp: number, ctx: Context): boolean { const span = trace.getSpan(ctx); if (!span || !span.spanContext().isValid || !span.spanContext().isSampled) { return false; } // Dynamic gating: Drop exemplar if quota is exceeded during error storms return this.limiter.allow(); } } export const exemplarFilter = new AdaptiveExemplarFilter(); const meterProvider = new MeterProvider({ exemplarFilter, readers: [ new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), ], views: [ new View({ instrumentType: InstrumentType.COUNTER, exemplarReservoirFactory: () => new FixedSizeExemplarReservoir(16), }), ], }); const meter = meterProvider.getMeter('billing-service'); const errorCounter = meter.createCounter('http_server_errors_total'); export function recordError(route: string, errorCode: string) { // Invariant preserved: Low-cardinality attributes; exemplars dynamically sampled via reservoir ceiling errorCounter.add(1, { route, error_code: errorCode, }, context.active()); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry High-Cardinality Dynamic Exemplar Retention and Adaptive Quota Standard

OpenTelemetry High-Cardinality Dynamic Exemplar Retention and Adaptive Quota Standard defines an industrial-grade behavioral standard for devops engineering. Enforces adaptive exemplar sampling and dynamic rate-limiting on OpenTelemetry metric streams to prevent backend telemetry degradation during distributed incidents. Mandates deterministic exemplar reservation reservoirs bounded by leaky-bucket quotas that throttle high-cardinality trace associations during error spikes. Ensures metric collection pipelines maintain zero-allocation invariants and resist OOM crashes under downstream telemetry partition events. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exemplar Sampling on Error Metrics Triggering Telemetry Backpressure and Collector OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exemplar Sampling on Error Metrics Triggering Telemetry Backpressure and Collector OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) wrapped with an adaptive rate-limiting or token-bucket ceiling to prevent metric stream bloat during error storms.

Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) wrapped with an adaptive rate-limiting or token-bucket ceiling to prevent metric stream bloat during error storms.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.