OpenTelemetry Bounded Exemplar Retention and Tail-Sampled Metric Invariant

--- description: OpenTelemetry Bounded Exemplar Retention and Tail-Sampled Metric Invariant - Restricts OpenTelemetry histogram exemplar retention to bounded, fixed-size memory reservoirs while enforcing zero high-cardinality metric labels. Deterministically binds high-cardinality trace contexts and tail-sampling policies exclusively to latency quantile outliers to eliminate memory bloat and metrics storage explosion. globs: src/telemetry/**/*.{ts,js}, src/observability/**/*.{ts,js}, otel-collector-config*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-exemplar-tail-sampling-standard" --- # OpenTelemetry Bounded Exemplar Retention and Tail-Sampled Metric Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Metric Label Cardinality with Head-Sampled Tracing and Leaky Exemplar Reservoirs ## System Constraints You are a Principal Software Architect enforcing production observability guardrails. Mandate bounded OpenTelemetry exemplar reservoirs, absolute low-cardinality metric label invariants, and explicit alignment between metric outlier buckets and collector tail-sampling configurations. ## Strict Guidelines - Engineers MUST NEVER inject dynamic or high-cardinality keys (e.g., `userId`, `tenantId`, `transactionId`, `orderId`) into metric dimension labels. - Engineers MUST ALWAYS route dynamic or high-cardinality metadata into active span attributes or trace-context exemplars. - Services MUST configure OpenTelemetry `MeterProvider` with explicit `ExemplarFilter` (e.g., `TraceBasedExemplarFilter`) and bounded `ExemplarReservoir` factories (e.g., `AlignedHistogramBucketExemplarReservoir`) to prevent linear heap degradation. - Exemplar storage MUST enforce strict memory bounds per bucket, capping total exemplars retained across all active histogram series. - Collector pipelines MUST deploy tail-sampling processors configured with explicit latency thresholds matching histogram upper-bound buckets to ensure traces referenced by exemplars are not dropped. ## Bad vs Good Implementations ### Configuring Bounded Histogram Exemplars with Outlier Trace Context #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly passing tenantId and userId into metric labels generates millions of unique time-series vectors in the metric store, leading to runaway memory consumption and monitoring pipeline outages without bounding exemplar memory. ```typescript import { MeterProvider } from '@opentelemetry/sdk-metrics'; import { trace } from '@opentelemetry/api'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('checkout-service'); const latencyHistogram = meter.createHistogram('http.server.duration'); export function recordLatencyBad(durationMs: number, tenantId: string, userId: string) { // CRITICAL FAILURE: Injects high-cardinality attributes directly into metric labels. // Causes Prometheus / M3DB time-series cardinality explosion and OOMs. latencyHistogram.record(durationMs, { tenant_id: tenantId, user_id: userId, route: '/api/v1/checkout' }); } ``` #### Production Standard (Correct) > **Standard**: Keeps metric series strictly bounded by filtering label keys to finite enums (route, status, method). High-cardinality fields remain on span attributes, linked to histogram outlier buckets via trace exemplars with zero metric memory bloat. ```typescript import { ExplicitBucketHistogramAggregation, MeterProvider, View } from '@opentelemetry/sdk-metrics'; import { trace, context } from '@opentelemetry/api'; const LATENCY_BOUNDARIES = [10, 50, 100, 250, 500, 1000, 2500, 5000]; // Enforce low-cardinality dimensions via Metric Views const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExplicitBucketHistogramAggregation(LATENCY_BOUNDARIES), attributeKeys: ['http.route', 'http.status_code', 'http.method'], }), ], }); const meter = meterProvider.getMeter('checkout-service'); const latencyHistogram = meter.createHistogram('http.server.duration', { unit: 'ms', description: 'HTTP server request execution duration', }); export function recordLatencyGood(durationMs: number, tenantId: string, userId: string) { const activeSpan = trace.getActiveSpan(); // Attach high-cardinality metadata strictly to the active trace span if (activeSpan && activeSpan.isRecording()) { activeSpan.setAttribute('tenant.id', tenantId); activeSpan.setAttribute('user.id', userId); } // Record metric with low-cardinality labels only. // Trace context (trace_id, span_id) is bound via OpenTelemetry Exemplar automatically. latencyHistogram.record( durationMs, { 'http.route': '/api/v1/checkout', 'http.status_code': 200, 'http.method': 'POST', }, context.active() ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Bounded Exemplar Retention and Tail-Sampled Metric Invariant

OpenTelemetry Bounded Exemplar Retention and Tail-Sampled Metric Invariant defines an industrial-grade behavioral standard for devops engineering. Restricts OpenTelemetry histogram exemplar retention to bounded, fixed-size memory reservoirs while enforcing zero high-cardinality metric labels. Deterministically binds high-cardinality trace contexts and tail-sampling policies exclusively to latency quantile outliers to eliminate memory bloat and metrics storage explosion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Metric Label Cardinality with Head-Sampled Tracing and Leaky Exemplar Reservoirs Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Metric Label Cardinality with Head-Sampled Tracing and Leaky Exemplar Reservoirs	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NEVER inject dynamic or high-cardinality keys (e.g., `userId`, `tenantId`, `transactionId`, `orderId`) into metric dimension labels.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.