OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard

--- description: OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard - Enforces strict memory-bounded sampling and rate-limiting on OpenTelemetry metric exemplars attached to high-cardinality latency measurements. Prevents telemetry collector and pipeline ingestion nodes from encountering Out-Of-Memory (OOM) crashes by restricting exemplar allocation to fixed-size reservoir samplers and filtering out non-sampled or sub-p99 traces. globs: src/telemetry/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/metrics/**/*.{ts,js}, src/instrumentation/**/*.{ts,js}, otel-collector/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-bounded-exemplar-trace-correlation" --- # OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Attaching Unfiltered Trace Contexts to Metrics via Unbounded Exemplar Reservoirs ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing the OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard. Unbounded exemplar collection leaks high-cardinality baggage and trace identifiers into time-series pipelines, overwhelming downstream collectors and storage backends (Prometheus, Cortex, Mimir). You must strictly enforce bounded reservoir algorithms, threshold-gated exemplar sampling, and rigorous dimension filtering across all metric instruments. ## Strict Guidelines - Services MUST ALWAYS configure fixed-capacity reservoir samplers (e.g., AlignedHistogramBucketExemplarReservoir or BoundedMemoryReservoir) when attaching trace contexts to metric histograms. - Engineers MUST NOT attach exemplars to metrics on every HTTP/RPC event; exemplars MUST be gated behind sampled span context checks and latency threshold predicates. - Instrumentation MUST limit exemplar attributes strictly to trace_id and span_id, explicitly stripping unbounded baggage or dynamic request headers. - Metrics aggregation configurations MUST enforce explicit bucket count boundaries and prevent unbounded dynamic label generation alongside exemplar tracking. - Telemetry pipelines MUST configure drop policies on collectors to prune redundant exemplars exceeding defined ingestion rate limits. ## Bad vs Good Implementations ### Bounded vs Unbounded Exemplar Recording in TypeScript Instrumentation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly attaching trace contexts and arbitrary user headers into metric attributes bypasses exemplar reservoir limits, turning time-series keys into high-cardinality memory sinks that crash collectors. ```typescript import { trace, context } from '@opentelemetry/api'; import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('api-gateway'); const latencyHistogram = meter.createHistogram('http_request_duration_ms'); export function recordRequestDurationBad(durationMs: number, reqHeaders: Record<string, string>) { const span = trace.getSpan(context.active()); // ANTI-PATTERN: Attaching raw span context and dynamic high-cardinality headers // on every single request without checking sampling state or bounds. latencyHistogram.record(durationMs, { trace_id: span?.spanContext().traceId ?? 'none', user_agent: reqHeaders['user-agent'] ?? 'unknown', user_id: reqHeaders['x-user-id'] ?? 'anonymous', exemplar_data: JSON.stringify(reqHeaders) }); } ``` #### Production Standard (Correct) > **Standard**: Validates sampling flags and restricts exemplar attachment exclusively to tail latency (>= p99 threshold), using low-cardinality standard attributes and native context passing to leverage SDK-managed bounded reservoirs. ```typescript import { trace, context, TraceFlags } from '@opentelemetry/api'; import { MeterProvider, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; const P99_THRESHOLD_MS = 250; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('api-gateway'); // Explicitly bounded histogram view to restrict aggregation memory meterProvider.addView( new View({ instrumentName: 'http_request_duration_ms', aggregation: new ExplicitBucketHistogramAggregation([5, 25, 50, 100, 250, 500, 1000, 2500]) }) ); const latencyHistogram = meter.createHistogram('http_request_duration_ms', { unit: 'ms', description: 'HTTP duration with bounded tail exemplars' }); export function recordRequestDurationGood(durationMs: number, method: string, status: number) { const spanContext = trace.getSpan(context.active())?.spanContext(); // INVARIANT: Exemplars are only correlated if span is sampled and latency crosses threshold const isSampled = spanContext && (spanContext.traceFlags & TraceFlags.SAMPLED) === TraceFlags.SAMPLED; const isTailLatency = durationMs >= P99_THRESHOLD_MS; const attributes = { 'http.request.method': method, 'http.response.status_code': status }; if (isSampled && isTailLatency) { // Bound exemplar correlation safely with minimal invariant attributes latencyHistogram.record(durationMs, attributes, context.active()); } else { latencyHistogram.record(durationMs, attributes); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard

OpenTelemetry Bounded Exemplar Trace-Metric Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded sampling and rate-limiting on OpenTelemetry metric exemplars attached to high-cardinality latency measurements. Prevents telemetry collector and pipeline ingestion nodes from encountering Out-Of-Memory (OOM) crashes by restricting exemplar allocation to fixed-size reservoir samplers and filtering out non-sampled or sub-p99 traces. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Attaching Unfiltered Trace Contexts to Metrics via Unbounded Exemplar Reservoirs Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Attaching Unfiltered Trace Contexts to Metrics via Unbounded Exemplar Reservoirs	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST ALWAYS configure fixed-capacity reservoir samplers (e.g., AlignedHistogramBucketExemplarReservoir or BoundedMemoryReservoir) when attaching trace contexts to metric histograms.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.