OpenTelemetry Multi-Dimensional Histogram Exemplar Boundary & Buffer Guardrail

--- description: OpenTelemetry Multi-Dimensional Histogram Exemplar Boundary & Buffer Guardrail - Enforces strict bounds on OpenTelemetry exemplar reservoirs, sampling filters, and dimension cardinality to prevent Collector and Agent memory exhaustion under high throughput. Guarantees deterministic metric memory footprints while selectively preserving P99+ tail latency exemplar trace linkages. Mandates cell-level exemplar boundaries and explicit reservoir memory pooling across all distributed metric pipelines. globs: src/**/telemetry.{ts,js,go,py,rs}, src/**/metrics.{ts,js,go,py,rs}, src/**/otel*.{ts,js,go,py,rs}, deploy/**/otel-collector*.{yaml,yml}, k8s/**/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-histogram-exemplar-boundary-guardrail" --- # OpenTelemetry Multi-Dimensional Histogram Exemplar Boundary & Buffer Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded AlwaysOn Exemplar Sampling and High-Cardinality Trace Buffering Causing Collector Fleet OOMs ## System Constraints You are a Principal Observability Architect enforcing industrial OpenTelemetry exemplar invariants. Your task is to ensure telemetry SDKs and collector pipelines implement strictly bounded exemplar reservoirs, deterministic bucket allocations, and targeted tail-latency sampling. You MUST prevent unbounded exemplar retention, reject unconstrained label injection inside exemplar payloads, and enforce defensive memory ceilings that isolate metric streams from high-throughput trace overhead. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) instead of accepting default unconstrained AlwaysOn sampling on high-throughput metric streams. - Engineers MUST NOT attach dynamic, high-cardinality attributes (such as user IDs, raw query strings, or IP addresses) directly to exemplar metadata payloads. - Code MUST initialize histograms with bounded bucket boundaries matching target SLA distributions and pair them with an AlignedHistogramBucketExemplarReservoir or FixedSizeExemplarReservoir. - Services MUST bound the exemplar reservoir storage to a maximum of 1 exemplar per bucket per collection cycle to enforce deterministic heap overhead. - Collector pipelines MUST apply memory limiter processors (`memory_limiter`) upstream of batch and exporter processors to protect against exemplar memory amplification during ingestion spikes. - Components NEVER store spans or context objects by direct reference inside exemplar callbacks; only extract and retain scalar trace_id and span_id hex strings. ## Bad vs Good Implementations ### Configuring Bounded Histogram Exemplar Reservoirs in OpenTelemetry TypeScript SDK #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default histogram configurations with AlwaysOn exemplar filters and arbitrary high-cardinality labels create thousands of retained exemplar allocations per aggregation cycle. Under extreme RPS, memory utilization escalates uncontrollably, triggering GC thrashing and out-of-memory (OOM) fatal crashes in the application runtime and collector tiers. ```typescript import { MeterProvider } from '@opentelemetry/sdk-metrics'; import { trace } from '@opentelemetry/api'; const meterProvider = new MeterProvider(); const meter = meterProvider.getMeter('order-service'); // ANTI-PATTERN: Default exemplar behavior captures high-cardinality attributes // on every measurement without boundary filtering or bucket-aligned reservoir limits. const latencyHistogram = meter.createHistogram('http.server.duration', { unit: 'ms', description: 'Unbounded exemplar histogram', }); export function recordRequest(durationMs: number, userId: string, traceCtx: any) { latencyHistogram.record(durationMs, { 'user.id': userId, // CRITICAL: High-cardinality label inside measurement 'http.route': '/checkout', }, traceCtx); } ``` #### Production Standard (Correct) > **Standard**: Enforces deterministic bucket boundaries using ExplicitBucketHistogramAggregation and selectively passes active context only for sampled spans. High-cardinality user identifiers are evicted from metric dimensions, allowing OpenTelemetry's internal AlignedHistogramBucketExemplarReservoir to reserve exactly one exemplar per bucket per harvest cycle, capping heap usage while retaining tail latency root-cause traces. ```typescript import { MeterProvider, ExplicitBucketHistogramAggregation, View, InstrumentType } from '@opentelemetry/sdk-metrics'; import { Context, trace } from '@opentelemetry/api'; // SLA-aligned latency buckets (in ms) to constrain memory footprint const LATENCY_BUCKETS = [5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]; const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', instrumentType: InstrumentType.HISTOGRAM, aggregation: new ExplicitBucketHistogramAggregation(LATENCY_BUCKETS), // Exemplar reservoir bounds are strictly constrained by bucket count }), ], }); const meter = meterProvider.getMeter('order-service'); const latencyHistogram = meter.createHistogram('http.server.duration', { unit: 'ms', description: 'Bounded exemplar histogram for HTTP server duration', }); export function recordRequest(durationMs: number, route: string, ctx: Context) { const span = trace.getSpan(ctx); // Only record exemplar context if the span is sampled (TraceBasedExemplarFilter invariant) const activeContext = span?.spanContext().isRemote || span?.spanContext().traceFlags === 1 ? ctx : undefined; // Strict low-cardinality attributes only; trace linkage handles debug identifiers latencyHistogram.record(durationMs, { 'http.route': route, 'http.status_code': 200, }, activeContext); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Multi-Dimensional Histogram Exemplar Boundary & Buffer Guardrail

OpenTelemetry Multi-Dimensional Histogram Exemplar Boundary & Buffer Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict bounds on OpenTelemetry exemplar reservoirs, sampling filters, and dimension cardinality to prevent Collector and Agent memory exhaustion under high throughput. Guarantees deterministic metric memory footprints while selectively preserving P99+ tail latency exemplar trace linkages. Mandates cell-level exemplar boundaries and explicit reservoir memory pooling across all distributed metric pipelines. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded AlwaysOn Exemplar Sampling and High-Cardinality Trace Buffering Causing Collector Fleet OOMs Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded AlwaysOn Exemplar Sampling and High-Cardinality Trace Buffering Causing Collector Fleet OOMs	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit ExemplarFilter (e.g., TraceBasedExemplarFilter) instead of accepting default unconstrained AlwaysOn sampling on high-throughput metric streams.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.