OpenTelemetry In-Process Tail-Based Exemplar Sampling Rate-Limiter

--- description: OpenTelemetry In-Process Tail-Based Exemplar Sampling Rate-Limiter - Enforces deterministic, in-process rate limiting and tail-based filtering for OpenTelemetry metric exemplars prior to export. Unconstrained exemplar attachment during high-throughput bursts injects unbounded high-cardinality trace contexts into metric series, exhausting collector memory and triggering OOM cascading failures across observability pipelines. globs: src/observability/**/*.{ts,js,go,java}, src/telemetry/**/*.{ts,js,go,java}, src/metrics/**/*.{ts,js,go,java}, src/instrumentation/**/*.{ts,js,go,java} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-in-process-exemplar-rate-limiter-guardrail" --- # OpenTelemetry In-Process Tail-Based Exemplar Sampling Rate-Limiter > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unchecked Metric Exemplar Attachment and Unbounded In-Memory Cardinality Spikes Under High QPS ## System Constraints You are a Principal Observability Architect enforcing strict in-process rate limiting and tail-filtering on OpenTelemetry metric exemplars. Mandate bounded exemplar reservoirs, strict token-bucket or sliding-window rate limiters per metric bucket, and tail-based selective retention (e.g., error spans, high latency thresholds) before exporting to downstream APM collectors. ## Strict Guidelines - Engineers MUST ALWAYS bound exemplar retention using a fixed-capacity reservoir paired with a high-resolution token-bucket or leaky-bucket rate limiter per metric stream. - Engineers MUST NOT attach raw SpanContext to metric updates without evaluating tail-based eligibility criteria (e.g., HTTP 5xx responses, trace latency exceeding p99 SLOs, or explicit error flags). - Code SHOULD implement an adaptive dropped-exemplar counter to maintain visibility into rate-limited exemplar drops without re-introducing cardinality pressure. - Services MUST enforce hard caps on exemplar payload attributes, stripping arbitrary request headers, query parameters, and unvetted payload fields. - Components NEVER instantiate unbounded in-memory exemplar buffers that scale linearly with incoming request throughput (QPS). ## Bad vs Good Implementations ### In-Process Exemplar Filter with Token-Bucket Rate Limiter and Error Tail Sampling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Unconditionally returning true in ExemplarFilter.shouldSample allocates and binds TraceContext to every metric update. In high-QPS production environments, this generates millions of short-lived exemplar objects, induces heavy GC pause times, and saturates downstream collector memory buffers with redundant normal-trace exemplars. ```typescript import { trace, context } from '@opentelemetry/api'; import { MeterProvider, ExemplarFilter } from '@opentelemetry/sdk-metrics'; // ANTI-PATTERN: AlwaysSampleExemplarFilter attaches every single active span context. // Under a 50k QPS burst, this floods Prometheus/OTel collector with millions of high-cardinality // exemplars, causing exponential memory growth, collector buffer drops, and OOM kills. export const meterProvider = new MeterProvider({ exemplarFilter: { shouldSample(value, bucketBounds, ctx) { return true; // Unconditionally samples trace context for every metric point } } }); ``` #### Production Standard (Correct) > **Standard**: Implements a TailSampledRateLimitedExemplarFilter that filters out non-critical nominal traffic, retains exemplars strictly for SLA breaches or errors, and restricts exemplar generation to a deterministic upper bound using an in-process token-bucket algorithm. ```typescript import { trace, context, Context } from '@opentelemetry/api'; import { ExemplarFilter } from '@opentelemetry/sdk-metrics'; class TokenBucketRateLimiter { private tokens: number; private lastRefill: number; constructor(private readonly capacity: number, private readonly refillRatePerMs: number) { this.tokens = capacity; this.lastRefill = Date.now(); } public tryAcquire(): boolean { const now = Date.now(); const elapsed = now - this.lastRefill; this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRatePerMs); this.lastRefill = now; if (this.tokens >= 1) { this.tokens -= 1; return true; } return false; } } export class TailSampledRateLimitedExemplarFilter implements ExemplarFilter { private readonly limiter: TokenBucketRateLimiter; private readonly latencyThresholdMs: number; constructor(maxExemplarsPerSec = 10, latencyThresholdMs = 500) { this.limiter = new TokenBucketRateLimiter(maxExemplarsPerSec, maxExemplarsPerSec / 1000); this.latencyThresholdMs = latencyThresholdMs; } public shouldSample(value: number, bucketBounds: number[], ctx: Context): boolean { const activeSpan = trace.getSpan(ctx); if (!activeSpan) return false; const spanContext = activeSpan.spanContext(); if (!spanContext.isRemote && !spanContext.traceFlags) return false; // Tail-filtering: prioritize high-latency breaches and errors const isHighLatency = value >= this.latencyThresholdMs; const isError = (activeSpan as any).status?.code === 2; // SpanStatusCode.ERROR if (!isHighLatency && !isError) { return false; } // Rate-limiting: prevent collector exhaustion during sustained degradation return this.limiter.tryAcquire(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry In-Process Tail-Based Exemplar Sampling Rate-Limiter

OpenTelemetry In-Process Tail-Based Exemplar Sampling Rate-Limiter defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, in-process rate limiting and tail-based filtering for OpenTelemetry metric exemplars prior to export. Unconstrained exemplar attachment during high-throughput bursts injects unbounded high-cardinality trace contexts into metric series, exhausting collector memory and triggering OOM cascading failures across observability pipelines. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unchecked Metric Exemplar Attachment and Unbounded In-Memory Cardinality Spikes Under High QPS Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unchecked Metric Exemplar Attachment and Unbounded In-Memory Cardinality Spikes Under High QPS	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS bound exemplar retention using a fixed-capacity reservoir paired with a high-resolution token-bucket or leaky-bucket rate limiter per metric stream.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.