OpenTelemetry Tail-Based Sampling Dynamic Ring-Buffer Circuit Breaker

--- description: OpenTelemetry Tail-Based Sampling Dynamic Ring-Buffer Circuit Breaker - Enforces backpressure-driven probabilistic sampling decay and strict memory-bounded ring buffers within OpenTelemetry tail-based sampling pipelines. Prevents collector node OOM cascading failures during traffic surges by dynamically decaying trace retention windows and dropping non-error traces when ring-buffer memory pressure hits critical high-water marks. globs: **/otel-collector*.{yaml,yml}, src/telemetry/**/*.{ts,js,go}, infra/k8s/**/*.{yaml,yml}, src/monitoring/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-dynamic-ring-buffer-breaker" --- # OpenTelemetry Tail-Based Sampling Dynamic Ring-Buffer Circuit Breaker > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded In-Memory Trace Buffering Without Memory Pressure Circuit Breaking ## System Constraints You are a Principal Observability and Infrastructure Reliability Engineer specializing in high-throughput distributed tracing and OpenTelemetry collector resilience. Mandate strict memory limits, dynamic ring-buffer slot bounds, and backpressure-driven probabilistic sampling decay for all tail-based sampling configurations. Reject any tail-sampling topology lacking memory-limiter guards or unbounded in-memory trace collection. ## Strict Guidelines - Engineers MUST ALWAYS place the `memory_limiter` processor before the `tail_sampling` processor in the OpenTelemetry Collector pipeline. - Engineers MUST NOT allow unbounded trace buffering; ring-buffer allocation MUST have strict capacity limits (`num_traces`) paired with hard heap allocation bounds. - Code SHOULD implement adaptive probabilistic decay that increases sample shedding exponentially as buffer saturation approaches the 80% high-water mark. - Services MUST prioritize complete traces with recorded errors or HTTP 5xx codes when evicting buffered spans under memory pressure. - Components NEVER hold open trace collection buffers past a deterministic maximum `decision_wait` timeout under load spikes. ## Bad vs Good Implementations ### Tail-Sampling Ring-Buffer Implementation with Dynamic Probabilistic Decay #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: This naive trace buffer maintains an unbounded Map without memory limits, size caps, or TTL expiration. When traffic spikes or microservices fail with high cardinality, memory consumption grows linearly until an out-of-memory crash occurs, causing a cascading failure across collector instances. ```typescript import { Span } from '@opentelemetry/api'; // ANTI-PATTERN: Unbounded Map buffering spans until completion. // Under traffic spikes, this induces heap exhaustion and triggers collector OOM crashes. export class NaiveTraceBuffer { private traces: Map<string, Span[]> = new Map(); public addSpan(traceId: string, span: Span): void { const spans = this.traces.get(traceId) ?? []; spans.push(span); this.traces.set(traceId, spans); } public evaluateTrace(traceId: string): Span[] | null { const spans = this.traces.get(traceId); if (!spans) return null; const hasError = spans.some((s) => (s as any).status?.code === 2); if (hasError) { this.traces.delete(traceId); return spans; } return null; } } ``` #### Production Standard (Correct) > **Standard**: Implements an explicit capacity ceiling, TTL pruning, and quadratic probabilistic shedding beyond the 80% high-water mark. Non-error spans are dropped preemptively under memory strain while preserving error traces, preventing OOM collapses. ```typescript import { Span, SpanStatusCode } from '@opentelemetry/api'; interface BufferedTrace { spans: Span[]; hasError: boolean; timestamp: number; } export class ResilientRingBufferBreaker { private readonly maxTraces: number; private readonly highWaterMark: number; private readonly maxTtlMs: number; private buffer: Map<string, BufferedTrace> = new Map(); constructor(maxTraces = 50_000, highWaterMarkRatio = 0.8, maxTtlMs = 30_000) { this.maxTraces = maxTraces; this.highWaterMark = Math.floor(maxTraces * highWaterMarkRatio); this.maxTtlMs = maxTtlMs; } public addSpan(traceId: string, span: Span): boolean { const now = Date.now(); const currentSize = this.buffer.size; // Circuit Breaker: Probabilistic decay when surpassing high-water mark if (currentSize >= this.highWaterMark) { const saturation = (currentSize - this.highWaterMark) / (this.maxTraces - this.highWaterMark); const dropProbability = Math.min(1.0, Math.pow(saturation, 2)); const isErrorSpan = (span as unknown as { status?: { code: SpanStatusCode } }).status?.code === SpanStatusCode.ERROR; if (!isErrorSpan && Math.random() < dropProbability) { // Shed non-error trace early to protect buffer health return false; } } // Hard saturation cap: evict expired or non-error traces immediately if (currentSize >= this.maxTraces) { this.evictOldestDegraded(now); } let trace = this.buffer.get(traceId); if (!trace) { trace = { spans: [], hasError: false, timestamp: now }; this.buffer.set(traceId, trace); } const isError = (span as unknown as { status?: { code: SpanStatusCode } }).status?.code === SpanStatusCode.ERROR; trace.spans.push(span); trace.hasError = trace.hasError || isError; return true; } private evictOldestDegraded(now: number): void { for (const [id, trace] of this.buffer.entries()) { if (!trace.hasError || now - trace.timestamp > this.maxTtlMs) { this.buffer.delete(id); return; } } // If all contain errors, drop the oldest trace deterministically const oldestKey = this.buffer.keys().next().value; if (oldestKey) this.buffer.delete(oldestKey); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Dynamic Ring-Buffer Circuit Breaker

OpenTelemetry Tail-Based Sampling Dynamic Ring-Buffer Circuit Breaker defines an industrial-grade behavioral standard for devops engineering. Enforces backpressure-driven probabilistic sampling decay and strict memory-bounded ring buffers within OpenTelemetry tail-based sampling pipelines. Prevents collector node OOM cascading failures during traffic surges by dynamically decaying trace retention windows and dropping non-error traces when ring-buffer memory pressure hits critical high-water marks. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded In-Memory Trace Buffering Without Memory Pressure Circuit Breaking Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded In-Memory Trace Buffering Without Memory Pressure Circuit Breaking	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS place the `memory_limiter` processor before the `tail_sampling` processor in the OpenTelemetry Collector pipeline.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.