Bounded Ring-Buffer and Circuit-Breaker Guardrails for OTLP Telemetry Pipelines

--- description: Bounded Ring-Buffer and Circuit-Breaker Guardrails for OTLP Telemetry Pipelines - Enforces strict memory-bounded ring buffers and circuit-breaker drop mechanics on all OpenTelemetry OTLP metric and trace export pipelines. Ensures that downstream collector unavailability or network degradation sheds telemetry load immediately rather than inducing heap exhaustion or blocking the application event loop. Guarantees the primary invariant that observability infrastructure must never compromise system availability. globs: src/**/tracing/**/*.{ts,js}, src/**/telemetry/**/*.{ts,js}, src/**/instrumentation/**/*.{ts,js}, src/**/observability/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-otlp-ring-buffer-circuit-breaker" --- # Bounded Ring-Buffer and Circuit-Breaker Guardrails for OTLP Telemetry Pipelines > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Telemetry Buffering Causing Heap Exhaustion and Microservice Brownouts ## System Constraints You are a Principal Software Architect enforcing zero-impact observability. Mandate that OpenTelemetry OTLP export pipelines use finite, pre-allocated bounded ring buffers with explicit load-shedding and active circuit-breaker trip states. Telemetry backpressure must never leak into worker threads or application execution loops. If an OTLP collector endpoint degrades, export drops MUST fail fast, record a single internal counter without logging floods, and release span references immediately to maintain deterministic heap ceilings. ## Strict Guidelines - Engineers MUST ALWAYS configure explicit limits on `maxQueueSize` and `maxExportBatchSize` on all OpenTelemetry BatchSpanProcessor and BatchMetricReader instances; defaults that allow unbounded memory growth are strictly prohibited. - Engineers MUST NOT use synchronous blocking export calls or unbounded promises within OTLP exporter transport handlers. - Pipelines MUST implement a Fail-Open / Drop-Oldest load-shedding policy: when telemetry buffers saturate, newly arriving or oldest pending spans MUST be silently dropped, incrementing an in-memory drop metric rather than allocating additional buffer space. - Export pipelines MUST wrap the OTLP exporter in a three-state circuit breaker (Closed, Open, Half-Open) that halts network attempts and drops payloads immediately after a defined threshold of consecutive export timeouts. - Export timeout parameters (`exportTimeoutMillis`) MUST be strictly lower than application request timeout budgets, with a recommended maximum between 1500ms and 3000ms. - Batch export failures MUST NEVER trigger uncaught promise rejections or bubble up exceptions into the runtime application execution context. ## Bad vs Good Implementations ### Unbounded OTLP BatchSpanProcessor Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring massive queue limits with prolonged export timeouts causes retained span buffers to survive minor GC cycles into tenured space. When collector degradation occurs, the application quickly hits heap limits and crashes under backpressure. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; // ANTI-PATTERN: Default or massive queue configurations with standard unbounded exporters. // If the OTLP endpoint fails or slows, spans accumulate in memory, leading to high GC pauses // and eventual node process OOM (Out Of Memory) crash. const exporter = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317', }); export const sdk = new NodeSDK({ // Default BatchSpanProcessor allows up to 2048 spans by default with no circuit-breaker. // A high-throughput service (>5k rps) buffering under endpoint latency stalls the V8 event loop. spanProcessor: new BatchSpanProcessor(exporter, { maxQueueSize: 65536, // DANGEROUS: High ceiling consumes critical heap exportTimeoutMillis: 30000, // DANGEROUS: Keeps sockets and buffers open for 30s }), }); ``` #### Production Standard (Correct) > **Standard**: Wraps the low-level OTLP exporter in a fail-fast circuit breaker and enforces a small, deterministic bounded buffer (2048 spans maximum with 2s timeout). Downstream backpressure is instantly neutralized by shedding telemetry, preventing memory spikes and event loop starvation. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { BatchSpanProcessor, SpanExporter, ReadableSpan } from '@opentelemetry/sdk-trace-base'; import { ExportResult, ExportResultCode } from '@opentelemetry/core'; // PRODUCTION FIX: Circuit-breaker wrapper with tight ring-buffer limits and fast-drop semantics class CircuitBreakerSpanExporter implements SpanExporter { private failureCount = 0; private nextAttempt = 0; private readonly failureThreshold = 5; private readonly cooldownMs = 30000; constructor(private readonly underlying: SpanExporter) {} export(spans: ReadableSpan[], resultCallback: (result: ExportResult) => void): void { const now = Date.now(); if (this.failureCount >= this.failureThreshold && now < this.nextAttempt) { // Circuit is OPEN: Drop immediately to protect memory and event loop return resultCallback({ code: ExportResultCode.FAILED, error: new Error('CircuitOpen: Dropping spans') }); } this.underlying.export(spans, (result) => { if (result.code === ExportResultCode.SUCCESS) { this.failureCount = 0; } else { this.failureCount += 1; if (this.failureCount >= this.failureThreshold) { this.nextAttempt = Date.now() + this.cooldownMs; } } resultCallback(result); }); } shutdown(): Promise<void> { return this.underlying.shutdown(); } } const baseExporter = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317', }); export const sdk = new NodeSDK({ spanProcessor: new BatchSpanProcessor(new CircuitBreakerSpanExporter(baseExporter), { maxQueueSize: 2048, // Strict upper memory bound maxExportBatchSize: 512, // Low batch payload size to bound network serialization overhead scheduledDelayMillis: 1000, // Regular deterministic flush cadence exportTimeoutMillis: 2000, // Fast failure: 2s timeout prevents backpressure accumulation }), }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Bounded Ring-Buffer and Circuit-Breaker Guardrails for OTLP Telemetry Pipelines

Bounded Ring-Buffer and Circuit-Breaker Guardrails for OTLP Telemetry Pipelines defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded ring buffers and circuit-breaker drop mechanics on all OpenTelemetry OTLP metric and trace export pipelines. Ensures that downstream collector unavailability or network degradation sheds telemetry load immediately rather than inducing heap exhaustion or blocking the application event loop. Guarantees the primary invariant that observability infrastructure must never compromise system availability. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Telemetry Buffering Causing Heap Exhaustion and Microservice Brownouts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Telemetry Buffering Causing Heap Exhaustion and Microservice Brownouts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure explicit limits on `maxQueueSize` and `maxExportBatchSize` on all OpenTelemetry BatchSpanProcessor and BatchMetricReader instances

Engineers MUST ALWAYS configure explicit limits on `maxQueueSize` and `maxExportBatchSize` on all OpenTelemetry BatchSpanProcessor and BatchMetricReader instances; defaults that allow unbounded memory growth are strictly prohibited.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.