OpenTelemetry OTLP Push Flow-Control Watermark Invariant

--- description: OpenTelemetry OTLP Push Flow-Control Watermark Invariant - Enforces strict high- and low-watermark flow-control policies on OpenTelemetry OTLP metric and span batch processors to prevent process OOM crashes during downstream ingestion degradation. Requires non-blocking enqueue strategies that shed telemetry based on buffer capacity thresholds before exporter thread pools or event loops block. globs: src/**/telemetry.{ts,js}, src/**/tracing.{ts,js}, src/**/instrumentation.{ts,js}, src/**/otel*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-otlp-push-flow-control-watermark-invariant" --- # OpenTelemetry OTLP Push Flow-Control Watermark Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Telemetry Buffering Causing Process Termination via Ingestion Backpressure ## System Constraints You are a Principal Observability and Infrastructure Engineer enforcing resilient telemetry pipeline architecture. You must mandate bounded memory buffers, explicit high-watermark drop strategies, and non-blocking exports across all OpenTelemetry BatchSpanProcessor and PeriodicExportingMetricReader configurations. Never allow infinite queue sizes, unbounded retries, or synchronous blocking exports that saturate application runtime event loops. ## Strict Guidelines - Engineers MUST ALWAYS configure explicit `maxQueueSize` and `maxExportBatchSize` on `BatchSpanProcessor` and `PeriodicExportingMetricReader`. - Engineers MUST NOT allow default unbounded or disproportionately large queues that exceed 10% of total allocated process heap memory under maximum telemetry volume. - Code MUST implement a drop-on-saturation watermark invariant: when the internal queue depth crosses the high-watermark threshold (default 80%), low-priority spans and debug metrics must be shed before buffer exhaustion. - Services MUST configure export timeouts (`exportTimeoutMillis`) strictly lower than the upstream request timeout to avoid cascading latency amplification during exporter connection delays. - Components NEVER block request execution threads or node event loops when the OTLP exporter encounters 429 (Too Many Requests) or network timeouts. ## Bad vs Good Implementations ### OTLP Push Pipeline Configuration with Watermark Backpressure and Bounded Queues #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Defaulting OTLP batch processors without explicit queue boundaries, export timeouts, and buffer ceiling protections leads to unbounded memory growth when the collector is slow or unreachable, eventually crashing the host application via OOM. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; // ANTI-PATTERN: Default configurations allow queues to expand without safety margins, // missing explicit timeouts and watermark shed logic. Under gRPC backpressure, heap expands to OOM. const traceExporter = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317', }); export const sdk = new NodeSDK({ traceExporter, // Uses default BatchSpanProcessor configuration: maxQueueSize=2048, default timeout of 30s // When upstream fails, batches pile up and exporter calls exhaust available sockets. spanProcessor: new BatchSpanProcessor(traceExporter), }); sdk.start(); ``` #### Production Standard (Correct) > **Standard**: Configures an immutable upper bound for queue capacities and export latency. Timeouts are constrained to 5000ms, and queues cap element retention to prevent memory exhaustion when downstream collectors saturate. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { BatchSpanProcessor, BufferConfig } from '@opentelemetry/sdk-trace-base'; import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api'; diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.WARN); // INVARIANT: Max 10% of average heap target. For 512MB heap limit, queue capped at 4096 elements. const MAX_QUEUE_SIZE = 4096; const MAX_BATCH_SIZE = 512; const EXPORT_TIMEOUT_MS = 5000; // Strict bounding prevents socket retention const SCHEDULED_DELAY_MS = 2000; const traceExporter = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317', timeoutMillis: EXPORT_TIMEOUT_MS, }); const bufferConfig: BufferConfig = { maxQueueSize: MAX_QUEUE_SIZE, maxExportBatchSize: MAX_BATCH_SIZE, scheduledDelayMillis: SCHEDULED_DELAY_MS, exportTimeoutMillis: EXPORT_TIMEOUT_MS, }; export const sdk = new NodeSDK({ traceExporter, spanProcessor: new BatchSpanProcessor(traceExporter, bufferConfig), }); // Graceful drain invariant on SIGTERM ensures queues flush within exit budget process.on('SIGTERM', () => { sdk.shutdown() .then(() => process.exit(0)) .catch((err) => { diag.error('Error shutting down OTel SDK', err); process.exit(1); }); }); sdk.start(); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry OTLP Push Flow-Control Watermark Invariant

OpenTelemetry OTLP Push Flow-Control Watermark Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces strict high- and low-watermark flow-control policies on OpenTelemetry OTLP metric and span batch processors to prevent process OOM crashes during downstream ingestion degradation. Requires non-blocking enqueue strategies that shed telemetry based on buffer capacity thresholds before exporter thread pools or event loops block. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Telemetry Buffering Causing Process Termination via Ingestion Backpressure Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Telemetry Buffering Causing Process Termination via Ingestion Backpressure	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure explicit `maxQueueSize` and `maxExportBatchSize` on `BatchSpanProcessor` and `PeriodicExportingMetricReader`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.