OpenTelemetry Arrow Stream Memory-Bounded Ingest Guardrail

--- description: OpenTelemetry Arrow Stream Memory-Bounded Ingest Guardrail - Enforces hard memory-bounded ring buffer allocations and upstream gRPC flow-control backpressure on high-throughput OTLP Arrow columnar streams. Prevents Linux Out-Of-Memory (OOM) killer evictions and collector process termination during sudden cross-cluster telemetry surges by rejecting or throttling unbuffered record batches prior to deserialization. globs: src/telemetry/**/*.ts, src/collector/**/*.ts, src/ingest/**/*.ts, config/otel/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-arrow-ingest-backpressure-guardrail" --- # OpenTelemetry Arrow Stream Memory-Bounded Ingest Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Dynamic Batch Ingestion Leading to Native Heap Exhaustion and Collector SIGKILL ## System Constraints You are a Principal Engineer enforcing the OpenTelemetry Arrow-Encoded Stream Ingest Backpressure Guardrail. Mandate that all Arrow telemetry ingestion pipelines enforce strict byte-bounded ring buffer limits, preemptive memory watermark checks, and immediate gRPC backpressure propagation (RESOURCE_EXHAUSTED) to prevent kernel OOM killer termination during traffic spikes. ## Strict Guidelines - Engineers MUST ALWAYS bound telemetry queues by physical byte size rather than item/record count. - Engineers MUST NOT allocate unbounded in-memory queues or dynamic arrays to hold raw Apache Arrow RecordBatches. - Code SHOULD enforce dual high/low memory watermarks to trigger upstream flow-control throttle and resume transitions. - Services MUST signal upstream senders with gRPC status code 8 (RESOURCE_EXHAUSTED) or TCP stream pauses when buffers exceed the high watermark. - Components NEVER hold references to deserialized Arrow record dictionaries once dispatched to pipeline processors. ## Bad vs Good Implementations ### Unbounded Item-Count Queue vs. Memory-Bounded Ring Buffer with Upstream Backpressure #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The handler limits queue capacity strictly by batch count rather than byte allocation. Wide columnar Arrow record batches can vary by orders of magnitude in physical byte size, causing total container resident memory (RSS) to breach cgroup limits and trigger an unrecoverable SIGKILL before reaching the MAX_BATCHES limit. ```typescript import { RecordBatch } from 'apache-arrow'; export class ArrowIngestionHandler { // ANTI-PATTERN: Queue is unbounded by byte size; record count metric is deceptive for columnar data private queue: RecordBatch[] = []; private readonly MAX_BATCHES = 1000; public async handleBatch(batch: RecordBatch): Promise<void> { if (this.queue.length >= this.MAX_BATCHES) { // Silently drops or blocks without upstream gRPC signal console.warn('Queue full, dropping batch'); return; } // Memory spikes catastrophically if batches contain wide string dictionaries this.queue.push(batch); } public dequeue(): RecordBatch | undefined { return this.queue.shift(); } } ``` #### Production Standard (Correct) > **Standard**: Tracks concrete buffer memory footprints via the Arrow internal chunk descriptors, enforces hard byte capacity ceilings alongside slot counts, and throws an explicit IngestExhaustionError (mapping to gRPC RESOURCE_EXHAUSTED) at the configured high watermark to force sender backpressure. ```typescript import { RecordBatch } from 'apache-arrow'; export class IngestExhaustionError extends Error { public readonly grpcCode = 8; // gRPC RESOURCE_EXHAUSTED constructor(message: string) { super(message); this.name = 'IngestExhaustionError'; } } export class MemoryBoundedArrowRingBuffer { private ring: (RecordBatch | null)[]; private head = 0; private tail = 0; private count = 0; private currentByteSize = 0; constructor( public readonly capacity: number, public readonly maxByteLimit: number, public readonly highWatermarkRatio: number = 0.85, public readonly lowWatermarkRatio: number = 0.50 ) { this.ring = new Array<RecordBatch | null>(capacity).fill(null); } public push(batch: RecordBatch): void { const batchBytes = batch.data.reduce((acc, data) => acc + data.byteLength, 0); const highWatermarkBytes = this.maxByteLimit * this.highWatermarkRatio; if (this.count === this.capacity || this.currentByteSize + batchBytes > highWatermarkBytes) { throw new IngestExhaustionError( `Ingest buffer saturated: current=${this.currentByteSize}B, incoming=${batchBytes}B, max=${this.maxByteLimit}B` ); } this.ring[this.tail] = batch; this.tail = (this.tail + 1) % this.capacity; this.count++; this.currentByteSize += batchBytes; } public pop(): RecordBatch | null { if (this.count === 0) return null; const batch = this.ring[this.head]; this.ring[this.head] = null; this.head = (this.head + 1) % this.capacity; this.count--; if (batch) { const batchBytes = batch.data.reduce((acc, data) => acc + data.byteLength, 0); this.currentByteSize = Math.max(0, this.currentByteSize - batchBytes); } return batch; } public get memoryUtilization(): number { return this.currentByteSize / this.maxByteLimit; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Arrow Stream Memory-Bounded Ingest Guardrail

OpenTelemetry Arrow Stream Memory-Bounded Ingest Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces hard memory-bounded ring buffer allocations and upstream gRPC flow-control backpressure on high-throughput OTLP Arrow columnar streams. Prevents Linux Out-Of-Memory (OOM) killer evictions and collector process termination during sudden cross-cluster telemetry surges by rejecting or throttling unbuffered record batches prior to deserialization. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Dynamic Batch Ingestion Leading to Native Heap Exhaustion and Collector SIGKILL Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Dynamic Batch Ingestion Leading to Native Heap Exhaustion and Collector SIGKILL	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS bound telemetry queues by physical byte size rather than item/record count.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.