OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded In-Memory Columnar Batching with Static Arrow Dictionary Vectors and Unchecked Backpressure) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard

--- description: OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard - Enforces deterministic memory allocation, dictionary lifecycle recycling, and backpressure-governed micro-batching for high-cardinality OTLP/Arrow streaming pipelines. Prevents catastrophic JVM/V8/Go heap fragmentation, unconstrained dictionary vector growth, and silent telemetry dropouts caused by asynchronous buffer starvation under burst loads. globs: src/telemetry/**/*.{ts,js,go}, src/observability/**/*.{ts,js,go}, config/otel-collector*.{yaml,yml}, deploy/k8s/otel*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otlp-arrow-streaming-microbatching-production-standard" --- # OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded In-Memory Columnar Batching with Static Arrow Dictionary Vectors and Unchecked Backpressure ## System Constraints You are a Principal Software Architect enforcing the OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard. Mandate explicit memory pool boundaries, strict Arrow dictionary eviction intervals, and backpressure propagation across all OTLP/Arrow columnar ingestion pipelines. ## Strict Guidelines - Engineers MUST ALWAYS enforce dual batch termination thresholds based on both maximum duration (e.g., <= 200ms) and aggregate byte size (e.g., <= 4MB), never relying solely on record counts. - Engineers MUST NOT permit unbounded dictionary vector retention across micro-batch boundaries; Arrow dictionaries MUST be reset or pruned when unique cardinality exceeds predefined slab limits. - Pipelines MUST propagate downstream backpressure synchronously to the upstream emitter using gRPC stream flow control or bounded reactive buffers, rejecting payloads with standard retryable codes (e.g., UNAVAILABLE / 429) rather than caching indefinitely. - Memory allocators MUST use pre-allocated slab or pooled off-heap allocators for Arrow RecordBatch generation to eliminate heap fragmentation under continuous reallocation. - Services MUST monitor and alert on `arrow_memory_allocated_bytes`, `arrow_batches_in_flight`, and `otlp_arrow_dropped_spans_total` to detect micro-batch saturation prior to OOM cascades. ## Bad vs Good Implementations ### OTLP/Arrow Micro-Batch Buffer with Pre-Allocated Sizing and Downstream Flow Control #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Queuing arbitrary JavaScript objects without byte limits triggers rapid heap fragmentation under high-cardinality telemetry. Converting entire arrays at arbitrary record counts ignores downstream stream backpressure and causes unhandled memory bloat. ```typescript import { RecordBatchStreamWriter, Table } from 'apache-arrow'; // ANTI-PATTERN: Unbounded in-memory queuing, no byte-size budgeting, // unbounded dictionary allocation causing V8 heap fragmentation and OOM drops. export class TelemetryIngestionBuffer { private queue: any[] = []; private writer: RecordBatchStreamWriter; constructor(destinationStream: NodeJS.WritableStream) { this.writer = RecordBatchStreamWriter.writeAll(destinationStream); } public async ingest(record: Record<string, unknown>): Promise<void> { // Naive unbounded push: absorbs traffic bursts indefinitely this.queue.push(record); if (this.queue.length >= 5000) { // Allocates new vectors on the heap without recycling buffers or checking memory limits const table = Table.from(this.queue); await this.writer.write(table); this.queue = []; // Forces heavy GC sweep of deallocated row objects } } } ``` #### Production Standard (Correct) > **Standard**: Enforces a hard 4MB memory boundary, uses typed Apache Arrow column builders to eliminate dynamic allocations, implements temporal batch flushing (200ms), and returns flow-control booleans back to the caller when the downstream consumer cannot keep up. ```typescript import { Builder, RecordBatchStreamWriter, makeBuilder, Utf8, Int64 } from 'apache-arrow'; // PRODUCTION INVARIANT: Bounded memory slab, explicit byte-budget micro-batching, // active backpressure propagation, and reusable column builders. export class ArrowMicroBatchStreamer { private readonly maxBytes: number = 4 * 1024 * 1024; // 4MB maximum batch size private readonly maxIntervalMs: number = 200; private currentBytes: number = 0; private flushTimer: NodeJS.Timeout | null = null; private isFlushing: boolean = false; constructor( private readonly writer: RecordBatchStreamWriter, private readonly nameBuilder = makeBuilder({ type: new Utf8() }), private readonly timestampBuilder = makeBuilder({ type: new Int64() }) ) {} public async ingest(name: string, timestampNs: bigint): Promise<boolean> { const estimatedRecordSize = name.length + 8; if (this.currentBytes + estimatedRecordSize > this.maxBytes) { const flushed = await this.flush(); if (!flushed) { // Propagate upstream backpressure: buffer full & downstream busy return false; } } this.nameBuilder.append(name); this.timestampBuilder.append(timestampNs); this.currentBytes += estimatedRecordSize; if (!this.flushTimer) { this.flushTimer = setTimeout(() => void this.flush(), this.maxIntervalMs); } return true; } public async flush(): Promise<boolean> { if (this.isFlushing || this.currentBytes === 0) return true; this.isFlushing = true; if (this.flushTimer) { clearTimeout(this.flushTimer); this.flushTimer = null; } try { const names = this.nameBuilder.finish().toVector(); const timestamps = this.timestampBuilder.finish().toVector(); this.nameBuilder.clear(); this.timestampBuilder.clear(); this.currentBytes = 0; // Enforce stream-level backpressure const canAcceptMore = this.writer.write({ name: names, timestamp: timestamps }); return canAcceptMore; } finally { this.isFlushing = false; } } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard

OpenTelemetry OTLP/Arrow Streaming Ingestion Micro-Batching Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory allocation, dictionary lifecycle recycling, and backpressure-governed micro-batching for high-cardinality OTLP/Arrow streaming pipelines. Prevents catastrophic JVM/V8/Go heap fragmentation, unconstrained dictionary vector growth, and silent telemetry dropouts caused by asynchronous buffer starvation under burst loads. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded In-Memory Columnar Batching with Static Arrow Dictionary Vectors and Unchecked Backpressure Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded In-Memory Columnar Batching with Static Arrow Dictionary Vectors and Unchecked Backpressure	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce dual batch termination thresholds based on both maximum duration (e.g., <= 200ms) and aggregate byte size (e.g., <= 4MB), never relying solely on record counts.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.