OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Arrow RecordBatch Allocation with Decoupled Push Exporters Leading to Collector Node OOM) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard

--- description: OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard - Enforces strict heap-bounded memory allocation and adaptive credit-based stream throttling across OpenTelemetry Arrow protocol pipelines. Prohibits unbound RecordBatch construction and downstream push buffers lacking reactive admission control. Prevents fatal Out-Of-Memory (OOM) collector termination during hyper-dimensional metric and trace cardinality spikes. globs: **/collector/*.go, **/exporter/arrowexporter/*.go, **/receiver/arrowreceiver/*.go, **/config/collector.yaml, **/otelcol/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-arrow-stream-backpressure-standard" --- # OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Arrow RecordBatch Allocation with Decoupled Push Exporters Leading to Collector Node OOM ## System Constraints You are a Principal Systems Architect specializing in telemetry stream processing, Apache Arrow columnar pipelines, and OpenTelemetry Collector resilience. Your role is to mandate strict memory invariants and backpressure mechanics across Arrow stream exporters, receivers, and batch processors. You strictly reject implementations that ingest telemetry without downstream credit checks or that create unbounded Arrow RecordBatches without integrating with the Arrow memory allocator's hard limit. ## Strict Guidelines - Engineers MUST ALWAYS enforce a hard memory budget on the underlying Arrow allocator (e.g., memory.NewCheckedAllocator or memory.NewGoAllocator with dynamic limit wrapping) configured below the container cgroup limit. - Engineers MUST ALWAYS integrate reactive stream backpressure using gRPC flow control or explicit token-bucket credit return loops before building new Arrow RecordBatches. - Engineers MUST NOT decouple ingestion queues from Arrow stream state using unbounded in-memory channels or naive batch processors that ignore downstream buffer saturation. - Engineers MUST ALWAYS configure the OpenTelemetry Memory Limiter Processor BEFORE the Arrow encoder pipeline to drop or return status ResourceExhausted (code 8) before memory allocation fails. - Engineers MUST ALWAYS ensure Arrow RecordBatches call Release() deterministically using deferred cleanup or explicit reference-counted lifecycle wrappers once emitted to the wire. ## Bad vs Good Implementations ### Unbounded Arrow Stream Batching vs. Budget-Aware Credit-Gated Arrow Stream Pipeline #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The bad example allocates Apache Arrow RecordBatches on the default allocator without validating available heap margins or checking downstream gRPC flow control credits. In bursts of high cardinality, multiple concurrent ExportTraces calls trigger explosive heap allocation faster than Go runtime GC or wire transmission can drain, resulting in unrecoverable SIGKILL from the OS cgroup OOM killer. ```go package exporter import ( "context" "github.com/apache/arrow/go/v14/arrow/memory" "go.opentelemetry.io/collector/pdata/ptrace" ) type ArrowExporter struct { allocator memory.Allocator } // Bad: Allocates record batches without checking memory limits or downstream credits func (e *ArrowExporter) ExportTraces(ctx context.Context, td ptrace.Traces) error { // Unchecked allocation: High-cardinality bursts trigger uncontrollable heap expansion record := buildArrowRecord(e.allocator, td) // Asynchronously pushed without verifying if receiver has consumed prior frames go func() { _ = sendOverWire(record) record.Release() // Release is deferred to an unmonitored goroutine }() return nil } ``` #### Production Standard (Correct) > **Standard**: The good example uses an explicit token-credit mechanism tied to downstream stream receipt, enforces maximum byte ceilings on the Arrow CheckedAllocator, and synchronously transfers records while guaranteeing immediate release. Downstream saturation immediately manifests as gRPC ResourceExhausted errors, protecting the node from OOM. ```go package exporter import ( "context" "errors" "github.com/apache/arrow/go/v14/arrow" "github.com/apache/arrow/go/v14/arrow/memory" "go.opentelemetry.io/collector/pdata/ptrace" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) type ArrowStreamExporter struct { allocator *memory.CheckedAllocator creditQueue chan struct{} // Flow control tokens bounded to downstream buffer depth } func (e *ArrowStreamExporter) ExportTraces(ctx context.Context, td ptrace.Traces) error { // 1. Enforce stream backpressure via credit acquisition select { case <-e.creditQueue: // Credit acquired, proceed with batching case <-ctx.Done(): return ctx.Err() default: // Downstream is saturated; reject early to trigger upstream queueing/backpressure return status.Error(codes.ResourceExhausted, "arrow stream backpressure limit reached") } // 2. Bound memory check before record allocation if e.allocator.AllocatedBytes() > maxArrowHeapLimit { e.creditQueue <- struct{}{} // Return credit return status.Error(codes.ResourceExhausted, "arrow allocator memory ceiling exceeded") } record, err := buildArrowRecordSafe(e.allocator, td) if err != nil { e.creditQueue <- struct{}{} return err } defer record.Release() // 3. Synchronous handoff to stream sender return e.sendSynchronousStream(ctx, record) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard

OpenTelemetry Arrow Protocol Columnar Record Batch Backpressure Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict heap-bounded memory allocation and adaptive credit-based stream throttling across OpenTelemetry Arrow protocol pipelines. Prohibits unbound RecordBatch construction and downstream push buffers lacking reactive admission control. Prevents fatal Out-Of-Memory (OOM) collector termination during hyper-dimensional metric and trace cardinality spikes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Arrow RecordBatch Allocation with Decoupled Push Exporters Leading to Collector Node OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Arrow RecordBatch Allocation with Decoupled Push Exporters Leading to Collector Node OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce a hard memory budget on the underlying Arrow allocator (e.g., memory.NewCheckedAllocator or memory.NewGoAllocator with dynamic limit wrapping) configured below the container cgroup limit.

Engineers MUST ALWAYS enforce a hard memory budget on the underlying Arrow allocator (e.g., memory.NewCheckedAllocator or memory.NewGoAllocator with dynamic limit wrapping) configured below the container cgroup limit.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.