OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Arrow In-Memory Record Accumulation with Missing or Misordered Memory Limiter Processors) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard

Enforces strict memory-bounded columnar batching, upstream gRPC backpressure, and deterministic memory limiter positioning in OpenTelemetry Collector topologies utilizing OTel-Arrow. Guarantees that high-throughput columnar transformations (row-to-record conversion) cannot breach container cgroup memory limits during multi-gigabit metric and trace bursts. Mandates early pipeline admission control, bounded stream allocations, and coordinated drain policies across downstream Arrow exporters.

--- description: OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard - Enforces strict memory-bounded columnar batching, upstream gRPC backpressure, and deterministic memory limiter positioning in OpenTelemetry Collector topologies utilizing OTel-Arrow. Guarantees that high-throughput columnar transformations (row-to-record conversion) cannot breach container cgroup memory limits during multi-gigabit metric and trace bursts. Mandates early pipeline admission control, bounded stream allocations, and coordinated drain policies across downstream Arrow exporters. globs: **/collector*.{yaml,yml}, **/otel*.{yaml,yml}, **/otel-collector*.{yaml,yml}, deployments/**/*.{yaml,yml}, k8s/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-arrow-columnar-backpressure-standard" --- # OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Arrow In-Memory Record Accumulation with Missing or Misordered Memory Limiter Processors ## System Constraints You are a Principal Software Architect enforcing the OTel Arrow Columnar Pipeline Backpressure Standard. Mandate that every OpenTelemetry Collector pipeline deploying Arrow columnar protocols strictly maintains memory safety invariants: the memory_limiter processor MUST be the first processor in every pipeline; Arrow receivers MUST configure explicit admission concurrency limits (max_concurrent_streams) and memory allocations; batch processors MUST enforce send_batch_max_size to avoid heap ballooning during Arrow RecordBatch synthesis; and upstream gRPC transport MUST actively push back via RESOURCE_EXHAUSTED status instead of accepting unbuffered payloads. ## Strict Guidelines - Pipelines MUST ALWAYS position `memory_limiter` as the very first entry under `processors` in traces, metrics, and logs pipelines to reject payloads before columnar Arrow deserialization allocates heap memory. - Collectors MUST NEVER configure `batch` processor without an explicit, non-zero `send_batch_max_size` that is strictly greater than or equal to `send_batch_size`. - The `memory_limiter` configuration MUST specify both `check_interval` (recommended <= 1s) and explicit `limit_mib` and `spike_limit_mib` calculated directly from the container cgroup limits (limit_mib should not exceed 80% of container RAM; spike_limit_mib should reserve 20%). - Receivers with Arrow support (`otelarrow`, `otlp` with arrow enabled) MUST configure bounded stream settings (`max_concurrent_streams`) and explicit memory limiters in their Arrow engine settings. - Arrow exporters MUST specify `sending_queue` with `enabled: true`, a finite `queue_size`, and `storage: null` (or bounded persistent storage) to prevent unbounded goroutine and channel accumulation under network degradation. - Engineers MUST NOT rely on default, zero-value OTel buffer sizes when ingesting telemetry exceeding 1 Gbps per collector replica. ## Bad vs Good Implementations ### Collector Pipeline: Unbounded Arrow Batching vs Memory-Bounded Guardrail #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing memory_limiter after batch forces the collector to already deserialize and accumulate records in memory before evaluating memory thresholds. The lack of send_batch_max_size allows multi-gigabyte batch accumulation during Arrow record synthesis, causing kernel OOM kills (exit code 137). ```yaml # Fatal Anti-Pattern: Missing memory_limiter at pipeline start, unbounded batch size, # and unconstrained otelarrow receiver causing OOM kills under burst traffic. receivers: otelarrow: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: timeout: 5s send_batch_size: 10000 # Missing send_batch_max_size allows infinite batch inflation memory_limiter: # ANTI-PATTERN: Placed AFTER batch processor; OOM occurs during batch aggregation check_interval: 5s limit_percentage: 95 exporters: otelarrow: endpoint: downstream-collector:4317 tls: insecure: true service: pipelines: metrics: receivers: [otelarrow] processors: [batch, memory_limiter] exporters: [otelarrow] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter runs prior to batching and Arrow columnar transformation, rejecting incoming traffic via gRPC backpressure (RESOURCE_EXHAUSTED) before allocation occurs. Explicit batch bounds and finite exporter queues guarantee absolute memory upper bounds under multi-gigabit bursts. ```yaml # Production Standard: Memory limiter placed first, strict Arrow receiver concurrency, # hard-capped batch sizes, and bounded exporter queues with drop/backpressure signaling. receivers: otelarrow: protocols: grpc: endpoint: 0.0.0.0:4317 max_concurrent_streams: 1024 include_metadata: true admission: request_limit_mib: 128 processors: memory_limiter: # MUST be the first processor in the pipeline check_interval: 500ms limit_mib: 3200 # 80% of 4096MiB cgroup limit spike_limit_mib: 800 # 20% reserved for burst handling and GC headroom batch: timeout: 1s send_batch_size: 8192 send_batch_max_size: 10240 exporters: otelarrow: endpoint: downstream-collector:4317 tls: insecure: false sending_queue: enabled: true num_consumers: 8 queue_size: 2000 retry_on_failure: enabled: true initial_interval: 1s max_interval: 10s max_elapsed_time: 30s service: pipelines: metrics: receivers: [otelarrow] processors: [memory_limiter, batch] exporters: [otelarrow] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard

OTel Arrow Columnar Pipeline Memory-Bounded Backpressure Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded columnar batching, upstream gRPC backpressure, and deterministic memory limiter positioning in OpenTelemetry Collector topologies utilizing OTel-Arrow. Guarantees that high-throughput columnar transformations (row-to-record conversion) cannot breach container cgroup memory limits during multi-gigabit metric and trace bursts. Mandates early pipeline admission control, bounded stream allocations, and coordinated drain policies across downstream Arrow exporters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Arrow In-Memory Record Accumulation with Missing or Misordered Memory Limiter Processors Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Arrow In-Memory Record Accumulation with Missing or Misordered Memory Limiter Processors	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST ALWAYS position `memory_limiter` as the very first entry under `processors` in traces, metrics, and logs pipelines to reject payloads before columnar Arrow deserialization allocates heap memory.

Pipelines MUST ALWAYS position `memory_limiter` as the very first entry under `processors` in traces, metrics, and logs pipelines to reject payloads before columnar Arrow deserialization allocates heap memory.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.