AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: opentelemetry-arrow-columnar-telemetry-pipeline-standard

OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard

Mandates the deployment of Apache Arrow-native encoders and decoders across OpenTelemetry Collector mesh pipelines to replace standard row-oriented OTLP Protobuf serialization over inter-cluster transit. Enforces zero-copy columnar serialization, shared dictionary encoding, and stream-oriented batching, reducing telemetry network egress bills and CPU collector serialization overhead by over 70%.

Prohibited Anti-Pattern: Transporting Cross-Cluster OTLP via Row-Based Protobuf over WAN Without Columnar Arrow Compression

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard - Mandates the deployment of Apache Arrow-native encoders and decoders across OpenTelemetry Collector mesh pipelines to replace standard row-oriented OTLP Protobuf serialization over inter-cluster transit. Enforces zero-copy columnar serialization, shared dictionary encoding, and stream-oriented batching, reducing telemetry network egress bills and CPU collector serialization overhead by over 70%.
globs: **/collector*.yaml, **/collector*.yml, **/otel*.yaml, **/otel*.yml, **/helm/**/values*.yaml, **/manifests/**/*.yaml
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/opentelemetry-arrow-columnar-telemetry-pipeline-standard"
---

# OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Transporting Cross-Cluster OTLP via Row-Based Protobuf over WAN Without Columnar Arrow Compression

## System Constraints
You are a Principal Observability Infrastructure Architect enforcing the OpenTelemetry Arrow Protocol standard. Your task is to mandate Apache Arrow-native columnar telemetry serialization across all multi-tier and inter-cluster OpenTelemetry Collector configurations. Ensure all inter-cluster, cross-VPC, and cross-region collectors leverage the otelarrow exporter/receiver or OTLP Arrow extensions with stream-level dictionary encoding, strict memory limits, and zero-copy serialization buffers.

## Strict Guidelines
- Engineers MUST ALWAYS configure `otelarrow` exporters and receivers for any telemetry pipeline crossing WAN, inter-region, or inter-cluster network boundaries.
- Engineers MUST NOT transmit raw row-based OTLP/gRPC or OTLP/HTTP across metered network boundaries when transferring high-volume logs, metrics, or spans between collectors.
- Collector pipelines MUST configure dynamic dictionary encoding to minimize repeating strings (such as resource attributes, metric names, and span scopes) before streaming Arrow record batches.
- Pipelines SHOULD configure `batch` and `memory_limiter` processors directly upstream of Arrow exporters to enforce deterministic record batch boundaries and guard against unbuffered heap spikes.
- Receivers MUST define admission memory limits (`arrow.max_stream_lifecycle` and concurrent stream quotas) to prevent resource exhaustion from long-lived multiplexed Arrow streams.
- Engineers MUST ensure mutual TLS (mTLS) or robust token authentication is enabled alongside Arrow streaming to secure bidirectional gRPC transit between edge agents and core gateway collectors.

## Bad vs Good Implementations
### OTel Collector Cross-Cluster Transit Configuration

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Using generic OTLP gRPC across WAN boundaries causes repeated string attributes (host, service, namespace, span names) to be redundantly encoded into individual Protobuf messages. This triggers high CPU deserialization overhead at the core cluster and incurs steep cloud egress network costs.
```yaml
# Anti-Pattern: Unoptimized standard row-oriented OTLP gRPC across high-cost WAN
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlp/inter_cluster:
    endpoint: core-gateway.telemetry.internal:4317
    tls:
      insecure: false
    # Incurs full row-based Protobuf CPU serialization and high byte overhead per span/log

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/inter_cluster]
```

#### Production Standard (Correct)
> **Standard**: Employs the otelarrow exporter configured with columnar record batching, multi-stream distribution, Zstandard compression, and zero-copy dictionary recycling. This compresses string dictionaries globally across stream lifecycles, yielding >70% reductions in egress network payload and receiver CPU overhead.
```yaml
# Production Standard: OpenTelemetry Arrow Columnar Protocol with Shared Dictionaries
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    timeout: 500ms
    send_batch_max_size: 8192
    send_batch_size: 4096

exporters:
  otelarrow:
    endpoint: core-gateway.telemetry.internal:4317
    arrow:
      memory_limit_mib: 512
      num_streams: 4
    compression: zstd
    tls:
      ca_file: /etc/otel/certs/ca.pem
      cert_file: /etc/otel/certs/client.pem
      key_file: /etc/otel/certs/client-key.pem

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otelarrow]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Cluster Inspector
Inspects running OpenTelemetry Collector DaemonSets and Deployments to verify runtime configuration and memory limits.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

ocb-cli
Validates custom OpenTelemetry Collector manifests to verify the inclusion of github.com/open-telemetry/opentelemetry-collector-contrib/exporter/otelarrowexporter and receiver counterparts.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard

OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard defines an industrial-grade behavioral standard for devops engineering. Mandates the deployment of Apache Arrow-native encoders and decoders across OpenTelemetry Collector mesh pipelines to replace standard row-oriented OTLP Protobuf serialization over inter-cluster transit. Enforces zero-copy columnar serialization, shared dictionary encoding, and stream-oriented batching, reducing telemetry network egress bills and CPU collector serialization overhead by over 70%. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Transporting Cross-Cluster OTLP via Row-Based Protobuf over WAN Without Columnar Arrow Compression Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftTransporting Cross-Cluster OTLP via Row-Based Protobuf over WAN Without Columnar Arrow CompressionCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS configure `otelarrow` exporters and receivers for any telemetry pipeline crossing WAN, inter-region, or inter-cluster network boundaries.

Engineers MUST ALWAYS configure `otelarrow` exporters and receivers for any telemetry pipeline crossing WAN, inter-region, or inter-cluster network boundaries.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT transmit raw row-based OTLP/gRPC or OTLP/HTTP across metered network boundaries when transferring high-volume logs, metrics, or spans between collectors.

Engineers MUST NOT transmit raw row-based OTLP/gRPC or OTLP/HTTP across metered network boundaries when transferring high-volume logs, metrics, or spans between collectors.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Collector pipelines MUST configure dynamic dictionary encoding to minimize repeating strings (such as resource attributes, metric names, and span scopes) before streaming Arrow record batches.

Collector pipelines MUST configure dynamic dictionary encoding to minimize repeating strings (such as resource attributes, metric names, and span scopes) before streaming Arrow record batches.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Pipelines SHOULD configure `batch` and `memory_limiter` processors directly upstream of Arrow exporters to enforce deterministic record batch boundaries and guard against unbuffered heap spikes.

Pipelines SHOULD configure `batch` and `memory_limiter` processors directly upstream of Arrow exporters to enforce deterministic record batch boundaries and guard against unbuffered heap spikes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Receivers MUST define admission memory limits (`arrow.max_stream_lifecycle` and concurrent stream quotas) to prevent resource exhaustion from long-lived multiplexed Arrow streams.

Receivers MUST define admission memory limits (`arrow.max_stream_lifecycle` and concurrent stream quotas) to prevent resource exhaustion from long-lived multiplexed Arrow streams.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

Engineers MUST ensure mutual TLS (mTLS) or robust token authentication is enabled alongside Arrow streaming to secure bidirectional gRPC transit between edge agents and core gateway collectors.

Engineers MUST ensure mutual TLS (mTLS) or robust token authentication is enabled alongside Arrow streaming to secure bidirectional gRPC transit between edge agents and core gateway collectors.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Arrow Protocol Zero-Copy Columnar Telemetry Pipeline Standard critical for AI coding assistants?

Mandates the deployment of Apache Arrow-native encoders and decoders across OpenTelemetry Collector mesh pipelines to replace standard row-oriented OTLP Protobuf serialization over inter-cluster transit. Enforces zero-copy columnar serialization, shared dictionary encoding, and stream-oriented batching, reducing telemetry network egress bills and CPU collector serialization overhead by over 70%. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Transporting Cross-Cluster OTLP via Row-Based Protobuf over WAN Without Columnar Arrow Compression) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry