AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: otel-collector-routing-connector-backpressure-invariant

OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant

Enforces declarative routing connectors in OpenTelemetry Collector configurations to dynamically isolate and load-shed trace telemetry during downstream saturation or traffic bursts. Guarantees that pipeline congestion on heavy trace streams fails safely via probabilistic sampling and circuit breaking without destabilizing critical metric and log delivery. Mandates decoupled queue-bounded memory pipelines between receivers and high-cardinality exporters.

Prohibited Anti-Pattern: Monolithic Shared-Pipeline Collector Configuration Lacking Dynamic Routing Connectors and Isolated Memory Queues

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant - Enforces declarative routing connectors in OpenTelemetry Collector configurations to dynamically isolate and load-shed trace telemetry during downstream saturation or traffic bursts. Guarantees that pipeline congestion on heavy trace streams fails safely via probabilistic sampling and circuit breaking without destabilizing critical metric and log delivery. Mandates decoupled queue-bounded memory pipelines between receivers and high-cardinality exporters.
globs: **/collector.yaml, **/otel-collector*.yaml, **/otel-collector*.yml, deploy/k8s/**/*.yaml, deploy/helm/**/*.yaml
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-collector-routing-connector-backpressure-invariant"
---

# OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Monolithic Shared-Pipeline Collector Configuration Lacking Dynamic Routing Connectors and Isolated Memory Queues

## System Constraints
You are a Principal Observability Infrastructure Architect enforcing strict reliability invariants for OpenTelemetry Collector deployments. You must prevent telemetry cascade failures where upstream span bursts exhaust collector memory buffers and trigger unmonitored drop cascades across all telemetry signals. Mandate dynamic routing connectors (`routing` or `spanmetrics` connectors), explicit batch processor bounds, queue-per-pipeline isolation, and memory_limiter invariants across every collector pipeline configuration.

## Strict Guidelines
- Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in every telemetry pipeline to guard against OOMKills before data deserialization buffers expand.
- Engineers MUST NOT multiplex traces and operational metrics through a single monolithic pipeline sharing an unbounded `sending_queue`.
- Engineers MUST configure declarative `routing` connectors to segment high-volume, non-critical trace paths away from high-priority business transactions and SLI/SLO metrics.
- Pipelines forwarding high-volume traces MUST declare tail-based or probabilistic sampling processors upstream of remote exporters to enforce backpressure load shedding.
- Exporters MUST declare explicit, bounded `sending_queue` blocks with `storage` persistence or explicit `queue_size` thresholds; defaulting to unbounded or unmonitored queues is strictly prohibited.
- The `memory_limiter` configuration MUST maintain at least a 20% margin between `check_interval` ceiling and container hard limits (`limit_percentage: 75`, `spike_limit_percentage: 20`).

## Bad vs Good Implementations
### Unbounded Monolithic Pipeline vs. Declarative Routing Connector with Dynamic Backpressure

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: The collector configuration lacks a memory_limiter processor and forwards raw traces directly into a shared exporter queue without traffic classification. If the downstream trace backend throttles, spans accumulate indefinitely in memory until the pod is OOMKilled by Kubernetes, concurrently blinding infrastructure monitoring by dropping operational metrics.
```yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s

exporters:
  otlp/backend:
    endpoint: tempo.internal:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    # CRITICAL FLAW: Single shared trace pipeline with no memory bounds
    # Downstream Tempo throttling triggers OOMKill and drops entire collector buffer
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/backend]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
```

#### Production Standard (Correct)
> **Standard**: Adheres to the dynamic backpressure invariant by placing memory_limiter at the entry point and routing spans through a declarative connector. High-volume background traces are segregated into traces/sampled with an upstream probabilistic load-shedder, while high-priority transactions are isolated into dedicated bounded queues, preventing telemetry loss during upstream burst traffic.
```yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  batch/fast:
    send_batch_size: 8192
    timeout: 200ms
  probabilistic_sampler/shed:
    sampling_percentage: 10.0

connectors:
  routing:
    default_pipelines: [traces/sampled]
    error_mode: ignore
    table:
      - statement: route() where attributes["priority"] == "high"
        pipelines: [traces/priority]

exporters:
  otlp/tempo_priority:
    endpoint: tempo.internal:4317
    sending_queue:
      enabled: true
      num_consumers: 16
      queue_size: 5000
  otlp/tempo_bulk:
    endpoint: tempo.internal:4317
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 1000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [routing]
    traces/priority:
      receivers: [routing]
      processors: [batch/fast]
      exporters: [otlp/tempo_priority]
    traces/sampled:
      receivers: [routing]
      processors: [probabilistic_sampler/shed, batch/fast]
      exporters: [otlp/tempo_bulk]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Cluster Controller MCP
Inspects running OpenTelemetry Collector Pod status, container memory limits, and restart metrics in Kubernetes.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OTel Collector Config Validator
Validates syntactical structure and component compatibility for OpenTelemetry Collector pipeline specifications.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant

OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces declarative routing connectors in OpenTelemetry Collector configurations to dynamically isolate and load-shed trace telemetry during downstream saturation or traffic bursts. Guarantees that pipeline congestion on heavy trace streams fails safely via probabilistic sampling and circuit breaking without destabilizing critical metric and log delivery. Mandates decoupled queue-bounded memory pipelines between receivers and high-cardinality exporters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Monolithic Shared-Pipeline Collector Configuration Lacking Dynamic Routing Connectors and Isolated Memory Queues Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftMonolithic Shared-Pipeline Collector Configuration Lacking Dynamic Routing Connectors and Isolated Memory QueuesCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in every telemetry pipeline to guard against OOMKills before data deserialization buffers expand.

Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in every telemetry pipeline to guard against OOMKills before data deserialization buffers expand.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT multiplex traces and operational metrics through a single monolithic pipeline sharing an unbounded `sending_queue`.

Engineers MUST NOT multiplex traces and operational metrics through a single monolithic pipeline sharing an unbounded `sending_queue`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Engineers MUST configure declarative `routing` connectors to segment high-volume, non-critical trace paths away from high-priority business transactions and SLI/SLO metrics.

Engineers MUST configure declarative `routing` connectors to segment high-volume, non-critical trace paths away from high-priority business transactions and SLI/SLO metrics.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Pipelines forwarding high-volume traces MUST declare tail-based or probabilistic sampling processors upstream of remote exporters to enforce backpressure load shedding.

Pipelines forwarding high-volume traces MUST declare tail-based or probabilistic sampling processors upstream of remote exporters to enforce backpressure load shedding.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Exporters MUST declare explicit, bounded `sending_queue` blocks with `storage` persistence or explicit `queue_size` thresholds

Exporters MUST declare explicit, bounded `sending_queue` blocks with `storage` persistence or explicit `queue_size` thresholds; defaulting to unbounded or unmonitored queues is strictly prohibited.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

The `memory_limiter` configuration MUST maintain at least a 20% margin between `check_interval` ceiling and container hard limits (`limit_percentage

The `memory_limiter` configuration MUST maintain at least a 20% margin between `check_interval` ceiling and container hard limits (`limit_percentage: 75`, `spike_limit_percentage: 20`).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Collector Routing Connector Dynamic Backpressure and Load-Shedding Invariant critical for AI coding assistants?

Enforces declarative routing connectors in OpenTelemetry Collector configurations to dynamically isolate and load-shed trace telemetry during downstream saturation or traffic bursts. Guarantees that pipeline congestion on heavy trace streams fails safely via probabilistic sampling and circuit breaking without destabilizing critical metric and log delivery. Mandates decoupled queue-bounded memory pipelines between receivers and high-cardinality exporters. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Monolithic Shared-Pipeline Collector Configuration Lacking Dynamic Routing Connectors and Isolated Memory Queues) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry