AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: otel-collector-adaptive-tail-sampling-memory-limiter-standard

OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard

Enforces coordinated memory management between the memory_limiter processor and tail_sampling trace buffers in OpenTelemetry Collector deployments. Prevents catastrophic OOMKilled crashes and cluster cascading failures during high-cardinality distributed trace bursts by mandating processor ordering, bounded trace cache sizing, and calibrated memory headroom thresholds.

Prohibited Anti-Pattern: Unbounded tail-sampling buffers configured downstream of uncalibrated or misordered memory limiters causing container OOM crashes under trace bursts

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard - Enforces coordinated memory management between the memory_limiter processor and tail_sampling trace buffers in OpenTelemetry Collector deployments. Prevents catastrophic OOMKilled crashes and cluster cascading failures during high-cardinality distributed trace bursts by mandating processor ordering, bounded trace cache sizing, and calibrated memory headroom thresholds.
globs: **/otel-collector*.{yaml,yml}, **/collector/**/*.{yaml,yml}, **/k8s/**/*.{yaml,yml}, **/helm/**/*.{yaml,yml}
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-collector-adaptive-tail-sampling-memory-limiter-standard"
---

# OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Unbounded tail-sampling buffers configured downstream of uncalibrated or misordered memory limiters causing container OOM crashes under trace bursts

## System Constraints
You are a Principal Software Architect enforcing OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standards. You must reject any OpenTelemetry Collector configuration where memory_limiter is missing, placed after tail_sampling, or decoupled from the tail-sampling buffer size calculations. Mandate strict pipeline order, bounded num_traces, hard memory limits calibrated against container cgroups, and proactive backpressure shedding.

## Strict Guidelines
- Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in the traces pipeline, strictly preceding `tail_sampling`.
- Engineers MUST NOT configure unbounded or arbitrarily large `num_traces` in `tail_sampling` without mathematical verification against `memory_limiter.limit_mib` and container cgroup memory limits.
- Collector pipelines MUST define `check_interval` (defaulting to 1s), `limit_percentage` (maximum 75-80%), and `spike_limit_percentage` (minimum 15-20%) within `memory_limiter` to absorb high-velocity allocation bursts.
- Tail-sampling `decision_wait` MUST NOT exceed 10s in high-throughput deployments without scaled memory budgets and distributed trace-id partitioning.
- Services MUST route traces through a load-balancing exporter with trace-ID consistent hashing upstream of tail-sampling nodes to prevent memory skew across collector replicas.
- Components NEVER rely on downstream exporters to provide backpressure to `tail_sampling`; local memory limits must proactively drop or probabilistically pass spans when critical thresholds are breached.

## Bad vs Good Implementations
### Collector Memory Limiter and Tail-Sampling Configuration

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Placing tail_sampling before memory_limiter allows the sampler to buffer millions of spans in heap memory before backpressure is evaluated. A 30s decision wait with 500,000 trace capacity and a 95% limit without a spike limit guarantees the container will exceed its Kubernetes memory limit and be OOMKilled before the 5-second check interval fires.
```yaml
# ANTI-PATTERN: Misordered processors, unbounded trace buffers, and missing spike protection
processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 500000 # Consumes several gigabytes under high cardinality
    expected_new_traces_per_sec: 10000
    policies:
      - name: sample-errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
  memory_limiter:
    # WRONG: Placed after tail_sampling in the pipeline and missing spike limits
    check_interval: 5s
    limit_percentage: 95

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, memory_limiter] # FAILS: tail_sampling buffers before limiter drops
      exporters: [otlp]
```

#### Production Standard (Correct)
> **Standard**: The memory_limiter is placed first in the pipeline with a 1-second check interval, a 75% limit, and a 20% spike limit, providing guaranteed headroom. The tail_sampling buffer is bounded to 30,000 traces with a 5-second decision wait, matched directly to available container RAM.
```yaml
# PRODUCTION STANDARD: Hardened processor ordering, bounded cache, and calibrated spike headroom
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 5s
    num_traces: 30000
    expected_new_traces_per_sec: 2000
    policies:
      - name: filter-errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: rate-limit-probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 5.0 }
  batch:
    send_batch_size: 8192
    timeout: 1s
    send_batch_max_size: 10240

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch] # CORRECT: limiter protects buffer; batch groups output
      exporters: [otlp]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes MCP Server
Inspect collector container memory limits, cgroup throttling, and OOMKilled events in real time.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OpenTelemetry Collector Config Validator
Validates processor configuration pipelines, memory allocations, and component schemas.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard

OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard defines an industrial-grade behavioral standard for devops engineering. Enforces coordinated memory management between the memory_limiter processor and tail_sampling trace buffers in OpenTelemetry Collector deployments. Prevents catastrophic OOMKilled crashes and cluster cascading failures during high-cardinality distributed trace bursts by mandating processor ordering, bounded trace cache sizing, and calibrated memory headroom thresholds. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded tail-sampling buffers configured downstream of uncalibrated or misordered memory limiters causing container OOM crashes under trace bursts Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftUnbounded tail-sampling buffers configured downstream of uncalibrated or misordered memory limiters causing container OOM crashes under trace burstsCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in the traces pipeline, strictly preceding `tail_sampling`.

Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in the traces pipeline, strictly preceding `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT configure unbounded or arbitrarily large `num_traces` in `tail_sampling` without mathematical verification against `memory_limiter.limit_mib` and container cgroup memory limits.

Engineers MUST NOT configure unbounded or arbitrarily large `num_traces` in `tail_sampling` without mathematical verification against `memory_limiter.limit_mib` and container cgroup memory limits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Collector pipelines MUST define `check_interval` (defaulting to 1s), `limit_percentage` (maximum 75-80%), and `spike_limit_percentage` (minimum 15-20%) within `memory_limiter` to absorb high-velocity allocation bursts.

Collector pipelines MUST define `check_interval` (defaulting to 1s), `limit_percentage` (maximum 75-80%), and `spike_limit_percentage` (minimum 15-20%) within `memory_limiter` to absorb high-velocity allocation bursts.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Tail-sampling `decision_wait` MUST NOT exceed 10s in high-throughput deployments without scaled memory budgets and distributed trace-id partitioning.

Tail-sampling `decision_wait` MUST NOT exceed 10s in high-throughput deployments without scaled memory budgets and distributed trace-id partitioning.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Services MUST route traces through a load-balancing exporter with trace-ID consistent hashing upstream of tail-sampling nodes to prevent memory skew across collector replicas.

Services MUST route traces through a load-balancing exporter with trace-ID consistent hashing upstream of tail-sampling nodes to prevent memory skew across collector replicas.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

Components NEVER rely on downstream exporters to provide backpressure to `tail_sampling`

Components NEVER rely on downstream exporters to provide backpressure to `tail_sampling`; local memory limits must proactively drop or probabilistically pass spans when critical thresholds are breached.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Collector Tail-Sampling Adaptive Buffer Limiter Standard critical for AI coding assistants?

Enforces coordinated memory management between the memory_limiter processor and tail_sampling trace buffers in OpenTelemetry Collector deployments. Prevents catastrophic OOMKilled crashes and cluster cascading failures during high-cardinality distributed trace bursts by mandating processor ordering, bounded trace cache sizing, and calibrated memory headroom thresholds. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded tail-sampling buffers configured downstream of uncalibrated or misordered memory limiters causing container OOM crashes under trace bursts) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry