OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard | AgentRules.org

Rules Stacks Clients Auditor Pruner Self-Heal RFCs Benchmark CLI CI/CD Authors Submit

This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →

devopscritical PriorityID: otel-tail-sampling-adaptive-memory-spillover-standard

OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard

Enforces deterministic memory safety in OpenTelemetry Collector deployments running tail-sampling processors. Prevents catastrophic OOMKilled crashes during trace volume surges by mandating upstream memory-limiter processors, bounded in-flight trace limits, and persistent disk-backed storage extensions for buffer spillover.

Prohibited Anti-Pattern: Unbounded In-Memory Tail-Sampling Buffer Leading to Node Eviction and Collector OOMKills

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard - Enforces deterministic memory safety in OpenTelemetry Collector deployments running tail-sampling processors. Prevents catastrophic OOMKilled crashes during trace volume surges by mandating upstream memory-limiter processors, bounded in-flight trace limits, and persistent disk-backed storage extensions for buffer spillover.
globs: **/otel-collector*.{yaml,yml}, **/collector*.{yaml,yml}, **/otelcol/**/*.{yaml,yml}, **/k8s/**/otel*.{yaml,yml}
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-tail-sampling-adaptive-memory-spillover-standard"
---

# OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Unbounded In-Memory Tail-Sampling Buffer Leading to Node Eviction and Collector OOMKills

## System Constraints
You are a Principal DevOps and Observability Infrastructure Architect enforcing memory safety and high-availability standards for OpenTelemetry Collector tail-sampling pipelines. Mandate strict ordering of memory_limiter processors, explicit trace capacity boundaries, cgroup-aligned GOMEMLIMIT configurations, and disk-backed queue spillover.

## Strict Guidelines
- Pipelines MUST ALWAYS position the `memory_limiter` processor before the `tail_sampling` processor in the trace pipeline processor sequence.
- The `tail_sampling` processor MUST define `num_traces` with an explicit upper bound calculated against peak span volume and memory headroom.
- The `tail_sampling` processor MUST NOT set `decision_wait` to excessive durations (>30s) without an active persistent spillover extension.
- Deployments MUST configure `file_storage` or persistent volume-backed queue extensions when buffers must survive transient upstream downstream backpressure.
- Kubernetes pod definitions MUST set `GOMEMLIMIT` to approximately 80% of the container cgroup memory limit to allow Go runtime garbage collection preemptive execution before cgroup termination.
- The `memory_limiter` configuration MUST declare `check_interval` (default 1s), `limit_percentage` (max 80%), and `spike_limit_percentage` (max 20%).

## Bad vs Good Implementations
### Unbounded Tail-Sampling Without Memory Guards (Vulnerable to OOM)

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: The tail_sampling processor retains spans in memory for 60 seconds with no upper bound on total in-memory traces (num_traces). During sudden traffic spikes, in-flight trace trees consume all heap memory. Because memory_limiter is absent, the collector process exceeds its Kubernetes cgroup memory ceiling and gets killed by the Linux OOM killer.
```yaml
# ANTI-PATTERN: tail_sampling declared without memory_limiter,
# unbounded num_traces, long decision_wait, and missing storage extensions.
processors:
  tail_sampling:
    decision_wait: 60s
    expected_new_traces_per_sec: 5000
    policies:
      - name: filter-errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch] # Missing memory_limiter upstream
      exporters: [otlp]
```

#### Production Standard (Correct)
> **Standard**: Memory safety is enforced across layers: memory_limiter triggers proactive load-shedding before heap exhaustion; tail_sampling enforces an upper limit of 50,000 concurrent traces and limits buffer duration to 10s; and the sending queue spills persistent data to file_storage during downstream backpressure.
```yaml
# PRODUCTION STANDARD: Pre-sampler memory_limiter, bounded num_traces,
# disk spillover via file_storage, and short decision windows.
extensions:
  file_storage:
    directory: /var/lib/otelcol/spool
    compaction:
      on_rebound: true

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 5000
    policies:
      - name: filter-errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: filter-latency
        type: numeric_attribute
        numeric_attribute: { key: http.status_code, value_condition: { greater_than_or_equal: 500 } }

  batch:
    send_batch_size: 8192
    timeout: 1s

exporters:
  otlp:
    endpoint: tempo:4317
    sending_queue:
      enabled: true
      storage: file_storage

service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]
```

Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Cluster Inspector

Inspects live OTel collector Pod OOMKilled events, container memory limits, and restart metrics.

Prometheus Metrics Server

Queries otelcol_processor_tail_sampling_count_traces_sampled and otelcol_process_memory_rss metrics to detect memory pressure.

Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OpenTelemetry Collector Config Validator

CLI validator that enforces correct processor ordering and bounds checking on collector YAML manifests.

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard

OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory safety in OpenTelemetry Collector deployments running tail-sampling processors. Prevents catastrophic OOMKilled crashes during trace volume surges by mandating upstream memory-limiter processors, bounded in-flight trace limits, and persistent disk-backed storage extensions for buffer spillover. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded In-Memory Tail-Sampling Buffer Leading to Node Eviction and Collector OOMKills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded In-Memory Tail-Sampling Buffer Leading to Node Eviction and Collector OOMKills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Pipelines MUST ALWAYS position the `memory_limiter` processor before the `tail_sampling` processor in the trace pipeline processor sequence.

Pipelines MUST ALWAYS position the `memory_limiter` processor before the `tail_sampling` processor in the trace pipeline processor sequence.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

2

The `tail_sampling` processor MUST define `num_traces` with an explicit upper bound calculated against peak span volume and memory headroom.

The `tail_sampling` processor MUST define `num_traces` with an explicit upper bound calculated against peak span volume and memory headroom.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

3

The `tail_sampling` processor MUST NOT set `decision_wait` to excessive durations (>30s) without an active persistent spillover extension.

The `tail_sampling` processor MUST NOT set `decision_wait` to excessive durations (>30s) without an active persistent spillover extension.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

4

Deployments MUST configure `file_storage` or persistent volume-backed queue extensions when buffers must survive transient upstream downstream backpressure.

Deployments MUST configure `file_storage` or persistent volume-backed queue extensions when buffers must survive transient upstream downstream backpressure.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

5

Kubernetes pod definitions MUST set `GOMEMLIMIT` to approximately 80% of the container cgroup memory limit to allow Go runtime garbage collection preemptive execution before cgroup termination.

Kubernetes pod definitions MUST set `GOMEMLIMIT` to approximately 80% of the container cgroup memory limit to allow Go runtime garbage collection preemptive execution before cgroup termination.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

6

The `memory_limiter` configuration MUST declare `check_interval` (default 1s), `limit_percentage` (max 80%), and `spike_limit_percentage` (max 20%).

The `memory_limiter` configuration MUST declare `check_interval` (default 1s), `limit_percentage` (max 80%), and `spike_limit_percentage` (max 20%).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Collector Tail-Sampling Adaptive Memory Spillover Standard critical for AI coding assistants?

Enforces deterministic memory safety in OpenTelemetry Collector deployments running tail-sampling processors. Prevents catastrophic OOMKilled crashes during trace volume surges by mandating upstream memory-limiter processors, bounded in-flight trace limits, and persistent disk-backed storage extensions for buffer spillover. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded In-Memory Tail-Sampling Buffer Leading to Node Eviction and Collector OOMKills) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry