AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: otel-collector-tail-sampling-memory-guardrail-opentelemetry-co-717

OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail

Enforces strict memory ballast, memory_limiter processor ordering, and bounded trace buffer sizing within OpenTelemetry Collector deployments. Prevents container OOMKills during cardinality bursts while ensuring high-latency spans and 5xx error traces are deterministically retained through isolated priority queues.

Prohibited Anti-Pattern: Unbounded tail-sampling buffers without upstream memory_limiter enforcement causing catastrophic collector OOM crash loops during traffic spikes.

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail - Enforces strict memory ballast, memory_limiter processor ordering, and bounded trace buffer sizing within OpenTelemetry Collector deployments. Prevents container OOMKills during cardinality bursts while ensuring high-latency spans and 5xx error traces are deterministically retained through isolated priority queues.
globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/otelcol*.yaml, **/otelcol*.yml, deployments/**/collector.yaml, k8s/**/otel*.yaml
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-collector-tail-sampling-memory-guardrail-opentelemetry-co-717"
---

# OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Unbounded tail-sampling buffers without upstream memory_limiter enforcement causing catastrophic collector OOM crash loops during traffic spikes.

## System Constraints
You are a Principal Observability Infrastructure Architect enforcing memory ceilings and tail-sampling reliability invariants in OpenTelemetry Collector topologies. Mandate strict ordering of processors, explicit memory ceilings, hard limits on trace arrival buffers, and dedicated fallback routes for anomalous spans.

## Strict Guidelines
- Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in the trace pipeline before `tail_sampling`.
- Engineers MUST configure `check_interval` to no greater than 1s and set `limit_percentage` to at most 80% (or `limit_mib` equivalent) of the Kubernetes pod container memory limit.
- Engineers MUST configure explicit upper bounds on `num_traces` and `expected_new_traces_per_sec` inside the `tail_sampling` processor configuration.
- Engineers MUST NOT run `tail_sampling` with default unconstrained memory buffers or omit the `spike_limit_percentage` parameter in `memory_limiter`.
- Engineers MUST configure sampling evaluation policies such that error conditions (HTTP 5xx, gRPC != 0) and high-latency thresholds (>99th percentile target) take precedence before probabilistic drop rules.
- Deployments MUST pair Kubernetes memory limits with `GOMEMLIMIT` environment variables set to 80-85% of total pod memory limits.

## Bad vs Good Implementations
### OpenTelemetry Collector Tail-Sampling Pipeline Configuration

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Omitting the memory_limiter processor as the first processor allows high-cardinality bursts or sustained span volume to fill the tail-sampling decision cache unchecked, exceeding Kubernetes container limits and triggering an unrecoverable SIGKILL (Exit Code 137).
```yaml
# INSECURE: memory_limiter is missing or misplaced, tail_sampling has unbound trace capacity
processors:
  tail_sampling:
    decision_wait: 30s
    expected_new_traces_per_sec: 10000
    policies:
      - name: sample-errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
  batch:
    send_batch_max_size: 1024
    timeout: 1s

service:
  pipelines:
    traces:
      receivers: [otlp]
      # Anti-pattern: tail_sampling consumes heap unbounded without memory_limiter protection
      processors: [tail_sampling, batch]
      exporters: [otlp]
```

#### Production Standard (Correct)
> **Standard**: Places memory_limiter at the beginning of the pipeline to shed load before tail-sampling queues can saturate heap. Bounds tail_sampling memory by capping decision_wait to 10s and num_traces to 50,000, while guaranteeing retention of latency spikes (>2s) and 5xx errors.
```yaml
# HARDENED: First-position memory_limiter with explicit ceilings and bounded tail-sampling cache
processors:
  memory_limiter:
    check_interval: 500ms
    limit_percentage: 75
    spike_limit_percentage: 15

  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 5000
    policies:
      - name: latency-anomaly-policy
        type: latency
        latency: { threshold_ms: 2000 }
      - name: error-status-policy
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: deterministic-probabilistic-fallback
        type: probabilistic
        probabilistic: { sampling_percentage: 5.0 }

  batch:
    send_batch_max_size: 2048
    timeout: 1s

service:
  pipelines:
    traces:
      receivers: [otlp]
      # Invariant: memory_limiter MUST precede tail_sampling
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Resource MCP Server
Inspects and validates Kubernetes pod spec memory limits, requests, and container OOMKilled events for OpenTelemetry Collector daemonsets/deployments.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OpenTelemetry Collector Config Validator
Validates OpenTelemetry collector configuration files via `otelcorecol validate` or `otelcol-contrib validate` prior to deployment.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail

OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory ballast, memory_limiter processor ordering, and bounded trace buffer sizing within OpenTelemetry Collector deployments. Prevents container OOMKills during cardinality bursts while ensuring high-latency spans and 5xx error traces are deterministically retained through isolated priority queues. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded tail-sampling buffers without upstream memory_limiter enforcement causing catastrophic collector OOM crash loops during traffic spikes. Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftUnbounded tail-sampling buffers without upstream memory_limiter enforcement causing catastrophic collector OOM crash loops during traffic spikes.Critical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in the trace pipeline before `tail_sampling`.

Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in the trace pipeline before `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST configure `check_interval` to no greater than 1s and set `limit_percentage` to at most 80% (or `limit_mib` equivalent) of the Kubernetes pod container memory limit.

Engineers MUST configure `check_interval` to no greater than 1s and set `limit_percentage` to at most 80% (or `limit_mib` equivalent) of the Kubernetes pod container memory limit.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Engineers MUST configure explicit upper bounds on `num_traces` and `expected_new_traces_per_sec` inside the `tail_sampling` processor configuration.

Engineers MUST configure explicit upper bounds on `num_traces` and `expected_new_traces_per_sec` inside the `tail_sampling` processor configuration.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Engineers MUST NOT run `tail_sampling` with default unconstrained memory buffers or omit the `spike_limit_percentage` parameter in `memory_limiter`.

Engineers MUST NOT run `tail_sampling` with default unconstrained memory buffers or omit the `spike_limit_percentage` parameter in `memory_limiter`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Engineers MUST configure sampling evaluation policies such that error conditions (HTTP 5xx, gRPC != 0) and high-latency thresholds (>99th percentile target) take precedence before probabilistic drop rules.

Engineers MUST configure sampling evaluation policies such that error conditions (HTTP 5xx, gRPC != 0) and high-latency thresholds (>99th percentile target) take precedence before probabilistic drop rules.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

Deployments MUST pair Kubernetes memory limits with `GOMEMLIMIT` environment variables set to 80-85% of total pod memory limits.

Deployments MUST pair Kubernetes memory limits with `GOMEMLIMIT` environment variables set to 80-85% of total pod memory limits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Collector Tail-Sampling Buffer Memory Ceiling Guardrail critical for AI coding assistants?

Enforces strict memory ballast, memory_limiter processor ordering, and bounded trace buffer sizing within OpenTelemetry Collector deployments. Prevents container OOMKills during cardinality bursts while ensuring high-latency spans and 5xx error traces are deterministically retained through isolated priority queues. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded tail-sampling buffers without upstream memory_limiter enforcement causing catastrophic collector OOM crash loops during traffic spikes.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry