AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: otel-collector-tail-sampling-memory-limit-standard

OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant

Enforces strict ordering and deterministic memory boundaries for OpenTelemetry Collector trace pipelines utilizing tail-based sampling. Prevents unrecoverable Out-Of-Memory (OOM) fatal kills during catastrophic upstream traffic spikes by strictly placing memory_limiter before tail_sampling and calculating bounded trace buffers against container cgroup limits.

Prohibited Anti-Pattern: Placing tail_sampling before memory_limiter with unbounded trace buffers, triggering catastrophic OOMKilled pipeline crashes during traffic surges.

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant - Enforces strict ordering and deterministic memory boundaries for OpenTelemetry Collector trace pipelines utilizing tail-based sampling. Prevents unrecoverable Out-Of-Memory (OOM) fatal kills during catastrophic upstream traffic spikes by strictly placing memory_limiter before tail_sampling and calculating bounded trace buffers against container cgroup limits.
globs: **/*otel*.{yaml,yml}, **/collector*.{yaml,yml}, **/helm/**/values*.{yaml,yml}, **/kustomize/**/otel*.{yaml,yml}
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-collector-tail-sampling-memory-limit-standard"
---

# OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Placing tail_sampling before memory_limiter with unbounded trace buffers, triggering catastrophic OOMKilled pipeline crashes during traffic surges.

## System Constraints
You are a Principal Software Architect specializing in hyperscale observability infrastructure. You mandate strict memory isolation, deterministic eviction policies, and resilient pipeline construction for OpenTelemetry Collectors. Under no circumstances should tail-sampling trace buffers operate without explicit bounds, nor may memory-limiter processors be omitted or sequenced after buffering stages.

## Strict Guidelines
- Pipelines MUST ALWAYS position 'memory_limiter' as the absolute FIRST processor in service.pipelines.traces to intercept and drop batches before memory-intensive deserialization and buffer aggregation.
- The 'tail_sampling' processor MUST ALWAYS specify an explicit 'num_traces' limit sized to consume at most 40% of the container memory budget calculated against the worst-case span count per trace tree.
- Container environments MUST ALWAYS define GOMEMLIMIT set to 80% of the cgroup memory limit, with the collector 'memory_limiter' check_interval set to 1s and limit_percentage capped at 75%.
- Pipelines MUST NOT rely on default buffer allocations for 'tail_sampling', as unbounded ring buffers will lead to immediate SIGKILL / OOMKilled errors under sudden traffic spikes.
- Engineers MUST configure the 'batch' processor AFTER the 'tail_sampling' processor so that discarded spans are not batched and queued needlessly into downstream exporters.
- When upstream traffic exceeds limits, 'tail_sampling' and 'memory_limiter' MUST be configured to emit dropping metrics (e.g., processor_tail_sampling_count_traces_sampled and processor_dropped_spans) to enable saturation alerting.

## Bad vs Good Implementations
### OpenTelemetry Collector Tail-Sampling Pipeline Configuration

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Placing tail_sampling before memory_limiter forces incoming spans to be retained in memory before the memory_limiter has an opportunity to shed load. Furthermore, omitting num_traces and setting check_interval to 10s ensures that a microburst will consume all heap space and trigger an unrecoverable cgroup OOM termination before the limiter ever checks usage.
```yaml
processors:
  tail_sampling:
    decision_wait: 30s
    # ANTI-PATTERN: num_traces omitted (defaults to unbounded or massive allocations)
    expected_new_traces_per_sec: 2000
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ ERROR ] }
  memory_limiter:
    # ANTI-PATTERN: High limit with missing spike margin and high check_interval
    check_interval: 10s
    limit_percentage: 95

service:
  pipelines:
    traces:
      receivers: [otlp]
      # ANTI-PATTERN: tail_sampling runs BEFORE memory_limiter, caching spans until OOM crash
      processors: [tail_sampling, memory_limiter, batch]
      exporters: [otlp]
```

#### Production Standard (Correct)
> **Standard**: memory_limiter is evaluated first, actively refusing new spans when memory reaches 75% capacity with a 20% spike headroom window checked every second. tail_sampling enforces a hard ceiling of 50,000 pending traces in its ring-buffer with a restrained 10s decision window, and the batch processor operates exclusively on spans that survived tail-sampling decisions.
```yaml
processors:
  memory_limiter:
    # PRODUCTION FIX: Intercepts spikes immediately with fast checks and explicit headroom
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    # PRODUCTION FIX: Strict deterministic ring buffer boundary
    num_traces: 50000
    expected_new_traces_per_sec: 5000
    policies:
      - name: drop-health-checks
        type: string_attribute
        string_attribute: { key: http.target, values: [ "/healthz", "/ready" ], enabled_regex_matching: false, invert_match: true }
      - name: error-rate-policy
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: probabilistic-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 10.0 }
  batch:
    send_batch_size: 8192
    timeout: 1s
    send_batch_max_size: 10240

service:
  pipelines:
    traces:
      receivers: [otlp]
      # PRODUCTION FIX: memory_limiter MUST precede buffering processors
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Prometheus Metrics MCP
Inspects otelcol_processor_dropped_spans and otelcol_process_memory_rss metrics to detect memory pressure and sampling drop rates during pipeline operations.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OpenTelemetry Collector Config Validator
CLI utility to lint and validate pipeline processor sequences, schema validity, and resource constraints against the official OpenTelemetry spec.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant

OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces strict ordering and deterministic memory boundaries for OpenTelemetry Collector trace pipelines utilizing tail-based sampling. Prevents unrecoverable Out-Of-Memory (OOM) fatal kills during catastrophic upstream traffic spikes by strictly placing memory_limiter before tail_sampling and calculating bounded trace buffers against container cgroup limits. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Placing tail_sampling before memory_limiter with unbounded trace buffers, triggering catastrophic OOMKilled pipeline crashes during traffic surges. Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftPlacing tail_sampling before memory_limiter with unbounded trace buffers, triggering catastrophic OOMKilled pipeline crashes during traffic surges.Critical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Pipelines MUST ALWAYS position 'memory_limiter' as the absolute FIRST processor in service.pipelines.traces to intercept and drop batches before memory-intensive deserialization and buffer aggregation.

Pipelines MUST ALWAYS position 'memory_limiter' as the absolute FIRST processor in service.pipelines.traces to intercept and drop batches before memory-intensive deserialization and buffer aggregation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

The 'tail_sampling' processor MUST ALWAYS specify an explicit 'num_traces' limit sized to consume at most 40% of the container memory budget calculated against the worst-case span count per trace tree.

The 'tail_sampling' processor MUST ALWAYS specify an explicit 'num_traces' limit sized to consume at most 40% of the container memory budget calculated against the worst-case span count per trace tree.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Container environments MUST ALWAYS define GOMEMLIMIT set to 80% of the cgroup memory limit, with the collector 'memory_limiter' check_interval set to 1s and limit_percentage capped at 75%.

Container environments MUST ALWAYS define GOMEMLIMIT set to 80% of the cgroup memory limit, with the collector 'memory_limiter' check_interval set to 1s and limit_percentage capped at 75%.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Pipelines MUST NOT rely on default buffer allocations for 'tail_sampling', as unbounded ring buffers will lead to immediate SIGKILL / OOMKilled errors under sudden traffic spikes.

Pipelines MUST NOT rely on default buffer allocations for 'tail_sampling', as unbounded ring buffers will lead to immediate SIGKILL / OOMKilled errors under sudden traffic spikes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Engineers MUST configure the 'batch' processor AFTER the 'tail_sampling' processor so that discarded spans are not batched and queued needlessly into downstream exporters.

Engineers MUST configure the 'batch' processor AFTER the 'tail_sampling' processor so that discarded spans are not batched and queued needlessly into downstream exporters.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

When upstream traffic exceeds limits, 'tail_sampling' and 'memory_limiter' MUST be configured to emit dropping metrics (e.g., processor_tail_sampling_count_traces_sampled and processor_dropped_spans) to enable saturation alerting.

When upstream traffic exceeds limits, 'tail_sampling' and 'memory_limiter' MUST be configured to emit dropping metrics (e.g., processor_tail_sampling_count_traces_sampled and processor_dropped_spans) to enable saturation alerting.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Collector Tail-Sampling Adaptive Ring-Buffer and Memory Limiter Invariant critical for AI coding assistants?

Enforces strict ordering and deterministic memory boundaries for OpenTelemetry Collector trace pipelines utilizing tail-based sampling. Prevents unrecoverable Out-Of-Memory (OOM) fatal kills during catastrophic upstream traffic spikes by strictly placing memory_limiter before tail_sampling and calculating bounded trace buffers against container cgroup limits. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Placing tail_sampling before memory_limiter with unbounded trace buffers, triggering catastrophic OOMKilled pipeline crashes during traffic surges.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry