OpenTelemetry Collector Tail Sampling Heap Eviction Circuit Breaker Standard

--- description: OpenTelemetry Collector Tail Sampling Heap Eviction Circuit Breaker Standard - Enforces memory-bounded tail sampling and deterministic load-shedding circuit breakers within OpenTelemetry Collector pipelines. Mandates strict ordering where the memory_limiter processor executes before the tail_sampling processor, alongside bounded trace caches and exporter queue limits, to prevent container OOM-kills during cascading upstream traffic spikes. globs: **/otel-collector*.{yaml,yml}, **/collector*.{yaml,yml}, k8s/**/otel*.yaml, deployments/**/collector*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-heap-circuit-breaker" --- # OpenTelemetry Collector Tail Sampling Heap Eviction Circuit Breaker Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Tail-Sampling Memory Buffers Inducing Collector SIGKILL Under Upstream Traffic Surges ## System Constraints You are a Principal Software Architect enforcing production-grade resilience standards for OpenTelemetry Collector deployments. Tail-based sampling requires in-memory trace buffering until completion or decision timeouts occur. Unbounded trace buffers combined with sudden upstream throughput bursts inevitably cause node-level cgroup OOM kills unless protected by proactive heap circuit breakers. You must enforce strict processor ordering, memory limiter configuration, explicit buffer caps, and export queue constraints. ## Strict Guidelines - Pipelines MUST configure `memory_limiter` as the FIRST processor in the traces pipeline before `tail_sampling`. - Engineers MUST specify explicit `check_interval`, `limit_percentage` (or `limit_mib`), and `spike_limit_percentage` (or `spike_limit_mib`) in the `memory_limiter` definition. - Engineers MUST NOT rely on default `tail_sampling` buffer sizes; `num_traces` and `expected_new_traces_per_sec` MUST be explicitly declared to bound memory allocation. - Engineers MUST ensure `decision_wait` is tuned to the shortest acceptable service SLA window (MUST NOT exceed 30 seconds without dedicated architectural review). - All downstream exporter blocks referenced by tail-sampling pipelines MUST define a bounded `sending_queue` with explicit `queue_size` and `storage: memory` limits. - Collector container environments MUST set `GOMEMLIMIT` to 80-85% of the cgroup memory limit and `GOGC=80` to coordinate Go runtime GC before the memory_limiter sheds traffic. ## Bad vs Good Implementations ### OpenTelemetry Collector Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing tail_sampling before memory_limiter allows upstream traffic spikes to flood trace buffers into memory without bounds. A 300s decision_wait without a capped num_traces guarantees uncontrollable heap growth, culminating in a Linux OOM killer termination (SIGKILL 137). ```yaml # CRITICAL VIOLATION: tail_sampling declared first without memory_limiter protection. # decision_wait is excessively long, num_traces is unconstrained, and exporter queue is unbounded. processors: tail_sampling: decision_wait: 300s expected_new_traces_per_sec: 10000 policies: - name: errors-policy type: status_code status_code: { status_codes: [ ERROR ] } exporters: otlp: endpoint: tempo.monitoring.svc.cluster.local:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [tail_sampling] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter acts as an immediate circuit breaker, rejecting or dropping inbound spans before they enter the tail-sampling trace cache. num_traces caps the heap allocation for incomplete traces, decision_wait is bounded to 10s, and the downstream sending queue is strictly sized. ```yaml # COMPLIANT: memory_limiter precedes tail_sampling with tight check intervals and spike buffers. # tail_sampling bounds trace cache capacity and exporter queues prevent memory deadlocks. processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2500 policies: - name: drop-health-checks type: string_attribute string_attribute: { key: http.target, values: [ "/healthz", "/ready" ], invert_match: true } - name: sample-errors type: status_code status_code: { status_codes: [ ERROR ] } - name: probabilistic-fallback type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_size: 8192 timeout: 1s send_batch_max_size: 10240 exporters: otlp: endpoint: tempo.monitoring.svc.cluster.local:4317 tls: insecure: true sending_queue: enabled: true num_consumers: 8 queue_size: 5000 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail Sampling Heap Eviction Circuit Breaker Standard

OpenTelemetry Collector Tail Sampling Heap Eviction Circuit Breaker Standard defines an industrial-grade behavioral standard for devops engineering. Enforces memory-bounded tail sampling and deterministic load-shedding circuit breakers within OpenTelemetry Collector pipelines. Mandates strict ordering where the memory_limiter processor executes before the tail_sampling processor, alongside bounded trace caches and exporter queue limits, to prevent container OOM-kills during cascading upstream traffic spikes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Tail-Sampling Memory Buffers Inducing Collector SIGKILL Under Upstream Traffic Surges Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Tail-Sampling Memory Buffers Inducing Collector SIGKILL Under Upstream Traffic Surges	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST configure `memory_limiter` as the FIRST processor in the traces pipeline before `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.