OpenTelemetry Tail-Based Sampling Bounded Ringbuffer & Backpressure Invariant

--- description: OpenTelemetry Tail-Based Sampling Bounded Ringbuffer & Backpressure Invariant - Enforces deterministic memory bounds and hard backpressure thresholds within OpenTelemetry Collector pipelines utilizing tail-based sampling. Tail sampling mandates caching incomplete trace spans in memory, which without strictly ordered memory limiters, ringbuffer sizing, and deterministic trace routing results in unrecoverable OOM collector cascades during high-cardinality trace spikes. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector-config*.yaml, **/collector-config*.yml, **/k8s/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-tail-sampling-bounded-ringbuffer-invariant" --- # OpenTelemetry Tail-Based Sampling Bounded Ringbuffer & Backpressure Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace State Accumulation Leading to Collector OOM Cascades ## System Constraints You are a Principal Observability and Infrastructure Engineer enforcing deterministic memory and backpressure safety across OpenTelemetry Collector deployments. Tail-based sampling requires retaining spans until trace completion, making pipelines exceptionally vulnerable to OOM kills during traffic surges or cardinality spikes. You must mandate strict pipeline ordering, memory_limiter placement, deterministic trace hashing, bounded trace cache sizing, and explicit drop strategies. ## Strict Guidelines - Engineers MUST ALWAYS position the `memory_limiter` processor as the FIRST processor in any pipeline containing `tail_sampling`. - Engineers MUST NEVER configure `tail_sampling` without explicitly setting both `num_traces` and `expected_new_traces_per_sec` tuned to available pod memory limits. - Pipelines MUST enforce upstream trace-consistent routing using the `loadbalancing` exporter configured with `routing_key: trace_id` before traffic reaches tail-sampling collector instances. - Collector deployments MUST align `memory_limiter.check_interval` (default 1s), `limit_percentage` (max 75-80%), and `spike_limit_percentage` (max 15-20%) directly with container cgroup limits. - Pipelines SHOULD implement an explicit `batch` processor immediately after `tail_sampling` to minimize network transit overhead on retained traces. - Engineers MUST NOT rely on default unbounded in-memory queues when routing spans into sampling pipelines. ## Bad vs Good Implementations ### Unbounded Tail Sampling Configuration Vulnerable to OOMKills #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting the memory_limiter processor before tail_sampling allows memory usage to surge beyond container cgroup thresholds during traffic spikes. Additionally, unconstrained memory allocation occurs without explicit num_traces ceilings. ```yaml # INSECURE: memory_limiter is missing or misordered, tail_sampling has unbounded/default trace allocations processors: tail_sampling: decision_wait: 30s expected_new_traces_per_sec: 2000 # Default num_traces is absent or excessive, allowing memory to balloon indefinitely policies: - name: sample-errors type: status_code status_code: { status_codes: [ ERROR ] } - name: probabilistic-sample type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_size: 8192 timeout: 1s service: pipelines: traces: receivers: [otlp] # ANTI-PATTERN: tail_sampling runs without prior memory_limiter protection processors: [tail_sampling, batch] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter operates as the primary gatekeeper to enforce backpressure drops before traces enter the tail-sampling state. `num_traces` is strictly bounded to prevent heap exhaustion. ```yaml # HARDENED: Bounded ringbuffer, memory limiter priority, and deterministic backpressure processors: # 1. memory_limiter MUST execute first to shed load before in-memory trace buffering memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 # 2. tail_sampling bounded to predictable memory allocations tail_sampling: decision_wait: 10s num_traces: 50000 # Strictly bounds in-flight trace ringbuffer memory expected_new_traces_per_sec: 5000 # Calibrates internal hashmap pre-allocation policies: - name: filter-errors type: status_code status_code: { status_codes: [ ERROR ] } - name: latency-outliers type: numeric_attribute numeric_attribute: { key: http.status_code, min_value: 500, max_value: 599 } - name: default-probabilistic type: probabilistic probabilistic: { sampling_percentage: 10.0 } batch: send_batch_size: 2048 timeout: 200ms service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Bounded Ringbuffer & Backpressure Invariant

OpenTelemetry Tail-Based Sampling Bounded Ringbuffer & Backpressure Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory bounds and hard backpressure thresholds within OpenTelemetry Collector pipelines utilizing tail-based sampling. Tail sampling mandates caching incomplete trace spans in memory, which without strictly ordered memory limiters, ringbuffer sizing, and deterministic trace routing results in unrecoverable OOM collector cascades during high-cardinality trace spikes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace State Accumulation Leading to Collector OOM Cascades Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace State Accumulation Leading to Collector OOM Cascades	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS position the `memory_limiter` processor as the FIRST processor in any pipeline containing `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.