OpenTelemetry Collector Tail-Based Sampling Memory Backpressure Standard

--- description: OpenTelemetry Collector Tail-Based Sampling Memory Backpressure Standard - Enforces deterministic memory safety boundaries and backpressure mechanics within OpenTelemetry Collector deployments running tail-based sampling. Mandates the placement of the memory_limiter processor strictly before stateful sampling processors and requires explicit calibration of trace buffer limits against container cgroup allocations to prevent OOMKilled events during high-concurrency traffic bursts. globs: **/otel-collector*.{yaml,yml}, **/collector/**/*.{yaml,yml}, deploy/kubernetes/**/*.{yaml,yml}, helm/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-backpressure-standard" --- # OpenTelemetry Collector Tail-Based Sampling Memory Backpressure Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Buffering unbounded in-flight traces in tail samplers without upstream memory gating, triggering catastrophic OOM crashes during sudden microservice traffic spikes. ## System Constraints You are a Principal Observability Infrastructure Architect enforcing memory backpressure invariants across OpenTelemetry Collector deployments. Tail-based sampling processors store entire trace batches in transient heap memory during their decision evaluation window. You must strictly mandate that all OpenTelemetry Collector pipelines position the memory_limiter processor upstream of tail_sampling, configure strict trace cache caps via expected_new_traces_per_sec, and align GC thresholds with container resource constraints. ## Strict Guidelines - Engineers MUST ALWAYS declare and order the 'memory_limiter' processor as the first processor in any pipeline containing 'tail_sampling'. - Engineers MUST configure explicit 'check_interval', 'limit_percentage' (or 'limit_mib'), and 'spike_limit_percentage' (or 'spike_limit_mib') in the 'memory_limiter' configuration matched to 75-80% of container cgroup memory limits. - Engineers MUST NOT configure 'tail_sampling' with an unbounded or zero 'expected_new_traces_per_sec' parameter. - Pipelines MUST maintain a 'decision_wait' configuration in 'tail_sampling' that does not exceed 30 seconds unless backed by an explicit downstream partitioned collector tier with external routing. - Pipelines MUST configure bounded queue sizes and droppable or backpressuring queue options in the 'batch' processor and downstream exporter 'sending_queue' configurations. - Collectors running tail-based sampling MUST set the 'GOMEMLIMIT' environment variable to 85% of container limits and 'GODEBUG=madvdontneed=1' to force immediate virtual memory release back to the OS. ## Bad vs Good Implementations ### Collector Trace Pipeline with Unbounded Tail Sampling (Vulnerable to OOM) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing memory_limiter after tail_sampling renders it useless against trace buffering spikes; tail_sampling holds uncommitted spans in memory for 30 seconds before passing them downstream. Without expected_new_traces_per_sec and upstream memory gating, a sudden surge in spans instantly drives memory consumption past the Kubernetes cgroup limit, killing the pod without shedding load. ```yaml processors: tail_sampling: decision_wait: 30s # Missing expected_new_traces_per_sec bounds policies: - name: sample_errors type: status_code status_code: { status_codes: [ ERROR ] } batch: send_batch_size: 8192 timeout: 5s # Memory limiter misplaced downstream of stateful buffering memory_limiter: check_interval: 1s limit_percentage: 95 service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch, memory_limiter] # FAILS: memory_limiter is after tail_sampling exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter evaluates heap constraints first, rejecting incoming trace ingestion before memory allocation can breach the cgroup ceiling. Setting 'expected_new_traces_per_sec: 15000' pre-allocates and bounds the internal ring buffer for the 10s decision window, and the downstream sending_queue prevents backpressure deadlocks. ```yaml processors: # Correct: memory_limiter acts as upstream gateway memory_limiter: check_interval: 500ms limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s expected_new_traces_per_sec: 15000 policies: - name: sample_errors type: status_code status_code: { status_codes: [ ERROR ] } - name: sample_probabilistic type: probabilistic probabilistic: { sampling_percentage: 10.0 } batch: send_batch_size: 4096 timeout: 1s exporters: otlp: endpoint: "tempo-distributor.monitoring:4317" sending_queue: enabled: true num_consumers: 8 queue_size: 5000 service: pipelines: traces: receivers: [otlp] # Correct order: memory_limiter sheds load before trace buffers allocate processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Based Sampling Memory Backpressure Standard

OpenTelemetry Collector Tail-Based Sampling Memory Backpressure Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic memory safety boundaries and backpressure mechanics within OpenTelemetry Collector deployments running tail-based sampling. Mandates the placement of the memory_limiter processor strictly before stateful sampling processors and requires explicit calibration of trace buffer limits against container cgroup allocations to prevent OOMKilled events during high-concurrency traffic bursts. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Buffering unbounded in-flight traces in tail samplers without upstream memory gating, triggering catastrophic OOM crashes during sudden microservice traffic spikes. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Buffering unbounded in-flight traces in tail samplers without upstream memory gating, triggering catastrophic OOM crashes during sudden microservice traffic spikes.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare and order the 'memory_limiter' processor as the first processor in any pipeline containing 'tail_sampling'.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.