OpenTelemetry Collector Tail-Based Trace Sampling Buffer Allocation Guardrail

Imposes deterministic decision windows, bounded in-memory trace buffers, and memory ballast limits on distributed tail-based samplers to capture 100% of P99 tail anomalies without collector out-of-memory cascading failures. Tail sampling processors accumulate spans across distributed trace boundaries before committing drop/keep evaluations, making memory consumption a function of span throughput, trace duration, and evaluation latency. Without explicit memory limiter thresholds, max_traces bounds, and consistent trace-ID hashing across collector topologies, asymmetric routing causes heap saturation and catastrophic OOM kills.

--- description: OpenTelemetry Collector Tail-Based Trace Sampling Buffer Allocation Guardrail - Imposes deterministic decision windows, bounded in-memory trace buffers, and memory ballast limits on distributed tail-based samplers to capture 100% of P99 tail anomalies without collector out-of-memory cascading failures. Tail sampling processors accumulate spans across distributed trace boundaries before committing drop/keep evaluations, making memory consumption a function of span throughput, trace duration, and evaluation latency. Without explicit memory limiter thresholds, max_traces bounds, and consistent trace-ID hashing across collector topologies, asymmetric routing causes heap saturation and catastrophic OOM kills. globs: **/collector/*.yaml, **/collector/*.yml, **/otel-collector*.yaml, **/otel-collector*.yml, **/deployments/**/*.yaml, **/k8s/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-buffer-guardrail" --- # OpenTelemetry Collector Tail-Based Trace Sampling Buffer Allocation Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace Buffer Accumulation and Missing Memory Limiter Extension Causing Collector OOM Collapse ## System Constraints You are a Principal Observability Infrastructure Architect enforcing strict memory bounded execution for OpenTelemetry Collector tail-based trace sampling. Unbounded buffer configurations or absent memory limiter processors directly cause catastrophic out-of-memory eviction spirals in production collector fleets. You must ensure all collector configurations enforce hard memory thresholds, deterministic trace decision latency windows, and load-balancing trace routers across collector tiers. ## Strict Guidelines - Engineers MUST ALWAYS configure the 'memory_limiter' processor as the first processor in the pipeline, setting 'check_interval: 1s', 'limit_percentage: 75', and 'spike_limit_percentage: 20' relative to Kubernetes cgroup limits. - Engineers MUST NEVER configure 'tail_sampling' processor without setting 'num_traces' and 'expected_new_traces_per_sec' bounded directly by the container's heap allocation. - Engineers MUST configure 'decision_wait' to a deterministic duration (e.g., 10s to 30s) sufficient to catch P99.9 distributed spans, but strictly bounded to prevent heap exhaustion during throughput spikes. - Pipelines utilizing tail-based sampling MUST receive spans routed via an upstream OpenTelemetry Collector layer running the 'loadbalancing' exporter configured with 'routing_key: trace_id' to prevent partial trace fragmentation. - Engineers MUST configure explicit drop policies or inverted priority chains where error status, latency thresholds, and rare attributes take precedence over percentage-based probabilistic drop rules. - The 'batch' processor MUST ALWAYS be placed AFTER the 'tail_sampling' processor in pipelines to avoid buffering un-sampled spans in memory twice. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail Sampling Configuration: Unbounded Buffer vs Deterministic Memory Bounded #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The bad configuration buffers spans for 300 seconds without specifying num_traces limits or utilizing the memory_limiter processor. Placing the batch processor before tail_sampling retains raw span vectors in memory, guaranteeing collector OOM kills when span arrival rates spike or when slow traces accumulate. ```yaml # ANTI-PATTERN: No memory_limiter, unbounded num_traces, batch processor before tail_sampling, # and missing trace-ID load balancing causing memory leaks and OOM under high volume. processors: batch: send_batch_size: 10000 timeout: 10s tail_sampling: decision_wait: 300s # Catastrophic: buffers spans for 5 minutes without memory bounds policies: - name: latency-policy type: numeric_attribute numeric_attribute: { key: http.status_code, value_condition: { greater_than_or_equal: 500 } } service: pipelines: traces: receivers: [otlp] processors: [batch, tail_sampling] # Anti-pattern: batching before sampling inflates memory exporters: [otlp/backend] ``` #### Production Standard (Correct) > **Standard**: Enforces memory bounds with memory_limiter as the first step, calculates memory limits using a 20% ballast, restricts trace accumulation to 50,000 active traces with a 15-second decision wait, guarantees P99 error and latency retention, and places the batch processor after tail sampling so discarded spans never consume batch buffer resources. ```yaml # PRODUCTION INVARIANT: Strict memory_limiter, bounded num_traces, deterministic decision window, # and proper processor ordering (memory_limiter -> tail_sampling -> batch). extensions: health_check: memory_ballast: size_in_percentage: 20 processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 tail_sampling: decision_wait: 15s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: - name: errors-policy type: status_code status_code: { status_codes: [ERROR] } - name: p99-latency-policy type: latency latency: { threshold_ms: 1500 } - name: probabilistic-sample type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_max_size: 2048 send_batch_size: 1024 timeout: 1s service: extensions: [health_check, memory_ballast] pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp/backend] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Based Trace Sampling Buffer Allocation Guardrail

OpenTelemetry Collector Tail-Based Trace Sampling Buffer Allocation Guardrail defines an industrial-grade behavioral standard for devops engineering. Imposes deterministic decision windows, bounded in-memory trace buffers, and memory ballast limits on distributed tail-based samplers to capture 100% of P99 tail anomalies without collector out-of-memory cascading failures. Tail sampling processors accumulate spans across distributed trace boundaries before committing drop/keep evaluations, making memory consumption a function of span throughput, trace duration, and evaluation latency. Without explicit memory limiter thresholds, max_traces bounds, and consistent trace-ID hashing across collector topologies, asymmetric routing causes heap saturation and catastrophic OOM kills. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace Buffer Accumulation and Missing Memory Limiter Extension Causing Collector OOM Collapse Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace Buffer Accumulation and Missing Memory Limiter Extension Causing Collector OOM Collapse	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure the 'memory_limiter' processor as the first processor in the pipeline, setting 'check_interval

Engineers MUST ALWAYS configure the 'memory_limiter' processor as the first processor in the pipeline, setting 'check_interval: 1s', 'limit_percentage: 75', and 'spike_limit_percentage: 20' relative to Kubernetes cgroup limits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.