Mandatory Pre-Processor Memory Limiting and Deterministic Routing for Tail-Based Trace Sampling

--- description: Mandatory Pre-Processor Memory Limiting and Deterministic Routing for Tail-Based Trace Sampling - Tail-based sampling buffers complete trace trees within a sliding evaluation window, introducing critical vulnerability to OOM termination during traffic bursts. This guardrail mandates strict processor ordering with `memory_limiter` placed first, upstream `trace_id` load-balanced routing, and tiered sampling policies that preserve 100% of errors and latency anomalies while downsampling healthy traffic. globs: **/otel*.{yaml,yml}, **/collector*.{yaml,yml}, **/helm/**/*.{yaml,yml}, **/k8s/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-memory-guardrail" --- # Mandatory Pre-Processor Memory Limiting and Deterministic Routing for Tail-Based Trace Sampling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Tail Sampling Buffering Without Memory Limiter Precedence Causing Silent Dropped Traces and OOM Kills ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing production-grade OpenTelemetry Collector tail-based trace sampling. You must reject any configuration where memory_limiter does not precede tail_sampling, where error/latency capture rates are less than 100%, or where trace routing lacks trace_id hash-affinity. ## Strict Guidelines - Engineers MUST ALWAYS place the `memory_limiter` processor as the FIRST processor in the `pipelines.traces.processors` array, strictly before `tail_sampling` and `batch`. - Engineers MUST NOT deploy tail-based sampling behind standard round-robin or layer-4 load balancers; collectors MUST receive traces partitioned deterministically via the `loadbalancingexporter` keyed by `trace_id`. - Configurations MUST define explicit 100% sample rate evaluation rules for all spans matching `status_code == STATUS_CODE_ERROR` and HTTP status code attributes `>= 500` or `>= 400`. - Configurations MUST define an explicit latency-based policy guaranteeing 100% capture for spans exceeding SLA latency thresholds (e.g., > 1000ms). - Services MUST configure `memory_limiter` with a `check_interval` of at most `1s`, `limit_percentage` of at most `80%`, and a `spike_limit_percentage` of at least `20%` to prevent container OOMKills during evaluation windows. - Components NEVER place the `batch` processor before `tail_sampling`, as batching early aggregates disparate traces into memory chunks that bypass per-trace decision timeouts. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail Sampling and Memory Limiter Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing memory_limiter after tail_sampling fails to protect the buffer from memory exhaustion during ingestion spikes. Batching before tail sampling fragments evaluation windows. Using only a probabilistic policy loses 90% of production errors and latency spikes. ```yaml # ANTI-PATTERN: memory_limiter placed after tail_sampling or omitted; # tail_sampling processes unconstrained volume without prioritizing errors processors: batch: send_batch_size: 8192 timeout: 1s tail_sampling: decision_wait: 10s num_traces: 100000 expected_new_traces_per_sec: 2000 policies: - name: probabilistic-sampling type: probabilistic probabilistic: sampling_percentage: 10 memory_limiter: check_interval: 5s limit_percentage: 95 service: pipelines: traces: receivers: [otlp] # FATAL: batch and tail_sampling run before memory_limiter. # High-volume bursts cause collector OOM before limits take effect. # Errors are sampled out probabilistically (90% data loss on outages). processors: [batch, tail_sampling, memory_limiter] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: Enforces verified production invariants and compliant error handling. ```yaml # PRODUCTION INVARIANT: memory_limiter first, explicit error & latency retention, # bounded probabilistic downsampling for healthy traces processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 5s num_traces: 50000 expected_new_traces_per_sec: 5000 policies: # Invariant: 100% retention for explicit OpenTelemetry trace errors - name: retain-errors type: status_code status_code: status_codes: [ERROR] # Invariant: 100% retention for HTTP error status codes - name: retain-http-5xx type: numeric_attribute numeric_attribute: key: http.status_code min_value: 500 max_value: 599 # Invariant: 100% retention for latency anomalies breaching SLO - name: retain-latency-breaches type: latency latency: threshold_ms: 1500 # Shed bulk normal traces while retaining representative baseline - name: sample-healthy-baseline type: probabilistic probabilistic: sampling_percentage: 2.5 batch: send_batch_max_size: 2048 send_batch_size: 1024 timeout: 500ms service: pipelines: traces: receivers: [otlp] # Correct Order: Protect memory -> Evaluate full traces -> Batch downstream egress processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Mandatory Pre-Processor Memory Limiting and Deterministic Routing for Tail-Based Trace Sampling

Mandatory Pre-Processor Memory Limiting and Deterministic Routing for Tail-Based Trace Sampling defines an industrial-grade behavioral standard for devops engineering. Tail-based sampling buffers complete trace trees within a sliding evaluation window, introducing critical vulnerability to OOM termination during traffic bursts. This guardrail mandates strict processor ordering with `memory_limiter` placed first, upstream `trace_id` load-balanced routing, and tiered sampling policies that preserve 100% of errors and latency anomalies while downsampling healthy traffic. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Tail Sampling Buffering Without Memory Limiter Precedence Causing Silent Dropped Traces and OOM Kills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Tail Sampling Buffering Without Memory Limiter Precedence Causing Silent Dropped Traces and OOM Kills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS place the `memory_limiter` processor as the FIRST processor in the `pipelines.traces.processors` array, strictly before `tail_sampling` and `batch`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.