Deterministic Backpressure and Load Shedding in OpenTelemetry Collector Tail-Based Sampling

--- description: Deterministic Backpressure and Load Shedding in OpenTelemetry Collector Tail-Based Sampling - Enforces deterministic load shedding and explicit memory isolation within OpenTelemetry Collector tail-sampling pipelines during traffic spikes and incident cascades. Prevents OOM kills by mandating memory_limiter placement before stateful buffers and configuring finite memory-bounded trace reserves with priority-tiered drop rules. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector/*.yaml, **/collector/*.yml, deploy/k8s/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-backpressure-invariant" --- # Deterministic Backpressure and Load Shedding in OpenTelemetry Collector Tail-Based Sampling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded State Buffering in Tail Sampler Preceding Memory Limiter Inducing Pipeline OOM Kills ## System Constraints You are a Principal Observability Infrastructure Architect enforcing strict tail-based sampling backpressure invariants on OpenTelemetry Collector configurations. Mandate that stateful trace aggregation buffers never exceed predictable memory ceilings during telemetry storms. Ensure that memory_limiter processors strictly precede tail_sampling processors in trace pipelines, explicit maximum trace counts are set, and lower-tier telemetry (e.g., polling, non-error GET requests, health checks) is aggressively shed before degrading anomalous or error traces. ## Strict Guidelines - Pipelines MUST ALWAYS position the `memory_limiter` processor as the very first processor in any traces pipeline utilizing `tail_sampling`. - The `tail_sampling` processor MUST ALWAYS define `num_traces` and `expected_new_traces_per_sec` tuned strictly below container memory allocations. - Engineers MUST NOT configure unbounded decision waits (`decision_wait`) exceeding 30s without deterministic routing to tiered fallback storage. - Configurations MUST define prioritized policy ordering where deterministic high-frequency non-critical spans (health checks, liveness probes) are dropped or downsampled before stateful evaluation of business spans. - Components NEVER rely on downstream exporter queues (`batch` processor or exporter retry queues) to apply backpressure to `tail_sampling` memory tables; explicit drop behavior MUST be declared at the collector ingress. ## Bad vs Good Implementations ### Collector Tail Sampling Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing tail_sampling before memory_limiter allows the tail sampler to accumulate trace state in memory uninhibited. During an incident, unbounded trace tree accumulation exhausts heap space before the memory_limiter can reject incoming batches, resulting in Linux OOM kills and zero telemetry egress. ```yaml processors: tail_sampling: decision_wait: 30s # ANTI-PATTERN: Missing num_traces and expected_new_traces_per_sec # Buffer will grow unbounded during traffic spikes policies: - name: sample-errors type: status_code status_code: { status_codes: [ ERROR ] } memory_limiter: # ANTI-PATTERN: Placed after tail_sampling or omitted entirely check_interval: 1s limit_percentage: 80 spike_limit_percentage: 20 service: pipelines: traces: receivers: [otlp] # ANTI-PATTERN: tail_sampling evaluated before memory_limiter enforcement processors: [tail_sampling, memory_limiter, batch] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter processor is strictly first, proactively shedding new connections if heap thresholds are breached. The tail_sampling processor specifies explicit bounds (`num_traces: 50000`), drops non-diagnostic traffic first via explicit filter policies, and protects against trace accumulation leaks during massive cascading failures. ```yaml processors: memory_limiter: check_interval: 500ms limit_percentage: 75 spike_limit_percentage: 15 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: # 1. Deterministic drop of low-priority telemetry before deep inspection - name: drop-health-checks type: string_attribute string_attribute: key: http.route values: ["/healthz", "/livez", "/readyz", "/metrics"] enabled_regex_matching: false invert_match: false # 2. Retain 100% of errors - name: retain-errors type: status_code status_code: { status_codes: [ ERROR ] } # 3. Retain slow traces - name: retain-high-latency type: latency latency: { threshold_ms: 1500 } # 4. Probabilistic tail-drop fallback for benign traffic - name: sample-normal-traffic type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_max_size: 2048 timeout: 1s service: pipelines: traces: receivers: [otlp] # Correct invariant: memory limits enforced first, followed by bounded tail sampling processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic Backpressure and Load Shedding in OpenTelemetry Collector Tail-Based Sampling

Deterministic Backpressure and Load Shedding in OpenTelemetry Collector Tail-Based Sampling defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic load shedding and explicit memory isolation within OpenTelemetry Collector tail-sampling pipelines during traffic spikes and incident cascades. Prevents OOM kills by mandating memory_limiter placement before stateful buffers and configuring finite memory-bounded trace reserves with priority-tiered drop rules. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded State Buffering in Tail Sampler Preceding Memory Limiter Inducing Pipeline OOM Kills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded State Buffering in Tail Sampler Preceding Memory Limiter Inducing Pipeline OOM Kills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST ALWAYS position the `memory_limiter` processor as the very first processor in any traces pipeline utilizing `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.