OpenTelemetry Tail-Based Sampling Sliding-Window Memory Eviction Firewall

--- description: OpenTelemetry Tail-Based Sampling Sliding-Window Memory Eviction Firewall - Enforces deterministic bounded sliding windows, memory hard-limits, and prioritized trace eviction inside OpenTelemetry Collector tail-sampling pipelines. Prevents JVM/Go runtime heap exhaustion and catastrophic silent drop storms by orchestrating circuit breakers, backpressure signaling, and deterministic reservoir eviction before telemetry buffers saturate system resources. globs: **/collector.yaml, **/otelcol*.yaml, **/*sampling*.{ts,go,yaml,json}, **/telemetry/**/*.{ts,go,yaml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-sliding-window-eviction-firewall" --- # OpenTelemetry Tail-Based Sampling Sliding-Window Memory Eviction Firewall > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded tail-sampling buffers without proactive memory circuit-breakers causing heap exhaustion and silent trace starvation during downstream error storms ## System Constraints You are a Principal Observability Infrastructure Architect enforcing strict memory-bounded tail-sampling pipelines for OpenTelemetry Collectors. You must ensure that high-throughput distributed tracing setups employ deterministic sliding-window buffers, explicit memory ballast or hard limits via memory_limiter processors, and priority-stratified shed policies so error surges never trigger OOM crashes or silent, non-audited trace drops. ## Strict Guidelines - Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in every pipeline preceding `tail_sampling` to enforce hard threshold cutoffs before trace allocation. - Engineers MUST specify explicit `expected_new_traces_per_sec` and `num_traces` limits within the `tail_sampling` processor configuration to establish a strictly bounded memory footprint. - Engineers MUST NOT configure tail-sampling evaluation windows (`decision_wait`) without corresponding `trace_idle_time` cutoffs, preventing slow or orphaned spans from permanently pinning collector memory. - Services MUST prioritize retention of 5xx errors and critical-path spans over high-volume success paths using prioritized multi-tier sampling rules with emergency drop fallbacks. - Components NEVER drop telemetry silently during heap pressure; all evictions and circuit-breaker triggers MUST be instrumented via collector internal metrics (`otelcol_processor_tail_sampling_traces_dropped_total` and `otelcol_processor_dropped_spans`). ## Bad vs Good Implementations ### Unbounded Collector Tail Sampling vs Deterministic Memory-Firewalled Collector Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing tail_sampling before memory_limiter allows memory allocations to overwhelm the Go heap before the limiter checks. Unbounded decision_wait without trace limits causes runaway heap exhaustion during downstream error spikes. ```yaml # INSECURE / UNSTABLE: Lacks memory bounds, order inverted, unbounded decision wait processors: tail_sampling: decision_wait: 300s # Stalls traces for 5 minutes without buffer caps expected_new_traces_per_sec: 0 # Unbounded allocation policies: - name: sample_errors type: status_code status_code: { status_codes: [ ERROR ] } memory_limiter: check_interval: 5s # Placed after tail_sampling: OOM occurs before check runs limit_percentage: 95 service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, memory_limiter] # Inverted processor chain exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter guards the pipeline intake with frequent check intervals (250ms) and bounded spikes. The tail_sampling processor defines explicit upper bounds on trace count (50000) and duration (10s), ensuring deterministic memory footprints even in extreme incident storms. ```yaml # HARDENED: Bounded sliding window, deterministic memory limiter firewall processors: memory_limiter: check_interval: 250ms limit_percentage: 75 # Proactive circuit-breaker tripping spike_limit_percentage: 15 tail_sampling: decision_wait: 10s # Bounded window num_traces: 50000 # Hard ceiling on in-flight memory traces expected_new_traces_per_sec: 5000 policies: - name: drop_high_volume_noise type: filter # Drop high-frequency healthchecks first under load - name: keep_catastrophic_errors type: status_code status_code: { status_codes: [ ERROR ] } - name: probabilistic_fallback type: probabilistic probabilistic: { sampling_percentage: 5.0 } service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling] # Limiter precedes sampler exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Sliding-Window Memory Eviction Firewall

OpenTelemetry Tail-Based Sampling Sliding-Window Memory Eviction Firewall defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic bounded sliding windows, memory hard-limits, and prioritized trace eviction inside OpenTelemetry Collector tail-sampling pipelines. Prevents JVM/Go runtime heap exhaustion and catastrophic silent drop storms by orchestrating circuit breakers, backpressure signaling, and deterministic reservoir eviction before telemetry buffers saturate system resources. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded tail-sampling buffers without proactive memory circuit-breakers causing heap exhaustion and silent trace starvation during downstream error storms Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded tail-sampling buffers without proactive memory circuit-breakers causing heap exhaustion and silent trace starvation during downstream error storms	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare the `memory_limiter` processor as the very first processor in every pipeline preceding `tail_sampling` to enforce hard threshold cutoffs before trace allocation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.