OpenTelemetry Collector Tail-Based Sampling Memory Buffer Guardrails

--- description: OpenTelemetry Collector Tail-Based Sampling Memory Buffer Guardrails - Enforces strict memory bounds, ingestion load shedding, and decoupled pipeline queues for OpenTelemetry Collector tail-based sampling configurations. Prevents Out-Of-Memory (OOM) fatal crashes and head-of-line blocking during multi-second trace evaluation windows under high span cardinality. Guarantees predictable trace aggregation buffers by strictly harmonizing `memory_limiter`, `num_traces`, `expected_new_traces_per_sec`, and pipeline batch size constraints. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector-config*.yaml, **/collector-config*.yml, deployments/**/otel*.yaml, k8s/**/otel*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-memory-buffer-guardrails" --- # OpenTelemetry Collector Tail-Based Sampling Memory Buffer Guardrails > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace Aggregation Buffering with Missing Memory Limiting and Downstream Pipeline Lockup ## System Constraints You are a Principal Observability Infrastructure Engineer enforcing memory bounds and preventing backpressure deadlocks in OpenTelemetry Collector deployments. Tail-based sampling requires retaining spans in-memory across distributed transaction latencies, creating extreme vulnerability to OOM termination and head-of-line blocking under trace volume spikes. You must mandate strict pipeline ordering, memory_limiter placement, bounded trace capacity, and proper load-balancer routing topologies. ## Strict Guidelines - Processors MUST ALWAYS declare `memory_limiter` as the very first processor in the traces pipeline before `tail_sampling` to drop spans or trigger backpressure before unconstrained memory allocation occurs. - Engineers MUST set `check_interval` to no greater than `1s`, `limit_percentage` to a maximum of `75%` (or 80% if hard limits are strictly isolated), and `spike_limit_percentage` to `20%` inside `memory_limiter` to absorb rapid ingest surges. - Configurations MUST define `num_traces` and `expected_new_traces_per_sec` explicitly in `tail_sampling`; `num_traces` MUST NOT exceed `(Trace Ingestion Rate / sec) * decision_wait * 1.5` to strictly bound trace map footprint. - The tail-sampling processor MUST define a non-zero `decision_wait` bounded between `5s` and `30s` alongside an explicit eviction strategy to prevent zombie spans from leaking memory indefinitely. - Downstream exporters MUST utilize explicit `sending_queue` blocks with `enabled: true`, non-zero `queue_size`, and bounded `num_consumers` to isolate sampling decision latency from external collector/backend I/O latency. - Architectures MUST use a two-tier collector topology: Tier 1 collectors (load-balancing exporter keyed by `trace_id`) MUST route to Tier 2 collectors running tail-based sampling to guarantee complete trace assembly without distributed trace-ID partition leaks. ## Bad vs Good Implementations ### Tail-Based Sampling Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting the memory_limiter processor leaves the collector completely unprotected against OOM kills under load spikes. Placing tail_sampling first causes span aggregation buffers to balloon indefinitely. Missing explicit num_traces causes memory hash maps to reallocate dynamically, and omitting an asynchronous sending_queue in the exporter forces tail-sampling thread pools to block on downstream backend delays. ```yaml # INSECURE: No memory_limiter, tail_sampling placed before guards, unbounded queue receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } processors: tail_sampling: decision_wait: 30s # Missing num_traces and expected_new_traces_per_sec leads to unbounded map growth expected_traversals: 0 policies: - name: errors-policy type: status_code status_code: { status_codes: [ ERROR ] } batch: send_batch_size: 8192 timeout: 1s exporters: otlp/tempo: endpoint: tempo.monitoring.svc.cluster.local:4317 # Unbounded downstream blocking; stalls tail-sampler when tempo experiences backpressure service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] # Missing memory_limiter as first processor! exporters: [otlp/tempo] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter is placed first to refuse spans before memory pressure causes an unrecoverable kernel SIGKILL. The tail_sampling processor strictly bounds internal state via decision_wait: 10s, num_traces: 50000, and expected_new_traces_per_sec: 2500. The exporter specifies a bounded sending_queue with dedicated worker consumers, preventing network latency or downstream storage delays from head-of-line blocking the sampling evaluation engine. ```yaml # RESILIENT: Bounded memory footprint, proactive shedding, isolated queues receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2500 policies: - name: drop-health-checks type: filter filter: conditions: - field: attributes["http.target"] value: ^/(healthz|metrics|live)$ operation: regex invert_match: true - name: sample-errors type: status_code status_code: { status_codes: [ ERROR ] } - name: sample-high-latency type: latency latency: { threshold_ms: 2000 } - name: probabilistic-fallback type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_size: 2048 timeout: 500ms send_batch_max_size: 4096 exporters: otlp/tempo: endpoint: tempo.monitoring.svc.cluster.local:4317 sending_queue: enabled: true num_consumers: 8 queue_size: 5000 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp/tempo] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Based Sampling Memory Buffer Guardrails

OpenTelemetry Collector Tail-Based Sampling Memory Buffer Guardrails defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory bounds, ingestion load shedding, and decoupled pipeline queues for OpenTelemetry Collector tail-based sampling configurations. Prevents Out-Of-Memory (OOM) fatal crashes and head-of-line blocking during multi-second trace evaluation windows under high span cardinality. Guarantees predictable trace aggregation buffers by strictly harmonizing `memory_limiter`, `num_traces`, `expected_new_traces_per_sec`, and pipeline batch size constraints. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace Aggregation Buffering with Missing Memory Limiting and Downstream Pipeline Lockup Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace Aggregation Buffering with Missing Memory Limiting and Downstream Pipeline Lockup	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Processors MUST ALWAYS declare `memory_limiter` as the very first processor in the traces pipeline before `tail_sampling` to drop spans or trigger backpressure before unconstrained memory allocation occurs.

Processors MUST ALWAYS declare `memory_limiter` as the very first processor in the traces pipeline before `tail_sampling` to drop spans or trigger backpressure before unconstrained memory allocation occurs.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.