OpenTelemetry Collector Tail-Based Sampling Memory Limiter & Circuit Breaker Invariant

--- description: OpenTelemetry Collector Tail-Based Sampling Memory Limiter & Circuit Breaker Invariant - Enforces deterministic placement of the memory_limiter processor upstream of tail_sampling and exporter buffers in OpenTelemetry Collector topologies. Mandates bounded trace decision caches, proactive memory ballast thresholds, and drop policies to eliminate out-of-memory (OOM) crashes under downstream storage backpressure. Guarantees pipeline stability during cardinality spikes and network degradation. globs: **/otel-collector*.{yaml,yml}, **/collector*.{yaml,yml}, deploy/**/otel*.{yaml,yml}, k8s/**/otel*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-memory-circuit-breaker-invariant" --- # OpenTelemetry Collector Tail-Based Sampling Memory Limiter & Circuit Breaker Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Placing Tail Sampling Before Memory Limiter with Unbounded In-Flight Caches Causing Telemetry OOM Cascades ## System Constraints You are a Principal DevOps and Observability Infrastructure Architect enforcing memory safety invariants on OpenTelemetry Collector pipelines. Mandate that memory_limiter precedes tail_sampling and batch processors, ensure tail_sampling buffer ceilings are rigidly defined, and verify exporter backpressure shedding mechanisms. ## Strict Guidelines - Engineers MUST ALWAYS declare 'memory_limiter' as the absolute first processor in the traces pipeline to discard incoming telemetry before expensive tail-sampling allocation occurs. - Engineers MUST NOT deploy 'tail_sampling' with missing or default unbounded 'num_traces' and 'expected_new_traces_per_sec' parameters. - Pipelines SHOULD define 'check_interval' at 1s or less, setting 'limit_percentage' at 75-80% and 'spike_limit_percentage' at 15-20% relative to cgroup memory ceilings. - Exporters MUST configure bounded 'sending_queue' blocks with 'storage: null' or bounded persistent queues to immediately reject new spans when downstream backpressure is sustained. - Components NEVER buffer spans in-memory across the 'decision_wait' period without dynamic circuit breaking when collector memory enters warning state. ## Bad vs Good Implementations ### OpenTelemetry Collector Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Tail sampling maintains an in-memory buffer for the entire 30s decision_wait. Without memory_limiter at the entry point of the pipeline and without trace cache sizing limits, high trace ingest rates or downstream storage backpressure will cause unrestricted heap expansion, triggering an immediate OS/cgroup SIGKILL. ```yaml processors: tail_sampling: decision_wait: 30s # ANTI-PATTERN: Missing num_traces and expected_new_traces_per_sec boundaries expected_traps: 0 expected_new_traces_per_sec: 0 policies: - name: sample_errors type: status_code status_code: { status_codes: [ ERROR ] } batch: send_batch_max_size: 10000 timeout: 10s service: pipelines: traces: receivers: [otlp] # ANTI-PATTERN: memory_limiter is absent; downstream backpressure causes tail_sampling buffer to balloon until cgroup OOM-kill processors: [tail_sampling, batch] exporters: [otlp/tempo] ``` #### Production Standard (Correct) > **Standard**: Placing memory_limiter first triggers shedding before spans hit the tail_sampling state cache. Trace caches are bounded by explicit limits (num_traces: 50000), decision_wait is constrained to 10s, and the exporter queue is capped (queue_size: 1000), guaranteeing deterministic memory behavior and zero OOM incidents under backpressure. ```yaml processors: # INVARIANT: memory_limiter must be first to shed load before memory allocation memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 5000 policies: - name: sample_errors type: status_code status_code: { status_codes: [ ERROR ] } batch: send_batch_size: 2048 timeout: 1s exporters: otlp/tempo: endpoint: tempo.monitoring.svc.cluster.local:4317 sending_queue: enabled: true num_consumers: 4 queue_size: 1000 service: pipelines: traces: receivers: [otlp] # INVARIANT: Strict processor sequence: memory_limiter -> tail_sampling -> batch processors: [memory_limiter, tail_sampling, batch] exporters: [otlp/tempo] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Based Sampling Memory Limiter & Circuit Breaker Invariant

OpenTelemetry Collector Tail-Based Sampling Memory Limiter & Circuit Breaker Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic placement of the memory_limiter processor upstream of tail_sampling and exporter buffers in OpenTelemetry Collector topologies. Mandates bounded trace decision caches, proactive memory ballast thresholds, and drop policies to eliminate out-of-memory (OOM) crashes under downstream storage backpressure. Guarantees pipeline stability during cardinality spikes and network degradation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Placing Tail Sampling Before Memory Limiter with Unbounded In-Flight Caches Causing Telemetry OOM Cascades Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Placing Tail Sampling Before Memory Limiter with Unbounded In-Flight Caches Causing Telemetry OOM Cascades	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare 'memory_limiter' as the absolute first processor in the traces pipeline to discard incoming telemetry before expensive tail-sampling allocation occurs.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.