OpenTelemetry Tail-Based Sampling Bounded In-Memory Trace Cache Guardrail

--- description: OpenTelemetry Tail-Based Sampling Bounded In-Memory Trace Cache Guardrail - Enforces strict memory limits and drop-tail eviction strategies on OpenTelemetry collectors executing tail-based sampling under cascading service failure cascades. Prevents Out-Of-Memory (OOM) collector termination by bounding concurrent trace aggregation buffers, mandating deterministic memory limits, and configuring backpressure-safe sampling thresholds. globs: deploy/otel-collector/**/*.{yaml,yml}, k8s/monitoring/**/*.{yaml,yml}, helm/**/*.{yaml,yml}, src/observability/**/*.{ts,go,yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-bounded-cache-standard" --- # OpenTelemetry Tail-Based Sampling Bounded In-Memory Trace Cache Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Collector Heap Exhaustion via Indefinite Span Aggregation in Tail-Sampling Processors ## System Constraints You are a Principal Observability Architect enforcing strict memory ceiling and eviction invariants on OpenTelemetry Collector configurations employing tail-based sampling. Tail-based sampling requires holding entire traces in volatile memory across configurable decision latency windows. Under anomalous cascading microservice failures, span volume surges non-linearly, causing trace queues to saturate heap memory, trigger OOM kills, and permanently blind operational telemetry. You must enforce explicit `num_traces` ceilings, explicit `memory_limiter` boundaries configured strictly prior to tail sampling in pipelines, and deterministic early-drop policies when buffers reach saturation. ## Strict Guidelines - Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in any pipeline preceding `tail_sampling`. - Engineers MUST NEVER configure `tail_sampling` with unbounded `num_traces` or `expected_new_traces_per_sec` without explicit caps matching pod cgroup limits. - Collector container configurations MUST allocate `check_interval`, `limit_percentage`, and `spike_limit_percentage` such that `limit_percentage` + `spike_limit_percentage` strictly equals or remains below 85% of total pod memory limits. - Tail sampling configurations MUST set `decision_wait` to a finite duration (maximum 30s) and explicitly configure `num_traces` to bound heap usage to: (Allocated Heap * 0.6) / (Average Trace Size Bytes). - Pipelines MUST configure fallback routing or load-balancing exporters with upstream trace-ID consistent routing to guarantee all spans of a trace arrive at the same collector instance without triggering distributed cache amplification. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Based Sampling Memory Boundary Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The pipeline omits the memory_limiter processor and sets an exorbitant decision_wait of 300s without a defined num_traces cap. During a cascading microservice failure, millions of spans saturate memory, causing the OS kernel to OOMKill the collector, resulting in complete observability loss. ```yaml # Anti-Pattern: Missing memory_limiter before tail_sampling, unbounded num_traces, dangerous decision_wait processors: tail_sampling: decision_wait: 300s # Too high: holds traces in memory for 5 minutes # Missing num_traces: defaults to unbound or huge memory allocation expected_new_traces_per_sec: 10000 policies: - name: sample-errors type: status_code status_code: { status_codes: [ ERROR ] } service: pipelines: traces: receivers: [otlp] processors: [tail_sampling] # Missing memory_limiter guardrail exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: Enforces memory ceilings via the memory_limiter processor positioned before tail_sampling. Limits the aggregated concurrent trace count to 50,000 with a bounded 10s decision window, ensuring memory usage never exceeds 85% of container cgroups during failure storms. ```yaml # Invariant Compliant: Strict memory_limiter upfront, bounded num_traces, short decision window processors: memory_limiter: check_interval: 1s limit_percentage: 70 spike_limit_percentage: 15 tail_sampling: decision_wait: 10s num_traces: 50000 # Hard ceiling on in-memory concurrent trace aggregation expected_new_traces_per_sec: 2000 policies: - name: drop-debug-traces type: numeric_attribute numeric_attribute: { key: "http.status_code", value_condition: { equal_to: 200 } } - name: error-policy type: status_code status_code: { status_codes: [ ERROR ] } service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling] # memory_limiter MUST precede sampling exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Bounded In-Memory Trace Cache Guardrail

OpenTelemetry Tail-Based Sampling Bounded In-Memory Trace Cache Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory limits and drop-tail eviction strategies on OpenTelemetry collectors executing tail-based sampling under cascading service failure cascades. Prevents Out-Of-Memory (OOM) collector termination by bounding concurrent trace aggregation buffers, mandating deterministic memory limits, and configuring backpressure-safe sampling thresholds. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Collector Heap Exhaustion via Indefinite Span Aggregation in Tail-Sampling Processors Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Collector Heap Exhaustion via Indefinite Span Aggregation in Tail-Sampling Processors	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in any pipeline preceding `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.