OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard

--- description: OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard - Enforces strict memory-bounded trace buffers and priority-aware adaptive eviction in OpenTelemetry collector pipelines. Ensures anomalous spans, HTTP 5xx errors, and high-latency traces are deterministically preserved during traffic spikes while non-critical root and intermediate spans are pruned to avoid collector OOM kills. Mandates consistent hash-based trace routing to guarantee complete trace assembly before sampling decisions are made. globs: **/collector/*.yaml, **/otel-collector*.yaml, **/deployment/otel/**/*.yaml, src/observability/**/*.ts, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-adaptive-eviction-standard" --- # OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Collector Memory Buffers and Random Drop Shedding Under Ingress Surges ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing the OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard. You must mandate strict memory-limiter controls preceding all tail-sampling stages, ensure traces are partitioned using load-balancing exporters by TraceID, and enforce tiered multi-policy sampling that prioritizes preservation of spans with error status codes and latency breaches above nominal traffic. ## Strict Guidelines - Engineers MUST ALWAYS configure the `memory_limiter` processor BEFORE the `tail_sampling` processor in the collector trace pipeline. - Engineers MUST NOT allow collector memory usage to exceed hard limits without defined ballast memory allocations or ballast-free soft/hard percentages (e.g., check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20). - Services MUST ensure upstream load-balancing collectors use consistent hashing on `trace_id` so that all spans belonging to a trace converge on the same collector instance evaluating tail-sampling policies. - Collector configurations MUST prioritize policy evaluation orders such that error-matching (`status_code: ERROR`) and high-latency threshold conditions execute before probabilistic rate-limiting rules. - Pipelines NEVER drop incomplete traces arbitrarily without recording the eviction metric (`otelcol_processor_tail_sampling_traces_dropped_total`) and flagging partial traces with an eviction span attribute. ## Bad vs Good Implementations ### Configuring Memory-Limited Priority Tail Sampling with Adaptive Fallbacks #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing tail_sampling before memory_limiter exposes the collector buffer to unbounded allocations during traffic spikes, leading to OS-level OOM kills. The single probabilistic policy discards traces containing HTTP 5xx errors and critical latency anomalies indiscriminately. ```yaml # Anti-Pattern: tail_sampling runs before memory_limiter without load-balancer trace routing # Missing error-prioritized rules; crashes under traffic bursts or indiscriminately drops 5xx traces processors: tail_sampling: decision_wait: 30s num_traces: 1000000 # Unbounded allocation causing OOM expected_new_traces_per_sec: 50000 policies: - name: probabilistic type: probabilistic probabilistic: sampling_percentage: 10 memory_limiter: check_interval: 5s limit_mib: 4096 service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, memory_limiter] # WRONG ORDER: Tail buffer consumes memory first exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: Memory limiter executes before buffering to protect collector nodes from heap exhaustion. Tail-sampling evaluates explicit error retention and latency thresholds first, only applying probabilistic reduction to baseline healthy traffic. ```yaml # Production Standard: Memory Limiter preceding Prioritized Tail-Based Policy Pipeline processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 5000 policies: - name: drop-health-checks type: string_attribute string_attribute: key: http.target values: ["/healthz", "/metrics", "/ready"] enabled_regex_matching: false invert_match: true - name: retain-errors type: status_code status_code: status_codes: [ERROR] - name: retain-high-latency type: latency latency: threshold_ms: 2000 - name: adaptive-probabilistic-fallback type: probabilistic probabilistic: sampling_percentage: 5.0 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling] # Correct sequence: bounds memory before buffering exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard

OpenTelemetry Tail-Based Sampling Adaptive Eviction Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded trace buffers and priority-aware adaptive eviction in OpenTelemetry collector pipelines. Ensures anomalous spans, HTTP 5xx errors, and high-latency traces are deterministically preserved during traffic spikes while non-critical root and intermediate spans are pruned to avoid collector OOM kills. Mandates consistent hash-based trace routing to guarantee complete trace assembly before sampling decisions are made. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Collector Memory Buffers and Random Drop Shedding Under Ingress Surges Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Collector Memory Buffers and Random Drop Shedding Under Ingress Surges	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure the `memory_limiter` processor BEFORE the `tail_sampling` processor in the collector trace pipeline.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.