Deterministic Ring-Buffered Tail Sampling with P99.9 Exemplar Retention

--- description: Deterministic Ring-Buffered Tail Sampling with P99.9 Exemplar Retention - Enforces memory-bounded tail-based trace sampling in OpenTelemetry Collectors while guaranteeing 100% trace graph fidelity for P99.9 latency outliers and distributed errors. Mandates upstream memory-limiter guards coupled with deterministic ring-buffer windowing (`decision_wait`, `num_traces`, `expected_new_traces_per_sec`) to eliminate collector OOMs during incident traffic spikes. globs: **/otel-collector*.{yaml,yml}, **/collector/**/*.{yaml,yml}, deploy/k8s/**/*.yaml, infra/monitoring/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-p999-retention" --- # Deterministic Ring-Buffered Tail Sampling with P99.9 Exemplar Retention > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace Buffering and Head-Based Sampling Discarding Tail Latency Root Causes ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing OpenTelemetry Tail-Based Sampling with P99.9 Trace Exemplar Retention. Enforce deterministic ring buffers, memory limiter ordering, and strict tail-sampling policies across all OpenTelemetry Collector pipelines. ## Strict Guidelines - Collectors MUST place the `memory_limiter` processor BEFORE the `tail_sampling` processor in the trace pipeline to prevent process crashes during traffic spikes. - Engineers MUST size the tail sampling ring buffer deterministically using `num_traces = expected_new_traces_per_sec * decision_wait * safety_factor (1.5)`. - The `tail_sampling` processor MUST configure prioritized retention rules: 100% retention on error statuses (`status_code: ERROR`), 100% retention on spans matching HTTP 5xx attributes, and 100% retention on durations exceeding the P99.9 latency SLA. - Engineers MUST set `decision_wait` to at least 1.5x the highest expected distributed trace span duration to prevent orphaned child spans and split-trace graphs. - Services MUST enable Exemplar Reservoir allocation in SDK metrics instrumentation to link P99.9 latency metrics directly to tail-sampled Trace IDs. - Pipelines MUST NEVER rely on pure head-based probabilistic sampling for mission-critical paths, as random drops extinguish outlier visibility. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Based Sampling Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Lacks the `memory_limiter` processor before `tail_sampling`, making the collector susceptible to sudden out-of-memory (OOM) crashes under burst traffic. The short 2-second decision window drops delayed spans, fragmenting trace topologies, while a crude 5% probabilistic sampler discards 95% of critical P99.9 latency anomalies and errors. ```yaml # Anti-Pattern: Missing memory limiter, arbitrary buffer sizing, head-based sampling drops P99.9 outliers receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } processors: tail_sampling: decision_wait: 2s # Too short: truncates async distributed spans policies: - name: random-probabilistic type: probabilistic probabilistic: { sampling_percentage: 5.0 } # Drops 95% of P99.9 tail anomalies pipelines: traces: receivers: [otlp] processors: [tail_sampling] # Missing memory_limiter: high risk of OOM during bursts exporters: [otlp/backend] ``` #### Production Standard (Correct) > **Standard**: Places `memory_limiter` ahead of `tail_sampling` to dynamically backpressure or drop load before memory exhaustion. The ring-buffer dimensions (`num_traces`, `decision_wait`, `expected_new_traces_per_sec`) accommodate complete trace lifecycles up to 10 seconds. Invariant retention rules guarantee 100% of P99.9 latency spikes (>2500ms) and error conditions are preserved for root-cause analysis while keeping general traffic capped at 1%. ```yaml # Production-Grade: Strict memory limiter ordering, sized ring buffer, and deterministic P99.9 + error retention receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 } processors: memory_limiter: check_interval: 250ms limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 150000 expected_new_traces_per_sec: 10000 policies: - name: retention-p999-latency type: latency latency: { threshold_ms: 2500 } # Explicit P99.9 SLA threshold - name: retention-errors type: status_code status_code: { status_codes: [ERROR] } - name: retention-http-5xx type: numeric_attribute numeric_attribute: { key: http.status_code, min_value: 500, max_value: 599 } - name: default-probabilistic-sample type: probabilistic probabilistic: { sampling_percentage: 1.0 } batch: send_batch_max_size: 2048 timeout: 500ms service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp/backend] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic Ring-Buffered Tail Sampling with P99.9 Exemplar Retention

Deterministic Ring-Buffered Tail Sampling with P99.9 Exemplar Retention defines an industrial-grade behavioral standard for devops engineering. Enforces memory-bounded tail-based trace sampling in OpenTelemetry Collectors while guaranteeing 100% trace graph fidelity for P99.9 latency outliers and distributed errors. Mandates upstream memory-limiter guards coupled with deterministic ring-buffer windowing (`decision_wait`, `num_traces`, `expected_new_traces_per_sec`) to eliminate collector OOMs during incident traffic spikes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace Buffering and Head-Based Sampling Discarding Tail Latency Root Causes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace Buffering and Head-Based Sampling Discarding Tail Latency Root Causes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Collectors MUST place the `memory_limiter` processor BEFORE the `tail_sampling` processor in the trace pipeline to prevent process crashes during traffic spikes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.