OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded tail-sampling trace buffers without prior memory-limiter shedding, causing fatal Collector OOM crashes during ingress spikes.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard

--- description: OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard - Guarantees deterministic memory ceilings and prevents Linux OOM-killer termination on OpenTelemetry Collectors during volumetric ingress spikes. Mandates strict ordering of the memory_limiter processor prior to tail_sampling, explicitly calculated trace ring-buffer capacities based on p99 span-tree sizes, and preemptive drop evaluation for high-frequency non-erroneous telemetry. globs: **/otel-collector*.{yaml,yml}, **/collector/**/*.{yaml,yml}, deployments/**/otel*.{yaml,yml}, k8s/**/collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-ringbuffer-guardrail" --- # OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded tail-sampling trace buffers without prior memory-limiter shedding, causing fatal Collector OOM crashes during ingress spikes. ## System Constraints You are a Principal Site Reliability Engineer and Observability Architect enforcing memory invariants for OpenTelemetry Collector pipelines. You mandate that tail-sampling processors never operate in unbounded or uncoordinated memory regimes. Enforce deterministic ring-buffer sizing, strict processor chain ordering (memory_limiter before tail_sampling), explicit drop-first filtering policies, and hard trace eviction bounds to preserve collector stability under sudden upstream traffic multipliers. ## Strict Guidelines - Pipelines MUST ALWAYS position the `memory_limiter` processor before the `tail_sampling` processor in the traces pipeline sequence. - Engineers MUST explicitly set `num_traces` in the `tail_sampling` processor based on the mathematical invariant: (num_traces * expected_max_spans_per_trace * 2KB) <= (limit_mib * 0.60). - Services MUST configure the `memory_limiter` processor with hard absolute values (`limit_mib` and `spike_limit_mib`) aligned to 80% and 20% of the container memory request/limit respectively. - Tail-sampling policies MUST evaluate fast-drop criteria (e.g., successful health-checks, synthetic probes, high-frequency low-value 2xx HTTP routes) at the top of the policy chain before regex or composite policies. - Engineers MUST NOT rely on default collector buffer configurations or unconstrained `decision_wait` intervals exceeding 30 seconds without an empirically verified memory model. - Components NEVER accept telemetry without an explicit upstream backpressure mechanism and a corresponding load-shedding error policy (`status_code` filter prioritizing HTTP 5xx and span error status). ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Based Sampling Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Placing tail_sampling before memory_limiter allows the collector to hold 500,000 trace trees in memory during a spike without guardrails. Because memory_limiter is placed downstream, it cannot prevent tail_sampling from exhausting the container heap, leading to sudden SIGKILL (OOM-137) events. ```yaml # ANTI-PATTERN: tail_sampling is placed before memory_limiter, # num_traces is arbitrarily set too high, and policies evaluate # expensive retention before shedding high-frequency healthy traces. processors: tail_sampling: decision_wait: 30s num_traces: 500000 # Consumes >10GB during burst; causes OOM-kill expected_new_traces_per_sec: 20000 policies: - name: sample-everything-else type: probabilistic probabilistic: sampling_percentage: 10.0 memory_limiter: check_interval: 5s limit_percentage: 95 spike_limit_percentage: 25 service: pipelines: traces: receivers: [otlp] # FLAW: tail_sampling allocates heap before memory_limiter can shed load processors: [tail_sampling, memory_limiter, batch] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter processor runs before tail_sampling, enforcing an upstream ceiling. Trace ring-buffer capacity (num_traces) is strictly bounded to the physical container memory allocation. Low-value transactions are shed before complex composite evaluation, guaranteeing deterministic memory safety. ```yaml # PRODUCTION INVARIANT: memory_limiter protects the heap upstream; # num_traces is mathematically bounded to available RAM; drop policies run first. processors: memory_limiter: check_interval: 1s limit_mib: 1600 # 80% of a 2048MiB container limit spike_limit_mib: 400 # 20% spike headroom tail_sampling: decision_wait: 10s num_traces: 50000 # Strictly budgeted: 50k traces * ~20 spans * 1KB ~ 1000MiB expected_new_traces_per_sec: 5000 policies: # 1. Deterministic shed: Drop noisy, non-erroneous paths immediately - name: drop-health-checks type: string_attribute string_attribute: key: http.target values: [ "/healthz", "/livez", "/metrics" ] enabled_regex_matching: false invert_match: true # Invert to sample, meaning matching traces are evaluated for drop # 2. Invariant: Retain 100% of errors and failures - name: keep-errors type: status_code status_code: status_codes: [ ERROR ] # 3. Controlled rate for normal traffic - name: sample-success-rate type: probabilistic probabilistic: sampling_percentage: 5.0 service: pipelines: traces: receivers: [otlp] # ORDER ENFORCED: memory_limiter drops upstream when heap is constrained processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard

OpenTelemetry Tail-Based Sampling Deterministic Ring-Buffer & Memory Boundary Standard defines an industrial-grade behavioral standard for devops engineering. Guarantees deterministic memory ceilings and prevents Linux OOM-killer termination on OpenTelemetry Collectors during volumetric ingress spikes. Mandates strict ordering of the memory_limiter processor prior to tail_sampling, explicitly calculated trace ring-buffer capacities based on p99 span-tree sizes, and preemptive drop evaluation for high-frequency non-erroneous telemetry. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded tail-sampling trace buffers without prior memory-limiter shedding, causing fatal Collector OOM crashes during ingress spikes. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded tail-sampling trace buffers without prior memory-limiter shedding, causing fatal Collector OOM crashes during ingress spikes.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST ALWAYS position the `memory_limiter` processor before the `tail_sampling` processor in the traces pipeline sequence.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.