OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Trace In-Memory Buffers and Missing Memory Limiter Triggering Collector OOMs During Traffic Surges) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant

Enforces strict memory limits and deterministic eviction fallbacks across OpenTelemetry Collector deployments running tail-based sampling processors. Tail-based sampling requires buffering spans in-memory until trace completion; unconstrained trace buffers or misconfigured memory ballast processors under high traffic bursts trigger catastrophic out-of-memory (OOM) kills. This rule mandates pre-allocated bounds, memory limiter processor placement, trace wait-time caps, and fallback sampling routes to ensure collector tier resilience.

--- description: OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant - Enforces strict memory limits and deterministic eviction fallbacks across OpenTelemetry Collector deployments running tail-based sampling processors. Tail-based sampling requires buffering spans in-memory until trace completion; unconstrained trace buffers or misconfigured memory ballast processors under high traffic bursts trigger catastrophic out-of-memory (OOM) kills. This rule mandates pre-allocated bounds, memory limiter processor placement, trace wait-time caps, and fallback sampling routes to ensure collector tier resilience. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector-config*.yaml, **/collector-config*.yml, **/*sampling*.yaml, src/observability/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-memory-pressure-guardrail" --- # OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Trace In-Memory Buffers and Missing Memory Limiter Triggering Collector OOMs During Traffic Surges ## System Constraints You are a Principal Observability Infrastructure Architect enforcing OpenTelemetry Tail-Based Sampling resilience. Unbounded buffering of spans during high throughput leads to uncontrolled heap spikes and collector pod crashes, causing complete telemetry blackout. You must mandate that every pipeline with a tail_sampling processor places the memory_limiter processor strictly before it, calculates hard memory limits based on container cgroups, enforces a bounded num_traces and expected_new_traces_per_sec, sets deterministic decision_wait limits, and defines a fallback drop/sample policy when buffers saturate. ## Strict Guidelines - Pipelines MUST place the `memory_limiter` processor as the FIRST processor in the telemetry pipeline, strictly prior to `tail_sampling`. - The `memory_limiter` MUST specify both `check_interval` (defaulting between 1s and 2s) and `limit_percentage` (max 75-80%) or `limit_mib` calibrated against container cgroups. - The `tail_sampling` processor MUST explicitly define `num_traces`, `expected_new_traces_per_sec`, and bounded `decision_wait` (recommended <= 30s) to prevent unbounded map allocations in memory. - Engineers MUST NOT rely on unbounded default settings in `tail_sampling` or omit container cgroup memory limits in Kubernetes pod specs. - Engineers MUST configure fallback drop or probabilistic head sampling rules when the collector encounters `spike_limit_percentage` violations. - Collector infrastructure MUST export `otelcol_processor_dropped_spans` and `otelcol_processor_tail_sampling_early_drops` metrics to alert on trace starvation before OOM occurs. ## Bad vs Good Implementations ### Collector Configuration: Unbounded Tail Sampling vs. Memory-Guarded Bounded Pipeline #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Without memory_limiter, incoming spans accumulate unrestricted in the collector's heap. A 300-second decision wait during traffic spikes easily exceeds memory limits, leading to OOM-kill cycles (SIGKILL 137) and cascading telemetry data loss across the fleet. ```yaml # INSECURE / UNSTABLE: Missing memory_limiter before tail_sampling, # unbounded num_traces, and excessive decision_wait allowing heap exhaustion. processors: tail_sampling: decision_wait: 300s # 5 minutes: holds millions of spans in RAM expected_new_traces_per_sec: 100000 policies: - name: error-policy type: status_code status_code: { status_codes: [ ERROR ] } service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] # OOM killer strikes during burst exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter precedes tail_sampling to drop or refuse data proactively before heap exhaustion. The tail_sampling processor defines explicit bounds for buffer sizes (num_traces: 50000), reasonable decision_wait (10s), and deterministic policies. ```yaml # SECURE & RESILIENT: Strict memory_limiter ordering, bounded wait times, # explicit memory limits, and bounded trace allocation. processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 5000 policies: - name: latency-drop-threshold type: latency latency: { threshold_ms: 15000 } - name: error-policy type: status_code status_code: { status_codes: [ ERROR ] } - name: probabilistic-fallback type: probabilistic probabilistic: { sampling_percentage: 5.0 } batch: send_batch_size: 8192 timeout: 1s send_batch_max_size: 10240 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ``` ### TypeScript Collector Deployment Manifest Validator for Memory Limiter Safety #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Validating only the presence of tail_sampling without verifying pipeline processor ordering, memory_limiter settings, or bounded trace parameters allows dangerous configurations into production. ```typescript import * as fs from 'fs'; import * as yaml from 'js-yaml'; export function validateCollectorConfig(filePath: string): boolean { const config = yaml.load(fs.readFileSync(filePath, 'utf8')) as any; // Flaw: Only checks if tail_sampling exists, ignores order and memory_limiter return Boolean(config?.processors?.tail_sampling); } ``` #### Production Standard (Correct) > **Standard**: Validates that memory_limiter is defined, placed before tail_sampling in the trace pipeline, bounds num_traces to prevent memory starvation, and constrains decision_wait to safe operational windows. ```typescript import * as fs from 'fs'; import * as yaml from 'js-yaml'; interface CollectorConfig { processors: Record<string, Record<string, unknown>>; service?: { pipelines?: { traces?: { processors?: string[]; }; }; }; } export function validateCollectorConfig(filePath: string): { valid: boolean; errors: string[] } { const raw = fs.readFileSync(filePath, 'utf8'); const config = yaml.load(raw) as CollectorConfig; const errors: string[] = []; const traceProcessors = config?.service?.pipelines?.traces?.processors ?? []; const tailSamplingIndex = traceProcessors.indexOf('tail_sampling'); const memoryLimiterIndex = traceProcessors.indexOf('memory_limiter'); if (tailSamplingIndex !== -1) { if (memoryLimiterIndex === -1) { errors.push('memory_limiter processor MUST be present when tail_sampling is configured.'); } else if (memoryLimiterIndex > tailSamplingIndex) { errors.push('memory_limiter processor MUST precede tail_sampling in the pipeline.'); } const tailSampling = config.processors?.tail_sampling as Record<string, unknown> | undefined; if (!tailSampling?.num_traces) { errors.push('tail_sampling MUST specify num_traces to bound in-memory buffer capacity.'); } const decisionWait = typeof tailSampling?.decision_wait === 'string' ? parseInt(tailSampling.decision_wait, 10) : 0; if (decisionWait > 30) { errors.push('tail_sampling decision_wait MUST NOT exceed 30s to prevent trace accumulation.'); } } return { valid: errors.length === 0, errors }; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant

OpenTelemetry Tail-Based Sampling Bounded Buffer and Memory Eviction Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory limits and deterministic eviction fallbacks across OpenTelemetry Collector deployments running tail-based sampling processors. Tail-based sampling requires buffering spans in-memory until trace completion; unconstrained trace buffers or misconfigured memory ballast processors under high traffic bursts trigger catastrophic out-of-memory (OOM) kills. This rule mandates pre-allocated bounds, memory limiter processor placement, trace wait-time caps, and fallback sampling routes to ensure collector tier resilience. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Trace In-Memory Buffers and Missing Memory Limiter Triggering Collector OOMs During Traffic Surges Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Trace In-Memory Buffers and Missing Memory Limiter Triggering Collector OOMs During Traffic Surges	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST place the `memory_limiter` processor as the FIRST processor in the telemetry pipeline, strictly prior to `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.