OpenTelemetry Collector Tail-Sampling Reservoir & Cardinality Guardrail

--- description: OpenTelemetry Collector Tail-Sampling Reservoir & Cardinality Guardrail - Enforces strict memory-bounded tail-sampling policies, memory limiter thresholds, and adaptive reservoir limits within OpenTelemetry Collector deployments. Prevents out-of-memory crashes caused by high-cardinality trace dimensions and volumetric bursts during systemic cascading failures. globs: **/otel-collector*.{yaml,yml}, **/collector/*.{yaml,yml}, **/helm/**/otel*.{yaml,yml}, **/k8s/**/otel*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-adaptive-tail-sampling-cardinality" --- # OpenTelemetry Collector Tail-Sampling Reservoir & Cardinality Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Head/Tail Sampling and Uncapped Attribute Cardinality Triggering Collector OOM Panics ## System Constraints You are a Principal Reliability Architect enforcing OpenTelemetry Collector resilience invariants. Mandate non-negotiable memory limiter configurations, strict tail-sampling trace evaluation bounds, dynamic attribute stripping to control high-cardinality explosion, and rate-limiting reservoir caps across all pipeline definitions. ## Strict Guidelines - Pipelines MUST place the `memory_limiter` processor as the FIRST processor in every trace pipeline to intercept allocations before any downstream buffering or parsing occurs. - The `memory_limiter` MUST specify both `check_interval` (recommended: 1s to 5s) and strict percentages: `limit_percentage` MUST NOT exceed 80%, and `spike_limit_percentage` MUST NOT exceed 20%. - Tail-sampling configurations MUST define an explicit `expected_new_traces_per_sec` and `num_traces` buffer cap to prevent unconstrained hash map expansion under traffic spikes. - Engineers MUST configure a `rate_limiting` or `probabilistic` policy within the `tail_sampling` processor to act as a fallback safety reservoir when error traces overwhelm the cluster. - High-cardinality attributes (such as user IDs, raw query strings, IP addresses, or unique UUIDs) MUST be sanitized or dropped via the `transform` or `attributes` processor before tail-sampling evaluation and exporter dispatch. - Pipelines MUST NOT rely solely on head-based probabilistic sampling at the SDK layer when compliance or error-capture SLAs mandate tail evaluation, nor configure tail-sampling without backpressure-aware exporter queues (`sending_queue` with explicit `queue_size`). ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Sampling Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omits the `memory_limiter` processor, allows an unbounded `decision_wait` without setting `num_traces` or `expected_new_traces_per_sec`, and uses an unmetered status_code policy that captures 100% of errors during an incident, exhausting the collector memory buffer and crashing the agent. ```yaml # Anti-Pattern: Missing memory limiter, unbounded tail sampling, and raw cardinality forwarding receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: tail_sampling: decision_wait: 30s policies: - name: always-keep-errors type: status_code status_code: { status_codes: [ ERROR ] } exporters: otlp: endpoint: tempo:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] # Lacks memory_limiter as first processor # Uncapped error volume will exhaust collector memory pool during outage processors: [tail_sampling] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: Places `memory_limiter` first to preempt allocation bursts, uses `transform` to drop unindexed high-cardinality attributes, sizes the tail-sampling memory footprint via explicit trace capacity (`num_traces: 50000`), and composes the error sampling policy with a rate-limiting reservoir to ensure survivability during incident cascades. ```yaml # Resilient Pattern: Memory limiter, cardinality sanitation, and rate-bounded reservoir tail sampling receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: # Invariant: memory_limiter MUST be the first processor in the pipeline memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 transform: error_mode: ignore trace_statements: - context: span statements: # Drop unbound high-cardinality fields before indexing/storage - delete_key(attributes, "db.statement") - delete_key(attributes, "http.request.header.authorization") - truncate_all(attributes, 1024) tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: - name: error-bounded-reservoir type: and and: and_sub_policy: - name: error-status type: status_code status_code: { status_codes: [ ERROR ] } - name: rate-limit-reservoir type: rate_limiting rate_limiting: { spans_per_second: 500 } - name: baseline-probabilistic type: probabilistic probabilistic: { sampling_percentage: 5.0 } exporters: otlp: endpoint: tempo:4317 sending_queue: enabled: true num_consumers: 10 queue_size: 5000 retry_on_failure: enabled: true max_elapsed_time: 60s service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, transform, tail_sampling] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling Reservoir & Cardinality Guardrail

OpenTelemetry Collector Tail-Sampling Reservoir & Cardinality Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory-bounded tail-sampling policies, memory limiter thresholds, and adaptive reservoir limits within OpenTelemetry Collector deployments. Prevents out-of-memory crashes caused by high-cardinality trace dimensions and volumetric bursts during systemic cascading failures. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Head/Tail Sampling and Uncapped Attribute Cardinality Triggering Collector OOM Panics Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Head/Tail Sampling and Uncapped Attribute Cardinality Triggering Collector OOM Panics	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST place the `memory_limiter` processor as the FIRST processor in every trace pipeline to intercept allocations before any downstream buffering or parsing occurs.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.