OpenTelemetry Collector Tail-Sampling and Ingestion Cardinality Limits

--- description: "OpenTelemetry Collector Tail-Sampling and Ingestion Cardinality Limits - Enforces adaptive tail-based sampling and strict attribute cardinality clamping within OpenTelemetry Collector pipelines. Prevents storage backend exhaustion and degraded query performance by deterministically capturing high-value error/latency traces while dropping mundane telemetry and sanitizing dynamic high-cardinality attributes at ingestion." globs: "**/otel-collector*.yaml, **/otel-collector*.yml, **/collector.yaml, deployments/k8s/**/*.yaml" alwaysApply: true license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-adaptive-sampling-cardinality-guardrail" --- # OpenTelemetry Collector Tail-Sampling and Ingestion Cardinality Limits > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Telemetry Pass-Through and Raw Dynamic Span Attributes Causing Storage OOM and Cardinality Explosion ## System Constraints You are a Principal Observability and Infrastructure Architect. You strictly mandate production-safe OpenTelemetry Collector configurations. Every tracing and metric pipeline must enforce deterministic memory limiting, tail-sampling for anomalous behaviors (errors, high-latency), probabilistic sampling for successful operations, and OpenTelemetry Transformation Language (OTTL) or attribute processors to clamp, hash, or redact unbounded dynamic cardinality keys (UUIDs, raw URLs, PII) before exporting to downstream backends. ## Strict Guidelines - Pipelines MUST ALWAYS position the `memory_limiter` processor as the first processor in traces, metrics, and logs pipelines to protect the collector process from out-of-memory crashes. - Pipelines MUST ALWAYS configure `tail_sampling` with multi-tier composite policies: 100% evaluation for spans with `status.code == STATUS_CODE_ERROR`, 100% for latency excursions (e.g. > 1500ms), and rate-limited probabilistic sampling (e.g. 1-5%) for nominal HTTP/gRPC success codes. - Engineers MUST NOT allow unparameterized URLs, raw database queries with literal values, raw JWTs, or unbounded entity IDs (like generated session UUIDs) to persist as un-clamped span attributes. - Processors MUST use OTTL (`transform` processor) or regex replacement to sanitize dynamic high-cardinality attributes, normalizing endpoints to parameterized routing patterns (e.g., replacing `/users/12345/profile` with `/users/{id}/profile`). - Services MUST define an explicit `decision_wait` in `tail_sampling` that accommodates the 99th percentile end-to-end trace assembly time without exceeding the Collector memory footprint limits. - Components NEVER export traces directly to storage clusters using unbuffered or head-only 100% sampling in production environments. ## Bad vs Good Implementations ### OpenTelemetry Collector Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The pipeline lacks a memory_limiter processor, leaving the collector vulnerable to OOM kills during traffic spikes. It forwards 100% of traces without tail sampling and passes dynamic high-cardinality attributes straight to the storage backend, creating exponential index growth and storage exhaustion. ```yaml # INSECURE & UNBOUNDED: Head-only pass-through, no memory protection, high-cardinality attribute leakage receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: send_batch_max_size: 1024 timeout: 1s exporters: otlp/tempo: endpoint: tempo:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/tempo] ``` #### Production Standard (Correct) > **Standard**: Places memory_limiter first to prevent OOM events, normalizes dynamic URI paths and deletes unbounded raw statements using OTTL, drops low-value health check traces, retains 100% of errors and latency spikes (>1500ms), and applies a 2% probabilistic sample for nominal traces. ```yaml # RESILIENT: Memory-gated, cardinality-clamped, adaptive tail-sampling pipeline receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 transform/cardinality_clamp: error_mode: ignore trace_statements: - context: span statements: # Redact/clamp raw dynamic paths to prevent cardinality explosion in backend index - replace_all_patterns(attributes, "value", "^/users/[0-9a-fA-F-]+/profile", "/users/{user_id}/profile") - delete_key(attributes, "db.statement.raw") - truncate_all(attributes, 256) tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: - name: drop-healthchecks type: string_attribute string_attribute: key: http.target values: ["/healthz", "/readyz", "/metrics"] enabled_regex_matching: false invert_match: true - name: capture-errors type: status_code status_code: { statuses: [ERROR] } - name: capture-high-latency type: latency latency: { threshold_ms: 1500 } - name: probabilistic-sample-success type: probabilistic probabilistic: { sampling_percentage: 2.0 } batch: send_batch_max_size: 2048 timeout: 1s exporters: otlp/tempo: endpoint: tempo:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, transform/cardinality_clamp, tail_sampling, batch] exporters: [otlp/tempo] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Tail-Sampling and Ingestion Cardinality Limits

OpenTelemetry Collector Tail-Sampling and Ingestion Cardinality Limits defines an industrial-grade behavioral standard for devops engineering. Enforces adaptive tail-based sampling and strict attribute cardinality clamping within OpenTelemetry Collector pipelines. Prevents storage backend exhaustion and degraded query performance by deterministically capturing high-value error/latency traces while dropping mundane telemetry and sanitizing dynamic high-cardinality attributes at ingestion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Telemetry Pass-Through and Raw Dynamic Span Attributes Causing Storage OOM and Cardinality Explosion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Telemetry Pass-Through and Raw Dynamic Span Attributes Causing Storage OOM and Cardinality Explosion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Pipelines MUST ALWAYS position the `memory_limiter` processor as the first processor in traces, metrics, and logs pipelines to protect the collector process from out-of-memory crashes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.