OpenTelemetry Tail-Sampling Exemplar and Error Retention Invariant

--- description: OpenTelemetry Tail-Sampling Exemplar and Error Retention Invariant - Enforces deterministic retention of all distributed traces associated with high-percentile latency (p99+) and HTTP 5xx/gRPC errors within OpenTelemetry Collector tail-sampling pipelines. Prevents broken metric-to-trace exemplar correlation in observability backends caused by premature probabilistic drop policies. globs: **/otel-collector*.{yaml,yml}, **/collector*.{yaml,yml}, **/otel/*.{yaml,yml}, deployments/**/otel*.{yaml,yml}, helm/**/values*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-tail-sampling-exemplar-retention" --- # OpenTelemetry Tail-Sampling Exemplar and Error Retention Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Dropping p99 and 5xx traces through downstream probabilistic tail-sampling, creating orphan metric exemplars ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing OpenTelemetry Collector tail-sampling guarantees. You mandate that any OpenTelemetry Collector configuration MUST guarantee 100% retention for spans matching error conditions (HTTP 5xx, gRPC != 0, span status error) and high-latency thresholds (p99 SLA violations) before any probabilistic sampling policy is evaluated. Traces linked to metric exemplars MUST NOT return 404 in storage backends like Jaeger or Tempo. ## Strict Guidelines - Engineers MUST ALWAYS order tail-sampling policies such that deterministic error status rules and latency threshold rules appear BEFORE probabilistic or rate-limiting rules. - Engineers MUST configure 'decision_wait' to at least 1.5x the maximum expected p99 trace duration (typically 10s-30s) to prevent premature partial trace sampling evaluations. - Collector pipelines MUST define a 100% sample percentage for spans with 'status.code == STATUS_CODE_ERROR' or HTTP status >= 500. - Services MUST NOT use head-based probabilistic sampling if Prometheus/OTel exemplars are recorded at client runtimes; trace sampling MUST be deferred to collector tail-sampling. - Components NEVER drop traces with 'forced-sampling=true' or 'sampled=true' attributes injected by upstream debug or testing harnesses. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Sampling Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Evaluating probabilistic sampling first drops up to 95% of critical 5xx errors and p99 latency spikes before they reach error or latency policies. This causes Prometheus/Grafana metric exemplars to point to missing traces (404 errors in Tempo/Jaeger). Additionally, a 2s decision_wait truncates long-running traces. ```yaml processors: tail_sampling: decision_wait: 2s num_traces: 10000 expected_new_traces_per_sec: 2000 # ANTI-PATTERN: Probabilistic sampling evaluated first drops # p99 anomalies and 5xx errors before specialized rules match policies: - name: probabilistic-sample type: probabilistic probabilistic: sampling_percentage: 5.0 - name: error-conditions type: status_code status_code: status_codes: [ERROR] - name: latency-p99 type: latency latency: threshold_ms: 2000 ``` #### Production Standard (Correct) > **Standard**: Error conditions and high-latency thresholds are prioritized at 100% retention. All traces containing span errors, HTTP 5xx responses, or >1500ms duration are guaranteed preservation. Only normal traces fall through to the 2% probabilistic sampler, ensuring all metric exemplars map directly to intact distributed traces. ```yaml processors: tail_sampling: decision_wait: 15s num_traces: 50000 expected_new_traces_per_sec: 5000 # PRODUCTION FIX: Deterministic retention for errors and p99 latency # evaluated ahead of probabilistic baseline to guarantee exemplar correlation policies: - name: retain-status-errors type: status_code status_code: status_codes: [ERROR] - name: retain-http-5xx type: numeric_attribute numeric_attribute: key: http.response.status_code min_value: 500 max_value: 599 - name: retain-p99-latency type: latency latency: threshold_ms: 1500 - name: fallback-probabilistic-sampling type: probabilistic probabilistic: sampling_percentage: 2.0 ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Sampling Exemplar and Error Retention Invariant

OpenTelemetry Tail-Sampling Exemplar and Error Retention Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic retention of all distributed traces associated with high-percentile latency (p99+) and HTTP 5xx/gRPC errors within OpenTelemetry Collector tail-sampling pipelines. Prevents broken metric-to-trace exemplar correlation in observability backends caused by premature probabilistic drop policies. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Dropping p99 and 5xx traces through downstream probabilistic tail-sampling, creating orphan metric exemplars Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Dropping p99 and 5xx traces through downstream probabilistic tail-sampling, creating orphan metric exemplars	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS order tail-sampling policies such that deterministic error status rules and latency threshold rules appear BEFORE probabilistic or rate-limiting rules.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.