OpenTelemetry Tail-Based Sampling and Trace Error Preservation Guardrail

--- description: OpenTelemetry Tail-Based Sampling and Trace Error Preservation Guardrail - Enforces 100% trace capture for all error trees and high-latency anomalies via OpenTelemetry Collector tail-sampling policies while downsampling nominal traffic to protect ingest quotas. Prevents distributed trace fragmentation caused by aggressive upstream head-based sampling and premature collector decision windows. globs: **/otel-collector*.{yaml,yml}, **/collector/**/*.{yaml,yml}, src/telemetry/**/*.{ts,js,go,py}, src/instrumentation/**/*.{ts,js,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-tail-sampling-error-preservation-standard" --- # OpenTelemetry Tail-Based Sampling and Trace Error Preservation Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Premature Head-Based Probabilistic Dropping of Distributed Traces and Incomplete Leaf-Span Error Capture ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing the OpenTelemetry Tail-Based Trace Sampling Error-Preservation Standard. Mandate that application SDKs always emit 100% of spans upstream (head-sampling disabled or set to 1.0) and that routing collectors employ stateful tail-sampling processors configured to unconditionally preserve 100% of error traces (status_code=ERROR, HTTP 5xx, or exception events) and latency outliers, while deterministically downsampling nominal spans. Enforce strict decision-wait window buffers and trace-ID routing affinity. ## Strict Guidelines - Engineers MUST ALWAYS configure edge/application OpenTelemetry SDKs with `AlwaysOnSampler` (or parent-based equivalent) to delegate sampling decisions entirely to the collector layer. - Engineers MUST NOT apply probabilistic head-based sampling at the application layer when root-cause trace context across downstream microservices is required. - Collector pipelines MUST configure the `tail_sampling` processor with policy rules evaluating `status_code` (`ERROR`), explicit error attributes (`error=true`, `http.status_code >= 500`), and latency thresholds before falling back to probabilistic rules. - The `tail_sampling` processor MUST define a `decision_wait` window that strictly exceeds the 99th percentile end-to-end distributed transaction duration (minimum recommended: 10s-30s) to prevent orphan child span collection. - Collector instances executing tail sampling MUST sit behind a trace-ID-consistent load-balancing layer (e.g., OpenTelemetry Load-Balancing Exporter using `routing_key: trace_id`) to ensure all spans for a trace arrive at the same collector instance. - Application code MUST explicitly set span status to `StatusCode.ERROR` and record exceptions using `span.recordException(err)` rather than merely logging errors to stdout. - Components NEVER drop root spans of failed requests; any error anywhere in the distributed trace tree MUST trigger the retention of the entire root-to-leaf trace execution path. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Sampling Processor Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Probabilistic sampling executes blind downsampling before a trace completes. If an error occurs in downstream services (e.g., database failure or HTTP 500), 95% of those failing traces are permanently dropped, making distributed debugging impossible. ```yaml # INSECURE/DEFECTIVE: Naive probabilistic head sampling drops 99% of error traces processors: probabilistic_sampler: sampling_percentage: 5.0 service: pipelines: traces: receivers: [otlp] processors: [probabilistic_sampler, batch] exporters: [otlp/backend] ``` #### Production Standard (Correct) > **Standard**: The tail_sampling processor retains spans in memory for 30s across all services. If any span within the trace registers an ERROR status, HTTP 5xx code, or exceeds 2000ms latency, 100% of the entire trace tree is preserved, while nominal healthchecks and nominal baseline requests are safely downsampled. ```yaml # PRODUCTION STANDARD: Tail-based sampling guarantees 100% error and latency capture processors: tail_sampling: decision_wait: 30s num_traces: 100000 expected_new_traces_per_sec: 5000 policies: - name: drop-healthchecks type: string_attribute string_attribute: { key: http.target, values: [ "/healthz", "/metrics" ], enabled_regex_matching: false, invert_match: false } - name: preserve-errors type: status_code status_code: { status_codes: [ ERROR ] } - name: preserve-http-server-errors type: numeric_attribute numeric_attribute: { key: http.status_code, min_value: 500, max_value: 599 } - name: preserve-latency-anomalies type: latency latency: { threshold_ms: 2000 } - name: sample-nominal-traffic type: probabilistic probabilistic: { sampling_percentage: 1.0 } service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] exporters: [otlp/backend] ``` ### Application-Side Semantic Error and Exception Recording (TypeScript) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Failing to invoke span.setStatus({ code: SpanStatusCode.ERROR }) and span.recordException(err) keeps the span status as UNSET. The downstream tail sampler will not recognize this trace as an error and will subject it to probabilistic dropping. ```typescript import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('checkout-service'); export async function handlePayment(paymentId: string) { return tracer.startActiveSpan('handlePayment', async (span) => { try { return await processCharge(paymentId); } catch (err: any) { // VIOLATION: Logging without setting OTel span status leaves span nominal (UNSET), // causing the tail sampler to misclassify the trace as successful. console.error('Payment failed', err); throw err; } finally { span.end(); } }); } ``` #### Production Standard (Correct) > **Standard**: Explicitly marks the span status as SpanStatusCode.ERROR and records structured exception events. This guarantees that the collector's tail-sampling processor status_code policy matches and preserves the full trace context. ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; const tracer = trace.getTracer('checkout-service'); export async function handlePayment(paymentId: string) { return tracer.startActiveSpan('handlePayment', async (span) => { try { const result = await processCharge(paymentId); span.setStatus({ code: SpanStatusCode.OK }); return result; } catch (err: unknown) { const error = err instanceof Error ? err : new Error(String(err)); // Mandated invariant: Record exception details and explicitly flag ERROR status span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, message: error.message, }); span.setAttribute('error', true); throw error; } finally { span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Based Sampling and Trace Error Preservation Guardrail

OpenTelemetry Tail-Based Sampling and Trace Error Preservation Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces 100% trace capture for all error trees and high-latency anomalies via OpenTelemetry Collector tail-sampling policies while downsampling nominal traffic to protect ingest quotas. Prevents distributed trace fragmentation caused by aggressive upstream head-based sampling and premature collector decision windows. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Premature Head-Based Probabilistic Dropping of Distributed Traces and Incomplete Leaf-Span Error Capture Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Premature Head-Based Probabilistic Dropping of Distributed Traces and Incomplete Leaf-Span Error Capture	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure edge/application OpenTelemetry SDKs with `AlwaysOnSampler` (or parent-based equivalent) to delegate sampling decisions entirely to the collector layer.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.