OpenTelemetry v1.28 Tail-Based Adaptive Sampling and Semantic Span Retention

--- description: OpenTelemetry v1.28 Tail-Based Adaptive Sampling and Semantic Span Retention - Enforces 100% trace capture for anomalous, high-latency, and error-state spans using OpenTelemetry Collector tail-based sampling while dynamically downsampling nominal transactions. Mandates compliance with OpenTelemetry Semantic Conventions v1.28 HTTP/RPC attribute standards to avoid dropping critical failure telemetry during cost-optimization passes. globs: otel-collector-config*.{yaml,yml}, src/telemetry/**/*.{ts,js,py,go}, deployment/k8s/**/*.{yaml,yml}, deploy/docker-compose*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-v1-28-tail-based-adaptive-sampling" --- # OpenTelemetry v1.28 Tail-Based Adaptive Sampling and Semantic Span Retention > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind head-based probabilistic sampling dropping rare production anomalies and violating v1.28 semantic conventions ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Semantic Conventions v1.28 and robust tail-based sampling architectures. You must mandate 100% capture of error spans, HTTP 5xx responses, and tail-latency outliers (p99+), while safely downsampling nominal high-throughput spans (p50/p90) to maintain strict egress bandwidth and storage constraints. Never permit upstream head-based sampling drops on unmeasured traces, and ensure all spans populate v1.28 standardized semantic attributes (e.g., `http.response.status_code`, `error.type`, `server.address`). ## Strict Guidelines - Collectors MUST configure `tail_sampling` processor policies evaluating both `status_code: ERROR` and `numeric_attribute` for `http.response.status_code` >= 500 at 100% retention. - Engineers MUST emit OpenTelemetry Semantic Conventions v1.28 compliant keys (e.g., `http.response.status_code` and `error.type`), avoiding deprecated v1.20 keys (`http.status_code`). - Nominal (2xx/3xx) HTTP transactions MUST be dynamically downsampled using probabilistic or rate-limiting policies (e.g., sample_percentage: 1-5%) only after latency thresholds pass. - Tracing pipelines MUST configure `latency` policies in the tail-sampling processor targeting p99 thresholds (e.g., threshold_ms: 1500) to force retention of slow requests. - Root SDK initializers MUST NOT use static head-based `TraceIdRatioBased` sampling below 1.0 when delegating retention decisions to downstream Collector tail samplers. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Based Sampling Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Simple probabilistic sampling drops 90% of all traces indiscriminately, discarding rare latency spikes and transient error traces before inspection. ```yaml processors: probabilistic_sampler: # ANTI-PATTERN: Blind 10% head-style probabilistic drop loses 90% of rare edge-case bugs sampling_percentage: 10.0 pipelines: traces: receivers: [otlp] processors: [probabilistic_sampler, batch] exporters: [otlp/tempo] ``` #### Production Standard (Correct) > **Standard**: Tail sampling buffers traces for 10 seconds, retaining 100% of errors, 5xx status codes, and latency outliers over 1.5s, while downsampling healthy traffic to 2% to conserve costs. ```yaml processors: tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: - name: drop-healthchecks type: string_attribute string_attribute: key: url.path values: ["/healthz", "/readyz", "/metrics"] enabled_regex_matching: false invert_match: true - name: keep-errors-status type: status_code status_code: { statuses: [ERROR] } - name: keep-5xx-http type: numeric_attribute numeric_attribute: key: http.response.status_code min_value: 500 max_value: 599 - name: keep-p99-latency type: latency latency: { threshold_ms: 1500 } - name: downsample-nominal type: probabilistic probabilistic: { sampling_percentage: 2.0 } pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] exporters: [otlp/tempo] ``` ### SDK Initialization Preserving Complete Traces for Tail Analysis (Node.js/TypeScript) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Client-side ratio sampling permanently discards 95% of traces before errors occur, making tail-based collection at the collector boundary impossible. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-node'; // ANTI-PATTERN: Head-sampling drops traces at the client before tail sampler can evaluate errors const sdk = new NodeSDK({ sampler: new TraceIdRatioBasedSampler(0.05), }); sdk.start(); ``` #### Production Standard (Correct) > **Standard**: Configures an AlwaysOnSampler at the application process layer, offloading selective retention and semantic analysis to the tail-sampling collector tier. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { AlwaysOnSampler } from '@opentelemetry/sdk-trace-node'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; // CORRECT: Always emit downstream to allow Collector tail-sampler to make the retention decision const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4317' }), sampler: new AlwaysOnSampler(), }); sdk.start(); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry v1.28 Tail-Based Adaptive Sampling and Semantic Span Retention

OpenTelemetry v1.28 Tail-Based Adaptive Sampling and Semantic Span Retention defines an industrial-grade behavioral standard for devops engineering. Enforces 100% trace capture for anomalous, high-latency, and error-state spans using OpenTelemetry Collector tail-based sampling while dynamically downsampling nominal transactions. Mandates compliance with OpenTelemetry Semantic Conventions v1.28 HTTP/RPC attribute standards to avoid dropping critical failure telemetry during cost-optimization passes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind head-based probabilistic sampling dropping rare production anomalies and violating v1.28 semantic conventions Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind head-based probabilistic sampling dropping rare production anomalies and violating v1.28 semantic conventions	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Collectors MUST configure `tail_sampling` processor policies evaluating both `status_code

Collectors MUST configure `tail_sampling` processor policies evaluating both `status_code: ERROR` and `numeric_attribute` for `http.response.status_code` >= 500 at 100% retention.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.