Deterministic OTel Tail-Based Sampling & Profiling Retention

--- description: Deterministic OTel Tail-Based Sampling & Profiling Retention - Enforces a two-tier OpenTelemetry Collector topology using consistent traceID hashing (loadbalancing exporter) and tail_sampling processor rules to guarantee 100% capture of error traces and p99.9 latency anomalies. Prevents runaway network egress costs and vendor bill shock while maintaining deterministic correlation between distributed traces and continuous CPU/memory profiles via trace-context baggage injection. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/k8s/**/*.yaml, **/observability/**/*.ts, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-tail-sampling-continuous-profiling-guardrail" --- # Deterministic OTel Tail-Based Sampling & Profiling Retention > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Naive head-based sampling dropping fatal anomalies or unbuffered collector-egress exploding telemetry bills ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing deterministic tail-based trace sampling and continuous profiling correlation across OpenTelemetry collectors. You mandate strict routing via trace-id consistent hashing, memory-delimited tail-sampling queues, zero data-loss on HTTP 5xx or unhandled exceptions, dynamic latency thresholding, and continuous profile linking via span context. ## Strict Guidelines - Collectors MUST deploy in a two-tier topology: Tier 1 (DaemonSet/Sidecar) with `loadbalancingexporter` keyed by `trace_id`, and Tier 2 (StatefulSet/Deployment) running `tail_sampling`. - Engineers MUST NOT run `tail_sampling` on Tier 1 collectors without trace-id affinity, as fragmented spans will break trace-level sampling decisions. - The `tail_sampling` processor MUST configure explicit rules: 100% retention on `status_code: ERROR`, 100% on latency breaches (> p99 threshold), and inverted probabilistic drop for health checks and high-frequency 2xx spans. - The `tail_sampling` configuration MUST specify an explicit `decision_wait` (e.g., 10s-30s) and bounded `num_traces` to prevent collector out-of-memory (OOM) crashes under high throughput. - Application runtimes MUST link continuous profiling (e.g., Pyroscope, Parca) with OpenTelemetry spans by injecting `trace_id` and `span_id` labels into active thread contexts during CPU/heap profiling samples. - Batch processors MUST ALWAYS precede exporters and follow tail-sampling processors to amortize HTTP/gRPC overhead on preserved traces. ## Bad vs Good Implementations ### Collector Tail Sampling Pipeline vs Naive Pass-Through #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly batching without tail-sampling either causes catastrophic vendor egress costs (retaining 100% of high-volume healthy 200 OK spans) or risks dropping critical error and high-latency traces if head-sampling is applied at the application runtime. ```yaml # Tier 2 Collector: Naive Head-Sampling or Unfiltered Egress Pipeline receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } processors: batch: timeout: 1s send_batch_size: 8192 exporters: otlp/vendor: endpoint: ingress.observability.vendor.com:4317 service: pipelines: traces: receivers: [otlp] # Lacks tail-sampling: 100% egress sent to vendor or 1% head-sampled # Dropping 99% of rare fatal errors or causing a 5-figure cloud bill processors: [batch] exporters: [otlp/vendor] ``` #### Production Standard (Correct) > **Standard**: Deterministic tail sampling evaluates traces in-memory for 10 seconds. It captures 100% of errors and latencies exceeding 750ms, discards noisy synthetic endpoints, and downsamples regular 200 OK traffic to 1%, eliminating collector egress inflation without losing observability. ```yaml # Tier 2 Collector: Deterministic Tail-Based Sampling Pipeline receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } processors: tail_sampling: decision_wait: 10s num_traces: 50000 expected_new_traces_per_sec: 2000 policies: - name: drop-noisy-healthchecks type: string_attribute string_attribute: { key: http.target, values: ["/healthz", "/readyz", "/metrics"], enabled_regex_matching: false, invert_match: true } - name: sample-errors-100-percent type: status_code status_code: { status_codes: [ERROR] } - name: sample-latency-p99 type: latency latency: { threshold_ms: 750 } - name: probabilistic-sample-healthy type: probabilistic probabilistic: { sampling_percentage: 1.0 } batch: timeout: 5s send_batch_size: 4096 exporters: otlp/vendor: endpoint: ingress.observability.vendor.com:4317 service: pipelines: traces: receivers: [otlp] processors: [tail_sampling, batch] exporters: [otlp/vendor] ``` ### Application Profiling-Trace Correlation Injection (Node.js/TypeScript) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The continuous profiler operates independently from the active OpenTelemetry span context. When the collector captures a latency anomaly trace, engineers cannot pivot to thread execution profiles for the culprit span. ```typescript import { trace } from '@opentelemetry/api'; import Pyroscope from '@pyroscope/nodejs'; Pyroscope.init({ serverAddress: 'http://pyroscope:4040', appName: 'payment-svc' }); export async function processPayment(payload: any) { const tracer = trace.getTracer('payment-svc'); return tracer.startActiveSpan('processPayment', async (span) => { // Anti-pattern: Continuous profiler runs blind to active trace IDs // CPU spikes cannot be correlated to the specific trace or customer transaction const res = await executeTransaction(payload); span.end(); return res; }); } ``` #### Production Standard (Correct) > **Standard**: Wraps the unit of execution with active trace_id and span_id labels inside the profiler and flags span status to ERROR on failure. When tail-sampling captures the trace, the profiler matches the span timeline directly. ```typescript import { trace, context } from '@opentelemetry/api'; import Pyroscope from '@pyroscope/nodejs'; Pyroscope.init({ serverAddress: 'http://pyroscope:4040', appName: 'payment-svc' }); export async function processPayment(payload: any) { const tracer = trace.getTracer('payment-svc'); return tracer.startActiveSpan('processPayment', async (span) => { const spanContext = span.spanContext(); // Invariant: Bind continuous profile labels to span context for bidirectional drill-down return Pyroscope.wrapExecutionWithLabels( { trace_id: spanContext.traceId, span_id: spanContext.spanId, }, async () => { try { const res = await executeTransaction(payload); return res; } catch (err: any) { span.recordException(err); span.setStatus({ code: 2, message: err.message }); // 2 = ERROR, guaranteed capture by tail_sampling throw err; } finally { span.end(); } } ); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic OTel Tail-Based Sampling & Profiling Retention

Deterministic OTel Tail-Based Sampling & Profiling Retention defines an industrial-grade behavioral standard for devops engineering. Enforces a two-tier OpenTelemetry Collector topology using consistent traceID hashing (loadbalancing exporter) and tail_sampling processor rules to guarantee 100% capture of error traces and p99.9 latency anomalies. Prevents runaway network egress costs and vendor bill shock while maintaining deterministic correlation between distributed traces and continuous CPU/memory profiles via trace-context baggage injection. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Naive head-based sampling dropping fatal anomalies or unbuffered collector-egress exploding telemetry bills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Naive head-based sampling dropping fatal anomalies or unbuffered collector-egress exploding telemetry bills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Collectors MUST deploy in a two-tier topology

Collectors MUST deploy in a two-tier topology: Tier 1 (DaemonSet/Sidecar) with `loadbalancingexporter` keyed by `trace_id`, and Tier 2 (StatefulSet/Deployment) running `tail_sampling`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.