OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant

--- description: OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant - Enforces strict distributed context propagation using W3C Baggage to dynamically adjust head-based trace sampling weights at ingress based on tenant tier SLA and downstream circuit breaker status. Prevents telemetry debt and silent data loss by synchronizing baggage-driven sampling probabilities across downstream tail-sampling collectors. globs: src/telemetry/**/*.ts, src/middleware/tracing/**/*.ts, src/ingress/**/*.ts, packages/observability/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-baggage-dynamic-sampling-invariant" --- # OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static Head-Sampling or Dropping W3C Baggage Resulting in Blind Spots and Ingestion Cost Explosions ## System Constraints You are a Principal Observability and Systems Architect enforcing the OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant. You must reject any ingress handler, middleware, or tracer configuration that strips baggage headers, computes sampling decisions statically without tenant awareness, or mutates OpenTelemetry Baggage outside immutable context APIs. Mandate precise baggage propagation and adaptive sampling weights to reconcile edge-ingress telemetry with collector tail-sampling pipelines. ## Strict Guidelines - Services MUST ALWAYS extract incoming W3C TraceContext and Baggage headers at ingress boundaries before creating root spans. - Engineers MUST inject tenant tier ('enterprise', 'pro', 'free') and dynamic sampling weight directives into W3C Baggage so downstream microservices and OTel Collectors retain sampling context. - Ingress middleware MUST dynamically compute head-sampling probabilities by intersecting tenant tier SLO guarantees with real-time downstream circuit breaker degradation metrics. - Engineers MUST NOT mutate active Baggage directly in place; context mutations MUST use immutable propagation chains (`propagation.setBaggage(ctx, updatedBaggage)`). - Services MUST NOT drop baggage headers during outgoing client requests (HTTP, gRPC, or message bus publishers). - Collectors and egress filters MUST read the baggage-defined sampling weights when executing tail-sampling decisions (e.g., preserving 100% of enterprise-tier errors and down-sampling health checks). ## Bad vs Good Implementations ### Ingress Sampling Middleware with Dynamic Baggage Propagation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Fails to extract or propagate incoming W3C Baggage, drops tenant tier metadata, makes an uncoordinated static head-sampling decision, and isolates downstream collectors from knowing the intended sampling fidelity. ```typescript import { trace, context } from '@opentelemetry/api'; import { Request, Response, NextFunction } from 'express'; export function tracingMiddleware(req: Request, res: Response, next: NextFunction) { // Anti-Pattern: Static root span creation, ignoring incoming baggage // and stripping context from downstream collectors. const tracer = trace.getTracer('ingress-edge'); const span = tracer.startSpan(`HTTP ${req.method} ${req.path}`); // Blind static sampling decision bypassing tenant tiers and downstream pressure if (Math.random() > 0.05) { span.end(); return next(); } context.with(trace.setSpan(context.active(), span), () => { res.on('finish', () => span.end()); next(); }); } ``` #### Production Standard (Correct) > **Standard**: Correctly extracts incoming propagation headers, computes dynamic sampling weights from tenant SLAs and circuit-breaker telemetry, sets deterministic W3C Baggage attributes, and retains complete distributed context for downstream tail-sampling collectors. ```typescript import { trace, context, propagation, SpanStatusCode } from '@opentelemetry/api'; import { Request, Response, NextFunction } from 'express'; import { getDownstreamHealthMetrics } from '../metrics/downstreamHealth'; export function dynamicSamplingMiddleware(req: Request, res: Response, next: NextFunction) { // Extract existing TraceContext and Baggage from wire const incomingCtx = propagation.extract(context.active(), req.headers); const tenantTier = (req.headers['x-tenant-tier'] as string) || 'standard'; // Dynamic weight calculation based on downstream health state const downstreamDegraded = getDownstreamHealthMetrics().hasDegradedDependencies(); const sampleWeight = tenantTier === 'enterprise' ? 1.0 : downstreamDegraded ? 0.01 : 0.1; const isSampled = Math.random() <= sampleWeight; // Immutable baggage population for tail collector and downstream microservices const updatedBaggage = propagation.createBaggage({ 'sampling.weight': { value: sampleWeight.toString() }, 'tenant.tier': { value: tenantTier }, 'system.degraded': { value: downstreamDegraded.toString() } }); const enrichedCtx = propagation.setBaggage(incomingCtx, updatedBaggage); const tracer = trace.getTracer('ingress-edge'); const span = tracer.startSpan(`HTTP ${req.method} ${req.path}`, { attributes: { 'tenant.tier': tenantTier, 'sampling.weight': sampleWeight, 'sampling.edge_decision': isSampled ? 'sampled' : 'discarded' } }, enrichedCtx); context.with(trace.setSpan(enrichedCtx, span), () => { res.on('finish', () => { if (res.statusCode >= 500) { span.setStatus({ code: SpanStatusCode.ERROR, message: `HTTP ${res.statusCode}` }); } span.end(); }); next(); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant

OpenTelemetry Baggage Dynamic Head-and-Tail Sampling Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces strict distributed context propagation using W3C Baggage to dynamically adjust head-based trace sampling weights at ingress based on tenant tier SLA and downstream circuit breaker status. Prevents telemetry debt and silent data loss by synchronizing baggage-driven sampling probabilities across downstream tail-sampling collectors. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static Head-Sampling or Dropping W3C Baggage Resulting in Blind Spots and Ingestion Cost Explosions Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static Head-Sampling or Dropping W3C Baggage Resulting in Blind Spots and Ingestion Cost Explosions	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST ALWAYS extract incoming W3C TraceContext and Baggage headers at ingress boundaries before creating root spans.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.