OpenTelemetry Context Baggage & Collector Tail-Sampling Topology Guardrail

--- description: OpenTelemetry Context Baggage & Collector Tail-Sampling Topology Guardrail - Enforces strict delineation between W3C Baggage and Span Attributes alongside two-tier OpenTelemetry Collector tail-based sampling. Baggage propagation must be minimal to avoid HTTP header bloat, while high-cardinality debugging metadata must be recorded as non-propagated span attributes evaluated for retention only when error or SLA thresholds are violated. globs: src/**/*.{ts,js,py,go,java}, k8s/**/*.yaml, deploy/**/*.yaml, collector/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-baggage-tail-sampling-guardrail" --- # OpenTelemetry Context Baggage & Collector Tail-Sampling Topology Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Baggage Injection and Head-Based Eager Sampling Leading to Network Header Bloat and Backend Ingestion Saturation ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry context propagation and tail-based sampling standards. You prevent network overhead and storage exhaustion by ensuring engineers never use W3C Baggage for high-cardinality payloads, verify that downstream sampling decisions are deferred to collector pools, and guarantee trace consistency using traceID-hashed load balancing. ## Strict Guidelines - Engineers MUST NOT inject high-cardinality metadata, JSON blobs, or operational payloads into W3C Baggage; baggage is strictly reserved for low-cardinality routing keys (e.g., tenant_id, region). - Services MUST record ephemeral debug contexts, payload sizes, and SLA indicators as local Span Attributes, never propagating them across process boundaries via baggage. - Collector topologies MUST implement a two-layer deployment: an agent tier routing traces via loadbalancingexporter (keyed by traceID) to an upstream collector tier running the tail_sampling processor. - The tail_sampling processor MUST evaluate error statuses, latency thresholds (e.g., > 1500ms), and percentage-based baseline traffic, dropping uninteresting nominal traces to preserve storage. - Services MUST NOT implement eager head-based drops in application code if downstream SLA evaluation is required; traces must emit with a ParentBased(AlwaysOn) or proportional root decision into the collector mesh. ## Bad vs Good Implementations ### Context Baggage Injection vs. Span Attributes with Tail-Sampling Attributes #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Injecting serialized request payloads and high-cardinality debug switches into Baggage inflates HTTP headers on every downstream network call, risks RFC header size limits, and bypasses centralized collector-level sampling architectures. ```typescript import { propagation, context, trace } from '@opentelemetry/api'; export async function handleIncomingOrder(orderPayload: Record<string, unknown>, userId: string) { const tracer = trace.getTracer('checkout-service'); return tracer.startActiveSpan('process_order', async (span) => { // ANTI-PATTERN: Serializing large payloads or debug telemetry into Baggage propagates // bloated HTTP headers ('baggage: ...') downstream to all downstream microservices. let currentContext = context.active(); currentContext = propagation.setBaggage( currentContext, propagation.createBaggage({ userId: { value: userId }, rawPayload: { value: JSON.stringify(orderPayload) }, debugTraceAll: { value: 'true' } }) ); // Context leakage via network headers await fetch('https://payment.internal/charge', { headers: { ...propagation.getBaggage(currentContext) } as Record<string, string> }); span.end(); }); } ``` #### Production Standard (Correct) > **Standard**: Reduces baggage to minimal routing context (tenant_id), attaches operational SLA attributes directly to the span, and sets an explicit error state when latency exceeds the contract threshold, allowing the tail-sampling collector to selectively retain the entire trace trace. ```typescript import { propagation, context, trace, SpanStatusCode } from '@opentelemetry/api'; export async function handleIncomingOrder(orderPayload: Record<string, unknown>, tenantId: string) { const tracer = trace.getTracer('checkout-service'); return tracer.startActiveSpan('process_order', async (span) => { const startTime = Date.now(); try { // INVARIANT: Only propagate immutable, low-cardinality routing/context keys in Baggage const minimalBaggage = propagation.createBaggage({ tenant_id: { value: tenantId } }); const enrichedContext = propagation.setBaggage(context.active(), minimalBaggage); // High-cardinality attributes belong ONLY on the local span for Collector evaluation span.setAttribute('order.items_count', (orderPayload.items as unknown[])?.length ?? 0); span.setAttribute('app.slo.target_ms', 1200); const response = await context.with(enrichedContext, () => fetch('https://payment.internal/charge') ); const durationMs = Date.now() - startTime; span.setAttribute('duration_ms', durationMs); if (!response.ok || durationMs > 1200) { // Signal failure to tail-sampler; span is captured by collector policy span.setStatus({ code: SpanStatusCode.ERROR, message: 'SLA violation or payment failure' }); } } catch (err: unknown) { span.recordException(err instanceof Error ? err : new Error(String(err))); span.setStatus({ code: SpanStatusCode.ERROR }); throw err; } finally { span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Context Baggage & Collector Tail-Sampling Topology Guardrail

OpenTelemetry Context Baggage & Collector Tail-Sampling Topology Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict delineation between W3C Baggage and Span Attributes alongside two-tier OpenTelemetry Collector tail-based sampling. Baggage propagation must be minimal to avoid HTTP header bloat, while high-cardinality debugging metadata must be recorded as non-propagated span attributes evaluated for retention only when error or SLA thresholds are violated. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Baggage Injection and Head-Based Eager Sampling Leading to Network Header Bloat and Backend Ingestion Saturation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Baggage Injection and Head-Based Eager Sampling Leading to Network Header Bloat and Backend Ingestion Saturation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NOT inject high-cardinality metadata, JSON blobs, or operational payloads into W3C Baggage

Engineers MUST NOT inject high-cardinality metadata, JSON blobs, or operational payloads into W3C Baggage; baggage is strictly reserved for low-cardinality routing keys (e.g., tenant_id, region).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.