OpenTelemetry W3C Baggage Ingress Quarantine & High-Cardinality Guardrail

--- description: OpenTelemetry W3C Baggage Ingress Quarantine & High-Cardinality Guardrail - Enforces strict extraction, validation, and sanitization of W3C Baggage headers at edge API ingress points before promoting attributes to telemetry spans or metrics. Unchecked propagation of arbitrary baggage values directly into span attributes triggers exponential metric cardinality explosions, degrading OpenTelemetry Collector nodes and causing downstream TSDB/Cortex OOM failures. globs: src/middleware/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/telemetry/**/*.{ts,js}, src/tracing/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-w3c-baggage-cardinality-quarantine-standard" --- # OpenTelemetry W3C Baggage Ingress Quarantine & High-Cardinality Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind extraction and promotion of raw W3C baggage keys to span attributes and metric dimensions without allowlisting or cardinality caps ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing strict W3C Baggage isolation and validation at all edge boundaries. OpenTelemetry Baggage carries application metadata across process boundaries, but untrusted external clients can inject unbounded unique dimensions (user IDs, JWT tokens, timestamps) into the 'baggage' header. If promoted unconditionally to span attributes or Prometheus metric dimensions, this induces catastrophic high-cardinality OOM in downstream collectors (e.g., OpenTelemetry Collector, Prometheus, Datadog, Grafana Mimir). You must enforce static dimension allowlists, maximum payload byte caps, sanitization, and explicit quarantine patterns at all ingress interfaces. ## Strict Guidelines - Edge middleware MUST NEVER unconditionally promote raw W3C Baggage entries to OpenTelemetry Span attributes or downstream Metric labels. - Services MUST validate all incoming W3C Baggage headers against an explicit, immutable allowlist schema specifying permissible keys and regex-validated value formats. - Ingress controllers MUST drop or quarantine any baggage key exceeding 32 characters or value exceeding 128 characters to prevent buffer bloat and memory inflation attacks. - Telemetry collectors MUST drop unvalidated baggage entries before forwarding traces to downstream telemetry pipelines, ensuring dropped counts are recorded on a bounded 'otel_baggage_quarantined_total' counter. - Engineers MUST configure BaggagePropagator with custom composite extractors that strip untrusted external baggage while preserving verified internal trace state headers. ## Bad vs Good Implementations ### Unfiltered Baggage Attribute Promotion vs. Quarantined Ingress Validation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Iterating over all raw baggage entries and injecting them directly into span attributes creates a massive security and cardinality vulnerability. Malicious actors or misconfigured clients can send thousands of unique keys or multi-kilobyte values, exhausting Collector memory buffers and crashing indexed metric stores. ```typescript import { Request, Response, NextFunction } from 'express'; import { trace, propagation, context } from '@opentelemetry/api'; // ANTI-PATTERN: Blindly extracts untrusted baggage and maps every key-value // pair directly to span attributes and context without size or allowlist checks. export function telemetryMiddleware(req: Request, res: Response, next: NextFunction): void { const activeContext = propagation.extract(context.active(), req.headers); const currentBaggage = propagation.getBaggage(activeContext); const currentSpan = trace.getSpan(activeContext); if (currentBaggage && currentSpan) { for (const [key, entry] of currentBaggage.getAllEntries()) { // CRITICAL BUG: Injecting external high-cardinality baggage directly into span attributes // Allows external clients to send random keys/values, crashing TSDBs and Collectors via OOM. currentSpan.setAttribute(`baggage.${key}`, entry.value); } } context.with(activeContext, () => next()); } ``` #### Production Standard (Correct) > **Standard**: Strictly filters incoming baggage against an allowlist with strict regex validations, caps the maximum permissible baggage entries, isolates sanitized values under a controlled namespace ('app.baggage.*'), and increments an observable rejection counter for untrusted keys. ```typescript import { Request, Response, NextFunction } from 'express'; import { trace, propagation, context, createBaggage } from '@opentelemetry/api'; import { metrics } from '@opentelemetry/api'; const ALLOWED_BAGGAGE_KEYS = new Map<string, RegExp>([ ['tenant_id', /^[a-zA-Z0-9_-]{1,36}$/], ['env', /^(production|staging|development)$/], ['region', /^[a-z]{2}-[a-z]+-\d{1}$/] ]); const MAX_BAGGAGE_ENTRIES = 5; const quarantinedCounter = metrics .getMeter('ingress-telemetry') .createCounter('otel_baggage_quarantined_total', { description: 'Count of quarantined or dropped W3C baggage attributes' }); export function quarantinedTelemetryMiddleware(req: Request, res: Response, next: NextFunction): void { const extractedContext = propagation.extract(context.active(), req.headers); const incomingBaggage = propagation.getBaggage(extractedContext); const span = trace.getSpan(extractedContext); const sanitizedEntries: Record<string, { value: string }> = {}; let validCount = 0; if (incomingBaggage) { for (const [key, entry] of incomingBaggage.getAllEntries()) { const pattern = ALLOWED_BAGGAGE_KEYS.get(key); if (pattern && pattern.test(entry.value) && validCount < MAX_BAGGAGE_ENTRIES) { sanitizedEntries[key] = { value: entry.value }; if (span) { span.setAttribute(`app.baggage.${key}`, entry.value); } validCount++; } else { quarantinedCounter.add(1, { rejected_key: key }); } } } const sanitizedBaggage = createBaggage(sanitizedEntries); const sanitizedContext = propagation.setBaggage(extractedContext, sanitizedBaggage); context.with(sanitizedContext, () => next()); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry W3C Baggage Ingress Quarantine & High-Cardinality Guardrail

OpenTelemetry W3C Baggage Ingress Quarantine & High-Cardinality Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict extraction, validation, and sanitization of W3C Baggage headers at edge API ingress points before promoting attributes to telemetry spans or metrics. Unchecked propagation of arbitrary baggage values directly into span attributes triggers exponential metric cardinality explosions, degrading OpenTelemetry Collector nodes and causing downstream TSDB/Cortex OOM failures. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind extraction and promotion of raw W3C baggage keys to span attributes and metric dimensions without allowlisting or cardinality caps Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind extraction and promotion of raw W3C baggage keys to span attributes and metric dimensions without allowlisting or cardinality caps	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Edge middleware MUST NEVER unconditionally promote raw W3C Baggage entries to OpenTelemetry Span attributes or downstream Metric labels.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.