OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant

--- description: OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant - Enforces unbroken context propagation of multi-tenant boundaries and security clearance levels across asynchronous message brokers, background workers, and RPC boundaries via W3C Baggage. Mandates explicit allowlisting and sanitization of baggage entries to guarantee traceability while strictly preventing the ingestion of unbounded high-cardinality keys or PII into downstream telemetry systems. globs: src/telemetry/**/*.{ts,js,py,go}, src/middleware/**/*.{ts,js,py,go}, src/messaging/**/*.{ts,js,py,go}, src/queues/**/*.{ts,js,py,go}, src/http/**/*.{ts,js,py,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-w3c-baggage-multi-tenant-lineage-guardrail" --- # OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Baggage Over-Propagation, Tenant Leakage, and Broken Asynchronous Trace Lineage ## System Constraints You are a Principal Software Architect enforcing the OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant. Your responsibility is to guarantee that multi-tenant isolation contexts, security compliance tags, and distributed trace parentage persist unbroken across asynchronous message brokers (e.g., Kafka, RabbitMQ, SQS) and RPC clients. You must prohibit developers from dropping context across async boundaries, blindly dumping raw HTTP headers into W3C Baggage, or attaching unbounded high-cardinality metadata (such as customer emails, payload hashes, or raw tokens) that degrade telemetry storage or violate privacy boundaries. ## Strict Guidelines - Engineers MUST ALWAYS serialize and inject both W3C TraceContext ('traceparent', 'tracestate') and W3C Baggage ('baggage') into message headers or RPC metadata when dispatching asynchronous tasks or queue events. - Engineers MUST ALWAYS extract existing W3C TraceContext and Baggage headers into an active OpenTelemetry context upon message consumption before initializing any consumer execution span. - Engineers MUST NOT copy all incoming W3C Baggage keys into span attributes or metrics tags; span attributes MUST be populated strictly through an approved allowlist (e.g., 'tenant.id', 'security.tier', 'environment'). - Engineers MUST NEVER append unbounded, dynamic, or high-cardinality identifiers (such as session tokens, user UUIDs, request payload hashes, or PII) into W3C Baggage entries. - Services MUST reject or sanitize incoming W3C Baggage headers exceeding 8192 bytes or containing non-RFC compliant delimiters, invalid UTF-8 bytes, or unauthorized tenant context mutations. - Components NEVER drop context across fire-and-forget or background execution threads; the active context MUST be explicitly bound using 'context.with()' or corresponding language runtime primitives. ## Bad vs Good Implementations ### Asynchronous Message Production and Consumption with Safe Baggage Allowlisting #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The publisher fails to inject the active trace and baggage contexts into the Kafka record headers, breaking end-to-end lineage across the broker. The consumer subsequently starts an orphaned root span, destroying distributed causality. Furthermore, the consumer blindly logs the raw baggage string into span attributes, violating tenancy boundaries and risking high-cardinality blowouts and PII leakage. ```typescript import { trace } from '@opentelemetry/api'; import { ProducerRecord } from 'kafkajs'; // Anti-pattern: Producing message without propagating context or baggage, // and blindly reflecting unvalidated headers into span tags. export async function publishTenantEvent(producer: any, tenantId: string, payload: any) { const tracer = trace.getTracer('billing-service'); return tracer.startActiveSpan('publishTenantEvent', async (span) => { // Dropping W3C baggage and trace context: headers remain empty const record: ProducerRecord = { topic: 'billing-events', messages: [{ value: JSON.stringify(payload) }], }; await producer.send(record); span.end(); }); } export async function consumeTenantEvent(message: any) { const tracer = trace.getTracer('billing-service'); // Root span created without parentage from headers return tracer.startActiveSpan('consumeTenantEvent', async (span) => { // High-cardinality anti-pattern: blindly copying raw unvalidated baggage if (message.headers?.baggage) { span.setAttribute('raw.baggage', message.headers.baggage.toString()); } span.end(); }); } ``` #### Production Standard (Correct) > **Standard**: The publisher attaches validated low-cardinality tenancy and security metadata via OpenTelemetry Baggage and correctly injects standard W3C headers into the message carrier. The consumer extracts the parent trace context and baggage, ensures context continuity across the asynchronous boundary via 'context.with', and restricts span attribute projection strictly to an approved allowlist, preventing cardinality inflation. ```typescript import { context, propagation, trace, Baggage, BaggageEntry } from '@opentelemetry/api'; import { ProducerRecord } from 'kafkajs'; const ALLOWED_BAGGAGE_KEYS = new Set(['tenant.id', 'security.classification', 'region.id']); export async function publishTenantEvent( producer: any, tenantId: string, securityClassification: string, payload: any ): Promise<void> { const tracer = trace.getTracer('billing-service'); await tracer.startActiveSpan('publishTenantEvent', async (span) => { try { // Build sanitized, low-cardinality baggage let currentBaggage = propagation.getBaggage(context.active()) || propagation.createBaggage(); currentBaggage = currentBaggage.setEntry('tenant.id', { value: tenantId }); currentBaggage = currentBaggage.setEntry('security.classification', { value: securityClassification }); const contextWithBaggage = propagation.setBaggage(context.active(), currentBaggage); const carrier: Record<string, string> = {}; // Inject W3C TraceContext and Baggage into carrier propagation.inject(contextWithBaggage, carrier); const record: ProducerRecord = { topic: 'billing-events', messages: [{ value: JSON.stringify(payload), headers: Object.entries(carrier).reduce((acc, [k, v]) => ({ ...acc, [k]: Buffer.from(v) }), {}) }], }; await producer.send(record); } finally { span.end(); } }); } export async function consumeTenantEvent(rawHeaders: Record<string, any>, handler: () => Promise<void>): Promise<void> { const tracer = trace.getTracer('billing-service'); const carrier: Record<string, string> = {}; for (const [key, value] of Object.entries(rawHeaders)) { if (value) carrier[key.toLowerCase()] = Buffer.isBuffer(value) ? value.toString('utf-8') : String(value); } // Extract incoming W3C Context and Baggage const extractedContext = propagation.extract(context.active(), carrier); await context.with(extractedContext, async () => { await tracer.startActiveSpan('consumeTenantEvent', async (span) => { try { const baggage = propagation.getBaggage(context.active()); if (baggage) { // Invariant: Allowlist-only projection to span attributes ALLOWED_BAGGAGE_KEYS.forEach((key) => { const entry = baggage.getEntry(key); if (entry) { span.setAttribute(key, entry.value); } }); } await handler(); } finally { span.end(); } }); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant

OpenTelemetry W3C Baggage Distributed Multi-Tenant Lineage Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces unbroken context propagation of multi-tenant boundaries and security clearance levels across asynchronous message brokers, background workers, and RPC boundaries via W3C Baggage. Mandates explicit allowlisting and sanitization of baggage entries to guarantee traceability while strictly preventing the ingestion of unbounded high-cardinality keys or PII into downstream telemetry systems. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Baggage Over-Propagation, Tenant Leakage, and Broken Asynchronous Trace Lineage Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Baggage Over-Propagation, Tenant Leakage, and Broken Asynchronous Trace Lineage	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS serialize and inject both W3C TraceContext ('traceparent', 'tracestate') and W3C Baggage ('baggage') into message headers or RPC metadata when dispatching asynchronous tasks or queue events.

Engineers MUST ALWAYS serialize and inject both W3C TraceContext ('traceparent', 'tracestate') and W3C Baggage ('baggage') into message headers or RPC metadata when dispatching asynchronous tasks or queue events.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.