W3C TraceContext and Baggage Propagation Continuity for Kafka and Outbox Pipelines

--- description: W3C TraceContext and Baggage Propagation Continuity for Kafka and Outbox Pipelines - Enforces strict W3C TraceContext (`traceparent`, `tracestate`) and Baggage wire-level propagation across transactional outbox storage and Kafka producer/consumer records. Guarantees immutable propagation of multi-tenant baggage and causal trace links across asynchronous message boundaries without header dropping, buffer corruption, or thread-context leakage. globs: src/infrastructure/kafka/**/*.ts, src/infrastructure/outbox/**/*.ts, src/messaging/**/*.ts, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-w3c-kafka-tracecontext-baggage-propagation" --- # W3C TraceContext and Baggage Propagation Continuity for Kafka and Outbox Pipelines > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Dropping Trace Context Across Outbox Boundaries or Mutating Active Baggage via Unpropagated Kafka Headers ## System Constraints You are a Principal Distributed Systems and Telemetry Architect. Your role is to mandate uncompromising OpenTelemetry W3C TraceContext and Baggage propagation across asynchronous Kafka pipelines and Transactional Outbox workers. You strictly prohibit inline span disconnection, raw string header omission, unbuffered byte transformations that strip traceparent/baggage, and context pollution in concurrent batch consumers. ## Strict Guidelines - Engineers MUST ALWAYS serialize OpenTelemetry Context (`traceparent`, `tracestate`, `baggage`) into the transactional outbox payload using standard W3C text map injectors (`W3CTraceContextPropagator`, `W3CBaggagePropagator`). - Engineers MUST NOT instantiate Kafka producers or consumers without configuring an explicit `TextMapPropagator` getter/setter adapter capable of encoding and decoding Kafka binary/string header pairs. - Services MUST extract parent context from incoming Kafka record headers, establish a CONSUMER span with appropriate `Link` or parent hierarchy, and execute downstream tenant logic strictly within the bound active context. - Components NEVER drop or selectively filter the `baggage` header when relaying events through intermediary topics, CDC engines, or dead-letter queues (DLQ). - Batch consumers MUST isolate trace contexts per individual record; they MUST NOT allow context leaked from record N to bleed into record N+1 during parallel or sequential batch processing. - Outbox relay publishers MUST read trace context from outbox storage and inject it as native Kafka record headers (`Buffer.from(val, 'utf-8')`) rather than solely burying telemetry in the JSON event payload. ## Bad vs Good Implementations ### Kafka Producer & Outbox Relay: Complete Context & Baggage Injection #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Fails to propagate traceparent or baggage into Kafka headers. Downstream consumers will generate an orphaned root span and lose all tenant metadata defined in upstream W3C baggage. ```typescript import { ProducerRecord } from 'kafkajs'; import { trace } from '@opentelemetry/api'; export async function publishOutboxEvent(outboxRow: { id: string; payload: string; topic: string }) { // Anti-pattern: Loses parent baggage and creates a disconnected span with no header propagation const tracer = trace.getTracer('outbox-relay'); return tracer.startActiveSpan('publish-event', async (span) => { const record: ProducerRecord = { topic: outboxRow.topic, messages: [{ key: outboxRow.id, value: outboxRow.payload }] }; span.end(); return record; }); } ``` #### Production Standard (Correct) > **Standard**: Extracts stored context from the outbox record, injects both TraceContext and multi-tenant Baggage into a standards-compliant carrier, and binds them as binary-safe UTF-8 Kafka headers. ```typescript import { ProducerRecord, IHeaders } from 'kafkajs'; import { context, propagation, defaultTextMapSetter } from '@opentelemetry/api'; interface OutboxEntity { id: string; topic: string; payload: string; traceCarrierJson?: string; } export function buildKafkaRecordWithContext(outboxRow: OutboxEntity): ProducerRecord { // Restore extracted context from outbox carrier storage or fallback to current active context const savedCarrier: Record<string, string> = outboxRow.traceCarrierJson ? JSON.parse(outboxRow.traceCarrierJson) : {}; const parentContext = propagation.extract(context.active(), savedCarrier); const headersCarrier: Record<string, string> = {}; // Inject traceparent, tracestate, and baggage into temporary string carrier propagation.inject(parentContext, headersCarrier, defaultTextMapSetter); // Transform to Kafka byte headers while preserving full W3C specs const kafkaHeaders: IHeaders = Object.entries(headersCarrier).reduce((acc, [key, val]) => { acc[key] = Buffer.from(val, 'utf-8'); return acc; }, {} as IHeaders); return { topic: outboxRow.topic, messages: [{ key: outboxRow.id, value: outboxRow.payload, headers: kafkaHeaders }] }; } ``` ### Kafka Consumer: Proper Context Extraction and Tenant Baggage Scope Isolation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Disregards message headers entirely, causing complete trace continuity failure and unpropagated tenant isolation contexts. ```typescript import { EachMessagePayload } from 'kafkajs'; import { trace } from '@opentelemetry/api'; export async function handleKafkaMessage({ message }: EachMessagePayload) { // Anti-pattern: Completely ignores record headers and trace context const tracer = trace.getTracer('kafka-consumer'); await tracer.startActiveSpan('process-message', async (span) => { const payload = JSON.parse(message.value!.toString()); // Tenant baggage is missing; cross-tenant execution occurs without auditability await processOrder(payload); span.end(); }); } ``` #### Production Standard (Correct) > **Standard**: Decodes headers from Kafka bytes, extracts active W3C trace and baggage contexts, establishes a CONSUMER span linking to upstream causality, and accesses baggage within isolated asynchronous scope. ```typescript import { EachMessagePayload } from 'kafkajs'; import { context, propagation, trace, SpanKind, defaultTextMapGetter, getBaggage } from '@opentelemetry/api'; export async function handleKafkaMessage({ topic, partition, message }: EachMessagePayload) { // Flatten Kafka record headers from Buffer to string map const carrier: Record<string, string> = {}; if (message.headers) { for (const [key, value] of Object.entries(message.headers)) { if (value) { carrier[key] = Buffer.isBuffer(value) ? value.toString('utf-8') : value.toString(); } } } const extractedContext = propagation.extract(context.active(), carrier, defaultTextMapGetter); const tracer = trace.getTracer('kafka-consumer'); await context.with(extractedContext, async () => { await tracer.startActiveSpan( `${topic} receive`, { kind: SpanKind.CONSUMER, attributes: { 'messaging.system': 'kafka', 'messaging.destination': topic, 'messaging.kafka.partition': partition, 'messaging.kafka.offset': message.offset } }, async (span) => { try { const tenantId = getBaggage(context.active())?.getEntry('tenant.id')?.value; const payload = JSON.parse(message.value!.toString()); await processOrder(payload, tenantId); } catch (error) { span.recordException(error as Error); throw error; } finally { span.end(); } } ); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: W3C TraceContext and Baggage Propagation Continuity for Kafka and Outbox Pipelines

W3C TraceContext and Baggage Propagation Continuity for Kafka and Outbox Pipelines defines an industrial-grade behavioral standard for devops engineering. Enforces strict W3C TraceContext (`traceparent`, `tracestate`) and Baggage wire-level propagation across transactional outbox storage and Kafka producer/consumer records. Guarantees immutable propagation of multi-tenant baggage and causal trace links across asynchronous message boundaries without header dropping, buffer corruption, or thread-context leakage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Dropping Trace Context Across Outbox Boundaries or Mutating Active Baggage via Unpropagated Kafka Headers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Dropping Trace Context Across Outbox Boundaries or Mutating Active Baggage via Unpropagated Kafka Headers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS serialize OpenTelemetry Context (`traceparent`, `tracestate`, `baggage`) into the transactional outbox payload using standard W3C text map injectors (`W3CTraceContextPropagator`, `W3CBaggagePropagator`).

Engineers MUST ALWAYS serialize OpenTelemetry Context (`traceparent`, `tracestate`, `baggage`) into the transactional outbox payload using standard W3C text map injectors (`W3CTraceContextPropagator`, `W3CBaggagePropagator`).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.