OpenTelemetry Trace Context and Baggage Propagation Across Asynchronous Task Queues

--- description: OpenTelemetry Trace Context and Baggage Propagation Across Asynchronous Task Queues - Enforces deterministic causal distributed tracing and transactional baggage preservation across asynchronous worker pools, task queues, and decentralized event boundaries. Mandates explicit W3C TraceContext and Baggage injection at enqueue time and root context extraction with span linking at execution time. Eliminates broken distributed causal graphs and context leakage across shared concurrency runtimes. globs: src/queues/**/*.{ts,js}, src/workers/**/*.{ts,js}, src/jobs/**/*.{ts,js}, src/messaging/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-async-queue-context-propagation" --- # OpenTelemetry Trace Context and Baggage Propagation Across Asynchronous Task Queues > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Silent Context Severing and Blind Asynchronous Worker Execution ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry standards across asynchronous execution boundaries. You must strictly mandate bidirectional W3C traceparent, tracestate, and baggage injection into message metadata on task publication, and explicit context extraction with context-scoped execution on worker consumption. Never allow implicit or headless queue operations. ## Strict Guidelines - Engineers MUST ALWAYS inject active OpenTelemetry trace context and baggage into message metadata using `propagation.inject(context.active(), carrier)` immediately prior to enqueuing a task. - Engineers MUST NOT rely on runtime ambient context surviving across process boundaries, asynchronous thread pools, or event loop queues without explicit serializable metadata carriers. - Services MUST extract the trace context on the consumer side via `propagation.extract(ROOT_CONTEXT, message.headers)` before instantiating child spans or invoking business logic. - Workers MUST execute task payloads inside `context.with(extractedContext, ...)` to maintain execution-scoped storage and prevent context leakage across reused worker processes. - Code SHOULD designate producer operations as `SpanKind.PRODUCER` and consumer execution entry points as `SpanKind.CONSUMER` to conform with OpenTelemetry Semantic Conventions. ## Bad vs Good Implementations ### Asynchronous Queue Context Propagation (Producer & Consumer) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The producer fails to serialize the active OpenTelemetry context into the job metadata, and the worker initiates a detached root span. This severs causality, drops upstream baggage attributes (such as tenant or correlation IDs), and creates fragmented traces. ```typescript import { Queue, Worker, Job } from 'bullmq'; import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('order-service'); const orderQueue = new Queue('orders'); // BAD: Enqueues job without injecting trace context or baggage into payload/metadata export async function enqueueOrder(orderId: string, correlationId: string) { return tracer.startActiveSpan('enqueueOrder', async (span) => { await orderQueue.add('process-order', { orderId, correlationId }); span.end(); }); } // BAD: Worker executes in an orphaned context; breaks the distributed causal trace new Worker('orders', async (job: Job) => { return tracer.startActiveSpan('processOrderWorker', async (span) => { // Orphan root span: traceparent and upstream baggage are completely lost await processOrder(job.data.orderId); span.end(); }); }); async function processOrder(id: string) {} ``` #### Production Standard (Correct) > **Standard**: The producer injects the active context into `job.data.traceCarrier`. The consumer extracts this carrier to reconstruct the exact upstream context using `ROOT_CONTEXT`, executing within `context.with()` and opening a `SpanKind.CONSUMER` child span that retains end-to-end distributed causality. ```typescript import { Queue, Worker, Job } from 'bullmq'; import { trace, context, propagation, SpanKind, ROOT_CONTEXT } from '@opentelemetry/api'; const tracer = trace.getTracer('order-service'); const orderQueue = new Queue('orders'); interface QueuePayload<T> { data: T; traceCarrier: Record<string, string>; } // GOOD: Injects current context (traceparent, baggage) into carrier metadata export async function enqueueOrder(orderId: string) { return tracer.startActiveSpan('enqueueOrder', { kind: SpanKind.PRODUCER }, async (span) => { try { const traceCarrier: Record<string, string> = {}; propagation.inject(context.active(), traceCarrier); await orderQueue.add('process-order', { data: { orderId }, traceCarrier, }); } finally { span.end(); } }); } // GOOD: Extracts carrier into parent context and establishes linked consumer span new Worker('orders', async (job: Job<QueuePayload<{ orderId: string }>>) => { const extractedContext = propagation.extract(ROOT_CONTEXT, job.data.traceCarrier); return context.with(extractedContext, async () => { return tracer.startActiveSpan( 'processOrderWorker', { kind: SpanKind.CONSUMER }, async (span) => { try { await processOrder(job.data.data.orderId); } catch (err) { span.recordException(err as Error); throw err; } finally { span.end(); } } ); }); }); async function processOrder(id: string) {} ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Trace Context and Baggage Propagation Across Asynchronous Task Queues

OpenTelemetry Trace Context and Baggage Propagation Across Asynchronous Task Queues defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic causal distributed tracing and transactional baggage preservation across asynchronous worker pools, task queues, and decentralized event boundaries. Mandates explicit W3C TraceContext and Baggage injection at enqueue time and root context extraction with span linking at execution time. Eliminates broken distributed causal graphs and context leakage across shared concurrency runtimes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Silent Context Severing and Blind Asynchronous Worker Execution Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Silent Context Severing and Blind Asynchronous Worker Execution	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS inject active OpenTelemetry trace context and baggage into message metadata using `propagation.inject(context.active(), carrier)` immediately prior to enqueuing a task.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.