OpenTelemetry Span Link Propagation for Detached Asynchronous Queue Consumers

--- description: OpenTelemetry Span Link Propagation for Detached Asynchronous Queue Consumers - Enforces asynchronous batch and queue consumers to initialize decoupled trace contexts linked causally via Span Links instead of inheriting parent context directly. Prevents broken distributed traces, skewed latency metrics, and incomplete error telemetry when consumer execution outlives publisher scope or processes multi-origin batches under cascading failures. globs: src/queues/**/*.{ts,js}, src/consumers/**/*.{ts,js}, src/workers/**/*.{ts,js}, src/events/**/*.{ts,js}, src/messaging/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-non-parented-span-link-propagation-standard" --- # OpenTelemetry Span Link Propagation for Detached Asynchronous Queue Consumers > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Direct Parent-Child Context Inheritance in Batch Queue Consumers Causing Context Leaks and Distorted Latencies ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry standards across asynchronous message brokers and decoupled queue consumers. In event-driven distributed systems, child spans must not directly inherit producer contexts across asynchronous batch boundaries or detached consumer retries. You must mandate the extraction of carrier metadata into explicit OpenTelemetry Span Links on newly rooted consumer transactions, preserving causal ancestry without distorting producer trace durations, cross-tenant contexts, or cascading error domains. ## Strict Guidelines - Engineers MUST ALWAYS extract incoming message carrier headers using `propagation.extract()` and attach the resolved `SpanContext` to the consumer span as an item in `links`, rather than passing it as the parent `Context`. - Engineers MUST NOT bind batch consumer transactions to a single message's trace context as its active parent, which corrupts trace duration and obscures multi-message boundaries. - Services processing batches MUST initialize an independent root consumer span (`SpanKind.CONSUMER`) with links referencing every extracted message trace in the batch payload. - Engineers MUST record failure cascades, dead-letter re-queues, and consumer retries with dedicated error events on the consumer span without overwriting or mutating the producer's original span status. - Workers MUST preserve carrier metadata across dead-letter queue (DLQ) and retry routing by propagating both the original producer context and preceding consumer fault context as span attributes or secondary links. ## Bad vs Good Implementations ### Batch Queue Consumer Context Extraction and Span Link Binding #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Adopting the first message's context as the parent trace context distorts the producer trace timeline, couples the batch consumer lifecycle to arbitrary messages, and strips causal tracking for all subsequent items in the batch. ```typescript import { trace, context, defaultTextMapGetter } from '@opentelemetry/api'; interface QueueMessage { id: string; headers: Record<string, string>; payload: unknown; } export async function processQueueMessages(messages: QueueMessage[]): Promise<void> { const tracer = trace.getTracer('payment-consumer'); // ANTI-PATTERN: Adopting the first message's trace context as the active parent. // This invalidates latency bounds, breaks trace trees for other messages in the batch, // and falsely binds detached failure cascades to the first producer's lifecycle. const firstCarrier = messages[0]?.headers ?? {}; const extractedContext = context.propagation.extract(context.active(), firstCarrier, defaultTextMapGetter); await context.with(extractedContext, async () => { const span = tracer.startSpan('processBatch'); try { for (const message of messages) { // Sub-tasks falsely appear as direct descendants of message[0] await handleMessage(message); } span.setStatus({ code: 1 }); } catch (err: unknown) { span.recordException(err as Error); span.setStatus({ code: 2, message: (err as Error).message }); throw err; } finally { span.end(); } }); } async function handleMessage(_msg: QueueMessage): Promise<void> { /* processing logic */ } ``` #### Production Standard (Correct) > **Standard**: Creates a distinct root consumer span using `context.ROOT_CONTEXT` and populates the `links` array with validated `SpanContext` references from all incoming batch messages. This guarantees accurate telemetry metrics while retaining complete lineage across asynchronous boundaries. ```typescript import { trace, context, defaultTextMapGetter, Link, SpanKind, SpanStatusCode } from '@opentelemetry/api'; interface QueueMessage { id: string; headers: Record<string, string>; payload: unknown; } export async function processQueueMessages(messages: QueueMessage[]): Promise<void> { const tracer = trace.getTracer('payment-consumer'); // INVARIANT FIX: Extract each carrier context independently and convert to non-parented Span Links. const links: Link[] = messages.flatMap((message) => { const extracted = context.propagation.extract(context.active(), message.headers, defaultTextMapGetter); const spanCtx = trace.getSpanContext(extracted); return spanCtx && spanCtx.isValid ? [{ context: spanCtx, attributes: { 'messaging.message.id': message.id } }] : []; }); // Root consumer span decoupled from publisher lifecycles, maintaining bidirectional causal links const consumerSpan = tracer.startSpan('processBatch', { kind: SpanKind.CONSUMER, links, attributes: { 'messaging.batch.size': messages.length, 'messaging.system': 'rabbitmq' } }, context.ROOT_CONTEXT); const consumerContext = trace.setSpan(context.ROOT_CONTEXT, consumerSpan); try { await context.with(consumerContext, async () => { for (const message of messages) { const messageSpan = tracer.startSpan(`processMessage:${message.id}`, { kind: SpanKind.INTERNAL }); try { await handleMessage(message); messageSpan.setStatus({ code: SpanStatusCode.OK }); } catch (error: unknown) { messageSpan.recordException(error as Error); messageSpan.setStatus({ code: SpanStatusCode.ERROR, message: error instanceof Error ? error.message : 'Processing failure' }); throw error; } finally { messageSpan.end(); } } }); consumerSpan.setStatus({ code: SpanStatusCode.OK }); } catch (batchError: unknown) { consumerSpan.recordException(batchError as Error); consumerSpan.setStatus({ code: SpanStatusCode.ERROR, message: batchError instanceof Error ? batchError.message : 'Batch execution failed' }); throw batchError; } finally { consumerSpan.end(); } } async function handleMessage(_msg: QueueMessage): Promise<void> { /* processing logic */ } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Span Link Propagation for Detached Asynchronous Queue Consumers

OpenTelemetry Span Link Propagation for Detached Asynchronous Queue Consumers defines an industrial-grade behavioral standard for devops engineering. Enforces asynchronous batch and queue consumers to initialize decoupled trace contexts linked causally via Span Links instead of inheriting parent context directly. Prevents broken distributed traces, skewed latency metrics, and incomplete error telemetry when consumer execution outlives publisher scope or processes multi-origin batches under cascading failures. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Direct Parent-Child Context Inheritance in Batch Queue Consumers Causing Context Leaks and Distorted Latencies Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Direct Parent-Child Context Inheritance in Batch Queue Consumers Causing Context Leaks and Distorted Latencies	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS extract incoming message carrier headers using `propagation.extract()` and attach the resolved `SpanContext` to the consumer span as an item in `links`, rather than passing it as the parent `Context`.

Engineers MUST ALWAYS extract incoming message carrier headers using `propagation.extract()` and attach the resolved `SpanContext` to the consumer span as an item in `links`, rather than passing it as the parent `Context`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.