OpenTelemetry Span Link DAG Invariant for Decoupled Topologies

--- description: OpenTelemetry Span Link DAG Invariant for Decoupled Topologies - Preserves causality across asynchronous, fan-out, and event-driven choreographies by enforcing OpenTelemetry Span Links instead of synchronous parent-child hierarchies. Prevent trace graph corruption, artificial latency aggregation, and severed causal directed acyclic graphs (DAGs) across message brokers, job queues, and decoupled streams. globs: src/consumers/**/*.{ts,js}, src/workers/**/*.{ts,js}, src/events/**/*.{ts,js}, src/messaging/**/*.{ts,js}, src/queues/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-span-link-dag-invariant" --- # OpenTelemetry Span Link DAG Invariant for Decoupled Topologies > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Parenting Asynchronous Consumers Directly to Producer Spans via Synchronous Context Propagation ## System Constraints You are a Principal Observability and Distributed Systems Architect enforcing the OpenTelemetry Span Link DAG Invariant. You must reject any attempt to make asynchronous background jobs, consumer message processors, or batched handlers direct synchronous children of transient producer spans. Instead, you mandate starting independent consumer root spans that causal-link back to the extracted producer contexts. ## Strict Guidelines - Engineers MUST ALWAYS extract incoming trace contexts from message payloads or headers into a distinct Link object using `trace.setSpanContext(ROOT_CONTEXT, extractedContext)`. - Engineers MUST NOT pass the extracted producer `Context` as the parent scope in `tracer.startActiveSpan()` when consuming asynchronous, decoupled event-driven tasks. - Engineers MUST start consumer spans with `root: true` or directly attached to `ROOT_CONTEXT` when the operation does not synchronously block the producer. - Services handling batch operations MUST attach multiple `Link` entries to a single processing span rather than choosing an arbitrary message as the parent. - Components NEVER represent independent lifecycles (e.g., publish-subscribe, delayed queues, dead-letter re-drives) as synchronous child spans, which distorts trace duration statistics and breaks waterfall visualizations. ## Bad vs Good Implementations ### Kafka Event Consumer Context Propagation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly passing extractedContext as the active parent binds the consumer's execution window to the upstream producer's trace tree. This causes trace timeline fragmentation, artificial duration skew in APM tooling, and corrupts fan-out DAGs where multiple independent workers process the same event asynchronously. ```typescript import { trace, propagation, SpanKind } from '@opentelemetry/api'; import { EachMessagePayload } from 'kafkajs'; const tracer = trace.getTracer('orders-consumer'); export async function handleOrderEvent({ message }: EachMessagePayload): Promise<void> { // ANTI-PATTERN: Activating the producer's context directly makes the consumer a synchronous child span. // This falsely couples the consumer duration to the producer trace and corrupts the root trace hierarchy. const extractedContext = propagation.extract(trace.getContext(), message.headers, { get: (headers, key) => headers?.[key]?.toString(), keys: (headers) => Object.keys(headers || {}), }); await tracer.startActiveSpan('process_order_event', { kind: SpanKind.CONSUMER }, extractedContext, async (span) => { try { await processMessage(message.value); span.setStatus({ code: 1 }); } catch (err) { span.recordException(err as Error); throw err; } finally { span.end(); } }); } ``` #### Production Standard (Correct) > **Standard**: Starting a root span with causal Span Links preserves cross-service lineage across asynchronous message boundaries while maintaining accurate trace durations and clean DAG visualizations in Jaeger, Tempo, or Honeycomb. ```typescript import { trace, propagation, SpanKind, ROOT_CONTEXT } from '@opentelemetry/api'; import { EachMessagePayload } from 'kafkajs'; const tracer = trace.getTracer('orders-consumer'); export async function handleOrderEvent({ message }: EachMessagePayload): Promise<void> { // PRODUCTION INVARIANT: Extract remote context and explicitly model it as a causal Span Link const extractedContext = propagation.extract(ROOT_CONTEXT, message.headers, { get: (headers, key) => headers?.[key]?.toString(), keys: (headers) => Object.keys(headers || {}), }); const producerSpanContext = trace.getSpan(extractedContext)?.spanContext(); const links = producerSpanContext ? [{ context: producerSpanContext }] : []; // Start an independent trace root with Span Links to maintain causality without lifecycle coupling await tracer.startActiveSpan( 'process_order_event', { kind: SpanKind.CONSUMER, root: true, links, attributes: { 'messaging.system': 'kafka', 'messaging.operation': 'process', }, }, ROOT_CONTEXT, async (span) => { try { await processMessage(message.value); span.setStatus({ code: 1 }); } catch (err) { span.recordException(err as Error); span.setStatus({ code: 2, message: (err as Error).message }); throw err; } finally { span.end(); } } ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Span Link DAG Invariant for Decoupled Topologies

OpenTelemetry Span Link DAG Invariant for Decoupled Topologies defines an industrial-grade behavioral standard for devops engineering. Preserves causality across asynchronous, fan-out, and event-driven choreographies by enforcing OpenTelemetry Span Links instead of synchronous parent-child hierarchies. Prevent trace graph corruption, artificial latency aggregation, and severed causal directed acyclic graphs (DAGs) across message brokers, job queues, and decoupled streams. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Parenting Asynchronous Consumers Directly to Producer Spans via Synchronous Context Propagation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Parenting Asynchronous Consumers Directly to Producer Spans via Synchronous Context Propagation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS extract incoming trace contexts from message payloads or headers into a distinct Link object using `trace.setSpanContext(ROOT_CONTEXT, extractedContext)`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.