NATS JetStream 2.11 Clustered Message Deduplication Window Invariant

--- description: NATS JetStream 2.11 Clustered Message Deduplication Window Invariant - Enforces deterministic, non-zero deduplication windows and client-side monotonic Nats-Msg-Id generation across Raft-replicated JetStream streams. Prevents at-least-once message explosion and silent double-execution during network partitions, cluster failovers, and slow consumer re-transmissions. Mandates aligned StreamConfig duplicate_window durations that match or exceed max consumer ack wait and producer retry horizons. globs: src/**/nats/**/*.{ts,js,go,py}, src/**/messaging/**/*.{ts,js,go,py}, src/**/streams/**/*.{ts,js,go,py}, src/**/events/**/*.{ts,js,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/nats-jetstream-deduplication-window-invariant" --- # NATS JetStream 2.11 Clustered Message Deduplication Window Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Publishing to JetStream without Nats-Msg-Id headers or using duplicate_window shorter than producer retry timeouts ## System Constraints You are a Principal Engineer enforcing NATS JetStream 2.11 Clustered Message Deduplication Window Invariants. You must ensure that all JetStream publishers supply deterministic, globally unique `Nats-Msg-Id` headers generated from transaction invariants, and that all stream declarations define an explicit `duplicate_window` that strictly dominates the client retry backoff envelope and consumer `ack_wait`. ## Strict Guidelines - Engineers MUST ALWAYS supply an explicit, deterministic `Nats-Msg-Id` header (or MsgId publish option) on every JetStream publish operation. - Engineers MUST NOT rely on auto-generated UUIDs at retry-time; `Nats-Msg-Id` MUST be derived deterministically from the business transaction domain key (e.g., `tx_<tenant>_<order_id>_<sequence>`). - Services MUST configure the stream's `duplicate_window` (defaults to 2 minutes in NATS) to strictly exceed the maximum producer retry timeout envelope (`max_retry_backoff * max_attempts`). - Engineers MUST NOT use ephemeral in-memory sequences as `Nats-Msg-Id` across service restarts, as Raft-replicated consumers will misidentify replay as new messages once the sequence resets. - Code MUST handle the `PubAck` response and inspect `ack.duplicate` to short-circuit downstream side-effects without throwing unexpected errors. - Stream configuration MUST NEVER set `duplicate_window: 0` or disable deduplication on streams targeted by transactional state-mutation pipelines. ## Bad vs Good Implementations ### JetStream Publishing and Stream Definition with Guaranteed Deduplication Window #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Publishing without an explicit Nats-Msg-Id allows network-partitioned ACKs to result in duplicates. If the publish succeeded on the leader broker but the TCP ACK dropped, the subsequent retry creates an un-deduplicated identical message on the cluster, triggering double-debit or duplicate fulfillment. ```typescript import { connect, JSONCodec } from 'nats'; export async function publishOrderEvent(nc: any, order: { id: string; amount: number }) { const js = nc.jetstream(); const jc = JSONCodec(); // ANTI-PATTERN: No Nats-Msg-Id provided; publish retries on transient errors // will cause silent double-execution in Raft-replicated consumers. let attempts = 0; while (attempts < 5) { try { const ack = await js.publish('orders.created', jc.encode(order)); return ack; } catch (err) { attempts++; await new Promise((res) => setTimeout(res, 1000 * Math.pow(2, attempts))); } } throw new Error('Failed to publish order event'); } ``` #### Production Standard (Correct) > **Standard**: Injects a deterministic domain-based 'Nats-Msg-Id' invariant into the message header and configures a 5-minute Raft-replicated duplicate window that strictly dominates the 31-second retry budget. Inspects ack.duplicate to gracefully absorb cluster-level deduplications without raising operational faults. ```typescript import { connect, JSONCodec, headers, JetStreamClient, StreamConfig, RetentionPolicy, StorageType } from 'nats'; export const ORDER_STREAM_CONFIG: Partial<StreamConfig> = { name: 'ORDERS', subjects: ['orders.>'], retention: RetentionPolicy.Limits, storage: StorageType.File, num_replicas: 3, // duplicate_window: 5 minutes (300,000,000,000 ns), strictly exceeding total retry budget (31s) duplicate_window: 300_000_000_000, }; export async function publishOrderEvent( js: JetStreamClient, order: { id: string; tenantId: string; version: number; amount: number } ) { const jc = JSONCodec(); // GUARANTEED INVARIANT: Monotonic, idempotent deduplication key derived from domain entity const deduplicationId = `tx_${order.tenantId}_${order.id}_v${order.version}`; const h = headers(); h.set('Nats-Msg-Id', deduplicationId); const maxAttempts = 5; for (let attempt = 1; attempt <= maxAttempts; attempt++) { try { const ack = await js.publish('orders.created', jc.encode(order), { headers: h }); if (ack.duplicate) { // Cluster recognized this ID inside duplicate_window; safe to treat as idempotent success return { success: true, duplicate: true, seq: ack.seq }; } return { success: true, duplicate: false, seq: ack.seq }; } catch (err: any) { if (attempt === maxAttempts) { throw new Error(`Exhausted retries publishing ${deduplicationId}: ${err.message}`); } // Exponential backoff with capped jitter within deduplication window budget const backoffMs = Math.min(1000 * Math.pow(2, attempt), 10_000); await new Promise((res) => setTimeout(res, backoffMs)); } } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: NATS JetStream 2.11 Clustered Message Deduplication Window Invariant

NATS JetStream 2.11 Clustered Message Deduplication Window Invariant defines an industrial-grade behavioral standard for backend engineering. Enforces deterministic, non-zero deduplication windows and client-side monotonic Nats-Msg-Id generation across Raft-replicated JetStream streams. Prevents at-least-once message explosion and silent double-execution during network partitions, cluster failovers, and slow consumer re-transmissions. Mandates aligned StreamConfig duplicate_window durations that match or exceed max consumer ack wait and producer retry horizons. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Publishing to JetStream without Nats-Msg-Id headers or using duplicate_window shorter than producer retry timeouts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Publishing to JetStream without Nats-Msg-Id headers or using duplicate_window shorter than producer retry timeouts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS supply an explicit, deterministic `Nats-Msg-Id` header (or MsgId publish option) on every JetStream publish operation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.