NATS JetStream Sliding Deduplication Window & Message Hash Invariant

--- description: NATS JetStream Sliding Deduplication Window & Message Hash Invariant - Enforces deterministic deduplication keys (Nats-Msg-Id) and guarantees stream sliding deduplication windows cover maximum edge retry backoff intervals. Prevents duplicate state mutation across network-partitioned edge topologies by asserting canonical sequence hashing and broker-side deduplication verification. globs: src/**/*.{ts,js,go}, infra/**/*.{json,yaml,yml,tf} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/nats-jetstream-deduplication-window-invariant-nats-jetstream-i-448" --- # NATS JetStream Sliding Deduplication Window & Message Hash Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Ephemerally Generated Message IDs and Truncated Deduplication Windows Inducing Double-Processing ## System Constraints You are a Principal Software Architect enforcing NATS JetStream Idempotency Window and Distributed Deduplication Invariants. You must ensure all JetStream publishers inject deterministic, idempotent deduplication headers (Nats-Msg-Id) derived from canonical business payloads or sequence identifiers, and verify that stream definitions configure duplicate_window to strictly exceed edge retry and backoff horizons. ## Strict Guidelines - Producers MUST ALWAYS set the `Nats-Msg-Id` header using a deterministic digest (e.g., canonical payload SHA-256 or aggregate root UUID + sequence version), NEVER an ephemeral `crypto.randomUUID()` generated per attempt inside a retry loop. - Engineers MUST configure JetStream `duplicate_window` to encompass the maximum total retry duration (initial backoff + exponential multipliers + edge partition jitter) plus safety margin. - Publishers MUST inspect the JetStream `PubAck` response; if `pubAck.duplicate === true`, the system MUST log the idempotency catch and treat the publication as successful without re-triggering business workflows. - Edge topologies MUST NOT rely on broker timestamps for deduplication; causal ordering and sequence identity MUST remain tied to deterministic headers across disconnected-mode buffer flushes. - JetStream streams MUST NOT set `duplicate_window` to zero or omit it when consumer processing demands exactly-once or at-least-once semantic integrity. - Consumer implementations MUST preserve idempotent handlers with local persistent transaction records to defend against edge deduplication window expiration. ## Bad vs Good Implementations ### JetStream Producer Deterministic Header & Sliding Window Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Generating a non-deterministic random UUID for each retry attempt defeats JetStream broker-side deduplication. If an edge partition experiences an unacknowledged publish that actually reached the broker, the subsequent retry with a new random UUID produces duplicate execution and data corruption. ```typescript import { connect, headers } from 'nats'; import { randomUUID } from 'node:crypto'; export async function publishEdgeTelemetry(nc: any, streamName: string, event: { deviceId: string; sequence: number; payload: any }) { const js = nc.jetstream(); // ANTI-PATTERN: Generating a fresh UUID per attempt defeats JetStream deduplication on retries. // ANTI-PATTERN: No header set, or random ID resets JetStream's duplicate window tracking. let attempts = 0; while (attempts < 5) { try { const h = headers(); h.set('Nats-Msg-Id', randomUUID()); // Causes duplicate writes upon partition self-healing await js.publish(`telemetry.${event.deviceId}`, JSON.stringify(event.payload), { headers: h }); break; } catch (err) { attempts++; await new Promise(r => setTimeout(r, Math.pow(2, attempts) * 1000)); } } } ``` #### Production Standard (Correct) > **Standard**: The good implementation constructs a deterministic deduplication ID derived from edge device sequence primitives. In the event of network partition timeouts or edge retry flushes, JetStream detects the identical `Nats-Msg-Id`, rejects duplicate insertion into the storage tier, and flags `ack.duplicate = true`. ```typescript import { connect, headers, JetStreamClient, natsHasher } from 'nats'; import { createHash } from 'node:crypto'; interface EdgeEvent { deviceId: string; sequenceNumber: number; timestamp: number; payload: Record<string, unknown>; } export function computeDeduplicationId(event: EdgeEvent): string { // INVARIANT: Deterministic hash incorporating origin identity and monotonically increasing sequence const rawKey = `${event.deviceId}:${event.sequenceNumber}:${event.timestamp}`; return createHash('sha256').update(rawKey).digest('hex'); } export async function publishEdgeTelemetrySafe( js: JetStreamClient, subject: string, event: EdgeEvent, maxRetries = 4 ): Promise<{ duplicate: boolean; seq: number }> { const deduplicationId = computeDeduplicationId(event); const h = headers(); h.set('Nats-Msg-Id', deduplicationId); let attempt = 0; const baseDelayMs = 500; while (attempt <= maxRetries) { try { const ack = await js.publish(subject, JSON.stringify(event.payload), { headers: h, timeout: 3000 }); // Verify if broker flagged message as a duplicate within the sliding window if (ack.duplicate) { return { duplicate: true, seq: ack.seq }; } return { duplicate: false, seq: ack.seq }; } catch (err) { attempt++; if (attempt > maxRetries) throw err; const jitter = Math.floor(Math.random() * 200); const delay = Math.min(baseDelayMs * Math.pow(2, attempt) + jitter, 10000); await new Promise((res) => setTimeout(res, delay)); } } throw new Error('Unreachable: retry limit exceeded'); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: NATS JetStream Sliding Deduplication Window & Message Hash Invariant

NATS JetStream Sliding Deduplication Window & Message Hash Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic deduplication keys (Nats-Msg-Id) and guarantees stream sliding deduplication windows cover maximum edge retry backoff intervals. Prevents duplicate state mutation across network-partitioned edge topologies by asserting canonical sequence hashing and broker-side deduplication verification. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Ephemerally Generated Message IDs and Truncated Deduplication Windows Inducing Double-Processing Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Ephemerally Generated Message IDs and Truncated Deduplication Windows Inducing Double-Processing	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Producers MUST ALWAYS set the `Nats-Msg-Id` header using a deterministic digest (e.g., canonical payload SHA-256 or aggregate root UUID + sequence version), NEVER an ephemeral `crypto.randomUUID()` generated per attempt inside a retry loop.

Producers MUST ALWAYS set the `Nats-Msg-Id` header using a deterministic digest (e.g., canonical payload SHA-256 or aggregate root UUID + sequence version), NEVER an ephemeral `crypto.randomUUID()` generated per attempt inside a retry loop.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.