Temporal Durable Execution Determinism & Replay Verification Standard

--- description: Temporal Durable Execution Determinism & Replay Verification Standard - Enforces strict workflow determinism invariants, backward-compatible migration patterns using workflow.patched(), and automated history replay verification in CI/CD. Eliminates NonDeterministicWorkflowPolicy errors, replay divergence, and state corruption across distributed asynchronous workers undergoing rolling deployments. globs: src/workflows/**/*.{ts,js}, src/temporal/**/*.{ts,js}, tests/temporal/**/*.{ts,js}, workflows/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/temporal-workflow-determinism-replay-migration-standard" --- # Temporal Durable Execution Determinism & Replay Verification Standard > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unversioned Workflow Topology Mutations and Non-Deterministic Runtime Operations Causing Production Replay Panics ## System Constraints You are a Principal Distributed Systems Architect enforcing Temporal Durable Execution Replay Verification & State Migration Standards. Mandate absolute determinism inside workflow functions: forbid native I/O, volatile clock access, uncontrolled randomness, and unversioned AST modifications. Enforce patching protocols and CI replay testing against production histories. ## Strict Guidelines - Engineers MUST ALWAYS isolate all side effects, network requests, database transactions, and random or clock-dependent values inside Activities, never directly inside Workflow functions. - Engineers MUST ALWAYS wrap workflow topology alterations (adding, reordering, or removing activities, child workflows, timers, or signals) in `workflow.patched(patchId)` to maintain backward replay compatibility for in-flight executions. - Engineers MUST NOT delete a `workflow.patched(patchId)` conditional branch until all historical workflows executed on previous code versions have definitively reached a terminal status (Completed, Failed, or Canceled). - Engineers MUST run automated pre-merge regression tests using `WorkflowReplayer.replayHistory()` against exported production history JSON fixtures for every modified workflow definition. - Services MUST utilize Temporal Worker Versioning (Deployment IDs / Build IDs) when shipping changes that structurally obsolete deprecated workflow code branches. - Workflow functions MUST NEVER access global mutable state, mutate parameters passed by reference across asynchronous yields, or use unseeded math/crypto primitives (e.g., `Math.random()`, `crypto.randomUUID()`, `Date.now()`). - Engineers MUST replace native time and asynchronous primitives with workflow-safe constructs: use `workflow.sleep()` instead of `setTimeout`, `workflow.condition()` instead of polling loops, and deterministically sorted iterations instead of non-deterministic object key traversals. ## Bad vs Good Implementations ### Workflow Modification: Unversioned Topology Mutation vs workflow.patched Migration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly introducing the 'runFraudCheck' activity causes active workflows replaying from past history to encounter an ActivityTaskScheduled event where none was recorded, throwing NonDeterministicWorkflowError. Using native Date.now() and crypto.randomUUID() introduces non-deterministic values that diverge between the initial execution and subsequent worker replay cycles. ```typescript import { proxyActivities, sleep } from '@temporalio/workflow'; import type * as activities from '../activities'; const { chargeCustomer, sendReceipt, runFraudCheck } = proxyActivities<typeof activities>({ startToCloseTimeout: '1 minute', }); // ANTI-PATTERN: Direct topology mutation and non-deterministic native clock/UUID usage export async function processOrderWorkflow(orderId: string, amount: number): Promise<void> { // FAILS REPLAY: runFraudCheck was added directly in path of in-flight workflows await runFraudCheck(orderId); // FAILS REPLAY: Date.now() returns current epoch during replay, deviating from recorded execution const timestamp = Date.now(); const traceId = crypto.randomUUID(); // FAILS REPLAY: Generates non-deterministic value await chargeCustomer(orderId, amount, traceId, timestamp); await sendReceipt(orderId); } ``` #### Production Standard (Correct) > **Standard**: Uses 'patched(FRAUD_CHECK_PATCH_ID)' to conditionally schedule 'runFraudCheck'. Active executions created before the change bypass the branch during replay without divergence, while new executions run the updated logic. Eliminates native non-deterministic APIs in favor of Temporal-safe deterministic utilities or Activity isolation. ```typescript import { proxyActivities, patched, deterministicRandom } from '@temporalio/workflow'; import type * as activities from '../activities'; const { chargeCustomer, sendReceipt, runFraudCheck } = proxyActivities<typeof activities>({ startToCloseTimeout: '1 minute', }); const FRAUD_CHECK_PATCH_ID = 'fraud-check-step-v1'; // PRODUCTION FIX: Safe continuous migration via workflow.patched() export async function processOrderWorkflow(orderId: string, amount: number): Promise<void> { // In-flight workflows initiated prior to this patch evaluate to false and skip during replay if (patched(FRAUD_CHECK_PATCH_ID)) { await runFraudCheck(orderId); } // Temporal SDK deterministic random seed generator const rng = deterministicRandom(); const traceId = `trace-${Math.floor(rng() * 1000000)}`; // Side effects like wall-clock timestamps must either be generated in activities or supplied by workflow state await chargeCustomer(orderId, amount, traceId); await sendReceipt(orderId); } ``` ### CI/CD Guardrail: Temporal Workflow History Replayer Test #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Testing workflow execution from scratch verifies only the forward-facing happy path. It fails to assert that existing, in-flight workflow histories from previous deployment versions can be successfully replayed by the modified code without throwing NonDeterministicWorkflowError. ```typescript // ANTI-PATTERN: Testing workflow solely with unit mocks without verifying history replay import { TestWorkflowEnvironment } from '@temporalio/testing'; import { Worker } from '@temporalio/worker'; import { processOrderWorkflow } from './order-workflow'; describe('OrderWorkflow Unit Test', () => { it('executes successfully', async () => { const testEnv = await TestWorkflowEnvironment.createLocal(); try { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test-queue', workflowsPath: require.resolve('./order-workflow'), }); // This does not validate that historical production event streams remain compatible! expect(worker).toBeDefined(); } finally { await testEnv.teardown(); } }); }); ``` #### Production Standard (Correct) > **Standard**: Validates the current workflow code against a real production event history JSON dump using Worker.runReplayHistory. If a developer alters workflow event sequencing without a valid patch, this test immediately fails in the CI pipeline prior to merging. ```typescript // PRODUCTION FIX: Mandatory Replay Verification Test against Production JSON History import { Worker } from '@temporalio/worker'; import * as fs from 'fs/promises'; import * as path from 'path'; describe('OrderWorkflow Replay Verification', () => { it('replays historical production executions without workflow divergence', async () => { const historyFixturePath = path.resolve(__dirname, '../fixtures/order-workflow-history.json'); const rawHistory = await fs.readFile(historyFixturePath, 'utf-8'); const historyJson = JSON.parse(rawHistory); // WorkflowReplayer asserts that executing the current workflow code matches the recorded history events precisely await Worker.runReplayHistory( { workflowsPath: require.resolve('./order-workflow'), }, historyJson ); }); }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Temporal Durable Execution Determinism & Replay Verification Standard

Temporal Durable Execution Determinism & Replay Verification Standard defines an industrial-grade behavioral standard for backend engineering. Enforces strict workflow determinism invariants, backward-compatible migration patterns using workflow.patched(), and automated history replay verification in CI/CD. Eliminates NonDeterministicWorkflowPolicy errors, replay divergence, and state corruption across distributed asynchronous workers undergoing rolling deployments. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unversioned Workflow Topology Mutations and Non-Deterministic Runtime Operations Causing Production Replay Panics Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unversioned Workflow Topology Mutations and Non-Deterministic Runtime Operations Causing Production Replay Panics	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS isolate all side effects, network requests, database transactions, and random or clock-dependent values inside Activities, never directly inside Workflow functions.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.