DuckDB-Wasm Dedicated Worker & Memory Pool Isolation Guardrail

--- description: DuckDB-Wasm Dedicated Worker & Memory Pool Isolation Guardrail - Restricts all in-browser analytical compute with DuckDB-Wasm to isolated Web Worker execution contexts bounded by strict WebAssembly memory pools and watchdog monitors. Prohibits main-thread database instantiation and unconstrained buffer growth that induce rendering pipeline starvation and fatal browser tab terminations. globs: src/**/*.{ts,tsx}, src/workers/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/duckdb-wasm-worker-memory-isolation" --- # DuckDB-Wasm Dedicated Worker & Memory Pool Isolation Guardrail > **Severity**: CRITICAL | **Category**: frontend > **Anti-Pattern Warning**: Direct Main-Thread DuckDB Instantiation and Uncapped WebAssembly Memory Allocation ## System Constraints You are a Principal Software Architect enforcing production-grade DuckDB-Wasm isolation standards. Analytical SQL queries executed in-browser can rapidly exhaust the 32-bit WebAssembly linear memory pool (~2-4GB browser ceiling) and block the single-threaded UI event loop. Mandate strict execution inside dedicated Web Workers, enforce query memory caps via SQL pragma/config, propagate AbortSignals to terminate runaway worker threads, and transfer results using zero-copy Arrow IPC buffers. ## Strict Guidelines - Engineers MUST ALWAYS instantiate DuckDB-Wasm via an isolated Web Worker (e.g., `duckdb.createWorker` or dedicated Worker module) using `AsyncDuckDB` rather than running instances on the main browser thread. - Engineers MUST ALWAYS configure explicit linear memory ceilings immediately upon connection initialization (e.g., `SET max_memory = '512MB';` and `SET threads TO 2;`) to prevent unbounded memory growth and unexpected tab crashes. - Engineers MUST NOT transfer query results across the worker boundary via unstructured JSON serialization (`toArray()`); transfer MUST utilize Arrow IPC stream buffers (`tableToIPC()`) or transferable `ArrayBuffer` objects. - Services MUST implement a watchdog supervisor that accepts an `AbortSignal` and proactively terminates the underlying `Worker` instance if a query times out or approaches browser memory quotas. - Components NEVER bind UI lifecycle hooks directly to raw DuckDB connection lifecycles; interactions must flow through an asynchronous worker client facade with explicit tear-down routines. ## Bad vs Good Implementations ### Main-Thread Execution vs Isolated Web Worker with Hard Memory Caps #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Instantiating DuckDB-Wasm and executing queries directly on the main thread causes significant frame jank (blocking layout/paint). Omitting memory limits (`max_memory`) allows complex analytical queries to breach the 4GB WebAssembly 32-bit limit, instantly crashing the tab without recoverable errors. Converting Arrow tables to JSON objects via `.toArray()` duplicates memory allocations. ```typescript import * as duckdb from '@duckdb/duckdb-wasm'; // ANTI-PATTERN: Running DuckDB-Wasm directly on the main UI thread with no memory pool caps export async function runClientQuery(sql: string) { const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles(); const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES); // Synchronous main-thread logger and instantiate blocks DOM render frames const logger = new duckdb.ConsoleLogger(); const db = new duckdb.AsyncDuckDB(logger, new Worker(bundle.mainWorker!)); await db.instantiate(bundle.mainModule, bundle.pthreadWorker); const conn = await db.connect(); // Uncapped memory query triggers OOM and converts all rows to large main-thread JS objects const result = await conn.query(sql); return result.toArray().map((row) => row.toJSON()); } ``` #### Production Standard (Correct) > **Standard**: The database is instantiated in an isolated Web Worker, shielding the main thread from UI frame jank. Queries set `max_memory` to prevent exceeding Wasm linear memory thresholds. The client accepts an `AbortSignal` that immediately terminates the worker thread if the user navigates away or cancels, avoiding runaway allocations. Returns Arrow IPC bytes (`Uint8Array`) to minimize garbage collection overhead. ```typescript import * as duckdb from '@duckdb/duckdb-wasm'; export interface QueryExecutionOptions { memoryLimit?: string; signal?: AbortSignal; } // PRODUCTION INVARIANT: Dedicated Worker client with hard memory ceiling and abort lifecycle export class IsolatedDuckDBClient { private worker: Worker | null = null; private db: duckdb.AsyncDuckDB | null = null; public async initialize(): Promise<void> { const bundles = duckdb.getJsDelivrBundles(); const bundle = await duckdb.selectBundle(bundles); // Create worker dedicated purely to database wasm compute this.worker = new Worker(bundle.mainWorker!); const logger = new duckdb.VoidLogger(); this.db = new duckdb.AsyncDuckDB(logger, this.worker); await this.db.instantiate(bundle.mainModule, bundle.pthreadWorker); } public async executeQuery( sql: string, options: QueryExecutionOptions = {} ): Promise<Uint8Array> { if (!this.db || !this.worker) { throw new Error('DuckDB Worker is not initialized'); } const conn = await this.db.connect(); const limit = options.memoryLimit ?? '512MB'; try { // Enforce hard memory ceiling to prevent tab termination await conn.query(`SET max_memory = '${limit}';`); if (options.signal?.aborted) { throw new DOMException('Query aborted prior to execution', 'AbortError'); } const abortHandler = () => { // Hard kill the worker immediately on client abort this.worker?.terminate(); this.worker = null; this.db = null; }; options.signal?.addEventListener('abort', abortHandler, { once: true }); const table = await conn.query(sql); options.signal?.removeEventListener('abort', abortHandler); // Zero-copy Arrow IPC serialization without main-thread JSON ballooning return duckdb.tableToIPC(table); } finally { await conn.close(); } } public async dispose(): Promise<void> { if (this.db) { await this.db.terminate(); } if (this.worker) { this.worker.terminate(); } this.worker = null; this.db = null; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: DuckDB-Wasm Dedicated Worker & Memory Pool Isolation Guardrail

DuckDB-Wasm Dedicated Worker & Memory Pool Isolation Guardrail defines an industrial-grade behavioral standard for frontend engineering. Restricts all in-browser analytical compute with DuckDB-Wasm to isolated Web Worker execution contexts bounded by strict WebAssembly memory pools and watchdog monitors. Prohibits main-thread database instantiation and unconstrained buffer growth that induce rendering pipeline starvation and fatal browser tab terminations. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Direct Main-Thread DuckDB Instantiation and Uncapped WebAssembly Memory Allocation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Direct Main-Thread DuckDB Instantiation and Uncapped WebAssembly Memory Allocation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS instantiate DuckDB-Wasm via an isolated Web Worker (e.g., `duckdb.createWorker` or dedicated Worker module) using `AsyncDuckDB` rather than running instances on the main browser thread.

Engineers MUST ALWAYS instantiate DuckDB-Wasm via an isolated Web Worker (e.g., `duckdb.createWorker` or dedicated Worker module) using `AsyncDuckDB` rather than running instances on the main browser thread.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.