DuckDB-WASM Linear Memory Arena Ceilings and Disk Spilling Guardrail

--- description: DuckDB-WASM Linear Memory Arena Ceilings and Disk Spilling Guardrail - Enforces strict WebAssembly linear memory limits and Origin Private File System (OPFS)/IndexedDB disk-spilling configurations for DuckDB-WASM instances. Prevents browser tab crashes and uncatchable WebAssembly.Memory out-of-bounds faults by setting hard memory budgets and redirecting intermediate query scratch buffers to browser-backed storage. globs: src/**/duckdb*.{ts,tsx}, src/**/*worker*.{ts,tsx}, src/**/analytics/**/*.{ts,tsx}, src/**/olap/**/*.{ts,tsx} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/duckdb-wasm-memory-spilling-guardrail" --- # DuckDB-WASM Linear Memory Arena Ceilings and Disk Spilling Guardrail > **Severity**: CRITICAL | **Category**: frontend > **Anti-Pattern Warning**: Unbounded DuckDB-WASM In-Memory Execution Without Memory Limits or Scratch Filesystem Tiering ## System Constraints You are a Principal Engineer enforcing DuckDB-WASM Linear Memory Arena Spilling Guardrails. WebAssembly 32-bit linear memory is constrained to a hard 2GB-4GB boundary, but browser renderer processes routinely terminate tabs when heap allocations cross 1.5GB to 2GB. You must mandate strict memory ceiling configurations, OPFS/IndexedDB-backed temporary spill directories, query streaming via Arrow record batches, and strict worker-thread isolation. ## Strict Guidelines - Engineers MUST ALWAYS configure DuckDB runtime parameters with explicit memory ceilings ('SET max_memory = <limit>') calculated as a safe fraction of available browser resources (e.g., maximum 512MB-1024MB). - Engineers MUST ALWAYS initialize an Origin Private File System (OPFS) or IndexedDB-backed scratch mount and assign 'SET temp_directory = <path>' to enable external sorting and hash-join spilling. - Engineers MUST ALWAYS isolate DuckDB-WASM instantiations inside a dedicated Web Worker to prevent UI thread lockups and renderer crashes. - Engineers MUST NEVER invoke '.toArray()' or eagerly materialize full record sets into the JavaScript heap; all tabular data MUST be consumed via streaming Arrow chunk readers or Paginated LIMIT/OFFSET cursors. - Services MUST explicitly call 'connection.close()' and release intermediate WASM buffers upon query completion or worker lifecycle termination. - Components NEVER execute raw analytical OLAP joins directly on the main thread or without memory budget estimation guards. ## Bad vs Good Implementations ### Configuring Memory Ceilings and OPFS Disk Spilling in Web Worker #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: DuckDB-WASM is instantiated without memory configuration constraints or a physical spill directory. When large datasets or heavy hash joins exceed WebAssembly linear memory boundaries (~2GB or renderer quotas), the browser process throws an unrecoverable RuntimeError: memory access out of bounds or kills the tab. Furthermore, calling .toArray() copies the entire dataset into the V8 engine heap, doubling memory pressure. ```typescript import * as duckdb from '@duckdb/duckdb-wasm'; export async function executeQuery(sql: string) { // Anti-pattern: Default in-memory allocation without limits or scratch filesystem const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles(); const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES); const worker = new Worker(bundle.mainWorker!); const logger = new duckdb.VoidLogger(); const db = new duckdb.AsyncDuckDB(logger, worker); await db.instantiate(bundle.mainModule); const conn = await db.connect(); // Unbounded memory usage on joins and aggregations leads to uncatchable WASM OOM const result = await conn.query(sql); // Eagerly materializes entire Arrow record set into V8 JS memory const rows = result.toArray(); return rows; } ``` #### Production Standard (Correct) > **Standard**: Configures a rigid memory ceiling (768MB) via 'SET max_memory' and initializes an explicit persistent scratch directory via 'SET temp_directory' before running analytical queries. Result batches are streamed asynchronously rather than eagerly materialized with .toArray(), ensuring linear WebAssembly memory and V8 heap remain well below browser crash thresholds. ```typescript import * as duckdb from '@duckdb/duckdb-wasm'; import { AsyncDuckDBConnection } from '@duckdb/duckdb-wasm'; const WASM_MEMORY_LIMIT_BYTES = 768 * 1024 * 1024; // 768MB safe ceiling const SCRATCH_PATH = '/duckdb_scratch'; export async function initializeGuardedDuckDB(): Promise<duckdb.AsyncDuckDB> { const bundles = duckdb.getJsDelivrBundles(); const bundle = await duckdb.selectBundle(bundles); // Instantiate in dedicated worker context with memory budget validation const worker = new Worker(bundle.mainWorker!); const db = new duckdb.AsyncDuckDB(new duckdb.VoidLogger(), worker); await db.instantiate(bundle.mainModule, bundle.pthreadWorker); // Mount OPFS/IndexedDB directory for temporary disk spilling await db.registerEmptyFileBuffer(`${SCRATCH_PATH}/spill.tmp`); const conn = await db.connect(); try { // Pin memory ceiling and route spill buffers outside linear WASM memory await conn.query(`SET max_memory = '${Math.floor(WASM_MEMORY_LIMIT_BYTES / (1024 * 1024))}MB';`); await conn.query(`SET temp_directory = '${SCRATCH_PATH}';`); await conn.query(`SET preserve_insertion_order = false;`); } finally { await conn.close(); } return db; } export async function* streamQueryResults( conn: AsyncDuckDBConnection, sql: string, batchSize: number = 2048 ): AsyncGenerator<Record<string, unknown>[]> { // Stream results via batches instead of eager materialization into V8 heap const reader = await conn.send(sql); for await (const batch of reader) { const rows: Record<string, unknown>[] = []; const numRows = batch.numRows; for (let i = 0; i < numRows; i++) { const row = batch.get(i); if (row) rows.push(row.toJSON()); if (rows.length >= batchSize) { yield rows; rows.length = 0; } } if (rows.length > 0) { yield rows; } } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: DuckDB-WASM Linear Memory Arena Ceilings and Disk Spilling Guardrail

DuckDB-WASM Linear Memory Arena Ceilings and Disk Spilling Guardrail defines an industrial-grade behavioral standard for frontend engineering. Enforces strict WebAssembly linear memory limits and Origin Private File System (OPFS)/IndexedDB disk-spilling configurations for DuckDB-WASM instances. Prevents browser tab crashes and uncatchable WebAssembly.Memory out-of-bounds faults by setting hard memory budgets and redirecting intermediate query scratch buffers to browser-backed storage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded DuckDB-WASM In-Memory Execution Without Memory Limits or Scratch Filesystem Tiering Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded DuckDB-WASM In-Memory Execution Without Memory Limits or Scratch Filesystem Tiering	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure DuckDB runtime parameters with explicit memory ceilings ('SET max_memory = <limit>') calculated as a safe fraction of available browser resources (e.g., maximum 512MB-1024MB).

Engineers MUST ALWAYS configure DuckDB runtime parameters with explicit memory ceilings ('SET max_memory = <limit>') calculated as a safe fraction of available browser resources (e.g., maximum 512MB-1024MB).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.