Bun 1.2 Embedded SQLite WAL Concurrency & S3 Zero-Copy Streaming Invariants

--- description: Bun 1.2 Embedded SQLite WAL Concurrency & S3 Zero-Copy Streaming Invariants - Enforces non-blocking embedded database and object storage patterns in Bun 1.2+. Mandates Write-Ahead Logging (WAL) with strict busy timeouts and prepared statements for 'bun:sqlite', alongside zero-allocation ReadableStream piping for Bun S3 native clients to prevent event-loop stalls and out-of-memory heap panics. globs: src/**/*.ts, server/**/*.ts, lib/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/bun-1-2-sqlite-wal-s3-streaming-standard" --- # Bun 1.2 Embedded SQLite WAL Concurrency & S3 Zero-Copy Streaming Invariants > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Buffering S3 payloads entirely in Node.js memory buffers and executing ad-hoc un-journaled bun:sqlite queries without WAL pragma locks. ## System Constraints You are a Principal Backend Architect specializing in Bun 1.2 high-throughput systems. Enforce strict concurrency invariants for bun:sqlite and Bun.s3 native storage. Mandate explicit WAL initialization, busy timeout configurations, reusable prepared statement zero-copy queries, and chunked ReadableStream backpressure pipes for S3 multipart upload and download lifecycles. ## Strict Guidelines - Engineers MUST ALWAYS set 'PRAGMA journal_mode = WAL;' and 'PRAGMA busy_timeout = 5000;' immediately upon instantiating a 'bun:sqlite' Database instance. - Engineers MUST instantiate SQLite prepared statements once at module or class scope using 'db.query()' and reuse them across requests; NEVER execute unparameterized ad-hoc SQL strings dynamically. - Engineers MUST use 'query.values()' or 'query.as(Type)' to leverage Bun's fast zero-copy C++ binding layer whenever reading tabular or structured rows. - Services MUST stream S3 uploads and downloads directly using Bun S3 Native API ('s3.file(...).stream()' or 's3.write(...)') without buffering full payloads into an ArrayBuffer or Buffer in JS memory. - Engineers MUST NOT run write transactions inside long-lived asynchronous closures that await network I/O or S3 streaming; keep database transaction boundaries strictly CPU-bound and synchronous. - Database instances MUST be treated as singletons per process or worker thread; NEVER instantiate a new Database instance per incoming HTTP request. ## Bad vs Good Implementations ### SQLite WAL Setup and S3 Native Streaming Pipeline #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Re-instantiates the SQLite database on each request without WAL mode or busy timeout pragmas, causing immediate lock contention under concurrent load. Buffers the complete payload into V8/JSC heap memory before uploading to S3, bypassing Bun 1.2 native streaming and causing out-of-memory crashes on large files. ```typescript import { Database } from "bun:sqlite"; import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3"; // ANTI-PATTERN: Database recreated per request, no WAL, blocking memory buffering export async function handleUpload(req: Request) { const db = new Database("app.db"); // Instantiated per request // Memory leak: buffers entire multi-GB file into RAM const blob = await req.arrayBuffer(); const s3 = new S3Client({ region: "us-east-1" }); await s3.send(new PutObjectCommand({ Bucket: "data-bucket", Key: "uploads/file.bin", Body: Buffer.from(blob) })); // Ad-hoc query without WAL or busy timeout; prone to SQLITE_BUSY crashes db.run(`INSERT INTO metadata (size, created_at) VALUES (${blob.byteLength}, ${Date.now()})`); return new Response("Uploaded"); } ``` #### Production Standard (Correct) > **Standard**: Initializes a singleton SQLite connection with WAL mode and busy timeout to guarantee safe concurrent reads/writes. Utilizes a pre-compiled parameterized query statement and streams incoming HTTP request chunks directly into Bun's native S3 client without buffering byte data into heap memory. ```typescript import { Database } from "bun:sqlite"; import { s3 } from "bun"; // PRODUCTION INVARIANT: Singleton DB initialized with WAL, busy_timeout, and synchronous pragmas const db = new Database("app.db", { create: true, strict: true }); db.run("PRAGMA journal_mode = WAL;"); db.run("PRAGMA busy_timeout = 5000;"); db.run("PRAGMA synchronous = NORMAL;"); // Pre-compiled prepared statement for zero-copy execution const insertMetadataStmt = db.query< { id: number }, [key: string, size: number, uploadedAt: number] >("INSERT INTO metadata (key, size, uploaded_at) VALUES (?, ?, ?) RETURNING id;"); const bucket = s3(); export async function handleUpload(req: Request): Promise<Response> { if (!req.body) { return new Response("Missing body", { status: 400 }); } const key = `uploads/${crypto.randomUUID()}.bin`; const destination = bucket.file(key); // Zero-copy stream directly from the incoming HTTP stream to S3 const bytesWritten = await destination.write(req.body); // Atomic, synchronous DB insert without awaiting across lock boundaries const result = insertMetadataStmt.get(key, bytesWritten, Date.now()); return Response.json({ success: true, fileId: result?.id, bytesWritten }, { status: 201 }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Bun 1.2 Embedded SQLite WAL Concurrency & S3 Zero-Copy Streaming Invariants

Bun 1.2 Embedded SQLite WAL Concurrency & S3 Zero-Copy Streaming Invariants defines an industrial-grade behavioral standard for backend engineering. Enforces non-blocking embedded database and object storage patterns in Bun 1.2+. Mandates Write-Ahead Logging (WAL) with strict busy timeouts and prepared statements for 'bun:sqlite', alongside zero-allocation ReadableStream piping for Bun S3 native clients to prevent event-loop stalls and out-of-memory heap panics. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Buffering S3 payloads entirely in Node.js memory buffers and executing ad-hoc un-journaled bun:sqlite queries without WAL pragma locks. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Buffering S3 payloads entirely in Node.js memory buffers and executing ad-hoc un-journaled bun:sqlite queries without WAL pragma locks.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS set 'PRAGMA journal_mode = WAL

Engineers MUST ALWAYS set 'PRAGMA journal_mode = WAL;' and 'PRAGMA busy_timeout = 5000;' immediately upon instantiating a 'bun:sqlite' Database instance.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.