Zero-Copy Byte-Stream Serialization and Backpressure Guardrails for Bun 1.2 Worker Pools

--- description: Zero-Copy Byte-Stream Serialization and Backpressure Guardrails for Bun 1.2 Worker Pools - Mandates explicit ownership transfers using ArrayBuffer transfer lists and strict backpressure propagation via ReadableStream and TransformStream over native Bun 1.2 multithreaded worker boundaries. Eliminates high-frequency structured clone CPU penalties, buffer duplication spikes, and unbounded heap exhaustion under stream contention. globs: src/**/*.ts, src/workers/**/*.ts, workers/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/bun-zero-copy-worker-stream-backpressure" --- # Zero-Copy Byte-Stream Serialization and Backpressure Guardrails for Bun 1.2 Worker Pools > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded ArrayBuffer Structured Cloning and Lost Backpressure Across Native Worker Boundaries ## System Constraints You are a Principal Engineer enforcing Bun 1.2 Zero-Copy Stream Processing & Multi-Threaded Worker Pools. Mandate explicit memory transfer lists over Web Worker boundaries, reject deep structured cloning of Uint8Array/ArrayBuffer buffers, and enforce pull-based or highWaterMark-controlled stream backpressure at all pipeline stages. ## Strict Guidelines - Engineers MUST ALWAYS pass the underlying `ArrayBuffer` in the transfer list of `Worker.postMessage(msg, [msg.buffer])` to transfer ownership without memory allocation. - Engineers MUST NOT rely on implicit structured cloning for payloads exceeding 64KB when dispatching byte streams across worker threads. - Code MUST verify that transferred ArrayBuffers are detached in the sender context (`buffer.byteLength === 0`) after `postMessage` execution. - Services MUST route chunked payloads through `TransformStream` or `ReadableStream` pipelines using standard highWaterMark limits rather than manual asynchronous emitter queues. - Components NEVER buffer an entire incoming stream using `await response.arrayBuffer()` or `await Bun.readableStreamToArrayBuffer()` before delegating work to worker pools; streams must be piped or chunk-transferred concurrently. ## Bad vs Good Implementations ### Transferring Streamed Chunks to a Worker Without Deep Cloning #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Passing a TypedArray directly in the message payload without a transfer list causes Bun to execute a deep structured clone of the memory page. This duplicates heap allocations and introduces heavy CPU latency across worker thread boundaries. ```typescript import { Worker } from "bun"; const worker = new Worker(new URL("./processor.ts", import.meta.url).href); export async function processPayload(stream: ReadableStream<Uint8Array>) { const reader = stream.getReader(); while (true) { const { done, value } = await reader.read(); if (done) break; // ANTI-PATTERN: Value is copied via deep structured clone. // Doubles memory footprint and blocks event loop on large buffers. worker.postMessage({ type: "CHUNK", data: value }); } } ``` #### Production Standard (Correct) > **Standard**: Transferring the underlying ArrayBuffer via the second argument transfer list relinquishes ownership in the sender context, enabling zero-copy pointer transfers across the Bun worker boundary. ```typescript import { Worker } from "bun"; const worker = new Worker(new URL("./processor.ts", import.meta.url).href); export async function processPayload(stream: ReadableStream<Uint8Array>) { const reader = stream.getReader(); try { while (true) { const { done, value } = await reader.read(); if (done) { worker.postMessage({ type: "EOF" }); break; } // PROD PATTERN: Zero-copy transfer of the underlying ArrayBuffer const buffer = value.buffer.slice( value.byteOffset, value.byteOffset + value.byteLength ); worker.postMessage( { type: "CHUNK", buffer, byteLength: buffer.byteLength }, [buffer] ); } } finally { reader.releaseLock(); } } ``` ### Enforcing Downstream Backpressure in Worker Stream Pipelines #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Failing to await worker readiness or evaluate consumer backpressure allows the source stream to saturate the worker's incoming message queue without bound, causing OOM crashes under high network throughput. ```typescript export function createWorkerTransformStream(worker: Worker): TransformStream<Uint8Array, Uint8Array> { // ANTI-PATTERN: Unbounded push without respecting controller.desiredSize or drain signals return new TransformStream({ transform(chunk) { worker.postMessage(chunk.buffer, [chunk.buffer]); } }); } ``` #### Production Standard (Correct) > **Standard**: Applies an explicit acknowledgment token strategy coupled with a concurrency credit window, pausing stream ingestion when in-flight buffers exceed the specified highWaterMark. ```typescript export function createWorkerTransformStream( worker: Worker, highWaterMark = 16 ): TransformStream<Uint8Array, Uint8Array> { let inFlight = 0; let resumeResolve: (() => void) | null = null; worker.onmessage = (event: MessageEvent) => { if (event.data.type === "ACK") { inFlight--; if (inFlight < highWaterMark && resumeResolve) { const resolve = resumeResolve; resumeResolve = null; resolve(); } } }; return new TransformStream<Uint8Array, Uint8Array>( { async transform(chunk, controller) { if (inFlight >= highWaterMark) { await new Promise<void>((resolve) => { resumeResolve = resolve; }); } const buffer = chunk.buffer; inFlight++; worker.postMessage({ type: "DATA", buffer }, [buffer]); } }, { highWaterMark: 1 }, { highWaterMark } ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Zero-Copy Byte-Stream Serialization and Backpressure Guardrails for Bun 1.2 Worker Pools

Zero-Copy Byte-Stream Serialization and Backpressure Guardrails for Bun 1.2 Worker Pools defines an industrial-grade behavioral standard for backend engineering. Mandates explicit ownership transfers using ArrayBuffer transfer lists and strict backpressure propagation via ReadableStream and TransformStream over native Bun 1.2 multithreaded worker boundaries. Eliminates high-frequency structured clone CPU penalties, buffer duplication spikes, and unbounded heap exhaustion under stream contention. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded ArrayBuffer Structured Cloning and Lost Backpressure Across Native Worker Boundaries Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded ArrayBuffer Structured Cloning and Lost Backpressure Across Native Worker Boundaries	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS pass the underlying `ArrayBuffer` in the transfer list of `Worker.postMessage(msg, [msg.buffer])` to transfer ownership without memory allocation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.