Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded io.ReadAll and Heap-Escaping Buffer Allocation Under Webhook Bursts) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi

--- description: Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi - Enforces memory-bounded chunked stream consumption using sync.Pool byte slice recycling, http.MaxBytesReader payload caps, and buffered concurrency tokens on Go Chi routes. Eliminates heap allocations and GC thrashing during heavy webhook burst traffic by processing payloads in fixed-size reusable buffers while shedding load with HTTP 429/503 when worker capacity saturates. globs: cmd/**/*.go, internal/**/*.go, pkg/**/*.go, **/*handler*.go, **/*router*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/go-chi-zero-allocation-stream-ingestion" --- # Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded io.ReadAll and Heap-Escaping Buffer Allocation Under Webhook Bursts ## System Constraints You are a Principal Go Systems Architect enforcing zero-allocation chunked stream ingestion pipelines and strict backpressure clamps on Go Chi HTTP services. Mandate the use of sync.Pool buffer cycling, http.MaxBytesReader boundaries, and non-blocking backpressure semaphores to maintain a flat memory profile under multi-gigabit webhook bursts. ## Strict Guidelines - Engineers MUST NEVER invoke io.ReadAll, ioutil.ReadAll, or bytes.Buffer without a predefined capacity bound on incoming request bodies. - Engineers MUST ALWAYS wrap incoming r.Body instances with http.MaxBytesReader before any read operation to reject payload inflation attacks at the wire level. - Handlers MUST cycle fixed-size scratch buffers (e.g., 32KB or 64KB) via sync.Pool and return them immediately using defer blocks to avoid heap allocation spikes. - Services MUST gate concurrent streaming ingestion using a capacity-bounded channel semaphore, returning HTTP 429 (Too Many Requests) or HTTP 503 with a Retry-After header when saturated. - Components MUST stream chunked data directly into downstream sinks (e.g., hashers, disk, object store) using io.CopyBuffer rather than accumulating intermediate byte slices in memory. - Engineers MUST verify that byte slices retrieved from sync.Pool are zeroed or resliced to [:0] and strictly never retained beyond the scope of the HTTP handler execution. ## Bad vs Good Implementations ### Go Chi Webhook Stream Ingestion Pipeline #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Using io.ReadAll buffers the entire payload onto the Go runtime heap without length constraints. During concurrent webhook spikes, this triggers unconstrained heap allocations, heavy GC stop-the-world pauses, and fatal OOM container terminations. ```go package webhook import ( "io" "net/http" "github.com/go-chi/chi/v5" ) func MountRoutes(r chi.Router) { r.Post("/webhooks/ingest", func(w http.ResponseWriter, r *http.Request) { // Flaw: Unbounded memory allocation; causes massive GC spikes // and heap-exhaustion OOM crashes under high-throughput bursts. data, err := io.ReadAll(r.Body) if err != nil { http.Error(w, "failed to read payload", http.StatusInternalServerError) return } defer r.Body.Close() processPayload(data) w.WriteHeader(http.StatusAccepted) }) } func processPayload(b []byte) { /* processes payload */ } ``` #### Production Standard (Correct) > **Standard**: Wraps r.Body with http.MaxBytesReader to enforce a strict memory boundary, clamps concurrent execution via a channel semaphore with immediate 429 shedding, and reuses 32KB scratch buffers via sync.Pool with io.CopyBuffer, ensuring zero dynamic heap allocations per chunk. ```go package webhook import ( "crypto/sha256" "errors" "io" "net/http" "sync" "github.com/go-chi/chi/v5" ) const ( MaxPayloadBytes = 10 * 1024 * 1024 // 10MB hard ceiling ChunkBufferSize = 32 * 1024 // 32KB chunk MaxConcurrency = 256 ) var ( bufferPool = sync.Pool{ New: func() any { b := make([]byte, ChunkBufferSize) return &b }, } sem = make(chan struct{}, MaxConcurrency) ) func MountRoutes(r chi.Router) { r.Post("/webhooks/ingest", func(w http.ResponseWriter, r *http.Request) { // Non-blocking backpressure clamp select { case sem <- struct{}{}: defer func() { <-sem }() default: w.Header().Set("Retry-After", "5") http.Error(w, "Ingestion pipeline saturated", http.StatusTooManyRequests) return } // Clamp stream bounds to reject payload inflation r.Body = http.MaxBytesReader(w, r.Body, MaxPayloadBytes) defer r.Body.Close() bufPtr := bufferPool.Get().(*[]byte) defer bufferPool.Put(bufPtr) hasher := sha256.New() written, err := io.CopyBuffer(hasher, r.Body, *bufPtr) if err != nil { var maxBytesErr *http.MaxBytesError if errors.As(err, &maxBytesErr) { http.Error(w, "Payload exceeds limit", http.StatusRequestEntityTooLarge) return } http.Error(w, "Stream processing failure", http.StatusBadRequest) return } if written == 0 { http.Error(w, "Empty payload", http.StatusBadRequest) return } w.WriteHeader(http.StatusAccepted) }) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi

Zero-Allocation Chunked Stream Ingestion & Backpressure Clamp for Go Chi defines an industrial-grade behavioral standard for backend engineering. Enforces memory-bounded chunked stream consumption using sync.Pool byte slice recycling, http.MaxBytesReader payload caps, and buffered concurrency tokens on Go Chi routes. Eliminates heap allocations and GC thrashing during heavy webhook burst traffic by processing payloads in fixed-size reusable buffers while shedding load with HTTP 429/503 when worker capacity saturates. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded io.ReadAll and Heap-Escaping Buffer Allocation Under Webhook Bursts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded io.ReadAll and Heap-Escaping Buffer Allocation Under Webhook Bursts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NEVER invoke io.ReadAll, ioutil.ReadAll, or bytes.Buffer without a predefined capacity bound on incoming request bodies.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.