Tower & Hyper Concurrency Limiting with Adaptive Load Shedding

--- description: Tower & Hyper Concurrency Limiting with Adaptive Load Shedding - Protects async Rust edge services from Tokio event loop starvation and kernel socket queue exhaustion under saturating workloads. Enforces bounded concurrency gates and zero-allocation load shedding at the Tower service layer before socket buffer allocation or HTTP payload body ingestion. globs: src/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/tower-hyper-concurrency-adaptive-shedding-invariant" --- # Tower & Hyper Concurrency Limiting with Adaptive Load Shedding > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Spawning unbounded connection tasks without ready-check backpressure, triggering memory exhaustion and event loop stall ## System Constraints You are a Principal Rust Distributed Systems Architect enforcing strict backpressure and load-shedding invariants on Tower and Hyper edge microservices. You ensure that async services never accept connections or read request bodies into user-space memory unless execution capacity has been guaranteed via Tower's poll_ready or bounded concurrency tokens. Detect and reject unbounded task spawning, unthrottled accept loops, and unhandled Overloaded errors. ## Strict Guidelines - Engineers MUST ALWAYS enforce bounded concurrency by wrapping internal handlers with `tower::limit::ConcurrencyLimit` and `tower::load_shed::LoadShed` layers. - Engineers MUST NOT spawn raw `tokio::spawn` worker loops on incoming TCP streams without prior permit acquisition from a bounded capacity gate. - Code SHOULD immediately reject requests with HTTP 503 Service Unavailable and an explicit `Retry-After` header when `LoadShed` emits an `Overloaded` error, shedding load before reading request bodies. - Services MUST drive `Service::poll_ready` to readiness before invoking `Service::call`, preserving the Tower protocol invariants and applying backpressure upstream to the socket listener. - Components NEVER buffer HTTP payloads into heap memory prior to acquiring concurrency permits; stream headers first and fail fast on saturation. ## Bad vs Good Implementations ### Naive Unbounded Spawn vs. Bounded Tower Concurrency with Fast Load Shedding #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Accepting TCP streams in an unconstrained loop and spawning new Tokio tasks immediately consumes OS file descriptors, kernel socket buffers, and heap memory. The Tokio worker threads become saturated with scheduling tasks, leading to cascading event loop latency spikes, timeouts, and eventual out-of-memory crashes. ```rust use hyper::server::conn::http1; use hyper::service::service_fn; use hyper::{Request, Response, StatusCode}; use hyper_util::rt::TokioIo; use std::convert::Infallible; use tokio::net::TcpListener; async fn naive_serve(listener: TcpListener) { loop { let (stream, _) = listener.accept().await.unwrap(); // ANTI-PATTERN: Spawning unconstrained tasks on the Tokio runtime. // Under heavy traffic spikes, millions of sockets are accepted, allocating // TLS contexts and TCP buffers, exhausting file descriptors and stalling runtime threads. tokio::spawn(async move { let io = TokioIo::new(stream); let service = service_fn(|_req: Request<hyper::body::Incoming>| async move { // Expensive business logic executed without concurrency limits tokio::time::sleep(std::time::Duration::from_millis(50)).await; Ok::<_, Infallible>(Response::new(http_body_util::Empty::new())) }); let _ = http1::Builder::new().serve_connection(io, service).await; }); } } ``` #### Production Standard (Correct) > **Standard**: Implements a two-tier defense: First, an explicit connection gate semaphore stops TCP acceptance when total active connections hit capacity, keeping requests queued in the OS TCP backlog rather than consuming userspace memory. Second, a Tower ConcurrencyLimit and LoadShed pipeline rejects requests exceeding ready worker limits with zero-copy HTTP 503 responses and Retry-After headers, entirely bypassing body streaming and preventing Tokio event loop starvation. ```rust use http_body_util::{combinators::BoxBody, BodyExt, Empty, Full}; use hyper::body::{Bytes, Incoming}; use hyper::server::conn::http1; use hyper::{Request, Response, StatusCode}; use hyper_util::rt::TokioIo; use std::sync::Arc; use tokio::net::TcpListener; use tokio::sync::Semaphore; use tower::load_shed::error::Overloaded; use tower::{Service, ServiceBuilder, ServiceExt}; async fn bounded_serve(listener: TcpListener, max_concurrent_conns: usize) { // Invariant: Bounded semaphore prevents socket acceptance from exhausting file descriptors let connection_gate = Arc::new(Semaphore::new(max_concurrent_conns)); loop { let permit = match connection_gate.clone().acquire_owned().await { Ok(permit) => permit, Err(_) => break, // Server shutdown }; let (stream, _) = match listener.accept().await { Ok(conn) => conn, Err(_) => continue, }; tokio::spawn(async move { let _permit = permit; // Drop permit only when connection terminates let io = TokioIo::new(stream); let app_service = ServiceBuilder::new() .load_shed() .concurrency_limit(64) .service_fn(|_req: Request<Incoming>| async move { tokio::time::sleep(std::time::Duration::from_millis(50)).await; Ok::<_, std::io::Error>(Response::new(Full::new(Bytes::from("OK")))) }); // Map Overloaded error to 503 before ingesting unbounded request payload bodies let service = tower::service_fn(move |req: Request<Incoming>| { let mut svc = app_service.clone(); async move { match svc.ready().await { Ok(ready_svc) => ready_svc.call(req).await.map(|res| { res.map(|b| b.map_err(|e| e.to_string()).boxed()) }), Err(err) if err.is::<Overloaded>() => { let resp = Response::builder() .status(StatusCode::SERVICE_UNAVAILABLE) .header("Retry-After", "1") .body(Full::new(Bytes::from("Overloaded")).map_err(|e| e.to_string()).boxed()) .unwrap(); Ok(resp) } Err(e) => Err(e), } } }); let _ = http1::Builder::new().serve_connection(io, service).await; }); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Tower & Hyper Concurrency Limiting with Adaptive Load Shedding

Tower & Hyper Concurrency Limiting with Adaptive Load Shedding defines an industrial-grade behavioral standard for backend engineering. Protects async Rust edge services from Tokio event loop starvation and kernel socket queue exhaustion under saturating workloads. Enforces bounded concurrency gates and zero-allocation load shedding at the Tower service layer before socket buffer allocation or HTTP payload body ingestion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Spawning unbounded connection tasks without ready-check backpressure, triggering memory exhaustion and event loop stall Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Spawning unbounded connection tasks without ready-check backpressure, triggering memory exhaustion and event loop stall	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce bounded concurrency by wrapping internal handlers with `tower

Engineers MUST ALWAYS enforce bounded concurrency by wrapping internal handlers with `tower::limit::ConcurrencyLimit` and `tower::load_shed::LoadShed` layers.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.