Axum 0.8 Tower Concurrency Limit and Ingress Load-Shedding Invariant

--- description: Axum 0.8 Tower Concurrency Limit and Ingress Load-Shedding Invariant - Eliminates distributed thread starvation and Tokio executor deadlocks by strictly bounding middleware queue latency and ingress worker concurrency. Mandates deterministic load shedding with explicit HTTP 503 error handling to fail fast under cascade failure scenarios. globs: src/**/*.rs, Cargo.toml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-tower-concurrency-load-shedding-invariant" --- # Axum 0.8 Tower Concurrency Limit and Ingress Load-Shedding Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded Task Buffering and Raw Concurrency Limits Causing Tokio Executor Deadlocks ## System Constraints You are a Principal Software Architect enforcing Axum 0.8 and Tower concurrency-limit, buffer-bounding, and load-shedding invariants. Mandate that every concurrency bottleneck is protected by fail-fast load shedding, strict timeout bounding, and explicit error translation before saturating the Tokio multi-threaded runtime. ## Strict Guidelines - Engineers MUST ALWAYS compose `ConcurrencyLimitLayer` with an immediately adjacent `LoadShedLayer` or `HandleErrorLayer` to reject saturated requests with HTTP 503 Service Unavailable instead of queuing indefinitely. - Engineers MUST NOT introduce unbounded `tower::buffer::BufferLayer` instances or unbounded channels ahead of concurrency barriers, as they hide queue latency and induce OOM crashes under overload. - Code MUST apply load-shedding and concurrency limiting at the outermost boundary of the Axum router before heavy request body buffering, state extraction, or TLS termination tasks consume Tokio budget. - Services MUST enforce strict request timeouts via `tower_http::timeout::TimeoutLayer` across all concurrency-bounded routes to guarantee that stalled downstream handlers release concurrency permits. - Components NEVER allow dynamic `tower::BoxError` failures produced by Tower middleware layers to reach hyper/axum unhandled, which abruptly drops HTTP/2 streams and obscures client diagnostics. ## Bad vs Good Implementations ### Ingress Concurrency Limiting and Fail-Fast Load Shedding in Axum 0.8 #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Applying ConcurrencyLimitLayer without LoadShedLayer and HandleErrorLayer causes incoming requests exceeding 128 in-flight tasks to block the underlying transport layer polling. Under sustained burst traffic, clients hang until connection timeouts trigger, saturating Tokio worker task queues and starving internal runtime threads. ```rust use axum::{routing::get, Router}; use tower::limit::ConcurrencyLimitLayer; // ANTI-PATTERN: Naked ConcurrencyLimitLayer without LoadShedLayer or HandleErrorLayer. // Excess requests will stall in memory waiting for poll_ready, exhausting file descriptors // and causing Tokio task starvation without failing fast. pub fn build_app() -> Router { Router::new() .route("/api/resource", get(|| async { "ok" })) .layer(ConcurrencyLimitLayer::new(128)) } ``` #### Production Standard (Correct) > **Standard**: By sequencing HandleErrorLayer outside of TimeoutLayer, LoadShedLayer, and ConcurrencyLimitLayer, requests exceeding 128 concurrent slots trigger an immediate Overloaded error without blocking. The error is converted deterministically to an HTTP 503 response, preserving Tokio thread pools and preventing cascade failures. ```rust use axum::{ http::StatusCode, response::IntoResponse, routing::get, BoxError, Router, }; use std::time::Duration; use tower::{ load_shed::LoadShedLayer, limit::ConcurrencyLimitLayer, timeout::TimeoutLayer, ServiceBuilder, }; async fn handle_service_error(err: BoxError) -> impl IntoResponse { if err.is::<tower::load_shed::error::Overloaded>() { (StatusCode::SERVICE_UNAVAILABLE, "Server overloaded: request shed.") } else if err.is::<tower::timeout::error::Elapsed>() { (StatusCode::GATEWAY_TIMEOUT, "Downstream processing timed out.") } else { (StatusCode::INTERNAL_SERVER_ERROR, "Unhandled middleware error.") } } pub fn build_app() -> Router { let middleware_stack = ServiceBuilder::new() .layer(tower::layer::layer_fn(|service| { tower::util::BoxCloneService::new(service) })) .layer(axum::error_handling::HandleErrorLayer::new(handle_service_error)) .layer(TimeoutLayer::new(Duration::from_secs(5))) .layer(LoadShedLayer::new()) .layer(ConcurrencyLimitLayer::new(128)); Router::new() .route("/api/resource", get(|| async { "ok" })) .layer(middleware_stack) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 Tower Concurrency Limit and Ingress Load-Shedding Invariant

Axum 0.8 Tower Concurrency Limit and Ingress Load-Shedding Invariant defines an industrial-grade behavioral standard for backend engineering. Eliminates distributed thread starvation and Tokio executor deadlocks by strictly bounding middleware queue latency and ingress worker concurrency. Mandates deterministic load shedding with explicit HTTP 503 error handling to fail fast under cascade failure scenarios. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Task Buffering and Raw Concurrency Limits Causing Tokio Executor Deadlocks Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Task Buffering and Raw Concurrency Limits Causing Tokio Executor Deadlocks	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS compose `ConcurrencyLimitLayer` with an immediately adjacent `LoadShedLayer` or `HandleErrorLayer` to reject saturated requests with HTTP 503 Service Unavailable instead of queuing indefinitely.

Engineers MUST ALWAYS compose `ConcurrencyLimitLayer` with an immediately adjacent `LoadShedLayer` or `HandleErrorLayer` to reject saturated requests with HTTP 503 Service Unavailable instead of queuing indefinitely.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.