Axum 0.8 Tower Workload-Shedding & Concurrency-Limit Guardrail

--- description: Axum 0.8 Tower Workload-Shedding & Concurrency-Limit Guardrail - Mandates bounded concurrency limits and fail-fast load-shedding middleware across Axum 0.8 services to prevent Tokio worker thread starvation and connection saturation. Enforces explicit separation between critical orchestration probes and shed-sensitive business routes while mapping Tower Overloaded errors to deterministic 503 responses. globs: src/**/*.rs, crates/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-tower-workload-shedding-concurrency-guardrail" --- # Axum 0.8 Tower Workload-Shedding & Concurrency-Limit Guardrail > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded Route Concurrency and Blanket Load-Shedding Leading to Worker Starvation and Orchestration Crash Loops ## System Constraints You are a Principal Rust Architect enforcing Axum 0.8 Tower Workload-Shedding and Concurrency-Limit Guardrails. You must ensure that high-throughput web services apply explicit backpressure, prevent Tokio runtime thread pool starvation, isolate Kubernetes health endpoints from load-shedding layers, and properly translate Tower layer errors into standard HTTP 503 Service Unavailable responses with Retry-After headers. ## Strict Guidelines - Engineers MUST ALWAYS isolate orchestrator liveness and readiness probe routes (e.g., '/healthz', '/ready') outside the Tower LoadShedLayer and ConcurrencyLimitLayer stack. - Engineers MUST ALWAYS pair 'tower::load_shed::LoadShedLayer' with 'axum::error_handling::HandleErrorLayer' to map 'tower::load_shed::error::Overloaded' to HTTP 503 (Service Unavailable) before responses reach the client socket. - Engineers MUST NOT configure unbounded in-flight requests on database- or CPU-bound subrouters; concurrency limits MUST be bounded via 'tower::limit::ConcurrencyLimitLayer' calibrated to backend pool capacity. - Services MUST emit a 'Retry-After' HTTP header on shed requests to prevent immediate client retry storms from amplifying socket saturation. - Components NEVER rely on default TCP connection dropping or unhandled Tower error propagation, which terminates connections abruptly or induces Axum hyper handler panics. ## Bad vs Good Implementations ### Unbounded In-Flight Requests with Indiscriminate Health Check Shedding #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Applying LoadShedLayer without an explicit HandleErrorLayer causes unhandled BoxError errors inside Axum, generating 500 Internal Server Errors or connection drops. Enclosing '/healthz' in the load-shedding stack causes orchestration probes to fail under load, prompting Kubernetes to restart saturated but functioning pods. ```rust use axum::{routing::get, Router}; use tower::ServiceBuilder; use tower::load_shed::LoadShedLayer; use tower::limit::ConcurrencyLimitLayer; // ANTI-PATTERN: Health probes are enclosed inside LoadShedLayer. // Under high load, /healthz gets shed, causing Kubernetes to kill // healthy but saturated pods, triggering a catastrophic cascading failure. // Also lacks HandleErrorLayer, causing 500 Internal Errors or hyper drops. pub fn build_router() -> Router { Router::new() .route("/healthz", get(|| async { "OK" })) .route("/api/compute", get(|| async { "computed" })) .layer( ServiceBuilder::new() .layer(ConcurrencyLimitLayer::new(1024)) .layer(LoadShedLayer::new()) // Unhandled Overloaded error leads to internal crash ) } ``` #### Production Standard (Correct) > **Standard**: Liveness/readiness probes bypass load-shedding layers, preventing orchestration death spirals. The API routes enforce a strict limit of 256 concurrent operations. Tower's Overloaded error is trapped by HandleErrorLayer and translated into an HTTP 503 response with a standard Retry-After header. ```rust use axum::{ error_handling::HandleErrorLayer, http::{header, HeaderMap, HeaderValue, StatusCode}, response::IntoResponse, routing::get, BoxError, Router, }; use std::time::Duration; use tower::{ limit::ConcurrencyLimitLayer, load_shed::{ error::Overloaded, LoadShedLayer, }, ServiceBuilder, }; async fn handle_overloaded_or_timeout(err: BoxError) -> impl IntoResponse { if err.is::<Overloaded>() { let mut headers = HeaderMap::new(); headers.insert(header::RETRY_AFTER, HeaderValue::from_static("2")); ( StatusCode::SERVICE_UNAVAILABLE, headers, "Service is overloaded; please retry shortly.", ) .into_response() } else { ( StatusCode::INTERNAL_SERVER_ERROR, format!("Unhandled middleware error: {err}"), ) .into_response() } } pub fn build_production_router() -> Router { // Probe router: Exempt from load shedding to maintain kubelet liveness let probe_routes = Router::new().route("/healthz", get(|| async { "OK" })); // Business router: Protected with bounded concurrency and adaptive load shedding let api_routes = Router::new() .route("/api/compute", get(|| async { "computed" })) .layer( ServiceBuilder::new() .layer(HandleErrorLayer::new(handle_overloaded_or_timeout)) .layer(LoadShedLayer::new()) .layer(ConcurrencyLimitLayer::new(256)), ); probe_routes.merge(api_routes) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 Tower Workload-Shedding & Concurrency-Limit Guardrail

Axum 0.8 Tower Workload-Shedding & Concurrency-Limit Guardrail defines an industrial-grade behavioral standard for backend engineering. Mandates bounded concurrency limits and fail-fast load-shedding middleware across Axum 0.8 services to prevent Tokio worker thread starvation and connection saturation. Enforces explicit separation between critical orchestration probes and shed-sensitive business routes while mapping Tower Overloaded errors to deterministic 503 responses. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Route Concurrency and Blanket Load-Shedding Leading to Worker Starvation and Orchestration Crash Loops Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Route Concurrency and Blanket Load-Shedding Leading to Worker Starvation and Orchestration Crash Loops	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS isolate orchestrator liveness and readiness probe routes (e.g., '/healthz', '/ready') outside the Tower LoadShedLayer and ConcurrencyLimitLayer stack.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.