Axum 0.8 Tower Service Graceful Connection Draining Invariant

--- description: Axum 0.8 Tower Service Graceful Connection Draining Invariant - Prevents dropped in-flight streaming requests, truncated SSE/WebSocket responses, and connection resets (ECONNRESET) during Kubernetes rolling upgrades. Enforces coordination between POSIX termination signals (SIGINT/SIGTERM), CancellationToken cascades across Tower middleware, and hyper-util connection graceful draining timeouts before binding closure. globs: src/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-0-8-tower-graceful-connection-draining" --- # Axum 0.8 Tower Service Graceful Connection Draining Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Naive axum::serve with_graceful_shutdown Drops In-Flight Hyper Streams and WebSocket Sessions on SIGTERM ## System Constraints You are a Principal Rust Systems Architect specializing in production Axum 0.8 and Hyper 1.x network servers. You enforce strict lifecycle invariants for zero-downtime rolling upgrades in containerized environments (Kubernetes). You mandate explicit signal handling with tokio::signal, layered Toko CancellationToken propagation to long-lived Tower streams and SSE handlers, and hard-bounded Hyper connection graceful draining timeouts to eliminate premature socket resets (RST packets) and truncated HTTP payloads. ## Strict Guidelines - Engineers MUST ALWAYS listen for both SIGINT and SIGTERM explicitly on Unix platforms, rather than relying exclusively on tokio::signal::ctrl_c() which fails to capture Kubernetes preStop / termination signals. - Engineers MUST NOT rely solely on axum::serve(...).with_graceful_shutdown(...) when handling Server-Sent Events (SSE), WebSockets, or unbounded request/response streaming bodies, as with_graceful_shutdown only stops accepting new connections and signals hyper to drain, which hangs indefinitely unless stream loops select on a cancellation token. - Services MUST propagate a tokio_util::sync::CancellationToken or watch::Receiver to all long-lived streaming Tower services and Axum routes via Router::with_state or Tower Extensions. - Engineers MUST configure a post-shutdown drain deadline or timeout using tokio::time::timeout wrapping the server future or hyper's connection drain future to prevent Pod termination deadlocks that trigger SIGKILL. - Components NEVER bind socket listeners without an explicit TCP SO_REUSEADDR option where zero-downtime hot-restarts or rapid process respawns are required. ## Bad vs Good Implementations ### Naive ctrl_c termination with uncoordinated streaming handlers #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: This implementation fails in two ways during Kubernetes rolling deployments: first, tokio::signal::ctrl_c() does not monitor SIGTERM, which Kubernetes sends to initiate pod termination. Second, infinite streaming handlers (SSE, WebSockets) possess no termination awareness; hyper will wait indefinitely for active connections to self-terminate, causing Kubernetes to reach its terminationGracePeriodSeconds and emit SIGKILL, abruptly dropping all in-flight connections with TCP resets. ```rust use axum::{routing::get, Router}; use tokio_stream::wrappers::IntervalStream; use axum::response::sse::{Event, Sse}; use std::time::Duration; use std::convert::Infallible; async fn sse_handler() -> Sse<impl tokio_stream::Stream<Item = Result<Event, Infallible>>> { // ANTI-PATTERN: Infinite stream has no shutdown hook or token. // During SIGTERM, axum::serve waits indefinitely for this stream to finish, // eventually forcing Kubernetes to issue a hard SIGKILL and dropping client connections. let stream = tokio_stream::wrappers::IntervalStream::new(tokio::time::interval(Duration::from_secs(1))) .map(|_| Ok(Event::default().data("tick"))); Sse::new(stream) } #[tokio::main] async fn main() { let app = Router::new().route("/events", get(sse_handler)); let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap(); // ANTI-PATTERN: ctrl_c does not capture SIGTERM in K8s, and axum::serve has no drain timeout. axum::serve(listener, app) .with_graceful_shutdown(async { tokio::signal::ctrl_c().await.ok(); }) .await .unwrap(); } ``` #### Production Standard (Correct) > **Standard**: Listens explicitly for both SIGTERM and SIGINT. Propagates a CancellationToken to streaming endpoints using tokio_stream::StreamExt::take_until, ensuring streaming loops break immediately when shutdown initiates. Finally, bounds the entire axum::serve future with tokio::time::timeout to guarantee complete socket drainage before Kubernetes issues an uncoordinated SIGKILL. ```rust use axum::{extract::State, routing::get, response::sse::{Event, Sse}, Router}; use tokio_util::sync::CancellationToken; use tokio_stream::{Stream, StreamExt}; use std::{convert::Infallible, sync::Arc, time::Duration}; #[derive(Clone)] struct AppState { shutdown_token: CancellationToken, } async fn sse_handler( State(state): State<Arc<AppState>>, ) -> Sse<impl Stream<Item = Result<Event, Infallible>>> { let token = state.shutdown_token.clone(); let interval = tokio_stream::wrappers::IntervalStream::new(tokio::time::interval(Duration::from_secs(1))); // Coordinated stream: terminates cleanly when shutdown_token is triggered let stream = interval .take_until(async move { token.cancelled().await }) .map(|_| Ok(Event::default().data("heartbeat"))); Sse::new(stream) } async fn shutdown_signal(token: CancellationToken) { let ctrl_c = async { tokio::signal::ctrl_c().await.expect("failed to install Ctrl+C handler"); }; #[cfg(unix)] let terminate = async { tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) .expect("failed to install SIGTERM handler") .recv() .await; }; #[cfg(not(unix))] let terminate = std::future::pending::<()>(); tokio::select! { _ = ctrl_c => {}, _ = terminate => {}, } // Signal all long-lived streams/tasks to stop producing token.cancel(); } #[tokio::main] async fn main() { let shutdown_token = CancellationToken::new(); let state = Arc::new(AppState { shutdown_token: shutdown_token.clone() }); let app = Router::new().route("/events", get(sse_handler)).with_state(state); let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap(); let server = axum::serve(listener, app) .with_graceful_shutdown(shutdown_signal(shutdown_token.clone())); // Bounded draining guard: enforce Hyper shutdown finishes within 25s (before standard 30s K8s SIGKILL) if let Err(_) = tokio::time::timeout(Duration::from_secs(25), server).await { eprintln!("Server drain deadline reached; forcing listener termination"); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 Tower Service Graceful Connection Draining Invariant

Axum 0.8 Tower Service Graceful Connection Draining Invariant defines an industrial-grade behavioral standard for backend engineering. Prevents dropped in-flight streaming requests, truncated SSE/WebSocket responses, and connection resets (ECONNRESET) during Kubernetes rolling upgrades. Enforces coordination between POSIX termination signals (SIGINT/SIGTERM), CancellationToken cascades across Tower middleware, and hyper-util connection graceful draining timeouts before binding closure. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Naive axum::serve with_graceful_shutdown Drops In-Flight Hyper Streams and WebSocket Sessions on SIGTERM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Naive axum::serve with_graceful_shutdown Drops In-Flight Hyper Streams and WebSocket Sessions on SIGTERM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS listen for both SIGINT and SIGTERM explicitly on Unix platforms, rather than relying exclusively on tokio

Engineers MUST ALWAYS listen for both SIGINT and SIGTERM explicitly on Unix platforms, rather than relying exclusively on tokio::signal::ctrl_c() which fails to capture Kubernetes preStop / termination signals.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.