Axum 0.8 & Hyper Two-Phase Graceful Connection Drain Invariant

--- description: Axum 0.8 & Hyper Two-Phase Graceful Connection Drain Invariant - Mandates a deterministic two-phase connection drain architecture for Axum 0.8 and Hyper servers during rolling and blue-green deployments. Enforces an ingress propagation buffer before stopping the listener and coordinates Tokio cancellation tokens with Hyper HTTP/1.1 and HTTP/2 connection drain states to eliminate dropped TCP packets and HTTP 502 Bad Gateway errors. globs: src/**/*.rs, crates/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-hyper-graceful-connection-drain-axum-0-8-gracefu-205" --- # Axum 0.8 & Hyper Two-Phase Graceful Connection Drain Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Immediate Listener Teardown on SIGTERM Causing Ingress Connection Drops and RST Packet Spikes ## System Constraints You are a Principal Systems Engineer enforcing strict Axum 0.8 and Hyper connection drain invariants. When designing or reviewing Rust HTTP server shutdown paths, you must reject any architecture that immediately terminates the listener or drops connection pools upon receiving OS termination signals (SIGTERM/SIGINT). You must require a two-phase drain protocol: Phase 1 decouples readiness probes and provides an ingress deregistration delay while continuing to accept traffic; Phase 2 invokes Hyper's graceful connection drain with a bounded timeout via Tokio CancellationToken to safely exhaust in-flight multiplexed HTTP/2 streams and HTTP/1.1 keep-alives. ## Strict Guidelines - Services MUST implement a two-phase shutdown sequence: Phase 1 marks readiness probes unhealthy and sleeps for the ingress synchronization window (e.g., 5-15s); Phase 2 executes Hyper's graceful connection drain. - Engineers MUST ALWAYS pass a `tokio_util::sync::CancellationToken` or dedicated watch channel into `axum::serve(...).with_graceful_shutdown(...)` rather than binding directly to a bare `tokio::signal::ctrl_c()` future. - Engineers MUST NOT allow active Tokio runtimes to drop while in-flight Hyper connections are completing their graceful drain; the outer runtime loop MUST await the server future to its completion. - Services MUST enforce a bounded hard-stop timeout (via `tokio::time::timeout`) around the graceful drain phase to prevent leaked connections or stuck long-polling requests from blocking container termination indefinitely. - Components NEVER immediately abort TCP sockets or drop background worker dependencies (such as database connection pools or tracer flushes) before the `axum::serve` future has completely resolved. ## Bad vs Good Implementations ### Naive Axum Server Termination vs Two-Phase Ingress-Aware Connection Drain #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly binding graceful shutdown to OS signals stops accepting incoming connections before the cloud load balancer or ingress controller (e.g., Envoy, Nginx, AWS ALB) has removed the pod from endpoints. Requests in flight or routed during the propagation window fail immediately with connection refused or reset errors. ```rust use axum::{routing::get, Router}; use tokio::net::TcpListener; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let app = Router::new().route("/healthz", get(|| async { "OK" })); let listener = TcpListener::bind("0.0.0.0:8080").await?; // ANTI-PATTERN: On SIGTERM, listener stops immediately. // In-flight HTTP/2 streams receive RST, and Kubernetes ingress // continues forwarding requests for several seconds, causing client 502s. axum::serve(listener, app) .with_graceful_shutdown(shutdown_signal()) .await?; Ok(()) } async fn shutdown_signal() { tokio::signal::ctrl_c().await.expect("failed to install CTRL+C handler"); } ``` #### Production Standard (Correct) > **Standard**: Implements the two-phase drain invariant: upon receiving SIGTERM, the server flips its readiness flag and waits for 10 seconds, allowing upstream ingress controllers to reconcile endpoints and stop dispatching traffic. Then, the `CancellationToken` triggers Hyper's graceful drain under a 30-second bounded timeout, ensuring existing HTTP/1 and HTTP/2 streams complete without RST flags. ```rust use axum::{extract::State, http::StatusCode, routing::get, Router}; use std::{sync::{atomic::{AtomicBool, Ordering}, Arc}, time::Duration}; use tokio::{net::TcpListener, signal, time::sleep}; use tokio_util::sync::CancellationToken; #[derive(Clone)] struct AppState { is_shutting_down: Arc<AtomicBool>, } async fn readiness_handler(State(state): State<AppState>) -> (StatusCode, &'static str) { if state.is_shutting_down.load(Ordering::Relaxed) { (StatusCode::SERVICE_UNAVAILABLE, "Draining") } else { (StatusCode::OK, "Healthy") } } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let state = AppState { is_shutting_down: Arc::new(AtomicBool::new(false)) }; let app = Router::new() .route("/ready", get(readiness_handler)) .with_state(state.clone()); let listener = TcpListener::bind("0.0.0.0:8080").await?; let drain_token = CancellationToken::new(); let server_drain_token = drain_token.clone(); let shutdown_coordinator = tokio::spawn(async move { wait_for_termination_signal().await; // Phase 1: Mark unhealthy so K8s readiness probe fails & wait for ingress mesh updates state.is_shutting_down.store(true, Ordering::SeqCst); tracing::info!("Readiness marked false; awaiting ingress endpoint propagation..."); sleep(Duration::from_secs(10)).await; // Phase 2: Signal Axum/Hyper to stop accepting new conns and drain active streams drain_token.cancel(); }); let server = axum::serve(listener, app) .with_graceful_shutdown(async move { server_drain_token.cancelled().await; tracing::info!("Beginning Hyper connection drain protocol"); }); // Bound the server drain phase to prevent indefinite stalls tokio::select! { res = server => res?, _ = sleep(Duration::from_secs(30)) => { tracing::warn!("Drain timeout exceeded; forcing shutdown"); } } let _ = shutdown_coordinator.await; Ok(()) } async fn wait_for_termination_signal() { let ctrl_c = async { signal::ctrl_c().await.expect("failed to listen for ctrl_c"); }; #[cfg(unix)] let sigterm = async { signal::unix::signal(signal::unix::SignalKind::terminate()) .expect("failed to install SIGTERM handler") .recv() .await; }; #[cfg(not(unix))] let sigterm = std::future::pending::<()>(); tokio::select! { _ = ctrl_c => {}, _ = sigterm => {}, } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 & Hyper Two-Phase Graceful Connection Drain Invariant

Axum 0.8 & Hyper Two-Phase Graceful Connection Drain Invariant defines an industrial-grade behavioral standard for backend engineering. Mandates a deterministic two-phase connection drain architecture for Axum 0.8 and Hyper servers during rolling and blue-green deployments. Enforces an ingress propagation buffer before stopping the listener and coordinates Tokio cancellation tokens with Hyper HTTP/1.1 and HTTP/2 connection drain states to eliminate dropped TCP packets and HTTP 502 Bad Gateway errors. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Immediate Listener Teardown on SIGTERM Causing Ingress Connection Drops and RST Packet Spikes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Immediate Listener Teardown on SIGTERM Causing Ingress Connection Drops and RST Packet Spikes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST implement a two-phase shutdown sequence

Services MUST implement a two-phase shutdown sequence: Phase 1 marks readiness probes unhealthy and sleeps for the ingress synchronization window (e.g., 5-15s); Phase 2 executes Hyper's graceful connection drain.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.