Axum 0.8 Multi-Phase Graceful Drain & Rolling Deployment Invariant

--- description: Axum 0.8 Multi-Phase Graceful Drain & Rolling Deployment Invariant - Enforces zero-downtime rolling deployments in Axum 0.8 by mandating a coordinated multi-phase drain lifecycle: load-balancer deregistration quiescence, bounded HTTP/1.1 and HTTP/2 stream completion, and ordered downstream connection pool termination. globs: src/**/*.rs, Cargo.toml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-08-multi-phase-graceful-drain" --- # Axum 0.8 Multi-Phase Graceful Drain & Rolling Deployment Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Immediate SIGTERM Termination Dropping Inflight Requests and Leaking Multiplexed HTTP/2 Streams ## System Constraints You are a Principal Rust Architect enforcing the Axum 0.8 Multi-Phase Graceful Drain Invariant. Axum 0.8 microservices deployed in orchestrated environments (Kubernetes, ECS) must never drop inflight HTTP/1.1 requests or HTTP/2 multiplexed streams during rolling restarts. You must enforce that: (1) Readiness probes fail immediately upon SIGTERM while the server continues serving existing and late-arriving traffic for an ingress propagation delay, (2) `axum::serve` graceful drain is bounded by a hard timeout to prevent hung streams from exceeding orchestration termination grace periods, and (3) Downstream dependencies (SQL connection pools, async task trackers, Tokio cancellation tokens) are drained strictly after the HTTP server drain completes, never prematurely or concurrently. ## Strict Guidelines - Engineers MUST ALWAYS implement a multi-phase shutdown sequence: Phase 1: mark readiness probe false and sleep for ingress sync delay; Phase 2: signal Axum server shutdown with a bounded timeout; Phase 3: drain background workers and close connection pools. - Engineers MUST NOT bind `axum::serve(...).with_graceful_shutdown(...)` directly to raw OS signals without an ingress quiescence delay, as upstream load balancers take seconds to propagate endpoint removals. - Code MUST wrap the Axum graceful drain future in `tokio::time::timeout` to prevent non-compliant HTTP/2 clients or zombie TCP connections from blocking process termination beyond Kubernetes `terminationGracePeriodSeconds`. - Services MUST keep database connection pools (e.g., `sqlx::Pool`) and gRPC clients alive and operational until the Axum server future has fully completed its drain. - Components NEVER abort Tokio `CancellationToken` instances governing inflight request processing before `axum::serve` successfully returns. ## Bad vs Good Implementations ### Naive Graceful Shutdown vs Multi-Phase Orchestrated Drain #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly triggering shutdown on SIGTERM causes race conditions: upstream load balancers route traffic to the terminating pod for several seconds, resulting in HTTP 502s. Inflight requests may also hang indefinitely until the container runtime issues a SIGKILL. ```rust use axum::{routing::get, Router}; use tokio::net::TcpListener; #[tokio::main] async fn main() { let app = Router::new().route("/work", get(|| async { "done" })); let listener = TcpListener::bind("0.0.0.0:3000").await.unwrap(); // ANTI-PATTERN: Binds directly to ctrl_c/SIGTERM without quiescence delay. // In Kubernetes, kube-proxy / ingress controllers take 2-10s to update iptables. // Requests routed during this window will fail with connection refused or reset. // Furthermore, there is no hard timeout; hung HTTP/2 streams cause SIGKILL. axum::serve(listener, app) .with_graceful_shutdown(async { tokio::signal::ctrl_c().await.unwrap(); }) .await .unwrap(); // Inflight requests requiring downstream resources are abruptly terminated here } ``` #### Production Standard (Correct) > **Standard**: Decouples shutdown into an explicit 3-phase lifecycle: (1) Ingress quiescence via readiness probe invalidation + sleep, (2) Bounded Axum server drain with a hard fallback deadline, (3) Ordered teardown of downstream database connection pools and tokens after all HTTP/2 streams finish. ```rust use axum::{routing::get, Router}; use std::{sync::{atomic::{AtomicBool, Ordering}, Arc}, time::Duration}; use tokio::{net::TcpListener, signal::unix::{signal, SignalKind}, sync::watch, time::sleep}; use tokio_util::sync::CancellationToken; struct AppState { is_ready: AtomicBool, cancel_token: CancellationToken, } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let state = Arc::new(AppState { is_ready: AtomicBool::new(true), cancel_token: CancellationToken::new(), }); let app = Router::new() .route("/healthz/ready", get(|s: axum::extract::State<Arc<AppState>>| async move { if s.is_ready.load(Ordering::Acquire) { axum::http::StatusCode::OK } else { axum::http::StatusCode::SERVICE_UNAVAILABLE } })) .route("/api/v1/work", get(|| async { "done" })) .with_state(state.clone()); let listener = TcpListener::bind("0.0.0.0:3000").await?; let (drain_tx, drain_rx) = watch::channel(()); // Spawn signal coordinator let state_clone = state.clone(); tokio::spawn(async move { let mut sigterm = signal(SignalKind::terminate()).expect("failed to install SIGTERM handler"); sigterm.recv().await; // Phase 1: Fail readiness probe and wait for load-balancer propagation (e.g., 5s) state_clone.is_ready.store(false, Ordering::Release); sleep(Duration::from_secs(5)).await; // Phase 2: Signal Axum to stop accepting new connections and drain inflight let _ = drain_tx.send(()); }); // Run server with hard drain ceiling (e.g., 20s) to guarantee exit before SIGKILL let server = axum::serve(listener, app) .with_graceful_shutdown(async move { let mut rx = drain_rx; let _ = rx.changed().await; }); tokio::select! { res = server => res?, _ = sleep(Duration::from_secs(25)) => { eprintln!("Graceful drain deadline reached; forcing termination"); } } // Phase 3: Cancel background tasks and drain DB pools strictly after HTTP drain state.cancel_token.cancel(); // db_pool.close().await; Ok(()) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 Multi-Phase Graceful Drain & Rolling Deployment Invariant

Axum 0.8 Multi-Phase Graceful Drain & Rolling Deployment Invariant defines an industrial-grade behavioral standard for backend engineering. Enforces zero-downtime rolling deployments in Axum 0.8 by mandating a coordinated multi-phase drain lifecycle: load-balancer deregistration quiescence, bounded HTTP/1.1 and HTTP/2 stream completion, and ordered downstream connection pool termination. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Immediate SIGTERM Termination Dropping Inflight Requests and Leaking Multiplexed HTTP/2 Streams Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Immediate SIGTERM Termination Dropping Inflight Requests and Leaking Multiplexed HTTP/2 Streams	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement a multi-phase shutdown sequence

Engineers MUST ALWAYS implement a multi-phase shutdown sequence: Phase 1: mark readiness probe false and sleep for ingress sync delay; Phase 2: signal Axum server shutdown with a bounded timeout; Phase 3: drain background workers and close connection pools.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.