Axum 0.8 / Hyper v1 Deterministic Multiplexed Stream Graceful Drain Barrier

--- description: Axum 0.8 / Hyper v1 Deterministic Multiplexed Stream Graceful Drain Barrier - Enforces deterministic cancellation propagation and controlled HTTP/2 and HTTP/3 multiplex stream drainage during Kubernetes rolling updates. Prevents dropped in-flight requests, connection reset errors (RST_STREAM/GOAWAY races), and half-open socket leaks when pods transition into Terminating state before ingress re-routing completes. globs: src/main.rs, src/server.rs, src/lib.rs, src/bin/**/*.rs, src/infrastructure/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-0-8-hyper-v1-multiplex-stream-drain-barrier" --- # Axum 0.8 / Hyper v1 Deterministic Multiplexed Stream Graceful Drain Barrier > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Naive axum::serve with_graceful_shutdown dropping in-flight HTTP/2 streams and racing SIGTERM ingress pruning ## System Constraints You are a Principal Rust Systems Architect specializing in networking runtimes, Hyper v1, and Kubernetes ingress lifecycle semantics. You mandate strict, deterministic shutdown barriers for Axum 0.8+ services handling multiplexed HTTP/2 and HTTP/3 streams. You must reject naive signal handling that immediately tears down the acceptor or aborts long-lived streaming connections without propagating hyper graceful shutdown signals and waiting for readiness probe detachment. ## Strict Guidelines - Services MUST implement a dual-phase shutdown barrier: Phase 1 initiates a configurable quiescence delay (typically 5-15s) after SIGTERM to allow Kubernetes Service/EndpointSlice propagation before closing the TCP listener. - Engineers MUST configure Hyper v1 auto-server connection builders with `.graceful_shutdown()` explicitly hooked into a `tokio_util::sync::CancellationToken` or hyper connection tracker, avoiding abrupt task drops. - Engineers MUST NOT rely on plain `axum::serve(listener, app).with_graceful_shutdown(...)` alone if multiplexed streams (SSE, gRPC, WebSocket) run indefinitely; a secondary hard-timeout deadline barrier MUST enforce upper-bound drainage. - Code MUST ensure that during the quiescence phase, readiness probes (`/readyz`) immediately return HTTP 503 or fail health checks, while liveness probes (`/healthz`) and existing in-flight multiplexed streams remain fully serviced. - Services MUST emit a Hyper GOAWAY frame with `NO_ERROR` and the last processed stream ID immediately upon entering the active drain phase, preventing clients from scheduling new streams on existing connections while existing streams run to completion. ## Bad vs Good Implementations ### Naive Axum 0.8 Server Shutdown vs Two-Phase Hyper-v1 Drain Barrier #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: When Kubernetes sends SIGTERM, kube-proxy and cloud load balancers take seconds to prune endpoints. Naively binding graceful shutdown directly to SIGTERM causes immediate listener closure and aggressive connection termination, resulting in HTTP 502/504 errors on clients and abrupt RST_STREAM on multiplexed HTTP/2 channels. ```rust use axum::{routing::get, Router}; use tokio::signal; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let app = Router::new().route("/data", get(|| async { "payload" })); let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?; // CRITICAL FLAW: Immediately tears down the listener and all multiplexed // HTTP/2 streams upon SIGTERM. Any in-flight stream is aborted with ECONNRESET // or RST_STREAM before Kube-proxy removes the pod from the EndpointSlice. axum::serve(listener, app) .with_graceful_shutdown(async { signal::ctrl_c().await.expect("failed to listen for ctrl_c"); }) .await?; Ok(()) } ``` #### Production Standard (Correct) > **Standard**: Decouples SIGTERM receipt from connection termination. Phase 1 flips the readiness probe to 503 and waits for ingress traffic to reroute. Phase 2 initiates Hyper's graceful drain via CancellationToken, followed by a hard upper-bound timeout to protect against permanently hung multiplexed streams. ```rust use axum::{extract::State, http::StatusCode, routing::get, Router}; use std::{sync::{atomic::{AtomicBool, Ordering}, Arc}, time::Duration}; use tokio::{net::TcpListener, signal::unix::{signal, SignalKind}, sync::watch}; use tokio_util::sync::CancellationToken; #[derive(Clone)] struct AppState { is_terminating: Arc<AtomicBool>, } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let state = AppState { is_terminating: Arc::new(AtomicBool::new(false)) }; let app = Router::new() .route("/healthz", get(|| async { StatusCode::OK })) .route("/readyz", get(|State(s): State<AppState>| async move { if s.is_terminating.load(Ordering::Acquire) { StatusCode::SERVICE_UNAVAILABLE } else { StatusCode::OK } })) .with_state(state.clone()); let listener = TcpListener::bind("0.0.0.0:8080").await?; let drain_token = CancellationToken::new(); let drain_token_clone = drain_token.clone(); tokio::spawn(async move { let mut sigterm = signal(SignalKind::terminate()).expect("sigterm stream fail"); sigterm.recv().await; // Phase 1: Mark unhealthy so Kubelet stops routing new connections state.is_terminating.store(true, Ordering::Release); // Barrier delay matching Kubernetes EndpointSlice convergence tokio::time::sleep(Duration::from_secs(10)).await; // Phase 2: Signal Axum/Hyper to emit GOAWAY and drain streams drain_token_clone.cancel(); }); let server = axum::serve(listener, app) .with_graceful_shutdown(async move { drain_token.cancelled().await; }); // Hard timeout barrier for stubborn in-flight HTTP/2 streams tokio::select! { res = server => res?, _ = tokio::time::sleep(Duration::from_secs(30)) => { eprintln!("Drain deadline exceeded; forcing process termination."); } } Ok(()) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 / Hyper v1 Deterministic Multiplexed Stream Graceful Drain Barrier

Axum 0.8 / Hyper v1 Deterministic Multiplexed Stream Graceful Drain Barrier defines an industrial-grade behavioral standard for backend engineering. Enforces deterministic cancellation propagation and controlled HTTP/2 and HTTP/3 multiplex stream drainage during Kubernetes rolling updates. Prevents dropped in-flight requests, connection reset errors (RST_STREAM/GOAWAY races), and half-open socket leaks when pods transition into Terminating state before ingress re-routing completes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Naive axum::serve with_graceful_shutdown dropping in-flight HTTP/2 streams and racing SIGTERM ingress pruning Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Naive axum::serve with_graceful_shutdown dropping in-flight HTTP/2 streams and racing SIGTERM ingress pruning	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST implement a dual-phase shutdown barrier

Services MUST implement a dual-phase shutdown barrier: Phase 1 initiates a configurable quiescence delay (typically 5-15s) after SIGTERM to allow Kubernetes Service/EndpointSlice propagation before closing the TCP listener.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.