Axum Hyper-v1 Multiplexed HTTP/2 Stream Drain Governor

--- description: Axum Hyper-v1 Multiplexed HTTP/2 Stream Drain Governor - Enforces a deterministic two-phase HTTP/2 connection drain protocol with bounded eviction deadlines on Axum Hyper-v1 servers during SIGTERM signals. Prevents persistent multiplexed keep-alive streams from stalling rolling blue/green deployments and inducing abrupt SIGKILL dropped connections. globs: src/**/*.rs, crates/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-hyper-v1-http2-stream-drain-governor" --- # Axum Hyper-v1 Multiplexed HTTP/2 Stream Drain Governor > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded HTTP/2 Keep-Alive Connection Deadlock During Container Termination ## System Constraints You are a Principal Systems Engineer enforcing deterministic HTTP/2 socket drain governance in Axum and Hyper v1. You mandate strict multi-phase shutdown lifecycles to prevent long-lived multiplexed streams from delaying blue/green deploy rollouts. Never allow bare `axum::serve().with_graceful_shutdown()` without connection-level GOAWAY signaling and hard eviction timeouts. ## Strict Guidelines - Engineers MUST ALWAYS implement a two-phase drain for HTTP/2: emit initial GOAWAY frames with max stream ID to stop new streams, then enforce an upper-bounded deadline to complete existing streams before socket teardown. - Engineers MUST NOT rely exclusively on standard `axum::serve(listener, router).with_graceful_shutdown(...)` when serving HTTP/2, as active multiplexed streams without server-side idle/keep-alive timeouts will block shutdown until orchestrator SIGKILL. - Services MUST configure Hyper v1 HTTP/2 keep-alive intervals (`http2_keep_alive_interval`), keep-alive timeouts (`http2_keep_alive_timeout`), and max idle timeouts to evict dead peer connections proactively. - Drain routines MUST integrate a bounded timeout (`tokio::time::timeout`) strictly lower than container orchestrator `terminationGracePeriodSeconds` (recommended: 75% of grace period) to ensure zero unhandled connection drops. - Connection trackers MUST supervise active socket handles using `tokio_util::task::TaskTracker` or atomic counters, triggering aggressive TCP RST or stream aborts if the drain window expires. ## Bad vs Good Implementations ### Anti-Pattern: Naive axum::serve Graceful Shutdown with Indefinite HTTP/2 Holding #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Hyper v1 won't proactively terminate active HTTP/2 connections without explicit stream-level drain governors or timeouts. When clients or ingress proxies pool connections, the server task waits indefinitely for streams to close naturally, exceeding orchestrator termination grace periods and causing sudden, dirty socket terminations. ```rust use axum::{routing::get, Router}; use tokio::net::TcpListener; use tokio::signal::unix::{signal, SignalKind}; pub async fn run_server() { let app = Router::new().route("/health", get(|| async { "OK" })); let listener = TcpListener::bind("0.0.0.0:8080").await.unwrap(); // CRITICAL FLAW: Standard with_graceful_shutdown waits indefinitely for // multiplexed HTTP/2 streams and clients holding keep-alive connections. // During blue/green deploys, the pod stalls until Kubernetes SIGKILL. axum::serve(listener, app) .with_graceful_shutdown(async { let mut sigterm = signal(SignalKind::terminate()).unwrap(); sigterm.recv().await; }) .await .unwrap(); } ``` #### Production Standard (Correct) > **Standard**: Uses Hyper-Util connection management with explicit HTTP/2 keep-alive parameterization and a deterministic TaskTracker. On SIGTERM, the loop exits, active connections receive `.graceful_shutdown()` (transmitting GOAWAY frames to clients), and socket drain time is strictly bounded by DRAIN_DEADLINE before orchestrator SIGKILL. ```rust use axum::{routing::get, Router}; use hyper_util::rt::{TokioExecutor, TokioIo}; use hyper_util::server::conn::auto::Builder as ConnBuilder; use std::sync::Arc; use std::time::Duration; use tokio::net::TcpListener; use tokio::signal::unix::{signal, SignalKind}; use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; const DRAIN_DEADLINE: Duration = Duration::from_secs(20); pub async fn run_server() -> Result<(), Box<dyn std::error::Error + Send + Sync>> { let app = Router::new().route("/health", get(|| async { "OK" })); let listener = TcpListener::bind("0.0.0.0:8080").await?; let token = CancellationToken::new(); let tracker = TaskTracker::new(); let executor = TokioExecutor::new(); let mut auto_builder = ConnBuilder::new(executor); auto_builder .http2() .keep_alive_interval(Some(Duration::from_secs(10))) .keep_alive_timeout(Duration::from_secs(5)) .max_concurrent_streams(Some(256)); let auto_builder = Arc::new(auto_builder); let shutdown_token = token.clone(); tokio::spawn(async move { let mut sigterm = signal(SignalKind::terminate()).expect("sigterm register failed"); sigterm.recv().await; tracing::info!("SIGTERM received: initiating deterministic HTTP/2 drain governor"); shutdown_token.cancel(); }); loop { tokio::select! { _ = token.cancelled() => break, res = listener.accept() => { let (stream, _) = match res { Ok(s) => s, Err(_) => continue }; let io = TokioIo::new(stream); let builder = Arc::clone(&auto_builder); let service = app.clone(); let conn_token = token.clone(); tracker.spawn(async move { let conn = builder.serve_connection_with_upgrades(io, service); tokio::pin!(conn); tokio::select! { res = conn.as_mut() => { let _ = res; } _ = conn_token.cancelled() => { // Emit GOAWAY and allow bounded graceful stream completion conn.as_mut().graceful_shutdown(); let _ = tokio::time::timeout(DRAIN_DEADLINE, conn.as_mut()).await; } } }); } } } drop(listener); tracker.close(); // Strict orchestrator guardrail: evict all remaining tasks before SIGKILL if tokio::time::timeout(DRAIN_DEADLINE, tracker.wait()).await.is_err() { tracing::warn!("HTTP/2 drain window elapsed: forcibly severing surviving multiplexed sockets"); } Ok(()) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum Hyper-v1 Multiplexed HTTP/2 Stream Drain Governor

Axum Hyper-v1 Multiplexed HTTP/2 Stream Drain Governor defines an industrial-grade behavioral standard for backend engineering. Enforces a deterministic two-phase HTTP/2 connection drain protocol with bounded eviction deadlines on Axum Hyper-v1 servers during SIGTERM signals. Prevents persistent multiplexed keep-alive streams from stalling rolling blue/green deployments and inducing abrupt SIGKILL dropped connections. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded HTTP/2 Keep-Alive Connection Deadlock During Container Termination Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded HTTP/2 Keep-Alive Connection Deadlock During Container Termination	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement a two-phase drain for HTTP/2

Engineers MUST ALWAYS implement a two-phase drain for HTTP/2: emit initial GOAWAY frames with max stream ID to stop new streams, then enforce an upper-bounded deadline to complete existing streams before socket teardown.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.