Rust Axum HTTP/2 GOAWAY Graceful Connection Draining Standard

--- description: Rust Axum HTTP/2 GOAWAY Graceful Connection Draining Standard - Enforces RFC 9113 compliant HTTP/2 GOAWAY frame sequencing and deterministic stream draining in Rust Axum microservices. Guarantees zero dropped in-flight multiplexed requests and clean client reconnection signaling during rolling blue/green deployments and pod evictions. globs: src/**/*.rs, Cargo.toml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/rust-axum-http2-goaway-graceful-draining-standard" --- # Rust Axum HTTP/2 GOAWAY Graceful Connection Draining Standard > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Naive Server Termination Dropping In-Flight HTTP/2 Streams via Premature TCP RST/FIN ## System Constraints You are a Principal Systems Architect enforcing the Rust Axum Graceful HTTP/2 GOAWAY Connection Draining Standard. You must strictly mandate that all Axum services handle POSIX termination signals (SIGTERM/SIGINT) via a two-phase connection drain: first signaling upstream ingress removal while accepting in-flight frames, followed by triggering Hyper's HTTP/2 graceful shutdown (GOAWAY frame dispatch) with a bounded draining timeout. Never allow raw process exits, abrupt listener drops, or unbounded server shutdowns that cause HTTP/2 stream errors like REFUSED_STREAM or stream cancellation exceptions. ## Strict Guidelines - Engineers MUST ALWAYS decouple ingress deregistration from socket termination by enforcing a pre-drain dwell period (sleep) immediately after receiving SIGTERM to permit mesh/load-balancer endpoint propagation. - Engineers MUST ALWAYS use Hyper's connection-level graceful shutdown or Axum's integrated graceful shutdown with hyper_util to ensure HTTP/2 GOAWAY frames with the maximum stream ID (2^31 - 1) and subsequent actual last-stream ID are properly dispatched to peers. - Engineers MUST NOT invoke `std::process::exit` or terminate the Tokio runtime while active HTTP/2 connections remain un-drained. - Code MUST wrap the graceful drain sequence in a hard timeout (`tokio::time::timeout`) to prevent hung client connections from deadlocking rolling deployments beyond orchestration termination grace periods. - Services MUST expose an atomic liveness/readiness health endpoint that immediately flips to unhealthy (HTTP 503) upon signal receipt while preserving existing connection servicing. - Components NEVER drop connection handles prematurely; connections MUST be tracked or delegated to a graceful driver task. ## Bad vs Good Implementations ### Naive Axum Shutdown Causing HTTP/2 Stream Truncation vs. Industrial Two-Phase GOAWAY Drain #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: When SIGTERM is received, the server stops listening immediately. Existing multiplexed HTTP/2 streams are severed via TCP FIN/RST without allowing load balancers (e.g., Envoy, AWS ALB) time to remove the endpoint from rotation, causing client-side 502/503 errors and dropped RPCs. ```rust use axum::{routing::get, Router}; use tokio::signal; #[tokio::main] async fn main() { let app = Router::new().route("/health", get(|| async { "OK" })); let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await.unwrap(); // ANTI-PATTERN: Naively listening for SIGINT/SIGTERM without a pre-drain window // causes Kubernetes/Ingress to continue routing new TCP/HTTP2 frames to a socket // that is already closing. Furthermore, terminating the runtime immediately // drops active HTTP/2 streams without sending GOAWAY frames, triggering client // connection resets (ECONNRESET/RST_STREAM). axum::serve(listener, app) .with_graceful_shutdown(async { signal::ctrl_c().await.expect("failed to listen for event"); println!("Termination signal received, exiting immediately"); }) .await .unwrap(); } ``` #### Production Standard (Correct) > **Standard**: Implements an industrial two-phase drain: First, SIGTERM marks the readiness probe unhealthy and pauses for network routing convergence. Next, cancellation signals Axum/Hyper to send HTTP/2 GOAWAY frames with stream draining. Finally, an outer timeout prevents deadlocks. ```rust use axum::{routing::get, Router}; use std::{sync::atomic::{AtomicBool, Ordering}, sync::Arc, time::Duration}; use tokio::signal::unix::{signal, SignalKind}; use tokio_util::sync::CancellationToken; static IS_DRAINING: AtomicBool = AtomicBool::new(false); async fn readiness_check() -> (hyper::StatusCode, &'static str) { if IS_DRAINING.load(Ordering::Acquire) { (hyper::StatusCode::SERVICE_UNAVAILABLE, "Draining") } else { (hyper::StatusCode::OK, "Ready") } } async fn shutdown_signal_sequence(token: CancellationToken) { let mut sigterm = signal(SignalKind::terminate()).expect("failed to bind SIGTERM"); let mut sigint = signal(SignalKind::interrupt()).expect("failed to bind SIGINT"); tokio::select! { _ = sigterm.recv() => tracing::info!("SIGTERM received"), _ = sigint.recv() => tracing::info!("SIGINT received"), }; // Phase 1: Mark readiness probe unhealthy for mesh/ingress deregistration IS_DRAINING.store(true, Ordering::Release); tracing::info!("Marked unhealthy. Entering pre-drain dwell period..."); // Wait for Ingress/Kubernetes iptables/endpoints propagation (e.g., 10 seconds) tokio::time::sleep(Duration::from_secs(10)).await; tracing::info!("Starting Axum/Hyper HTTP/2 GOAWAY connection drain..."); token.cancel(); } #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error>> { let token = CancellationToken::new(); let shutdown_token = token.clone(); let app = Router::new() .route("/ready", get(readiness_check)) .route("/api/v1/work", get(|| async { "Processed" })); let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?; tokio::spawn(shutdown_signal_sequence(shutdown_token)); // Phase 2: Hyper sends HTTP/2 GOAWAY, closes listeners, and finishes existing streams let server = axum::serve(listener, app) .with_graceful_shutdown(async move { token.cancelled().await; }); // Enforce an absolute bound on stream completion to prevent orphaned hangs if let Err(_) = tokio::time::timeout(Duration::from_secs(30), server).await { tracing::error!("Graceful drain timed out after 30s. Forcing exit."); std::process::exit(1); } tracing::info!("Clean shutdown complete. Zero dropped frames."); Ok(()) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Rust Axum HTTP/2 GOAWAY Graceful Connection Draining Standard

Rust Axum HTTP/2 GOAWAY Graceful Connection Draining Standard defines an industrial-grade behavioral standard for backend engineering. Enforces RFC 9113 compliant HTTP/2 GOAWAY frame sequencing and deterministic stream draining in Rust Axum microservices. Guarantees zero dropped in-flight multiplexed requests and clean client reconnection signaling during rolling blue/green deployments and pod evictions. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Naive Server Termination Dropping In-Flight HTTP/2 Streams via Premature TCP RST/FIN Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Naive Server Termination Dropping In-Flight HTTP/2 Streams via Premature TCP RST/FIN	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS decouple ingress deregistration from socket termination by enforcing a pre-drain dwell period (sleep) immediately after receiving SIGTERM to permit mesh/load-balancer endpoint propagation.

Engineers MUST ALWAYS decouple ingress deregistration from socket termination by enforcing a pre-drain dwell period (sleep) immediately after receiving SIGTERM to permit mesh/load-balancer endpoint propagation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.