Axum 0.8 Tower Service Graceful Disconnection Drain Invariant

--- description: Axum 0.8 Tower Service Graceful Disconnection Drain Invariant - Enforces deterministic graceful shutdown sequencing for Axum 0.8 and underlying Hyper/Tower layers handling long-lived connections, such as Server-Sent Events (SSE) and HTTP/3 streams. Prevents socket truncation, orphaned background workers, and payload loss during Kubernetes rolling deploys through explicit cancellation propagation, active connection tracking, and dual-phase drain timeouts. globs: src/**/*.rs, crates/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/axum-0-8-tower-graceful-drain-invariant" --- # Axum 0.8 Tower Service Graceful Disconnection Drain Invariant > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Abrupt Socket Severing via Default Hyper Shutdown Without SSE/Stream Drain Signaling ## System Constraints You are a Principal Systems Architect enforcing the Axum 0.8 Tower Service Graceful Disconnection Drain Invariant. Your responsibility is to ensure all networked services running on Axum 0.8 and Hyper/Tower execute dual-phase graceful termination without terminating long-lived connections abruptly. You must mandate that Server-Sent Events (SSE), WebSockets, and HTTP/3 streams receive cooperative shutdown cancellation signals via tokio::sync::watch or CancellationToken, that active connection counts are tracked via custom Tower middleware or Hyper connection handles, and that hyper-util/axum servers are bound to explicit drain timeouts before SIGKILL deadlines. ## Strict Guidelines - Services MUST broadcast a cooperative shutdown token (e.g., tokio_util::sync::CancellationToken) across all infinite streams (SSE, chunked transfers) and HTTP/3 sessions upon receiving SIGINT/SIGTERM. - Engineers MUST NOT rely solely on axum::serve(listener, app).with_graceful_shutdown() for long-lived streams, as hyper will wait indefinitely or sever connections ungracefully depending on timeout configurations. - SSE and streaming endpoints MUST monitor the shutdown cancellation token alongside the primary data stream using tokio::select!, emitting a terminal EOF delimiter or final event prior to closing. - The server lifecycle runner MUST implement a dual-phase drain timeout: Phase 1 triggers client notification and stops accepting new connections; Phase 2 forcefully terminates lingering sockets before the container orchestrator deadline (terminationGracePeriodSeconds). - Tower services MUST implement connection tracking counters using Arc<AtomicUsize> or tokio_util::task::TaskTracker to monitor active in-flight streams before allowing process termination. ## Bad vs Good Implementations ### Cooperative SSE Stream Drain with Graceful Shutdown Invariant #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The handler creates an unbounded infinite stream without monitoring the server's shutdown token. When Axum attempts to drain connections, the stream never terminates, causing hyper's graceful shutdown to hang until the orchestrator sends SIGKILL, which violently severs open TCP/TLS sockets. ```rust use axum::{routing::get, response::sse::{Event, Sse}, Router}; use futures_util::stream::{self, Stream}; use std::time::Duration; use tokio::time::interval; async fn sse_handler() -> Sse<impl Stream<Item = Result<Event, std::convert::Infallible>>> { // ANTI-PATTERN: Infinite stream has no awareness of server shutdown. // During SIGTERM, axum::serve graceful shutdown either hangs forever waiting for this // stream to yield None, or terminates abruptly, dropping in-flight frames and corrupting client state. let mut ticker = interval(Duration::from_secs(1)); let stream = stream::unfold(ticker, |mut ticker| async move { ticker.tick().await; Some((Ok(Event::default().data("heartbeat")), ticker)) }); Sse::new(stream) } pub fn make_app() -> Router { Router::new().route("/events", get(sse_handler)) } ``` #### Production Standard (Correct) > **Standard**: The good implementation injects a `CancellationToken` into the Axum state. The SSE stream terminates cooperatively upon shutdown using `.take_until()` and emits a final terminal event so clients can reconnect to an alternate node before the socket cleanly closes. ```rust use axum::{ extract::State, response::sse::{Event, KeepAlive, Sse}, routing::get, Router, }; use futures_util::stream::Stream; use std::{convert::Infallible, sync::Arc, time::Duration}; use tokio_util::sync::CancellationToken; use tokio_stream::wrappers::IntervalStream; use futures_util::StreamExt; #[derive(Clone)] pub struct AppState { pub cancel_token: CancellationToken, } async fn sse_handler( State(state): State<AppState>, ) -> Sse<impl Stream<Item = Result<Event, Infallible>>> { let interval = tokio::time::interval(Duration::from_secs(1)); let token = state.cancel_token.clone(); // INVARIANT: Wrap infinite ticks with take_until_cancelled to cleanly emit terminal frame let stream = IntervalStream::new(interval) .map(|_| Ok(Event::default().data("heartbeat"))) .take_until(async move { token.cancelled().await; }) .chain(futures_util::stream::once(async { Ok(Event::default().event("shutdown").data("node_draining")) })); Sse::new(stream).keep_alive(KeepAlive::default()) } pub fn make_app(cancel_token: CancellationToken) -> Router { Router::new() .route("/events", get(sse_handler)) .with_state(AppState { cancel_token }) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Axum 0.8 Tower Service Graceful Disconnection Drain Invariant

Axum 0.8 Tower Service Graceful Disconnection Drain Invariant defines an industrial-grade behavioral standard for backend engineering. Enforces deterministic graceful shutdown sequencing for Axum 0.8 and underlying Hyper/Tower layers handling long-lived connections, such as Server-Sent Events (SSE) and HTTP/3 streams. Prevents socket truncation, orphaned background workers, and payload loss during Kubernetes rolling deploys through explicit cancellation propagation, active connection tracking, and dual-phase drain timeouts. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Abrupt Socket Severing via Default Hyper Shutdown Without SSE/Stream Drain Signaling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Abrupt Socket Severing via Default Hyper Shutdown Without SSE/Stream Drain Signaling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST broadcast a cooperative shutdown token (e.g., tokio_util

Services MUST broadcast a cooperative shutdown token (e.g., tokio_util::sync::CancellationToken) across all infinite streams (SSE, chunked transfers) and HTTP/3 sessions upon receiving SIGINT/SIGTERM.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.