Tokio & Tower Buffer Backpressure and Circular-Wait Deadlock Prevention

--- description: Tokio & Tower Buffer Backpressure and Circular-Wait Deadlock Prevention - Enforces strict ordering, worker isolation, and reservation-dispatch pairing for Tower Buffer middleware in Tokio-based Rust RPC clients and servers. Prevents re-entrant channel deadlocks, circular-wait lockups, and unmetered memory exhaustion in layered interceptor topologies. globs: src/**/*.rs, crates/**/*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/tokio-tower-buffer-backpressure-deadlock-guardrail" --- # Tokio & Tower Buffer Backpressure and Circular-Wait Deadlock Prevention > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Interleaved Awaits Between poll_ready and call in Tower Buffers Causing Worker Starvation and Circular-Wait Deadlocks ## System Constraints You are a Principal Rust Systems Architect specializing in Tokio runtime internals and Tower service middleware composition. Enforce strict invariants regarding `tower::buffer::Buffer` channels, readiness permits, and task execution domains. Mandate that `poll_ready` reservations are never held across unrelated asynchronous yield points, re-entrant buffered RPC topologies are structurally forbidden, and bounded backpressure is coupled with timeout and load-shedding barriers. ## Strict Guidelines - Engineers MUST ALWAYS invoke `Service::call` immediately after `ServiceExt::ready` or `Service::poll_ready` resolves to `Ok(())`, with ZERO intervening `.await` points or fallible async operations. - Engineers MUST NOT invoke a `tower::buffer::Buffer` service from inside the execution context of the buffer's own inner worker task (re-entrant invocation causes instant single-worker deadlock). - Services MUST decouple circular RPC dependencies using independent client channel pools, out-of-band notification queues, or non-buffered point-to-point connections rather than nested middleware buffers. - Code SHOULD prefer `tower::util::ServiceExt::oneshot(service, request)` when consuming cloned multi-threaded service handles to guarantee atomic readiness reservation and request dispatch. - Engineers MUST NEVER hold a mutable reference or permit acquired during `poll_ready` across an IPC/RPC round-trip, database transaction, or external mutex guard. - Middleware stacks MUST pair `tower::buffer::Buffer` with upstream `tower::limit::concurrency` or `tower::load_shed::LoadShed` layers to prevent unbounded queue saturation and thread starvation under heavy load. ## Bad vs Good Implementations ### Tower Service Readiness Reservation and Dispatch #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Calling `ready().await` reserves a capacity slot in the Tower `Buffer` channel. Awaiting `auth_token_fetcher` while holding that reservation blocks the service slot. If the token fetcher itself triggers an RPC through the same buffered client, a circular-wait deadlock occurs. ```rust use tower::{Service, ServiceExt}; use tokio::sync::Mutex; use std::sync::Arc; async fn execute_reentrant_call<S, Req>( client: Arc<Mutex<tower::buffer::Buffer<S, Req>>>, req: Req, auth_token_fetcher: impl std::future::Future<Output = String>, ) -> Result<S::Response, S::Error> where S: Service<Req> + Send + 'static, Req: Send + 'static, S::Future: Send + 'static, S::Error: Send + 'static, { let mut svc = client.lock().await; // ANTI-PATTERN: Reserving buffer channel capacity, then yielding the runtime // during a network fetch. If the buffer is near capacity or auth_token_fetcher // calls back into the same client, this deadlocks the worker. svc.ready().await?; let _token = auth_token_fetcher.await; // Async yield holds the buffer permit! svc.call(req).await } ``` #### Production Standard (Correct) > **Standard**: All external asynchronous dependencies are fully resolved before reserving capacity. Using `client.oneshot(req).await` ensures the `poll_ready` permit acquisition and the `call` dispatch occur atomically without interleaved suspension points. ```rust use tower::{Service, ServiceExt}; async fn execute_safe_call<S, Req>( client: tower::buffer::Buffer<S, Req>, req: Req, auth_token_fetcher: impl std::future::Future<Output = String>, ) -> Result<S::Response, S::Error> where S: Service<Req> + Send + 'static, Req: Send + 'static, S::Future: Send + 'static, S::Error: Send + 'static, { // Invariant: Resolve all auxiliary async work BEFORE acquiring buffer permit let _token = auth_token_fetcher.await; // Atomically reserve buffer permit and dispatch request with no intervening yield client.oneshot(req).await } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Tokio & Tower Buffer Backpressure and Circular-Wait Deadlock Prevention

Tokio & Tower Buffer Backpressure and Circular-Wait Deadlock Prevention defines an industrial-grade behavioral standard for backend engineering. Enforces strict ordering, worker isolation, and reservation-dispatch pairing for Tower Buffer middleware in Tokio-based Rust RPC clients and servers. Prevents re-entrant channel deadlocks, circular-wait lockups, and unmetered memory exhaustion in layered interceptor topologies. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Interleaved Awaits Between poll_ready and call in Tower Buffers Causing Worker Starvation and Circular-Wait Deadlocks Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Interleaved Awaits Between poll_ready and call in Tower Buffers Causing Worker Starvation and Circular-Wait Deadlocks	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS invoke `Service

Engineers MUST ALWAYS invoke `Service::call` immediately after `ServiceExt::ready` or `Service::poll_ready` resolves to `Ok(())`, with ZERO intervening `.await` points or fallible async operations.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.