FastAPI AnyIO Worker Thread Pool Starvation & Concurrency Partitioning Guardrail

Prevents catastrophic worker thread starvation and cascading event-loop deadlocks in FastAPI applications. In FastAPI, standard synchronous route handlers (`def`) and untuned `run_in_threadpool` or `anyio.to_thread.run_sync` calls draw from AnyIO's shared global `CapacityLimiter` (defaulting to 40 threads), leading to full thread pool exhaustion under heavy sync I/O. This standard enforces non-blocking async primitives, custom isolated thread pools via dedicated `CapacityLimiter` instances, and strict tuning of global AnyIO concurrency limits during application lifespan.

--- description: FastAPI AnyIO Worker Thread Pool Starvation & Concurrency Partitioning Guardrail - Prevents catastrophic worker thread starvation and cascading event-loop deadlocks in FastAPI applications. In FastAPI, standard synchronous route handlers (`def`) and untuned `run_in_threadpool` or `anyio.to_thread.run_sync` calls draw from AnyIO's shared global `CapacityLimiter` (defaulting to 40 threads), leading to full thread pool exhaustion under heavy sync I/O. This standard enforces non-blocking async primitives, custom isolated thread pools via dedicated `CapacityLimiter` instances, and strict tuning of global AnyIO concurrency limits during application lifespan. globs: **/*.py, app/**/*.py, src/**/*.py alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/fastapi-anyio-threadpool-starvation-guardrail" --- # FastAPI AnyIO Worker Thread Pool Starvation & Concurrency Partitioning Guardrail > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Saturating FastAPI's Global 40-Thread AnyIO Worker Pool via Unbounded Sync I/O ## System Constraints You are a Principal Python Architect specializing in modern high-throughput asynchronous microservices. You strictly enforce zero-starvation concurrency patterns in FastAPI and AnyIO. You mandate non-blocking async drivers, explicit thread pool partitioning with dedicated CapacityLimiter instances, and lifespan-level thread pool limits. ## Strict Guidelines - Engineers MUST NOT define FastAPI route handlers or dependencies with standard 'def' when handling high-concurrency external I/O (e.g., HTTP requests, database transactions, disk reads); native 'async def' handlers with async-native drivers MUST be used instead. - Engineers MUST NEVER execute blocking synchronous operations inside 'async def' functions without explicitly offloading via 'anyio.to_thread.run_sync' or 'starlette.concurrency.run_in_threadpool'. - Engineers MUST NOT offload high-latency or slow third-party synchronous operations to the default AnyIO worker thread pool without providing an isolated 'anyio.CapacityLimiter' instance. - Engineers MUST tune the global AnyIO thread capacity limiter ('anyio.to_thread.current_default_thread_limiter().total_tokens') during application lifespan startup if benchmarked workloads demonstrably require a higher baseline capacity than the default 40 tokens. - Services MUST isolate CPU-bound tasks from I/O-bound sync operations by deploying separate ProcessPoolExecutors or Celery/ARQ background workers instead of AnyIO thread pools. - Components MUST enforce explicit timeouts (e.g., via 'anyio.fail_after' or 'asyncio.timeout') across all offloaded thread pool executions to prevent thread retention under upstream failure modes. ## Bad vs Good Implementations ### Preventing Thread Pool Exhaustion from Blocking Sync I/O in FastAPI #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The synchronous route handler consumes a thread from the global AnyIO pool (capped at 40). A surge of slow HTTP requests fully exhausts this pool, starving internal dependencies, database sessions, and filesystem calls. The second route blocks the entire asyncio event loop directly, stopping heartbeats and concurrent I/O. ```python from fastapi import FastAPI import requests import time app = FastAPI() # ANTI-PATTERN: Standard 'def' routes run on AnyIO's shared 40-thread worker pool. # Under 50+ concurrent requests, all 40 worker threads become pinned by slow sync HTTP # or sleep calls, causing all other sync routes and background tasks to deadlock. @app.get("/external-data") def get_external_data(): response = requests.get("https://api.legacy-upstream.internal/slow", timeout=10) return response.json() # ANTI-PATTERN: Blocking call inside 'async def' freezes the entire asyncio event loop @app.get("/blocking-in-async") async def blocking_in_async(): time.sleep(2) # Blocks event loop for all concurrent users return {"status": "complete"} ``` #### Production Standard (Correct) > **Standard**: Replaces synchronous network calls with native non-blocking `httpx.AsyncClient`. Where synchronous third-party code is unavoidable, it isolates the execution to a custom `CapacityLimiter` with an active `fail_after` timeout, ensuring the primary worker pool of 100 tokens remains completely unblocked. ```python from contextlib import asynccontextmanager import anyio import httpx from fastapi import FastAPI # ISOLATION: Dedicated capacity limiter prevents legacy sync workloads # from exhausting the global worker pool. LEGACY_SYNC_LIMITER = anyio.CapacityLimiter(10) @asynccontextmanager async def lifespan(app: FastAPI): # PRODUCTION GUARDRAIL: Adjust global AnyIO thread limiter on startup limiter = anyio.to_thread.current_default_thread_limiter() limiter.total_tokens = 100 async with httpx.AsyncClient(timeout=10.0) as client: app.state.http_client = client yield app = FastAPI(lifespan=lifespan) # PATTERN 1: Native async I/O avoids AnyIO worker thread pool entirely @app.get("/external-data") async def get_external_data(): client: httpx.AsyncClient = app.state.http_client response = await client.get("https://api.upstream.internal/data") response.raise_for_status() return response.json() # PATTERN 2: Isolated sync worker execution using dedicated limiter and timeout def _legacy_blocking_operation(payload: dict) -> dict: # Dedicated third-party library without async support import legacy_sdk return legacy_sdk.process(payload) @app.post("/legacy-bridge") async def legacy_bridge(payload: dict): with anyio.fail_after(5.0): return await anyio.to_thread.run_sync( _legacy_blocking_operation, payload, limiter=LEGACY_SYNC_LIMITER ) ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: FastAPI AnyIO Worker Thread Pool Starvation & Concurrency Partitioning Guardrail

FastAPI AnyIO Worker Thread Pool Starvation & Concurrency Partitioning Guardrail defines an industrial-grade behavioral standard for backend engineering. Prevents catastrophic worker thread starvation and cascading event-loop deadlocks in FastAPI applications. In FastAPI, standard synchronous route handlers (`def`) and untuned `run_in_threadpool` or `anyio.to_thread.run_sync` calls draw from AnyIO's shared global `CapacityLimiter` (defaulting to 40 threads), leading to full thread pool exhaustion under heavy sync I/O. This standard enforces non-blocking async primitives, custom isolated thread pools via dedicated `CapacityLimiter` instances, and strict tuning of global AnyIO concurrency limits during application lifespan. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Saturating FastAPI's Global 40-Thread AnyIO Worker Pool via Unbounded Sync I/O Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Saturating FastAPI's Global 40-Thread AnyIO Worker Pool via Unbounded Sync I/O	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NOT define FastAPI route handlers or dependencies with standard 'def' when handling high-concurrency external I/O (e.g., HTTP requests, database transactions, disk reads)

Engineers MUST NOT define FastAPI route handlers or dependencies with standard 'def' when handling high-concurrency external I/O (e.g., HTTP requests, database transactions, disk reads); native 'async def' handlers with async-native drivers MUST be used instead.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.