vLLM PagedAttention Multi-Tenant KV-Cache Bound and Preemption Guardrail

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Dynamic Max-Token Ingestion Triggering PagedAttention Block Exhaustion and Cascading GPU Preemption Thrashing) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

--- description: "vLLM PagedAttention Multi-Tenant KV-Cache Bound and Preemption Guardrail - Enforces strict tenant-isolated token reservation and PagedAttention block budgeting before scheduling requests to vLLM inference engines. Prevents sudden GPU out-of-memory cascades and thrashing preemption loops by binding dynamic context lengths against a hard watermark threshold." globs: "src/inference/**/*.{ts,tsx}, src/gateways/**/*.{ts,tsx}, src/services/**/*.{ts,tsx}, src/llm/**/*.{ts,tsx}" alwaysApply: true license: "CC-BY-4.0" source: "https://agentrules.org/rules/vllm-pagedattention-kvcache-throttling-standard" --- # vLLM PagedAttention Multi-Tenant KV-Cache Bound and Preemption Guardrail > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded Dynamic Max-Token Ingestion Triggering PagedAttention Block Exhaustion and Cascading GPU Preemption Thrashing ## System Constraints You are a Principal Software Architect enforcing the vLLM PagedAttention KV-Cache Multi-Tenant Memory Throttling Standard. Your primary responsibility is guaranteeing GPU stability and fair multi-tenant scheduling across vLLM inference worker pools. You must enforce strict token budget pre-allocation, reserve memory headrooms for KV block expansion, reject requests exceeding per-tenant block quotas, and prevent uncoordinated sequence preemptions that cause catastrophic CUDA out-of-memory cascades. ## Strict Guidelines - Engineers MUST ALWAYS pre-allocate and reserve physical KV-cache blocks based on prompt length plus bounded max_tokens before enqueuing requests to the vLLM scheduler. - Engineers MUST NOT allow unbounded or default max_tokens parameters from multi-tenant consumers without clamping them to tenant-tier ceiling limits. - Services MUST track active PagedAttention GPU memory utilization and enforce a minimum 15% safety watermark of free KV blocks before accepting new prompt contexts. - Code SHOULD implement proactive 429 (Too Many Requests) backpressure when tenant block budgets or global watermark thresholds are breached, rather than allowing vLLM internal preemption to degrade into recompute thrashing. - Components NEVER rely solely on vLLM internal swap-space fallback, as high-concurrency host memory swapping severely degrades P99 TTFT and causes downstream request timeouts. ## Bad vs Good Implementations ### Multi-Tenant Admission Controller with KV-Cache Watermark Reservation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Client requests specify unbounded or oversized max_tokens without pre-admission reservation. When multiple tenants execute concurrently, the physical KV blocks managed by PagedAttention are quickly exhausted, forcing the scheduler into continuous preempt-and-recompute loops, escalating latency and triggering CUDA allocations outside allocated pools. ```typescript import { Request, Response } from 'express'; import axios from 'axios'; // ANTI-PATTERN: Forwards arbitrary max_tokens directly to vLLM without block reservations. // Under heavy concurrent load, KV-cache blocks deplete rapidly, causing vLLM to thrash // via recomputation/swapping or crash with CUDA OOM. export async function handleGenerate(req: Request, res: Response): Promise<void> { const { prompt, max_tokens, tenantId } = req.body; const vllmResponse = await axios.post('http://vllm-engine:8000/v1/completions', { model: 'meta-llama/Llama-3-70b-Instruct', prompt, max_tokens: max_tokens || 8192 // Catastrophic: Unchecked context allocation triggers block exhaustion }); res.json(vllmResponse.data); } ``` #### Production Standard (Correct) > **Standard**: The production implementation computes the exact PagedAttention blocks required, clamps tokens to a strict tenant cap, verifies that global free KV block availability meets the minimum 15% headroom, and tracks tenant-level block quotas before delegating to the vLLM engine. ```typescript import { Request, Response } from 'express'; import axios from 'axios'; interface TenantQuotaManager { canAllocateBlocks(tenantId: string, requiredBlocks: number): Promise<boolean>; reserveBlocks(tenantId: string, requestId: string, requiredBlocks: number): Promise<void>; releaseBlocks(tenantId: string, requestId: string): Promise<void>; getGpuFreeBlockRatio(): Promise<number>; } const BLOCK_SIZE = 16; const MIN_FREE_BLOCK_WATERMARK = 0.15; // 15% safety headroom const TENANT_MAX_TOKEN_CAP = 2048; declare const quotaManager: TenantQuotaManager; export async function handleGenerate(req: Request, res: Response): Promise<void> { const { prompt, max_tokens, tenantId } = req.body; const requestId = req.header('x-request-id') || crypto.randomUUID(); // 1. Clamp output tokens to tenant maximum allowable bounds const clampedMaxTokens = Math.min(max_tokens ?? 512, TENANT_MAX_TOKEN_CAP); // 2. Approximate token count (1 token ≈ 4 chars conservative heuristic if tokenizer not inlined) const estimatedPromptTokens = Math.ceil(prompt.length / 4); const totalTokens = estimatedPromptTokens + clampedMaxTokens; const requiredBlocks = Math.ceil(totalTokens / BLOCK_SIZE); // 3. Enforce global GPU KV-cache watermark guardrail const freeRatio = await quotaManager.getGpuFreeBlockRatio(); if (freeRatio < MIN_FREE_BLOCK_WATERMARK) { res.status(503).json({ error: 'Inference pool KV cache capacity degraded below safety watermark. Retry with backoff.', code: 'KV_CACHE_WATERMARK_EXCEEDED' }); return; } // 4. Check tenant-isolated block allocation limits const hasTenantBudget = await quotaManager.canAllocateBlocks(tenantId, requiredBlocks); if (!hasTenantBudget) { res.status(429).json({ error: 'Tenant KV block budget exceeded.', code: 'TENANT_BLOCK_QUOTA_EXHAUSTED' }); return; } await quotaManager.reserveBlocks(tenantId, requestId, requiredBlocks); try { const vllmResponse = await axios.post('http://vllm-engine:8000/v1/completions', { model: 'meta-llama/Llama-3-70b-Instruct', prompt, max_tokens: clampedMaxTokens }, { timeout: 30000 }); res.json(vllmResponse.data); } finally { await quotaManager.releaseBlocks(tenantId, requestId); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: vLLM PagedAttention Multi-Tenant KV-Cache Bound and Preemption Guardrail

vLLM PagedAttention Multi-Tenant KV-Cache Bound and Preemption Guardrail defines an industrial-grade behavioral standard for backend engineering. Enforces strict tenant-isolated token reservation and PagedAttention block budgeting before scheduling requests to vLLM inference engines. Prevents sudden GPU out-of-memory cascades and thrashing preemption loops by binding dynamic context lengths against a hard watermark threshold. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Dynamic Max-Token Ingestion Triggering PagedAttention Block Exhaustion and Cascading GPU Preemption Thrashing Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Dynamic Max-Token Ingestion Triggering PagedAttention Block Exhaustion and Cascading GPU Preemption Thrashing	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS pre-allocate and reserve physical KV-cache blocks based on prompt length plus bounded max_tokens before enqueuing requests to the vLLM scheduler.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.