AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
backendcritical PriorityID: litellm-proxy-resilience-quota-routing

LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing

Enforces enterprise-grade resilience and financial governance across LiteLLM Proxy gateway integrations. Mandates deterministic model-tier fallback routing, bounded semantic cache TTLs with similarity thresholds, and strict upstream token budget enforcement to eliminate unmitigated failover loops and cascade billing overages.

Prohibited Anti-Pattern: Unbounded Primary-Model Routing with Indefinite Semantic Caching and Unchecked Rate-Limit Exhaustion

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing - Enforces enterprise-grade resilience and financial governance across LiteLLM Proxy gateway integrations. Mandates deterministic model-tier fallback routing, bounded semantic cache TTLs with similarity thresholds, and strict upstream token budget enforcement to eliminate unmitigated failover loops and cascade billing overages.
globs: src/**/llm/**/*.{ts,tsx}, src/**/gateway/**/*.{ts,tsx}, src/**/services/ai/**/*.{ts,tsx}, config/litellm*.yaml
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/litellm-proxy-resilience-quota-routing"
---

# LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing

> **Severity**: CRITICAL | **Category**: backend
> **Anti-Pattern Warning**: Unbounded Primary-Model Routing with Indefinite Semantic Caching and Unchecked Rate-Limit Exhaustion

## System Constraints
You are a Principal Software Architect enforcing production invariants on LiteLLM Proxy integrations. Mandate multi-tier fallback routing, deterministic token quota budgeting per tenant, bounded semantic cache TTLs with strict similarity distance metrics, and graceful degradation during provider degradation.

## Strict Guidelines
- Services MUST route inference requests through virtual model groups (e.g., 'production-tier-1') configured with multi-provider fallbacks (e.g., OpenAI -> Azure -> Anthropic) rather than querying concrete upstream provider model IDs directly.
- Engineers MUST specify bounded TTLs (maximum 86400 seconds) and similarity score thresholds (minimum 0.85 cosine similarity) for semantic cache configurations to prevent prompt drift and stale responses.
- Engineers MUST attach tenant budgeting identifiers ('user', 'team_id', or 'end_user_id') and hard spend caps to proxy invocation headers to ensure the proxy's BudgetManager halts execution before quota exhaustion.
- Code MUST handle HTTP 429 (Rate Limit) and HTTP 503 (Service Unavailable) status codes with randomized exponential backoff and circuit-breaker tripping, rather than aggressive immediate retries.
- Components NEVER execute raw unbounded streaming requests without client-side timeout guards (AbortController) and downstream disconnect hooks to prevent phantom token burn.

## Bad vs Good Implementations
### LiteLLM Proxy Client Call with Budgeting, Semantic Cache Control, and Failover

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Directly targets a specific upstream model ('gpt-4o') bypassing fallback router groups. Fails to pass tenant metadata or budget constraints, leaving downstream spend unbounded. Omits semantic caching controls, risk parameters, and timeout abort signals, causing thread starvation and cascading quota exhaustion during provider outages.
```typescript
import OpenAI from 'openai';

// ANTI-PATTERN: Direct concrete model usage, missing cache headers, no budget tracking, unbounded call
const litellm = new OpenAI({
  baseURL: 'https://litellm.internal.net/v1',
  apiKey: process.env.LITELLM_MASTER_KEY,
});

export async function generateCompletion(prompt: string) {
  const response = await litellm.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }],
  });
  return response.choices[0].message.content;
}
```

#### Production Standard (Correct)
> **Standard**: Targets a virtual model group configured in LiteLLM for zero-downtime multi-provider fallbacks. Enforces per-request financial bounds via 'X-LiteLLM-MaxBudget' and tenant identifiers. Constrains semantic cache freshness with explicit TTL and high similarity threshold, while enforcing network timeouts.
```typescript
import OpenAI from 'openai';

const litellm = new OpenAI({
  baseURL: process.env.LITELLM_PROXY_BASE_URL,
  apiKey: process.env.LITELLM_VIRTUAL_KEY,
  timeout: 15_000,
  maxRetries: 0 // Retries must be managed by LiteLLM Proxy router
});

interface CompletionOptions {
  tenantId: string;
  userId: string;
  maxBudgetUSD?: number;
  signal?: AbortSignal;
}

export async function generateCompletion(prompt: string, options: CompletionOptions) {
  // INVARIANT: Route to logical group, attach tenant metadata, enforce bounded cache TTL
  const response = await litellm.chat.completions.create({
    model: 'primary-analysis-group', // Virtual model group backed by fallbacks
    messages: [{ role: 'user', content: prompt }],
    user: options.userId,
    extra_headers: {
      'X-LiteLLM-TeamId': options.tenantId,
      'X-LiteLLM-MaxBudget': options.maxBudgetUSD ? options.maxBudgetUSD.toFixed(4) : '0.0500',
      'X-LiteLLM-Cache-TTL': '3600', // 1 hour bounded semantic cache TTL
      'X-LiteLLM-Semantic-Threshold': '0.90'
    }
  }, { signal: options.signal });

  return response.choices[0]?.message?.content ?? '';
}
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Prometheus Gateway MCP
Inspects LiteLLM proxy latency, error rates, and token consumption metrics.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

LiteLLM Admin CLI
CLI utility for validating router fallback groups, budget allocations, and cache status.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing

LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing defines an industrial-grade behavioral standard for backend engineering. Enforces enterprise-grade resilience and financial governance across LiteLLM Proxy gateway integrations. Mandates deterministic model-tier fallback routing, bounded semantic cache TTLs with similarity thresholds, and strict upstream token budget enforcement to eliminate unmitigated failover loops and cascade billing overages. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Primary-Model Routing with Indefinite Semantic Caching and Unchecked Rate-Limit Exhaustion Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftUnbounded Primary-Model Routing with Indefinite Semantic Caching and Unchecked Rate-Limit ExhaustionCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Services MUST route inference requests through virtual model groups (e.g., 'production-tier-1') configured with multi-provider fallbacks (e.g., OpenAI -> Azure -> Anthropic) rather than querying concrete upstream provider model IDs directly.

Services MUST route inference requests through virtual model groups (e.g., 'production-tier-1') configured with multi-provider fallbacks (e.g., OpenAI -> Azure -> Anthropic) rather than querying concrete upstream provider model IDs directly.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST specify bounded TTLs (maximum 86400 seconds) and similarity score thresholds (minimum 0.85 cosine similarity) for semantic cache configurations to prevent prompt drift and stale responses.

Engineers MUST specify bounded TTLs (maximum 86400 seconds) and similarity score thresholds (minimum 0.85 cosine similarity) for semantic cache configurations to prevent prompt drift and stale responses.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Engineers MUST attach tenant budgeting identifiers ('user', 'team_id', or 'end_user_id') and hard spend caps to proxy invocation headers to ensure the proxy's BudgetManager halts execution before quota exhaustion.

Engineers MUST attach tenant budgeting identifiers ('user', 'team_id', or 'end_user_id') and hard spend caps to proxy invocation headers to ensure the proxy's BudgetManager halts execution before quota exhaustion.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Code MUST handle HTTP 429 (Rate Limit) and HTTP 503 (Service Unavailable) status codes with randomized exponential backoff and circuit-breaker tripping, rather than aggressive immediate retries.

Code MUST handle HTTP 429 (Rate Limit) and HTTP 503 (Service Unavailable) status codes with randomized exponential backoff and circuit-breaker tripping, rather than aggressive immediate retries.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Components NEVER execute raw unbounded streaming requests without client-side timeout guards (AbortController) and downstream disconnect hooks to prevent phantom token burn.

Components NEVER execute raw unbounded streaming requests without client-side timeout guards (AbortController) and downstream disconnect hooks to prevent phantom token burn.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is LiteLLM Proxy Multi-Provider Failover, Semantic Cache TTLs, and Budget Routing critical for AI coding assistants?

Enforces enterprise-grade resilience and financial governance across LiteLLM Proxy gateway integrations. Mandates deterministic model-tier fallback routing, bounded semantic cache TTLs with similarity thresholds, and strict upstream token budget enforcement to eliminate unmitigated failover loops and cascade billing overages. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Primary-Model Routing with Indefinite Semantic Caching and Unchecked Rate-Limit Exhaustion) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related BACKEND Rules

View all rules →
backend
42

Fastify 5 Schema Compilers & Encapsulation Production Standard

Fastify relies on lexical encapsulation contexts to sandbox decorators, hooks, and compiled validation/serialization schemas without memory leaks or cross-tenant bleed. In Fastify 5, misuse of global compilers or failing to encapsulate plugin boundaries causes compiled `fast-json-stringify` serializers and Ajv validators to either crash on duplicate IDs, leak private data across sibling scopes, or bypass JIT compilation optimizations. Production services must enforce strict encapsulation hierarchies and type-safe schema compilation across every routing subtree.

backend
68

Bun Embedded SQLite WAL Mode & Transaction Synchronization Standard

Production database standard for bun:sqlite. Mandates enabling Write-Ahead Logging (PRAGMA journal_mode = WAL), busy timeouts, and using db.transaction() for atomic operations without lock starvation.

backend
68

Drizzle ORM Batch API & Interactive Transaction Isolation Standard

Production database standard for Drizzle ORM (PostgreSQL & MySQL). Mandates using db.batch() over HTTP drivers to prevent round-trip waterfalls, explicit db.transaction() with serializable/read-committed isolation, and strictly avoids unawaited query promises.

1 Rule in Stack

LiteLLM