Envoy Gateway 1.2+ Dynamic Outlier Detection & Egress Circuit-Breaking Standard

--- description: Envoy Gateway 1.2+ Dynamic Outlier Detection & Egress Circuit-Breaking Standard - Enforces deterministic outlier detection and egress circuit breaking via Envoy Gateway 1.2+ BackendTrafficPolicy CRDs across all canary and upstream service meshes. Proactively isolates failing pods and halts cascading failovers during statistical anomaly windows while preserving baseline traffic availability. globs: **/k8s/**/*.yaml, **/manifests/**/*.yaml, **/gateway/**/*.yaml, **/helm/**/*.yaml, deploy/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/envoy-gateway-dynamic-outlier-detection-circuit-breaking" --- # Envoy Gateway 1.2+ Dynamic Outlier Detection & Egress Circuit-Breaking Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Egress Retries Without Outlier Detection Triggering Canary Collapse ## System Constraints You are a Principal Infrastructure Architect enforcing resilient ingress/egress configurations using Envoy Gateway 1.2+. You must ensure every HTTPRoute and Backend reference is safeguarded with a dedicated BackendTrafficPolicy that mandates statistical outlier detection, ejection safety bounds, and connection pool saturation limits to arrest cascading topology failures. ## Strict Guidelines - Engineers MUST ALWAYS attach a valid `BackendTrafficPolicy` (API version `gateway.envoyproxy.io/v1alpha1`) to target HTTPRoutes referencing upstream services. - Engineers MUST configure `outlierDetection` with both `consecutive5xxErrors` and explicit interval/base ejection windows to remove degraded pods deterministically. - Engineers MUST NEVER set `maxEjectionPercent` higher than 50% in production to prevent complete cluster starvation under systemic network partition. - Engineers MUST define explicit `circuitBreaker` thresholds for `maxConnections`, `maxPendingRequests`, and `maxRequests` to prevent upstream connection queue saturation. - Services MUST declare split canary weights using distinct TargetRefs or Backend clusters to allow outlier detection to isolate canary instances independently. ## Bad vs Good Implementations ### Unbounded Route Configuration with Default Retries (Vulnerable) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Routing traffic to canary pods without an accompanying BackendTrafficPolicy allows failing or slow canary replicas to consume all gateway connection buffers, causing cascading timeouts to healthy baseline pods. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: order-service-route namespace: production spec: parentRefs: - name: eg-gateway rules: - matches: - path: type: PathPrefix value: /api/v1/orders backendRefs: - name: order-service-baseline port: 8080 weight: 90 - name: order-service-canary port: 8080 weight: 10 # Missing BackendTrafficPolicy: canary failures will exhaust gateway connection pools # and trigger cascading gateway-wide 503 errors. ``` #### Production Standard (Correct) > **Standard**: The BackendTrafficPolicy enforces connection and pending request bounds alongside proactive outlier detection. If canary pods emit consecutive 5xx/gateway errors within 10 seconds, Envoy dynamically ejects them for 30 seconds up to a safe 50% cap. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: order-service-route namespace: production spec: parentRefs: - name: eg-gateway rules: - matches: - path: type: PathPrefix value: /api/v1/orders backendRefs: - name: order-service-baseline port: 8080 weight: 90 - name: order-service-canary port: 8080 weight: 10 --- apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: order-service-resilience namespace: production spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: order-service-route circuitBreaker: maxConnections: 1024 maxPendingRequests: 128 maxRequests: 2048 maxRetries: 3 outlierDetection: consecutive5xxErrors: 3 interval: 10s baseEjectionTime: 30s maxEjectionPercent: 50 consecutiveGatewayErrors: 3 ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Envoy Gateway 1.2+ Dynamic Outlier Detection & Egress Circuit-Breaking Standard

Envoy Gateway 1.2+ Dynamic Outlier Detection & Egress Circuit-Breaking Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic outlier detection and egress circuit breaking via Envoy Gateway 1.2+ BackendTrafficPolicy CRDs across all canary and upstream service meshes. Proactively isolates failing pods and halts cascading failovers during statistical anomaly windows while preserving baseline traffic availability. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Egress Retries Without Outlier Detection Triggering Canary Collapse Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Egress Retries Without Outlier Detection Triggering Canary Collapse	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS attach a valid `BackendTrafficPolicy` (API version `gateway.envoyproxy.io/v1alpha1`) to target HTTPRoutes referencing upstream services.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.