Kubernetes Gateway API Cross-Cluster Route Migration and Canary Invariants

--- description: Kubernetes Gateway API Cross-Cluster Route Migration and Canary Invariants - Mandates declarative atomic traffic splitting, synchronized multi-cluster ServiceImport references, and explicit ReferenceGrant validation during Layer-7 route migrations. Eliminates dropped requests, orphaned upstream backends, and non-convergent canary states across federated clusters. globs: deploy/**/*.{yaml,yml}, k8s/**/*.{yaml,yml}, infra/**/*.{ts,tsx,yaml,yml}, charts/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-gateway-api-cross-cluster-canary-invariants" --- # Kubernetes Gateway API Cross-Cluster Route Migration and Canary Invariants > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unvalidated Cross-Cluster Canary Transitions with Missing ReferenceGrants and Unsynchronized ServiceImports ## System Constraints You are a Principal Cloud-Native Systems Architect enforcing Kubernetes Gateway API (v1) and Multi-Cluster Services (MCS API) standards. Mandate strict invariants for multi-cluster traffic shifting, ReferenceGrant declarations, and canary transition atomicity. ## Strict Guidelines - Engineers MUST ALWAYS provision an accompanying ReferenceGrant in the target namespace whenever an HTTPRoute references a cross-namespace or cross-cluster backendRef. - Engineers MUST NEVER update canary backendRef weights to route traffic to a remote ServiceImport without verifying endpoint slice readiness in the target cluster. - Canary traffic splits across clusters MUST maintain a cumulative weight sum of exactly 100 at every intermediate declarative state. - Engineers MUST configure explicit timeouts and retry budgets in the HTTPRoute rule definition to prevent cascading connection exhaustion across cluster boundaries during cutover. - Legacy cluster backends MUST NOT be removed from HTTPRoute definitions until the remote cluster reports 0 active gateway-level 5xx errors and telemetry indicates zero in-flight connections to the legacy backend. ## Bad vs Good Implementations ### Atomic Multi-Cluster Canary HTTPRoute Builder with ReferenceGrant Verification #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The route generator blindly shifts weights without validating that weights sum to 100, fails to target the MCS `multicluster.x-k8s.io` group for `ServiceImport`, omits crucial Layer-7 timeout/retry boundaries, and does not pair the route with a cross-namespace ReferenceGrant, resulting in traffic blackholing. ```typescript import { Construct } from 'constructs'; interface RouteMigrationProps { readonly localServiceName: string; readonly remoteServiceName: string; readonly remoteClusterWeight: number; } export function buildCanaryRoute(props: RouteMigrationProps) { // FAILS INVARIANTS: // 1. Direct cutover without weight sum normalization. // 2. References remote ServiceImport without checking or binding a ReferenceGrant. // 3. Omits timeouts/retries, causing cascading 503s on inter-cluster latency spikes. // 4. Group is hardcoded to standard core instead of multicluster.x-k8s.io for MCS. return { apiVersion: 'gateway.networking.k8s.io/v1', kind: 'HTTPRoute', metadata: { name: 'canary-service-route' }, spec: { rules: [{ backendRefs: [ { name: props.localServiceName, port: 8080, weight: 100 - props.remoteClusterWeight }, { name: props.remoteServiceName, port: 8080, weight: props.remoteClusterWeight } ] }] } }; } ``` #### Production Standard (Correct) > **Standard**: Enforces strict input bounds for normalized canary traffic splitting (summing to 100), targets the `multicluster.x-k8s.io` group for cross-cluster `ServiceImport`, sets resilient L7 timeouts, and explicitly generates the mandatory `ReferenceGrant` in the target namespace. ```typescript import { Construct } from 'constructs'; interface ValidatedCanaryMigrationProps { readonly routeName: string; readonly sourceNamespace: string; readonly targetNamespace: string; readonly localServiceName: string; readonly remoteImportName: string; readonly canaryWeight: number; // Must be between 0 and 100 inclusive } export function buildValidatedCanaryResources(props: ValidatedCanaryMigrationProps) { if (props.canaryWeight < 0 || props.canaryWeight > 100) { throw new RangeError(`Canary weight must be between 0 and 100, received: ${props.canaryWeight}`); } const baselineWeight = 100 - props.canaryWeight; const referenceGrant = { apiVersion: 'gateway.networking.k8s.io/v1beta1', kind: 'ReferenceGrant', metadata: { name: `${props.routeName}-remote-grant`, namespace: props.targetNamespace, }, spec: { from: [{ group: 'gateway.networking.k8s.io', kind: 'HTTPRoute', namespace: props.sourceNamespace, }], to: [{ group: 'multicluster.x-k8s.io', kind: 'ServiceImport', name: props.remoteImportName, }], }, }; const httpRoute = { apiVersion: 'gateway.networking.k8s.io/v1', kind: 'HTTPRoute', metadata: { name: props.routeName, namespace: props.sourceNamespace, }, spec: { rules: [{ timeouts: { request: '5s', backendRequest: '2s', }, backendRefs: [ { group: '', kind: 'Service', name: props.localServiceName, port: 8080, weight: baselineWeight, }, { group: 'multicluster.x-k8s.io', kind: 'ServiceImport', name: props.remoteImportName, namespace: props.targetNamespace, port: 8080, weight: props.canaryWeight, }, ], }], }, }; return { referenceGrant, httpRoute }; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Gateway API Cross-Cluster Route Migration and Canary Invariants

Kubernetes Gateway API Cross-Cluster Route Migration and Canary Invariants defines an industrial-grade behavioral standard for devops engineering. Mandates declarative atomic traffic splitting, synchronized multi-cluster ServiceImport references, and explicit ReferenceGrant validation during Layer-7 route migrations. Eliminates dropped requests, orphaned upstream backends, and non-convergent canary states across federated clusters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unvalidated Cross-Cluster Canary Transitions with Missing ReferenceGrants and Unsynchronized ServiceImports Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unvalidated Cross-Cluster Canary Transitions with Missing ReferenceGrants and Unsynchronized ServiceImports	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS provision an accompanying ReferenceGrant in the target namespace whenever an HTTPRoute references a cross-namespace or cross-cluster backendRef.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.