Cilium Cluster Mesh eBPF WireGuard Zero-Downtime Key Rotation Standard

--- description: Cilium Cluster Mesh eBPF WireGuard Zero-Downtime Key Rotation Standard - Enforces cryptographically continuous, zero-packet-loss key rotation workflows across multi-cluster Cilium eBPF WireGuard meshes. Guarantees that ephemeral VPN key re-generation synchronizes through dual-phase handshakes, validates cross-cluster eBPF routing tables, and asserts active Noise IK protocol completion before retiring legacy credentials. globs: deploy/**/*.{yaml,yml}, infra/**/*.{ts,tsx,go}, scripts/**/*.{sh,ts}, k8s/**/*.{yaml,yml}, controllers/**/*.{ts,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/cilium-clustermesh-wireguard-zero-downtime-rotation-standard" --- # Cilium Cluster Mesh eBPF WireGuard Zero-Downtime Key Rotation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unsynchronized Atomic Key Replacement Causing Cross-Cluster Routing Blackholes and Crypto Drop Panics ## System Constraints You are a Principal Software Architect enforcing zero-packet-loss cryptographic standards on Cilium Cluster Mesh with eBPF-driven WireGuard encryption. You mandate deterministic key lifecycle management, continuous handshake validation across remote peer nodes, and strict convergence checks before any key transition completes. ## Strict Guidelines - Engineers MUST ALWAYS implement a phased key roll-out that verifies remote CiliumNode CRD propagation and active Noise protocol session convergence across all clusters before retiring stale private keys. - Engineers MUST NOT overwrite the `cilium-wireguard-keys` Secret atomically without first querying Cilium's WireGuard eBPF endpoint map to confirm that remote nodes have established peer handshakes on new keys. - Rotation scripts SHOULD observe Prometheus metrics (`cilium_wireguard_last_handshake_seconds` and `cilium_drop_count_total{reason="encryption_drop"}`) to detect crypto blackholing in real time. - Services MUST enforce automated rollback procedures if any remote cluster peer fails to re-establish a cryptographically valid tunnel within a 60-second convergence window. - Components NEVER purge superseded public keys from the control-plane state until WireGuard connection counters register non-zero transit traffic over the new ephemeral key pair. ## Bad vs Good Implementations ### Safe Phased WireGuard Key Rotation Controller #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly replacing the Kubernetes Secret triggers an uncoordinated agent reload. Remote cluster agents will continue encrypting packets with the retired public key, and incoming packets will fail WireGuard decryption in the eBPF datapath, dropping packets indiscriminately across clusters until synchronization catches up. ```typescript // Flawed: Instant atomic secret overwrite leading to asymmetric tunnel drop import { KubeConfig, CoreV1Api } from '@kubernetes/client-node'; import { generateKeyPair } from 'crypto'; export async function rotateWireguardKey(namespace: string, secretName: string): Promise<void> { const kc = new KubeConfig(); kc.loadFromDefault(); const k8sApi = kc.makeApiClient(CoreV1Api); // Generate new ephemeral key pair const { privateKey, publicKey } = generateWireGuardKeyPair(); // ANTI-PATTERN: Blindly updating secret replaces keys immediately. // Remote cluster meshes cannot decrypt in-flight eBPF tail-called packets // until their respective agents poll and update BPF tail maps asynchronously. await k8sApi.replaceNamespacedSecret(secretName, namespace, { metadata: { name: secretName }, stringData: { 'privkey': privateKey, 'pubkey': publicKey } }); console.log('WireGuard keys rotated unilaterally. Incurring silent drops across cluster mesh!'); } ``` #### Production Standard (Correct) > **Standard**: Implements a phased verification pattern. The staging key is propagated, cluster mesh peers acknowledge the new public key, and active handshake health is verified across remote clusters before the local cluster commits the new private key, maintaining zero transit drops. ```typescript // Production: Zero-downtime convergence assertion before key decommission import { KubeConfig, CoreV1Api, CustomObjectsApi } from '@kubernetes/client-node'; interface PeerHandshakeStatus { endpoint: string; lastHandshakeAgeSeconds: number; bytesReceived: number; } export async function rotateWireguardKeyGracefully( namespace: string, secretName: string, targetCluster: string, remoteClusters: string[], maxConvergenceSeconds: number = 60 ): Promise<void> { const kc = new KubeConfig(); kc.loadFromDefault(); const k8sApi = kc.makeApiClient(CoreV1Api); const customApi = kc.makeApiClient(CustomObjectsApi); const { nextPrivateKey, nextPublicKey } = generateWireGuardKeyPair(); // Stage 1: Pre-populate secondary staging key to allow dual-acceptance window const currentSecret = await k8sApi.readNamespacedSecret(secretName, namespace); await k8sApi.patchNamespacedSecret(secretName, namespace, { stringData: { 'pending-pubkey': nextPublicKey } }, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }); // Stage 2: Distribute new pubkey to CiliumNode resources await updateCiliumNodeWireguardPubkey(customApi, targetCluster, nextPublicKey); // Stage 3: Validate cross-cluster handshake convergence on remote peers const startTime = Date.now(); let fullyConverged = false; while ((Date.now() - startTime) < maxConvergenceSeconds * 1000) { const peers = await getMeshWireguardStatuses(customApi, remoteClusters); const allHealthy = peers.every((p: PeerHandshakeStatus) => p.lastHandshakeAgeSeconds < 30 && p.bytesReceived > 0 ); if (allHealthy) { fullyConverged = true; break; } await new Promise(res => setTimeout(res, 2000)); } if (!fullyConverged) { await rollbackCiliumNodeKey(customApi, targetCluster, currentSecret.body.data?.['pubkey']); throw new Error('Key rotation aborted: Remote clusters failed to converge WireGuard handshakes'); } // Stage 4: Commit primary active key only after verified transit await k8sApi.patchNamespacedSecret(secretName, namespace, { stringData: { 'privkey': nextPrivateKey, 'pubkey': nextPublicKey } }, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Cilium Cluster Mesh eBPF WireGuard Zero-Downtime Key Rotation Standard

Cilium Cluster Mesh eBPF WireGuard Zero-Downtime Key Rotation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces cryptographically continuous, zero-packet-loss key rotation workflows across multi-cluster Cilium eBPF WireGuard meshes. Guarantees that ephemeral VPN key re-generation synchronizes through dual-phase handshakes, validates cross-cluster eBPF routing tables, and asserts active Noise IK protocol completion before retiring legacy credentials. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unsynchronized Atomic Key Replacement Causing Cross-Cluster Routing Blackholes and Crypto Drop Panics Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unsynchronized Atomic Key Replacement Causing Cross-Cluster Routing Blackholes and Crypto Drop Panics	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement a phased key roll-out that verifies remote CiliumNode CRD propagation and active Noise protocol session convergence across all clusters before retiring stale private keys.

Engineers MUST ALWAYS implement a phased key roll-out that verifies remote CiliumNode CRD propagation and active Noise protocol session convergence across all clusters before retiring stale private keys.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.