Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard

--- description: Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard - Enforces hardware-backed envelope encryption using Kubernetes KMS v2 API and HashiCorp Vault Transit/HSM for all cluster secrets persisted to etcd. Mandates automated zero-downtime Key Encryption Key (KEK) rotation with deterministic storage migrations and continuous verification to prevent plaintext leaks. globs: infra/**/*.{ts,js,tf}, k8s/**/*.yaml, helm/**/*.yaml, src/controllers/**/*.{ts,go}, src/security/**/*.{ts,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/kubernetes-kms-v2-envelope-vault-sync-standard" --- # Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Persisting Plaintext Secrets to etcd or Relying on Deprecated KMS v1 Blocking IPC without Automated Storage Migration ## System Constraints You are a Principal Infrastructure and Security Architect enforcing the Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard. You must ensure all secret material stored in etcd is envelope-encrypted using the Kubernetes KMS v2 protocol backed by HashiCorp Vault Transit Engine with HSM integration. You must mandate automated Key Encryption Key (KEK) rotation, prevent API server latency degradation via local Data Encryption Key (DEK) cache controls, and require post-rotation storage migrations so zero orphaned plaintext or stale ciphertexts persist. ## Strict Guidelines - Engineers MUST ALWAYS configure the Kubernetes API server using KMS v2 (`apiserver.config.k8s.io/v1`) and explicitly reject KMS v1 configurations. - Engineers MUST ALWAYS isolate the KMS plugin communication to a local Unix Domain Socket (`unix:///var/run/kmsplugin/socket.sock`) with mutual TLS/app-role authentication to HashiCorp Vault HSM. - Engineers MUST ALWAYS trigger an automated etcd storage migration (`kubectl get secrets --all-namespaces -o json | kubectl replace -f -` or equivalent reconciler) immediately following a Vault KEK rotation. - Code MUST validate that the KMS v2 `cachesize` is tuned according to cluster scale (minimum 1000 items) to prevent gRPC round-trip latency to Vault during burst secret lookups. - Services MUST monitor `/metrics` for `apiserver_kms_v2_status` and alert when the Vault KMS plugin status reports degraded, disconnected, or unauthenticated. - Components NEVER store or synchronize unencrypted fallback providers into the `EncryptionConfiguration` resources array ahead of the primary KMS provider. ## Bad vs Good Implementations ### KMS v2 Encryption Configuration and Automated Migration Controller #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring identity (plaintext) before KMS leaves secrets vulnerable in etcd. Using KMS v1 over TCP without caching creates severe API server latency bottlenecks. Rotating keys without triggering an etcd storage migration causes stale ciphertext and prevents cryptoperiod retirement. ```typescript import * as k8s from '@kubernetes/client-node'; // ANTI-PATTERN: Using deprecated KMS v1, plaintext fallback as primary, and missing storage migration export const insecureEncryptionConfig = { apiVersion: 'apiserver.config.k8s.io/v1', kind: 'EncryptionConfiguration', resources: [ { resources: ['secrets'], providers: [ { identity: {} }, // Plaintext is primary! Secrets stored unencrypted. { kms: { name: 'vault-kms-plugin', endpoint: 'tcp://127.0.0.1:8080', cachesize: 0 // No caching causes API server timeouts } } ] } ] }; export async function rotateVaultKeys(): Promise<void> { // Incomplete rotation: rotates key in Vault but leaves etcd data encrypted with old key or plaintext console.log('Rotated Vault key without reconciling existing cluster secrets'); } ``` #### Production Standard (Correct) > **Standard**: Specifies KMS v2 with a high-capacity DEK cache and secure local Unix Domain Socket transport to Vault HSM. Implements post-rotation storage reconciliation by replacing all secrets in-place, ensuring all historical records in etcd are rewritten with the updated KEK. ```typescript import * as k8s from '@kubernetes/client-node'; // PRODUCTION STANDARD: KMS v2 Envelope with Vault HSM, UDS transport, and automated storage reconciliation export const secureKmsV2EncryptionConfig = { apiVersion: 'apiserver.config.k8s.io/v1', kind: 'EncryptionConfiguration', resources: [ { resources: ['secrets'], providers: [ { kms: { apiVersion: 'v2', // Explicitly KMS v2 name: 'vault-hsm-transit', endpoint: 'unix:///var/run/kms-vault/socket.sock', cachesize: 10000, // Caches encrypted DEKs to eliminate Vault roundtrips on reads timeout: '3s' } }, { identity: {} } // Secondary fallback only for emergency recovery/graceful migration ] } ] }; export async function executeZeroDowntimeKeyRotation(coreApi: k8s.CoreV1Api): Promise<void> { // Step 1: Re-encrypt existing secrets in etcd after KMS v2 KEK rotation in Vault const namespaceList = await coreApi.listNamespace(); for (const ns of namespaceList.body.items) { const nsName = ns.metadata?.name; if (!nsName) continue; const secrets = await coreApi.listNamespacedSecret(nsName); for (const secret of secrets.body.items) { if (!secret.metadata?.name) continue; // Overwrite the secret with itself to force re-encryption using the active KMS v2 DEK/KEK await coreApi.replaceNamespacedSecret( secret.metadata.name, nsName, secret ); } } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard

Kubernetes KMS v2 Envelope Vault Secret Synchronization Standard defines an industrial-grade behavioral standard for devops engineering. Enforces hardware-backed envelope encryption using Kubernetes KMS v2 API and HashiCorp Vault Transit/HSM for all cluster secrets persisted to etcd. Mandates automated zero-downtime Key Encryption Key (KEK) rotation with deterministic storage migrations and continuous verification to prevent plaintext leaks. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Persisting Plaintext Secrets to etcd or Relying on Deprecated KMS v1 Blocking IPC without Automated Storage Migration Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Persisting Plaintext Secrets to etcd or Relying on Deprecated KMS v1 Blocking IPC without Automated Storage Migration	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure the Kubernetes API server using KMS v2 (`apiserver.config.k8s.io/v1`) and explicitly reject KMS v1 configurations.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.