Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard

--- description: Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard - Enforces deterministic coordination between Karpenter v1 node consolidation policies, application PodDisruptionBudgets (PDBs), and container lifecycle graceful termination hooks. Guarantees zero transient 502/504 connection drops during Spot node interruptions and consolidation drains by synchronizing terminationGracePeriodSeconds with ingress draining and preStop sleeps. globs: deploy/k8s/**/*.yaml, deploy/k8s/**/*.yml, infra/karpenter/**/*.yaml, infra/karpenter/**/*.yml, helm/**/*.yaml, helm/**/*.yml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/karpenter-v1-pdb-coordinated-drain-standard" --- # Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Uncoordinated Karpenter Node Drain Causing Abrupt Pod Termination and Upstream 502/504 Gateway Errors ## System Constraints You are a Principal Cloud Architect enforcing the Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard. Mandate that every application workload running on Karpenter-managed Spot or Consolidated nodes possesses an active PodDisruptionBudget (PDB), an explicit preStop hook delay allowing kube-proxy and Ingress controller endpoints to flush, and a terminationGracePeriodSeconds strictly calibrated above the sum of the preStop delay and application connection-draining timeout. Reject any NodePool configuration where consolidationPolicy or expireAfter conflicts with workload draining invariants. ## Strict Guidelines - Engineers MUST ALWAYS define a PodDisruptionBudget (`minAvailable` or `maxUnavailable`) alongside Deployments to ensure Karpenter node draining respects workload availability thresholds. - Engineers MUST configure a `preStop` hook with an active sleep (`sleep 10` to `sleep 20`) to allow Kubernetes endpoint propagation and Cloud Load Balancer / Ingress deregistration before the main process receives SIGTERM. - Engineers MUST ensure `terminationGracePeriodSeconds` is strictly greater than the sum of `preStop` sleep duration and the application's internal graceful shutdown deadline. - Engineers MUST NOT configure NodePool `consolidationPolicy: WhenEmptyOrUnderutilized` without a corresponding `expireAfter` or `budgets` array that prevents simultaneous draining of critical node sets. - Engineers MUST NEVER set `terminationGracePeriodSeconds` lower than 30s on production internet-facing services subject to Karpenter Spot disruption notices. ## Bad vs Good Implementations ### Karpenter NodePool and Application Manifest Draining Synchronization #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Karpenter drains the node upon consolidation or Spot interruption. Without a preStop hook, the Pod receives SIGTERM instantly before Ingress and EndpointSlice updates propagate, causing load balancers to route in-flight HTTP requests to a dying pod (502/504 Bad Gateway). Without a PDB, Karpenter can evict all replicas simultaneously. ```yaml # Anti-pattern: Missing preStop delay, uncoordinated timeouts, missing PDB apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m --- apiVersion: apps/v1 kind: Deployment metadata: name: payment-api spec: replicas: 4 template: spec: containers: - name: api image: payment-api:v1.2.0 # Missing lifecycle preStop and explicit terminationGracePeriodSeconds ports: - containerPort: 8080 ``` #### Production Standard (Correct) > **Standard**: The PodDisruptionBudget prevents Karpenter from evicting more than 25% of replicas at once. When eviction triggers, the preStop hook pauses SIGTERM for 15s to allow EndpointSlice updates across kube-proxy and ALBs. The 60s terminationGracePeriodSeconds provides 45s of remaining buffer for active requests to finish cleanly. ```yaml # Production Invariant: Synchronized PDB, preStop flush, and Karpenter v1 Disruption Budget apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-api-pdb spec: minAvailable: 75% selector: matchLabels: app: payment-api --- apiVersion: apps/v1 kind: Deployment metadata: name: payment-api spec: replicas: 4 selector: matchLabels: app: payment-api template: metadata: labels: app: payment-api spec: terminationGracePeriodSeconds: 60 containers: - name: api image: payment-api:v1.2.0 lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 15"] ports: - containerPort: 8080 --- apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m budgets: - nodes: "10%" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard

Karpenter v1 Consolidated Pod Disruption Budget Coordinated Drain Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic coordination between Karpenter v1 node consolidation policies, application PodDisruptionBudgets (PDBs), and container lifecycle graceful termination hooks. Guarantees zero transient 502/504 connection drops during Spot node interruptions and consolidation drains by synchronizing terminationGracePeriodSeconds with ingress draining and preStop sleeps. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Uncoordinated Karpenter Node Drain Causing Abrupt Pod Termination and Upstream 502/504 Gateway Errors Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Uncoordinated Karpenter Node Drain Causing Abrupt Pod Termination and Upstream 502/504 Gateway Errors	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS define a PodDisruptionBudget (`minAvailable` or `maxUnavailable`) alongside Deployments to ensure Karpenter node draining respects workload availability thresholds.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.