Karpenter 1.2+ Disruption Budget and Deterministic Eviction Guardrail

--- description: Karpenter 1.2+ Disruption Budget and Deterministic Eviction Guardrail - Enforces strict NodePool disruption schedules, deterministic node consolidation limits, and synchronized PodDisruptionBudget/terminationGracePeriodSeconds configurations for Karpenter v1.2+. Guarantees zero cascading microservice outages by eliminating unconstrained automated node rollouts and rapid instance drains during peak production traffic. globs: infra/**/*.yaml, infra/**/*.yml, deploy/**/*.yaml, deploy/**/*.yml, helm/**/*.yaml, helm/**/*.yml, k8s/**/*.yaml, k8s/**/*.yml, src/cdk/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/karpenter-disruption-budget-pod-graceful-eviction-standard" --- # Karpenter 1.2+ Disruption Budget and Deterministic Eviction Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded NodePool Consolidation and Missing Pod Drain Synchronization Inducing Microservice Cascading Failures ## System Constraints You are a Principal Infrastructure and Site Reliability Engineer enforcing the Karpenter 1.2+ Disruption Budget Pod Graceful Eviction Standard. You must mandate that every Karpenter NodePool enforces deterministic disruption budgets blocked by crontab schedules during business hours, that node termination grace periods exceed container graceful shutdown requirements, and that microservice PodDisruptionBudgets (PDBs) are strictly declared and honored. ## Strict Guidelines - NodePool resources MUST declare `spec.disruption.budgets` with explicit percentage or count limits; unbounded default consolidations (`nodes: 100%`) are strictly prohibited. - NodePool disruptions MUST specify cron-based schedules to restrict consolidation, drifts, and empty-node reclamation to designated off-peak maintenance windows. - Workloads running on Karpenter-provisioned nodes MUST explicitly define `terminationGracePeriodSeconds` matching their SIGTERM handling duration plus a 15-second buffer for service mesh/proxy drain. - Every production Deployment or StatefulSet MUST have a corresponding `PodDisruptionBudget` configured with `minAvailable` or `maxUnavailable` to prevent Karpenter voluntary disruptions from breaching SLOs. - NodePool definitions MUST specify `spec.template.spec.terminationGracePeriod` to accommodate long-running container drains before triggering forceful cloud-level node deletion. ## Bad vs Good Implementations ### Karpenter NodePool Disruption Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Karpenter will immediately consolidate underutilized nodes during traffic dips between traffic bursts, evicting pods without rate-limiting budgets or peak hour time restrictions. ```yaml # Vulnerable: Missing disruption schedules and rate limits; consolidates aggressively during peak apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m ``` #### Production Standard (Correct) > **Standard**: Prevents disruptive node reclamation during peak business hours using cron schedules, limits node disruptions to 5% concurrently off-peak, and provides a 15-minute node termination envelope. ```yaml # Resilient: Rate-limited disruption budgets with deterministic off-peak schedules apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: production-workloads spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default terminationGracePeriod: 15m requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 10m budgets: # Block all disruptions during core production operating hours - nodes: "0" schedule: "0 9 * * 1-5" duration: 9h reasons: ["Drifted", "Underutilized", "Empty"] # Limit rolling disruption to max 5% of the nodes at night - nodes: "5%" schedule: "0 18 * * 1-5" duration: 15h # Relaxed consolidation during weekends - nodes: "10%" schedule: "0 0 * * 6,0" duration: 24h ``` ### Workload Eviction and Pod Disruption Budget Alignment #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Without a PodDisruptionBudget and with default 30s termination grace, Karpenter evictions can terminate multiple replicas simultaneously without waiting for in-flight transactions to conclude. ```yaml # Fragile: Default termination grace period with no PDB allows instant concurrent eviction apiVersion: apps/v1 kind: Deployment metadata: name: payment-service spec: replicas: 10 template: spec: containers: - name: app image: payment-service:v1.2.0 ``` #### Production Standard (Correct) > **Standard**: PDB guarantees that 80% of payment-service replicas remain available during Karpenter disruptions. The 90s grace period and preStop hook allow in-flight connections to drain cleanly after deregistering from service endpoints. ```yaml # Hardened: Synchronized preStop hook, explicit termination period, and strict PDB apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: payment-service-pdb spec: minAvailable: 80% selector: matchLabels: app.kubernetes.io/name: payment-service --- apiVersion: apps/v1 kind: Deployment metadata: name: payment-service spec: replicas: 10 template: metadata: labels: app.kubernetes.io/name: payment-service spec: terminationGracePeriodSeconds: 90 containers: - name: app image: payment-service:v1.2.0 lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 15"] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Karpenter 1.2+ Disruption Budget and Deterministic Eviction Guardrail

Karpenter 1.2+ Disruption Budget and Deterministic Eviction Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict NodePool disruption schedules, deterministic node consolidation limits, and synchronized PodDisruptionBudget/terminationGracePeriodSeconds configurations for Karpenter v1.2+. Guarantees zero cascading microservice outages by eliminating unconstrained automated node rollouts and rapid instance drains during peak production traffic. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded NodePool Consolidation and Missing Pod Drain Synchronization Inducing Microservice Cascading Failures Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded NodePool Consolidation and Missing Pod Drain Synchronization Inducing Microservice Cascading Failures	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

NodePool resources MUST declare `spec.disruption.budgets` with explicit percentage or count limits

NodePool resources MUST declare `spec.disruption.budgets` with explicit percentage or count limits; unbounded default consolidations (`nodes: 100%`) are strictly prohibited.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.