Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Blind In-Place Memory Contraction Causing Instant Kernel OOM Kills and cgroup v2 Lock Contention) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail

--- description: Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail - Enforces strict validation, phased downscaling, and runtime cgroup headroom checks during live Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal container OOM kills and kernel cgroup lock contention caused by aggressive memory downscaling beneath active working set sizes or without proper resizePolicy declarations. globs: k8s/**/*.yaml, deploy/**/*.yaml, src/operators/**/*.ts, src/controllers/**/*.ts, src/autoscale/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-in-place-resource-mutation-guardrail" --- # Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind In-Place Memory Contraction Causing Instant Kernel OOM Kills and cgroup v2 Lock Contention ## System Constraints You are a Principal Software Architect enforcing production-grade Kubernetes In-Place Pod Resource Mutation standards. You must ensure all automated live CPU and memory patch operations evaluate active runtime memory usage (working_set_bytes), enforce non-disruptive resize policies (NotRequired), verify Node Allocatable margins, and throttle step-down limits to prevent SIGKILL terminations and kernel-level cgroup locks. ## Strict Guidelines - Engineers MUST ALWAYS define container `resizePolicy` explicitly with `restartPolicy: NotRequired` for CPU and Memory when in-place scaling is intended. - Engineers MUST NOT reduce container `resources.limits.memory` below the current `container_memory_working_set_bytes` plus an operational buffer (minimum 15% headroom). - Services MUST execute memory downscaling via iterative decrements (max 10-20% delta per phase with verification intervals) to allow the kernel page cache and runtime runtimes (e.g., V8, JVM) to reclaim and release uncommitted memory. - Controllers MUST verify the target Node's allocatable capacity and inspect `pod.status.resize` before patching `spec.containers[*].resources` to prevent scheduler-induced Pod rejections or `Infeasible` states. - Components NEVER issue in-place mutation patches while `pod.status.resize` is `Infeasible` or `Proposed` without resolving the underlying scheduling constraint. ## Bad vs Good Implementations ### In-Place Pod Resize Controller Patch with Runtime Guardrails #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly patching `spec.containers[*].resources` without inspecting current live memory usage causes an immediate OOM kill if the new limit drops below active resident memory. It also ignores `pod.status.resize` and misses `resizePolicy: NotRequired`, potentially triggering unintended pod termination and node-level cgroup lock contention. ```typescript import * as k8s from '@kubernetes/client-node'; // ANTI-PATTERN: Patching resources directly without reading working_set_bytes, // checking resizePolicy, or verifying in-progress resize status. export async function mutatePodResourcesUnsafe( k8sApi: k8s.CoreV1Api, namespace: string, podName: string, containerName: string, newMemoryLimit: string, newCpuLimit: string ): Promise<void> { const patch = { spec: { containers: [{ name: containerName, resources: { limits: { memory: newMemoryLimit, cpu: newCpuLimit }, requests: { memory: newMemoryLimit, cpu: newCpuLimit } } }] } }; // Instantly drops limit: if newMemoryLimit < working_set_bytes, kernel OOMs the container immediately. await k8sApi.patchNamespacedPod( podName, namespace, patch, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } } ); } ``` #### Production Standard (Correct) > **Standard**: Validates the pod's current resize state, checks that container `resizePolicy` avoids unexpected restarts, and queries real-time cgroup `working_set_bytes` to enforce a 15% headroom buffer, safely avoiding both immediate SIGKILL OOM events and kernel cgroup lock contention. ```typescript import * as k8s from '@kubernetes/client-node'; interface ResourceMetrics { containerName: string; memoryWorkingSetBytes: bigint; } export async function mutatePodResourcesSafe( k8sApi: k8s.CoreV1Api, namespace: string, podName: string, containerName: string, targetMemoryBytes: bigint, targetCpuMillis: number, getLiveMetrics: (pod: string, container: string) => Promise<ResourceMetrics> ): Promise<void> { const { body: pod } = await k8sApi.readNamespacedPod(podName, namespace); const container = pod.spec?.containers.find(c => c.name === containerName); if (!container) { throw new Error(`Container ${containerName} not found in pod ${podName}`); } // 1. Invariant: Resize policy must be NotRequired to prevent restart const memPolicy = container.resizePolicy?.find(p => p.resourceName === 'memory'); if (memPolicy?.restartPolicy === 'RestartContainer') { throw new Error(`Unsafe mutation: memory resizePolicy requires container restart`); } // 2. Invariant: Pod must not have pending or infeasible resize transitions if (pod.status?.resize === 'Infeasible' || pod.status?.resize === 'Proposed') { throw new Error(`Pod ${podName} has active/infeasible resize state: ${pod.status.resize}`); } // 3. Invariant: Memory downscale must retain a minimum 15% safety margin over current working set const metrics = await getLiveMetrics(podName, containerName); const minimumSafeMemory = (metrics.memoryWorkingSetBytes * 115n) / 100n; if (targetMemoryBytes < minimumSafeMemory) { throw new Error( `Target memory (${targetMemoryBytes}B) falls below safe active headroom (${minimumSafeMemory}B). ` + `Risk of instant kernel OOM kill.` ); } // 4. Construct strategic merge patch with explicit in-place constraints const patch = { spec: { containers: [{ name: containerName, resources: { limits: { memory: `${targetMemoryBytes}`, cpu: `${targetCpuMillis}m` }, requests: { memory: `${targetMemoryBytes}`, cpu: `${targetCpuMillis}m` } } }] } }; await k8sApi.patchNamespacedPod( podName, namespace, patch, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } } ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail

Kubernetes In-Place Pod Resource Mutation & Safe-Drain Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict validation, phased downscaling, and runtime cgroup headroom checks during live Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal container OOM kills and kernel cgroup lock contention caused by aggressive memory downscaling beneath active working set sizes or without proper resizePolicy declarations. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind In-Place Memory Contraction Causing Instant Kernel OOM Kills and cgroup v2 Lock Contention Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind In-Place Memory Contraction Causing Instant Kernel OOM Kills and cgroup v2 Lock Contention	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS define container `resizePolicy` explicitly with `restartPolicy

Engineers MUST ALWAYS define container `resizePolicy` explicitly with `restartPolicy: NotRequired` for CPU and Memory when in-place scaling is intended.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.