Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard

--- description: Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard - Enforces a two-phase staging protocol for live vertical scaling under KEP-1287 (In-Place Pod Vertical Scaling). Prevents kernel OOM-killer invocations by guaranteeing that container runtime cgroup v2 memory.max boundaries and kubelet allocated status are observed before expanding workload memory allocations (e.g., JVM heap, V8 heap, or cache pools). globs: infra/**/*.yaml, deploy/**/*.yaml, k8s/**/*.yaml, src/autoscaling/**/*.{ts,go,py}, src/controllers/**/*.{ts,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-in-place-resize-memory-headroom-standard" --- # Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Simultaneously Increasing Workload Heap Consumption and Pod Resource Limits Triggering Instant cgroup v2 OOM Kills ## System Constraints You are a Principal DevOps and Systems Architect enforcing the Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard. You must strictly enforce asynchronous staging of memory limits prior to workload heap expansions, verify container cgroup v2 updates via downward API or status probes, and ensure resize policies specify RestartNotRequired with verified node allocatable headroom. ## Strict Guidelines - Engineers MUST ALWAYS set resizePolicy for memory to 'RestartNotRequired' when leveraging KEP-1287 in-place vertical scaling. - Engineers MUST NOT increase runtime application heap allocations (e.g., MaxRAMPercentage, V8 max_old_space_size) until 'status.containerStatuses[].resourcesAllocated' matches the target spec AND the cgroup v2 'memory.max' file is verified updated. - Workload controllers SHOULD implement a dynamic polling or event-driven hook watching `/sys/fs/cgroup/memory.max` before dispatching SIGUSR1 or JMX commands that expand in-memory buffers. - Automation services MUST verify that target nodes possess unallocated allocatable capacity before patching 'spec.containers[].resources.limits.memory' to avoid Infeasible resize status deadlocks. - Platform operators NEVER downscale memory limits below 'status.containerStatuses[].resources.requests' without initiating a coordinated proactive cache eviction to avoid synchronous kernel OOM faults. ## Bad vs Good Implementations ### Atomic Heap Expansion without Verifying cgroup v2 Allocation (Anti-Pattern) vs Staged Headroom Verification (Compliant) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Patching the Kubernetes Pod resource limits does not immediately update the container's cgroup limits on the host. If the workload expands memory usage before kubelet and container runtime (e.g., containerd) update the underlying cgroup v2 `memory.max`, the Linux kernel OOM killer triggers instantly. ```typescript import * as k8s from '@kubernetes/client-node'; export async function scaleContainerMemory(k8sApi: k8s.CoreV1Api, podName: string, namespace: string, newMemMiB: number) { // Anti-Pattern: Patching pod spec and immediately commanding the application runtime to consume expanded memory. // In-place pod resize is asynchronous; cgroup limits are NOT updated immediately. await k8sApi.patchNamespacedPod( podName, namespace, { spec: { containers: [{ name: 'app', resources: { limits: { memory: `${newMemMiB}Mi` } } }] } }, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } } ); // Workload immediately consumes memory before container runtime updates memory.max -> Kernel OOM Kill! await triggerWorkloadMemoryExpansion(podName, newMemMiB); } async function triggerWorkloadMemoryExpansion(podName: string, targetMiB: number): Promise<void> { /* Triggers heap growth via IPC/HTTP */ } ``` #### Production Standard (Correct) > **Standard**: The compliant code explicitly sets `restartPolicy: 'RestartNotRequired'`, patches the resource spec, and iteratively waits for `status.containerStatuses[].allocatedResources` to confirm that the kubelet and runtime have applied the new cgroup v2 memory limits before signalling the workload to expand its memory footprint. ```typescript import * as k8s from '@kubernetes/client-node'; export async function scaleContainerMemoryStaged( k8sApi: k8s.CoreV1Api, podName: string, namespace: string, containerName: string, newMemBytes: number ): Promise<void> { const targetMemString = `${Math.floor(newMemBytes / (1024 * 1024))}Mi`; // Step 1: Request In-Place Pod Resize via Strategic Merge Patch await k8sApi.patchNamespacedPod( podName, namespace, { spec: { containers: [ { name: containerName, resizePolicy: [{ resourceName: 'memory', restartPolicy: 'RestartNotRequired' }], resources: { limits: { memory: targetMemString }, requests: { memory: targetMemString } } } ] } }, undefined, undefined, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } } ); // Step 2: Poll pod status until containerStatuses reflects allocated resources const maxRetries = 30; let allocated = false; for (let i = 0; i < maxRetries; i++) { const pod = await k8sApi.readNamespacedPodStatus(podName, namespace); const cStatus = pod.body.status?.containerStatuses?.find(c => c.name === containerName); const allocMem = cStatus?.allocatedResources?.memory; if (allocMem === targetMemString) { allocated = true; break; } await new Promise(resolve => setTimeout(resolve, 1000)); } if (!allocated) { throw new Error(`In-Place Resize timed out or rejected for container: ${containerName}`); } // Step 3: Workload is signaled safely only AFTER cgroup allocation is guaranteed await triggerWorkloadMemoryExpansion(podName, Math.floor(newMemBytes / (1024 * 1024))); } async function triggerWorkloadMemoryExpansion(podName: string, targetMiB: number): Promise<void> { /* Safely notify the workload over internal IPC to expand heap limit */ } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard

Kubernetes In-Place Pod Resize Dynamic Memory Headroom Standard defines an industrial-grade behavioral standard for devops engineering. Enforces a two-phase staging protocol for live vertical scaling under KEP-1287 (In-Place Pod Vertical Scaling). Prevents kernel OOM-killer invocations by guaranteeing that container runtime cgroup v2 memory.max boundaries and kubelet allocated status are observed before expanding workload memory allocations (e.g., JVM heap, V8 heap, or cache pools). Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Simultaneously Increasing Workload Heap Consumption and Pod Resource Limits Triggering Instant cgroup v2 OOM Kills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Simultaneously Increasing Workload Heap Consumption and Pod Resource Limits Triggering Instant cgroup v2 OOM Kills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS set resizePolicy for memory to 'RestartNotRequired' when leveraging KEP-1287 in-place vertical scaling.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.