Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Immediate memory downscaling or tight-bound resizing that collapses cgroup-v2 memory.max below working set plus dirty page buffers, triggering instantaneous kernel OOM kills.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant

--- description: Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant - When leveraging Kubernetes In-Place Pod Vertical Scaling (KEP-1287), dynamic adjustments to pod resource limits directly mutate cgroup-v2 memory controllers (memory.max, memory.high) without container restarts. Engineers must enforce a calculated memory buffer headroom and sequencing invariant during downscaling and upscaling to prevent kernel OOM-killer invocations caused by dirty page reclaim latency, page cache pressure, and memory allocation races under high load. globs: deployments/**/*.yaml, k8s/**/*.yaml, helm/**/*.yaml, src/controllers/**/*.go, src/autoscaling/**/*.ts, src/autoscaling/**/*.py alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-in-place-pod-resize-cgroupv2-memory-headroom-invariant" --- # Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Immediate memory downscaling or tight-bound resizing that collapses cgroup-v2 memory.max below working set plus dirty page buffers, triggering instantaneous kernel OOM kills. ## System Constraints You are a Principal Infrastructure & Systems Architect enforcing strict invariants for Kubernetes In-Place Pod Resize (KEP-1287) operating on cgroup-v2 systems. When performing dynamic pod memory mutations without restarts, you must enforce explicit delta throttling, safe order-of-operations (requests before limits during downscale, limits before requests during upscale), and a guaranteed 20% or 256MiB (whichever is greater) headroom above memory.current (working_set + inactive_file) to prevent immediate kernel OOM killer triggers. ## Strict Guidelines - Engineers MUST ALWAYS inspect cgroup-v2 `memory.current`, `memory.stat` (`anon` + `file_dirty`), and `memory.high` before initiating an in-place patch to `spec.containers[*].resources.limits.memory`. - Engineers MUST NOT reduce `limits.memory` below `memory.current + max(256MiB, 0.20 * memory.current)` in a single mutation step; multi-phase staged downscaling with cooldown intervals MUST be used. - Engineers MUST configure `resizePolicy` for memory explicitly to `RestartContainer: NotRequired` only when the underlying runtime and runtime application actively listen to and respect memory notifications or handle dynamic heap resizing. - Services performing automated in-place resizing MUST enforce strict ordering: when upscaling, mutate `limits` before `requests`; when downscaling, mutate `requests` before `limits` to maintain the invariant `requests <= limits` at all intermediate states. - Components NEVER issue concurrent limit reductions while dirty page writeback (`nr_dirty` / `file_dirty`) exceeds 5% of targeted cgroup memory limit. ## Bad vs Good Implementations ### In-Place Resize Patch Controller with cgroup-v2 Headroom Validation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly applying the target memory limit without checking real-time cgroup-v2 memory consumption (`memory.current` and dirty cache) instantly triggers an unrecoverable kernel OOM kill if the active memory usage exceeds the target during downscaling. ```typescript import * as k8s from '@kubernetes/client-node'; interface ResizeParams { namespace: string; podName: string; containerName: string; targetMemoryBytes: number; } export async function resizePodMemoryDirectly(k8sApi: k8s.CoreV1Api, params: ResizeParams): Promise<void> { // ANTI-PATTERN: Patching memory limits directly to target without evaluating // cgroup-v2 memory.current, active page cache, or dirty pages causes an instantaneous // kernel OOM kill when targetMemoryBytes <= active working set. const patch = { spec: { containers: [{ name: params.containerName, resources: { limits: { memory: `${params.targetMemoryBytes}` }, requests: { memory: `${params.targetMemoryBytes}` } } }] } }; const options = { headers: { 'Content-Type': k8s.PatchUtils.PATCH_STRATEGIC_MERGE_PATCH } }; await k8sApi.patchNamespacedPod(params.podName, params.namespace, patch, undefined, undefined, undefined, undefined, options); } ``` #### Production Standard (Correct) > **Standard**: Validates cgroup-v2 live memory metrics against the minimum required buffer (max(256MiB, 20%)) and rejects downscaling if pending dirty page flushes could cause an instantaneous kernel cgroup OOM. ```typescript import * as k8s from '@kubernetes/client-node'; interface CgroupV2MemorySnapshot { currentBytes: number; fileDirtyBytes: number; } interface SafeResizeParams { namespace: string; podName: string; containerName: string; targetLimitBytes: number; targetRequestBytes: number; currentCgroupUsage: CgroupV2MemorySnapshot; } const MIN_HEADROOM_BYTES = 256 * 1024 * 1024; // 256 MiB const HEADROOM_RATIO = 0.20; // 20% safety margin export async function resizePodMemorySafely( k8sApi: k8s.CoreV1Api, params: SafeResizeParams ): Promise<void> { const dynamicHeadroom = Math.max(MIN_HEADROOM_BYTES, params.currentCgroupUsage.currentBytes * HEADROOM_RATIO); const safeFloor = params.currentCgroupUsage.currentBytes + dynamicHeadroom; // INVARIANT: Prevent downscaling below cgroup-v2 safe operational headroom if (params.targetLimitBytes < safeFloor) { throw new Error( `In-place downscale rejected: targetLimit (${params.targetLimitBytes} B) violates ` + `cgroup-v2 headroom floor (${safeFloor} B) based on memory.current (${params.currentCgroupUsage.currentBytes} B).` ); } // INVARIANT: Do not downscale memory while kernel dirty page writeback is active if (params.currentCgroupUsage.fileDirtyBytes > (params.targetLimitBytes * 0.05)) { throw new Error('In-place resize deferred: cgroup dirty page writeback exceeds 5% threshold'); } const patchPayload = { spec: { containers: [{ name: params.containerName, resources: { requests: { memory: `${params.targetRequestBytes}` }, limits: { memory: `${params.targetLimitBytes}` } } }] } }; const options = { headers: { 'Content-Type': k8s.PatchUtils.PATCH_STRATEGIC_MERGE_PATCH } }; await k8sApi.patchNamespacedPod( params.podName, params.namespace, patchPayload, undefined, undefined, undefined, undefined, options ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant

Kubernetes In-Place Pod Resize cgroup-v2 Memory Buffer Headroom Invariant defines an industrial-grade behavioral standard for devops engineering. When leveraging Kubernetes In-Place Pod Vertical Scaling (KEP-1287), dynamic adjustments to pod resource limits directly mutate cgroup-v2 memory controllers (memory.max, memory.high) without container restarts. Engineers must enforce a calculated memory buffer headroom and sequencing invariant during downscaling and upscaling to prevent kernel OOM-killer invocations caused by dirty page reclaim latency, page cache pressure, and memory allocation races under high load. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Immediate memory downscaling or tight-bound resizing that collapses cgroup-v2 memory.max below working set plus dirty page buffers, triggering instantaneous kernel OOM kills. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Immediate memory downscaling or tight-bound resizing that collapses cgroup-v2 memory.max below working set plus dirty page buffers, triggering instantaneous kernel OOM kills.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS inspect cgroup-v2 `memory.current`, `memory.stat` (`anon` + `file_dirty`), and `memory.high` before initiating an in-place patch to `spec.containers[*].resources.limits.memory`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.