Kubernetes In-Place Pod Resizing cgroup v2 Dynamic Memory Rebalancing

--- description: Kubernetes In-Place Pod Resizing cgroup v2 Dynamic Memory Rebalancing - Enforces strict runtime synchronization between Kubernetes In-Place Pod Resizing (cgroup v2 `memory.max` / `memory.high`) and application runtime memory limits (Go `debug.SetMemoryLimit`, HotSpot JVM dynamic MXBeans/ZGC). Prevents immediate OOMKills during dynamic downscaling and eliminates severe memory underutilization during dynamic upscaling without pod restarts. globs: deploy/kubernetes/**/*.yaml, deploy/helm/**/*.yaml, cmd/**/*.go, pkg/**/*.go, src/main/**/*.java, src/main/**/*.kt alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-inplace-resize-cgroupv2-dynamic-memory" --- # Kubernetes In-Place Pod Resizing cgroup v2 Dynamic Memory Rebalancing > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static Memory Initialization and Blind Downscaling Inducing Silent OOMKills ## System Constraints You are a Principal SRE and Infrastructure Architect enforcing production invariants for Kubernetes In-Place Pod Resizing under cgroup v2. Mandate that application runtimes dynamically detect cgroup v2 boundary mutations, subscribe to kernel memory notifications, proactively release allocations prior to container-engine downscaling, and dynamically recalculate runtime target bounds without pod restarts. ## Strict Guidelines - Workloads MUST configure `resizePolicy` for `memory` to `RestartContainer: NotRequired` in pod templates when utilizing Kubernetes In-Place Pod Resizing. - Go applications MUST use `runtime/debug.SetMemoryLimit` backed by an active cgroup v2 fsnotify/inotify watcher on `/sys/fs/cgroup/memory.max` and `/sys/fs/cgroup/memory.high`, accounting for container headroom. - JVM services MUST run on OpenJDK 21+ with dynamically reconfigurable memory limits or explicit external sidecar/agent orchestrators monitoring cgroup v2 events, and MUST proactively invoke `System.gc()` or trigger dynamic ZGC/Shenandoah uncommit before accepting downscaled limits. - Engineers MUST NOT rely on static initialization-time memory detection (`automaxprocs`, `-XX:MaxRAMPercentage`, or single-shot `cgroups` inspection) when in-place resizing is enabled. - Services MUST subtract safety buffer headroom (minimum 10-15% or 128MiB-256MiB) from `memory.max` when calculating runtime target allocations to accommodate non-heap memory and kernel page cache. - Monitoring agents MUST listen to `memory.events` (specifically `high`, `max`, and `oom_kill` counters) to preemptively force memory reclamation when utilization breaches `memory.high`. ## Bad vs Good Implementations ### Go Dynamic cgroup v2 In-Place Resize Memory Adapter #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Initialization-time inspection fails to track live cgroup v2 mutations made by the Kubelet during In-Place Pod Resizing. Downsizing `memory.max` below current runtime allocation instantly invokes kernel OOM killer before GC can react. ```go package main import ( "log" "os" "runtime/debug" "strconv" "strings" ) // Bad: Reads cgroup limit once at startup. When Kubernetes executes an In-Place // Pod Resize, the process fails to adapt. Downscaling triggers instant OOMKill; // upscaling leaves memory stranded and unusable. func init() { data, err := os.ReadFile("/sys/fs/cgroup/memory.max") if err == nil { val := strings.TrimSpace(string(data)) if limit, err := strconv.ParseInt(val, 10, 64); err == nil { // Static initialization only debug.SetMemoryLimit(int64(float64(limit) * 0.85)) log.Printf("Static memory limit set to: %d", limit) } } } ``` #### Production Standard (Correct) > **Standard**: Subscribes to `/sys/fs/cgroup/memory.max` write notifications via inotify/fsnotify and maintains a safety poll ticker. Automatically recalibrates Go's soft memory limit and forces memory return (`debug.FreeOSMemory`) before kernel memory limits are enforced. ```go package main import ( "context" "log" "os" "runtime/debug" "strconv" "strings" "time" "github.com/fsnotify/fsnotify" ) // Good: Continuously watches cgroup v2 memory boundary changes via fsnotify // and dynamically re-applies runtime/debug.SetMemoryLimit with explicit headroom. func StartCgroupV2MemoryRebalancer(ctx context.Context, headroomRatio float64) error { const cgroupMaxPath = "/sys/fs/cgroup/memory.max" rebalance := func() { data, err := os.ReadFile(cgroupMaxPath) if err != nil { return } val := strings.TrimSpace(string(data)) if val == "max" { debug.SetMemoryLimit(-1) // Unconstrained return } limit, err := strconv.ParseInt(val, 10, 64) if err != nil || limit <= 0 { return } // Reserve headroom for off-heap, thread stacks, and page cache target := int64(float64(limit) * (1.0 - headroomRatio)) old := debug.SetMemoryLimit(target) debug.FreeOSMemory() // Encourage immediate release back to OS log.Printf("cgroup v2 resized: adjusted limit from %d to %d (raw: %d)", old, target, limit) } watcher, err := fsnotify.NewWatcher() if err != nil { return err } rebalance() if err := watcher.Add(cgroupMaxPath); err != nil { watcher.Close() return err } go func() { defer watcher.Close() ticker := time.NewTicker(30 * time.Second) // Fallback poll defer ticker.Stop() for { select { case <-ctx.Done(): return case event, ok := <-watcher.Events: if !ok { return } if event.Has(fsnotify.Write) { rebalance() } case <-ticker.C: rebalance() } } }() return nil } ``` ### Kubernetes Pod Spec In-Place Resize Policy Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default Kubernetes behavior when resizing container resources restarts the container unless explicitly designated otherwise. Without `resizePolicy`, changing memory bounds causes an unwanted pod restart. ```yaml apiVersion: v1 kind: Pod metadata: name: payment-processor spec: containers: - name: app image: payment-processor:v2.1.0 resources: requests: memory: "2Gi" cpu: "1" limits: memory: "4Gi" cpu: "2" # Bad: Omits resizePolicy. Defaults to RestartContainer upon resizing limits, # defeating the entire purpose of in-place resize and causing availability churn. ``` #### Production Standard (Correct) > **Standard**: Explicitly declares `resizePolicy` with `restartPolicy: NotRequired` for memory and CPU, allowing Kubelet and containerd to mutate cgroup v2 limits live without terminating the application process. ```yaml apiVersion: v1 kind: Pod metadata: name: payment-processor spec: containers: - name: app image: payment-processor:v2.1.0 resizePolicy: - resourceName: memory restartPolicy: NotRequired - resourceName: cpu restartPolicy: NotRequired resources: requests: memory: "2Gi" cpu: "1" limits: memory: "4Gi" cpu: "2" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes In-Place Pod Resizing cgroup v2 Dynamic Memory Rebalancing

Kubernetes In-Place Pod Resizing cgroup v2 Dynamic Memory Rebalancing defines an industrial-grade behavioral standard for devops engineering. Enforces strict runtime synchronization between Kubernetes In-Place Pod Resizing (cgroup v2 `memory.max` / `memory.high`) and application runtime memory limits (Go `debug.SetMemoryLimit`, HotSpot JVM dynamic MXBeans/ZGC). Prevents immediate OOMKills during dynamic downscaling and eliminates severe memory underutilization during dynamic upscaling without pod restarts. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static Memory Initialization and Blind Downscaling Inducing Silent OOMKills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static Memory Initialization and Blind Downscaling Inducing Silent OOMKills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Workloads MUST configure `resizePolicy` for `memory` to `RestartContainer

Workloads MUST configure `resizePolicy` for `memory` to `RestartContainer: NotRequired` in pod templates when utilizing Kubernetes In-Place Pod Resizing.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.