AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: k8s-inplace-vertical-scaling-cgroupv2-guardrail

Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail

Enforces strict resource adjustment sequences and kernel-level Pressure Stall Information (PSI) synchronization during Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal kernel Out-Of-Memory (OOM) invocations and multi-second CFS CPU throttling collapses when shifting cgroupv2 memory.max, memory.high, and cpu.max controllers without restarting containers.

Prohibited Anti-Pattern: Directly Patching Pod Spec Limits Below Active Cgroupv2 Working Sets Inducing Kernel OOM Kills

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail - Enforces strict resource adjustment sequences and kernel-level Pressure Stall Information (PSI) synchronization during Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal kernel Out-Of-Memory (OOM) invocations and multi-second CFS CPU throttling collapses when shifting cgroupv2 memory.max, memory.high, and cpu.max controllers without restarting containers.
globs: deploy/**/*.yaml, manifests/**/*.yaml, pkg/controllers/**/*.go, src/controllers/**/*.ts, infra/k8s/**/*.json
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/k8s-inplace-vertical-scaling-cgroupv2-guardrail"
---

# Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Directly Patching Pod Spec Limits Below Active Cgroupv2 Working Sets Inducing Kernel OOM Kills

## System Constraints
You are a Principal Software Architect enforcing zero-downtime, kernel-safe Kubernetes In-Place Pod Vertical Scaling (KEP-1287) across cgroupv2 nodes. You must prevent premature memory clipping, uncoordinated CPU quota decreases, and missing resizePolicy configurations that trigger kernel OOM kills or severe CFS scheduler stalls.

## Strict Guidelines
- Engineers MUST ALWAYS configure container 'resizePolicy' explicitly with 'resourceName: cpu' and 'resourceName: memory' set to 'NotRequired' or 'RestartContainer' based on workload runtime capabilities.
- Engineers MUST NOT apply immediate memory.max reductions below the active 'memory.current' plus page cache reclamation buffer; dynamic memory downscaling MUST be preceded by staged 'memory.high' throttling signals.
- Services MUST monitor Linux Pressure Stall Information (PSI) metrics ('some' and 'full' memory/CPU stalls) via the node runtime before executing in-place downscale patches.
- Controllers MUST execute resource updates sequentially: memory limits MUST expand before workload memory footprint grows during scale-up, and CPU allocations MUST step down gradually during scale-down to avoid thread starvation.
- Components NEVER assume pod resource spec updates synchronously reflect active runtime cgroup boundaries; controllers MUST poll 'containerStatuses[].allocatedResources' and 'containerStatuses[].resources' before declaring a scale operation complete.

## Bad vs Good Implementations
### Dynamic Pod Resize Controller Step Logic

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Directly applying a low memory limit without verifying current cgroupv2 'memory.current' and PSI thresholds causes the Linux kernel to immediately trigger an uncatchable synchronous OOM kill. Furthermore, missing container resizePolicy specifications and lack of status convergence validation can cause undefined runtime state transitions.
```typescript
import * as k8s from '@kubernetes/client-node';

// ANTI-PATTERN: Naively patching Pod resources directly to desired target without
// inspecting active working set or staged cgroupv2 memory limits
export async function resizePodResources(
  k8sApi: k8s.CoreV1Api,
  namespace: string,
  podName: string,
  containerName: string,
  targetMemoryMb: number,
  targetCpuCores: number
): Promise<void> {
  // Blindly updating the pod spec limits
  // If targetMemoryMb < current active anonymous memory, kernel invokes immediate OOM kill!
  await k8sApi.patchNamespacedPod(
    podName,
    namespace,
    {
      spec: {
        containers: [{
          name: containerName,
          resources: {
            limits: {
              memory: `${targetMemoryMb}Mi`,
              cpu: `${targetCpuCores}`,
            },
            requests: {
              memory: `${targetMemoryMb}Mi`,
              cpu: `${targetCpuCores}`,
            }
          }
        }]
      }
    },
    undefined,
    undefined,
    undefined,
    undefined,
    undefined,
    { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }
  );
}
```

#### Production Standard (Correct)
> **Standard**: Validates active memory usage and Pressure Stall Information (PSI) prior to initiating resize, enforcing explicit 'NotRequired' resize policies to guarantee zero-restart live scaling while ensuring limits never fall below safe working-set margins.
```typescript
import * as k8s from '@kubernetes/client-node';

interface CgroupPsiMetrics {
  memorySomeAvg10: number;
  memoryCurrentBytes: number;
}

// PRODUCTION INVARIANT: Staged resizing with working set buffer validation and resizePolicy compliance
export async function resizePodInPlaceSafely(
  k8sApi: k8s.CoreV1Api,
  namespace: string,
  podName: string,
  containerName: string,
  targetMemoryBytes: number,
  targetCpuMillis: number,
  cgroupMetrics: CgroupPsiMetrics
): Promise<void> {
  const pod = (await k8sApi.readNamespacedPod(podName, namespace)).body;
  const container = pod.spec?.containers.find(c => c.name === containerName);
  
  if (!container) throw new Error(`Container ${containerName} not found in ${podName}`);

  // 1. Guard against downscaling below active memory footprint (working set + 20% safety margin)
  const minSafeMemory = cgroupMetrics.memoryCurrentBytes * 1.2;
  if (targetMemoryBytes < minSafeMemory) {
    throw new Error(
      `In-place downscale rejected: target ${targetMemoryBytes}B is below safe watermark ${minSafeMemory}B.`
    );
  }

  // 2. Reject scale adjustments if PSI memory pressure is already elevated (>15% stall)
  if (cgroupMetrics.memorySomeAvg10 > 15.0) {
    throw new Error(`Transient pressure stall detected (PSI: ${cgroupMetrics.memorySomeAvg10}%). Deferring resize.`);
  }

  // 3. Patch pod with explicit resizePolicy guarantees
  const patchPayload = {
    spec: {
      containers: [{
        name: containerName,
        resizePolicy: [
          { resourceName: 'cpu', restartPolicy: 'NotRequired' },
          { resourceName: 'memory', restartPolicy: 'NotRequired' }
        ],
        resources: {
          limits: {
            memory: `${Math.floor(targetMemoryBytes / (1024 * 1024))}Mi`,
            cpu: `${targetCpuMillis}m`
          },
          requests: {
            memory: `${Math.floor(targetMemoryBytes / (1024 * 1024))}Mi`,
            cpu: `${targetCpuMillis}m`
          }
        }
      }]
    }
  };

  await k8sApi.patchNamespacedPod(
    podName,
    namespace,
    patchPayload,
    undefined,
    undefined,
    undefined,
    undefined,
    undefined,
    { headers: { 'Content-Type': 'application/strategic-merge-patch+json' } }
  );
}
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Cluster Inspector
Inspects live pod specs, allocatedResources statuses, and node CRI cgroup configurations.
Prometheus Metrics Inspector
Queries real-time container_memory_working_set_bytes and node cgroupv2 PSI metrics.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

Kubectl In-Place Resize Validator
Verifies that Kubelet InPlacePodVerticalScaling feature gate is active and tracks subresource status conditions.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail

Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict resource adjustment sequences and kernel-level Pressure Stall Information (PSI) synchronization during Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal kernel Out-Of-Memory (OOM) invocations and multi-second CFS CPU throttling collapses when shifting cgroupv2 memory.max, memory.high, and cpu.max controllers without restarting containers. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Directly Patching Pod Spec Limits Below Active Cgroupv2 Working Sets Inducing Kernel OOM Kills Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftDirectly Patching Pod Spec Limits Below Active Cgroupv2 Working Sets Inducing Kernel OOM KillsCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS configure container 'resizePolicy' explicitly with 'resourceName

Engineers MUST ALWAYS configure container 'resizePolicy' explicitly with 'resourceName: cpu' and 'resourceName: memory' set to 'NotRequired' or 'RestartContainer' based on workload runtime capabilities.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT apply immediate memory.max reductions below the active 'memory.current' plus page cache reclamation buffer

Engineers MUST NOT apply immediate memory.max reductions below the active 'memory.current' plus page cache reclamation buffer; dynamic memory downscaling MUST be preceded by staged 'memory.high' throttling signals.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Services MUST monitor Linux Pressure Stall Information (PSI) metrics ('some' and 'full' memory/CPU stalls) via the node runtime before executing in-place downscale patches.

Services MUST monitor Linux Pressure Stall Information (PSI) metrics ('some' and 'full' memory/CPU stalls) via the node runtime before executing in-place downscale patches.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Controllers MUST execute resource updates sequentially

Controllers MUST execute resource updates sequentially: memory limits MUST expand before workload memory footprint grows during scale-up, and CPU allocations MUST step down gradually during scale-down to avoid thread starvation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Components NEVER assume pod resource spec updates synchronously reflect active runtime cgroup boundaries

Components NEVER assume pod resource spec updates synchronously reflect active runtime cgroup boundaries; controllers MUST poll 'containerStatuses[].allocatedResources' and 'containerStatuses[].resources' before declaring a scale operation complete.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is Kubernetes In-Place Pod Vertical Scaling Cgroupv2 Dynamic Throttling Guardrail critical for AI coding assistants?

Enforces strict resource adjustment sequences and kernel-level Pressure Stall Information (PSI) synchronization during Kubernetes In-Place Pod Vertical Scaling (KEP-1287). Prevents fatal kernel Out-Of-Memory (OOM) invocations and multi-second CFS CPU throttling collapses when shifting cgroupv2 memory.max, memory.high, and cpu.max controllers without restarting containers. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Directly Patching Pod Spec Limits Below Active Cgroupv2 Working Sets Inducing Kernel OOM Kills) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

Kubernetes