Cgroup v2 Proactive memory.high Kernel Reclaim Backpressure Standard

--- description: Cgroup v2 Proactive memory.high Kernel Reclaim Backpressure Standard - Enforces proactive kernel reclaim and synchronous allocator throttling via cgroup v2 memory.high thresholds set below absolute hard ceilings (memory.max). This prevents sudden burst-induced Out-Of-Memory (OOM) killer terminations by inducing gradual backpressure, reclaim compaction, and application-level load shedding prior to unrecoverable SIGKILL events. globs: deploy/**/*.yaml, k8s/**/*.yaml, helm/**/*.yaml, systemd/**/*.slice, systemd/**/*.service, **/kubelet-config.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/cgroup-v2-memory-high-backpressure-production-standard" --- # Cgroup v2 Proactive memory.high Kernel Reclaim Backpressure Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Binary OOM Cliff Falloff Caused by Sole Reliance on memory.max Without memory.high Throttling Thresholds ## System Constraints You are a Principal DevOps and Linux Kernel Reliability Engineer enforcing Cgroup v2 Proactive Memory Reclaim and Backpressure architectures. All container definitions, systemd slices, and Kubernetes workload profiles must eliminate binary OOM cliffs by configuring cgroup v2 memory.high backpressure boundaries situated between 80% and 90% of memory.max. Never allow workloads to operate solely with unthrottled allocations up to memory.max without kernel-level compaction and proactive paging headroom. ## Strict Guidelines - Engineers MUST ALWAYS configure cgroup v2 'memory.high' at 80% to 90% of 'memory.max' (or equivalent container memory limits) to induce proactive synchronous kernel reclaim. - Engineers MUST NOT rely exclusively on hard limits ('memory.max' or unannotated Kubernetes 'resources.limits.memory') that cause instant SIGKILL under memory allocation spikes. - Kubernetes configurations MUST leverage MemoryQoS feature gates or container runtime annotations (e.g., container.apparmor.security.beta.kubernetes.io or CRI-O/containerd annotations) to compute and enforce 'memory.high'. - Services running under systemd MUST set 'MemoryHigh=' in their respective slice or service unit files alongside 'MemoryMax='. - Observability configurations MUST monitor 'memory.events: high' counters and alert on sustained throttle durations before 'memory.events: oom' is reached. - Applications running in containers SHOULD intercept cgroup PSI (Pressure Stall Information) at '/proc/pressure/memory' to shed non-essential internal caches when kernel reclaim latency increases. ## Bad vs Good Implementations ### Kubernetes Pod with Cgroup v2 MemoryQoS and memory.high Annotations #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring only memory.limits maps directly to cgroup v2 memory.max without memory.high. When a memory spike occurs, the process experiences no pre-emptive kernel throttling or proactive asynchronous reclaim, leading to an immediate unrecoverable OOM-kill (exit code 137). ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout-service spec: template: spec: containers: - name: app image: checkout-service:v2.1.0 resources: requests: memory: "1Gi" # ANTI-PATTERN: Setting only hard limit maps directly to memory.max in cgroup v2. # Under sudden burst allocations, kernel fails to reclaim in time and invokes OOM killer immediately. limits: memory: "2Gi" ``` #### Production Standard (Correct) > **Standard**: Establishes a proactive memory.high threshold at 85% of memory.max (~1.7GiB). When consumption passes memory.high, the Linux kernel subjects allocating threads to incremental sleep delays (backpressure) and proactively evicts clean page caches, giving the application time to self-throttle and avoiding SIGKILL. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout-service annotations: # Induces cgroup v2 memory.high throttling at 85% of limit memory.qos.container.resource/high-fraction: "0.85" spec: template: metadata: annotations: # Ensure CRI runtime configures memory.high backpressure headroom io.kubernetes.cri-o.MemoryHigh: "1825361100" # ~1.7GiB (85% of 2GiB) spec: containers: - name: app image: checkout-service:v2.1.0 resources: requests: memory: "1Gi" limits: memory: "2Gi" ``` ### Systemd Service Slice Cgroup v2 Resource Guardrail #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting MemoryHigh allows worker threads to execute unconstrained rapid slab/heap allocations right up to 8GB, at which point the kernel invokes OOM-killer immediately without pre-emptive paging or allocator delays. ```ini # ANTI-PATTERN: Only MemoryMax is specified. [Unit] Description=High-Throughput Ingestion Worker [Service] ExecStart=/usr/local/bin/worker MemoryMax=8G ``` #### Production Standard (Correct) > **Standard**: MemoryHigh enforces proactive kernel reclaim and allocator throttling at 6.8GB. MemoryLow protects critical working sets from premature reclamation, and systemd-oomd monitors memory pressure stalls deterministically. ```ini [Unit] Description=High-Throughput Ingestion Worker [Service] ExecStart=/usr/local/bin/worker # Establish soft backpressure at 85% and hard termination ceiling MemoryLow=2G MemoryHigh=6.8G MemoryMax=8G # Expose PSI metrics to the unit slice ManagedOOMMemoryPressure=kill ManagedOOMMemoryPressureLimit=80% ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Cgroup v2 Proactive memory.high Kernel Reclaim Backpressure Standard

Cgroup v2 Proactive memory.high Kernel Reclaim Backpressure Standard defines an industrial-grade behavioral standard for devops engineering. Enforces proactive kernel reclaim and synchronous allocator throttling via cgroup v2 memory.high thresholds set below absolute hard ceilings (memory.max). This prevents sudden burst-induced Out-Of-Memory (OOM) killer terminations by inducing gradual backpressure, reclaim compaction, and application-level load shedding prior to unrecoverable SIGKILL events. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Binary OOM Cliff Falloff Caused by Sole Reliance on memory.max Without memory.high Throttling Thresholds Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Binary OOM Cliff Falloff Caused by Sole Reliance on memory.max Without memory.high Throttling Thresholds	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure cgroup v2 'memory.high' at 80% to 90% of 'memory.max' (or equivalent container memory limits) to induce proactive synchronous kernel reclaim.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.