eBPF Cgroup v2 PSI-Driven Memory Stall Proactive Pod Throttling

--- description: eBPF Cgroup v2 PSI-Driven Memory Stall Proactive Pod Throttling - Mandates kernel-level Pressure Stall Information (PSI) monitoring via eBPF on cgroup v2 hierarchies to proactively shed non-critical background jobs, telemetry, and caches before the Linux kernel OOM killer triggers. Prevents catastrophic OOM kill loops (exit code 137) and kernel direct reclaim CPU starvation through cooperative application-level backpressure. globs: **/daemon/**/*.{ts,js,go}, **/telemetry/**/*.{ts,js,go}, **/infra/**/*.{ts,js,yaml}, **/k8s/**/*.{ts,yaml,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-cgroupv2-psi-memory-stall-pod-throttling" --- # eBPF Cgroup v2 PSI-Driven Memory Stall Proactive Pod Throttling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Relying on static RSS threshold polling or passive OOM-killer invocations instead of kernel memory pressure stall signals ## System Constraints You are a Principal Software Architect enforcing proactive memory protection standards using eBPF, cgroup v2, and Pressure Stall Information (PSI). You mandate that production container runtimes do not rely on passive OOM kills or static RSS polling. Instead, systems must listen for eBPF ring buffer events or non-blocking epoll triggers on cgroup v2 PSI memory stall events (`some`/`full`) to cooperatively shed asynchronous telemetry and non-essential background workloads before entering direct reclaim starvation or OOM termination. ## Strict Guidelines - Engineers MUST ALWAYS inspect cgroup v2 `memory.pressure` (`some` and `full` stall metrics) rather than static resident set size (RSS) or `memory.current` percentage, because page caches often occupy near 100% memory without inducing reclaim stalls. - Engineers MUST use eBPF ring buffers (`BPF_MAP_TYPE_RINGBUF`) or non-blocking `epoll()` on the PSI file descriptor for event-driven latency mitigation; periodic interval polling of `/sys/fs/cgroup` files in user-space is strictly prohibited. - Services MUST define tiered shedding priorities: priority 0 (in-flight critical customer requests), priority 1 (secondary sync APIs), priority 2 (asynchronous task consumers), and priority 3 (telemetry export, debug tracing, and in-memory read caches). - Components NEVER drop critical request transactions during PSI `some` pressure states; only priority 2 and priority 3 tasks (such as OpenTelemetry batch spans and background image warmers) must be dropped or throttled. - Engineers MUST configure PSI threshold monitors with hysteresis damping (sliding window) to prevent rapid oscillation (thrashing) between throttled and unthrottled execution states. ## Bad vs Good Implementations ### Application Throttling via PSI Event Subscription vs Naive Memory Polling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Periodic user-space polling using `process.memoryUsage()` or total OS memory does not track the container's cgroup v2 boundary or kernel reclaim stall duration. By the time a 5-second poll detects high memory, direct memory reclaim has already frozen threads, or the kernel OOM killer has terminated the container. ```typescript import { setInterval } from 'node:timers'; import * as os from 'node:os'; // ANTI-PATTERN: Polling process.memoryUsage() or RSS fails to detect kernel // reclaim stalls and triggers arbitrary shedding or falls victim to OOM kills. class NaiveTelemetryCollector { private buffer: Array<Record<string, unknown>> = []; constructor() { setInterval(() => { const used = process.memoryUsage().heapUsed / os.totalmem(); if (used > 0.85) { // Unreliable: RSS and heapUsed do not reflect cgroup v2 limits or page-cache thrashing this.buffer = []; } }, 5000); // 5-second polling window is far too slow to prevent microsecond kernel OOM spikes } public record(telemetry: Record<string, unknown>) { this.buffer.push(telemetry); } } ``` #### Production Standard (Correct) > **Standard**: Connects directly to an eBPF/PSI event stream providing kernel stall severity metrics. When kernel threads experience stalls exceeding 20ms in the window, non-critical telemetry buffers are shed immediately, freeing uncommitted buffers and preventing direct reclaim latency degradation and SIGKILL terminations. ```typescript import { EventEmitter } from 'node:events'; interface MemoryPressureEvent { level: 'nominal' | 'some' | 'full'; stallDurationUs: number; timestamp: number; } // PRODUCTION INVARIANT: Subscribes to an eBPF/PSI IPC socket bridge // to shed non-critical telemetry and pause workers before kernel OOM killer activates. export class ResilientTelemetryManager { private telemetryBuffer: Array<Record<string, unknown>> = []; private isThrottled = false; private readonly maxBufferSize = 5000; constructor(private readonly psiBridge: EventEmitter) { this.psiBridge.on('memoryPressure', (event: MemoryPressureEvent) => { this.handleMemoryPressure(event); }); } private handleMemoryPressure(event: MemoryPressureEvent): void { if (event.level === 'full' || (event.level === 'some' && event.stallDurationUs > 20_000)) { // Severe kernel stall detected: immediately purge non-critical telemetry this.isThrottled = true; this.telemetryBuffer = []; } else if (event.level === 'nominal') { this.isThrottled = false; } } public record(telemetry: Record<string, unknown>): boolean { if (this.isThrottled) { // Proactively drop priority-3 telemetry to relieve page-cache pressure return false; } if (this.telemetryBuffer.length >= this.maxBufferSize) { this.telemetryBuffer.shift(); } this.telemetryBuffer.push(telemetry); return true; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF Cgroup v2 PSI-Driven Memory Stall Proactive Pod Throttling

eBPF Cgroup v2 PSI-Driven Memory Stall Proactive Pod Throttling defines an industrial-grade behavioral standard for devops engineering. Mandates kernel-level Pressure Stall Information (PSI) monitoring via eBPF on cgroup v2 hierarchies to proactively shed non-critical background jobs, telemetry, and caches before the Linux kernel OOM killer triggers. Prevents catastrophic OOM kill loops (exit code 137) and kernel direct reclaim CPU starvation through cooperative application-level backpressure. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Relying on static RSS threshold polling or passive OOM-killer invocations instead of kernel memory pressure stall signals Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Relying on static RSS threshold polling or passive OOM-killer invocations instead of kernel memory pressure stall signals	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS inspect cgroup v2 `memory.pressure` (`some` and `full` stall metrics) rather than static resident set size (RSS) or `memory.current` percentage, because page caches often occupy near 100% memory without inducing reclaim stalls.

Engineers MUST ALWAYS inspect cgroup v2 `memory.pressure` (`some` and `full` stall metrics) rather than static resident set size (RSS) or `memory.current` percentage, because page caches often occupy near 100% memory without inducing reclaim stalls.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.