Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Guardrail

--- description: Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Guardrail - Enforces deterministic structured parameters, CEL-based hardware constraint filtering, and localized lifecycle binding in Kubernetes Dynamic Resource Allocation (DRA) for heterogeneous GPU/NPU workloads. Prevents device memory fragmentation, interconnect topology mismatches, and scheduler starvation caused by legacy static device allocations during dynamic multi-tenant LLM inference batching. globs: deploy/**/*.yaml, k8s/**/*.yaml, helm/**/*.yaml, infra/**/*.ts, manifests/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-gpu-partitioning-invariant-kubernetes-dynam-238" --- # Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unstructured static GPU slicing and unconstrained ResourceClaimTemplates causing device fragmentation and scheduler starvation. ## System Constraints You are a Principal Infrastructure Architect enforcing Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Invariants. Mandate strict adherence to resource.k8s.io APIs, Common Expression Language (CEL) hardware matching, NVLink/interconnect topology awareness, and deterministic claim deallocation. Eliminate all legacy static device plugin oversubscription and unpartitioned multi-tenant hardware sharing across heterogeneous AI clusters. ## Strict Guidelines - Engineers MUST ALWAYS define GPU/NPU requirements using `resource.k8s.io` ResourceClaims or ResourceClaimTemplates rather than legacy opaque integer resources (e.g., `nvidia.com/gpu`). - Engineers MUST ALWAYS enforce CEL selector constraints on memory capacity, interconnect bandwidth, and exact hardware capability (e.g., compute capability >= 9.0) in dynamic slice requests. - Engineers MUST NEVER use unconstrained MIG or time-slicing profiles across heterogeneous tenants within the same physical device without explicit memory and SM isolation policies. - Services MUST bound the lifecycle of dynamically allocated partitions to the Pod lifecycle via `spec.resourceClaims[].resourceClaimTemplateName` to eliminate orphaned GPU resource allocations. - Components NEVER bypass DRA driver topology constraints when scheduling distributed multi-GPU/NPU tensor-parallel inference workloads. ## Bad vs Good Implementations ### Kubernetes Pod Resource Claim with DRA vs. Legacy Static GPU Slicing #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Legacy opaque integer device allocations cannot enforce memory bounds, NVLink adjacency, or structured hardware partitioning. In a heterogeneous cluster, this causes scheduler starvation, co-located memory fragmentation, and massive tail-latency spikes during dynamic LLM batching surges. ```yaml apiVersion: v1 kind: Pod metadata: name: llm-inference-worker spec: containers: - name: vllm-engine image: vllm/vllm-openai:latest resources: limits: # ANTI-PATTERN: Opaque integer slicing lacks memory guarantees, # topology constraints, and leads to fragmentation under dynamic batching. nvidia.com/gpu: "2" requests: nvidia.com/gpu: "2" ``` #### Production Standard (Correct) > **Standard**: DRA ResourceClaimTemplates dynamically negotiate explicit hardware slices with CEL expressions, guaranteeing minimum dedicated VRAM, microarchitecture version, and NVLink topology. Claim binding is scoped to the pod lifecycle, preventing memory fragmentation and zombie claim lockup. ```yaml apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: llm-inference-claim-template spec: spec: devices: requests: - name: partitioned-gpus deviceClassName: gpu.nvidia.com count: 2 selectors: - cel: # INVARIANT: Strict CEL-based partitioning, memory guarantees, and topology matching expression: >- device.capacity["memory"].quantity() >= quantity("40Gi") && device.attributes["computeCapability"].versionNumber >= version("9.0.0") && device.attributes["nvlinkConnected"] == true --- apiVersion: v1 kind: Pod metadata: name: llm-inference-worker spec: resourceClaims: - name: gpu-claim resourceClaimTemplateName: llm-inference-claim-template containers: - name: vllm-engine image: vllm/vllm-openai:latest resources: claims: - name: gpu-claim ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Guardrail

Kubernetes Dynamic Resource Allocation (DRA) GPU/NPU Partitioning Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic structured parameters, CEL-based hardware constraint filtering, and localized lifecycle binding in Kubernetes Dynamic Resource Allocation (DRA) for heterogeneous GPU/NPU workloads. Prevents device memory fragmentation, interconnect topology mismatches, and scheduler starvation caused by legacy static device allocations during dynamic multi-tenant LLM inference batching. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unstructured static GPU slicing and unconstrained ResourceClaimTemplates causing device fragmentation and scheduler starvation. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unstructured static GPU slicing and unconstrained ResourceClaimTemplates causing device fragmentation and scheduler starvation.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS define GPU/NPU requirements using `resource.k8s.io` ResourceClaims or ResourceClaimTemplates rather than legacy opaque integer resources (e.g., `nvidia.com/gpu`).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.