Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariant

--- description: Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariant - Enforces deterministic, structured GPU hardware slicing via Kubernetes Dynamic Resource Allocation (DRA) ResourceSlices and structured parameters rather than uncoordinated device plugin sharing or unpartitioned nvidia.com/gpu limits. Eliminates noisy-neighbor compute and high-bandwidth memory (HBM) cache contention across collocated AI inference workloads by strictly bounding MIG or spatial GPU partitions at scheduling time. globs: deployments/**/*.{yaml,yml,json,ts}, helm/**/*.{yaml,yml,json,ts}, infra/**/*.{yaml,yml,json,ts}, src/k8s/**/*.{yaml,yml,json,ts} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-gpu-partitioning-invariant" --- # Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Uncoordinated Device Plugin GPU Over-subscription and Unpartitioned Integer Device Allocation ## System Constraints You are a Principal Cloud Native Infrastructure Architect and AI Platform Specialist enforcing Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariants. Mandate structured parameters and ResourceClaimTemplates for all GPU-dependent inference workloads. Prevent noisy-neighbor starvation by forbidding legacy opaque integer resources (e.g., nvidia.com/gpu: 1) where multi-tenant or collocated partitioned workloads are required. Ensure all compute and memory slice profiles are deterministically scheduled via resource.k8s.io APIs. ## Strict Guidelines - Engineers MUST ALWAYS declare GPU requirements using ResourceClaims or ResourceClaimTemplates bound via resource.k8s.io/v1beta1 (or higher) with structured device partitioning parameters. - Engineers MUST NOT rely on deprecated nvidia.com/gpu or nvidia.com/gpu-shared integer resources when collocating latency-sensitive inference workloads. - Manifests MUST specify exact hardware profile attributes (e.g., compute slices, HBM memory capacities, SM counts) using selector-based constraints in the claim spec. - Deployments MUST configure device sharing policies explicitly as exclusive slices within the structured parameters to guarantee compute and memory bandwidth isolation. - AI inference pod specs MUST bind container resources using the resources.claims mapping rather than conventional resources.limits or resources.requests. ## Bad vs Good Implementations ### DRA ResourceClaimTemplate vs Legacy Opaque GPU Limit Allocation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Using opaque integer resource limits like 'nvidia.com/gpu: 1' bypasses dynamic hardware-slice validation. It forces either complete single-tenant reservation of an entire GPU (causing extreme under-utilization) or relies on non-isolated shared plugins that lack HBM memory bandwidth and SM isolation, resulting in tail-latency spikes from noisy neighbors. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: llm-inference-service spec: replicas: 2 template: spec: containers: - name: vllm-engine image: vllm/vllm-openai:latest resources: limits: # Anti-pattern: Opaque device plugin allocation causes uncoordinated # full-GPU reservation or arbitrary unpartitioned sharing with no HBM/SM isolation. nvidia.com/gpu: "1" ``` #### Production Standard (Correct) > **Standard**: Adheres to the DRA invariant by defining a structured ResourceClaimTemplate with explicit CEL device selectors for a deterministic slice profile ('1g.10gb'). The pod binds this claim deterministically, guaranteeing isolation of streaming multiprocessors (SMs) and HBM without noisy-neighbor contention. ```yaml apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: gpu-mig-slice-claim spec: spec: devices: requests: - name: partitioned-gpu deviceClassName: gpu.nvidia.com selectors: - cel: # Invariant: Guarantee exact hardware compute and memory bounds expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'" --- apiVersion: apps/v1 kind: Deployment metadata: name: llm-inference-service spec: replicas: 2 template: spec: resourceClaims: - name: gpu-slice resourceClaimTemplateName: gpu-mig-slice-claim containers: - name: vllm-engine image: vllm/vllm-openai:latest resources: claims: - name: gpu-slice ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariant

Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, structured GPU hardware slicing via Kubernetes Dynamic Resource Allocation (DRA) ResourceSlices and structured parameters rather than uncoordinated device plugin sharing or unpartitioned nvidia.com/gpu limits. Eliminates noisy-neighbor compute and high-bandwidth memory (HBM) cache contention across collocated AI inference workloads by strictly bounding MIG or spatial GPU partitions at scheduling time. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Uncoordinated Device Plugin GPU Over-subscription and Unpartitioned Integer Device Allocation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Uncoordinated Device Plugin GPU Over-subscription and Unpartitioned Integer Device Allocation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare GPU requirements using ResourceClaims or ResourceClaimTemplates bound via resource.k8s.io/v1beta1 (or higher) with structured device partitioning parameters.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.