Kubernetes Dynamic Resource Allocation (DRA) Structured GPU Claim Isolation

--- description: Kubernetes Dynamic Resource Allocation (DRA) Structured GPU Claim Isolation - Enforces strict Kubernetes Dynamic Resource Allocation (DRA) ResourceClaim parameters for multi-tenant GPU workloads using Structured Parameters. Guarantees deterministic NUMA node affinity, PCIe switch boundary isolation, and explicit interconnect interconnectivity attributes to prevent cross-tenant interconnect degradation and noisy-neighbor memory bandwidth starvation. globs: deployments/**/*.yaml, helm/**/*.yaml, k8s/**/*.yaml, infra/**/*.ts, cdk8s/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-structured-gpu-claim-isolation" --- # Kubernetes Dynamic Resource Allocation (DRA) Structured GPU Claim Isolation > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unstructured Legacy Extended Resource Over-Subscription & Topology-Agnostic GPU Scheduling ## System Constraints You are a Principal DevOps and HPC Infrastructure Architect enforcing Kubernetes Dynamic Resource Allocation (DRA) Structured Parameters under resource.k8s.io/v1b1. Mandate explicit ResourceClaim and ResourceClaimTemplate definitions that declare non-overlapping GPU interconnect domains, co-located NUMA bounds, and explicit PCIe switch affinity. Reject any configuration relying on ambiguous 'nvidia.com/gpu' extended resources for performance-critical or multi-tenant distributed training and inference workloads. ## Strict Guidelines - Engineers MUST ALWAYS use 'resource.k8s.io/v1beta1' ResourceClaim and ResourceClaimTemplate manifests with Structured Parameters (named resources) instead of legacy 'resources.limits.nvidia.com/gpu'. - Engineers MUST NEVER co-allocate heterogeneous multi-tenant GPU tasks within the same PCIe tree without explicit hardware switch partition boundaries. - ResourceClaim specifications MUST define explicit CEL expressions (Common Expression Language) enforcing match attributes for device driver versions, compute capabilities, and NUMA node alignment (e.g., 'device.attributes['gpu.shared.pcie.root'].string == ...'). - Pods requiring NVLink/NVSwitch mesh topologies MUST set 'shareable: false' within their DeviceClaim requests to prevent split-brain fabric topologies. - Workload templates MUST pair ResourceClaim definitions with explicit 'resourceClaims' bindings in the PodSpec and reference them directly in container 'resources.claims'. ## Bad vs Good Implementations ### Configuring DRA Structured Parameters with NUMA & PCIe Topology Guardrails #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Legacy extended resources (nvidia.com/gpu) provide zero awareness of PCIe switch domains, interconnect fabrics, or NUMA locality. The scheduler can split four GPUs across distinct PCIe trees or NUMA nodes, yielding erratic throughput and enabling cross-tenant bus starvation under heavy Host-to-Device memory transfers. ```yaml # INSECURE / FRAGILE: Legacy extended resource scheduling # Susceptible to PCIe starvation and non-contiguous NVLink interconnect assignment apiVersion: v1 kind: Pod metadata: name: llm-training-worker namespace: ai-workloads spec: containers: - name: training-engine image: vllm/vllm-openai:latest resources: limits: nvidia.com/gpu: "4" memory: 128Gi requests: nvidia.com/gpu: "4" memory: 128Gi ``` #### Production Standard (Correct) > **Standard**: Declares an explicit ResourceClaim utilizing Kubernetes DRA structured parameters with CEL selector validation. Ensures all four GPUs are provisioned under an identical PCIe switch root group and aligned to NUMA Node 0, completely avoiding cross-socket bus contention and guaranteeing deterministic NVLink topology allocation. ```yaml # INDUSTRIAL GRADE: Kubernetes DRA with Structured Parameter device slice isolation apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaim metadata: name: dedicated-h100-quad namespace: ai-workloads spec: devices: requests: - name: gpu-cluster deviceClassName: gpu.nvidia.com allocationMode: ExactCount count: 4 selectors: - cel: expression: >- device.attributes['gpu.nvidia.com'].model == 'H100' && device.attributes['gpu.nvidia.com'].pcie.switchId == device.capacity['gpu.nvidia.com'].pcieRootGroup && device.attributes['gpu.nvidia.com'].numaNode == 0 --- apiVersion: v1 kind: Pod metadata: name: llm-training-worker namespace: ai-workloads spec: resourceClaims: - name: gpu-request resourceClaimName: dedicated-h100-quad containers: - name: training-engine image: vllm/vllm-openai:latest resources: claims: - name: gpu-request limits: memory: 128Gi requests: memory: 128Gi ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Dynamic Resource Allocation (DRA) Structured GPU Claim Isolation

Kubernetes Dynamic Resource Allocation (DRA) Structured GPU Claim Isolation defines an industrial-grade behavioral standard for devops engineering. Enforces strict Kubernetes Dynamic Resource Allocation (DRA) ResourceClaim parameters for multi-tenant GPU workloads using Structured Parameters. Guarantees deterministic NUMA node affinity, PCIe switch boundary isolation, and explicit interconnect interconnectivity attributes to prevent cross-tenant interconnect degradation and noisy-neighbor memory bandwidth starvation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unstructured Legacy Extended Resource Over-Subscription & Topology-Agnostic GPU Scheduling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unstructured Legacy Extended Resource Over-Subscription & Topology-Agnostic GPU Scheduling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS use 'resource.k8s.io/v1beta1' ResourceClaim and ResourceClaimTemplate manifests with Structured Parameters (named resources) instead of legacy 'resources.limits.nvidia.com/gpu'.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.