Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrail

--- description: Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrail - Governs atomic, lock-free GPU and NPU device allocation at the kubelet and scheduler level to prevent fractional allocation leaks and deadlocks across distributed AI inference workloads. Mandates structured ResourceClaim templates with atomic all-or-nothing allocation semantics and explicit device sharing policies instead of legacy fragmented extended resources. globs: deployments/**/*.{yaml,yml,json}, manifests/**/*.{yaml,yml,json}, infra/**/*.{yaml,yml,json,ts}, helm/**/*.{yaml,yml}, src/controllers/**/*.{go,ts} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-multi-accelerator-claim-guardrail" --- # Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Partial GPU/NPU Allocation via Fragmented Extended Resources Inducing Distributed Deadlocks and Orphaned Device Leaks ## System Constraints You are a Principal Platform Engineer enforcing Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrails. Mandate atomic, lock-free accelerator device assignment via ResourceClaims, structured ResourceClaimTemplates, and CEL-based device selectors. Prevent fractional allocation fragmentation, race-conditioned device plug-ins, and asymmetric multi-node deadlocks across distributed LLM inference and training deployments. ## Strict Guidelines - Engineers MUST ALWAYS declare accelerator requirements using Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaimTemplates rather than fragmented extended resources (e.g., nvidia.com/gpu, google.com/tpu) for multi-accelerator topologies. - Engineers MUST NEVER permit asymmetric or partial accelerator claim binding across multi-pod distributed inference gangs; pods in a distributed tensor-parallel group MUST bind atomically or fail entirely. - Claims MUST enforce device affinity and interconnect constraints (e.g., NVLink domain, PCIe switch locality, NPU mesh mesh topology) using Common Expression Language (CEL) selectors in the DeviceClass or ResourceClaim parameters. - Custom DRA Driver controllers MUST implement lock-free, idempotent NodePrepareResources and NodeUnprepareResources gRPC endpoints to guarantee zero device handle leaks during abrupt container eviction or preemption. - Workloads MUST NOT combine legacy extended resource limits with DRA claims in the same pod specification to prevent double-counting and resource starvation at the kubelet device manager. - ResourceClaims targeting shared accelerators (MIG slices, MPS threads, fractional NPU cores) MUST specify an explicit sharing strategy with deterministic cleanup guarantees upon pod termination. ## Bad vs Good Implementations ### Migrating Fragile Extended GPU Requests to Atomic Structured DRA ResourceClaimTemplates #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Legacy extended resources bypass the DRA scheduling framework. They cannot evaluate topology constraints (e.g., ensuring all 4 GPUs are on the same NVLink bridge), lack atomic binding guarantees for distributed workloads, and lead to fractional scheduling deadlocks when clusters are constrained. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: distributed-llama-inference spec: replicas: 2 template: spec: containers: - name: vllm-worker image: vllm/vllm-openai:v0.6.0 resources: limits: # Anti-pattern: Non-atomic legacy extended resources # Does not guarantee NVLink topology, device locality, or atomic gang claim nvidia.com/gpu: "4" requests: nvidia.com/gpu: "4" ``` #### Production Standard (Correct) > **Standard**: Leverages Kubernetes DRA ResourceClaimTemplates with structured CEL selectors. The scheduler evaluates hardware topology atomically, ensuring all 4 GPUs satisfy NVLink interconnect requirements and bind in a lock-free, all-or-nothing transaction before container runtime handoff. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: distributed-llama-inference spec: replicas: 2 template: spec: resourceClaims: - name: accelerator-gang resourceClaimTemplateName: h100-nvlink-quad containers: - name: vllm-worker image: vllm/vllm-openai:v0.6.0 resources: claims: - name: accelerator-gang --- apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: h100-nvlink-quad spec: spec: devices: requests: - name: gpu-group deviceClassName: gpu.nvidia.com count: 4 selectors: - cel: # Invariant: Atomic multi-GPU allocation restricted to fully connected NVLink domain expression: device.attributes['gpu.nvidia.com'].nvlinkBandwidthGbps >= 900 ``` ### Idempotent DRA Driver Device Unpreparation in Go Controller #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Prematurely deletes the internal allocation tracking entry before confirming device hardware teardown. If device reset fails or times out, subsequent retries fail to find the claim UID, permanently leaking the physical accelerator into an unallocatable state. ```go // Anti-pattern: State-mutating device cleanup assuming synchronous unmount func (d *Driver) NodeUnprepareResources(ctx context.Context, req *drapb.NodeUnprepareResourcesRequest) (*drapb.NodeUnprepareResourcesResponse, error) { for _, claim := range req.Claims { // Unsafe: direct slice mutation without checking actual hardware state devID := d.allocatedDevices[claim.Uid] delete(d.allocatedDevices, claim.Uid) // If hardware reset fails, the device is orphaned from state but still locked in kernel if err := d.hwInterface.ResetDevice(devID); err != nil { return nil, err // Causes kubelet unprepare retry loop with leaked state } } return &drapb.NodeUnprepareResourcesResponse{}, nil } ``` #### Production Standard (Correct) > **Standard**: Implements an idempotent teardown pattern adhering to the DRA CDI (Container Device Interface) lifecycle. Hardware state is safely flushed and verified before internal registries are updated, returning per-claim error structures instead of failing the entire batch. ```go // Invariant: Idempotent teardown reconciling physical state before clearing metadata func (d *Driver) NodeUnprepareResources(ctx context.Context, req *drapb.NodeUnprepareResourcesRequest) (*drapb.NodeUnprepareResourcesResponse, error) { resp := &drapb.NodeUnprepareResourcesResponse{ Claims: make(map[string]*drapb.NodeUnprepareResourceResponse), } for _, claim := range req.Claims { devID, exists := d.getAllocatedDeviceID(claim.Uid) if !exists { // Idempotent success: Already cleaned or unknown claim resp.Claims[claim.Uid] = &drapb.NodeUnprepareResourceResponse{} continue } // Reset physical accelerator state; ensure no lingering memory mappings or compute processes if err := d.hwInterface.DrainAndReset(ctx, devID); err != nil { resp.Claims[claim.Uid] = &drapb.NodeUnprepareResourceResponse{ Error: fmt.Sprintf("hardware reset failed for device %s: %v", devID, err), } continue } // Safely deregister from internal tracker only after physical release succeeds d.releaseDeviceID(claim.Uid) resp.Claims[claim.Uid] = &drapb.NodeUnprepareResourceResponse{} } return resp, nil } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrail

Kubernetes Dynamic Resource Allocation (DRA) Multi-Accelerator Claim Guardrail defines an industrial-grade behavioral standard for devops engineering. Governs atomic, lock-free GPU and NPU device allocation at the kubelet and scheduler level to prevent fractional allocation leaks and deadlocks across distributed AI inference workloads. Mandates structured ResourceClaim templates with atomic all-or-nothing allocation semantics and explicit device sharing policies instead of legacy fragmented extended resources. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Partial GPU/NPU Allocation via Fragmented Extended Resources Inducing Distributed Deadlocks and Orphaned Device Leaks Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Partial GPU/NPU Allocation via Fragmented Extended Resources Inducing Distributed Deadlocks and Orphaned Device Leaks	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare accelerator requirements using Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaimTemplates rather than fragmented extended resources (e.g., nvidia.com/gpu, google.com/tpu) for multi-accelerator topologies.

Engineers MUST ALWAYS declare accelerator requirements using Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaimTemplates rather than fragmented extended resources (e.g., nvidia.com/gpu, google.com/tpu) for multi-accelerator topologies.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.