Kubernetes DRA Dynamic Device Claim Atomic Pre-Binding and Health Validation Invariant

--- description: Kubernetes DRA Dynamic Device Claim Atomic Pre-Binding and Health Validation Invariant - Enforces atomic pre-binding hardware validation and driver lease verification within Kubernetes Dynamic Resource Allocation (DRA) controllers and CDI plugins prior to pod container admission. Prevents silent multi-tenant resource leakage, zombie device locks, and co-tenant memory exposure on GPU and NPU clusters by mandating hardware-level sanitization and lease verification before updating NodeResourceSlice and DeviceClaim allocations. globs: pkg/dra/**/*.go, pkg/controller/resourceclaim/**/*.go, controllers/dra/**/*.go, pkg/driver/**/*.go, deploy/helm/**/*.yaml, config/crd/bases/resource.k8s.io_*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-device-claim-prebind-invariant" --- # Kubernetes DRA Dynamic Device Claim Atomic Pre-Binding and Health Validation Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Optimistic DRA Claim Binding Without Atomic Hardware Health Verification and PCIe Reset Handshake ## System Constraints You are a Principal Infrastructure & Systems Architect enforcing strict Kubernetes Dynamic Resource Allocation (DRA) invariants (resource.k8s.io/v1alpha3 or v1beta1). You must reject any controller, mutating webhook, or CDI runtime implementation that performs optimistic ResourceClaim scheduling or status binding without synchronous, atomic hardware health probing, firmware error checking (such as PCIe AER, ECC fatal/non-fatal traps), memory zeroing, and lease acquisition before reporting devices as allocated to kube-scheduler or kubelet. ## Strict Guidelines - DRA driver controllers MUST execute synchronous low-level hardware health checks (e.g., PCIe link speed, NVLink topology, ECC error status) before writing the `.status.allocation` field of a `ResourceClaim`. - Engineers MUST NOT mark a `ResourceClaim` as allocated or ready when an underlying accelerator device is in an un-reset, poisoned, or degraded health state. - DRA Node Preparation plugins (CDI/kubelet plugin interface) MUST invoke complete device sanitization (e.g., GPU memory zeroing, compute engine reset via FLR or NVML reset) prior to returning a successful `NodePrepareResource` gRPC response. - Controllers MUST acquire an explicit lease/reservation lock with monotonic fencing tokens before assigning physical UUIDs into `NodeResourceSlice` or `ResourceClaimAllocationResult` to avoid split-brain concurrent allocation across dual-driver or hybrid scheduler configurations. - ResourceClaim status updates MUST use optimistic concurrency control (`metadata.resourceVersion` with retry-on-conflict) and must roll back physical reservations atomically if the Kubernetes API server update fails. - Engineers MUST configure explicit timeouts and transactional context cancellations for all driver-level RPCs; a timed-out device reset must put the target device into a quarantined NodeResourceSlice state rather than permitting pod container launch. ## Bad vs Good Implementations ### Unsafe Optimistic DeviceClaim Allocation (Anti-Pattern) vs. Atomic Pre-Bound Health Verification (Production Invariant) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The anti-pattern optimistically sets the status.allocation on the ResourceClaim without verifying physical device health or wiping hardware memory. If the GPU contains leftover tenant weights in VRAM or has uncorrected PCIe ECC errors, the incoming pod will either experience silent hardware panics or access leaked tenant memory across multi-tenant boundaries. ```typescript import { KubeConfig, CustomObjectsApi } from '@kubernetes/client-node'; interface DeviceAllocationRequest { claimUid: string; nodeName: string; requestedDevices: string[]; } export async function allocateClaimOptimistic( k8sApi: CustomObjectsApi, req: DeviceAllocationRequest ): Promise<void> { // ANTI-PATTERN: Optimistically binding claim without validating underlying // device health, ECC status, or clearing previous tenant memory footprint. const patch = { status: { allocation: { nodeSelector: { nodeSelectorTerms: [{ matchFields: [{ key: 'metadata.name', operator: 'In', values: [req.nodeName] }] }] }, devices: { results: req.requestedDevices.map(id => ({ driver: 'gpu.vendor.com/dra', pool: 'default-gpus', device: id })) } } } }; // Blindly updating the ResourceClaim allocation directly to the API server await k8sApi.patchNamespacedCustomObjectStatus( 'resource.k8s.io', 'v1alpha3', 'default', 'resourceclaims', req.claimUid, patch, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/merge-patch+json' } } ); } ``` #### Production Standard (Correct) > **Standard**: The production invariant verifies that the target accelerator undergoes a physical health check and memory scrub (Function Level Reset) before binding. It claims a monotonic fencing lease to prevent double-booking, leverages resourceVersion checks for optimistic concurrency, and guarantees rollback of the hardware lease if the Kubernetes API commit fails. ```typescript import { CustomObjectsApi } from '@kubernetes/client-node'; interface DeviceState { uuid: string; isHealthy: boolean; eccFatalErrors: number; fencingToken: bigint; } interface HardwareDriverClient { verifyHealthAndReset(deviceUuid: string): Promise<DeviceState>; acquireAtomicLease(deviceUuid: string, fencingToken: bigint): Promise<boolean>; releaseLease(deviceUuid: string): Promise<void>; } export async function allocateClaimWithPreBindInvariant( k8sApi: CustomObjectsApi, driverClient: HardwareDriverClient, claimName: string, claimNamespace: string, claimResourceVersion: string, nodeName: string, targetDeviceUuid: string, expectedFencingToken: bigint ): Promise<void> { // 1. Synchronously execute hardware reset (FLR/VRAM zeroing) and health probe const devState = await driverClient.verifyHealthAndReset(targetDeviceUuid); if (!devState.isHealthy || devState.eccFatalErrors > 0) { throw new Error(`Device ${targetDeviceUuid} failed pre-bind health invariant. Allocation aborted.`); } // 2. Acquire monotonic lease to prevent concurrent double-booking const leaseAcquired = await driverClient.acquireAtomicLease(targetDeviceUuid, expectedFencingToken); if (!leaseAcquired) { throw new Error(`Atomic lease contention for device ${targetDeviceUuid}.`); } try { // 3. Atomically patch ResourceClaim allocation with optimistic concurrency control const patch = { metadata: { resourceVersion: claimResourceVersion }, status: { allocation: { nodeSelector: { nodeSelectorTerms: [{ matchFields: [{ key: 'metadata.name', operator: 'In', values: [nodeName] }] }] }, devices: { results: [{ driver: 'gpu.vendor.com/dra', pool: 'isolated-h100', device: devState.uuid }] } } } }; await k8sApi.patchNamespacedCustomObjectStatus( 'resource.k8s.io', 'v1alpha3', claimNamespace, 'resourceclaims', claimName, patch, undefined, undefined, undefined, { headers: { 'Content-Type': 'application/merge-patch+json' } } ); } catch (error) { // 4. Compensation rollback: Release driver lease on API failure to prevent zombie locks await driverClient.releaseLease(targetDeviceUuid); throw new Error(`ResourceClaim allocation failed. Rolled back hardware lease: ${(error as Error).message}`); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes DRA Dynamic Device Claim Atomic Pre-Binding and Health Validation Invariant

Kubernetes DRA Dynamic Device Claim Atomic Pre-Binding and Health Validation Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces atomic pre-binding hardware validation and driver lease verification within Kubernetes Dynamic Resource Allocation (DRA) controllers and CDI plugins prior to pod container admission. Prevents silent multi-tenant resource leakage, zombie device locks, and co-tenant memory exposure on GPU and NPU clusters by mandating hardware-level sanitization and lease verification before updating NodeResourceSlice and DeviceClaim allocations. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Optimistic DRA Claim Binding Without Atomic Hardware Health Verification and PCIe Reset Handshake Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Optimistic DRA Claim Binding Without Atomic Hardware Health Verification and PCIe Reset Handshake	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

DRA driver controllers MUST execute synchronous low-level hardware health checks (e.g., PCIe link speed, NVLink topology, ECC error status) before writing the `.status.allocation` field of a `ResourceClaim`.

DRA driver controllers MUST execute synchronous low-level hardware health checks (e.g., PCIe link speed, NVLink topology, ECC error status) before writing the `.status.allocation` field of a `ResourceClaim`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.