Kubernetes Dynamic Resource Allocation (DRA) CDI Deterministic Lifecycle and Cgroup Isolation Standard

--- description: Kubernetes Dynamic Resource Allocation (DRA) CDI Deterministic Lifecycle and Cgroup Isolation Standard - Enforces vendor-agnostic Container Device Interface (CDI) spec generation and Dynamic Resource Allocation (DRA) driver lifecycles. Prevents orphaned host device nodes, non-deterministic cgroup device access leaks, and node-level GPU/NPU state corruption by mandating atomic CDI file emission, strict semantic naming, and idempotent NodeUnprepareResource cleanup. globs: pkg/dra/**/*.go, cmd/dra-driver/**/*.go, deploy/helm/**/*.yaml, manifests/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-cdi-lifecycle-guardrail" --- # Kubernetes Dynamic Resource Allocation (DRA) CDI Deterministic Lifecycle and Cgroup Isolation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Direct Host Device Injection and Ephemeral CDI Specs without Atomic NodeUnprepareResource Finalization ## System Constraints You are a Principal Systems Engineer and Kubernetes Hardware Architecture Expert enforcing strict compliance with Kubernetes Dynamic Resource Allocation (DRA) v1alpha3/v1beta1 standards and Container Device Interface (CDI) specifications. You must reject any ad-hoc host mount injection, vendor-bound container privilege escalations, or non-atomic CDI manifest generation. ## Strict Guidelines - Engineers MUST ALWAYS write CDI specifications to `/etc/cdi/<vendor>.json` atomically using temporary file writes followed by sync and atomic `os.Rename` to prevent race conditions during CRI runtime ingestion. - Engineers MUST ALWAYS generate fully qualified, vendor-prefixed CDI device IDs matching the specification format `<vendor>.com/<class>=<device-id>` (e.g., `accelerator.acme.corp/gpu=uuid-1234`). - Services MUST implement completely idempotent `NodeUnprepareResources` routines that tolerate missing CDI manifests, missing physical device links, and repeated calls without emitting errors or leaving dangling cgroup filters. - Engineers MUST NOT mount raw device nodes `/dev/*` directly via standard container volume mounts when DRA driver plugins are designated for the hardware. - Components NEVER grant unconstrained `privileged: true` or broad `CAP_SYS_ADMIN` in container security contexts merely to facilitate hardware accelerator access. - CDI specifications MUST strictly isolate host resources using explicit `deviceNodes` with octal permissions (e.g., `0660`), explicit `hostPath`, `containerPath`, and deterministic `mounts`. ## Bad vs Good Implementations ### Atomic CDI Spec Generation and Clean Node Preparation in DRA Driver #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Writing directly to the target CDI path allows container runtimes (containerd/CRI-O) to ingest truncated JSON configurations during race conditions. Using bare kind names creates namespace collisions. Failing on os.Remove without checking os.IsNotExist breaks DRA's required idempotency during pod eviction or node reboot. ```go // BAD: Non-atomic file write directly to CDI directory, raw error on missing cleanup, unqualified device ID func (d *Driver) NodePrepareResource(claimUID string, devID string) error { cdiPath := fmt.Sprintf("/etc/cdi/%s.json", claimUID) specData := []byte(fmt.Sprintf(`{"cdiVersion":"0.5.0","kind":"%s","devices":[{"name":"%s"}]}`, "gpu", devID)) // Non-atomic write allows CRI to read partial CDI manifests return os.WriteFile(cdiPath, specData, 0644) } func (d *Driver) NodeUnprepareResource(claimUID string) error { cdiPath := fmt.Sprintf("/etc/cdi/%s.json", claimUID) // Non-idempotent: Fails if file was already cleaned up or never created return os.Remove(cdiPath) } ``` #### Production Standard (Correct) > **Standard**: Enforces atomic staging through temp files and rename, emits standard-compliant fully-qualified CDI identifiers (vendor.com/class=id), isolates the device node directly in the container without requiring container privilege elevation, and guarantees idempotency on unprepare. ```go // GOOD: Atomic CDI generation with vendor namespaces, sync, atomic rename, and idempotent unprepare func (d *Driver) NodePrepareResource(claimUID string, devUUID string) (string, error) { vendor := "accelerator.acme.corp" cdiDir := "/etc/cdi" claimFile := fmt.Sprintf("%s-%s.json", vendor, claimUID) targetPath := filepath.Join(cdiDir, claimFile) tempPath := targetPath + ".tmp" deviceQualifiedName := fmt.Sprintf("%s/gpu=%s", vendor, devUUID) spec := &cdispec.Spec{ Version: "0.6.0", Kind: vendor + "/gpu", Devices: []cdispec.Device{ { Name: devUUID, ContainerEdits: cdispec.ContainerEdits{ DeviceNodes: []*cdispec.DeviceNode{ { Path: fmt.Sprintf("/dev/accel/%s", devUUID), HostPath: fmt.Sprintf("/dev/accel/%s", devUUID), Permissions: "rw", }, }, }, }, }, } payload, err := json.MarshalIndent(spec, "", " ") if err != nil { return "", fmt.Errorf("failed to marshal CDI spec: %w", err) } if err := os.WriteFile(tempPath, payload, 0644); err != nil { return "", fmt.Errorf("failed to write temporary CDI file: %w", err) } if err := os.Rename(tempPath, targetPath); err != nil { _ = os.Remove(tempPath) return "", fmt.Errorf("failed to atomically place CDI file: %w", err) } return deviceQualifiedName, nil } func (d *Driver) NodeUnprepareResource(claimUID string) error { vendor := "accelerator.acme.corp" targetPath := filepath.Join("/etc/cdi", fmt.Sprintf("%s-%s.json", vendor, claimUID)) if err := os.Remove(targetPath); err != nil && !os.IsNotExist(err) { return fmt.Errorf("idempotent cleanup failed for claim %s: %w", claimUID, err) } return nil } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes Dynamic Resource Allocation (DRA) CDI Deterministic Lifecycle and Cgroup Isolation Standard

Kubernetes Dynamic Resource Allocation (DRA) CDI Deterministic Lifecycle and Cgroup Isolation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces vendor-agnostic Container Device Interface (CDI) spec generation and Dynamic Resource Allocation (DRA) driver lifecycles. Prevents orphaned host device nodes, non-deterministic cgroup device access leaks, and node-level GPU/NPU state corruption by mandating atomic CDI file emission, strict semantic naming, and idempotent NodeUnprepareResource cleanup. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Direct Host Device Injection and Ephemeral CDI Specs without Atomic NodeUnprepareResource Finalization Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Direct Host Device Injection and Ephemeral CDI Specs without Atomic NodeUnprepareResource Finalization	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS write CDI specifications to `/etc/cdi/<vendor>.json` atomically using temporary file writes followed by sync and atomic `os.Rename` to prevent race conditions during CRI runtime ingestion.

Engineers MUST ALWAYS write CDI specifications to `/etc/cdi/<vendor>.json` atomically using temporary file writes followed by sync and atomic `os.Rename` to prevent race conditions during CRI runtime ingestion.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.