Kubernetes DRA Deterministic Sliced-Accelerator Lifecycle & Memory Erasure Standard

--- description: Kubernetes DRA Deterministic Sliced-Accelerator Lifecycle & Memory Erasure Standard - Enforces strict Kubernetes Dynamic Resource Allocation (DRA) ResourceClaim parameters for partitioned GPU/TPU accelerators (e.g., NVIDIA MIG or Google TPU slices). Guarantees hardware-level memory scrubbing, deterministic spatial/temporal partitioning, and non-shared PCIe isolation boundaries to prevent cross-tenant state leakage and noisy-neighbor interference. globs: deployments/**/*.{yaml,yml,json,ts}, infra/**/*.{yaml,yml,json,ts}, helm/**/*.{yaml,yml,json,ts}, cdk/**/*.{ts,tsx,js,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-multi-instance-accelerator-standard" --- # Kubernetes DRA Deterministic Sliced-Accelerator Lifecycle & Memory Erasure Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unsanitized Resource Sharing via Legacy Extended Resources with Leaked VRAM Buffers ## System Constraints You are a Principal Infrastructure & Security Architect enforcing Kubernetes Dynamic Resource Allocation (DRA) standards for multi-instance accelerators (NVIDIA MIG, Google Cloud TPU). You must prevent dirty-memory reuse, state leakage across partitioned hardware, and non-deterministic resource scheduling. Mandate resource.k8s.io ResourceClaims with explicit lifecycle deallocation hooks, isolated compute slices, and zero-fill memory guarantees across all workload definitions. ## Strict Guidelines - Engineers MUST ALWAYS declare sliced accelerator dependencies using Kubernetes Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaim or ResourceClaimTemplate manifests rather than legacy resource requests (e.g., nvidia.com/gpu or nvidia.com/mig-*). - ResourceClaims MUST explicitly declare profile-driven allocation parameters (e.g., profile: '1g.10gb') and enforce memory sanitization policies ('WipeOnDeallocate' / 'ZeroFillMemoryOnRelease') within the DRA driver configuration. - Engineers MUST NOT allow ResourceClaims with 'allocationMode: ExactCount' to omit device sharing restrictions; 'allowMultipleAllocations' MUST explicitly be set to false for multi-tenant AI training/inference workloads. - Pods utilizing DRA claims MUST declare structured container-level resource volume claims, ensuring device affinity constraints and PCIe bus isolation match the cluster NUMA topology. - Custom operators and admission controllers MUST reject ResourceClaimTemplates that lack cryptographic slice isolation tags and driver-enforced sanitization lifecycle hooks. ## Bad vs Good Implementations ### Configuring Sliced GPU Workload via Kubernetes DRA vs Legacy MIG Requests #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Legacy extended resource requests (nvidia.com/mig-*) lack hardware-level DRA lifecycle hooks. When pods are evicted or replaced, the device memory is not deterministically wiped by the control plane, permitting potential exfiltration of sensitive weights or KV-cache state between multi-tenant workloads. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: tenant-inference-worker namespace: tenant-a spec: replicas: 1 template: spec: containers: - name: inference-engine image: tensorrt-llm:latest resources: limits: # Flawed: Legacy extended resources do not guarantee VRAM sanitization # between pods, allowing residual tensor weights/KV cache to persist. nvidia.com/mig-1g.10gb: 1 ``` #### Production Standard (Correct) > **Standard**: Leverages Kubernetes DRA (ResourceClaimTemplate) with explicit CEL constraints and 'allowMultipleAllocations: false'. The DRA driver hooks into the accelerator lifecycle to execute an unbind memory-scrub sequence upon deallocation, ensuring complete tenant isolation and deterministic resource limits. ```yaml apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: isolated-mig-claim-template namespace: tenant-a spec: spec: devices: requests: - name: dedicated-slice deviceClassName: gpu.nvidia.com selectors: - cel: expression: "device.attributes['gpu.nvidia.com'].profile == '1g.10gb'" allocationMode: ExactCount count: 1 # Invariant: Enforce hardware isolation, no sharing, and guaranteed zeroization allowMultipleAllocations: false --- apiVersion: apps/v1 kind: Deployment metadata: name: tenant-inference-worker namespace: tenant-a spec: replicas: 1 template: spec: resourceClaims: - name: accelerator-slice resourceClaimTemplateName: isolated-mig-claim-template containers: - name: inference-engine image: tensorrt-llm:latest resources: claims: - name: accelerator-slice ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kubernetes DRA Deterministic Sliced-Accelerator Lifecycle & Memory Erasure Standard

Kubernetes DRA Deterministic Sliced-Accelerator Lifecycle & Memory Erasure Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict Kubernetes Dynamic Resource Allocation (DRA) ResourceClaim parameters for partitioned GPU/TPU accelerators (e.g., NVIDIA MIG or Google TPU slices). Guarantees hardware-level memory scrubbing, deterministic spatial/temporal partitioning, and non-shared PCIe isolation boundaries to prevent cross-tenant state leakage and noisy-neighbor interference. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unsanitized Resource Sharing via Legacy Extended Resources with Leaked VRAM Buffers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unsanitized Resource Sharing via Legacy Extended Resources with Leaked VRAM Buffers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare sliced accelerator dependencies using Kubernetes Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaim or ResourceClaimTemplate manifests rather than legacy resource requests (e.g., nvidia.com/gpu or nvidia.com/mig-*).

Engineers MUST ALWAYS declare sliced accelerator dependencies using Kubernetes Dynamic Resource Allocation (resource.k8s.io/v1alpha3 or v1beta1) ResourceClaim or ResourceClaimTemplate manifests rather than legacy resource requests (e.g., nvidia.com/gpu or nvidia.com/mig-*).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.