Deterministic PCIe Topology Isolation and Memory Reservation in Kubernetes DRA

--- description: Deterministic PCIe Topology Isolation and Memory Reservation in Kubernetes DRA - Enforces strict hardware-aligned partition boundaries using Kubernetes Dynamic Resource Allocation (DRA) resource claims for multi-tenant GPU workloads. Prevents non-deterministic host-to-device PCIe bandwidth saturation, cross-NUMA cross-talk, and OOM cascades caused by unreserved MIG slices or unpinned NVLink topologies in multi-tenant heterogenous clusters. globs: infra/**/*.yaml, infra/**/*.yml, deploy/k8s/**/*.yaml, src/platform/orchestration/**/*.ts, src/platform/infrastructure/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/k8s-dra-gpu-partitioning-boundary-standard" --- # Deterministic PCIe Topology Isolation and Memory Reservation in Kubernetes DRA > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unconstrained Multi-Tenant GPU Allocation via Legacy Extended Resources Bypassing PCIe Topology and Memory Hard Limits ## System Constraints You are a Principal Infrastructure Architect enforcing Kubernetes Dynamic Resource Allocation (DRA) GPU Partitioning Boundaries. Mandate that all GPU allocations for heterogeneous multi-tenant clusters reject legacy 'nvidia.com/gpu' integer counts in favor of explicit ResourceClaim manifests targeting exact PCIe root complex affinities, strict memory-slice reservations (MIG/MPS), and NVLink domain constraints. Never permit pods to cross NUMA node boundaries or multiplex uneven memory footprints onto shared physical sockets without absolute hardware-enforced isolation. ## Strict Guidelines - Engineers MUST ALWAYS declare GPU demands via ResourceClaimTemplate or ResourceClaim CRDs referencing dynamic resource allocation (DRA) driver classes (e.g., gpu.nvidia.com) rather than legacy pod resources.limits. - Engineers MUST ALWAYS enforce strict PCIe tree isolation by configuring topology selectors that align the allocated GPU device with the pod's CPU NUMA node and local NIC device. - Engineers MUST NEVER co-locate latency-sensitive training or inference workloads on shared multi-instance GPU (MIG) slices without explicit dedicated memory reservations and isolated PCIe execution engines. - Code generating Kubernetes Pod specs SHOULD automatically inject DeviceClaim selectors specifying deterministic constraints for interconnect (NVLink vs PCIe peer-to-peer) bandwidth. - Platform orchestrators MUST validate that memory reservations (GiB) and streaming multiprocessor (SM) counts are immutable and statically provisioned at allocation time to avoid noisy-neighbor OOM evictions. ## Bad vs Good Implementations ### K8s Spec: Migration from Non-Deterministic Extended GPU Resources to DRA Device Claim with PCIe Affinity #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Legacy 'nvidia.com/gpu' allocation allocates opaque integer units. Devices may be attached to disparate PCIe root complexes or cross-socket NUMA nodes, causing severe interconnect bottlenecks, memory bandwidth degradation, and no physical memory partitioning guarantees. ```typescript import * as k8s from '@kubernetes/client-node'; // ANTI-PATTERN: Using legacy extended resources that ignore NUMA alignment, // PCIe topology boundaries, and memory reservations in multi-tenant pools export function createInferencePod(name: string, tenantId: string): k8s.V1Pod { return { apiVersion: 'v1', kind: 'Pod', metadata: { name, labels: { tenant: tenantId } }, spec: { containers: [{ name: 'model-runner', image: 'registry.enterprise.ai/llm-serving:v2.1', resources: { limits: { 'nvidia.com/gpu': '2' // Non-deterministic: devices may span disparate NUMA nodes or PCIe switches } } }] } }; } ``` #### Production Standard (Correct) > **Standard**: Implements Kubernetes Dynamic Resource Allocation using CEL selectors to mandate explicit memory reservations, strict MIG partition isolation, and unified PCIe/NUMA root complex affinity to eliminate cross-talk and memory starvation. ```typescript import * as k8s from '@kubernetes/client-node'; // PRODUCTION INVARIANT: Kubernetes DRA (resource.k8s.io/v1alpha3 or v1beta1) with exact PCIe/NUMA topology isolation export function createInferencePod(name: string, tenantId: string): k8s.V1Pod { const claimName = `${name}-gpu-partition-claim`; return { apiVersion: 'v1', kind: 'Pod', metadata: { name, labels: { tenant: tenantId } }, spec: { resourceClaims: [{ name: 'gpu-partition', resourceClaimName: claimName }], containers: [{ name: 'model-runner', image: 'registry.enterprise.ai/llm-serving:v2.1', resources: { claims: [{ name: 'gpu-partition' }] } }] } }; } export function createGpuDeviceClaim(claimName: string, namespace: string, minMemGiB: number) { return { apiVersion: 'resource.k8s.io/v1beta1', kind: 'ResourceClaim', metadata: { name: claimName, namespace }, spec: { devices: { requests: [{ name: 'isolated-mig-engine', deviceClassName: 'gpu.nvidia.com', selectors: [{ cel: { expression: `device.attributes['gpu.nvidia.com'].memorySize >= ${minMemGiB}Gi && ` + `device.attributes['gpu.nvidia.com'].migSlice == true && ` + `device.topology.pcie.rootComplex == node.topology.numa[0].pcieRoot` } }] }] } } }; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic PCIe Topology Isolation and Memory Reservation in Kubernetes DRA

Deterministic PCIe Topology Isolation and Memory Reservation in Kubernetes DRA defines an industrial-grade behavioral standard for devops engineering. Enforces strict hardware-aligned partition boundaries using Kubernetes Dynamic Resource Allocation (DRA) resource claims for multi-tenant GPU workloads. Prevents non-deterministic host-to-device PCIe bandwidth saturation, cross-NUMA cross-talk, and OOM cascades caused by unreserved MIG slices or unpinned NVLink topologies in multi-tenant heterogenous clusters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unconstrained Multi-Tenant GPU Allocation via Legacy Extended Resources Bypassing PCIe Topology and Memory Hard Limits Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unconstrained Multi-Tenant GPU Allocation via Legacy Extended Resources Bypassing PCIe Topology and Memory Hard Limits	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare GPU demands via ResourceClaimTemplate or ResourceClaim CRDs referencing dynamic resource allocation (DRA) driver classes (e.g., gpu.nvidia.com) rather than legacy pod resources.limits.

Engineers MUST ALWAYS declare GPU demands via ResourceClaimTemplate or ResourceClaim CRDs referencing dynamic resource allocation (DRA) driver classes (e.g., gpu.nvidia.com) rather than legacy pod resources.limits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.