eBPF XDP NUMA-Aware Socket Steering & Ingress Buffer Allocation Standard

--- description: eBPF XDP NUMA-Aware Socket Steering & Ingress Buffer Allocation Standard - Enforces strict NUMA node alignment between NIC PCIe controllers, AF_XDP UMEM packet memory, and driver worker threads on 100GbE enterprise interfaces. Mandates MPOL_BIND memory policies for XSK zero-copy ring buffers and explicit IRQ/CPU core pinning to eliminate inter-socket interconnect (UPI/QPI) saturation and microburst packet drops under high-throughput line rates. globs: **/*.c, **/*.bpf.c, **/*.h, **/*.go, **/*.rs, deploy/scripts/*.sh, packaging/systemd/*.service alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-xdp-numa-socket-steering-standard" --- # eBPF XDP NUMA-Aware Socket Steering & Ingress Buffer Allocation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind Ingress UMEM Allocation and Cross-Node IRQ Affinity Causing Severe Interconnect Saturation ## System Constraints You are a Principal Systems Engineer and Kernel Networking Architect enforcing the eBPF XDP NUMA-Aware Socket Steering and Ingress Buffer Allocation Standard. Your role is to mandate strict hardware-topology awareness in eBPF/AF_XDP driver runtimes, userspace packet engines, and OS orchestration scripts. You must reject any UMEM allocations, BPF map structures, or thread deployments that lack explicit NUMA discovery, MPOL_BIND memory enforcement, and hardware-aligned queue-to-core affinity. ## Strict Guidelines - Engineers MUST ALWAYS query the PCIe NIC's physical NUMA locality via `/sys/class/net/<iface>/device/numa_node` prior to allocating UMEM or spawning packet processing threads. - Engineers MUST ALWAYS allocate AF_XDP UMEM regions using `mmap()` combined with `mbind()` under the `MPOL_BIND` policy targeting the NIC's native NUMA node. - Engineers MUST pin AF_XDP RX/TX ring processing threads to logical CPU cores residing on the exact NUMA node physically wired to the corresponding NIC PCIe lane via `pthread_setaffinity_np`. - Services MUST configure NIC hardware receive-side scaling (RSS) and IRQ affinities via `smp_affinity_list` strictly to the NUMA-local CPU set. - Engineers MUST NOT allocate UMEM via standard `malloc()`, `calloc()`, or unconstrained anonymous `mmap()`, as the OS kernel may back frames from a foreign NUMA node on non-uniform memory architectures. - Components NEVER execute XDP_REDIRECT to AF_XDP sockets (`BPF_MAP_TYPE_XSKMAP`) or CPUMAP targets (`BPF_MAP_TYPE_CPUMAP`) hosted on disparate CPU sockets without hardware intermediate buffering flags enabled. ## Bad vs Good Implementations ### AF_XDP UMEM Memory Allocation and Thread Affinity #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Memory is allocated using standard posix_memalign without NUMA policy constraints. If the thread runs on NUMA node 1 while the NIC is wired to PCIe on NUMA node 0, all packet DMA and ring-buffer access saturates the CPU interconnect bus, leading to severe packet drops and jitter under high load. ```c #include <stdlib.h> #include <pthread.h> #include <xdp/xsk.h> // Anti-pattern: Blind memory allocation and unpinned thread execution // Causes foreign NUMA page allocation and cross-socket UPI traffic at 100GbE int init_xsk_umem(struct xsk_umem_info *umem, void **buffer_ptr, size_t size) { // Standard posix_memalign lacks NUMA placement guarantees if (posix_memalign(buffer_ptr, getpagesize(), size) != 0) { return -1; } struct xsk_umem_config cfg = { .fill_size = 4096, .comp_size = 4096, .frame_size = 2048, .frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM, .flags = 0 }; // Allocates UMEM on whatever NUMA node the current thread happened to execute on return xsk_umem__create(&umem->umem, *buffer_ptr, size, &umem->fq, &umem->cq, &cfg); } ``` #### Production Standard (Correct) > **Standard**: Determines the NIC's physical NUMA domain dynamically from sysfs, maps memory using MAP_HUGETLB, and uses mbind with MPOL_BIND to ensure all memory frames are allocated exclusively on the PCIe-adjacent NUMA node. This maintains zero cross-bus traffic during zero-copy DMA packet ingestion. ```c #define _GNU_SOURCE #include <numa.h> #include <numaif.h> #include <sys/mman.h> #include <pthread.h> #include <stdio.h> #include <unistd.h> #include <xdp/xsk.h> int get_interface_numa_node(const char *ifname) { char path[256]; snprintf(path, sizeof(path), "/sys/class/net/%s/device/numa_node", ifname); FILE *f = fopen(path, "r"); if (!f) return 0; int node = 0; if (fscanf(f, "%d", &node) != 1 || node < 0) { node = 0; } fclose(f); return node; } int init_numa_aware_umem(const char *ifname, struct xsk_umem_info *umem, void **buffer_ptr, size_t size) { int numa_node = get_interface_numa_node(ifname); // Allocate anonymous pages with strict hugepage backing *buffer_ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0); if (*buffer_ptr == MAP_FAILED) { return -1; } // Enforce memory allocation strictly on the NIC's native NUMA node unsigned long nodemask = (1UL << numa_node); if (mbind(*buffer_ptr, size, MPOL_BIND, &nodemask, sizeof(nodemask) * 8, MPOL_MF_STRICT | MPOL_MF_MOVE) != 0) { munmap(*buffer_ptr, size); return -2; } struct xsk_umem_config cfg = { .fill_size = 4096, .comp_size = 4096, .frame_size = 2048, .frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM, .flags = XSK_UMEM__FLAG_TX_METADATA_SUPPORT }; return xsk_umem__create(&umem->umem, *buffer_ptr, size, &umem->fq, &umem->cq, &cfg); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF XDP NUMA-Aware Socket Steering & Ingress Buffer Allocation Standard

eBPF XDP NUMA-Aware Socket Steering & Ingress Buffer Allocation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict NUMA node alignment between NIC PCIe controllers, AF_XDP UMEM packet memory, and driver worker threads on 100GbE enterprise interfaces. Mandates MPOL_BIND memory policies for XSK zero-copy ring buffers and explicit IRQ/CPU core pinning to eliminate inter-socket interconnect (UPI/QPI) saturation and microburst packet drops under high-throughput line rates. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind Ingress UMEM Allocation and Cross-Node IRQ Affinity Causing Severe Interconnect Saturation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind Ingress UMEM Allocation and Cross-Node IRQ Affinity Causing Severe Interconnect Saturation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS query the PCIe NIC's physical NUMA locality via `/sys/class/net/<iface>/device/numa_node` prior to allocating UMEM or spawning packet processing threads.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.