eBPF TC Egress Packet Scheduling & Multi-Tenant Bandwidth Isolation

--- description: eBPF TC Egress Packet Scheduling & Multi-Tenant Bandwidth Isolation - Enforces deterministic kernel-space egress traffic shaping and tenant rate limiting via eBPF Traffic Control (TC) BPF_PROG_TYPE_SCHED_ACT programs. Prevents noisy-neighbor egress bandwidth starvation on multi-tenant Kubernetes nodes by binding Token Bucket Filter (TBF) and Earliest Departure Time (EDT) scheduling algorithms directly to virtual ethernet and physical root interfaces. globs: bpf/**/*.c, bpf/**/*.h, pkg/ebpf/**/*.go, pkg/network/**/*.go, deploy/daemonset/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-tc-egress-bandwidth-scheduling-standard" --- # eBPF TC Egress Packet Scheduling & Multi-Tenant Bandwidth Isolation > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Relying on Userspace Rate Limiters or Best-Effort CNI QoS Without Kernel-Level TC Egress Queue Enforcement ## System Constraints You are a Principal Linux Kernel and Network Architect enforcing eBPF TC (Traffic Control) Packet Scheduling & Egress Shaping Standards in multi-tenant Kubernetes clusters. You must ensure all egress traffic is shaped directly in kernel space using BPF TC egress hooks with EDT or atomic Token Bucket algorithms. Prohibit unbounded egress queues, userspace packet throttling, missing tail-drop metrics, and unpinned BPF maps that break across daemon restarts. ## Strict Guidelines - Engineers MUST ALWAYS attach eBPF egress shapers to the TC subsystem via BPF_TC_EGRESS hooks on host interfaces and container veth pairs using direct-action mode (da). - Engineers MUST ALWAYS compute delivery deadlines using skb->tstamp via Earliest Departure Time (EDT) coupled with FQ (Fair Queueing) qdisc rather than spinning or sleeping inside kernel programs. - Engineers MUST ALWAYS use atomic BPF map lookups (BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_LRU_HASH) with __sync_fetch_and_add for token replenishment to eliminate race conditions across concurrent CPU cores. - Engineers MUST NOT rely on userspace traffic proxying, iptables string-matching, or cgroupv1 blkio/net_cls controllers for network egress rate isolation. - Code SHOULD return TC_ACT_SHOT on token exhaustion or quota breach and emit a perf event / ring buffer metric to register throttled packets immediately. - Services MUST pin BPF rate-limit maps under /sys/fs/bpf to guarantee state persistence during CNI daemon agent restarts and upgrades. - Components NEVER drop control plane traffic (ARP, ICMP, DHCP, DNS, and internal health checks) regardless of tenant egress saturation. ## Bad vs Good Implementations ### Kernel TC eBPF Tenant Rate Limiter using Monotonic Clocks and Atomic Tokens #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The bad example mutates tenant quota state non-atomically across concurrent cores, ignores token replenishment calculations based on ktime, fails to protect essential infrastructure traffic, and passes packets through when quotas are depleted. ```c #include <linux/bpf.h> #include <linux/pkt_cls.h> #include <bpf/bpf_helpers.h> struct tenant_quota { __u64 tokens; __u64 last_time; }; struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 1024); __type(key, __u32); /* cgroup_id */ __type(value, struct tenant_quota); } rate_map SEC(".maps"); SEC("tc") int tc_egress_bad(struct __sk_buff *skb) { __u32 cgroup_id = bpf_skb_cgroup_id(skb); struct tenant_quota *q = bpf_map_lookup_elem(&rate_map, &cgroup_id); /* Non-atomic state mutation leads to severe multi-core race conditions */ if (q) { if (q->tokens < skb->len) { return TC_ACT_OK; /* Fails open, leading to noisy-neighbor saturation */ } q->tokens -= skb->len; } return TC_ACT_OK; } ``` #### Production Standard (Correct) > **Standard**: The good example evaluates skb egress at TC stage, safeguards ARP control-plane frames, accounts for nanosecond-level token replenishment, enforces burst bounds, and executes deterministic tail-drops (TC_ACT_SHOT) while pinning maps to survive reloads. ```c #include <linux/bpf.h> #include <linux/pkt_cls.h> #include <linux/if_ether.h> #include <linux/ip.h> #include <bpf/bpf_helpers.h> #define RATE_BPS (100 * 1024 * 1024 / 8) /* 100 Mbps in bytes/sec */ #define BURST_BYTES (256 * 1024) #define NSEC_PER_SEC 1000000000ULL struct tenant_bucket { __u64 tokens; __u64 last_update_ns; }; struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __uint(max_entries, 16384); __type(key, __u32); /* pod cgroup_id */ __type(value, struct tenant_bucket); __uint(pinning, LIBBPF_PIN_BY_NAME); } pod_egress_rates SEC(".maps"); SEC("tc") int tc_egress_shaped(struct __sk_buff *skb) { void *data = (void *)(long)skb->data; void *data_end = (void *)(long)skb->data_end; /* Bypass control plane traffic (e.g., ARP) */ struct ethhdr *eth = data; if ((void *)(eth + 1) > data_end) return TC_ACT_OK; if (eth->h_proto == __constant_htons(ETH_P_ARP)) return TC_ACT_OK; __u32 cgroup_id = bpf_skb_cgroup_id(skb); if (cgroup_id == 0) return TC_ACT_OK; __u64 now = bpf_ktime_get_ns(); struct tenant_bucket *bucket = bpf_map_lookup_elem(&pod_egress_rates, &cgroup_id); if (!bucket) { struct tenant_bucket init = { .tokens = BURST_BYTES, .last_update_ns = now }; bpf_map_update_elem(&pod_egress_rates, &cgroup_id, &init, BPF_NOEXIST); bucket = bpf_map_lookup_elem(&pod_egress_rates, &cgroup_id); if (!bucket) return TC_ACT_OK; } __u64 elapsed = now - bucket->last_update_ns; __u64 replenished = (elapsed * RATE_BPS) / NSEC_PER_SEC; __u64 current_tokens = bucket->tokens + replenished; if (current_tokens > BURST_BYTES) current_tokens = BURST_BYTES; if (current_tokens < skb->len) return TC_ACT_SHOT; /* Hard egress clamp prevents starvation */ bucket->tokens = current_tokens - skb->len; bucket->last_update_ns = now; return TC_ACT_OK; } char _license[] SEC("license") = "Dual BSD/GPL"; ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF TC Egress Packet Scheduling & Multi-Tenant Bandwidth Isolation

eBPF TC Egress Packet Scheduling & Multi-Tenant Bandwidth Isolation defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic kernel-space egress traffic shaping and tenant rate limiting via eBPF Traffic Control (TC) BPF_PROG_TYPE_SCHED_ACT programs. Prevents noisy-neighbor egress bandwidth starvation on multi-tenant Kubernetes nodes by binding Token Bucket Filter (TBF) and Earliest Departure Time (EDT) scheduling algorithms directly to virtual ethernet and physical root interfaces. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Relying on Userspace Rate Limiters or Best-Effort CNI QoS Without Kernel-Level TC Egress Queue Enforcement Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Relying on Userspace Rate Limiters or Best-Effort CNI QoS Without Kernel-Level TC Egress Queue Enforcement	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS attach eBPF egress shapers to the TC subsystem via BPF_TC_EGRESS hooks on host interfaces and container veth pairs using direct-action mode (da).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.