eBPF TCX Multi-Program Packet Re-Injection Scheduling & Deterministic Chain Ordering

--- description: eBPF TCX Multi-Program Packet Re-Injection Scheduling & Deterministic Chain Ordering - Guarantees deterministic execution ordering across coexisting eBPF TCX (Traffic Control BPF link) attachments on network interfaces to eliminate silent packet loss and circular re-injection loops. Enforces explicit relative link positioning flags (BPF_F_BEFORE/BPF_F_AFTER) alongside strict packet-marking invariants to track re-injection depth across kernel ingress filter links. globs: **/*.c, **/*.go, **/*.rs, pkg/ebpf/**/*.go, bpf/**/*.c alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-tcx-multiprogram-reinjection-scheduling-invariant" --- # eBPF TCX Multi-Program Packet Re-Injection Scheduling & Deterministic Chain Ordering > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Non-Deterministic TCX Head Preemption and Unbounded Ingress Re-Injection Loops ## System Constraints You are a Principal Linux Kernel Systems Architect enforcing invariants for eBPF TCX (Traffic Control BPF Link API, Linux 6.6+) multi-program chains. Mandate deterministic attachment ordering, explicit neighbor-relative link anchoring, and recursion-safe packet re-injection paths. ## Strict Guidelines - Engineers MUST ALWAYS specify explicit attachment ordering using BPF_F_BEFORE, BPF_F_AFTER, or BPF_F_ID when creating TCX links (BPF_LINK_TYPE_TCX) instead of relying on default head-of-chain preemption. - Engineers MUST NOT issue bpf_redirect() or bpf_clone_redirect() back into the same interface ingress without encoding a decrementing TTL or deterministic recursion barrier in skb->mark or cb[]. - Code SHOULD return TCX_NEXT (or TC_ACT_UNSPEC in legacy wrappers) when a program does not decisively drop or redirect, ensuring subsequent TCX filter programs in the kernel execution chain receive the skb. - Services MUST query the existing TCX link graph via BPF_PROG_QUERY or link iterator before attachment to assert that relative target links exist and avoid ENOLINK/EINVAL runtime crashes. - Components NEVER execute unchecked bpf_redirect_peer() across veth pairs without inspecting ingress SKB loop flags, preventing infinite kernel softirq livelocks. ## Bad vs Good Implementations ### Go/cilium-ebpf TCX Multi-Program Ingress Attachment & Chain Anchoring #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching a TCX program without explicit anchoring inserts it blindly at the head of the chain. If another security or routing program exists, packets may be dropped or redirected before downstream inspection runs, and lack of recursion checks causes infinite packet loops on redirected frames. ```go // BAD: Implicit attachment assumes sole ownership of the TC ingress chain. // Defaults to inserting at the head, clobbering existing filter chains and causing // non-deterministic ordering with other agents (e.g., Cilium, Istio ambient, Calico). func attachIngressFilter(ifindex int, prog *ebpf.Program) (link.Link, error) { opts := link.TCXOptions{ Interface: ifindex, Program: prog, Attach: ebpf.AttachTCXIngress, // Missing RelativeFD, Anchor, and Flags: // Program blindly preempts all existing ingress filters. } l, err := link.AttachTCX(opts) if err != nil { return nil, err } return l, nil } ``` #### Production Standard (Correct) > **Standard**: The good implementation uses `link.TCXAfter` with a validated `RelativeFD`, guaranteeing that the program executes strictly after the referenced link. This prevents silent clobbering and ensures proper chain continuity. ```go // GOOD: Deterministically queries the existing TCX chain, verifies conflicting links, // anchors relative to existing infrastructure links, and establishes a strict execution order. func attachIngressFilter(ifindex int, prog *ebpf.Program, targetLink link.Link) (link.Link, error) { tcxLink, ok := targetLink.(interface{ FD() int }) if !ok { return nil, fmt.Errorf("target link does not expose valid file descriptor for relative anchoring") } opts := link.TCXOptions{ Interface: ifindex, Program: prog, Attach: ebpf.AttachTCXIngress, RelativeFD: tcxLink.FD(), Anchor: link.TCXAfter, // Explicitly sequence after baseline routing filter } l, err := link.AttachTCX(opts) if err != nil { return nil, fmt.Errorf("failed to attach TCX filter deterministically: %w", err) } return l, nil } ``` ### C Kernel-Space TCX Packet Filter with Re-injection Loop Breaker #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Re-injecting a packet onto the same ingress interface without checking metadata (skb->mark) causes an unbounded kernel softirq loop (ksoftirqd pegged at 100%). Returning TCX_PASS halts the execution chain for other coexisting TCX links. ```c // BAD: Blind re-injection with bpf_redirect on ingress causes recursive loop // and returns TCX_PASS, terminating the TCX chain without notifying downstream links. #include <linux/bpf.h> #include <bpf/bpf_helpers.h> SEC("tc/ingress") int bad_ingress_filter(struct __sk_buff *skb) { // Re-inject without checking if this packet has already been redirected bpf_redirect(skb->ifindex, 0); return TCX_PASS; // Terminates multi-prog chain prematurely } ``` #### Production Standard (Correct) > **Standard**: The safe kernel program checks `skb->mark` for a magic loop signature and enforces a maximum hop ceiling. Returning `TCX_NEXT` (-1) preserves the multi-program execution order so peer TCX programs on the same interface can process the packet. ```c // GOOD: Recursion guard via skb->mark inspection and deterministic TCX_NEXT propagation #include <linux/bpf.h> #include <bpf/bpf_helpers.h> #define REINJECT_MAGIC 0xbeef0000 #define REINJECT_MASK 0xffff0000 #define HOP_COUNT_MASK 0x000000ff #define MAX_REINJECT_HOPS 3 #define TCX_NEXT -1 #define TCX_PASS 0 #define TCX_DROP 2 SEC("tcx/ingress") int safe_ingress_filter(struct __sk_buff *skb) { __u32 mark = skb->mark; // Verify whether this packet was previously re-injected by our chain if ((mark & REINJECT_MASK) == REINJECT_MAGIC) { __u32 hops = mark & HOP_COUNT_MASK; if (hops >= MAX_REINJECT_HOPS) { // Hard recursion ceiling reached; drop packet to break infinite loop return TCX_DROP; } // Increment hop counter and preserve downstream propagation skb->mark = REINJECT_MAGIC | (hops + 1); return TCX_NEXT; } // Mark the packet before redirection to enforce recursion invariance skb->mark = REINJECT_MAGIC | 1; // Forward to target veth interface or pass to remaining TCX links return TCX_NEXT; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF TCX Multi-Program Packet Re-Injection Scheduling & Deterministic Chain Ordering

eBPF TCX Multi-Program Packet Re-Injection Scheduling & Deterministic Chain Ordering defines an industrial-grade behavioral standard for devops engineering. Guarantees deterministic execution ordering across coexisting eBPF TCX (Traffic Control BPF link) attachments on network interfaces to eliminate silent packet loss and circular re-injection loops. Enforces explicit relative link positioning flags (BPF_F_BEFORE/BPF_F_AFTER) alongside strict packet-marking invariants to track re-injection depth across kernel ingress filter links. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Non-Deterministic TCX Head Preemption and Unbounded Ingress Re-Injection Loops Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Non-Deterministic TCX Head Preemption and Unbounded Ingress Re-Injection Loops	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS specify explicit attachment ordering using BPF_F_BEFORE, BPF_F_AFTER, or BPF_F_ID when creating TCX links (BPF_LINK_TYPE_TCX) instead of relying on default head-of-chain preemption.

Engineers MUST ALWAYS specify explicit attachment ordering using BPF_F_BEFORE, BPF_F_AFTER, or BPF_F_ID when creating TCX links (BPF_LINK_TYPE_TCX) instead of relying on default head-of-chain preemption.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.