Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation

--- description: Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation - Enforces Direct Server Return (DSR) or Local external traffic policies alongside scaled BPF connection tracking maps and deterministic ephemeral port ranges. This prevents silent packet drops, connection blackholing, and cascading node partitions under high-concurrency connection churn in Cilium-managed Kubernetes clusters. globs: **/cilium*.yaml, **/helm/**/cilium/**, **/values*.yaml, **/infra/**/cilium*.tf, **/k8s/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/cilium-ebpf-host-routing-snat-exhaustion-mitigation" --- # Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Default SNAT NodePort Routing with Default BPF Conntrack Table Limits Under High Concurrency ## System Constraints You are a Principal Infrastructure Architect enforcing Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation. Mandate calibrated BPF conntrack table boundaries, Direct Server Return (DSR) or Local external traffic policies, aggressive TCP conntrack garbage collection, and expanded ephemeral source port allocations across all Kubernetes and Helm configurations. Reject uncalibrated default conntrack map sizes and unconstrained cluster-wide SNAT node-hopping. ## Strict Guidelines - Engineers MUST ALWAYS enforce `bpf.ctMapEntriesGlobalTCP` and `bpf.ctMapEntriesGlobalAny` sizing based on peak concurrent connection models (minimum 1,048,576 for TCP in production) rather than relying on default limits. - Engineers MUST NOT permit `loadBalancer.mode: snat` for high-throughput public ingress services without explicit ephemeral port reservation and BPF masquerade tuning. - Code SHOULD configure `loadBalancer.mode: dsr` or `loadBalancer.mode: hybrid` with native routing (`routingMode: native`) to preserve client source IPs and eliminate return-path SNAT state exhaustion. - Services MUST enforce `externalTrafficPolicy: Local` on all high-concurrency NodePort or LoadBalancer services to prevent inter-node hops and avoid node-level SNAT overhead. - Components NEVER execute with Linux host default ephemeral port ranges (`net.ipv4.ip_local_port_range`) alongside Cilium BPF masquerading without expanding the pool to `10240 65535` and tuning `conntrackGCInterval` to 60 seconds or lower. ## Bad vs Good Implementations ### Cilium Helm Values: Default SNAT vs Hardened DSR and Conntrack Tuning #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default SNAT mode rewrites backend traffic across nodes, allocating source ports on intermediate nodes. Under high concurrency or microbursts, the unscaled BPF conntrack tables fill up, resulting in kernel drops with 'BPF CT: Map full, dropping packet' errors and severe cluster-wide packet loss. ```yaml # Anti-Pattern: Default Helm values lead to SNAT exhaustion under heavy load tunnel: disabled routingMode: native bpf: masquerade: true hostRouting: true # Default conntrack limits (default: 524288 / 262144) will exhaust under bursts # loadBalancer.mode defaults to snat, rewriting back-and-forth across nodes loadBalancer: mode: snat ``` #### Production Standard (Correct) > **Standard**: Configures Direct Server Return (DSR) using native eBPF host routing, eliminating return-path SNAT entirely. Conntrack maps are scaled to handle over 1 million concurrent flows, garbage collection intervals are halved, and the Maglev consistent hash scheduler provides resilient, deterministic load balancing across worker nodes. ```yaml # Hardened Production Invariant: Direct Server Return with Scaled Conntrack tunnel: disabled routingMode: native autoDirectNodeRoutes: true bpf: masquerade: true hostRouting: true ctMapEntriesGlobalTCP: 1048576 ctMapEntriesGlobalAny: 524288 conntrackGCInterval: "30s" tcpTimeoutEstablished: "3600s" tcpTimeoutFinWait: "30s" loadBalancer: mode: dsr acceleration: native algorithm: maglev dsrDispatch: opt nodePort: enabled: true range: "30000,32767" ``` ### Ingress Service Traffic Policy: Cluster vs Local Anti-Pattern #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: externalTrafficPolicy: Cluster causes random inter-node traffic forwarding. The ingress node must SNAT the packet before forwarding it to the target pod on another node, multiplying connection tracking entries and exhausting source ports. ```yaml apiVersion: v1 kind: Service metadata: name: ingress-gateway namespace: ingress-system spec: type: LoadBalancer externalTrafficPolicy: Cluster ports: - port: 443 targetPort: 8443 protocol: TCP ``` #### Production Standard (Correct) > **Standard**: Setting externalTrafficPolicy: Local ensures the external load balancer routes traffic only to nodes running local pod instances. This bypasses secondary inter-node hops, preserves client IP without SNAT, and prevents conntrack map saturation. ```yaml apiVersion: v1 kind: Service metadata: name: ingress-gateway namespace: ingress-system spec: type: LoadBalancer externalTrafficPolicy: Local allocateLoadBalancerNodePorts: true ports: - port: 443 targetPort: 8443 protocol: TCP ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation

Cilium eBPF Host-Routing NodePort SNAT Exhaustion Mitigation defines an industrial-grade behavioral standard for devops engineering. Enforces Direct Server Return (DSR) or Local external traffic policies alongside scaled BPF connection tracking maps and deterministic ephemeral port ranges. This prevents silent packet drops, connection blackholing, and cascading node partitions under high-concurrency connection churn in Cilium-managed Kubernetes clusters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Default SNAT NodePort Routing with Default BPF Conntrack Table Limits Under High Concurrency Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Default SNAT NodePort Routing with Default BPF Conntrack Table Limits Under High Concurrency	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce `bpf.ctMapEntriesGlobalTCP` and `bpf.ctMapEntriesGlobalAny` sizing based on peak concurrent connection models (minimum 1,048,576 for TCP in production) rather than relying on default limits.

Engineers MUST ALWAYS enforce `bpf.ctMapEntriesGlobalTCP` and `bpf.ctMapEntriesGlobalAny` sizing based on peak concurrent connection models (minimum 1,048,576 for TCP in production) rather than relying on default limits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.