eBPF-Driven Kubernetes Node Ephemeral Port Exhaustion Socket Guardrail

--- description: eBPF-Driven Kubernetes Node Ephemeral Port Exhaustion Socket Guardrail - Intercepts and regulates rapid SNAT allocation churn and ephemeral port depletion at the Linux kernel socket layer before pod traffic degrades. Prevents node-level silent socket drops and Conntrack table thrashing caused by un-pooled HTTP/gRPC egress clients and misconfigured conntrack sysctls. globs: daemonsets/**/*.yaml, bpf/**/*.c, bpf/**/*.bpf.c, deploy/k8s/**/*.yaml, pkg/ebpf/**/*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-k8s-ephemeral-port-exhaustion-socket-guardrail" --- # eBPF-Driven Kubernetes Node Ephemeral Port Exhaustion Socket Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Un-pooled TCP/UDP Pod Egress Triggering Node Ephemeral Port Exhaustion and Silent Kernel Drops ## System Constraints You are a Principal Systems and Infrastructure Architect enforcing industrial-grade Linux kernel and Kubernetes network invariants. Your objective is to mandate proactive socket-level telemetry and enforcement using eBPF to halt node-level ephemeral port exhaustion (EADDRNOTAVAIL), conntrack table saturation, and SNAT port collisions before pod-to-pod and pod-to-external communication degrades. ## Strict Guidelines - Engineers MUST ALWAYS deploy eBPF probes at `sock:inet_csk_get_port` or attach `BPF_PROG_TYPE_CGROUP_SOCK_ADDR` hooks (`connect4`/`connect6`) to monitor and throttle socket allocation velocities per cgroup/pod. - Engineers MUST NOT allow daemonsets or application pods to omit `net.ipv4.ip_local_port_range` tuning and connection reuse configurations when driving high outbound request volumes. - eBPF programs MUST maintain a lockless BPF LRU hash map (`BPF_MAP_TYPE_LRU_HASH`) tracking active 4-tuples and egress allocation frequency to preemptively return `-EADDRNOTAVAIL` or trigger backpressure prior to conntrack ring-buffer overflow. - Kubernetes node bootstrap scripts MUST set `net.ipv4.tcp_tw_reuse = 1` and configure `net.netfilter.nf_conntrack_max` to at least 4x the total ephemeral socket capacity. - Services MUST implement TCP Keep-Alive, HTTP persistent pooling, or gRPC subchannel re-use to limit the `TIME_WAIT` socket churn on the host network namespace. - DaemonSets embedding eBPF programs MUST verify `BPF_F_NO_PREALLOC` flags and memory limits via `rlimit` (RLIMIT_MEMLOCK) to prevent kernel out-of-memory panics during high socket creation spikes. ## Bad vs Good Implementations ### eBPF Socket Connect Hook with Ephemeral Port Exhaustion Guard #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Passive tracepoints cannot intercept or short-circuit socket allocation failure cascades. When ephemeral ports are depleted, the kernel falls into spinlocks attempting to locate free ports in the hash table, driving high softirq latency and dropping SYN packets. ```c #include <linux/bpf.h> #include <bpf/bpf_helpers.h> // ANTI-PATTERN: Passive tracing without enforcing quota or detecting SNAT pool exhaustion SEC("tracepoint/sock/inet_csk_get_port") int trace_inet_csk_get_port(void *ctx) { // Merely increments a counter without rate-limiting or blocking bpf_printk("Socket port requested\n"); return 0; } ``` #### Production Standard (Correct) > **Standard**: Implements an active `cgroup/connect4` BPF hook that measures and enforces socket limits per cgroup slice. If a misbehaving pod approaches exhaustion limits, the kernel denies socket creation cleanly before SNAT allocation degrades the root network namespace. ```c #include <vmlinux.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> #define MAX_PORTS_PER_CGROUP 28000 #define EADDRNOTAVAIL 99 struct { __uint(type, BPF_MAP_TYPE_CGROUP_ARRAY); __type(key, u32); __type(value, u32); __uint(max_entries, 1024); } cgroup_map SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __type(key, u64); /* cgroup_id */ __type(value, u32); /* active_outbound_sockets */ __uint(max_entries, 65536); } socket_tracker SEC(".maps"); SEC("cgroup/connect4") int guard_port_exhaustion(struct bpf_sock_addr *ctx) { u64 cgroup_id = bpf_get_current_cgroup_id(); u32 *active_ports = bpf_map_lookup_elem(&socket_tracker, &cgroup_id); if (active_ports && *active_ports >= MAX_PORTS_PER_CGROUP) { /* Proactively deny egress connection to prevent kernel socket exhaustion */ return 0; /* Drops connection initiation early */ } u32 initial_count = 1; if (!active_ports) { bpf_map_update_elem(&socket_tracker, &cgroup_id, &initial_count, BPF_NOEXIST); } else { __sync_fetch_and_add(active_ports, 1); } return 1; } char _license[] SEC("license") = "GPL"; ``` ### Node Tuning DaemonSet with Conntrack and Ephemeral Port Guardrails #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Disabling tcp_tw_reuse and leaving default ephemeral port limits (32768-60999) on high-density nodes causes rapid exhaustion under bursty microservice communication, causing node-wide outbound network blackouts. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: node-network-init spec: template: spec: containers: - name: sysctl-tuner image: busybox # ANTI-PATTERN: Default small ephemeral range and unoptimized conntrack command: ["sh", "-c", "sysctl -w net.ipv4.tcp_tw_reuse=0"] ``` #### Production Standard (Correct) > **Standard**: Expands the local ephemeral port range, safely permits reuse of TIME_WAIT sockets for outgoing connections, expands conntrack tables to absorb traffic spikes, and drops conntrack TIME_WAIT state timeout from 120s to 30s. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: node-network-guard namespace: kube-system spec: selector: matchLabels: app: node-network-guard template: metadata: labels: app: node-network-guard spec: hostNetwork: true initContainers: - name: sysctl-tune image: alpine:3.19 securityContext: privileged: true command: ["sh", "-c", "sysctl -w net.ipv4.ip_local_port_range='10240 65535' && sysctl -w net.ipv4.tcp_tw_reuse=1 && sysctl -w net.netfilter.nf_conntrack_max=1048576 && sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30"] containers: - name: pause image: registry.k8s.io/pause:3.9 ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF-Driven Kubernetes Node Ephemeral Port Exhaustion Socket Guardrail

eBPF-Driven Kubernetes Node Ephemeral Port Exhaustion Socket Guardrail defines an industrial-grade behavioral standard for devops engineering. Intercepts and regulates rapid SNAT allocation churn and ephemeral port depletion at the Linux kernel socket layer before pod traffic degrades. Prevents node-level silent socket drops and Conntrack table thrashing caused by un-pooled HTTP/gRPC egress clients and misconfigured conntrack sysctls. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Un-pooled TCP/UDP Pod Egress Triggering Node Ephemeral Port Exhaustion and Silent Kernel Drops Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Un-pooled TCP/UDP Pod Egress Triggering Node Ephemeral Port Exhaustion and Silent Kernel Drops	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS deploy eBPF probes at `sock

Engineers MUST ALWAYS deploy eBPF probes at `sock:inet_csk_get_port` or attach `BPF_PROG_TYPE_CGROUP_SOCK_ADDR` hooks (`connect4`/`connect6`) to monitor and throttle socket allocation velocities per cgroup/pod.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.