Cilium Tetragon eBPF Host-Level Kernel CPU and Cache Budget Governor

--- description: Cilium Tetragon eBPF Host-Level Kernel CPU and Cache Budget Governor - Enforces strict host-level CPU and L3 cache overhead governance for Cilium Tetragon eBPF tracing programs. Prevents kernel-space telemetry runaway and soft lockups by dynamically throttling, degrading tracing resolution, or detaching non-essential kprobes/tracepoints when host resource budgets breach safety thresholds. globs: deployments/**/*.yaml, helm/**/*.yaml, crds/**/*.yaml, pkg/governor/**/*.go, src/controllers/**/*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/tetragon-ebpf-cpu-governor-guardrail" --- # Cilium Tetragon eBPF Host-Level Kernel CPU and Cache Budget Governor > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded kernel tracing triggers CPU starvation, ring buffer drops, and node instability under heavy syscall workloads. ## System Constraints You are a Principal Systems Engineer and Kernel Observability Architect enforcing runtime safety invariants for Cilium Tetragon eBPF deployments. Unmonitored eBPF programs, deep call-path stack unwinding, and high-frequency kprobes (e.g., sys_enter_read, sys_enter_write) can induce CPU pipeline stalls, thrash L3 CPU caches, and trigger kernel soft lockups under high-load production workloads. You MUST mandate autonomous dynamic overhead governance, rate-limiting in kernel space via BPF token buckets, strict per-CPU ring buffer configurations, and automated circuit-breaking that disables high-overhead TracingPolicies when host CPU or cache miss budgets exceed enterprise safety limits. ## Strict Guidelines - Engineers MUST ALWAYS implement kernel-space rate-limiting filters (e.g., rateLimit / bpf token buckets) inside TracingPolicy specifications for high-frequency kernel symbols. - Engineers MUST NOT attach raw kprobes or kretprobes to high-throughput I/O syscalls (e.g., read, write, recvfrom, epoll_wait) without explicit namespace, binary, or cgroup-level filtering. - TracingPolicy deployments MUST declare an automated feedback governor that monitors node-level L3 cache misses (perf subsystem) and CPU utilization, detaching non-critical probes if Tetragon CPU usage exceeds 2.5% of host capacity. - Services MUST configure explicit ring buffer memory limits (`rb-size` and `rb-size-total`) to prevent unbounded kernel memory allocations and continuous perf_event drops. - Components NEVER allow unbounded string or buffer extraction from kernel memory; field extractors must enforce maximum byte limits (e.g., `sizeArg: 256`) to protect L1/L2/L3 cache residency. ## Bad vs Good Implementations ### Autonomous Governor Controller for Tetragon TracingPolicy Life Cycle #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching an unfiltered kprobe to sys_enter_write across all host processes with unbounded string extraction will saturate the eBPF ring buffer, thrash CPU caches, and induce kernel soft lockups under write-intensive database or logging workloads. ```yaml apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: audit-io-operations spec: kprobes: - call: "sys_enter_write" syscall: true args: - index: 0 type: "int" - index: 1 type: "string" # Unbounded extraction of buffer triggers L3 cache thrashing selectors: - matchArgs: [] # Matches all processes globally without rate limits or namespaces ``` #### Production Standard (Correct) > **Standard**: The TracingPolicy restricts the probe to designated Kubernetes namespaces, binds buffer extraction to a fixed 64-byte envelope, uses an eBPF rate-limiting token bucket, and adds governor metadata labels to enable automated degradation if host CPU thresholds are crossed. ```yaml apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: governed-audit-io-operations labels: governor.tetragon.io/priority: "low-tier" governor.tetragon.io/degrade-order: "1" spec: kprobes: - call: "sys_enter_write" syscall: true args: - index: 0 type: "int" - index: 1 type: "char_buf" sizeArg: 64 # Strictly bounded payload inspection selectors: - matchNamespaces: - namespace: "production-payment" operator: "In" matchArgs: - index: 0 operator: "Equal" values: ["1", "2"] # Stdout/Stderr only rateLimit: rate: 50 unit: "minute" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Cilium Tetragon eBPF Host-Level Kernel CPU and Cache Budget Governor

Cilium Tetragon eBPF Host-Level Kernel CPU and Cache Budget Governor defines an industrial-grade behavioral standard for devops engineering. Enforces strict host-level CPU and L3 cache overhead governance for Cilium Tetragon eBPF tracing programs. Prevents kernel-space telemetry runaway and soft lockups by dynamically throttling, degrading tracing resolution, or detaching non-essential kprobes/tracepoints when host resource budgets breach safety thresholds. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded kernel tracing triggers CPU starvation, ring buffer drops, and node instability under heavy syscall workloads. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded kernel tracing triggers CPU starvation, ring buffer drops, and node instability under heavy syscall workloads.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement kernel-space rate-limiting filters (e.g., rateLimit / bpf token buckets) inside TracingPolicy specifications for high-frequency kernel symbols.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.