OpenTelemetry eBPF Dynamic Flow Tail-Sampling Kernel Governor

--- description: OpenTelemetry eBPF Dynamic Flow Tail-Sampling Kernel Governor - Enforces in-kernel ring-buffer backpressure and dynamic tail-sampling for high-cardinality eBPF network flow telemetry during microbursts. Prevents user-space collector out-of-memory (OOM) crashes by shedding deterministic, healthy flow events in kernel space while guaranteeing 100% trace extraction for anomalous, high-latency, or errored socket transactions. globs: **/*.c, **/*.bpf.c, **/*.bpf.h, pkg/ebpf/**/*.go, src/collector/**/*.rs, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-flow-tail-sampling-governor" --- # OpenTelemetry eBPF Dynamic Flow Tail-Sampling Kernel Governor > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Head-Sampling Ring Buffer Emission Leading to Collector Node OOM ## System Constraints You are a Principal Observability and Systems Architect enforcing kernel-level telemetry governance. Uncontrolled socket telemetry extraction via eBPF can easily exhaust memory bandwidth and crash collector daemons under TCP microburst conditions. You must enforce two-phase tail-sampling: compute latency windows and TCP error states directly in BPF maps, deferring trace emission until connection close or threshold breach, and dropping nominal keep-alive flows before they cross into user space. ## Strict Guidelines - Engineers MUST ALWAYS implement local BPF hash maps (e.g., BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_LRU_HASH) to hold active socket state until a transaction terminates or exceeds an anomaly threshold. - Engineers MUST NOT forward raw TCP packet headers directly to user space via BPF_MAP_TYPE_PERF_EVENT_ARRAY or BPF_MAP_TYPE_RINGBUF prior to classification. - Kernels MUST dynamically apply an adaptive token bucket governor that downsamples nominal HTTP 200 / TCP FIN flows when collector consumer queue depth breaches the high-watermark threshold (>= 80%). - Services MUST guarantee deterministic delivery of telemetry frames tagged with TCP retransmissions (TCP_CA_Loss), socket-level errors (sk_err != 0), or latencies exceeding service-level SLO thresholds. - Collectors MUST expose a shared memory or atomic state flag (via BPF_MAP_TYPE_ARRAY) communicating user-space heap pressure back to the in-kernel filter for real-time backpressure. ## Bad vs Good Implementations ### In-Kernel Adaptive Sampling vs. Indiscriminate Head-Sampling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Emitting an event per payload segment floods the ring buffer during high-throughput microbursts. This exhausts collector memory buffers and causes catastrophic telemetry loss via user-space OOM termination without capturing diagnostic context. ```c // BAD: Emits an event on every single socket transmission without state or burst protection SEC("kprobe/tcp_sendmsg") int BPF_KPROBE(trace_tcp_sendmsg, struct sock *sk, struct msghdr *msg, size_t size) { struct flow_event_t event = {}; event.pid = bpf_get_current_pid_tgid() >> 32; event.bytes = size; event.ts_ns = bpf_ktime_get_ns(); // Directly emits to user space ring buffer, inducing OOM during network microbursts bpf_ringbuf_output(&flow_events, &event, sizeof(event), 0); return 0; } ``` #### Production Standard (Correct) > **Standard**: Flow state is tracked inside kernel memory and governed by runtime backpressure parameters updated by the collector. Only anomalous flows (high latency, socket errors) or valid tail samples reach user space, preserving system stability and signal fidelity. ```c // GOOD: Aggregates in an LRU map and conditionally emits via adaptive tail-sampling SEC("kprobe/tcp_sendmsg") int BPF_KPROBE(trace_tcp_sendmsg, struct sock *sk, struct msghdr *msg, size_t size) { u64 sk_key = (u64)sk; struct flow_state *state = bpf_map_lookup_elem(&active_flows, &sk_key); u64 now = bpf_ktime_get_ns(); if (!state) { struct flow_state init = { .start_ns = now, .bytes = size, .is_anomalous = 0 }; bpf_map_update_elem(&active_flows, &sk_key, &init, BPF_ANY); return 0; } state->bytes += size; u64 duration = now - state->start_ns; // Read real-time backpressure governor signaled by user-space collector u32 zero = 0; struct governor_cfg *cfg = bpf_map_lookup_elem(&governor_control, &zero); u32 sample_rate = (cfg && cfg->shed_nominal) ? cfg->nominal_rate_pct : 100; // Anomaly criteria: SLO breach, socket error, or random sampling pass if (duration > SLO_LATENCY_NS || BPF_CORE_READ(sk, sk_err) != 0) { state->is_anomalous = 1; } // Defer user-space ringbuf submission: only emit immediately if anomalous and unbuffered if (state->is_anomalous || (bpf_get_prandom_u32() % 100) < sample_rate) { struct flow_event_t *event = bpf_ringbuf_reserve(&flow_events, sizeof(*event), 0); if (event) { event->duration_ns = duration; event->bytes = state->bytes; event->anomalous = state->is_anomalous; bpf_ringbuf_submit(event, 0); } } return 0; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Dynamic Flow Tail-Sampling Kernel Governor

OpenTelemetry eBPF Dynamic Flow Tail-Sampling Kernel Governor defines an industrial-grade behavioral standard for devops engineering. Enforces in-kernel ring-buffer backpressure and dynamic tail-sampling for high-cardinality eBPF network flow telemetry during microbursts. Prevents user-space collector out-of-memory (OOM) crashes by shedding deterministic, healthy flow events in kernel space while guaranteeing 100% trace extraction for anomalous, high-latency, or errored socket transactions. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Head-Sampling Ring Buffer Emission Leading to Collector Node OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Head-Sampling Ring Buffer Emission Leading to Collector Node OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement local BPF hash maps (e.g., BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_LRU_HASH) to hold active socket state until a transaction terminates or exceeds an anomaly threshold.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.