eBPF Continuous Profiling DWARF Unwinding Buffer-Pressure Throttling Invariant

--- description: eBPF Continuous Profiling DWARF Unwinding Buffer-Pressure Throttling Invariant - Enforces adaptive ring-buffer backpressure and sampling frequency throttling during DWARF unwinding in production continuous profilers. Uncontrolled symbolization and kernel-to-userspace stack trace streaming under high container density will cause kernel ring-buffer overflows, silent telemetry frame drops, and severe soft-lockup/CPU exhaustion. The architecture mandates feedback loops between userspace ring-buffer consumer watermark metrics and kernel BPF sampling interval maps. globs: src/profiler/**/*.c, src/profiler/**/*.rs, src/profiler/**/*.go, bpf/**/*.bpf.c, pkg/collector/**/*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-profiler-dwarf-buffer-throttling-invariant" --- # eBPF Continuous Profiling DWARF Unwinding Buffer-Pressure Throttling Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static-Rate Unwinding Without Adaptive Watermark Backpressure and Dynamic Subsampling ## System Constraints You are a Principal Systems & Observability Architect enforcing safety invariants for eBPF-based continuous profilers using DWARF/ORC unwinding. You must prevent telemetry loss and CPU spikes by mandating kernel-level sampling frequency throttling driven by userspace ring-buffer consumption watermarks and strict page fault boundaries. ## Strict Guidelines - Profilers MUST implement an adaptive BPF sampling frequency map (BPF_MAP_TYPE_ARRAY) that userspace dynamically updates based on epoll consumer lag and ring buffer utilization percentage. - Profilers MUST evaluate ring-buffer watermarks before initiating userspace memory reads via `bpf_probe_read_user()` or pushing stack chunks to `BPF_MAP_TYPE_RINGBUF`. - Profilers MUST NOT perform unthrottled or full-depth DWARF frame-pointerless unwinding when ring-buffer consumption crosses the critical high-watermark threshold (default >= 75% capacity). - BPF programs MUST track and export monotonic drop counters via dedicated per-CPU arrays (`perf_event_output` or dedicated stat structs) to guarantee auditability when throttling drops samples. - Userspace telemetry agents MUST gracefully degrade unwinding fidelity (e.g., fallback from full DWARF table walk to heuristic frame pointer or truncating max frame depth from 128 to 32) before discarding samples entirely. ## Bad vs Good Implementations ### Adaptive Ring-Buffer Backpressure & Dynamic Sampling Throttle in eBPF C #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The BPF program uses a fixed unwinding depth and blindly attempts to push to the ring buffer. Under heavy workload or high container density, userspace cannot consume samples fast enough. This leads to dropped frames without backpressure, wasting CPU cycles on doomed unwinding attempts and potentially saturating the memory bus. ```c // BAD: Fixed sampling, unmonitored ring buffer submission, unbounded unwinding #include <vmlinux.h> #include <bpf/bpf_tracing.h> struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 16 * 1024 * 1024); } stack_ringbuf SEC(".maps"); SEC("perf_event") int profile_sample(struct bpf_perf_event_data *ctx) { struct stack_record *rec; // Blindly allocates without checking consumer saturation or drop status rec = bpf_ringbuf_reserve(&stack_ringbuf, sizeof(*rec), 0); if (!rec) { // Drops sample silently, continues burning CPU on subsequent samples return 0; } // Always reads max depth (128 frames) using expensive DWARF userspace memory access #pragma unroll for (int i = 0; i < 128; i++) { // Heavy userspace dereferences causing page fault stalls rec->ips[i] = ctx->regs.ip; } bpf_ringbuf_submit(rec, 0); return 0; } ``` #### Production Standard (Correct) > **Standard**: Introduces an adaptive control plane between userspace ring-buffer consumer metrics and the eBPF kernel program. When userspace observes ring-buffer capacity exceeding safety thresholds, it downshifts the profiler into degraded (reduced unwinding depth) or suppressed states, eliminating CPU thrashing and recording dropped frames deterministically. ```c // GOOD: Watermark-aware adaptive throttling and unwinding backpressure #include <vmlinux.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> #define THROTTLE_MAP_IDX 0 #define STATUS_NORMAL 0 #define STATUS_DEGRADED 1 #define STATUS_SUPPRESSED 2 struct throttle_cfg { __u32 state; // Controlled by userspace telemetry consumer __u32 max_depth; // Dynamic stack unwinding limit __u64 dropped_frames; // Monotonic kernel telemetry drop counter }; struct { __uint(type, BPF_MAP_TYPE_ARRAY); __type(key, __u32); __type(value, struct throttle_cfg); __uint(max_entries, 1); } throttle_map SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 16 * 1024 * 1024); } stack_ringbuf SEC(".maps"); SEC("perf_event") int profile_sample(struct bpf_perf_event_data *ctx) { __u32 key = THROTTLE_MAP_IDX; struct throttle_cfg *cfg = bpf_map_lookup_elem(&throttle_map, &key); if (!cfg) return 0; // Circuit breaker: drops instantly if userspace reports critical ring-buffer pressure if (cfg->state == STATUS_SUPPRESSED) { __sync_fetch_and_add(&cfg->dropped_frames, 1); return 0; } // Restrict walk depth dynamically based on userspace drain watermark __u32 depth_limit = (cfg->state == STATUS_DEGRADED) ? 32 : cfg->max_depth; if (depth_limit > 128) depth_limit = 128; struct stack_record *rec = bpf_ringbuf_reserve(&stack_ringbuf, sizeof(*rec), 0); if (!rec) { __sync_fetch_and_add(&cfg->dropped_frames, 1); return 0; } // Execute bounded walk within verified throttle parameters rec->depth = 0; #pragma unroll for (int i = 0; i < 32; i++) { if (i >= depth_limit) break; rec->ips[i] = ctx->regs.ip; rec->depth++; } bpf_ringbuf_submit(rec, 0); return 0; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: eBPF Continuous Profiling DWARF Unwinding Buffer-Pressure Throttling Invariant

eBPF Continuous Profiling DWARF Unwinding Buffer-Pressure Throttling Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces adaptive ring-buffer backpressure and sampling frequency throttling during DWARF unwinding in production continuous profilers. Uncontrolled symbolization and kernel-to-userspace stack trace streaming under high container density will cause kernel ring-buffer overflows, silent telemetry frame drops, and severe soft-lockup/CPU exhaustion. The architecture mandates feedback loops between userspace ring-buffer consumer watermark metrics and kernel BPF sampling interval maps. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static-Rate Unwinding Without Adaptive Watermark Backpressure and Dynamic Subsampling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static-Rate Unwinding Without Adaptive Watermark Backpressure and Dynamic Subsampling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Profilers MUST implement an adaptive BPF sampling frequency map (BPF_MAP_TYPE_ARRAY) that userspace dynamically updates based on epoll consumer lag and ring buffer utilization percentage.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.