OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling

--- description: OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling - Enforces dynamic throttling on eBPF continuous profiling agents to limit ring-buffer saturation and kernel stack-unwinding CPU overhead. Under sustained host compute spikes, agents must autonomously reduce unwinding frequency, fall back to frame-pointer walking, or drop samples to preserve host node stability. globs: infra/profiling/**/*.yaml, deploy/daemonsets/**/*.yaml, src/profiling/**/*.ts, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiler-kernel-overhead-throttling" --- # OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded eBPF Stack Walking and Ring-Buffer Allocations Saturating Production Kernel Cores ## System Constraints You are a Principal Engineer enforcing OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling. Mandate strict CPU overhead budgets, dynamic sampling frequency attenuation, backpressure handling on perf/ring buffers, and fallback mechanics to guarantee zero node starvation caused by telemetry. ## Strict Guidelines - Engineers MUST ALWAYS enforce dynamic CPU utilization budgets for continuous profiling agents, capping total agent overhead to <= 1.5% of host CPU capacity. - Engineers MUST NOT permit static high-frequency sampling (e.g., rigid 99Hz or 100Hz) when the host CPU utilization crosses the critical threshold (>80%). - Code SHOULD dynamically downsample unwinding events and toggle DWARF unwinding off in favor of Frame Pointer walking or stack-depth truncation during high system load. - Services MUST monitor BPF ring-buffer drop rates (`BPF_MAP_TYPE_RINGBUF` / `BPF_MAP_TYPE_PERF_EVENT_ARRAY`) and trigger exponential backoff on sample generation when buffer saturation exceeds 5%. - Components NEVER lock the eBPF event submission pipeline synchronously in critical kernel execution paths without non-blocking fallback (`BPF_RB_NO_WAKEUP`). ## Bad vs Good Implementations ### Dynamic Throttling Controller for OTel Profiler Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configures a rigid 99Hz sampling frequency using resource-intensive DWARF unwinding without host load monitoring or ring-buffer overflow backpressure. This creates significant kernel compute overhead and risks kernel thread starvation during production peak loads. ```typescript import { ProfilerAgentConfig } from './types'; export function configureProfiler(): ProfilerAgentConfig { // Anti-Pattern: Static aggressive sampling without load adaptation or ring-buffer safeguards return { samplingFrequencyHz: 99, unwindStrategy: 'dwarf', maxStackDepth: 128, ringBufferSizeKb: 4096, dynamicThrottlingEnabled: false, cpuBudgetPercent: 10.0 }; } ``` #### Production Standard (Correct) > **Standard**: Implements an adaptive profiling controller that actively evaluates node CPU utilization and ring-buffer drop rates. It throttles sampling frequency, drops heavy DWARF unwinding for frame pointers, and enforces a strict 1.5% CPU overhead ceiling. ```typescript import { ProfilerAgentConfig, HostMetrics } from './types'; export function calculateAdaptiveProfilerConfig(metrics: HostMetrics): ProfilerAgentConfig { const MAX_CPU_BUDGET_PERCENT = 1.5; const HIGH_LOAD_THRESHOLD = 0.80; const CRITICAL_LOAD_THRESHOLD = 0.90; // Default healthy baseline let frequency = 19; let unwindStrategy: 'dwarf' | 'frame-pointer' = 'dwarf'; let maxDepth = 64; if (metrics.hostCpuUsageRatio >= CRITICAL_LOAD_THRESHOLD || metrics.ringBufferDropRate > 0.05) { // Critical overhead: attenuate severely to safeguard compute capacity frequency = 5; unwindStrategy = 'frame-pointer'; maxDepth = 32; } else if (metrics.hostCpuUsageRatio >= HIGH_LOAD_THRESHOLD) { // High load: downsample unwinding frequency frequency = 11; unwindStrategy = 'frame-pointer'; maxDepth = 48; } return { samplingFrequencyHz: frequency, unwindStrategy, maxStackDepth: maxDepth, ringBufferSizeKb: 8192, dynamicThrottlingEnabled: true, cpuBudgetPercent: MAX_CPU_BUDGET_PERCENT }; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling

OTel Continuous Profiling eBPF Stack-Unwinding Kernel Overhead Throttling defines an industrial-grade behavioral standard for devops engineering. Enforces dynamic throttling on eBPF continuous profiling agents to limit ring-buffer saturation and kernel stack-unwinding CPU overhead. Under sustained host compute spikes, agents must autonomously reduce unwinding frequency, fall back to frame-pointer walking, or drop samples to preserve host node stability. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded eBPF Stack Walking and Ring-Buffer Allocations Saturating Production Kernel Cores Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded eBPF Stack Walking and Ring-Buffer Allocations Saturating Production Kernel Cores	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce dynamic CPU utilization budgets for continuous profiling agents, capping total agent overhead to <= 1.5% of host CPU capacity.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.