OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Budget Guardrail

--- description: OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Budget Guardrail - Enforces an autonomous negative-feedback loop that bounds OpenTelemetry eBPF stack-unwinding agent CPU utilization strictly below 1.0% of total host capacity. During host CPU saturation or burst contention, the profiling daemon must dynamically back off sampling frequency from standard targets (e.g., 19 Hz) down to a degraded floor (e.g., 1-5 Hz) or trigger an emergency circuit breaker to prevent compounding node exhaustion. globs: deploy/k8s/**/*.{yaml,yml}, configs/profiling/**/*.{yaml,yml,json,toml}, src/observability/**/*.{ts,js,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-continuous-profiling-overhead-budget-guardrail" --- # OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Budget Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static Unthrottled eBPF Sampling Frequencies Causing Kernel Unwind Storms During Host CPU Contention ## System Constraints You are a Principal Systems and Observability Architect enforcing the OpenTelemetry eBPF Continuous Profiling Overhead Budget Guardrail. Mandate invariants that keep eBPF unwinding and collection daemon CPU consumption strictly below 1% of host capacity across user-space (unwind DWARF/frame pointers) and kernel-space execution. Enforce dynamic adaptive sampling, autonomous back-off thresholds, cgroup resource bounding, and fail-safe circuit breaking. ## Strict Guidelines - Profiling daemons MUST ALWAYS run within dedicated Linux cgroups with hard CPU limits set to no more than 1% of the node's total allocatable CPU (e.g., 40m CPU on a 4-core node). - Dynamic controllers MUST continuously evaluate host CPU load and agent self-consumption via `/proc/stat` or cgroup metrics at sub-second intervals. - Engineers MUST NOT deploy static high-frequency eBPF sampling rates (>19 Hz) without runtime negative-feedback throttling loops. - Profiling agents MUST dynamically scale down unwinding frequency from baseline (e.g., 19 Hz) to safe fallbacks (e.g., 5 Hz or 1 Hz) whenever overall host CPU utilization exceeds 80%. - Components NEVER continue user-space frame unwinding if native symbolization queues back up; unresolvable stacks MUST be shed immediately rather than retried in hot execution paths. - The agent MUST trigger an autonomous emergency circuit breaker that unloads or detaches BPF `PERF_EVENT` probes when host CPU load exceeds 95% or agent CPU consumption breaches the 1% threshold for more than 3 consecutive evaluation cycles. ## Bad vs Good Implementations ### Dynamic Adaptive Sampling Rate Controller for eBPF Profiling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring static 99 Hz eBPF sampling with unthrottled DWARF unwinding and undefined CPU limits causes the profiler to compete for CPU cycles during node saturation, creating cascading kernel lock contention and breaching the 1% overhead budget. ```typescript import { ProfilingAgent } from '@opentelemetry/experimental-profiling'; // ANTI-PATTERN: Static high-frequency sampling with no overhead protection or back-off export function startContinuousProfiling(): void { const agent = new ProfilingAgent({ sampleRateHz: 99, // Static aggressive rate creates severe kernel stack-unwinding load collectKernelStacks: true, collectUserStacks: true, unwindDwarf: true, cpuLimitPercent: undefined, // Unbounded CPU footprint }); agent.start(); // Daemon ignores host CPU pressure and continues unwinding during 100% saturation spikes } ``` #### Production Standard (Correct) > **Standard**: Implements an adaptive closed-loop controller that monitors agent and host CPU consumption every second, scaling sampling frequencies between 19 Hz and 1 Hz, and activating an emergency zero-rate circuit breaker if agent overhead exceeds 1% or host CPU exceeds 95%. ```typescript import { ProfilingAgent } from '@opentelemetry/experimental-profiling'; import * as os from 'node:os'; interface HostLoadMetrics { totalHostCpuPercent: number; agentCpuPercent: number; } export class AdaptiveProfilingSupervisor { private agent: ProfilingAgent; private timer: NodeJS.Timeout | null = null; private static readonly MAX_AGENT_CPU_PERCENT = 1.0; // Invariant: Strict <= 1% overhead private static readonly BASE_SAMPLE_RATE_HZ = 19; // Prime sampling rate to avoid harmonic bias private static readonly MIN_SAMPLE_RATE_HZ = 1; constructor() { this.agent = new ProfilingAgent({ sampleRateHz: AdaptiveProfilingSupervisor.BASE_SAMPLE_RATE_HZ, collectKernelStacks: true, collectUserStacks: true, enableDynamicRateAdjustment: true }); } public start(): void { this.agent.start(); this.timer = setInterval(() => this.enforceOverheadBudget(), 1000); } private enforceOverheadBudget(): void { const metrics = this.readHostMetrics(); // Circuit breaker: Emergency detach if agent breaches 1% or host hits critical saturation if (metrics.agentCpuPercent > AdaptiveProfilingSupervisor.MAX_AGENT_CPU_PERCENT || metrics.totalHostCpuPercent > 95.0) { this.agent.setSampleRate(0); // Pause unwinding completely return; } // Proportional back-off if host is under pressure if (metrics.totalHostCpuPercent > 80.0) { this.agent.setSampleRate(AdaptiveProfilingSupervisor.MIN_SAMPLE_RATE_HZ); } else { this.agent.setSampleRate(AdaptiveProfilingSupervisor.BASE_SAMPLE_RATE_HZ); } } private readHostMetrics(): HostLoadMetrics { // Implementation reading cgroup cpu.stat and /proc/stat return { totalHostCpuPercent: 45.0, agentCpuPercent: 0.35 }; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Budget Guardrail

OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Budget Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces an autonomous negative-feedback loop that bounds OpenTelemetry eBPF stack-unwinding agent CPU utilization strictly below 1.0% of total host capacity. During host CPU saturation or burst contention, the profiling daemon must dynamically back off sampling frequency from standard targets (e.g., 19 Hz) down to a degraded floor (e.g., 1-5 Hz) or trigger an emergency circuit breaker to prevent compounding node exhaustion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static Unthrottled eBPF Sampling Frequencies Causing Kernel Unwind Storms During Host CPU Contention Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static Unthrottled eBPF Sampling Frequencies Causing Kernel Unwind Storms During Host CPU Contention	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Profiling daemons MUST ALWAYS run within dedicated Linux cgroups with hard CPU limits set to no more than 1% of the node's total allocatable CPU (e.g., 40m CPU on a 4-core node).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.