OpenTelemetry eBPF Continuous Profiler Adaptive CFS Quota Throttling Guardrail

--- description: OpenTelemetry eBPF Continuous Profiler Adaptive CFS Quota Throttling Guardrail - Continuous eBPF stack-unwinding profiles introduce measurable kernel CPU overhead under high context-switch frequencies. Profiling agents must actively monitor target cgroup cpu.stat metrics and dynamically back off sampling frequency or pause unwind iterations when target cgroups exceed configurable CFS throttling thresholds, preventing profiling-induced cascade latency. globs: src/**/*.{ts,tsx,go,rs,c,cpp}, deploy/**/*.{yaml,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiler-cfs-adaptive-throttling" --- # OpenTelemetry eBPF Continuous Profiler Adaptive CFS Quota Throttling Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static high-frequency eBPF sampling causing kernel-level CFS starvation and cascade pod throttling ## System Constraints You are a Principal Systems & Observability Architect enforcing strict runtime safety for OpenTelemetry eBPF continuous profilers. You must enforce dynamic sampling back-off and unwinding suppression when workload containers experience CFS bandwidth exhaustion. ## Strict Guidelines - Engineers MUST ALWAYS poll target cgroup v2 cpu.stat (`nr_throttled` and `throttled_usec`) before configuring or updating BPF perf-event sampling frequency. - Engineers MUST NOT maintain static sampling frequencies (e.g., rigid 19Hz/99Hz) across heterogeneous workloads without adaptive back-off controls. - Code SHOULD implement an exponential backoff controller that decrements sampling frequency towards a safe lower bound (e.g., 5Hz or complete pause) when CFS throttled periods exceed 5% of total periods within a 1-second window. - Services MUST emit OpenTelemetry operational telemetry capturing profiler degradation events (`profiler.adaptive_backoff.active`, `profiler.samples_dropped_throttling`). - Components NEVER execute deep DWARF or frame-pointer unwinding via BPF helpers inside interrupt context if the target cgroup pressure stall information (PSI) indicates severe CPU pressure (`some avg10 > 25`). ## Bad vs Good Implementations ### Adaptive CFS Quota Aware Profiler Configuration Manager #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The profiler blindly maintains a high sampling rate (99Hz) without checking CFS quota metrics. If the target application enters CPU contention or runs out of CFS quota, the unwinding overhead in kernel space consumes quota allocation and exacerbates application degradation. ```typescript import { openSync, readFileSync } from 'node:fs'; interface ProfilerConfig { targetPid: number; samplingHz: number; } export class BpfProfilerManager { private samplingHz: number = 99; constructor(private readonly cgroupPath: string) {} // ANTI-PATTERN: Ignoring CFS throttling state and maintaining fixed high-frequency // sampling regardless of CPU starvation inside the target container cgroup. public reconcileProfiler(): ProfilerConfig { return { targetPid: 1, samplingHz: this.samplingHz, }; } } ``` #### Production Standard (Correct) > **Standard**: Reads the cgroup v2 cpu.stat delta. When the ratio of throttled periods exceeds 5%, the manager dynamically drops the unwinding rate to 5Hz to yield execution time back to the monitored workload. ```typescript import { readFileSync } from 'node:fs'; interface CgroupCpuStat { nrPeriods: number; nrThrottled: number; throttledUsec: bigint; } interface AdaptiveProfilerConfig { samplingHz: number; isSuppressed: boolean; throttleRatio: number; } export class AdaptiveBpfProfilerManager { private static readonly BASE_SAMPLING_HZ = 99; private static readonly MIN_SAMPLING_HZ = 5; private static readonly THROTTLE_THRESHOLD_RATIO = 0.05; // 5% period throttling private lastPeriods: number = 0; private lastThrottled: number = 0; constructor(private readonly cgroupPath: string) {} private readCgroupCpuStat(): CgroupCpuStat { const content = readFileSync(`${this.cgroupPath}/cpu.stat`, 'utf-8'); const lines = content.split('\n'); let nrPeriods = 0; let nrThrottled = 0; let throttledUsec = 0n; for (const line of lines) { const [key, val] = line.split(' '); if (key === 'nr_periods') nrPeriods = Number(val); if (key === 'nr_throttled') nrThrottled = Number(val); if (key === 'throttled_usec') throttledUsec = BigInt(val); } return { nrPeriods, nrThrottled, throttledUsec }; } public reconcileAdaptiveRate(): AdaptiveProfilerConfig { const current = this.readCgroupCpuStat(); const deltaPeriods = current.nrPeriods - this.lastPeriods; const deltaThrottled = current.nrThrottled - this.lastThrottled; this.lastPeriods = current.nrPeriods; this.lastThrottled = current.nrThrottled; const throttleRatio = deltaPeriods > 0 ? deltaThrottled / deltaPeriods : 0; if (throttleRatio >= AdaptiveBpfProfilerManager.THROTTLE_THRESHOLD_RATIO) { return { samplingHz: AdaptiveBpfProfilerManager.MIN_SAMPLING_HZ, isSuppressed: true, throttleRatio, }; } return { samplingHz: AdaptiveBpfProfilerManager.BASE_SAMPLING_HZ, isSuppressed: false, throttleRatio, }; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiler Adaptive CFS Quota Throttling Guardrail

OpenTelemetry eBPF Continuous Profiler Adaptive CFS Quota Throttling Guardrail defines an industrial-grade behavioral standard for devops engineering. Continuous eBPF stack-unwinding profiles introduce measurable kernel CPU overhead under high context-switch frequencies. Profiling agents must actively monitor target cgroup cpu.stat metrics and dynamically back off sampling frequency or pause unwind iterations when target cgroups exceed configurable CFS throttling thresholds, preventing profiling-induced cascade latency. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static high-frequency eBPF sampling causing kernel-level CFS starvation and cascade pod throttling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static high-frequency eBPF sampling causing kernel-level CFS starvation and cascade pod throttling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS poll target cgroup v2 cpu.stat (`nr_throttled` and `throttled_usec`) before configuring or updating BPF perf-event sampling frequency.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.