OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Throttling Standard

--- description: OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Throttling Standard - Enforces dynamic, feedback-loop sample-rate decay and kernel unwinding bypasses when OpenTelemetry continuous profiling agent host CPU overhead breaches 1.5%. Mitigates hypervisor steal and Completely Fair Scheduler (CFS) quota throttling on production Kubernetes nodes under heavy thread contention. globs: deploy/**/*.yaml, k8s/**/*.yaml, pkg/profiler/**/*.go, cmd/otel-profiler/**/*.go, bpf/**/*.c alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-overhead-throttling-standard" --- # OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Throttling Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static high-frequency eBPF stack unwinding triggering CFS quota starvation and hypervisor-level CPU throttling ## System Constraints You are a Principal Systems and Observability Architect enforcing OpenTelemetry eBPF profiling safety. Continuous profiling agents running on Linux worker nodes must never destabilize co-located workloads. You must strictly mandate dynamic, in-kernel unwinding sample-rate decay, automated circuit breaking at a 1.5% CPU overhead threshold, and resilient cgroup self-monitoring. ## Strict Guidelines - Services MUST ALWAYS calculate agent CPU utilization using high-resolution delta reads from `/sys/fs/cgroup/cpu.stat` (cgroups v2) or `/sys/fs/cgroup/cpu/cpuacct.usage` (cgroups v1) over a sliding 1-second window. - Engineers MUST configure an automated sample-rate decay algorithm that throttles the unwinding rate from default target (e.g., 19Hz or 99Hz) down to a minimum floor (e.g., 1Hz to 5Hz) when agent CPU exceeds 1.5% of total node allocatable capacity. - Code MUST synchronize the decayed sampling interval directly into an eBPF config map (e.g., `BPF_MAP_TYPE_ARRAY`) to drop stack collection within the in-kernel perf/tracepoint handler before triggering expensive DWARF or frame-pointer unwinding. - Agents MUST trigger an emergency bypass circuit-breaker that sets the sample rate to 0Hz for 30 seconds if CPU consumption breaches 2.5% or if hypervisor steal (`cpu.stat.nr_throttled`) spikes over 5% in a single evaluation window. - Engineers MUST NOT run static sampling rates without runtime self-overhead feedback loops in production DaemonSets. - Components NEVER allocate unbounded eBPF ring buffers (`perf_event_output` or `bpf_ringbuf`) without drop counters reported via OpenTelemetry native telemetry metrics. ## Bad vs Good Implementations ### Unthrottled Static Sampling vs. Dynamic In-Kernel Decay Controller #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The static profiler runs at a fixed 99Hz frequency regardless of node CPU pressure. It does not monitor cgroup consumption, lacks feedback loops to update kernel maps, and risks node-level CFS throttling when applications spawn many active threads. ```go package profiler import "time" type StaticProfiler struct { sampleFreqHz int } // BAD: Continuously unwinds stacks at a fixed 99Hz without observing agent CPU utilization. // Under heavy multithreaded load, DWARF unwinding in eBPF saturates the node's CPU quota, // driving the agent into CFS throttle and causing cascading latency on application pods. func NewProfiler() *StaticProfiler { return &StaticProfiler{sampleFreqHz: 99} } func (p *StaticProfiler) Run(stopCh <-chan struct{}) { // Fixed frequency push to BPF map; completely blind to node load and hypervisor steal updateBpfSampleRate(p.sampleFreqHz) <-stopCh } func updateBpfSampleRate(rate int) {} ``` #### Production Standard (Correct) > **Standard**: Calculates agent cgroup CPU usage every second relative to host capacity. If overhead crosses the 1.5% limit, it calculates a decay factor and syncs the lower frequency directly into the eBPF control array map, dropping unwinding events in-kernel before frame parsing. ```go package profiler import ( "context" "math" "time" ) const ( MaxPermissibleCPURatio = 0.015 // 1.5% CPU threshold EmergencyCeilingRatio = 0.025 // 2.5% Emergency breaker BaselineSampleRateHz = 19.0 FloorSampleRateHz = 1.0 ) type DynamicThrottler struct { cgroupMonitor CGroupUsageReader bpfConfigMap BPFArrayMap } // GOOD: Reads delta cgroup CPU usage every second and dynamically decays // in-kernel stack unwinding rate via an eBPF control map. func (t *DynamicThrottler) RunThrottlingLoop(ctx context.Context, numCores float64) { ticker := time.NewTicker(1 * time.Second) defer ticker.Stop() currentHz := BaselineSampleRateHz for { select { case <-ctx.Done(): return case <-ticker.C: usedDeltaSec, err := t.cgroupMonitor.DeltaCPUSeconds() if err != nil { continue } // CPU utilization normalized against total available CPU capacity overheadRatio := usedDeltaSec / numCores switch { case overheadRatio >= EmergencyCeilingRatio: // Emergency bypass: mute kernel trace points completely currentHz = 0.0 case overheadRatio > MaxPermissibleCPURatio: // Exponential decay when overhead exceeds 1.5% target decayFactor := MaxPermissibleCPURatio / overheadRatio currentHz = math.Max(FloorSampleRateHz, currentHz*decayFactor) default: // Gradual recovery toward baseline when within budget currentHz = math.Min(BaselineSampleRateHz, currentHz+1.0) } // Atomic write to BPF_MAP_TYPE_ARRAY checked in kernel-space tracepoint _ = t.bpfConfigMap.Update(0, uint32(math.Round(currentHz))) } } } type CGroupUsageReader interface { DeltaCPUSeconds() (float64, error) } type BPFArrayMap interface { Update(key uint32, value uint32) error } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Throttling Standard

OpenTelemetry eBPF Continuous Profiling Dynamic Overhead Throttling Standard defines an industrial-grade behavioral standard for devops engineering. Enforces dynamic, feedback-loop sample-rate decay and kernel unwinding bypasses when OpenTelemetry continuous profiling agent host CPU overhead breaches 1.5%. Mitigates hypervisor steal and Completely Fair Scheduler (CFS) quota throttling on production Kubernetes nodes under heavy thread contention. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static high-frequency eBPF stack unwinding triggering CFS quota starvation and hypervisor-level CPU throttling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static high-frequency eBPF stack unwinding triggering CFS quota starvation and hypervisor-level CPU throttling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST ALWAYS calculate agent CPU utilization using high-resolution delta reads from `/sys/fs/cgroup/cpu.stat` (cgroups v2) or `/sys/fs/cgroup/cpu/cpuacct.usage` (cgroups v1) over a sliding 1-second window.

Services MUST ALWAYS calculate agent CPU utilization using high-resolution delta reads from `/sys/fs/cgroup/cpu.stat` (cgroups v2) or `/sys/fs/cgroup/cpu/cpuacct.usage` (cgroups v1) over a sliding 1-second window.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.