OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded eBPF Stack Walk Collection During Thread Contention Cascades) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling

--- description: OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling - Mandates an automated runtime supervisor and eBPF sampling governor for OpenTelemetry continuous profiling agents in high-density Kubernetes environments. Enforces adaptive stack-unwinding frequency scaling and a failsafe circuit breaker to guarantee profiler CPU consumption stays strictly below 1% and RSS memory below 50MiB during severe thread contention and process-fork storms. Prevents kernel BPF ring-buffer lockups, cgroup throttling cascades, and noisy-neighbor node eviction. globs: deployments/**/*.yaml, k8s/**/*.yaml, configs/**/*.yaml, cmd/profiler/**/*.go, pkg/profiler/**/*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiler-overhead-circuit-breaker" --- # OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded eBPF Stack Walk Collection During Thread Contention Cascades ## System Constraints You are a Principal Systems and Infrastructure Architect enforcing strict resource overhead boundaries for continuous profiling agents. You must mandate an adaptive sampling governor and automatic circuit-breaking mechanism across all OpenTelemetry eBPF continuous profiling agents deployed into Kubernetes environments. Ensure that profiler agents automatically scale down their unwind frequency or suspend eBPF probe attachments whenever agent cgroup CPU usage approaches 1% or kernel ring-buffer drop rates indicate thread contention cascades. ## Strict Guidelines - Engineers MUST ALWAYS implement an active telemetry self-monitoring governor loop that samples the profiler's own cgroup `cpu.stat` and `memory.current` every 1 to 5 seconds. - Engineers MUST NOT configure static 99Hz or 100Hz eBPF sampling rates without enabling adaptive backoff down to 19Hz or 9Hz under thread contention or elevated host CPU load. - Profilers MUST immediately trip a circuit breaker and detach or suspend non-critical BPF perf events when agent CPU utilization exceeds 0.8% over a 10-second rolling window. - Profilers MUST drop sample frames in user space or BPF map space using lossy ring-buffer flags (e.g., `BPF_RB_NO_WAKEUP`) rather than blocking or spinning on full perf event buffers. - Deployments MUST configure strict Linux cgroup resource limits (`resources.limits.cpu: 100m`, `resources.limits.memory: 128Mi`) alongside explicit Kubernetes burst profiles to prevent unbounded memory allocation during stack-unwind table generation. - Services MUST emit an OpenTelemetry Gauge metric `profiler.overhead.cpu_ratio` and trip alerts when the circuit breaker enters the `Open` (suspended) state. ## Bad vs Good Implementations ### Go: eBPF Profiler Adaptive Sampling Governor and Circuit Breaker Loop #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The profiler attaches eBPF probes at an unyielding 99Hz across all CPUs with no self-overhead governor. During thread contention spikes or fork storms, unwinding stack traces in kernel space consumes substantial CPU, easily exceeding 1% cgroup limits and causing kernel perf buffer overflow or pod eviction. ```go // BAD: Fixed sampling rate with no self-monitoring or circuit breaking. // Under heavy context-switching or thread contention storms, BPF stack // unwinding saturates the node's CPU and bloats memory. package profiler import ( "context" "time" ) type Profiler struct { sampleFreqHz int } func NewProfiler() *Profiler { return &Profiler{sampleFreqHz: 99} } func (p *Profiler) Run(ctx context.Context) error { // Unconditionally attach eBPF perf event at 99Hz across all online CPUs if err := attachPerfEvents(p.sampleFreqHz); err != nil { return err } <-ctx.Done() return nil } func attachPerfEvents(hz int) error { return nil } ``` #### Production Standard (Correct) > **Standard**: Implements a dynamic self-monitoring feedback loop that evaluates cgroup CPU utilization every 2 seconds. If CPU usage reaches 75% of the 1% boundary (0.008 CPU ratio), it backs off sampling frequency to 19Hz. If it exceeds 0.8% total ratio, it trips the circuit breaker and detaches BPF programs, guaranteeing that profiler overhead never breaches the strict 1% threshold. ```go // GOOD: Production-grade overhead governor with an adaptive circuit breaker. // Dynamically scales sampling rate and suspends probes if self-overhead exceeds 0.8% CPU. package profiler import ( "context" "sync/atomic" "time" ) const ( MaxPermissibleCPURatio = 0.008 // 0.8% CPU threshold for circuit breaker BaselineSampleHz = 99 ThrottledSampleHz = 19 ) type CircuitState int32 const ( StateClosed CircuitState = iota // Normal operation StateDegraded // Throttled frequency StateTripped // Probes disabled ) type Governor struct { state atomic.Int32 currentHz atomic.Int32 cgroupUsage CgroupMetricsReader } type CgroupMetricsReader interface { GetCPUUsageRatio(window time.Duration) (float64, error) } func NewGovernor(reader CgroupMetricsReader) *Governor { g := &Governor{cgroupUsage: reader} g.state.Store(int32(StateClosed)) g.currentHz.Store(BaselineSampleHz) return g } func (g *Governor) MonitorLoop(ctx context.Context, updateBpfRate func(int) error, detachBpf func() error) { ticker := time.NewTicker(2 * time.Second) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: usage, err := g.cgroupUsage.GetCPUUsageRatio(2 * time.Second) if err != nil { continue } switch { case usage >= MaxPermissibleCPURatio: // Trip circuit breaker: detach probes to prevent node disruption if g.state.CompareAndSwap(int32(StateClosed), int32(StateTripped)) || g.state.CompareAndSwap(int32(StateDegraded), int32(StateTripped)) { _ = detachBpf() } case usage >= (MaxPermissibleCPURatio * 0.75): // Degradation zone: back off sampling frequency if g.state.CompareAndSwap(int32(StateClosed), int32(StateDegraded)) { g.currentHz.Store(ThrottledSampleHz) _ = updateBpfRate(ThrottledSampleHz) } case usage < (MaxPermissibleCPURatio * 0.40): // Safe zone: recover to baseline if g.state.Swap(int32(StateClosed)) != int32(StateClosed) { g.currentHz.Store(BaselineSampleHz) _ = updateBpfRate(BaselineSampleHz) } } } } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling

OTel eBPF Profiler Dynamic Overhead Circuit Breaker & Adaptive Rate Throttling defines an industrial-grade behavioral standard for devops engineering. Mandates an automated runtime supervisor and eBPF sampling governor for OpenTelemetry continuous profiling agents in high-density Kubernetes environments. Enforces adaptive stack-unwinding frequency scaling and a failsafe circuit breaker to guarantee profiler CPU consumption stays strictly below 1% and RSS memory below 50MiB during severe thread contention and process-fork storms. Prevents kernel BPF ring-buffer lockups, cgroup throttling cascades, and noisy-neighbor node eviction. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded eBPF Stack Walk Collection During Thread Contention Cascades Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded eBPF Stack Walk Collection During Thread Contention Cascades	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement an active telemetry self-monitoring governor loop that samples the profiler's own cgroup `cpu.stat` and `memory.current` every 1 to 5 seconds.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.