OpenTelemetry Continuous Profiling Dynamic Adaptive Sampling & eBPF Budget Invariant

--- description: OpenTelemetry Continuous Profiling Dynamic Adaptive Sampling & eBPF Budget Invariant - Continuous eBPF and runtime pprof CPU profilers must enforce dynamic adaptive sampling frequency scaling and hash-ring profile sharding based on real-time tail-latency (p99/p999) degradation and kernel overhead metrics. Unregulated static kernel-level sampling rates (e.g., rigid 99Hz timers across all cores) cause catastrophic cascading CPU cache eviction and ring-buffer drop events during tail-latency saturation events. globs: **/profiling/**/*.go, **/telemetry/**/*.go, **/profiler/**/*.rs, **/ebpf/**/*.c, **/otel-collector-config*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-profiling-dynamic-adaptive-sampling-rate-sharding-standard" --- # OpenTelemetry Continuous Profiling Dynamic Adaptive Sampling & eBPF Budget Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static Unsharded eBPF/Pprof Profiling Rates Inducing Observer Overhead Cascades Under Load ## System Constraints You are a Principal Systems Engineer and Site Reliability Architect enforcing strict invariants on OpenTelemetry continuous profiling agents (eBPF and runtime pprof). You must prevent profiling observer effect spirals during traffic spikes and p99/p999 latency regressions. Enforce dynamic sampling degradation, sharded target assignment via consistent hashing rings, and strict CPU/ring-buffer budget circuit-breaking. ## Strict Guidelines - Engineers MUST ALWAYS implement PID-controller or stepped adaptive frequency scaling that reduces eBPF/pprof sampling rates (e.g., stepped from 99Hz down to 19Hz or 0Hz) when node CPU utilization exceeds 80% or tail latency breaches configured SLAs. - Engineers MUST NOT run static high-frequency (>=99Hz) continuous kernel-space eBPF stack collection across 100% of service targets simultaneously in production clusters. - Code MUST partition high-cardinality profiling workloads across target nodes using a consistent hash ring with virtual nodes, ensuring no single daemon collector or backend pipeline experiences ring-buffer overflow (BPF_PERF_OUTPUT buffer overrun). - Services MUST integrate backpressure signaling from the OTel Collector OTLP/Profiles pipeline into the local agent, pausing or downsampling stack unwinding when memory budgets or export queues saturate. - Components NEVER drop eBPF perf events silently without exporting drop-rate metrics ('profiling.ebpf.lost_samples_total') to ensure sampling bias is observable in downstream flame graphs. ## Bad vs Good Implementations ### Adaptive Sampling Rate Regulator with Latency and CPU Throttling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The profiler runs at a rigid 99Hz sampling rate without checking node CPU pressure or tail-latency feedback. During a latency spike or high-load event, the unwinder competes with user workloads for CPU caches and registers, worsening the outage. ```go // Static profiling config that ignores host load and tail latency package profiler import "time" type StaticProfiler struct { sampleRateHz int } func NewStaticProfiler() *StaticProfiler { // Unbounded 99Hz sampling across all cores regardless of tail latency or CPU stress return &StaticProfiler{sampleRateHz: 99} } func (p *StaticProfiler) Run(stopCh <-chan struct{}) { ticker := time.NewTicker(time.Second / time.Duration(p.sampleRateHz)) defer ticker.Stop() for { select { case <-ticker.C: captureStackTrace() case <-stopCh: return } } } func captureStackTrace() {} ``` #### Production Standard (Correct) > **Standard**: Implements an active backoff mechanism that reduces profiler overhead down to safe boundaries (minRateHz) when tail-latency degraded signals or CPU threshold overruns occur, eliminating profiling-induced cascading failure. ```go // Dynamic adaptive rate regulator with backpressure and load shedding package profiler import ( "sync/atomic" "time" ) type AdaptiveProfiler struct { baseRateHz int64 minRateHz int64 currentRateHz int64 cpuThresholdPct float64 } func NewAdaptiveProfiler(baseHz, minHz int64, cpuThreshold float64) *AdaptiveProfiler { return &AdaptiveProfiler{ baseRateHz: baseHz, minRateHz: minHz, currentRateHz: baseHz, cpuThresholdPct: cpuThreshold, } } // AdjustSamplingBudget dynamically throttles sampling frequency based on host load and tail latency feedback func (p *AdaptiveProfiler) AdjustSamplingBudget(nodeCPUUsagePct float64, p99Degraded bool) int64 { if p99Degraded || nodeCPUUsagePct > p.cpuThresholdPct { // Exponential backoff to minimum safe kernel sampling floor newRate := atomic.LoadInt64(&p.currentRateHz) / 2 if newRate < p.minRateHz { newRate = p.minRateHz } atomic.StoreInt64(&p.currentRateHz, newRate) return newRate } // Gradual recovery toward baseline sampling budget current := atomic.LoadInt64(&p.currentRateHz) if current < p.baseRateHz { atomic.StoreInt64(&p.currentRateHz, current+1) } return atomic.LoadInt64(&p.currentRateHz) } func (p *AdaptiveProfiler) CurrentInterval() time.Duration { rate := atomic.LoadInt64(&p.currentRateHz) if rate <= 0 { return time.Hour // Effective circuit-breaker pause } return time.Second / time.Duration(rate) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling Dynamic Adaptive Sampling & eBPF Budget Invariant

OpenTelemetry Continuous Profiling Dynamic Adaptive Sampling & eBPF Budget Invariant defines an industrial-grade behavioral standard for devops engineering. Continuous eBPF and runtime pprof CPU profilers must enforce dynamic adaptive sampling frequency scaling and hash-ring profile sharding based on real-time tail-latency (p99/p999) degradation and kernel overhead metrics. Unregulated static kernel-level sampling rates (e.g., rigid 99Hz timers across all cores) cause catastrophic cascading CPU cache eviction and ring-buffer drop events during tail-latency saturation events. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static Unsharded eBPF/Pprof Profiling Rates Inducing Observer Overhead Cascades Under Load Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static Unsharded eBPF/Pprof Profiling Rates Inducing Observer Overhead Cascades Under Load	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement PID-controller or stepped adaptive frequency scaling that reduces eBPF/pprof sampling rates (e.g., stepped from 99Hz down to 19Hz or 0Hz) when node CPU utilization exceeds 80% or tail latency breaches configured SLAs.

Engineers MUST ALWAYS implement PID-controller or stepped adaptive frequency scaling that reduces eBPF/pprof sampling rates (e.g., stepped from 99Hz down to 19Hz or 0Hz) when node CPU utilization exceeds 80% or tail latency breaches configured SLAs.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.