Adaptive Sampling Budget and Dynamic Throttling for OpenTelemetry Continuous Profiling

--- description: Adaptive Sampling Budget and Dynamic Throttling for OpenTelemetry Continuous Profiling - Continuous stack profiling via OTLP/Pprof introduces measurable tail latency and collector ingestion bottlenecks if sampled at static rates during traffic surges. This policy enforces client-side adaptive sampling budgets that dynamically down-sample or suspend call-stack captures when CPU or memory headroom thresholds are breached. Production services must guarantee profiling overhead remains under 2% CPU and zero collector OTLP buffer dropouts. globs: **/telemetry/**/*.{ts,js,py,go}, **/profiler/**/*.{ts,js,py,go}, **/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-profiling-adaptive-sampling-budget" --- # Adaptive Sampling Budget and Dynamic Throttling for OpenTelemetry Continuous Profiling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Static high-frequency profiling during CPU saturation causing self-inflicted tail latency cascades and collector buffer eviction ## System Constraints You are a Principal Software Architect enforcing production-grade OpenTelemetry Continuous Profiling guardrails. Mandate that all continuous profiling integrations (pprof, Pyroscope OTLP, or native OTel Profiling SDKs) employ dynamic, feedback-driven sampling controls. Profiling MUST automatically shed collection frequencies and enforce byte-rate budgets under high host CPU utilization, garbage collection pressure, or collector pushback. ## Strict Guidelines - Engineers MUST ALWAYS enforce dynamic backoff for continuous profiling (e.g., dropping from 99Hz to 19Hz or 0Hz) when host or process CPU utilization exceeds 75% or heap usage exceeds 80%. - Engineers MUST NOT allow unthrottled, fixed-rate pprof or OTLP call-stack sampling in production environments. - Services MUST establish a strict maximum outbound bandwidth budget for profile exports (e.g., max 512KB/s per instance) with client-side frame dropping rather than memory buffering when saturated. - Collector pipelines MUST configure memory_limiter and batch processors upstream of continuous profiling receivers to enforce backpressure without crashing the collector daemon. - Profilers MUST report internal telemetry metrics (profiler_overhead_cpu_seconds_total, profiler_dropped_profiles_total) to monitor observability overhead in real time. - Components NEVER hold unbounded in-memory ring buffers of serialized profile chunks during collector OTLP retry loops. ## Bad vs Good Implementations ### Continuous Profiling Adaptive Sampling and Budget Controller #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: A static sample rate of 99Hz adds substantial overhead during heavy GC or high CPU utilization, compounding p99 latency spikes. The unbounded queue risks out-of-memory crashes if the OTLP collector experiences ingestion lag or throttling. ```typescript import { ProfilingAgent } from '@opentelemetry/experimental-profiler'; // ANTI-PATTERN: Fixed 99Hz sampling with unbounded export queues and no resource guardrails export function initializeProfiler(): void { const profiler = new ProfilingAgent({ serviceName: 'order-checkout', sampleRateHz: 99, // Static 99Hz unwinds stacks regardless of CPU saturation exportIntervalMs: 5000, endpoint: 'http://otel-collector.internal:4318/v1/profiles', exportTimeoutMs: 30000, maxQueueSize: 100000 // Dangerously large queue causes OOM under collector lag }); profiler.start(); } ``` #### Production Standard (Correct) > **Standard**: The adaptive controller proactively monitors process health, shedding unwinding rates from nominal to throttled or zero when CPU/heap thresholds are exceeded. It enforces strict queue bounds to prevent memory bloat and protects both the application and the OTLP collector. ```typescript import { ProfilingAgent } from '@opentelemetry/experimental-profiler'; import * as os from 'node:os'; interface ProfilingBudgetConfig { serviceName: string; endpoint: string; maxCpuThreshold: number; // e.g., 0.75 for 75% maxHeapThreshold: number; // e.g., 0.80 for 80% nominalSampleRateHz: number; throttledSampleRateHz: number; maxExportBandwidthBytesPerSec: number; } export class AdaptiveProfilerController { private profiler: ProfilingAgent; private monitorTimer: NodeJS.Timeout | null = null; private bytesExportedInWindow = 0; private windowStartMs = Date.now(); constructor(private readonly config: ProfilingBudgetConfig) { this.profiler = new ProfilingAgent({ serviceName: config.serviceName, sampleRateHz: config.nominalSampleRateHz, exportIntervalMs: 10000, endpoint: config.endpoint, maxQueueSize: 256 // Constrained memory footprint (circuit-breaker) }); } public start(): void { this.profiler.start(); this.monitorTimer = setInterval(() => this.rebalanceSamplingBudget(), 5000); } private rebalanceSamplingBudget(): void { const cpuUsage = this.getProcessCpuUsageFraction(); const memUsage = process.memoryUsage(); const heapFraction = memUsage.heapUsed / memUsage.heapTotal; // Circuit breaker: Suspend or throttle unwinding under resource exhaustion if (cpuUsage > this.config.maxCpuThreshold || heapFraction > this.config.maxHeapThreshold) { this.profiler.setSampleRateHz(0); // Complete shed under critical saturation } else if (cpuUsage > (this.config.maxCpuThreshold * 0.8)) { this.profiler.setSampleRateHz(this.config.throttledSampleRateHz); } else { this.profiler.setSampleRateHz(this.config.nominalSampleRateHz); } } private getProcessCpuUsageFraction(): number { const cpus = os.cpus().length; const loadAvg = os.loadavg()[0]; return cpus > 0 ? loadAvg / cpus : 0; } public stop(): void { if (this.monitorTimer) clearInterval(this.monitorTimer); this.profiler.stop(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Adaptive Sampling Budget and Dynamic Throttling for OpenTelemetry Continuous Profiling

Adaptive Sampling Budget and Dynamic Throttling for OpenTelemetry Continuous Profiling defines an industrial-grade behavioral standard for devops engineering. Continuous stack profiling via OTLP/Pprof introduces measurable tail latency and collector ingestion bottlenecks if sampled at static rates during traffic surges. This policy enforces client-side adaptive sampling budgets that dynamically down-sample or suspend call-stack captures when CPU or memory headroom thresholds are breached. Production services must guarantee profiling overhead remains under 2% CPU and zero collector OTLP buffer dropouts. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static high-frequency profiling during CPU saturation causing self-inflicted tail latency cascades and collector buffer eviction Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static high-frequency profiling during CPU saturation causing self-inflicted tail latency cascades and collector buffer eviction	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce dynamic backoff for continuous profiling (e.g., dropping from 99Hz to 19Hz or 0Hz) when host or process CPU utilization exceeds 75% or heap usage exceeds 80%.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.