OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrail

--- description: OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrail - Enforces adaptive sampling budgets and hardware thread accounting to strictly cap continuous profiling and native runtime stack unwinding overhead to under 1% total CPU allocation. Prevents telemetry-induced tail-latency spikes and CPU starvation under high concurrency by dynamically adjusting sampling frequency via a PID controller or sliding-window token bucket. globs: src/instrumentation/**/*.{ts,js,go,rs}, src/profiling/**/*.{ts,js,go,rs}, config/telemetry/**/*.{yaml,json}, deployments/**/*.{yaml,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-profiling-cpu-budget-guardrail" --- # OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Continuous Profiling Thread Execution Without Dynamic CPU Budget Throttling ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrails. Continuous profiling agents and unwinding engines MUST NEVER consume more than 1% of total node/container CPU time. You must reject any static high-frequency profiling configurations that fail to self-meter CPU consumption or lack adaptive backoff mechanisms during burst workloads. ## Strict Guidelines - Engineers MUST ALWAYS implement adaptive sampling algorithms (e.g., PID loop or token bucket) that measure self-CPU consumption and scale down sampling frequency when profiler CPU usage exceeds 1.0% of process total. - Engineers MUST NOT deploy continuous profiling with fixed sample frequencies (>19Hz) without hardware thread-level CPU accounting or an automatic fallback killswitch. - Profilers SHOULD monitor thread execution time using high-resolution POSIX thread clocks (CLOCK_THREAD_CPUTIME_ID) or native runtime counters (e.g., process.cpuUsage) within a rolling window under 5 seconds. - Services MUST gracefully degrade unwinding fidelity (e.g., skip frame symbolization, reduce max stack depth, or clamp collection from 99Hz down to 10Hz) before disabling collection entirely. - Components NEVER execute blocking symbol resolution, disk I/O, or remote OTLP export directly inside signal handlers or interrupted thread stack-walking routines. ## Bad vs Good Implementations ### Continuous Profiler Adapter with Adaptive CPU Budget Regulation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring a static high-frequency sampling rate (100Hz) with maximum stack depth without monitoring the profiler's own CPU consumption violates the 1% CPU overhead bound under high concurrency or deep stack execution. ```typescript import { ProfilingAgent } from '@opentelemetry/profiling-node'; // ANTI-PATTERN: Fixed 99Hz continuous profiling without self-monitoring. // Deep stack recursion and thread saturation will exceed 1% total CPU, // causing latency spikes and process throttling. export function startContinuousProfiling(): void { const profiler = new ProfilingAgent({ sampleIntervalMicroseconds: 10_000, // Static 100Hz maxStackDepth: 256, exportIntervalMs: 5000 }); profiler.start(); } ``` #### Production Standard (Correct) > **Standard**: Implements an adaptive closed-loop budget controller. It continuously evaluates the telemetry CPU fraction against the target ceiling (1%), scaling down the sampling frequency dynamically during high load and safely restoring it during normal conditions. ```typescript import { ProfilingAgent } from '@opentelemetry/profiling-node'; interface ProfilingBudgetConfig { maxCpuFraction: number; // e.g., 0.01 for 1% budget checkIntervalMs: number; minFrequencyHz: number; maxFrequencyHz: number; } export class AdaptiveProfilingSupervisor { private profiler: ProfilingAgent; private currentFrequencyHz: number; private lastCpuUsage: NodeJS.CpuUsage; private lastTimestamp: number; private timer?: NodeJS.Timeout; constructor(private readonly config: ProfilingBudgetConfig = { maxCpuFraction: 0.01, checkIntervalMs: 5000, minFrequencyHz: 10, maxFrequencyHz: 99 }) { this.currentFrequencyHz = config.maxFrequencyHz; this.lastCpuUsage = process.cpuUsage(); this.lastTimestamp = performance.now(); this.profiler = new ProfilingAgent({ sampleIntervalMicroseconds: Math.floor(1_000_000 / this.currentFrequencyHz), maxStackDepth: 64, exportIntervalMs: 15_000 }); } public start(): void { this.profiler.start(); this.timer = setInterval(() => this.reconcileCpuBudget(), this.config.checkIntervalMs); this.timer.unref(); } private reconcileCpuBudget(): void { const currentTimestamp = performance.now(); const wallTimeElapsedUs = (currentTimestamp - this.lastTimestamp) * 1000; const diffCpu = process.cpuUsage(this.lastCpuUsage); this.lastCpuUsage = process.cpuUsage(); this.lastTimestamp = currentTimestamp; const totalCpuTimeUs = diffCpu.user + diffCpu.system; // Overhead calculation normalized across active threads const cpuFraction = totalCpuTimeUs / (wallTimeElapsedUs * (process.env.CPU_COUNT ? Number(process.env.CPU_COUNT) : 1)); if (cpuFraction > this.config.maxCpuFraction) { // Overhead exceeded: aggressively halve sampling frequency down to floor this.currentFrequencyHz = Math.max(this.config.minFrequencyHz, Math.floor(this.currentFrequencyHz * 0.5)); this.profiler.setSampleIntervalMicroseconds(Math.floor(1_000_000 / this.currentFrequencyHz)); } else if (this.currentFrequencyHz < this.config.maxFrequencyHz) { // Safe overhead: incrementally ramp frequency back up this.currentFrequencyHz = Math.min(this.config.maxFrequencyHz, this.currentFrequencyHz + 5); this.profiler.setSampleIntervalMicroseconds(Math.floor(1_000_000 / this.currentFrequencyHz)); } } public stop(): void { if (this.timer) clearInterval(this.timer); this.profiler.stop(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrail

OpenTelemetry Continuous Profiling Thread CPU-Time Allocation Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces adaptive sampling budgets and hardware thread accounting to strictly cap continuous profiling and native runtime stack unwinding overhead to under 1% total CPU allocation. Prevents telemetry-induced tail-latency spikes and CPU starvation under high concurrency by dynamically adjusting sampling frequency via a PID controller or sliding-window token bucket. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Continuous Profiling Thread Execution Without Dynamic CPU Budget Throttling Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Continuous Profiling Thread Execution Without Dynamic CPU Budget Throttling	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement adaptive sampling algorithms (e.g., PID loop or token bucket) that measure self-CPU consumption and scale down sampling frequency when profiler CPU usage exceeds 1.0% of process total.

Engineers MUST ALWAYS implement adaptive sampling algorithms (e.g., PID loop or token bucket) that measure self-CPU consumption and scale down sampling frequency when profiler CPU usage exceeds 1.0% of process total.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.