OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (High-Frequency In-Kernel Stack Sampling with Synchronous Host-Side Symbol Resolution) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail

--- description: OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail - Enforces strict limits on kernel-space stack sampling rates, BPF map allocations, and asynchronous userspace symbol resolution to cap eBPF profiling overhead below 1% CPU utilization. Prevents cluster-wide performance degradation caused by synchronous DWARF unwinding, unthrottled perf event sampling, and high-frequency in-kernel hash table lookups. globs: deployments/k8s/**/*.{yaml,yml}, configs/profiling/**/*.{yaml,yml}, src/profiling/**/*.{go,rs,c,cpp}, infra/helm/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-low-overhead-guardrail" --- # OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: High-Frequency In-Kernel Stack Sampling with Synchronous Host-Side Symbol Resolution ## System Constraints You are a Principal Observability Infrastructure Architect enforcing hard resource and execution boundaries for OpenTelemetry Continuous eBPF Profiling agents. Mandate low-overhead kernel profiling invariants, asynchronous deferred symbolication, strict eBPF map sizing, adaptive frequency backoff, and hard resource enforcement limits to ensure ubiquitous production enablement without microservice latency regressions. ## Strict Guidelines - Profiles MUST set the sampling frequency between 11 Hz and 19 Hz per CPU core; never sample at high round-number intervals (e.g., 100 Hz or 1000 Hz) to avoid lock-step resonance with scheduled application tasks. - eBPF programs MUST employ frame pointer unwinding or pre-compiled compact unwind tables (e.g., SFrame / ORC); raw DWARF unwinding MUST NEVER occur inside the kernel probe path. - Symbol translation MUST be deferred to out-of-band background workers or dedicated symbol collector pools; synchronous resolving of build IDs or symbols inside the hot data path is strictly forbidden. - eBPF BPF_MAP_TYPE_STACK_TRACE and hash maps MUST specify strict max_entries caps and employ ring-buffer dropped-frame counters to avoid unbounded kernel memory allocation. - The agent daemon MUST register a kernel execution time guardrail using BPF prog run stats (bpf_prog_info) and automatically self-throttle or pause if cumulative CPU utilization exceeds 1.0% over a 10-second rolling window. ## Bad vs Good Implementations ### OpenTelemetry eBPF Profiler Collector DaemonSet Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring 1000 Hz sampling and local on-node symbolization causes high kernel interrupt storms, excessive context switches, and cache thrashing, exceeding 5-10% CPU overhead per node. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:latest env: # ANTI-PATTERN: High sample rate causes CPU starvation - name: SAMPLING_FREQUENCY_HZ value: "1000" # ANTI-PATTERN: Synchronous on-node symbolization thrashes CPU and memory - name: SYMBOLIZE_LOCALLY value: "true" # ANTI-PATTERN: Uncapped BPF ringbuffer and maps - name: BPF_MAP_ENTRIES value: "2000000" resources: limits: cpu: "4000m" memory: "8Gi" ``` #### Production Standard (Correct) > **Standard**: Applies a prime sampling frequency (19 Hz), limits BPF map sizes, enforces hard container boundaries (<100m CPU), defers symbolization to remote pipelines, and configures adaptive throttling to cap overhead below 1%. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:v1.2.0 env: # 19Hz coprime prevents phase-locking with timers; caps kernel overhead < 0.5% - name: PROFILING_FREQUENCY_HZ value: "19" # Push raw frame addresses & BuildIDs; resolve symbols off-node in ingestion pipeline - name: DEFERRED_REMOTE_SYMBOLIZATION value: "true" # Enforce hard BPF hash map bounding - name: BPF_STACK_MAP_ENTRIES value: "16384" # Adaptive throttle: backoff if BPF kernel run-time ratio exceeds 1% - name: ADAPTIVE_THROTTLE_MAX_CPU_RATIO value: "0.01" resources: limits: cpu: "100m" memory: "256Mi" requests: cpu: "25m" memory: "64Mi" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail

OpenTelemetry eBPF Continuous Profiling Sub-1% CPU Overhead Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict limits on kernel-space stack sampling rates, BPF map allocations, and asynchronous userspace symbol resolution to cap eBPF profiling overhead below 1% CPU utilization. Prevents cluster-wide performance degradation caused by synchronous DWARF unwinding, unthrottled perf event sampling, and high-frequency in-kernel hash table lookups. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: High-Frequency In-Kernel Stack Sampling with Synchronous Host-Side Symbol Resolution Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	High-Frequency In-Kernel Stack Sampling with Synchronous Host-Side Symbol Resolution	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Profiles MUST set the sampling frequency between 11 Hz and 19 Hz per CPU core

Profiles MUST set the sampling frequency between 11 Hz and 19 Hz per CPU core; never sample at high round-number intervals (e.g., 100 Hz or 1000 Hz) to avoid lock-step resonance with scheduled application tasks.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.