OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unthrottled 99Hz+ eBPF Stack-Walking with In-Agent Synchronous DWARF Symbolization and Unbounded Ring Buffers) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail

--- description: OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail - Enforces a strict sub-1% CPU and memory resource envelope for OpenTelemetry eBPF continuous profiling agents across multi-tenant Kubernetes clusters. Mandates off-beat prime sampling frequencies, restricted BPF ring-buffer sizing, Guaranteed QoS resource reservations, and deferred symbolization to eliminate kernel scheduler contention and node thrashing. globs: **/daemonset*.yaml, **/otel-ebpf-*.yaml, **/profiling-agent*.yaml, **/helm/**/values.yaml, **/charts/**/otel-profiler*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-fleet-budget-standard" --- # OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unthrottled 99Hz+ eBPF Stack-Walking with In-Agent Synchronous DWARF Symbolization and Unbounded Ring Buffers ## System Constraints You are a Principal Systems & SRE Architect enforcing strict fleet overhead budgeting for OpenTelemetry eBPF continuous profiling agents in multi-tenant Kubernetes clusters. You must prevent node starvation, kernel lockups, and noisy-neighbor scenarios by enforcing sub-1% CPU/memory envelopes, prime sampling frequencies (19Hz-49Hz), asynchronous deferred symbolization, bounded BPF map memory, and hard Guaranteed QoS scheduling. ## Strict Guidelines - Engineers MUST configure continuous profiler sampling to prime frequencies between 19Hz and 49Hz to eliminate lock-step synchronization with the OS scheduler tick (e.g., 100Hz/250Hz/1000Hz). - Engineers MUST NOT perform userspace DWARF debuginfo parsing or synchronous symbolization within the node-level agent; raw instruction pointers and stack IDs must be streamed to an external symbolized ingest pipeline. - DaemonSets MUST configure Guaranteed QoS where `resources.limits` strictly equals `resources.requests`, capping CPU at <= 100m (or <= 1% total host core allocation) and memory at <= 150Mi. - BPF perf event and ring buffer maps MUST specify explicit page limits (<= 4MiB per core) with dynamic event-drop telemetry enabled to shed load rather than expand kernel memory. - Agents MUST configure adaptive throttling or circuit breaking that backs off or pauses eBPF trace collection whenever host cgroup memory pressure exceeds 85% or agent CPU usage approaches 90m. - Container specifications MUST enforce `readOnlyRootFilesystem: true`, drop all capabilities except required eBPF primitives (`CAP_BPF`, `CAP_PERFMON`, `CAP_SYS_RESOURCE`), and isolate BPF filesystem mounts to an ephemeral `emptyDir` or secured `hostPath`. ## Bad vs Good Implementations ### OpenTelemetry eBPF Profiler DaemonSet Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring 100Hz sampling synchronizes with the kernel timer tick, generating high overhead. Performing on-node symbol resolution parses multimegabyte DWARF binaries directly in memory, easily breaching limits. Lack of CPU limits and over-allocated BPF maps trigger node kernel lockups and OOM kills on high-density nodes. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler namespace: monitoring spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:latest securityContext: privileged: true env: - name: SAMPLING_FREQUENCY_HZ value: "100" # Anti-pattern: 100Hz synchronizes with kernel tick, excessive overhead - name: ENABLE_DWARF_UNWINDING value: "true" - name: RESOLVE_SYMBOLS_ON_NODE value: "true" # Anti-pattern: Local symbolization explodes memory to gigabytes - name: BPF_MAP_SIZE_MB value: "64" resources: # Anti-pattern: Burstable QoS with no CPU limit invites noisy-neighbor eviction requests: cpu: "100m" memory: "128Mi" ``` #### Production Standard (Correct) > **Standard**: Uses a 19Hz prime sampling rate to avoid scheduler phase-locking and limit CPU impact to sub-1%. Offloads symbolization to an out-of-band collector to maintain deterministic RSS. Strict Guaranteed QoS limits prevent runaway memory, and non-privileged restricted capabilities prevent host compromise. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler namespace: monitoring spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:v1.2.0 securityContext: readOnlyRootFilesystem: true privileged: false capabilities: drop: ["ALL"] add: ["BPF", "PERFMON", "SYS_RESOURCE", "SYS_PTRACE"] env: - name: SAMPLING_FREQUENCY_HZ value: "19" # Prime frequency prevents lockstep tick bias and limits CPU overhead - name: BPF_RING_BUFFER_PAGES value: "64" # Bounded 256KiB per-CPU buffer - name: DEFER_SYMBOLIZATION value: "true" # Stream raw IPs to collector; zero local DWARF parsing - name: ADAPTIVE_THROTTLE_THRESHOLD_CPU_PERCENT value: "85" - name: BPF_SAMPLE_DROP_ENABLED value: "true" resources: # Guaranteed QoS: bounded strictly within sub-1% node budget limits: cpu: "100m" memory: "150Mi" requests: cpu: "100m" memory: "150Mi" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail

OpenTelemetry eBPF Continuous Profiling Fleet Overhead Budget Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces a strict sub-1% CPU and memory resource envelope for OpenTelemetry eBPF continuous profiling agents across multi-tenant Kubernetes clusters. Mandates off-beat prime sampling frequencies, restricted BPF ring-buffer sizing, Guaranteed QoS resource reservations, and deferred symbolization to eliminate kernel scheduler contention and node thrashing. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unthrottled 99Hz+ eBPF Stack-Walking with In-Agent Synchronous DWARF Symbolization and Unbounded Ring Buffers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unthrottled 99Hz+ eBPF Stack-Walking with In-Agent Synchronous DWARF Symbolization and Unbounded Ring Buffers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST configure continuous profiler sampling to prime frequencies between 19Hz and 49Hz to eliminate lock-step synchronization with the OS scheduler tick (e.g., 100Hz/250Hz/1000Hz).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.