OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Uncapped continuous eBPF stack collection triggering kernel ring buffer contention and CPU starvation) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement

--- description: OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement - Enforces strict CPU, memory, and telemetry ring-buffer quotas on OpenTelemetry eBPF continuous profiling agents deployed across high-density containerized nodes. Unrestricted sampling rates and unbounded userspace aggregation cause ring buffer drops, kernel scheduling latency, and CPU thrashing during traffic spikes. Telemetry infrastructure must implement hard sampling rate ceilings, memory limits, and automated backoff mechanisms to prevent kernel-level resource starvation. globs: deployments/**/*.yaml, helm/**/*.yaml, monitoring/**/*.{yaml,yml,json}, infra/**/*.tf, src/observability/**/*.{ts,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-budget-quotas" --- # OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Uncapped continuous eBPF stack collection triggering kernel ring buffer contention and CPU starvation ## System Constraints You are a Principal Observability Infrastructure Architect enforcing OpenTelemetry Continuous Profiling (eBPF) Overhead Budgeting & Quotas. Your objective is to ensure that eBPF profilers (e.g., OTel Profiling Agent / Beyla) never destabilize production kernel schedulers or starve host node workloads. Mandate rigid sampling frequency bounds (max 19Hz-49Hz), strict perf_event buffer memory locks, cgroup resource hard-caps, and proactive load-shedding circuit breakers. ## Strict Guidelines - Engineers MUST ALWAYS enforce a maximum profiling sampling frequency ceiling (e.g., between 19Hz and 49Hz) using non-harmonic prime frequencies to minimize lock-step synchronization with system clocks. - Engineers MUST configure hard cgroup limits (`resources.limits.cpu` <= 0.2 cores and `resources.limits.memory` <= 256Mi) for daemonset profiling agents on high-density production nodes. - Engineers MUST NOT run eBPF profilers with unbound `perf_event_open` page allocation (`max_map_entries` or ring buffer pages) to avoid exhausting non-swappable kernel locked memory (`RLIMIT_MEMLOCK`). - Services MUST enable dynamic backoff or adaptive rate shedding within the profiler daemon when host total CPU usage exceeds 85% capacity. - Deployment manifests MUST explicitly pin `RLIMIT_MEMLOCK` to a finite bound rather than allowing `infinity` without strict namespace-level eBPF limits. ## Bad vs Good Implementations ### OpenTelemetry eBPF Continuous Profiler Agent DaemonSet Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: A 250Hz sample rate across dense multi-core nodes generates catastrophic interrupt overhead and memory lock allocations. With unbounded collection caps and vast buffer allocations, kernel ring-buffer starvation impairs the Linux CFS scheduler and starves colocated application pods. ```yaml # INSECURE / UNBOUNDED: Causes CPU starvation and kernel memory exhaustion apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler namespace: observability spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:latest securityContext: privileged: true env: - name: SAMPLING_FREQUENCY_HZ value: "250" # Excessive sampling rate - name: SAMPLES_PER_SECOND_CAP value: "0" # Unbounded stack collection - name: PERF_BUFFER_PAGE_COUNT value: "512" # Consumes huge amount of non-swappable kernel mem ``` #### Production Standard (Correct) > **Standard**: Specifies a non-harmonic 19Hz sampling frequency with rigid cgroup limits, fine-grained capabilities instead of global privileged mode, a 1% host overhead threshold circuit breaker, and bound ring-buffer memory limits to preserve deterministic kernel scheduling. ```yaml # SECURE & RESILIENT: Hard quotas, prime frequency, and adaptive throttling apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler namespace: observability spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:0.11.0 securityContext: privileged: false capabilities: add: ["SYS_PTRACE", "PERFMON", "BPF"] resources: limits: cpu: "200m" memory: "256Mi" requests: cpu: "50m" memory: "64Mi" env: - name: SAMPLING_FREQUENCY_HZ value: "19" # Prime frequency prevents harmonic drift - name: OVERHEAD_BUDGET_CPU_PERCENT value: "1.0" # Circuit breaker: shed samples if > 1% host CPU - name: MAX_COLLECTED_SAMPLES_PER_SEC value: "1000" - name: PERF_BUFFER_PAGE_COUNT value: "64" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement

OpenTelemetry Continuous eBPF Profiling Overhead Budgeting and Quota Enforcement defines an industrial-grade behavioral standard for devops engineering. Enforces strict CPU, memory, and telemetry ring-buffer quotas on OpenTelemetry eBPF continuous profiling agents deployed across high-density containerized nodes. Unrestricted sampling rates and unbounded userspace aggregation cause ring buffer drops, kernel scheduling latency, and CPU thrashing during traffic spikes. Telemetry infrastructure must implement hard sampling rate ceilings, memory limits, and automated backoff mechanisms to prevent kernel-level resource starvation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Uncapped continuous eBPF stack collection triggering kernel ring buffer contention and CPU starvation Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Uncapped continuous eBPF stack collection triggering kernel ring buffer contention and CPU starvation	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce a maximum profiling sampling frequency ceiling (e.g., between 19Hz and 49Hz) using non-harmonic prime frequencies to minimize lock-step synchronization with system clocks.

Engineers MUST ALWAYS enforce a maximum profiling sampling frequency ceiling (e.g., between 19Hz and 49Hz) using non-harmonic prime frequencies to minimize lock-step synchronization with system clocks.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.