OpenTelemetry eBPF Continuous Profiler Ring Buffer Sizing and Drop Mitigation Guardrail

--- description: OpenTelemetry eBPF Continuous Profiler Ring Buffer Sizing and Drop Mitigation Guardrail - Enforces strict kernel ring buffer page sizing, adaptive sampling frequencies, and epoll wakeup watermarks for OpenTelemetry eBPF continuous profiling agents. Prevents silent kernel-space sample drop spikes (-ENOSPC) during micro-bursts and guarantees bounded agent CPU overhead during high-density native stack unwinding. Ensures system stability by preventing unbounded lock memory allocations and kernel tracing telemetry blind spots. globs: deployments/**/otel-profiler*.{yaml,yml}, k8s/**/daemonset*.{yaml,yml}, helm/**/values*.{yaml,yml}, configs/**/profiler.{yaml,yml,conf} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiler-ringbuf-overflow-guardrail" --- # OpenTelemetry eBPF Continuous Profiler Ring Buffer Sizing and Drop Mitigation Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Statically Undersized BPF Ring Buffers with Unthrottled High-Frequency Native Stack Unwinding ## System Constraints You are a Principal Observability & Systems Engineer enforcing kernel safety, bounded overhead, and reliable telemetry capture for OpenTelemetry eBPF Continuous Profilers. Mandate invariants preventing BPF ring buffer drop events, unbounded stack table growth, and CPU cache thrashing during high-frequency production micro-bursts. ## Strict Guidelines - Engineers MUST ALWAYS set eBPF ring buffer sizes to a strict power of 2 (e.g., 2048 pages / 8MB per online CPU group) to satisfy kernel memory mapping alignment invariants. - Engineers MUST NOT configure static sampling frequencies above 19Hz without enabling hardware interrupt adaptive backpressure or micro-burst throttle gates. - Services MUST configure explicit epoll wakeup thresholds (`wakeup_events` or `bpf_ringbuf_reserve` watermark) to prevent userspace wakeups on every single stack sample. - Configurations MUST define `RLIMIT_MEMLOCK` to `unlimited` or a strictly calculated locked-memory budget on profiling agent daemonsets to avoid silent eBPF map creation failures. - Components NEVER drop telemetry silently; agents MUST export Prometheus/OTel counters for `bpf_ringbuf_dropped_samples_total` and alert when the drop ratio exceeds 0.01% of captured samples. ## Bad vs Good Implementations ### OpenTelemetry eBPF Profiler DaemonSet Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: A non-power-of-two page count for BPF ring buffers breaks BPF map allocation invariants, a fixed 100Hz sampling rate triggers kernel-space -ENOSPC drops during CPU bursts, and missing memory lock permissions causes BPF load failures. ```yaml # INVARIANT VIOLATION: Sub-optimal ring buffer, high fixed sample rate, missing memlock and dropped metrics apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:0.11.0 env: - name: SAMPLES_PER_SECOND value: "100" # Over-sampling causing heavy micro-burst ring buffer contention - name: BPF_RINGBUF_PAGES value: "500" # Invalid non-power-of-two page count; causes runtime fallback or failure resources: limits: memory: 200Mi ``` #### Production Standard (Correct) > **Standard**: Adheres to power-of-two page allocation requirements, selects an asynchronous 19Hz sampling rate to prevent harmonic bias, implements batch wakeups to suppress scheduler noise, and provides necessary Linux capabilities. ```yaml # COMPLIANT: Bounded power-of-two ring buffer, safe sample frequency, adaptive backpressure, and memlock set apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: securityContext: sysctls: - name: kernel.perf_event_paranoid value: "1" containers: - name: profiler image: otel/ebpf-profiler:0.11.0 securityContext: privileged: false capabilities: add: ["SYS_ADMIN", "SYS_PTRACE", "SYS_RESOURCE", "BPF", "PERFMON"] env: - name: SAMPLES_PER_SECOND value: "19" # Prime frequency to avoid harmonic lockstep with cron jobs - name: BPF_RINGBUF_PAGES value: "2048" # 8MB per core allocation; strictly power of two - name: WAKEUP_WATERMARK_PAGES value: "128" # Batch userspace drain to reduce context switches - name: MONITOR_DROPPED_SAMPLES value: "true" resources: limits: memory: 1Gi requests: memory: 512Mi cpu: 100m ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiler Ring Buffer Sizing and Drop Mitigation Guardrail

OpenTelemetry eBPF Continuous Profiler Ring Buffer Sizing and Drop Mitigation Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict kernel ring buffer page sizing, adaptive sampling frequencies, and epoll wakeup watermarks for OpenTelemetry eBPF continuous profiling agents. Prevents silent kernel-space sample drop spikes (-ENOSPC) during micro-bursts and guarantees bounded agent CPU overhead during high-density native stack unwinding. Ensures system stability by preventing unbounded lock memory allocations and kernel tracing telemetry blind spots. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Statically Undersized BPF Ring Buffers with Unthrottled High-Frequency Native Stack Unwinding Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Statically Undersized BPF Ring Buffers with Unthrottled High-Frequency Native Stack Unwinding	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS set eBPF ring buffer sizes to a strict power of 2 (e.g., 2048 pages / 8MB per online CPU group) to satisfy kernel memory mapping alignment invariants.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.