OpenTelemetry eBPF Continuous Profiling Budget and Lossy Pprof Tail-Sampling Ceiling

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded eBPF sample rates causing collector OOMs and >5% CPU overhead in production nodes) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

--- description: OpenTelemetry eBPF Continuous Profiling Budget and Lossy Pprof Tail-Sampling Ceiling - Enforces strict resource limits on OpenTelemetry eBPF-based continuous profiling agents to cap kernel-space and user-space overhead at under 1% total CPU utilization. Mandates downstream collector pipelines to enforce lossy tail-sampling and backpressure ceilings on pprof ingestion to prevent collector OOM crashes during stack trace fan-out storms. globs: k8s/**/*.yaml, deploy/**/*.yaml, helm/**/*.yaml, collector/**/*.yaml, src/observability/**/*.{ts,go,yaml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-profiling-cpu-budget-pprof-ceiling" --- # OpenTelemetry eBPF Continuous Profiling Budget and Lossy Pprof Tail-Sampling Ceiling > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded eBPF sample rates causing collector OOMs and >5% CPU overhead in production nodes ## System Constraints You are a Principal Observability Infrastructure Architect enforcing strict CPU budgets on eBPF profiling agents and memory-bounded pprof ingestion pipelines. You must reject any profiling daemonset or OpenTelemetry Collector configuration that lacks hard sampling limits, adaptive rate-limiting, cgroup CPU quota ceilings under 100m cores, and tail-based stack frame aggregation discard policies. ## Strict Guidelines - Profiling DaemonSets MUST configure cgroup limits guaranteeing CPU utilization strictly under 1% of host capacity (typically <= 100m per node). - eBPF collection frequency MUST NOT exceed 19 Hz (19 samples per second per core) to prevent context-switching lockups in production kernels. - The OpenTelemetry Collector receiving pprof profiles MUST implement `memory_limiter` processor with hard `check_interval` and `limit_percentage` set to <= 80%. - Stack trace ingestion pipelines MUST implement probabilistic tail-sampling or adaptive lossy down-sampling when ingestion queues cross a 75% depth threshold. - Collector pipelines MUST NOT buffer uncompressed raw pprof frames in volatile heap memory without explicit batch size ceilings and persistent spill buffers. ## Bad vs Good Implementations ### OpenTelemetry Collector Profiling & eBPF DaemonSet Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The eBPF profiler samples at 100Hz without CPU limits, easily consuming 5-8% host CPU and triggering kernel throttling. Downstream, the collector lacks memory limiter and tail-sampling processors, exposing the aggregator to unbounded heap exhaustion during thread contention spikes. ```yaml # Anti-Pattern: Uncapped sampling frequency and missing collector backpressure apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:latest env: - name: SAMPLING_FREQUENCY value: "100" # Over 100Hz causes > 5% CPU overhead resources: {} --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: otel-collector-config.yaml: | receivers: otlp: protocols: grpc: processors: batch: exporters: otlphttp: endpoint: https://profiling-backend:443 service: pipelines: profiles: receivers: [otlp] processors: [batch] exporters: [otlphttp] ``` #### Production Standard (Correct) > **Standard**: Sets eBPF sampling rate to 19Hz (a prime interval reducing frequency aliasing) and enforces strict 100m CPU limits. Collector introduces `memory_limiter` and probabilistic tail-sampling ceilings to discard redundant stack traces under heavy profiling load, maintaining 1% budget bounds. ```yaml # Production Guardrail: Sub-1% CPU cap (19Hz) with lossy tail-sampling ceiling apiVersion: apps/v1 kind: DaemonSet metadata: name: otel-ebpf-profiler spec: template: spec: containers: - name: profiler image: otel/ebpf-profiler:1.2.0 env: - name: ESY_PROFILER_SAMPLING_RATE value: "19" # 19Hz avoids harmonic lockstep while remaining <1% CPU resources: limits: cpu: "100m" memory: "128Mi" requests: cpu: "25m" memory: "64Mi" --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: otel-collector-config.yaml: | receivers: otlp: protocols: grpc: max_concurrent_streams: 100 processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 tail_sampling: decision_wait: 5s expected_new_traces_per_sec: 2000 policies: - name: drop_high_frequency_stacks type: probabilistic probabilistic: { sampling_percentage: 10.0 } batch: send_batch_size: 512 timeout: 5s exporters: otlphttp: endpoint: https://profiling-backend:443 sending_queue: enabled: true queue_size: 1000 service: pipelines: profiles: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlphttp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Budget and Lossy Pprof Tail-Sampling Ceiling

OpenTelemetry eBPF Continuous Profiling Budget and Lossy Pprof Tail-Sampling Ceiling defines an industrial-grade behavioral standard for devops engineering. Enforces strict resource limits on OpenTelemetry eBPF-based continuous profiling agents to cap kernel-space and user-space overhead at under 1% total CPU utilization. Mandates downstream collector pipelines to enforce lossy tail-sampling and backpressure ceilings on pprof ingestion to prevent collector OOM crashes during stack trace fan-out storms. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded eBPF sample rates causing collector OOMs and >5% CPU overhead in production nodes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded eBPF sample rates causing collector OOMs and >5% CPU overhead in production nodes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Profiling DaemonSets MUST configure cgroup limits guaranteeing CPU utilization strictly under 1% of host capacity (typically <= 100m per node).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.