OpenTelemetry eBPF Continuous Profiling Adaptive Tail-Sampling Guardrail

--- description: OpenTelemetry eBPF Continuous Profiling Adaptive Tail-Sampling Guardrail - Governs kernel-level continuous profiling ingestion rates using adaptive tail-sampling to capture thread latency anomalies without exhausting telemetry storage backends or collector memory. Mandates local ring-buffer retention with p99-evaluated forwarding, strict bounded heap limits, and backpressure shedding across eBPF collectors. globs: otelcol/**/*.yaml, otelcol/**/*.yml, deploy/k8s/**/*.yaml, src/observability/**/*.ts, src/profiling/**/*.go, helm/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-tail-sampling-invariant" --- # OpenTelemetry eBPF Continuous Profiling Adaptive Tail-Sampling Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Naive head-sampling or unbounded forwarding of high-frequency eBPF stack traces leading to OOM crashes and storage blowouts. ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing invariants for OpenTelemetry eBPF Continuous Profiling. You must enforce deterministic memory bounding, circular ring-buffer retention in user space, and adaptive tail-sampling driven by execution duration and off-CPU latency thresholds. Never allow raw, unbounded emission of kernel stack unwinds directly to upstream telemetry endpoints. ## Strict Guidelines - Collectors MUST maintain an in-memory BPF ring-buffer or user-space circular trace buffer with a deterministic maximum memory cap (max_bytes) before making retention decisions. - Engineers MUST configure adaptive tail-sampling policies that evaluate trace/thread duration against latency thresholds (e.g., p99 execution duration >= 250ms, off-CPU scheduling latency spikes) before exporting continuous profiling profiles. - Services MUST emit aggregated pprof/OTel profiles with strict symbolization limits, enforcing stack folding and deduplication at the node collector level prior to network transit. - Collectors MUST NOT use uniform or head-based sampling for high-frequency eBPF unwinds (e.g., 100Hz CPU unwinds) without downsampling or tail-sampling gates. - Pipeline configurations MUST define explicit drop-tail backpressure strategies (e.g., memory_limiter processor combined with drop_oldest queueing) to prevent node-level resource contention under thread contention storms. ## Bad vs Good Implementations ### OpenTelemetry Collector Tail-Sampling Pipeline for Profiling Data #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Exporting raw 99Hz eBPF stack unwinds directly through a naive batch processor saturates downstream networks and storage backends. It lacks local memory boundaries, off-CPU anomaly filtering, and tail-sampling, triggering OOM kills during CPU/thread contention spikes. ```yaml # Anti-Pattern: Direct forwarding of unthrottled eBPF profiling without memory limits or tail-sampling receivers: ebpf_profiler: sampling_rate_hz: 99 collect_interval: 10s processors: batch: send_batch_size: 1000 timeout: 5s exporters: otlp: endpoint: profiling-storage.internal:4317 service: pipelines: profiles: receivers: [ebpf_profiler] processors: [batch] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: Implements an explicit memory limiter, ring-buffer constraints, and an adaptive tail-sampling processor that retains baseline profiles at 1% while capturing 100% of thread execution anomalies (>200ms duration or >50ms off-CPU blocked time). ```yaml # Invariant Compliant: Adaptive tail-sampling with strict memory barriers and latency anomaly triggers receivers: ebpf_profiler: sampling_rate_hz: 99 collect_interval: 5s ring_buffer_pages: 512 off_cpu_threshold_ms: 50 processors: memory_limiter: check_interval: 500ms limit_percentage: 75 spike_limit_percentage: 20 tail_sampling: decision_wait: 10s num_traces: 5000 expected_new_traces_per_sec: 1000 policies: - name: thread_latency_anomaly type: numeric_attribute numeric_attribute: key: profile.duration_ms min_value: 200 - name: off_cpu_block_anomaly type: numeric_attribute numeric_attribute: key: profile.off_cpu_blocked_ms min_value: 50 - name: baseline_rate_limit type: probabilistic probabilistic: sampling_percentage: 1.0 batch: send_batch_max_size: 256 timeout: 2s exporters: otlp: endpoint: profiling-storage.internal:4317 sending_queue: enabled: true num_consumers: 4 queue_size: 1000 storage: memory service: pipelines: profiles: receivers: [ebpf_profiler] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry eBPF Continuous Profiling Adaptive Tail-Sampling Guardrail

OpenTelemetry eBPF Continuous Profiling Adaptive Tail-Sampling Guardrail defines an industrial-grade behavioral standard for devops engineering. Governs kernel-level continuous profiling ingestion rates using adaptive tail-sampling to capture thread latency anomalies without exhausting telemetry storage backends or collector memory. Mandates local ring-buffer retention with p99-evaluated forwarding, strict bounded heap limits, and backpressure shedding across eBPF collectors. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Naive head-sampling or unbounded forwarding of high-frequency eBPF stack traces leading to OOM crashes and storage blowouts. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Naive head-sampling or unbounded forwarding of high-frequency eBPF stack traces leading to OOM crashes and storage blowouts.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Collectors MUST maintain an in-memory BPF ring-buffer or user-space circular trace buffer with a deterministic maximum memory cap (max_bytes) before making retention decisions.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.