OpenTelemetry Continuous Profiling eBPF Delta Buffer Memory Bounding

--- description: OpenTelemetry Continuous Profiling eBPF Delta Buffer Memory Bounding - Continuous eBPF profiling creates high-frequency stack trace deltas that can rapidly exhaust system memory under sustained CPU pressure or collector backpressure. This guardrail enforces deterministic byte-bounded ring buffers, explicit drop-on-saturation policies, and memory limiter invariants across all OTLP/Profiles aggregation pipelines to prevent node OOM crashes. globs: src/profiling/**/*.{ts,tsx,js}, src/observability/**/*.{ts,tsx,js}, src/collector/**/*.{ts,tsx,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-otlp-profiles-memory-bound-invariant" --- # OpenTelemetry Continuous Profiling eBPF Delta Buffer Memory Bounding > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded stack-trace delta aggregation maps without hard memory limits or sample drop metrics under eBPF load spikes. ## System Constraints You are a Principal Observability & Systems Architect enforcing strict memory invariants for continuous profiling and eBPF data ingestion pipelines. Mandate deterministic memory bounding for all stack trace delta buffers, bounded aggregation windows, explicit sample-dropping telemetry under backpressure, and absolute protection against node-level OOM events. ## Strict Guidelines - Engineers MUST ALWAYS enforce an immutable maximum byte budget on in-memory stack trace delta buffers before initiating OTLP exports. - Engineers MUST NOT allow eBPF delta aggregators to allocate unbounded Map or Array structures across flush intervals. - Pipelines MUST implement a deterministic drop policy (e.g., drop-tail or drop-oldest) with an incremented metric counter whenever the delta buffer reaches its high watermark. - Services MUST monitor heap pressure and halt local stack ingestion when total allocated profiling memory exceeds 75% of the configured cgroup/container limit. - Components NEVER block the main kernel/eBPF consumer loop waiting for network acknowledgment from remote OTLP collectors; flushing MUST be non-blocking and decoupled. ## Bad vs Good Implementations ### Bounded Profile Delta Buffer with Byte Accounting vs Unbounded Map #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The aggregator stores samples in an unbounded Map without tracking memory consumption or enforcing a maximum capacity. If network export latency rises or eBPF thread contention spikes, heap memory escalates unchecked, triggering a node OOM panic. ```typescript import { Profile, Sample } from '@opentelemetry/exporter-profiles-otlp-http'; // Anti-pattern: Unbounded accumulator map that grows infinitely under // CPU pressure or slow collector ingestion, causing host OOMs. export class UnboundedDeltaAggregator { private sampleBuffer = new Map<string, number>(); public recordSample(stackTraceKey: string, cpuWeightNs: number): void { const current = this.sampleBuffer.get(stackTraceKey) ?? 0; this.sampleBuffer.set(stackTraceKey, current + cpuWeightNs); } public async flush(exporter: { export: (data: unknown) => Promise<void> }): Promise<void> { const payload = Array.from(this.sampleBuffer.entries()).map(([stack, weight]) => ({ stack, weight, })); // Network stall here causes unbounded memory growth as recordSample continues await exporter.export(payload); this.sampleBuffer.clear(); } } ``` #### Production Standard (Correct) > **Standard**: Enforces a strict upper bound on both entry count and raw memory footprint. If eBPF throughput exceeds the buffer threshold, new unique stacks are dropped rather than exhausting heap memory, while recording dropped samples for telemetry alerting. ```typescript export interface BoundedAggregatorConfig { readonly maxBufferBytes: number; readonly maxEntries: number; } export class BoundedDeltaAggregator { private readonly sampleBuffer = new Map<string, number>(); private currentEstimatedBytes = 0; private droppedSamplesCount = 0; constructor(private readonly config: BoundedAggregatorConfig) {} public recordSample(stackTraceKey: string, cpuWeightNs: number): boolean { const existing = this.sampleBuffer.get(stackTraceKey); if (existing !== undefined) { this.sampleBuffer.set(stackTraceKey, existing + cpuWeightNs); return true; } // 64-bit integer (8 bytes) + string length * 2 (UTF-16) + map overhead (~64 bytes) const entryMemoryCost = stackTraceKey.length * 2 + 72; const projectedBytes = this.currentEstimatedBytes + entryMemoryCost; if ( projectedBytes > this.config.maxBufferBytes || this.sampleBuffer.size >= this.config.maxEntries ) { this.droppedSamplesCount++; return false; // Deterministic drop under ingestion backpressure } this.sampleBuffer.set(stackTraceKey, cpuWeightNs); this.currentEstimatedBytes = projectedBytes; return true; } public getDroppedSamplesCount(): number { return this.droppedSamplesCount; } public drain(): Array<{ stack: string; weight: number }> { const payload: Array<{ stack: string; weight: number }> = []; for (const [stack, weight] of this.sampleBuffer.entries()) { payload.push({ stack, weight }); } this.sampleBuffer.clear(); this.currentEstimatedBytes = 0; return payload; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling eBPF Delta Buffer Memory Bounding

OpenTelemetry Continuous Profiling eBPF Delta Buffer Memory Bounding defines an industrial-grade behavioral standard for devops engineering. Continuous eBPF profiling creates high-frequency stack trace deltas that can rapidly exhaust system memory under sustained CPU pressure or collector backpressure. This guardrail enforces deterministic byte-bounded ring buffers, explicit drop-on-saturation policies, and memory limiter invariants across all OTLP/Profiles aggregation pipelines to prevent node OOM crashes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded stack-trace delta aggregation maps without hard memory limits or sample drop metrics under eBPF load spikes. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded stack-trace delta aggregation maps without hard memory limits or sample drop metrics under eBPF load spikes.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce an immutable maximum byte budget on in-memory stack trace delta buffers before initiating OTLP exports.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.