OpenTelemetry Continuous Profiling Ring-Buffer Memory Cap Standard

--- description: OpenTelemetry Continuous Profiling Ring-Buffer Memory Cap Standard - Enforces strict off-heap, bounded ring-buffer memory allocations for OpenTelemetry eBPF continuous CPU stack sampling agents. Prohibits dynamic heap allocations, unbounded backpressure queues, and user-space memory growth during stack unwinding. Guarantees predictable p99 kernel-to-user context switch latency and prevents out-of-memory (OOM) eviction of production workloads. globs: **/profiler/**/*.{ts,go,c,rs}, **/telemetry/**/*.{ts,go,c,rs}, **/otel/**/*.{ts,go,c,rs}, **/daemonset.yaml, **/helm/**/values.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-profiling-ring-buffer-cap-standard" --- # OpenTelemetry Continuous Profiling Ring-Buffer Memory Cap Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded User-Space Buffering and Dynamic Allocations for eBPF Profiling Telemetry ## System Constraints You are a Principal Systems and Reliability Architect enforcing memory and scheduling safety invariants for OpenTelemetry continuous profiling. You must prohibit unbounded dynamic heap growth, ensure all eBPF profile sample collectors consume strictly pre-allocated, power-of-two page-aligned off-heap ring buffers with explicit overflow drop metrics, and prevent user-space serialization spikes from degrading kernel context-switch budgets. ## Strict Guidelines - Profiles MUST ALWAYS allocate memory from a fixed, pre-allocated off-heap ring buffer with a statically configured upper-bound limit (e.g., maximum 64MB per node/agent). - Stack collection agents MUST use non-blocking BPF ring buffer consumption patterns (`BPF_MAP_TYPE_RINGBUF`) with overwrite or drop semantics rather than blocking syscalls. - The agent MUST NOT invoke dynamic heap allocations (such as `malloc` or growing dynamic slices/vectors) inside the hot unwinding and sample ingestion loop. - Collector exporters MUST discard incoming frames using an atomic counter (`otel_profiling_dropped_samples_total`) when the downstream buffer reaches high-water mark (>= 85% capacity). - Profilers MUST enforce strict power-of-two page sizing (`page_size * 2^n`) for perf/ring buffers to avoid kernel virtual memory fragmentation and TLB thrashing. - Agents MUST NOT synchronize with or block on external network I/O; OTLP export threads MUST run isolated from the eBPF kernel-consumer worker pool via lock-free bounded single-producer single-consumer (SPSC) channels. ## Bad vs Good Implementations ### Bounded Pre-allocated Ring Buffer vs Dynamic Heap Appending #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Pushing raw profiling samples onto a dynamic JavaScript array creates unbounded heap allocations during high CPU bursts. Under sustained load or network delay, memory consumption escalates rapidly, triggering heavy garbage collection cycles and risking Node.js process OOM. ```typescript import { ProfilingCollector, RawSample } from '@opentelemetry/profiling-node'; export class DynamicProfilerService implements ProfilingCollector { // ANTI-PATTERN: Dynamic array grows unbounded under high sampling rates private sampleQueue: RawSample[] = []; private isExporting = false; public onStackSampleReceived(sample: RawSample): void { // Allocates on V8 heap inside the hot profiling path this.sampleQueue.push(sample); if (this.sampleQueue.length > 1000 && !this.isExporting) { this.flushSamples(); } } private async flushSamples(): Promise<void> { this.isExporting = true; // Unbounded serialization causes GC spikes and memory pressure const batch = this.sampleQueue.splice(0); await fetch('http://otel-collector:4318/v1/profiles', { method: 'POST', body: JSON.stringify(batch), }); this.isExporting = false; } } ``` #### Production Standard (Correct) > **Standard**: Uses a pre-allocated `SharedArrayBuffer` of fixed capacity. Memory is partitioned off-heap without allocating dynamic objects in the hot telemetry path. When capacity is exceeded, samples are dropped instantly and recorded via an atomic metric counter. ```typescript import { Counter, Meter } from '@opentelemetry/api'; export class FixedRingBufferProfiler { private readonly ringBuffer: SharedArrayBuffer; private readonly dataView: Uint8Array; private readonly capacity: number; private writeHead = 0; private readHead = 0; private readonly droppedSamplesMetric: Counter; constructor(meter: Meter, capacityBytes: number = 16 * 1024 * 1024) { // INVARIANT: Fixed off-heap buffer size capped strictly (16MB default) this.capacity = capacityBytes; this.ringBuffer = new SharedArrayBuffer(this.capacity); this.dataView = new Uint8Array(this.ringBuffer); this.droppedSamplesMetric = meter.createCounter('otel_profiling_dropped_samples_total', { description: 'Total dropped samples due to ring buffer saturation', }); } public offerSample(serializedFrame: Uint8Array): boolean { const sampleLength = serializedFrame.byteLength; const availableSpace = this.capacity - (this.writeHead - this.readHead); if (sampleLength > availableSpace) { // Strict drop policy: drop sample immediately without heap allocation this.droppedSamplesMetric.add(1); return false; } const offset = this.writeHead % this.capacity; this.dataView.set(serializedFrame, offset); this.writeHead += sampleLength; return true; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling Ring-Buffer Memory Cap Standard

OpenTelemetry Continuous Profiling Ring-Buffer Memory Cap Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict off-heap, bounded ring-buffer memory allocations for OpenTelemetry eBPF continuous CPU stack sampling agents. Prohibits dynamic heap allocations, unbounded backpressure queues, and user-space memory growth during stack unwinding. Guarantees predictable p99 kernel-to-user context switch latency and prevents out-of-memory (OOM) eviction of production workloads. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded User-Space Buffering and Dynamic Allocations for eBPF Profiling Telemetry Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded User-Space Buffering and Dynamic Allocations for eBPF Profiling Telemetry	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Profiles MUST ALWAYS allocate memory from a fixed, pre-allocated off-heap ring buffer with a statically configured upper-bound limit (e.g., maximum 64MB per node/agent).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.