OTel Collector Continuous Profiling Flame Graph Sampling and Memory Limiter Guardrail

--- description: OTel Collector Continuous Profiling Flame Graph Sampling and Memory Limiter Guardrail - Enforces strict memory limiter pre-filtering, backpressure buffers, and adaptive sample-rate governors for OpenTelemetry continuous profiling data streams. Prevents heap allocation telemetry bursts from exhausting Collector cgroup limits and causing catastrophic container runtime OOMKills during service memory spikes. globs: **/otel-collector*.{yaml,yml}, **/collector/*.{yaml,yml}, **/k8s/**/*.{yaml,yml}, deployments/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-continuous-profiling-sampling-standard" --- # OTel Collector Continuous Profiling Flame Graph Sampling and Memory Limiter Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Continuous Profiling Ingestion Without Upstream Limiting Induces Collector OOMKills ## System Constraints You are a Principal Software Architect enforcing production reliability standards for OpenTelemetry continuous profiling pipelines. You must mandate that all OTel Collector configurations handling pprof, eBPF, or flame-graph telemetry strictly sequence the memory_limiter processor as the first processor in the pipeline, enforce hard drop thresholds on profiling spans, set bounded batch buffer limits, and govern client-side heap profile sample rates to preclude OOMKills under adverse memory spikes. ## Strict Guidelines - Engineers MUST ALWAYS position the `memory_limiter` processor as the FIRST processor in the `pipelines.profiles` and `pipelines.traces` definitions. - Engineers MUST NOT configure profiling pipelines without explicit `check_interval`, `limit_percentage`, and `spike_limit_percentage` thresholds synchronized with the container's cgroup memory limits. - Services MUST govern application-level heap profile collection rates (e.g., `MemProfileRate` >= 512KB) and cap profile payload sizes prior to network transmission. - Code SHOULD configure the `batch` processor immediately following the `memory_limiter` with a concrete `send_batch_max_size` and non-zero `timeout` to avoid unbounded profile buffering. - Components NEVER accept raw, unaggregated heap allocation events without a downstream probabilistic sampler or rate-limiting filter. ## Bad vs Good Implementations ### OTel Collector Pipeline Configuration for Continuous Profiling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Without the memory_limiter processor positioned first, an application experiencing a rapid memory allocation spike will flood the collector with heavy heap profile frames, causing the collector to allocate faster than the Go runtime GC can reclaim, inevitably triggering an unrecoverable kernel OOMKill. ```yaml # Anti-Pattern: No memory limiter, unbounded batching, unthrottled profile ingestion receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: # Missing send_batch_max_size and memory bounding timeout: 10s exporters: otlphttp/profiling: endpoint: https://profiling-backend.internal:443 service: pipelines: profiles: receivers: [otlp] processors: [batch] # Anti-pattern: batch processor without preceding memory_limiter exporters: [otlphttp/profiling] ``` #### Production Standard (Correct) > **Standard**: The memory_limiter executes first, actively shedding load and returning gRPC status ResourceExhausted before the collector's cgroup memory limit is reached. The batch processor bounds buffer sizes, and the exporter enforces a finite queue size to prevent memory leaks during downstream outages. ```yaml # Production Standard: Enforced memory_limiter guardrail with adaptive head sampling receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 max_concurrent_streams: 128 processors: memory_limiter: check_interval: 250ms limit_percentage: 75 spike_limit_percentage: 20 probabilistic_sampler: sampling_percentage: 10.0 batch: send_batch_size: 256 send_batch_max_size: 512 timeout: 5s exporters: otlphttp/profiling: endpoint: https://profiling-backend.internal:443 sending_queue: enabled: true num_consumers: 4 queue_size: 1000 retry_on_failure: enabled: true initial_interval: 1s max_interval: 30s service: pipelines: profiles: receivers: [otlp] processors: [memory_limiter, probabilistic_sampler, batch] exporters: [otlphttp/profiling] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OTel Collector Continuous Profiling Flame Graph Sampling and Memory Limiter Guardrail

OTel Collector Continuous Profiling Flame Graph Sampling and Memory Limiter Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict memory limiter pre-filtering, backpressure buffers, and adaptive sample-rate governors for OpenTelemetry continuous profiling data streams. Prevents heap allocation telemetry bursts from exhausting Collector cgroup limits and causing catastrophic container runtime OOMKills during service memory spikes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Continuous Profiling Ingestion Without Upstream Limiting Induces Collector OOMKills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Continuous Profiling Ingestion Without Upstream Limiting Induces Collector OOMKills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS position the `memory_limiter` processor as the FIRST processor in the `pipelines.profiles` and `pipelines.traces` definitions.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.