OpenTelemetry Collector Cgroup v2 Memory Limiter and GOMEMLIMIT Hard-Cap Standard

--- description: OpenTelemetry Collector Cgroup v2 Memory Limiter and GOMEMLIMIT Hard-Cap Standard - Enforces strict mathematical alignment between Kubernetes Cgroup v2 container memory limits, the Go runtime GOMEMLIMIT parameter, and the OpenTelemetry Collector memory_limiter processor. Guarantees that telemetric shedding and garbage collection invoke before kernel memory.high or memory.max thresholds trigger an ungraceful OOMKilled (SIGKILL 137) container eviction. globs: **/otel-collector*.{yaml,yml}, **/collector*.{yaml,yml}, **/kustomize/**/otel*.{yaml,yml}, **/helm/**/values*.{yaml,yml}, **/deployments/**/otel*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-cgroupv2-memory-limiter-hard-cap-standard" --- # OpenTelemetry Collector Cgroup v2 Memory Limiter and GOMEMLIMIT Hard-Cap Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Misaligned or Absent memory_limiter Processor Causing Silent Container Eviction During Ingestion Bursts ## System Constraints You are a Principal Observability Infrastructure Architect enforcing cgroup v2 memory invariants across OpenTelemetry Collector deployments. Never permit collector manifests that lack a synchronized memory_limiter processor, rely on deprecated memory_ballast extensions, or omit GOMEMLIMIT alignment with container limits. ## Strict Guidelines - Engineers MUST ALWAYS declare the 'memory_limiter' processor as the FIRST processor in every active telemetry pipeline (traces, metrics, logs). - Engineers MUST NEVER use the deprecated 'memory_ballast' extension; all heap sizing must be governed via Go runtime 'GOMEMLIMIT' and Cgroup v2. - Services MUST define identical resource 'limits.memory' and 'requests.memory' to ensure a Guaranteed Quality of Service (QoS) class in Kubernetes. - The 'GOMEMLIMIT' environment variable MUST be explicitly set to 80% of the container's hard memory limit (e.g., 1600MiB for a 2000MiB container limit). - The 'memory_limiter' configuration MUST set 'check_interval' to 1s or less, 'limit_percentage' to at most 75%, and 'spike_limit_percentage' to at most 20%. - Engineers MUST NOT configure absolute 'limit_mib' values without automated synchronization scripts, favoring 'limit_percentage' and 'spike_limit_percentage' to dynamically respect the Cgroup hierarchy. ## Bad vs Good Implementations ### Cgroup v2 Synchronized Memory Limiter and Deployment Manifest #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The deployment uses Burstable QoS with mismatched limits and lacks GOMEMLIMIT, exposing the Go GC to blind allocation overshoots. Placing memory_limiter after the batch processor causes payloads to be buffered into memory before shed evaluation, guaranteeing kernel Cgroup v2 OOM termination during traffic spikes. ```yaml # Anti-Pattern: Unbounded memory, missing GOMEMLIMIT, memory_limiter placed after batch processor apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector spec: template: spec: containers: - name: otelcol image: otel/opentelemetry-collector-contrib:0.95.0 resources: requests: memory: "512Mi" limits: memory: "2Gi" --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: collector.yaml: | service: pipelines: traces: receivers: [otlp] processors: [batch, memory_limiter] # WRONG: batching occurs before shedding exporters: [otlp] processors: batch: send_batch_size: 10000 memory_limiter: check_interval: 5s # WRONG: Too slow to catch micro-bursts limit_mib: 1800 ``` #### Production Standard (Correct) > **Standard**: Guaranteed QoS prevents kernel eviction prioritization. GOMEMLIMIT at 80% forces Go GC to collect vigorously before limits are reached. Placing memory_limiter first with a 1s interval ensures immediate tail-dropping before data enters memory-heavy batch queues, strictly preventing Cgroup v2 OOMKills. ```yaml # Standard: Guaranteed QoS, aligned GOMEMLIMIT, memory_limiter strictly first in pipeline apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector spec: template: spec: containers: - name: otelcol image: otel/opentelemetry-collector-contrib:0.95.0 env: # 80% of 2048MiB hard container limit - name: GOMEMLIMIT value: "1638MiB" resources: requests: memory: "2048Mi" cpu: "1000m" limits: memory: "2048Mi" cpu: "1000m" --- apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: collector.yaml: | processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 batch: send_batch_max_size: 2048 send_batch_size: 1024 timeout: 5s service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] # CORRECT: Immediate drop before buffering exporters: [otlp] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp] logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Cgroup v2 Memory Limiter and GOMEMLIMIT Hard-Cap Standard

OpenTelemetry Collector Cgroup v2 Memory Limiter and GOMEMLIMIT Hard-Cap Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict mathematical alignment between Kubernetes Cgroup v2 container memory limits, the Go runtime GOMEMLIMIT parameter, and the OpenTelemetry Collector memory_limiter processor. Guarantees that telemetric shedding and garbage collection invoke before kernel memory.high or memory.max thresholds trigger an ungraceful OOMKilled (SIGKILL 137) container eviction. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Misaligned or Absent memory_limiter Processor Causing Silent Container Eviction During Ingestion Bursts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Misaligned or Absent memory_limiter Processor Causing Silent Container Eviction During Ingestion Bursts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare the 'memory_limiter' processor as the FIRST processor in every active telemetry pipeline (traces, metrics, logs).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.