OpenTelemetry Collector Processor Pipeline Order and Memory Guardrails

--- description: OpenTelemetry Collector Processor Pipeline Order and Memory Guardrails - Mandates strict ordering and configuration of OpenTelemetry Collector processor pipelines. Enforces memory_limiter as the non-negotiable first processor to prevent process crashes, requires OTTL-driven PII redaction before persistence or sampling, and mandates dedicated pipeline tiers for tail-based sampling. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector*.yaml, **/collector*.yml, deploy/k8s/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-processor-pipeline-safeguards" --- # OpenTelemetry Collector Processor Pipeline Order and Memory Guardrails > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Placing batch or sampling processors before memory limiters and exporting raw unredacted telemetry ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing OpenTelemetry Collector configuration standards. Every pipeline configuration must enforce deterministic processor sequencing: memory_limiter MUST precede all other processors to eliminate Out-Of-Memory (OOM) fatal crashes; transform processors MUST redact PII attributes using OTTL before telemetry hits sampling or exporting stages; and batch processors MUST be tuned alongside memory ceilings. Never emit unredacted attributes, unbounded batch sizes, or inverted processor pipelines. ## Strict Guidelines - Engineers MUST ALWAYS declare 'memory_limiter' as the first processor in every pipeline (traces, metrics, logs) before any compute-intensive or buffer-allocating processors. - Engineers MUST NOT place 'batch' or 'tail_sampling' processors before 'memory_limiter', as high ingest volume will cause unbounded heap growth and fatal SIGKILL termination. - Engineers MUST configure 'memory_limiter' with explicit 'check_interval', 'limit_percentage' (or 'limit_mib'), and 'spike_limit_percentage' (or 'spike_limit_mib') calibrated to the container memory ceiling. - Engineers MUST ALWAYS redact sensitive attributes (PII, tokens, auth headers) using the 'transform' processor via OpenTelemetry Transformation Language (OTTL) prior to any sampling or export step. - Pipelines using 'tail_sampling' MUST be deployed in an isolated collector cluster with trace-ID-consistent routing, and MUST execute after PII redaction but before the final 'batch' processor. - Batch processors MUST declare explicit 'send_batch_size', 'timeout', and 'send_batch_max_size' to ensure bounded memory envelopes during network backpressure. ## Bad vs Good Implementations ### OpenTelemetry Collector Processor Pipeline Order and Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The memory_limiter is placed last in the pipeline, which renders it ineffective against spikes buffering in tail_sampling and batch processors, leading to unpreventable container OOMKills. Furthermore, no transform/redaction processor is configured, causing sensitive authorization and user PII to leak to APM backends. ```yaml processors: batch: timeout: 10s tail_sampling: decision_wait: 10s expected_new_traces_per_sec: 2000 policies: [ { name: drop_healthchecks, type: string_attribute, string_attribute: { key: http.target, values: [ "/healthz" ], enabled_regex_matching: false, invert_match: true } } ] memory_limiter: check_interval: 5s limit_percentage: 80 spike_limit_percentage: 20 service: pipelines: traces: receivers: [otlp] # ANTI-PATTERN: memory_limiter is executed last. Tail sampling and batching # will consume unconstrained memory under load spikes, causing OOM kills. # Missing PII redaction transforms exposes credentials to downstream backends. processors: [tail_sampling, batch, memory_limiter] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The pipeline rigorously applies the memory_limiter first, shielding the collector from heap exhaustion. PII attributes are wiped next via OTTL transforms before the tail_sampling decision engine. Finally, batching executes with rigid upper bounds right before network export. ```yaml processors: # INVARIANT 1: memory_limiter must be first to evaluate allocation against limits memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 # INVARIANT 2: OTTL transform redacts PII before trace decisions or exports transform/pii_redaction: error_mode: ignore trace_statements: - context: span statements: - replace_pattern(attributes["http.request.header.authorization"], ".*", "[REDACTED]") - replace_pattern(attributes["user.email"], ".*", "[REDACTED]") - delete_key(attributes, "db.statement.parameters") # INVARIANT 3: Tail-based sampling operates on sanitized spans tail_sampling: decision_wait: 5s num_traces: 50000 expected_new_traces_per_sec: 5000 policies: - name: drop_healthchecks type: string_attribute string_attribute: key: http.target values: ["/healthz", "/ready"] invert_match: true # INVARIANT 4: Bounded batch processor immediately precedes export batch: send_batch_size: 8192 timeout: 1s send_batch_max_size: 10240 service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, transform/pii_redaction, tail_sampling, batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Processor Pipeline Order and Memory Guardrails

OpenTelemetry Collector Processor Pipeline Order and Memory Guardrails defines an industrial-grade behavioral standard for devops engineering. Mandates strict ordering and configuration of OpenTelemetry Collector processor pipelines. Enforces memory_limiter as the non-negotiable first processor to prevent process crashes, requires OTTL-driven PII redaction before persistence or sampling, and mandates dedicated pipeline tiers for tail-based sampling. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Placing batch or sampling processors before memory limiters and exporting raw unredacted telemetry Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Placing batch or sampling processors before memory limiters and exporting raw unredacted telemetry	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS declare 'memory_limiter' as the first processor in every pipeline (traces, metrics, logs) before any compute-intensive or buffer-allocating processors.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.