Zero-Loss Deterministic In-Flight PII Redaction for OTel Collector Pipelines

--- description: Zero-Loss Deterministic In-Flight PII Redaction for OTel Collector Pipelines - Enforces deterministic, zero-loss in-flight PII and credential scrubbing within OpenTelemetry Collector pipelines using OTTL (OpenTelemetry Transformation Language) transform processors. Prevents high-entropy tokens, JWTs, and regulated tenant data from leaking into downstream telemetry backends while guaranteeing span and log continuity without backpressure drops. globs: **/otel-collector*.{yaml,yml}, **/collector/*.{yaml,yml}, **/pipelines/**/*.{yaml,yml}, **/deployment/otel/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-stream-processor-zero-loss-pii-scrubber" --- # Zero-Loss Deterministic In-Flight PII Redaction for OTel Collector Pipelines > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Pass-Through Telemetry or Dropping Spans Due to Greedy Regex and Unsafe Failure Modes ## System Constraints You are a Principal Software Architect enforcing zero-loss in-flight telemetry sanitization across OpenTelemetry Collector pipelines. You mandate explicit, bounded, and deterministic OpenTelemetry Transformation Language (OTTL) transform configurations that strip high-entropy credentials, Authorization headers, connection strings, and regulated tenant PII before records reach memory buffers or export queues. Never allow raw telemetry pass-through or drop-on-error behavior. ## Strict Guidelines - Engineers MUST ALWAYS configure the `transform` processor with explicit OTTL context paths (`trace`, `log`, `metric`) rather than relying on destination telemetry SaaS scrubbing rules. - Engineers MUST NOT use unanchored, catastrophically backtracking regular expressions (e.g., greedy wildcard matching `.*` across entire JSON bodies) that induce collector CPU exhaustion or backpressure buffer drops. - Code SHOULD sanitize common credential keys deterministically across attributes, scope attributes, and resource attributes using OTTL `replace_all_patterns` or `set` masking functions. - Services MUST implement deterministic error handling with `error_mode: ignore` or explicit Dead-Letter Queue (DLQ) routing to ensure parsing errors redact and quarantine rather than silently dropping spans or crashing the collector daemon. - Components NEVER store or persist unredacted telemetry buffers to disk or retry queues destined for external multi-tenant backends. ## Bad vs Good Implementations ### Unsafe Telemetry Pipeline vs. Resilient Zero-Loss OTTL Redaction #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The pipeline sends unsanitized trace attributes (e.g., authorization headers, session cookies) directly to external vendors. The flawed log transformer uses an unbounded greedy regex without an explicit error_mode, introducing catastrophic backtracking vulnerabilities that cause collector CPU starvation and silent data loss. ```yaml # Vulnerable OTel configuration: Relies on downstream scrubbing or brittle regex processors: batch: send_batch_size: 8192 timeout: 5s pipelines: traces: receivers: [otlp] # Flaw: No in-flight sanitization; Authorization headers & API keys leak directly to external SaaS processors: [batch] exporters: [otlp/datadog] logs: receivers: [otlp] processors: [transform/flawed] exporters: [otlp/elastic] # Flawed transform: Catastrophic backtracking regex and silent drop on failure transform/flawed: log_statements: - context: log statements: # Greedy regex causes high collector CPU and drops incoming logs under load - replace_pattern(body, ".*token.*", "[REDACTED]") ``` #### Production Standard (Correct) > **Standard**: Applies memory-protected OTTL statements using non-greedy bounded matching to sanitize Authorization headers, database connection credentials, and emails across trace and log pipelines prior to batch buffering, ensuring zero unmasked persistent leaks and deterministic pipeline execution under heavy load. ```yaml # Hardened OTel configuration: In-flight OTTL stream redactor with zero-loss fallback processors: memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 20 transform/pii_scrubber: error_mode: ignore trace_statements: - context: span statements: # Mask authorization and session cookies across span attributes - set(attributes["http.request.header.authorization"], "[REDACTED]") where attributes["http.request.header.authorization"] != nil - set(attributes["http.request.header.cookie"], "[REDACTED]") where attributes["http.request.header.cookie"] != nil # Scrub standard bearer tokens and high-entropy API key patterns - replace_all_patterns(attributes, "value", "(?i)bearer\\s+[a-zA-Z0-9_\\-\\.~+/]+=*", "Bearer [REDACTED]") - replace_all_patterns(attributes, "value", "(?i)(api[_-]?key|secret|password)=\\S+", "$1=[REDACTED]") - context: resource statements: # Prevent host/container metadata credential leaks - replace_all_patterns(attributes, "value", "(?i)(postgres|mysql|mongodb)://[^:]+:[^@]+@", "$1://[REDACTED]:[REDACTED]@") log_statements: - context: log statements: - set(attributes["authorization"], "[REDACTED]") where attributes["authorization"] != nil - replace_all_patterns(attributes, "value", "(?i)[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}", "[REDACTED_EMAIL]") batch: send_batch_size: 8192 timeout: 1s pipelines: traces: receivers: [otlp] processors: [memory_limiter, transform/pii_scrubber, batch] exporters: [otlp/datadog] logs: receivers: [otlp] processors: [memory_limiter, transform/pii_scrubber, batch] exporters: [otlp/elastic] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Zero-Loss Deterministic In-Flight PII Redaction for OTel Collector Pipelines

Zero-Loss Deterministic In-Flight PII Redaction for OTel Collector Pipelines defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, zero-loss in-flight PII and credential scrubbing within OpenTelemetry Collector pipelines using OTTL (OpenTelemetry Transformation Language) transform processors. Prevents high-entropy tokens, JWTs, and regulated tenant data from leaking into downstream telemetry backends while guaranteeing span and log continuity without backpressure drops. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Pass-Through Telemetry or Dropping Spans Due to Greedy Regex and Unsafe Failure Modes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Pass-Through Telemetry or Dropping Spans Due to Greedy Regex and Unsafe Failure Modes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure the `transform` processor with explicit OTTL context paths (`trace`, `log`, `metric`) rather than relying on destination telemetry SaaS scrubbing rules.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.