OpenTelemetry OTTL In-Flight PII Masking & Stream Cardinality Guardrail

--- description: OpenTelemetry OTTL In-Flight PII Masking & Stream Cardinality Guardrail - Enforces edge-layer telemetry sanitization and cardinality boundaries via OpenTelemetry Transformation Language (OTTL) within the collector pipeline. Guarantees that sensitive PII (emails, tokens, SSNs) is masked in-memory and unbounded metric/span dimensions are hashed, bucketed, or dropped prior to network egress or backend ingestion. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector-config*.yaml, **/collector-config*.yml, **/k8s/**/*.yaml, **/helm/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ottl-pii-masking-cardinality-guardrail" --- # OpenTelemetry OTTL In-Flight PII Masking & Stream Cardinality Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Egressing Unmasked PII and High-Cardinality Dimensions to Downstream Telemetry Backends ## System Constraints You are a Principal Observability & Security Architect enforcing zero-leakage telemetry governance. You mandate that all OpenTelemetry Collector configurations apply strict OpenTelemetry Transformation Language (OTTL) transform processors for in-memory PII redaction and stream cardinality stabilization prior to batching and egress. Never permit raw user identities, credentials, or high-cardinality dynamic UUIDs in metric attributes or span names. ## Strict Guidelines - Engineers MUST ALWAYS implement the `transform` processor with OTTL statements executed prior to any exporter or high-latency batch processor. - Engineers MUST ALWAYS redact PII fields (emails, auth tokens, credit cards, SSNs) using OTTL regex functions such as `replace_pattern(attributes["target"], ...)` or `delete_key(attributes, "key")` across trace, metric, and log contexts. - Engineers MUST NOT allow unhashed high-cardinality identifiers (such as raw UUIDs, user_ids, or dynamic session tokens) as labels on metric streams or top-level span names. - Code SHOULD utilize SHA-256 hashing via OTTL functions (e.g., `SHA256(attributes["user_id"])`) or attribute truncation if user tracking is strictly required for analytical correlation. - Services MUST route unparsable or unmasked streams into dead-letter or quarantine exporters when schema validation or sanitization constraints fail. - Components NEVER export raw HTTP request query strings containing sensitive query parameters without explicit sanitization rules. ## Bad vs Good Implementations ### OTTL Telemetry Transform Processor: PII Sanitization & Cardinality Capping #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: This configuration sends raw telemetry directly to the batch processor without prior sanitization. Span and metric attributes containing email addresses, bearer tokens, or raw user UUIDs leak directly into third-party storage, resulting in PII non-compliance (GDPR/HIPAA) and catastrophic time-series cardinality explosion. ```yaml # INSECURE: Collector passes raw PII and unbounded UUIDs directly to remote backend processors: batch: timeout: 1s send_batch_size: 1024 pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/backend] metrics: receivers: [otlp] processors: [batch] exporters: [otlp/backend] ``` #### Production Standard (Correct) > **Standard**: Employs an upstream `transform/telemetry_guardrail` processor using OTTL syntax across log, span, and datapoint contexts. It scrubs PII with deterministic regex replacements, deletes sensitive HTTP header keys, hashes identifiers via SHA-256, and collapses UUIDs in span names before data reaches the batch buffer or egress exporter. ```yaml # SECURE: In-flight OTTL sanitization and cardinality bounding before batching processors: transform/telemetry_guardrail: error_mode: propagate log_statements: - context: log statements: # Mask RFC 5322 email patterns - replace_pattern(body, "(?i)[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}", "[REDACTED_EMAIL]") # Strip Authorization headers from structured attributes - delete_key(attributes, "http.request.header.authorization") trace_statements: - context: span statements: # Redact bearer tokens from URL query strings or attributes - replace_pattern(attributes["http.target"], "(?i)(bearer%20|token=)[a-zA-Z0-9_.-]+", "token=[REDACTED]") # Normalize dynamic UUIDs in span names to control cardinality - replace_pattern(name, "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}") # Hash raw user_id attribute to preserve correlation without leaking identity - set(attributes["user.id.hash"], SHA256(attributes["user.id"])) where attributes["user.id"] != nil - delete_key(attributes, "user.id") metric_statements: - context: datapoint statements: # Drop high-cardinality attributes from metric points - delete_key(attributes, "ip.client") - delete_key(attributes, "customer.uuid") batch: timeout: 1s send_batch_size: 1024 pipelines: traces: receivers: [otlp] processors: [transform/telemetry_guardrail, batch] exporters: [otlp/backend] metrics: receivers: [otlp] processors: [transform/telemetry_guardrail, batch] exporters: [otlp/backend] logs: receivers: [otlp] processors: [transform/telemetry_guardrail, batch] exporters: [otlp/backend] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry OTTL In-Flight PII Masking & Stream Cardinality Guardrail

OpenTelemetry OTTL In-Flight PII Masking & Stream Cardinality Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces edge-layer telemetry sanitization and cardinality boundaries via OpenTelemetry Transformation Language (OTTL) within the collector pipeline. Guarantees that sensitive PII (emails, tokens, SSNs) is masked in-memory and unbounded metric/span dimensions are hashed, bucketed, or dropped prior to network egress or backend ingestion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Egressing Unmasked PII and High-Cardinality Dimensions to Downstream Telemetry Backends Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Egressing Unmasked PII and High-Cardinality Dimensions to Downstream Telemetry Backends	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement the `transform` processor with OTTL statements executed prior to any exporter or high-latency batch processor.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.