OpenTelemetry Collector Dynamic Semantic Attribute Stripping

--- description: OpenTelemetry Collector Dynamic Semantic Attribute Stripping - Enforces deterministic attribute normalization and transforms within the OpenTelemetry Collector using the transform processor and OTTL (OpenTelemetry Transformation Language). Strips unbounded, high-cardinality metadata from spans, metrics, and exemplar contexts at the ingestion tier to prevent catastrophic downstream time-series explosions across Prometheus, Mimir, and Jaeger backends without truncating trace lineage. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector/*.yaml, **/k8s/**/otel*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-collector-dynamic-attribute-stripping" --- # OpenTelemetry Collector Dynamic Semantic Attribute Stripping > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Passing Unbounded High-Cardinality Attributes and Raw PII Direct to Downstream Time-Series Storage ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing deterministic OpenTelemetry Collector ingress guardrails. Mandate OTTL-based dynamic attribute stripping, exemplar sanitization, and metric label isolation to prevent downstream time-series database crashes (OOM) while strictly preserving distributed trace correlation IDs (trace_id, span_id). ## Strict Guidelines - Engineers MUST ALWAYS use the 'transform' processor with OTTL statements instead of broad-stroke filter processors when shedding cardinality, ensuring parent span linkages and span contexts remain intact. - Engineers MUST NOT allow dynamic user identifiers, raw UUIDs, unbounded database queries, or authorization headers (e.g., 'user.id', 'db.statement', 'http.request.header.authorization') to leak into metric labels or exemplar attributes. - Pipelines MUST truncate and hash required diagnostic attributes that contain dynamic parameters using OTTL hash or SHA functions rather than storing arbitrary raw strings. - Processors MUST explicitly delete or restrict exemplar attributes on incoming metrics pipelines before routing payloads to Prometheus/Mimir remote-write exporters. - Pipelines SHOULD maintain a strict allowlist of semantic attributes using OTTL 'keep_keys' on span and metric contexts instead of relying solely on reactive denylists. ## Bad vs Good Implementations ### OTTL Transform Processor Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly batching and exporting OTLP metrics and spans without attribute filtering forwards high-cardinality values (e.g., user GUIDs, session tokens, raw parameter strings) into metric exemplar and label indices, causing catastrophic memory exhaustion in downstream TSDB clusters. ```yaml processors: batch: send_batch_size: 8192 timeout: 5s # ANTI-PATTERN: Passthrough pipelines that forward arbitrary telemetry directly # allows user.id, raw SQL queries, and dynamic query params into Prometheus/Jaeger, # causing multi-million metric cardinality explosions and remote-write OOM crashes. service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/jaeger] metrics: receivers: [otlp] processors: [batch] exporters: [prometheusremotewrite] ``` #### Production Standard (Correct) > **Standard**: Implements an OTTL transform processor that masks raw query contents, hashes account identifiers, and enforces a strict attribute allowlist on metric datapoints. High-cardinality dimensions are purged prior to reaching downstream TSDB storage while trace propagation invariants (trace_id, span_id) remain uncompromised. ```yaml processors: batch: send_batch_size: 8192 timeout: 5s transform/strip_cardinality: error_mode: ignore trace_statements: - context: span statements: # Mask sensitive and unbounded database parameters - replace_pattern(attributes["db.statement"], "(?i)(values\\s*\$).*(\$)", "$$1?$$2") # Strip high-cardinality and PII attributes from spans while retaining span identity - delete_key(attributes, "http.request.header.authorization") - delete_key(attributes, "user.id") - delete_key(attributes, "session.id") - set(attributes["account.id.hashed"], SHA256(attributes["account.id"])) where attributes["account.id"] != nil - delete_key(attributes, "account.id") metric_statements: - context: datapoint statements: # Strip dangerous dynamic attributes that mutate metric label sets - delete_key(attributes, "user_id") - delete_key(attributes, "request_id") - delete_key(attributes, "client_ip") # Restrict metric labels to safe structural set - keep_keys(attributes, ["http_status_code", "http_method", "service_name", "net_host_name"]) service: pipelines: traces: receivers: [otlp] processors: [transform/strip_cardinality, batch] exporters: [otlp/jaeger] metrics: receivers: [otlp] processors: [transform/strip_cardinality, batch] exporters: [prometheusremotewrite] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector Dynamic Semantic Attribute Stripping

OpenTelemetry Collector Dynamic Semantic Attribute Stripping defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic attribute normalization and transforms within the OpenTelemetry Collector using the transform processor and OTTL (OpenTelemetry Transformation Language). Strips unbounded, high-cardinality metadata from spans, metrics, and exemplar contexts at the ingestion tier to prevent catastrophic downstream time-series explosions across Prometheus, Mimir, and Jaeger backends without truncating trace lineage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Passing Unbounded High-Cardinality Attributes and Raw PII Direct to Downstream Time-Series Storage Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Passing Unbounded High-Cardinality Attributes and Raw PII Direct to Downstream Time-Series Storage	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS use the 'transform' processor with OTTL statements instead of broad-stroke filter processors when shedding cardinality, ensuring parent span linkages and span contexts remain intact.

Engineers MUST ALWAYS use the 'transform' processor with OTTL statements instead of broad-stroke filter processors when shedding cardinality, ensuring parent span linkages and span contexts remain intact.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.