OTTL Cardinality Fencing and Dynamic Path Normalization

--- description: OTTL Cardinality Fencing and Dynamic Path Normalization - Enforces boundary-level telemetry sanitation using OpenTelemetry Transformation Language (OTTL) to scrub unbounded REST paths, numeric identifiers, UUIDs, and GraphQL variable payloads. Prevents downstream TSDB (e.g., Prometheus, M3DB, VictoriaMetrics) indexing degradation and OOM panics caused by high-cardinality metric label explosions. globs: **/otel-collector*.yaml, **/otel-collector*.yml, **/collector*.yaml, **/collector*.yml, **/conf/otel/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ottl-cardinality-fencing" --- # OTTL Cardinality Fencing and Dynamic Path Normalization > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Leaking Unparameterized URI Paths and Unbounded Query Strings Directly into TSDB Metric Dimensions ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Collector transformation standards. You mandate deterministic cardinality fencing at edge collectors using OTTL. Any OpenTelemetry Collector pipeline exporting metrics or span-derived metrics to a TSDB must scrub dynamic route segments, enforce key/value truncation, and redact unbounded parameters before downstream ingestion. ## Strict Guidelines - Engineers MUST ALWAYS implement OTTL transform processors at the collector boundary to scrub UUIDs, numeric IDs, and query strings from http.route, http.target, and url.path attributes. - Engineers MUST NOT allow unparameterized raw client request URIs (e.g., /api/v1/users/58291/checkout) to be converted directly into metric dimensions or Prometheus labels. - Services MUST configure the OTTL transform processor with error_mode: ignore to guarantee non-blocking pipeline delivery even on malformed attribute representations. - Collector pipelines SHOULD enforce attribute key counts and value length fencing using OTTL truncate_all and keep_keys statements to strictly cap dimension allocations. - Components NEVER export GraphQL operation queries containing inline literals; collectors MUST extract and retain only the sanitized operation name and normalized field path. ## Bad vs Good Implementations ### OTTL Path and Parameter Normalization Pipeline #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Raw request paths containing dynamic identifiers (such as UUIDs, hashes, and integer keys) flow unfiltered to Prometheus/M3DB. This generates millions of short-lived series, leading to extreme heap usage, frequent garbage collector stalls, and eventual catastrophic TSDB crash loops. ```yaml processors: batch: send_batch_size: 8192 timeout: 5s # ANTI-PATTERN: Emitting raw http.target or url.path directly to downstream metric scrapers # Every distinct user ID or resource ID creates a distinct time-series dimension service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [prometheus, otlp/tempo] ``` #### Production Standard (Correct) > **Standard**: Transforms raw request parameters into deterministic, bounded string templates ({id}, {uuid}, {object_id}) and enforces attribute length truncation before batching and export, guaranteeing flat cardinality curves across metrics stores. ```yaml processors: transform/cardinality_fence: error_mode: ignore trace_statements: - context: span statements: # Strip dynamic query strings - replace_pattern(attributes["http.target"], "\\?.*$", "") - replace_pattern(attributes["url.path"], "\\?.*$", "") # Normalize canonical UUIDs - replace_pattern(attributes["http.route"], "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}") - replace_pattern(attributes["http.target"], "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}") # Normalize variable integer and Mongo/BSON hex identifiers - replace_pattern(attributes["http.route"], "/[0-9]+(?=/|$)", "/{id}") - replace_pattern(attributes["http.route"], "/[0-9a-fA-F]{24}(?=/|$)", "/{object_id}") # Ensure bounded length on fallbacks - truncate_all(attributes, 128) metric_statements: - context: datapoint statements: - replace_pattern(attributes["http.route"], "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "{uuid}") - replace_pattern(attributes["http.route"], "/[0-9]+(?=/|$)", "/{id}") - truncate_all(attributes, 64) service: pipelines: traces: receivers: [otlp] processors: [transform/cardinality_fence, batch] exporters: [prometheus, otlp/tempo] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OTTL Cardinality Fencing and Dynamic Path Normalization

OTTL Cardinality Fencing and Dynamic Path Normalization defines an industrial-grade behavioral standard for devops engineering. Enforces boundary-level telemetry sanitation using OpenTelemetry Transformation Language (OTTL) to scrub unbounded REST paths, numeric identifiers, UUIDs, and GraphQL variable payloads. Prevents downstream TSDB (e.g., Prometheus, M3DB, VictoriaMetrics) indexing degradation and OOM panics caused by high-cardinality metric label explosions. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Leaking Unparameterized URI Paths and Unbounded Query Strings Directly into TSDB Metric Dimensions Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Leaking Unparameterized URI Paths and Unbounded Query Strings Directly into TSDB Metric Dimensions	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement OTTL transform processors at the collector boundary to scrub UUIDs, numeric IDs, and query strings from http.route, http.target, and url.path attributes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.