OpenTelemetry High-Cardinality Deterministic Hash Scrubbing & Pre-Aggregation Guardrail

--- description: OpenTelemetry High-Cardinality Deterministic Hash Scrubbing & Pre-Aggregation Guardrail - Downstream telemetry backends suffer catastrophic memory exhaustion and index explosion when unbounded identifiers (UUIDs, transaction IDs, email addresses) enter metric label dimensions. This guardrail mandates edge-level deterministic hashing and pre-aggregation reservoir sampling in OpenTelemetry Collector configurations and SDK interceptors. High-cardinality values must be normalized into fixed-size hash buckets or stripped prior to TSDB ingestion. globs: **/otel-collector*.{yaml,yml}, **/collector/*.{yaml,yml}, src/**/telemetry/**/*.{ts,js,go,py}, src/**/metrics/**/*.{ts,js,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-high-cardinality-hash-scrubbing-guardrail" --- # OpenTelemetry High-Cardinality Deterministic Hash Scrubbing & Pre-Aggregation Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Injecting Raw Unbounded Identifiers Directly into Metric Dimensions Without Edge Hash-Bucketing ## System Constraints You are a Principal Observability and Infrastructure Reliability Engineer enforcing deterministic cardinality controls on OpenTelemetry pipelines. You must forbid exporting raw unbounded identity attributes into metric labels or unsampled trace spans. Mandate OpenTelemetry Transformation Language (OTTL) or SDK-level hash bucketing (e.g., modulo partitioning of cryptographically hashed IDs) and reservoir sampling at edge collection boundaries before data touches downstream TSDBs or analytical stores. ## Strict Guidelines - Engineers MUST ALWAYS sanitize or hash-bucket unbounded variables (e.g., user_id, order_id, ip_address) before using them in metric aggregations or high-frequency span attributes. - Engineers MUST NOT emit unbounded entity IDs directly as metric dimensions to prevent TSDB index explosion and OOM kill loops. - OpenTelemetry Collector pipelines MUST configure the `transform` processor with OpenTelemetry Transformation Language (OTTL) to scrub high-cardinality fields into deterministic buckets (e.g., SHA-256 slice modulo bucket count). - Tracing pipelines SHOULD use tail-based sampling or edge reservoir sampling to bound trace volume containing high-cardinality debugging attributes. - Services MUST drop raw PII and unbounded UUIDs at the edge agent layer, preserving only sampled exemplars or deterministic bucket IDs for aggregation. ## Bad vs Good Implementations ### OpenTelemetry Collector OTTL Cardinality Scrubbing Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Raw customer_id and transaction_id are propagated into metric labels without truncation or bucketing. In a high-throughput system with millions of entities, this causes Prometheus TSDB index explosion, exponential RAM growth, and eventual collector/database OOM failure. ```yaml processors: metricstransform: transforms: - include: http.server.request.duration action: update # FLUID/UNBOUNDED CARDINALITY: Directly propagating raw customer_id and transaction_id new_name: http_server_requests_per_customer pipelines: metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [prometheus] ``` #### Production Standard (Correct) > **Standard**: The OTTL transform processor scrubs unbounded identifiers (user_id, transaction_id, email) at the edge. It converts raw user IDs into a bounded deterministic hex bucket using SHA-256 truncation, guaranteeing bounded cardinality and predictable memory utilization downstream. ```yaml processors: transform: error_mode: ignore metric_statements: - context: datapoint statements: # Deterministically hash user_id to a fixed 0-99 bucket to prevent TSDB cardinality explosion - set(attributes["user_bucket"], Concat(["bucket_", Substring(SHA256(attributes["user_id"]), 0, 2)])) where attributes["user_id"] != nil - delete_key(attributes, "user_id") - delete_key(attributes, "transaction_id") - delete_key(attributes, "email") filter/cardinality: metrics: exclude: match_type: strict metric_names: - raw.debug.events pipelines: metrics: receivers: [otlp] processors: [memory_limiter, transform, filter/cardinality, batch] exporters: [prometheus] ``` ### Node.js OpenTelemetry Metric View Cardinality Restriction #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching unique transaction identifiers ('order.id') and customer identifiers to metric recordings generates a new time series for every single operation, destroying TSDB query performance and crashing downstream monitoring infrastructure. ```typescript // ANTI-PATTERN: Emitting unbounded UUID as an explicit metric attribute dimension import { Meter } from '@opentelemetry/api'; export function recordCheckoutDuration(meter: Meter, durationMs: number, orderId: string, customerId: string) { const histogram = meter.createHistogram('checkout_duration_ms'); histogram.record(durationMs, { 'order.id': orderId, // High cardinality: Unique per checkout 'customer.id': customerId, // High cardinality: Millions of customers 'region': 'us-east-1' }); } ``` #### Production Standard (Correct) > **Standard**: High-cardinality identifiers are mapped to a bounded cohort bucket via deterministic modulus hashing. The order ID is completely excluded from metric dimensions, preserving TSDB stability while maintaining aggregate statistical viability. ```typescript import { Meter } from '@opentelemetry/api'; import { createHash } from 'node:crypto'; const BUCKET_COUNT = 100; function getDeterministicBucket(identifier: string, modulus: number): number { const hash = createHash('sha256').update(identifier).digest(); return hash.readUInt16BE(0) % modulus; } export function recordCheckoutDuration(meter: Meter, durationMs: number, orderId: string, customerId: string) { const histogram = meter.createHistogram('checkout_duration_ms'); // Deterministically bucket customer for cohort analysis while strictly bounding metric cardinality const customerBucket = getDeterministicBucket(customerId, BUCKET_COUNT); histogram.record(durationMs, { 'customer.bucket': `bucket_${customerBucket}`, // Bounded to 100 series max per region 'region': 'us-east-1' // order.id is deliberately omitted from metrics; trace context handles individual lookups }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry High-Cardinality Deterministic Hash Scrubbing & Pre-Aggregation Guardrail

OpenTelemetry High-Cardinality Deterministic Hash Scrubbing & Pre-Aggregation Guardrail defines an industrial-grade behavioral standard for devops engineering. Downstream telemetry backends suffer catastrophic memory exhaustion and index explosion when unbounded identifiers (UUIDs, transaction IDs, email addresses) enter metric label dimensions. This guardrail mandates edge-level deterministic hashing and pre-aggregation reservoir sampling in OpenTelemetry Collector configurations and SDK interceptors. High-cardinality values must be normalized into fixed-size hash buckets or stripped prior to TSDB ingestion. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Injecting Raw Unbounded Identifiers Directly into Metric Dimensions Without Edge Hash-Bucketing Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Injecting Raw Unbounded Identifiers Directly into Metric Dimensions Without Edge Hash-Bucketing	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS sanitize or hash-bucket unbounded variables (e.g., user_id, order_id, ip_address) before using them in metric aggregations or high-frequency span attributes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.