OpenTelemetry In-Process Span Attribute Cardinality Governor

--- description: OpenTelemetry In-Process Span Attribute Cardinality Governor - Enforces pre-export filtering and sanitization of dynamic OpenTelemetry span attributes directly within the in-process SDK boundary. Prevents downstream TSDB (e.g., Prometheus, Cortex, ClickHouse, Mimir) index exhaustion caused by unbounded dynamic values like raw UUIDs, timestamps, and unrestricted user input injected as metric labels or indexed span attributes. globs: src/**/telemetry/**/*.{ts,js,py,go}, src/**/tracing/**/*.{ts,js,py,go}, src/**/instrumentation/**/*.{ts,js,py,go}, src/**/observability/**/*.{ts,js,py,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-span-attribute-cardinality-governor" --- # OpenTelemetry In-Process Span Attribute Cardinality Governor > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unchecked dynamic values injected as high-cardinality span attributes causing downstream TSDB/APM ingest OOM and indexing death ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing strict span attribute cardinality boundaries. Unbounded dynamic attributes (such as raw UUIDs, email addresses, order IDs, or raw query parameters) attached to traces and downstream span-metrics cause explosive index growth, high memory churn, and catastrophic outages in downstream ingestion pipelines. Mandate custom in-process SpanProcessors that enforce strict key allowlists, attribute-count limits, and deterministic regex-based or categorical sanitizers prior to OTLP batch export. ## Strict Guidelines - Engineers MUST ALWAYS implement a custom SpanProcessor (or SpanExporter wrapper) in the OpenTelemetry SDK that drops or redacts high-cardinality values before spans reach the BatchSpanProcessor or network exporter. - Engineers MUST NOT inject unbounded entity identifiers (UUIDs, transaction IDs, session tokens, arbitrary user inputs) as raw span attribute keys or unmasked values when spans are projected into downstream TSDB metric labels. - Engineers MUST configure hard limits on span attributes using `SpanLimits` (e.g., `attributeCountLimit`, `attributeValueLengthLimit`) in the TracerProvider configuration to prevent payload bloat. - SpanProcessor implementations MUST replace dynamic path segments (e.g., `/users/3fa85f64-5717-4562-b3fc-2c963f66afa6`) with parameterized templates (e.g., `/users/:id`) in `http.route` and `http.target`. - Telemetry services MUST log a rate-limited warning metric or internal diagnostic event when an attribute key is dropped due to cardinality governor policy violations. ## Bad vs Good Implementations ### In-Process Cardinality Governor SpanProcessor Implementation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching dynamic keys (`cart.item.${itemSku}`) and raw dynamic IDs without span limits or an in-process governor causes combinatorial key explosion and high cardinality. Downstream TSDBs converting span attributes to exemplar labels will experience rapid index degradation and out-of-memory errors. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { trace } from '@opentelemetry/api'; // ANTI-PATTERN: Direct exporter without cardinality protection or span limits const sdk = new NodeSDK({ spanProcessor: new BatchSpanProcessor(new OTLPTraceExporter()), }); sdk.start(); // Downstream code leaks dynamic UUIDs and unbounded keys directly const tracer = trace.getTracer('checkout-service'); export function recordCheckout(userId: string, cartId: string, itemSku: string) { const span = tracer.startSpan('process_checkout'); // DANGEROUS: High-cardinality arbitrary attributes degrade TSDB indexing span.setAttribute(`cart.item.${itemSku}`, true); span.setAttribute('user.id', userId); // Raw UUID span.setAttribute('cart.id', cartId); // Raw UUID span.end(); } ``` #### Production Standard (Correct) > **Standard**: Implements an in-process SpanProcessor before the batch exporter. It enforces a strict allowlist of attribute keys, normalizes UUIDs and numeric IDs to fixed tokens, bounds value length, and strips arbitrary dynamic keys before they leave the process memory. ```typescript import { Span, SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; import { Context } from '@opentelemetry/api'; const ALLOWED_EXACT_KEYS = new Set(['http.method', 'http.status_code', 'service.name', 'rpc.method']); const CARDINALITY_MASKS = [ { pattern: /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i, replacement: ':uuid' }, { pattern: /^\d+$/, replacement: ':id' } ]; export class CardinalityGovernorSpanProcessor implements SpanProcessor { forceFlush(): Promise<void> { return Promise.resolve(); } shutdown(): Promise<void> { return Promise.resolve(); } onStart(_span: Span, _parentContext: Context): void {} onEnd(span: ReadableSpan): void { const writeableSpan = span as unknown as { attributes: Record<string, unknown> }; const sanitized: Record<string, unknown> = {}; for (const [key, value] of Object.entries(span.attributes)) { // Reject dynamic keys if (!ALLOWED_EXACT_KEYS.has(key) && !key.startsWith('app.safe.')) { continue; } // Sanitize string values to prevent TSDB metric label explosion if (typeof value === 'string') { let safeVal = value; for (const { pattern, replacement } of CARDINALITY_MASKS) { safeVal = safeVal.replace(pattern, replacement); } sanitized[key] = safeVal.slice(0, 128); // Bound string length } else { sanitized[key] = value; } } writeableSpan.attributes = sanitized; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry In-Process Span Attribute Cardinality Governor

OpenTelemetry In-Process Span Attribute Cardinality Governor defines an industrial-grade behavioral standard for devops engineering. Enforces pre-export filtering and sanitization of dynamic OpenTelemetry span attributes directly within the in-process SDK boundary. Prevents downstream TSDB (e.g., Prometheus, Cortex, ClickHouse, Mimir) index exhaustion caused by unbounded dynamic values like raw UUIDs, timestamps, and unrestricted user input injected as metric labels or indexed span attributes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unchecked dynamic values injected as high-cardinality span attributes causing downstream TSDB/APM ingest OOM and indexing death Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unchecked dynamic values injected as high-cardinality span attributes causing downstream TSDB/APM ingest OOM and indexing death	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS implement a custom SpanProcessor (or SpanExporter wrapper) in the OpenTelemetry SDK that drops or redacts high-cardinality values before spans reach the BatchSpanProcessor or network exporter.

Engineers MUST ALWAYS implement a custom SpanProcessor (or SpanExporter wrapper) in the OpenTelemetry SDK that drops or redacts high-cardinality values before spans reach the BatchSpanProcessor or network exporter.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.