OpenTelemetry High-Throughput Span Attribute Cardinality Throttling Governor

--- description: OpenTelemetry High-Throughput Span Attribute Cardinality Throttling Governor - Enforces strict bounds and dynamic bucketing on span attribute key-value pairs across high-throughput microsecond RPC paths. Unchecked high-cardinality metadata (e.g., raw UUIDs, unbounded customer inputs, arbitrary query params) causes exponential memory growth and downstream time-series backend crashes. Implements pre-allocated attribute filtering, LRU-bounded dimension clamping, and fallback rollups directly within the OpenTelemetry SpanProcessor lifecycle. globs: src/telemetry/**/*.ts, src/tracing/**/*.ts, src/observability/**/*.ts, src/middleware/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-span-cardinality-governor" --- # OpenTelemetry High-Throughput Span Attribute Cardinality Throttling Governor > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Injecting Dynamic Unbounded Identifiers Directly into OpenTelemetry Span Attributes ## System Constraints You are a Principal Observability Architect enforcing strict memory and cardinality guardrails on distributed tracing pipelines. You must prevent telemetry memory leaks and backend ingestion rejection by mandating deterministic attribute sanitization, maximum attribute count quotas, and fixed-capacity cardinality bucketing across all OpenTelemetry spans. ## Strict Guidelines - Engineers MUST ALWAYS sanitize dynamic runtime values into low-cardinality buckets or bounded categorical classifications before attaching them as span attributes. - Engineers MUST NOT inject raw user identifiers, dynamic session IDs, random nonces, or unbounded query strings into span attribute keys or values; these MUST be routed exclusively to trace baggage, span links, or log events. - Services MUST configure a custom OpenTelemetry `SpanProcessor` or wrap the Tracer to enforce hard bounds on `attributeCountLimit` (maximum 32 keys per span) and `attributeValueLengthLimit` (maximum 256 bytes). - Components NEVER attach attributes dynamically inside hot microsecond loops without a thread-safe LRU cardinality limiter or static enum whitelist validation. - Tracer configurations MUST drop or rollup unknown attributes exceeding the cardinality threshold to an `overflow: true` bucket rather than rejecting or accumulating unbounded memory. ## Bad vs Good Implementations ### Unbounded Attribute Injection vs Governed Cardinality Span Processor #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Attaching unique UUIDs, dynamic key names, and variable-length payloads to span attributes causes cardinality explosion in storage backends (e.g., Tempo, Jaeger, ClickHouse) and causes Node.js heap exhaustion under high-throughput RPC loads. ```typescript import { trace, Span } from '@opentelemetry/api'; const tracer = trace.getTracer('payment-rpc-tracer'); export async function handleRpcRequest(req: { txId: string; userId: string; rawPayload: string }): Promise<void> { // Anti-Pattern: Unchecked span attributes directly bind high-cardinality // random UUIDs, raw payloads, and ad-hoc dimension keys directly on hot RPC paths. const span = tracer.startSpan('process_payment'); try { span.setAttribute('rpc.tx_id', req.txId); // Unique per microsecond RPC call span.setAttribute('rpc.user_id', req.userId); // Millions of distinct keys span.setAttribute(`debug.${req.txId}`, req.rawPayload); // Dynamic attribute keys create schema explosion await executePayment(req); } finally { span.end(); } } async function executePayment(_: unknown): Promise<void> {} ``` #### Production Standard (Correct) > **Standard**: Implements an in-process governor via an OpenTelemetry SpanProcessor. It clamps total distinct attribute keys, truncates oversized values, and marks dropped telemetry with an overflow indicator to safeguard downstream tracing aggregators. ```typescript import { Span, SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; import { Context, AttributeValue } from '@opentelemetry/api'; const MAX_KEY_CARDINALITY = 500; const MAX_ATTR_VALUE_LENGTH = 128; export class CardinalityThrottlingSpanProcessor implements SpanProcessor { private readonly observedKeys = new Set<string>(); onStart(span: Span, _context: Context): void { // Mutate span attributes dynamically or wrap setters with strict enforcement const originalSetAttribute = span.setAttribute.bind(span); span.setAttribute = (key: string, value: AttributeValue): Span => { // Enforce low-cardinality keys and cap attribute explosion if (!this.observedKeys.has(key)) { if (this.observedKeys.size >= MAX_KEY_CARDINALITY) { originalSetAttribute('cardinality.governor.dropped', true); return span; } this.observedKeys.add(key); } // Truncate high-length string values to prevent memory spikes let sanitizedValue = value; if (typeof value === 'string' && value.length > MAX_ATTR_VALUE_LENGTH) { sanitizedValue = `${value.substring(0, MAX_ATTR_VALUE_LENGTH)}...[TRUNCATED]`; } return originalSetAttribute(key, sanitizedValue); }; } onEnd(_span: ReadableSpan): void {} async shutdown(): Promise<void> { this.observedKeys.clear(); } async forceFlush(): Promise<void> {} } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry High-Throughput Span Attribute Cardinality Throttling Governor

OpenTelemetry High-Throughput Span Attribute Cardinality Throttling Governor defines an industrial-grade behavioral standard for devops engineering. Enforces strict bounds and dynamic bucketing on span attribute key-value pairs across high-throughput microsecond RPC paths. Unchecked high-cardinality metadata (e.g., raw UUIDs, unbounded customer inputs, arbitrary query params) causes exponential memory growth and downstream time-series backend crashes. Implements pre-allocated attribute filtering, LRU-bounded dimension clamping, and fallback rollups directly within the OpenTelemetry SpanProcessor lifecycle. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Injecting Dynamic Unbounded Identifiers Directly into OpenTelemetry Span Attributes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Injecting Dynamic Unbounded Identifiers Directly into OpenTelemetry Span Attributes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS sanitize dynamic runtime values into low-cardinality buckets or bounded categorical classifications before attaching them as span attributes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.