OpenTelemetry Trace-Correlated Metric Cardinality Governor and Exemplar Safeguard

--- description: OpenTelemetry Trace-Correlated Metric Cardinality Governor and Exemplar Safeguard - Enforces deterministic metric cardinality budgets across all OpenTelemetry telemetry pipelines to protect time-series databases (TSDBs) from memory exhaustion during traffic anomalies. Dynamically normalizes or drops volatile labels (such as customer IDs, UUIDs, or raw query parameters) while systematically capturing trace context via exemplars on Prometheus and OTLP exporters. globs: src/**/telemetry/**/*.{ts,js}, src/**/metrics/**/*.{ts,js}, src/**/monitoring/**/*.{ts,js}, src/**/*telemetry*.{ts,js}, src/**/*metric*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-cardinality-governor-exemplar-preservation" --- # OpenTelemetry Trace-Correlated Metric Cardinality Governor and Exemplar Safeguard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Metric Attribute Pollution Causing TSDB Out-of-Memory Crashing Under Peak Traffic ## System Constraints You are a Principal Observability and Site Reliability Engineer enforcing OpenTelemetry metric cardinality governance. Uncontrolled time-series explosion destroys TSDB stability (Prometheus, Cortex, M3DB) during traffic spikes and incident investigations. You must mandate strict dynamic attribute shedding for high-cardinality values, route volatile entity identifiers into trace spans or metric exemplars, and enforce cardinality limiters at instrument or meter registration boundaries. ## Strict Guidelines - Engineers MUST NEVER inject dynamic, user-generated, or unbounded values (such as user IDs, session tokens, tenant IDs, email addresses, order IDs, or raw URL paths) as raw Metric attributes. - Engineers MUST capture fine-grained transactional identifiers inside OpenTelemetry Spans and attach current trace/span IDs as exemplars to aggregated metric recordings. - Metrics pipelines MUST configure explicit cardinality limiting processors (e.g., View aggregators with attribute filter allowlists or bounded LRU label caches) that clamp unexpected labels into a fallback value such as '__other__'. - Telemetry modules MUST utilize route templates (e.g., '/api/v1/users/:id') rather than literal resolved request URLs (e.g., '/api/v1/users/12839123') when recording HTTP metrics. - Exemplar filters MUST be explicitly bound to TraceContext (such as TraceContextExemplarFilter or AlwaysOnExemplarFilter) to ensure correlation survives attribute shedding. ## Bad vs Good Implementations ### Preventing Unbounded Metric Attributes via Exemplar Preservation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly injecting raw entity IDs (`customerId`, `invoiceId`) into metric label attributes causes unbounded time-series explosion (high cardinality), leading to TSDB OOM crashes under high traffic. ```typescript import { metrics, trace, context } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service'); const invoiceCounter = meter.createCounter('invoices_processed_total', { description: 'Tracks processed invoices', }); export function processInvoice(invoiceId: string, customerId: string, amount: number) { // CRITICAL BUG: Injecting customerId and invoiceId creates unbounded metric series. // Under peak anomaly or retry loops, this triggers a TSDB Out-Of-Memory (OOM) event. invoiceCounter.add(1, { 'customer.id': customerId, 'invoice.id': invoiceId, 'currency': 'USD' }); } ``` #### Production Standard (Correct) > **Standard**: High-cardinality attributes (`customerId`, `invoiceId`) are moved exclusively to the active Span. Metric attributes are clamped to a deterministic, low-cardinality set (`tier`, `currency`, `status`). The active context is passed to the metric `.add()` call, allowing the exporter to preserve trace-correlated exemplars without expanding TSDB series. ```typescript import { metrics, trace, context, SpanStatusCode } from '@opentelemetry/api'; const meter = metrics.getMeter('billing-service'); const tracer = trace.getTracer('billing-service'); const invoiceCounter = meter.createCounter('invoices_processed_total', { description: 'Tracks processed invoices with bounded cardinality', }); const ALLOWED_TIERS = new Set(['enterprise', 'pro', 'free']); export async function processInvoice( invoiceId: string, customerId: string, tier: string, amount: number ): Promise<void> { return tracer.startActiveSpan('processInvoice', async (span) => { try { // High-cardinality values are safely isolated to trace span attributes span.setAttribute('customer.id', customerId); span.setAttribute('invoice.id', invoiceId); span.setAttribute('invoice.amount', amount); // Cardinality Governor: Enforce strict bounded label allowlist const safeTier = ALLOWED_TIERS.has(tier) ? tier : '__other__'; // TraceContext is implicitly captured via exemplar filters on the active context invoiceCounter.add(1, { 'customer.tier': safeTier, 'currency': 'USD', 'status': 'success' }, context.active()); span.setStatus({ code: SpanStatusCode.OK }); } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR }); invoiceCounter.add(1, { 'customer.tier': 'unknown', 'currency': 'USD', 'status': 'error' }, context.active()); throw error; } finally { span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Trace-Correlated Metric Cardinality Governor and Exemplar Safeguard

OpenTelemetry Trace-Correlated Metric Cardinality Governor and Exemplar Safeguard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic metric cardinality budgets across all OpenTelemetry telemetry pipelines to protect time-series databases (TSDBs) from memory exhaustion during traffic anomalies. Dynamically normalizes or drops volatile labels (such as customer IDs, UUIDs, or raw query parameters) while systematically capturing trace context via exemplars on Prometheus and OTLP exporters. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Metric Attribute Pollution Causing TSDB Out-of-Memory Crashing Under Peak Traffic Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Metric Attribute Pollution Causing TSDB Out-of-Memory Crashing Under Peak Traffic	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NEVER inject dynamic, user-generated, or unbounded values (such as user IDs, session tokens, tenant IDs, email addresses, order IDs, or raw URL paths) as raw Metric attributes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.