OpenTelemetry Metric Cardinality Budgeting and Exemplar Correlation Standard

--- description: OpenTelemetry Metric Cardinality Budgeting and Exemplar Correlation Standard - Enforces strict metric attribute sanitization and dynamic cardinality bounding to prevent TSDB out-of-memory (OOM) failures. Ensures high-cardinality identifiers (UUIDs, raw URLs, user IDs) are stripped from metric label sets while deterministically retaining distributed trace context via OpenTelemetry Exemplar reservoirs on high-latency buckets. globs: src/**/*.{ts,js,py,go}, lib/telemetry/**/*.{ts,js,go}, config/otel/**/*.{yaml,json} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-cardinality-budget-exemplar-guardrail" --- # OpenTelemetry Metric Cardinality Budgeting and Exemplar Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Injecting Unbounded High-Cardinality Dimensions into Metric Labels Causing TSDB Meltdowns ## System Constraints You are a Principal Observability and Systems Architect enforcing strict OpenTelemetry Metric Cardinality Budgets and Exemplar Correlation. Unbounded metric labels exhaust TSDB memory, crash Prometheus/Mimir clusters, and inflate observability costs. You must reject any metric recording that injects user IDs, UUIDs, trace IDs, query strings, or raw URLs as metric label keys. Mandate parameterized routes and label whitelisting via MeterProvider Views, and enforce the passing of active OpenTelemetry Context so trace context is captured strictly as Exemplars. ## Strict Guidelines - Engineers MUST ALWAYS sanitize dimensions by mapping raw URLs and dynamic parameters into static route templates (e.g., '/orders/:orderId' instead of '/orders/9b1deb4d'). - Engineers MUST NOT inject unbounded identifiers (such as user_id, order_id, email, IP address, request_id, or trace_id) into metric label attributes. - Services MUST configure MeterProvider Views with an explicit attribute whitelist or an allowlist filter to drop ad-hoc unbounded dimensions before export. - Instrumentation MUST pass the active OpenTelemetry Context into `.record()` calls, allowing configured Exemplar Reservoirs (e.g., AlignedHistogramBucketExemplarReservoir) to capture trace context without generating metric series. - Histogram metrics SHOULD explicitly define low-cardinality explicit bucket boundaries tailored to service SLOs, avoiding default unbounded microsecond-level allocations. ## Bad vs Good Implementations ### HTTP Latency Metric Instrumentation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Recording high-cardinality attributes like raw URLs, user IDs, and trace IDs creates a new time series in the TSDB for every unique combination, resulting in an OOM crash of metric storage engines. ```typescript import { metrics, trace } from '@opentelemetry/api'; import { Request, Response } from 'express'; const meter = metrics.getMeter('http-service'); const latencyHistogram = meter.createHistogram('http_server_duration_ms'); export function recordHttpMetrics(req: Request, res: Response, durationMs: number): void { const activeSpan = trace.getActiveSpan(); // BAD: Injects raw URL, UUIDs, User ID, and Trace ID into metric labels. // This causes infinite TSDB series creation and memory exhaustion. latencyHistogram.record(durationMs, { path: req.originalUrl, // e.g. /api/v1/users/550e8400-e29b-41d4-a716-446655440000 userId: req.header('x-user-id') || 'anonymous', statusCode: res.statusCode.toString(), traceId: activeSpan?.spanContext().traceId || 'none' // Anti-pattern: traceId in labels }); } ``` #### Production Standard (Correct) > **Standard**: Normalizes dynamic path parameters to route templates, bounds labels to static enumerations, and passes the active OpenTelemetry context so exemplars attach trace identifiers without TSDB series explosion. ```typescript import { metrics, context } from '@opentelemetry/api'; import { Request, Response } from 'express'; const meter = metrics.getMeter('http-service'); // Histograms should use bounded bucket configurations registered via MeterProvider Views const latencyHistogram = meter.createHistogram('http_server_duration_ms', { description: 'Measures inbound HTTP request duration in milliseconds', unit: 'ms' }); export function recordHttpMetrics(req: Request, res: Response, durationMs: number): void { // GOOD: High-cardinality variables are masked; static route template is used. const matchedRoute = req.route?.path ? `${req.baseUrl}${req.route.path}` : 'UNMATCHED_ROUTE'; // GOOD: Trace context is passed via context.active(). The OpenTelemetry Exemplar reservoir // automatically correlates trace_id/span_id to histogram buckets without creating extra series. latencyHistogram.record( durationMs, { http_route: matchedRoute, http_method: req.method, http_status_code: res.statusCode.toString() }, context.active() ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Metric Cardinality Budgeting and Exemplar Correlation Standard

OpenTelemetry Metric Cardinality Budgeting and Exemplar Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict metric attribute sanitization and dynamic cardinality bounding to prevent TSDB out-of-memory (OOM) failures. Ensures high-cardinality identifiers (UUIDs, raw URLs, user IDs) are stripped from metric label sets while deterministically retaining distributed trace context via OpenTelemetry Exemplar reservoirs on high-latency buckets. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Injecting Unbounded High-Cardinality Dimensions into Metric Labels Causing TSDB Meltdowns Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Injecting Unbounded High-Cardinality Dimensions into Metric Labels Causing TSDB Meltdowns	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS sanitize dimensions by mapping raw URLs and dynamic parameters into static route templates (e.g., '/orders/

Engineers MUST ALWAYS sanitize dimensions by mapping raw URLs and dynamic parameters into static route templates (e.g., '/orders/:orderId' instead of '/orders/9b1deb4d').

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.