Prometheus 3.0 Native OTLP Ingestion and High-Cardinality Deduplication Guardrail

--- description: Prometheus 3.0 Native OTLP Ingestion and High-Cardinality Deduplication Guardrail - Enforces strict streaming time-windowed deduplication, label allowlisting, and exemplar normalization at the OpenTelemetry (OTLP) ingestion boundary for Prometheus 3.0. Prevents memory thrashing, TSDB head block WAL corruption, and ingestion dropouts caused by ephemeral attributes (e.g., trace IDs, customer GUIDs) masquerading as metric labels. globs: src/observability/**/*.{ts,js}, src/metrics/**/*.{ts,js}, otel-collector-config*.{yaml,yml}, prometheus*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/prometheus-v3-otlp-cardinality-dedup-standard" --- # Prometheus 3.0 Native OTLP Ingestion and High-Cardinality Deduplication Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded OTLP Resource Attribute Leakage Triggering Prometheus 3.0 TSDB Head OOM ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing Prometheus 3.0 Native OTLP Ingestion standards. Your core objective is safeguarding the TSDB from cardinality explosion and out-of-order sample drops. You mandate strict attribute sanitization, exemplar extraction for high-cardinality dimensions (trace_id, span_id, request_id), and streaming deduplication windows before metric submission to Prometheus 3.0. ## Strict Guidelines - Engineers MUST ALWAYS project high-cardinality dimensions (e.g., `trace_id`, `span_id`, `transaction_id`, `user_id`) into Exemplars rather than metric labels. - Engineers MUST NOT emit dynamic, unbounded labels (HTTP paths with raw UUIDs, email addresses, ephemeral container hashes) directly to Prometheus 3.0 OTLP endpoints. - Code SHOULD apply explicit OpenTelemetry Metric Views to drop unapproved attributes and enforce metric aggregation at the client boundary before transmission. - Services MUST configure cumulative-to-delta or delta-to-cumulative transformations consistently to prevent time-series resets and out-of-order sample rejections in Prometheus 3.0. - Components NEVER send duplicate timestamps for identical series within the same scraping or ingestion window; pipelines MUST implement streaming windowed deduplication. ## Bad vs Good Implementations ### OpenTelemetry Node.js SDK Metric View and Exemplar Normalization #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly injecting user_id, trace_id, and raw unparameterized HTTP paths into counter attributes results in an unbounded number of active time series. In Prometheus 3.0, this causes severe memory thrashing in the TSDB Head block, triggers continuous compaction loops, and leads to OOM crashes or massive sample dropouts. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; const exporter = new OTLPMetricExporter({ url: 'http://prometheus.monitoring.svc.cluster.local:9090/api/v1/otlp/v1/metrics', }); // ANTI-PATTERN: Default MeterProvider without Views or Exemplar mapping. // Dynamically records raw request-scoped identifiers directly into label dimensions. const meterProvider = new MeterProvider({ readers: [new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 5000 })], }); const meter = meterProvider.getMeter('checkout-service'); const counter = meter.createCounter('http_requests_total'); export function recordRequest(path: string, userId: string, traceId: string) { // Leaks unbounded cardinality directly into Prometheus TSDB labels! counter.add(1, { http_route: path, // e.g. /users/a3f1-4821-bc user_id: userId, trace_id: traceId, }); } ``` #### Production Standard (Correct) > **Standard**: Configures an OpenTelemetry View that strictly allowlists low-cardinality metric dimensions (route, status, method). Trace and span identifiers are passed through context and mapped to native Prometheus 3.0 exemplars instead of labels, guaranteeing bounded TSDB series while retaining full distributed trace drill-down capability. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; import { trace, context } from '@opentelemetry/api'; const exporter = new OTLPMetricExporter({ url: 'http://prometheus.monitoring.svc.cluster.local:9090/api/v1/otlp/v1/metrics', }); // PRODUCTION INVARIANT: Use Views to whitelist labels and drop unbounded dimensions const httpMetricView = new View({ meterName: 'checkout-service', instrumentName: 'http_requests_total', attributeKeys: ['http_route', 'http_status_code', 'http_method'], }); const meterProvider = new MeterProvider({ views: [httpMetricView], readers: [ new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 15000, }), ], }); const meter = meterProvider.getMeter('checkout-service'); const counter = meter.createCounter('http_requests_total'); export function recordRequest(parameterizedRoute: string, statusCode: number, method: string) { const activeSpan = trace.getSpan(context.active()); const spanContext = activeSpan?.spanContext(); // High-cardinality trace context automatically attached as Exemplar via SDK context, // while metric labels remain bounded and normalized. counter.add( 1, { http_route: parameterizedRoute, // e.g., '/users/:id' http_status_code: statusCode.toString(), http_method: method, }, context.active() ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Prometheus 3.0 Native OTLP Ingestion and High-Cardinality Deduplication Guardrail

Prometheus 3.0 Native OTLP Ingestion and High-Cardinality Deduplication Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict streaming time-windowed deduplication, label allowlisting, and exemplar normalization at the OpenTelemetry (OTLP) ingestion boundary for Prometheus 3.0. Prevents memory thrashing, TSDB head block WAL corruption, and ingestion dropouts caused by ephemeral attributes (e.g., trace IDs, customer GUIDs) masquerading as metric labels. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded OTLP Resource Attribute Leakage Triggering Prometheus 3.0 TSDB Head OOM Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded OTLP Resource Attribute Leakage Triggering Prometheus 3.0 TSDB Head OOM	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS project high-cardinality dimensions (e.g., `trace_id`, `span_id`, `transaction_id`, `user_id`) into Exemplars rather than metric labels.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.