Prometheus 3.0 & OpenTelemetry Native Exponential Histogram Bucketing Alignment

--- description: Prometheus 3.0 & OpenTelemetry Native Exponential Histogram Bucketing Alignment - Mandates zero-loss exponential histogram configurations across OpenTelemetry SDKs, OpenTelemetry Collectors, and Prometheus 3.0 scrapers. Prevents tail-latency estimation errors and metric cardinality explosion by enforcing synchronized scale parameters, zero thresholds, and native histogram transmission across all pipeline stages. globs: **/*otel*collector*.{yaml,yml}, **/otel*.{yaml,yml,ts,go,java}, **/prometheus*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-prometheus-native-exponential-histogram-alignment" --- # Prometheus 3.0 & OpenTelemetry Native Exponential Histogram Bucketing Alignment > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Down-converting high-dynamic-range exponential histograms into legacy explicit buckets or misaligning OTel scale and max-buckets across collector pipelines ## System Constraints You are a Principal Software Architect enforcing Prometheus 3.0 and OpenTelemetry Exponential Histogram alignment. You must strictly mandate that telemetry pipelines leverage Prometheus native histograms (exponential bucketing schema) and prevent any down-conversion to legacy explicit bucket counts. Enforce consistent scale parameters (maximum scale: 8 to 20), consistent bucket limits (max_buckets: 160), explicit zero-threshold definitions, and ensure that all exporter components set `send_native_histograms: true`. ## Strict Guidelines - Engineers MUST ALWAYS configure OpenTelemetry Collector exporters (`prometheus`, `otlphttp`) with `send_native_histograms: true` when routing to Prometheus 3.0+ endpoints. - Engineers MUST NOT convert Exponential Histograms to legacy explicit bucket histograms via collector processors or SDK aggregators for multi-order-of-magnitude latency metrics. - Services MUST configure OpenTelemetry SDK views with `ExplicitBucketHistogramAggregation` explicitly replaced by `ExponentialHistogramAggregation` with a maximum scale of at least 8 (recommended 8-12) and a max bucket count limit of 160. - Collector pipelines MUST define an explicit `zero_threshold` (e.g., `1e-9` or `1e-4` depending on unit) to ensure zero-bucket stability and avoid floating-point bucket oscillations near zero. - Components NEVER emit custom high-granularity explicit bucket bounds (e.g., >30 linear/log buckets) where native exponential histograms are supported by the target TSDB. ## Bad vs Good Implementations ### OpenTelemetry Collector and Prometheus Pipeline Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Forcing conversion to fixed explicit buckets destroys dynamic range resolution, introduces quantization errors in tail percentiles, and generates numerous separate time series for every bucket boundary instead of using Prometheus 3.0 compact sparse native histogram chunks. ```yaml # Anti-pattern: Falling back to explicit bucketing and dropping native histogram schemas receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: transform: error_mode: ignore metric_statements: - context: metric statements: # Anti-pattern: down-converting exponential histograms causes cardinality explosion # and destroys tail-latency precision (P99/P99.9 skew) - convert_exponential_to_explicit_histogram([0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]) exporters: prometheus: endpoint: "0.0.0.0:8889" # Missing send_native_histograms emits only legacy _bucket series with high cardinality otlphttp/prometheus: endpoint: "http://prometheus:9090/api/v1/otlp" ``` #### Production Standard (Correct) > **Standard**: Configuring send_native_histograms directly forwards OpenTelemetry exponential histogram layouts into Prometheus 3.0 native histograms without converting to legacy explicit buckets, retaining exact dynamic resolution at sub-10% bucket widths and cutting metric storage costs by up to 80%. ```yaml # Production Invariant: Zero-Loss Native Exponential Histogram Pipeline receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: send_batch_size: 8192 timeout: 5s exporters: prometheus: endpoint: "0.0.0.0:8889" namespace: "prod" # Mandate native histogram streaming for Prometheus 3.0 scraping send_native_histograms: true otlphttp/prometheus: endpoint: "http://prometheus:9090/api/v1/otlp" metrics: histograms: send_native_histograms: true service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [prometheus, otlphttp/prometheus] ``` ### OpenTelemetry SDK Metric Reader / View Configuration (TypeScript) #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Default SDK histograms use fixed, legacy explicit bucket boundaries (e.g., [0, 5, 10, 25, 50, ...]), causing severe tail truncation on latencies beyond the max boundary and wasting memory across service endpoints. ```typescript // Anti-pattern: SDK default explicit boundaries causing high-cardinality skew import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; const meterProvider = new MeterProvider(); meterProvider.addMetricReader(new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, // Uses default ExplicitBucketHistogramAggregation with legacy boundaries })); ``` #### Production Standard (Correct) > **Standard**: Configuring ExponentialHistogramAggregation with maxScale=8 and maxBuckets=160 produces standard exponential buckets directly convertible into Prometheus native histograms without data loss or cardinality explosion. ```typescript // Production Invariant: Standardized Exponential Aggregation with Scale and MaxBuckets import { MeterProvider, PeriodicExportingMetricReader, View, InstrumentType, ExponentialHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; const meterProvider = new MeterProvider({ views: [ new View({ instrumentType: InstrumentType.HISTOGRAM, aggregation: new ExponentialHistogramAggregation( 160, // maxBuckets: prevents unbounded memory allocations 8 // maxScale: guarantees <= 2^(2^-8) ≈ 0.27% resolution width ), }), ], }); meterProvider.addMetricReader(new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, })); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Prometheus 3.0 & OpenTelemetry Native Exponential Histogram Bucketing Alignment

Prometheus 3.0 & OpenTelemetry Native Exponential Histogram Bucketing Alignment defines an industrial-grade behavioral standard for devops engineering. Mandates zero-loss exponential histogram configurations across OpenTelemetry SDKs, OpenTelemetry Collectors, and Prometheus 3.0 scrapers. Prevents tail-latency estimation errors and metric cardinality explosion by enforcing synchronized scale parameters, zero thresholds, and native histogram transmission across all pipeline stages. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Down-converting high-dynamic-range exponential histograms into legacy explicit buckets or misaligning OTel scale and max-buckets across collector pipelines Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Down-converting high-dynamic-range exponential histograms into legacy explicit buckets or misaligning OTel scale and max-buckets across collector pipelines	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure OpenTelemetry Collector exporters (`prometheus`, `otlphttp`) with `send_native_histograms

Engineers MUST ALWAYS configure OpenTelemetry Collector exporters (`prometheus`, `otlphttp`) with `send_native_histograms: true` when routing to Prometheus 3.0+ endpoints.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.