OpenTelemetry Exponential Histogram Dynamic Scale Governor Standard

--- description: OpenTelemetry Exponential Histogram Dynamic Scale Governor Standard - Enforces strict bounds and dynamic downscaling on OpenTelemetry Exponential Histograms at both the SDK and OpenTelemetry Collector boundaries. Prevents catastrophic cardinality explosions and memory exhaustion in TSDB backends during severe tail-latency anomalies by constraining max bucket allocation and scale factors. globs: **/collector*.y*ml, **/otel*.y*ml, src/**/telemetry.{ts,js,go}, src/**/metrics.{ts,js,go}, src/**/instrumentation.{ts,js,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-exponential-histogram-dynamic-scale-governor" --- # OpenTelemetry Exponential Histogram Dynamic Scale Governor Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Exponential Histogram Scales Inducing OOM Panics and TSDB Write Rejection ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing strict metric cardinality and telemetry boundaries. Mandate that all OpenTelemetry Base-2 Exponential Histograms implement dynamic scale governing and hard bucket caps (max_buckets <= 160). Forbid default or unbounded scale configurations that permit metric explosion during high-latency incident states. ## Strict Guidelines - Engineers MUST ALWAYS configure an explicit `maxBuckets` limit (maximum 160 buckets per metric stream) when registering `ExponentialHistogramAggregation` in SDK Views. - Engineers MUST NOT permit unbounded or static high scale factors ($s > 4$) without automatic dynamic scale down-adaptation enabled. - Telemetry Collector pipelines MUST configure metric transform processors or filter processors to drop or dynamically clamp histogram scale factors prior to exporting to TSDB backends (e.g., Prometheus, M3DB, VictoriaMetrics). - Services MUST enforce bounded memory allocation per time-series stream by verifying bucket offsets do not span beyond configured exponential scale boundaries. - Components NEVER emit exponentially scaled histogram metrics directly to telemetry pipelines without verified downstream TSDB native histogram support. ## Bad vs Good Implementations ### SDK-Level Exponential Histogram Aggregation View Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting explicit maxBuckets and scale constraints allows the histogram to allocate dynamic buckets without upper bounds during extreme latency spikes, blowing up serialization payloads and TSDB ingest memory. ```typescript import { MeterProvider, View } from '@opentelemetry/sdk-metrics'; import { ExponentialHistogramAggregation } from '@opentelemetry/sdk-metrics'; // ANTI-PATTERN: Default or unconstrained exponential histogram. // During a major network partition or tail-latency spike, the bucket range // explodes across orders of magnitude, generating thousands of buckets // per stream and causing TSDB ingest buffer exhaustion (OOM). const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExponentialHistogramAggregation(), // Default allows excessive buckets }), ], }); ``` #### Production Standard (Correct) > **Standard**: Specifying maxBuckets (160) and a conservative maxScale (4) guarantees that the SDK dynamically reduces bucket scale resolution during latency variance, maintaining a strictly deterministic memory and network footprint. ```typescript import { MeterProvider, View } from '@opentelemetry/sdk-metrics'; import { ExponentialHistogramAggregation } from '@opentelemetry/sdk-metrics'; // PRODUCTION INVARIANT: Explicitly bounded exponential histogram governor. // maxBuckets is clamped to 160 (or TSDB threshold), and maxScale is constrained // so the SDK automatically downscales resolution factor (s -> s-1) dynamically // when extreme latency ranges occur, guaranteeing constant memory consumption. const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExponentialHistogramAggregation( 160, // maxBuckets: strict cardinality ceiling 4 // maxScale: reasonable resolution with dynamic downscaling ), }), ], }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Exponential Histogram Dynamic Scale Governor Standard

OpenTelemetry Exponential Histogram Dynamic Scale Governor Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict bounds and dynamic downscaling on OpenTelemetry Exponential Histograms at both the SDK and OpenTelemetry Collector boundaries. Prevents catastrophic cardinality explosions and memory exhaustion in TSDB backends during severe tail-latency anomalies by constraining max bucket allocation and scale factors. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Exponential Histogram Scales Inducing OOM Panics and TSDB Write Rejection Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Exponential Histogram Scales Inducing OOM Panics and TSDB Write Rejection	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure an explicit `maxBuckets` limit (maximum 160 buckets per metric stream) when registering `ExponentialHistogramAggregation` in SDK Views.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.