OpenTelemetry Exponential Histogram Dynamic Scale-Down & Memory Invariant

--- description: OpenTelemetry Exponential Histogram Dynamic Scale-Down & Memory Invariant - Enforces dynamic scale-down invariants for OpenTelemetry exponential histograms under extreme latency distributions, guaranteeing a bounded memory footprint via strict max_buckets caps while preserving sub-millisecond tail percentile accuracy. Mandates automatic scale decay and bucket collapsing rather than arbitrary sample dropping or unbounded linear bucket allocation. globs: src/**/telemetry/**/*.{ts,js,go,py}, src/**/metrics/**/*.{ts,js,go,py}, src/**/instrumentation/**/*.{ts,js,go,py} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-exponential-histogram-dynamic-scale-down" --- # OpenTelemetry Exponential Histogram Dynamic Scale-Down & Memory Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Bucket Expansion via Static High-Scale Exponential Histograms Causing OOM Kills ## System Constraints You are a Principal Software Architect enforcing production observability standards for telemetry systems. Your objective is to mandate OpenTelemetry Exponential Histogram Dynamic Scale-Down Invariants. Prevent telemetry aggregation out-of-memory crashes by strictly enforcing dynamic scale degradation, bounded bucket capacities (maxSize), and mathematically sound bucket compaction without data loss. ## Strict Guidelines - Services MUST ALWAYS enforce an upper bound on exponential histogram bucket sizes (maxSize / maxBuckets <= 320, default 160). - Telemetry systems MUST NOT deploy static high-resolution scales (scale > 3) without enabling dynamic scale-down logic when metric boundaries expand. - Engineers MUST configure exponential aggregations such that index difference (index_max - index_min + 1) exceeding maxSize triggers scale reduction by decrements of 1 until buckets merge to fit within capacity. - Code MUST NEVER drop or clamp outlier latency measurements to prevent histogram growth; dynamic scaling MUST be utilized to absorb extreme distribution variance. - Aggregation pipelines MUST preserve zero-bucket thresholds and separate positive/negative scales independently during downscale operations. ## Bad vs Good Implementations ### Configuring Exponential Histogram Aggregation with Dynamic Scale Down #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Failing to set maxSize causes unbounded bucket growth across wide latency spreads. When tail latencies span multiple orders of magnitude at scale=20, thousands of buckets are allocated per metric stream, exhausting process heap. ```typescript import { MeterProvider, View } from '@opentelemetry/sdk-metrics'; // ANTI-PATTERN: Fixed high-resolution exponential histogram without max bucket boundaries // Under extreme latency variance (e.g. 0.05ms to 60000ms), bucket count grows linearly // with scale=20, triggering severe memory leaks and collector OOM kills. const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', // @ts-expect-error - Missing maxSize guarantees unbounded memory growth aggregation: { type: 'EXPONENTIAL_HISTOGRAM', options: { maxScale: 20 // Fixed resolution, fails to downscale on high variance } } }) ] }); ``` #### Production Standard (Correct) > **Standard**: Configuring ExponentialHistogramAggregation with explicit maxSize (160) and maxScale (20) guarantees that when latency dispersion expands beyond available buckets, the SDK dynamically decrements the scale factor, halving bucket resolution through bit-shifting and keeping memory utilization strictly constant. ```typescript import { MeterProvider, View, ExponentialHistogramAggregation } from '@opentelemetry/sdk-metrics'; // PRODUCTION INVARIANT: Bounded bucket capacity with dynamic scale decay // Max buckets is capped at 160. As latency range increases, the aggregator // automatically scales down (e.g. scale 20 -> scale 6) by collapsing adjacent buckets, // bounding memory footprint to O(maxSize) while maintaining tail fidelity. const MAX_HISTOGRAM_BUCKETS = 160; const INITIAL_MAX_SCALE = 20; export const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: 'http.server.duration', aggregation: new ExponentialHistogramAggregation( MAX_HISTOGRAM_BUCKETS, // Enforces strict memory ceiling INITIAL_MAX_SCALE // High initial resolution; decays automatically on wide ranges ) }) ] }); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Exponential Histogram Dynamic Scale-Down & Memory Invariant

OpenTelemetry Exponential Histogram Dynamic Scale-Down & Memory Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces dynamic scale-down invariants for OpenTelemetry exponential histograms under extreme latency distributions, guaranteeing a bounded memory footprint via strict max_buckets caps while preserving sub-millisecond tail percentile accuracy. Mandates automatic scale decay and bucket collapsing rather than arbitrary sample dropping or unbounded linear bucket allocation. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Bucket Expansion via Static High-Scale Exponential Histograms Causing OOM Kills Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Bucket Expansion via Static High-Scale Exponential Histograms Causing OOM Kills	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST ALWAYS enforce an upper bound on exponential histogram bucket sizes (maxSize / maxBuckets <= 320, default 160).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.