OpenTelemetry Exponential Histogram Dynamic Downsampling & Memory Limiter Guardrail

--- description: OpenTelemetry Exponential Histogram Dynamic Downsampling & Memory Limiter Guardrail - Enforces strict upper bounds on exponential histogram scale and bucket capacities across OpenTelemetry metric readers and collectors. During high-cardinality metric spikes or wide dynamic ranges, unbounded bucket growth exhausts heap allocations, causing agent and gateway Out-Of-Memory (OOM) crashes. Systems must implement dynamic scale downsampling, deterministic bucket limits, and proactive memory-aware backpressure. globs: **/otel/**/*.{ts,js,json,yaml,yml}, **/metrics/**/*.{ts,js,json,yaml,yml}, **/telemetry/**/*.{ts,js,json,yaml,yml}, **/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-exponential-histogram-memory-limiter-production-standard" --- # OpenTelemetry Exponential Histogram Dynamic Downsampling & Memory Limiter Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unconstrained Exponential Histogram Max Buckets and Fixed High-Scale Allocations Causing Process Out-Of-Memory (OOM) Termination ## System Constraints You are a Principal Software Architect enforcing production reliability standards for OpenTelemetry metrics aggregation. You must ensure all exponential histogram metric configurations enforce rigid memory allocation bounds (`maxSize` / `maxScale`), prevent heap saturation under high-cardinality spikes, and configure memory limiter processors with deterministic downsampling policies. ## Strict Guidelines - Engineers MUST ALWAYS enforce explicit `maxSize` (maximum bucket count) constraints when configuring `ExponentialHistogramAggregation`, never exceeding 320 buckets per histogram stream. - Engineers MUST NOT configure unbounded initial scales (`maxScale` > 8 in Node/Go SDKs) without dedicated heap budget calculations, as each unit increment doubles bucket density and exponentially increases memory consumption under high dynamic range. - Metric collection pipelines MUST configure the OpenTelemetry Collector `memory_limiter` processor as the very first processor in metric pipelines to trigger garbage collection and ballast shedding before memory ceiling breaches. - Services MUST define MeterProvider `View` rules targeting high-volume duration metrics to apply bounded exponential histogram aggregations rather than unbounded default explicit bucket arrays. - Components NEVER allocate new metric series dynamically based on unvetted, high-cardinality user inputs or ephemeral IDs inside histogram metric streams. ## Bad vs Good Implementations ### Exponential Histogram Metric Reader Configuration #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting explicit ExponentialHistogramAggregation options and View constraints causes the runtime to allocate unconstrained bucket arrays when measurements span wide orders of magnitude. Under high-throughput cardinality spikes, Node.js heap rapidly saturates, killing the process with an OOM error. ```typescript import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // ANTI-PATTERN: Default or unconstrained aggregation allows exponential // bucket growth under dynamic ranges (e.g. tracking latency spanning nanoseconds to hours). // Without explicit maxScale and maxSize bounds, high-traffic spikes trigger OOM crashes. const meterProvider = new MeterProvider(); meterProvider.addMetricReader(new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, })); export const tracer = meterProvider.getMeter('api-service'); ``` #### Production Standard (Correct) > **Standard**: Configures a View that enforces an ExponentialHistogramAggregation with a strict `maxSize` of 160 buckets and a `maxScale` of 8. When measurements exceed resolution limits, the SDK dynamically downsamples scale rather than allocating additional memory buffers, guaranteeing deterministic memory footprints. ```typescript import { MeterProvider, PeriodicExportingMetricReader, View, InstrumentType, ExponentialHistogramAggregation } from '@opentelemetry/sdk-metrics'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'; // PRODUCTION INVARIANT: Constrain maximum buckets (maxSize) and dynamic scale (maxScale). // An allocation limit of 160 buckets per direction guarantees bounded memory usage // while dynamic downsampling retains statistical distribution accuracy under spikes. const boundedHistogramAggregation = new ExponentialHistogramAggregation( 160, // maxSize: Maximum buckets for positive/negative scales (strictly budgets heap) 8 // maxScale: Upper bound on resolution to restrict exponential allocation ); export const meterProvider = new MeterProvider({ views: [ new View({ instrumentName: '*_duration_*', instrumentType: InstrumentType.HISTOGRAM, aggregation: boundedHistogramAggregation }) ] }); meterProvider.addMetricReader(new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000 })); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Exponential Histogram Dynamic Downsampling & Memory Limiter Guardrail

OpenTelemetry Exponential Histogram Dynamic Downsampling & Memory Limiter Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict upper bounds on exponential histogram scale and bucket capacities across OpenTelemetry metric readers and collectors. During high-cardinality metric spikes or wide dynamic ranges, unbounded bucket growth exhausts heap allocations, causing agent and gateway Out-Of-Memory (OOM) crashes. Systems must implement dynamic scale downsampling, deterministic bucket limits, and proactive memory-aware backpressure. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unconstrained Exponential Histogram Max Buckets and Fixed High-Scale Allocations Causing Process Out-Of-Memory (OOM) Termination Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unconstrained Exponential Histogram Max Buckets and Fixed High-Scale Allocations Causing Process Out-Of-Memory (OOM) Termination	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce explicit `maxSize` (maximum bucket count) constraints when configuring `ExponentialHistogramAggregation`, never exceeding 320 buckets per histogram stream.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.