OpenTelemetry Adaptive In-Memory Ring Buffer Tail-Sampling and High-Cardinality Sanitization

--- description: OpenTelemetry Adaptive In-Memory Ring Buffer Tail-Sampling and High-Cardinality Sanitization - Enforces strict local memory-ring buffering and cardinality pruning for OpenTelemetry traces prior to network export. Restricts unbounded attribute explosion by enforcing OTel semantic convention compliance and selectively dropping non-error trace spans when heap utilization surpasses critical thresholds. globs: src/telemetry/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/instrumentation/**/*.{ts,js}, src/middleware/tracing/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-semantic-tail-sampling-ring-guardrail" --- # OpenTelemetry Adaptive In-Memory Ring Buffer Tail-Sampling and High-Cardinality Sanitization > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded Head-Sampling with Raw Ephemeral Attribute Insertion Leading to OOM and Exporter Congestion ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Semantic Conventions and Dynamic Tail-Sampling Guardrails. You mandate zero-leakage trace processing where spans pass through bounded circular memory rings with strict attribute whitelisting, non-blocking backpressure, and runtime cardinality limits before exporter transmission. ## Strict Guidelines - Engineers MUST ALWAYS enforce strict OpenTelemetry semantic conventions (e.g., `http.request.method`, `server.address`) and reject arbitrary raw dynamic keys. - Engineers MUST NOT push unindexed, highly cardinal payload objects, UUID arrays, or unbounded string values into span attributes without deterministic truncation or hashing. - Services MUST process all outbound spans through a fixed-size ring buffer implementing adaptive tail-sampling that preserves 100% of error-tagged traces while dynamically dropping benign trace segments during memory pressure. - Telemetry pipelines MUST evaluate runtime memory pressure via `process.memoryUsage()` or dedicated buffer metrics and reject non-critical spans when heap capacity crosses safe thresholds (e.g., >= 80%). - Components NEVER block synchronous user request paths when the telemetry exporter or local ring buffer encounters saturation; backpressure MUST fail-open by shedding debug spans immediately. ## Bad vs Good Implementations ### Unbounded Dynamic Span Insertion vs. Adaptive Ring Buffer with Semantic Pruning #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly writing large JSON blobs, dynamic timestamps as keys, and raw authentication tokens introduces massive cardinality spikes and security leaks, while direct egress via default batching risks Node.js OOM crashes under high load. ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; import { BatchSpanProcessor, OTLPTraceExporter } from '@opentelemetry/sdk-trace-node'; const tracer = trace.getTracer('payment-service'); export async function handlePayment(req: any) { const span = tracer.startSpan('process-payment'); try { // ANTI-PATTERN: Injecting unredacted, dynamic, high-cardinality payloads directly span.setAttributes({ 'user.payload': JSON.stringify(req.body), 'raw.jwt.token': req.headers.authorization, [`client.meta.${Date.now()}`]: req.ip }); await executePayment(req); span.setStatus({ code: SpanStatusCode.OK }); } catch (err: any) { span.recordException(err); span.setStatus({ code: SpanStatusCode.ERROR }); throw err; } finally { span.end(); // Eager export without memory ring or tail-sampling evaluation } } ``` #### Production Standard (Correct) > **Standard**: Implements an in-memory ring buffer that evaluates system memory pressure, strictly sanitizes attribute keys to standard dot-delimited conventions, caps payload byte sizes, and guarantees critical error retention while shedding low-priority traces. ```typescript import { Span, SpanStatusCode } from '@opentelemetry/api'; import { ReadableSpan, SpanProcessor } from '@opentelemetry/sdk-trace-base'; import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions'; interface RingConfig { maxRingSize: number; maxAttributeValueLength: number; memoryHighWatermarkBytes: number; } export class AdaptiveTailSamplingProcessor implements SpanProcessor { private ring: ReadableSpan[] = []; private readonly config: RingConfig; constructor(config: RingConfig) { this.config = config; } onStart(): void {} onEnd(span: ReadableSpan): void { // INVARIANT: Always sample error traces; prune successful spans under memory pressure const isError = span.status.code === SpanStatusCode.ERROR; const currentMemory = process.memoryUsage().heapUsed; if (!isError && currentMemory > this.config.memoryHighWatermarkBytes) { return; // Drop benign span under memory backpressure } if (this.ring.length >= this.config.maxRingSize) { this.ring.shift(); // Evict oldest span from ring buffer } this.ring.push(this.sanitizeSpan(span)); } private sanitizeSpan(span: ReadableSpan): ReadableSpan { const safeAttributes: Record<string, any> = {}; for (const [key, val] of Object.entries(span.attributes)) { // INVARIANT: Restrict key naming to standard dot-notation and limit length if (/^[a-z0-9]+(\.[a-z0-9]+)*$/.test(key)) { safeAttributes[key] = typeof val === 'string' ? val.slice(0, this.config.maxAttributeValueLength) : val; } } Object.defineProperty(span, 'attributes', { value: safeAttributes }); return span; } async shutdown(): Promise<void> { this.ring = []; } async forceFlush(): Promise<void> { // Flush sanitized spans to downstream exporter this.ring = []; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Adaptive In-Memory Ring Buffer Tail-Sampling and High-Cardinality Sanitization

OpenTelemetry Adaptive In-Memory Ring Buffer Tail-Sampling and High-Cardinality Sanitization defines an industrial-grade behavioral standard for devops engineering. Enforces strict local memory-ring buffering and cardinality pruning for OpenTelemetry traces prior to network export. Restricts unbounded attribute explosion by enforcing OTel semantic convention compliance and selectively dropping non-error trace spans when heap utilization surpasses critical thresholds. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Head-Sampling with Raw Ephemeral Attribute Insertion Leading to OOM and Exporter Congestion Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Head-Sampling with Raw Ephemeral Attribute Insertion Leading to OOM and Exporter Congestion	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce strict OpenTelemetry semantic conventions (e.g., `http.request.method`, `server.address`) and reject arbitrary raw dynamic keys.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.