Deterministic Span-Event Throttling and Memory Circuit-Breakers for Telemetry Pipelines

--- description: Deterministic Span-Event Throttling and Memory Circuit-Breakers for Telemetry Pipelines - Enforces strict cardinality ceilings, attribute size truncation, and memory circuit-breakers on structured OpenTelemetry span events. Prevents client-side heap exhaustion and cascading collector Out-Of-Memory (OOM) crashes during severe telemetry volume bursts and recursive error loops. globs: src/telemetry/**/*.{ts,js}, src/observability/**/*.{ts,js}, src/infrastructure/tracing/**/*.{ts,js}, **/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-span-event-backpressure-circuit-breaker" --- # Deterministic Span-Event Throttling and Memory Circuit-Breakers for Telemetry Pipelines > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unbounded span.addEvent invocations within tight loops emitting deep unindexed objects without backpressure ## System Constraints You are a Principal Observability and Infrastructure Architect enforcing deterministic backpressure and cardinality controls for OpenTelemetry span events. You must mandate strict event quotas per span, string truncation, memory threshold circuit-breakers, and safe batching primitives to ensure telemetry infrastructure never induces system instability or collector OOM failures. ## Strict Guidelines - Engineers MUST ALWAYS enforce strict hard caps on the total number of span events allowed per individual trace span (maximum 64 events per span). - Engineers MUST NOT serialize raw arbitrary objects, unbounded error stacks, or raw network payloads directly into span event attributes. - Code MUST truncate any string attribute value to a deterministic threshold (maximum 1024 characters) and reject dynamically generated or unindexed attribute key names. - Services MUST evaluate process heap allocation via a deterministic circuit breaker; telemetry events MUST be dropped immediately if available heap headroom falls below 15%. - Components NEVER allow span event submission to block or throw exceptions back into application execution paths under export backpressure. ## Bad vs Good Implementations ### High-Cardinality Unbounded Span Events vs. Circuit-Broken Rate-Limited Telemetry Guardrail #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Calling span.addEvent inside large loops without size constraints, event limits, or dynamic key sanitization causes exponential collector memory consumption. If item payloads are large or iterate thousands of times, the trace buffer will exhaust client memory or crash upstream OpenTelemetry collector nodes. ```typescript import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('payment-service'); export async function processBatchItems(items: Array<Record<string, unknown>>) { return tracer.startActiveSpan('processBatchItems', async (span) => { for (const [index, item] of items.entries()) { // CRITICAL DEFECT: Unbounded event emission in loop with raw payload // Under 10,000 items, this generates massive payload arrays, causing collector OOM span.addEvent('item_processed', { index, rawItem: JSON.stringify(item), timestamp: Date.now(), dynamicMetricKey: `user_metric_${item.userId}` }); } span.end(); }); } ``` #### Production Standard (Correct) > **Standard**: Implements an in-memory circuit-breaker that inspects heap pressure before allocating telemetry buffers. It enforces deterministic limits: a hard limit of 64 events per span, fixed semantic attribute keys, and strict string truncation to eliminate collector heap pressure and high-cardinality crashes. ```typescript import { trace, Span, Attributes } from '@opentelemetry/api'; import v8 from 'v8'; const tracer = trace.getTracer('payment-service'); const MAX_EVENTS_PER_SPAN = 64; const MAX_ATTR_STRING_LENGTH = 1024; const MEMORY_EXHAUSTION_THRESHOLD_RATIO = 0.85; class SpanEventGuardrail { private static eventCounters = new WeakMap<Span, number>(); private static isMemoryPressureCritical(): boolean { const stats = v8.getHeapStatistics(); return stats.used_heap_size / stats.heap_size_limit > MEMORY_EXHAUSTION_THRESHOLD_RATIO; } public static addSafeEvent(span: Span, name: string, attributes?: Attributes): boolean { if (this.isMemoryPressureCritical()) { return false; // Circuit-breaker tripped: drop telemetry to protect process stability } const currentCount = this.eventCounters.get(span) ?? 0; if (currentCount >= MAX_EVENTS_PER_SPAN) { if (currentCount === MAX_EVENTS_PER_SPAN) { span.setAttribute('telemetry.events_dropped', true); this.eventCounters.set(span, currentCount + 1); } return false; } const sanitizedAttributes: Attributes = {}; if (attributes) { for (const [key, value] of Object.entries(attributes)) { if (typeof value === 'string') { sanitizedAttributes[key] = value.slice(0, MAX_ATTR_STRING_LENGTH); } else if (typeof value === 'number' || typeof value === 'boolean') { sanitizedAttributes[key] = value; } } } span.addEvent(name, sanitizedAttributes); this.eventCounters.set(span, currentCount + 1); return true; } } export async function processBatchItems(items: Array<Record<string, unknown>>) { return tracer.startActiveSpan('processBatchItems', async (span) => { try { for (const [index, item] of items.entries()) { const safeUserId = typeof item.userId === 'string' ? item.userId : 'unknown'; SpanEventGuardrail.addSafeEvent(span, 'item_processed', { 'batch.index': index, 'user.id': safeUserId }); } } finally { span.end(); } }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic Span-Event Throttling and Memory Circuit-Breakers for Telemetry Pipelines

Deterministic Span-Event Throttling and Memory Circuit-Breakers for Telemetry Pipelines defines an industrial-grade behavioral standard for devops engineering. Enforces strict cardinality ceilings, attribute size truncation, and memory circuit-breakers on structured OpenTelemetry span events. Prevents client-side heap exhaustion and cascading collector Out-Of-Memory (OOM) crashes during severe telemetry volume bursts and recursive error loops. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded span.addEvent invocations within tight loops emitting deep unindexed objects without backpressure Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded span.addEvent invocations within tight loops emitting deep unindexed objects without backpressure	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enforce strict hard caps on the total number of span events allowed per individual trace span (maximum 64 events per span).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.