Architectural Invariant Design: OpenTelemetry Collector Dynamic Memory Shedding & Adaptive Tail Sampling Invariant
OpenTelemetry Collector Dynamic Memory Shedding & Adaptive Tail Sampling Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces survival invariants on OpenTelemetry Collector pipelines by prioritizing critical error traces and shedding non-erroneous telemetry under downstream backpressure. Prevents OOM kills through strict processor ordering, memory_limiter thresholds, and composite tail-sampling policies with prioritized fallback tiers. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.
Hazard Analysis: What Breaks Without This Rule
The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:
| Failure Mode | AI Symptom (The Anti-Pattern) | Severity | Production Impact & Remedy |
|---|---|---|---|
| AI Hallucination & Deprecation Drift | Unbounded Trace Buffering and Indiscriminate In-Memory Tail Sampling Inducing Outage Cascades and OOM Kills | Critical | Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository. |
| Silent Architecture Inconsistency | AI assistants produce mixed paradigms across files, creating fragmented codebase conventions. | Medium | Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails. |
Step-by-Step Production Directives
Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:
Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in every trace pipeline.
Engineers MUST ALWAYS declare the `memory_limiter` processor as the FIRST processor in every trace pipeline.
Engineers MUST configure `check_interval` to at most `1s` with `limit_percentage` set between 75% and 80%, and `spike_limit_percentage` set to 20% to prevent heap runaway before garbage collection cycles execute.
Engineers MUST configure `check_interval` to at most `1s` with `limit_percentage` set between 75% and 80%, and `spike_limit_percentage` set to 20% to prevent heap runaway before garbage collection cycles execute.
Engineers MUST NOT place `tail_sampling` ahead of `memory_limiter` or omit hard limits on `num_traces` in the tail sampling memory buffer.
Engineers MUST NOT place `tail_sampling` ahead of `memory_limiter` or omit hard limits on `num_traces` in the tail sampling memory buffer.
Pipelines MUST implement prioritized composite sampling where HTTP 5xx errors and gRPC status code exceptions are evaluated with 100% retention before applying probabilistic or rate-limiting filters to healthy traces.
Pipelines MUST implement prioritized composite sampling where HTTP 5xx errors and gRPC status code exceptions are evaluated with 100% retention before applying probabilistic or rate-limiting filters to healthy traces.
Exporters MUST configure bounded retry queues (`sending_queue` with explicit `queue_size` and non-infinite `retry_on_failure`) to prevent exporter buffers from cannibalizing the tail sampler's memory allocation.
Exporters MUST configure bounded retry queues (`sending_queue` with explicit `queue_size` and non-infinite `retry_on_failure`) to prevent exporter buffers from cannibalizing the tail sampler's memory allocation.
Collectors MUST emit and alert on `otelcol_processor_tail_sampling_early_drops` and `otelcol_processor_refused_spans` to indicate active memory shedding.
Collectors MUST emit and alert on `otelcol_processor_tail_sampling_early_drops` and `otelcol_processor_refused_spans` to indicate active memory shedding.
Frequently Asked Technical Questions
Common questions developers face when configuring AI coding assistants for this stack:
Q:Why is OpenTelemetry Collector Dynamic Memory Shedding & Adaptive Tail Sampling Invariant critical for AI coding assistants?
Enforces survival invariants on OpenTelemetry Collector pipelines by prioritizing critical error traces and shedding non-erroneous telemetry under downstream backpressure. Prevents OOM kills through strict processor ordering, memory_limiter thresholds, and composite tail-sampling policies with prioritized fallback tiers. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.
Q:How does adopting this rule prevent AI hallucinations?
This rule explicitly declares what the AI must avoid (Unbounded Trace Buffering and Indiscriminate In-Memory Tail Sampling Inducing Outage Cascades and OOM Kills) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.
Q:Which AI coding environments are supported?
This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).
Q:Can I combine this rule with other technology stacks?
Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.