AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
devopscritical PriorityID: otel-tail-sampling-consistent-hash-ring-invariant

OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant

Enforces deterministic span routing using consistent hashing keyed strictly on trace ID across two-tier OpenTelemetry collector architectures. Prevents partial trace trees and false-positive tail-sampling drops caused by routing disparate spans of the same trace to distinct collector nodes.

Prohibited Anti-Pattern: Stateless load-balancing (Round-Robin/Random/IP Hash) across tail-sampling collector instances fragmenting distributed traces.

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant - Enforces deterministic span routing using consistent hashing keyed strictly on trace ID across two-tier OpenTelemetry collector architectures. Prevents partial trace trees and false-positive tail-sampling drops caused by routing disparate spans of the same trace to distinct collector nodes.
globs: deployments/**/*.yaml, deployments/**/*.yml, k8s/**/*.yaml, k8s/**/*.yml, terraform/**/*.tf, collector-configs/**/*.yaml, collector-configs/**/*.yml
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-tail-sampling-consistent-hash-ring-invariant"
---

# OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant

> **Severity**: CRITICAL | **Category**: devops
> **Anti-Pattern Warning**: Stateless load-balancing (Round-Robin/Random/IP Hash) across tail-sampling collector instances fragmenting distributed traces.

## System Constraints
You are a Principal Infrastructure & Observability Architect enforcing OpenTelemetry Tail-Sampling Collector Cluster Invariants. Tail-sampling requires 100% of spans for a given trace_id to reach the exact same collector instance evaluating the decision rules. You must mandate a two-tier architecture where Tier 1 (stateless gateway/agent) routes spans strictly via the `loadbalancing` exporter using consistent hashing (`trace_id`) backed by dynamic DNS or K8s headless service discovery, and Tier 2 runs `tail_sampling` with memory ballasts and matched hash algorithms.

## Strict Guidelines
- Engineers MUST ALWAYS configure Tier 1 routing collectors with the `loadbalancing` exporter configured with `routing_key: trace_id` and `consistent_hashing` resolver.
- Engineers MUST NEVER route directly from workloads to a tail-sampling collector pool through standard Kubernetes ClusterIP Service or L4/L7 load balancers (e.g., AWS ALB, NGINX round-robin).
- Tier 1 exporter resolver MUST use headless Kubernetes Service endpoints (`k8s_resolver` or dynamic DNS `hostname`) to track active Tier 2 replicas without kube-proxy interference.
- Collector Tier 2 instances MUST implement `memory_limiter` processor before the `tail_sampling` processor to prevent OOM panics when buffering high-cardinality spans awaiting root decision windows.
- Services MUST ensure span export retries preserve intact span batches and trace_id headers to maintain consistent ring placement under transient downstream network pressure.
- Engineers MUST align the decision wait time (`decision_wait`) on Tier 2 with the longest expected distributed transaction duration plus P99 network latency.

## Bad vs Good Implementations
### Tier 1 Collector Configuration: Stateless Gateway to Tail-Sampling Cluster

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Routing spans via standard K8s ClusterIP or round-robin exporter scatters spans belonging to the same trace across multiple backend collector pods. Tail-sampling rules cannot evaluate full trace trees, leading to incorrect sampling decisions and broken traces.
```yaml
# ANTI-PATTERN: Standard OTLP exporter pointing to a Round-Robin K8s Service
# Breaks trace assembly; spans of trace X are split across Pod A, B, and C.
exporters:
  otlp/tier2:
    endpoint: "otel-tail-sampling-svc.observability.svc.cluster.local:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tier2]
```

#### Production Standard (Correct)
> **Standard**: The loadbalancing exporter deterministically hashes the trace_id onto an active collector ring discovered via the headless Kubernetes service, guaranteeing all spans for any given trace land on the exact same collector node.
```yaml
# PRODUCTION INVARIANT: Trace-ID based Consistent-Hashing Load-Balancing Exporter
exporters:
  loadbalancing:
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      k8s:
        service: otel-collector-tier2-headless
        ports:
          - 4317
    routing_key: "trace_id"

processors:
  batch:
    send_batch_size: 8192
    timeout: 1s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [loadbalancing]
```

### Tier 2 Collector Configuration: Deterministic Tail-Sampling Node

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Lacks the memory_limiter processor preceding tail_sampling. Under trace burst conditions, buffering traces for 10 seconds will trigger an unhandled OOMKilled event on the node, breaking the consistent hash ring.
```yaml
# ANTI-PATTERN: Tail-sampling configured without memory limits or proper buffer constraints
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp/tempo]
```

#### Production Standard (Correct)
> **Standard**: Guarantees cluster stability by ordering memory_limiter before tail_sampling, explicitly sizing the internal trace cache table (num_traces, expected_new_traces_per_sec), and preventing ring rebalancing caused by node memory crashes.
```yaml
# PRODUCTION INVARIANT: Safe buffering with Memory Limiter and Deterministic Tail-Sampling
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 5000
    policies:
      - name: error-filter
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: high-latency
        type: latency
        latency: { threshold_ms: 1500 }
  batch:
    send_batch_size: 4096
    timeout: 500ms

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kubernetes Cluster Inspector
Inspects headless services, pod readiness endpoints, and collector daemonset/deployment status.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OTel Collector Config Validator
CLI validator to verify OpenTelemetry Collector pipeline components and semantic configuration schemas.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant

OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic span routing using consistent hashing keyed strictly on trace ID across two-tier OpenTelemetry collector architectures. Prevents partial trace trees and false-positive tail-sampling drops caused by routing disparate spans of the same trace to distinct collector nodes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Stateless load-balancing (Round-Robin/Random/IP Hash) across tail-sampling collector instances fragmenting distributed traces. Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftStateless load-balancing (Round-Robin/Random/IP Hash) across tail-sampling collector instances fragmenting distributed traces.Critical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS configure Tier 1 routing collectors with the `loadbalancing` exporter configured with `routing_key

Engineers MUST ALWAYS configure Tier 1 routing collectors with the `loadbalancing` exporter configured with `routing_key: trace_id` and `consistent_hashing` resolver.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NEVER route directly from workloads to a tail-sampling collector pool through standard Kubernetes ClusterIP Service or L4/L7 load balancers (e.g., AWS ALB, NGINX round-robin).

Engineers MUST NEVER route directly from workloads to a tail-sampling collector pool through standard Kubernetes ClusterIP Service or L4/L7 load balancers (e.g., AWS ALB, NGINX round-robin).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Tier 1 exporter resolver MUST use headless Kubernetes Service endpoints (`k8s_resolver` or dynamic DNS `hostname`) to track active Tier 2 replicas without kube-proxy interference.

Tier 1 exporter resolver MUST use headless Kubernetes Service endpoints (`k8s_resolver` or dynamic DNS `hostname`) to track active Tier 2 replicas without kube-proxy interference.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Collector Tier 2 instances MUST implement `memory_limiter` processor before the `tail_sampling` processor to prevent OOM panics when buffering high-cardinality spans awaiting root decision windows.

Collector Tier 2 instances MUST implement `memory_limiter` processor before the `tail_sampling` processor to prevent OOM panics when buffering high-cardinality spans awaiting root decision windows.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Services MUST ensure span export retries preserve intact span batches and trace_id headers to maintain consistent ring placement under transient downstream network pressure.

Services MUST ensure span export retries preserve intact span batches and trace_id headers to maintain consistent ring placement under transient downstream network pressure.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
6

Engineers MUST align the decision wait time (`decision_wait`) on Tier 2 with the longest expected distributed transaction duration plus P99 network latency.

Engineers MUST align the decision wait time (`decision_wait`) on Tier 2 with the longest expected distributed transaction duration plus P99 network latency.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry Tail-Sampling Collector Cluster Consistent-Hash Ring Invariant critical for AI coding assistants?

Enforces deterministic span routing using consistent hashing keyed strictly on trace ID across two-tier OpenTelemetry collector architectures. Prevents partial trace trees and false-positive tail-sampling drops caused by routing disparate spans of the same trace to distinct collector nodes. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Stateless load-balancing (Round-Robin/Random/IP Hash) across tail-sampling collector instances fragmenting distributed traces.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related DEVOPS Rules

View all rules →
devops
86

Kubernetes Gateway API v1 HTTPRoute Canary and ReferenceGrant Invariants

Enforces strict conformance to Kubernetes Gateway API v1 specifications during migration from legacy Ingress controllers. Mandates explicit ReferenceGrant resources for cross-namespace routing, strict relative integer weighting for HTTPRoute canary releases, and explicit backendRef port declarations.

devops
118

ArgoCD ApplicationSet Matrix Generator Invariants & Cascade Deletion Guards

Enforces strict structural constraints on ArgoCD ApplicationSet Matrix generators combining cluster selectors and Git directories. Prevents catastrophic multi-cluster cascade pruning, accidental secret exposure through parameter interpolation, and runaway Application CR explosion via explicit sync policies, preservation finalizers, and parameter validation.

devops
40

Karpenter Resilient Topology, Interruption Queue, and Consolidation Budget Standard

Enforces deterministic disruption budgets, multi-instance-type diversification, and event-driven SQS interruption queue handling across Karpenter NodePools and EC2NodeClasses. Eliminates cascading eviction loops, instance starvation, and availability downtime during AWS Spot rebalances and underutilized node consolidation.

1 Rule in Stack

OpenTelemetry