Deterministic Metric Exemplar and Trace-Sampling Coupling Standard

--- description: Deterministic Metric Exemplar and Trace-Sampling Coupling Standard - Enforces deterministic coupling between high-cardinality tail-latency metric histograms and distributed trace sampling decisions. Guarantees that P99+ tail-latency metrics linked to trace exemplars resolve to fully materialized traces, eliminating 404 trace drops and ghost exemplars during production incident triage. globs: src/**/telemetry.{ts,js}, src/**/instrumentation.{ts,js}, src/**/otel*.{ts,js}, src/observability/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-metric-exemplar-trace-sampling-governance" --- # Deterministic Metric Exemplar and Trace-Sampling Coupling Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Decoupled Probabilistic Head-Sampling Yielding Ghost Exemplars and Missing P99 Trace Context ## System Constraints You are a Principal Software Architect enforcing OpenTelemetry Metric Exemplar Trace-Sampling Governance. You must ensure that high-cardinality metric aggregations (specifically tail latency histograms) maintain deterministic references to valid, fully persisted distributed traces. Decoupled configurations where probabilistic head-samplers drop traces while metric exemplar filters point to discarded trace IDs create unresolvable observability black holes. Every metric exemplar recorded MUST reference a trace that is guaranteed to be sampled and exported. ## Strict Guidelines - Engineers MUST ALWAYS couple ExemplarFilter decisions with the TracerProvider's sampling state, ensuring non-sampled traces are never attached as metric exemplars unless tail-sampling collector pipelines are explicitly configured to retain them. - Engineers MUST NOT configure decoupled, low-ratio head-sampling (e.g., `TraceIdRatioBased(0.01)`) without setting `TraceBasedExemplarFilter` or an exemplar-aware composite sampler, which otherwise emits 'ghost exemplars' that return 404 in APM backends. - Code MUST explicitly configure `MeterProvider` with a `View` defining explicit bucket boundaries for latency histograms and an `ExemplarReservoir` tailored for high-percentile latency capture. - Services MUST ensure span contexts passed to metric recording calls contain active, sampled trace contexts (`TraceFlags.SAMPLED`) when an exemplar is evaluated for retention. - Trace sampling configurations MUST use adaptive or `ParentBased` composite samplers that force trace retention whenever a downstream operation violates predefined SLA or latency thresholds. ## Bad vs Good Implementations ### Configuring Metric Exemplar Reservoirs with Coordinated Trace Sampling #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Using a naive low-ratio head sampler without configuring metric exemplar reservoirs or trace-exemplar coordination causes 'ghost exemplars'. High latency P99/P99.9 outliers will record metrics, but their corresponding distributed traces will be dropped at root span creation, leaving SREs with broken trace links during incidents. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { TraceIdRatioBasedSampler, ParentBasedSampler } from '@opentelemetry/sdk-trace-node'; import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // ANTI-PATTERN: Decoupled 1% head-sampling with default exemplar behavior. // 99% of requests that breach P99 latency will have their traces dropped by the sampler, // but the histogram will still record latency, leaving exemplars pointing to non-existent traces. export const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter(), metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 10000, }), // Decoupled head sampler drops 99% of spans indiscriminately sampler: new ParentBasedSampler({ root: new TraceIdRatioBasedSampler(0.01), }), }); sdk.start(); ``` #### Production Standard (Correct) > **Standard**: Enforces deterministic coupling by pairing `TraceBasedExemplarFilter` with a coordinated sampler. Metric exemplars are guaranteed to attach only to traces that are definitively sampled and exported, eliminating broken links in observability platforms. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { Sampler, SamplingDecision, SamplingResult, ParentBasedSampler, } from '@opentelemetry/sdk-trace-node'; import { MeterProvider, PeriodicExportingMetricReader, View, ExplicitBucketHistogramAggregation, TraceBasedExemplarFilter, } from '@opentelemetry/sdk-metrics'; import { Context, TraceFlags } from '@opentelemetry/api'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc'; // Deterministic coupling: Force sampling for SLA-critical paths and link to exemplars class TailSlaAwareSampler implements Sampler { private readonly baseRatioSampler = new ParentBasedSampler({ root: new (require('@opentelemetry/sdk-trace-node').TraceIdRatioBasedSampler)(0.05), }); shouldSample(context: Context, traceId: string, spanName: string, spanKind: number, attributes: Record<string, unknown>, links: unknown[]): SamplingResult { // Guarantee 100% sampling on critical API routes or explicit debug headers const isCriticalPath = typeof attributes['http.target'] === 'string' && attributes['http.target'].startsWith('/api/v1/checkout'); if (isCriticalPath) { return { decision: SamplingDecision.RECORD_AND_SAMPLED, attributes: { 'sampling.reason': 'critical_path' } }; } return this.baseRatioSampler.shouldSample(context, traceId, spanName, spanKind, attributes, links); } toString(): string { return 'TailSlaAwareSampler'; } } const metricReader = new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15000, }); const meterProvider = new MeterProvider({ readers: [metricReader], // INVARIANT: TraceBasedExemplarFilter ensures metric exemplars ONLY capture sampled traces exemplarFilter: new TraceBasedExemplarFilter(), views: [ new View({ instrumentName: '*latency*', aggregation: new ExplicitBucketHistogramAggregation([ 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, ]), }), ], }); export const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter(), meterProvider, sampler: new TailSlaAwareSampler(), }); sdk.start(); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Deterministic Metric Exemplar and Trace-Sampling Coupling Standard

Deterministic Metric Exemplar and Trace-Sampling Coupling Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic coupling between high-cardinality tail-latency metric histograms and distributed trace sampling decisions. Guarantees that P99+ tail-latency metrics linked to trace exemplars resolve to fully materialized traces, eliminating 404 trace drops and ghost exemplars during production incident triage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Decoupled Probabilistic Head-Sampling Yielding Ghost Exemplars and Missing P99 Trace Context Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Decoupled Probabilistic Head-Sampling Yielding Ghost Exemplars and Missing P99 Trace Context	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS couple ExemplarFilter decisions with the TracerProvider's sampling state, ensuring non-sampled traces are never attached as metric exemplars unless tail-sampling collector pipelines are explicitly configured to retain them.

Engineers MUST ALWAYS couple ExemplarFilter decisions with the TracerProvider's sampling state, ensuring non-sampled traces are never attached as metric exemplars unless tail-sampling collector pipelines are explicitly configured to retain them.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.