OpenTelemetry 1.28 Semantic Conventions & Tail-Based Trace Sampling Guardrail

--- description: OpenTelemetry 1.28 Semantic Conventions & Tail-Based Trace Sampling Guardrail - Enforces strict compliance with OpenTelemetry Semantic Conventions v1.28 across span attributes and collector pipelines. Guarantees deterministic 100% trace capture for anomalous P99 latency regressions and 5xx cascading errors while preventing cardinality explosion caused by unbounded span names, legacy attributes, and unparameterized URLs. globs: src/**/telemetry/**/*.{ts,js}, src/**/instrumentation/**/*.{ts,js}, src/**/middleware/**/*.{ts,js}, **/otel-collector*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-1-28-tail-sampling-invariant" --- # OpenTelemetry 1.28 Semantic Conventions & Tail-Based Trace Sampling Guardrail > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Cardinally Unbounded Head-Sampling with Deprecated v1.20 Attributes and Silently Dropped Failure Outliers ## System Constraints You are a Principal Observability Architect enforcing OpenTelemetry Semantic Conventions 1.28 and robust tail-based sampling policies. You must strictly mandate that all spans use modernized 1.28 attribute names ('http.response.status_code', 'http.request.method', 'url.path', 'error.type'), span names follow low-cardinality parameterized templates ('{http.request.method} {http.route}'), and telemetry pipelines retain 100% of 5xx errors and latency anomalies while discarding benign high-volume traces. ## Strict Guidelines - Engineers MUST ALWAYS use OpenTelemetry 1.28 stable HTTP/RPC attributes ('http.response.status_code', 'http.request.method', 'url.path', 'server.address') instead of deprecated legacy attributes ('http.status_code', 'http.method', 'http.url'). - Engineers MUST NOT construct span names containing dynamic identifiers, query strings, UUIDs, or raw path variables; span names MUST be static or parameterized route templates. - Services MUST emit 'error.type' and populate 'recordException' alongside 'http.response.status_code >= 500' to guarantee tail-sampling policy matchers capture the anomaly. - Tail-based sampling configurations in OpenTelemetry Collector MUST evaluate traces after completion and enforce 100% retention on criteria: error status, status code >= 500, or trace duration >= 2000ms. - Engineers MUST NEVER configure high-frequency production endpoints with pure head-based probabilistic sampling if P99 latency regressions and rare cascading 5xx bugs must be diagnosed. ## Bad vs Good Implementations ### OpenTelemetry 1.28 HTTP Instrumentation with Exception & Error Invariants #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Span name includes raw URL (high cardinality explosion). Uses deprecated semantic attributes ('http.status_code', 'http.method', 'http.url') which break OpenTelemetry 1.28 normalization. Lacks 'error.type' and exception record, preventing collector-side tail-sampling filters from triggering 100% retention on critical 5xx errors. ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; const tracer = trace.getTracer('orders-service'); export async function handleOrder(req: any, res: any) { // ANTI-PATTERN: High cardinality span name containing raw user input/query strings const span = tracer.startSpan(`GET ${req.originalUrl}`); try { // Deprecated OTel conventions (pre-1.21) causing telemetry fragmentation span.setAttribute('http.method', req.method); span.setAttribute('http.url', req.originalUrl); // Leaks tokens and query params span.setAttribute('http.status_code', 200); const data = await processOrder(req.params.orderId); return res.status(200).json(data); } catch (err: any) { // Fails to set error.type; tail sampler will miss the failure span.setAttribute('http.status_code', 500); span.setStatus({ code: SpanStatusCode.ERROR, message: err.message }); return res.status(500).json({ error: 'Internal Error' }); } finally { span.end(); } } ``` #### Production Standard (Correct) > **Standard**: Conforms to OpenTelemetry 1.28 conventions: uses low-cardinality parameterized span names, stable HTTP attributes ('http.request.method', 'http.route', 'http.response.status_code'), and records 'error.type' along with structured exceptions so collector tail-sampling rules reliably isolate P99 and 5xx incidents. ```typescript import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api'; const tracer = trace.getTracer('orders-service', '1.28.0'); export async function handleOrder(req: any, res: any) { // INVARIANT: Low-cardinality parameterized route template conformant with OTel 1.28 const route = req.route?.path ?? '/orders/:orderId'; const span = tracer.startSpan(`${req.method} ${route}`, { kind: SpanKind.SERVER, attributes: { 'http.request.method': req.method, 'http.route': route, 'url.path': req.baseUrl + req.path, 'server.address': req.hostname, }, }); try { const data = await processOrder(req.params.orderId); span.setAttribute('http.response.status_code', 200); return res.status(200).json(data); } catch (err: any) { const statusCode = err.status ?? 500; // Invariant attributes guaranteeing tail-sampling filter match span.setAttribute('http.response.status_code', statusCode); span.setAttribute('error.type', err.name || 'INTERNAL_ERROR'); span.recordException(err); span.setStatus({ code: SpanStatusCode.ERROR, message: err.message }); return res.status(statusCode).json({ error: err.message }); } finally { span.end(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry 1.28 Semantic Conventions & Tail-Based Trace Sampling Guardrail

OpenTelemetry 1.28 Semantic Conventions & Tail-Based Trace Sampling Guardrail defines an industrial-grade behavioral standard for devops engineering. Enforces strict compliance with OpenTelemetry Semantic Conventions v1.28 across span attributes and collector pipelines. Guarantees deterministic 100% trace capture for anomalous P99 latency regressions and 5xx cascading errors while preventing cardinality explosion caused by unbounded span names, legacy attributes, and unparameterized URLs. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Cardinally Unbounded Head-Sampling with Deprecated v1.20 Attributes and Silently Dropped Failure Outliers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Cardinally Unbounded Head-Sampling with Deprecated v1.20 Attributes and Silently Dropped Failure Outliers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS use OpenTelemetry 1.28 stable HTTP/RPC attributes ('http.response.status_code', 'http.request.method', 'url.path', 'server.address') instead of deprecated legacy attributes ('http.status_code', 'http.method', 'http.url').

Engineers MUST ALWAYS use OpenTelemetry 1.28 stable HTTP/RPC attributes ('http.response.status_code', 'http.request.method', 'url.path', 'server.address') instead of deprecated legacy attributes ('http.status_code', 'http.method', 'http.url').

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.