OpenTelemetry v1.28 GenAI Span Lineage & Distributed Token Budget Governance

--- description: OpenTelemetry v1.28 GenAI Span Lineage & Distributed Token Budget Governance - Mandates strict adherence to OpenTelemetry Semantic Conventions v1.28 for GenAI operations across all multi-agent orchestration spans. Enforces deterministic prompt template hashing, pre-execution token budget validation, and standard semantic attribute emission to prevent unbounded operational spend and broken telemetry lineage. globs: src/agents/**/*.ts, src/llm/**/*.ts, src/orchestration/**/*.ts, src/telemetry/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-v1-28-genai-tracing-token-budgeting-standard" --- # OpenTelemetry v1.28 GenAI Span Lineage & Distributed Token Budget Governance > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Ad-Hoc LLM Invocations with Unmonitored Spans, Unhashed Prompts, and Unbounded Token Consumption ## System Constraints You are a Principal Software Architect enforcing OpenTelemetry Semantic Conventions v1.28 GenAI Tracing and distributed token budget guardrails. You must reject any direct LLM client invocation that lacks an active GenAI telemetry span, bypasses deterministic prompt hash lineage calculation, or omits atomic token budget reservation checks. ## Strict Guidelines - Engineers MUST ALWAYS wrap every LLM generation, embedding, or tool call in an OpenTelemetry span conforming to v1.28 semantic conventions using SpanKind.CLIENT or SpanKind.INTERNAL. - Engineers MUST ALWAYS populate standard GenAI attributes: `gen_ai.system`, `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, and `gen_ai.usage.output_tokens`. - Engineers MUST NEVER record raw unmasked PII or sensitive proprietary inputs in span attributes; prompt content must be hashed into a deterministic lineage identifier (e.g., `gen_ai.prompt.hash` via SHA-256) unless explicitly operating in an isolated debug environment with scrubbing enabled. - Services MUST perform pre-execution token budget checks against the distributed budget manager before triggering LLM inference and deduct actual reported tokens upon span completion. - Components NEVER execute raw SDK client calls (e.g., `openai.chat.completions.create`) directly inside business logic without going through an instrumented, budget-aware LLM proxy client. ## Bad vs Good Implementations ### Unbounded Direct LLM Invocation vs. OTel v1.28 Instrumented & Budget-Enforced Execution #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The bad example calls the LLM provider directly without an OpenTelemetry span, violating OTel v1.28 GenAI conventions. There is no pre-flight token budget reservation, no prompt lineage hashing, and no recorded metrics for input/output token usage. ```typescript import OpenAI from 'openai'; const client = new OpenAI(); export async function runAgentStep(prompt: string): Promise<string> { // ANTI-PATTERN: Direct invocation, no OTel span, missing GenAI semantic attributes, // zero token budgeting, and unhashed prompt leakage into uninstrumented logs. const response = await client.chat.completions.create({ model: 'gpt-4o', messages: [{ role: 'user', content: prompt }], temperature: 0.7, }); return response.choices[0].message.content ?? ''; } ``` #### Production Standard (Correct) > **Standard**: The good example wraps execution in a compliant OpenTelemetry v1.28 span, hashes the prompt lineage using SHA-256 to safeguard privacy while enabling distributed cache tracing, strictly enforces token budget limits prior to network calls, and records usage attributes. ```typescript import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api'; import { createHash } from 'node:crypto'; import OpenAI from 'openai'; import { TokenBudgetExceededError, reserveTokenBudget, commitTokenUsage } from '../telemetry/budget'; const tracer = trace.getTracer('orchestrator.agent', '1.28.0'); const client = new OpenAI(); interface AgentExecutionOptions { sessionId: string; maxBudgetTokens: number; } export async function runAgentStep( prompt: string, options: AgentExecutionOptions ): Promise<string> { const promptHash = createHash('sha256').update(prompt).digest('hex'); const model = 'gpt-4o'; // Pre-execution token reservation guardrail const hasBudget = await reserveTokenBudget(options.sessionId, options.maxBudgetTokens); if (!hasBudget) { throw new TokenBudgetExceededError(`Session ${options.sessionId} exceeded token budget.`); } return tracer.startActiveSpan( `chat ${model}`, { kind: SpanKind.CLIENT, attributes: { 'gen_ai.system': 'openai', 'gen_ai.operation.name': 'chat', 'gen_ai.request.model': model, 'gen_ai.request.temperature': 0.7, 'gen_ai.prompt.hash': promptHash, 'session.id': options.sessionId, }, }, async (span) => { try { const response = await client.chat.completions.create({ model, messages: [{ role: 'user', content: prompt }], temperature: 0.7, }); const inputTokens = response.usage?.prompt_tokens ?? 0; const outputTokens = response.usage?.completion_tokens ?? 0; const responseModel = response.model ?? model; // Standard OTel v1.28 attribute population span.setAttributes({ 'gen_ai.response.model': responseModel, 'gen_ai.response.finish_reasons': response.choices.map((c) => c.finish_reason ?? 'unknown'), 'gen_ai.usage.input_tokens': inputTokens, 'gen_ai.usage.output_tokens': outputTokens, }); await commitTokenUsage(options.sessionId, inputTokens + outputTokens, options.maxBudgetTokens); span.setStatus({ code: SpanStatusCode.OK }); return response.choices[0].message.content ?? ''; } catch (error) { span.recordException(error as Error); span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message }); throw error; } finally { span.end(); } } ); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry v1.28 GenAI Span Lineage & Distributed Token Budget Governance

OpenTelemetry v1.28 GenAI Span Lineage & Distributed Token Budget Governance defines an industrial-grade behavioral standard for backend engineering. Mandates strict adherence to OpenTelemetry Semantic Conventions v1.28 for GenAI operations across all multi-agent orchestration spans. Enforces deterministic prompt template hashing, pre-execution token budget validation, and standard semantic attribute emission to prevent unbounded operational spend and broken telemetry lineage. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Ad-Hoc LLM Invocations with Unmonitored Spans, Unhashed Prompts, and Unbounded Token Consumption Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Ad-Hoc LLM Invocations with Unmonitored Spans, Unhashed Prompts, and Unbounded Token Consumption	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS wrap every LLM generation, embedding, or tool call in an OpenTelemetry span conforming to v1.28 semantic conventions using SpanKind.CLIENT or SpanKind.INTERNAL.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.