OpenTelemetry GenAI Semantic Convention Token Redaction Standard

--- description: OpenTelemetry GenAI Semantic Convention Token Redaction Standard - Enforces deterministic, pre-export sanitization and masking of PII, secrets, and raw vector payloads within OpenTelemetry GenAI semantic convention span attributes. Prohibits direct ingestion of unredacted prompt and completion telemetry into collectors, requiring custom span processors to scrub sensitive tokens before transport. globs: src/**/*.{ts,js}, instrumentation/**/*.{ts,js}, tracing/**/*.{ts,js} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-genai-token-redaction-standard" --- # OpenTelemetry GenAI Semantic Convention Token Redaction Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Exporting Unredacted Customer PII and Secrets in OpenTelemetry GenAI Span Attributes ## System Constraints You are a Principal Engineer and Security Architect enforcing the OpenTelemetry GenAI Semantic Convention Token Redaction Standard. You must reject any telemetry instrumentation that logs raw customer prompts, model completions, or high-dimensional embeddings directly into span attributes without deterministic sanitization. Mandate that an in-memory, pre-export SpanProcessor scrubs credentials, PII (SSN, emails, credit cards), and sensitive customer context from all `gen_ai.*` semantic attributes before dispatching spans to downstream OTLP exporters. ## Strict Guidelines - Engineers MUST ALWAYS register an in-pipeline sanitizing SpanProcessor before any OTLP BatchSpanProcessor or exporter is invoked. - Engineers MUST NOT directly assign raw user input to `gen_ai.prompt` or raw model outputs to `gen_ai.completion` without running through a verified deterministic token scrubber. - Span processors MUST mutate or drop sensitive span attributes matching `gen_ai.prompt`, `gen_ai.completion`, `gen_ai.system`, and `gen_ai.content.*` before spans reach serialization layers. - Services MUST log redaction metadata (e.g., `gen_ai.redaction.applied: true`, `gen_ai.redaction.rule_count: N`) to maintain auditability without exposing raw payload bytes. - High-entropy tokens such as API keys, bearer tokens, JWTs, and RFC-compliant email/credit card patterns MUST be replaced with deterministic surrogate tokens (e.g., `[REDACTED:API_KEY]`). - Telemetry pipelines NEVER store raw high-dimensional prompt embedding vectors directly in span attributes; only vector dimensionality and hash digests are permitted. ## Bad vs Good Implementations ### Deterministic OTel GenAI Span Sanitizer vs Raw Attribute Logging #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly sets `gen_ai.prompt` and `gen_ai.completion` with unredacted customer text. If the prompt contains API keys, passwords, or PII, they are permanently written to downstream OTLP observability storage without token masking. ```typescript import { trace } from '@opentelemetry/api'; const tracer = trace.getTracer('llm-service'); export async function executeLlmCall(userId: string, promptText: string): Promise<string> { return tracer.startActiveSpan('gen_ai.client.call', async (span) => { try { // CRITICAL FLAW: Raw customer prompt and PII directly injected into standard attributes span.setAttribute('gen_ai.system', 'openai'); span.setAttribute('gen_ai.request.model', 'gpt-4o'); span.setAttribute('gen_ai.prompt', promptText); span.setAttribute('enduser.id', userId); const completion = await callProviderApi(promptText); // CRITICAL FLAW: Raw completion exported to third-party APM backend span.setAttribute('gen_ai.completion', completion); return completion; } finally { span.end(); } }); } async function callProviderApi(prompt: string): Promise<string> { return 'Mock response for ' + prompt; } ``` #### Production Standard (Correct) > **Standard**: Implements defense-in-depth: a custom OpenTelemetry `SpanProcessor` intercepts and deterministically scrubs all `gen_ai.*` prompt attributes before export, and values are scrubbed at the call site with PII replaced by explicit tokens. ```typescript import { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'; import { trace, Span } from '@opentelemetry/api'; const PII_PATTERNS: Array<{ regex: RegExp; mask: string }> = [ { regex: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, mask: '[REDACTED:EMAIL]' }, { regex: /\b\d{3}-\d{2}-\d{4}\b/g, mask: '[REDACTED:SSN]' }, { regex: /sk-[A-Za-z0-9_-]{20,}/g, mask: '[REDACTED:SECRET_KEY]' }, ]; export function redactText(input: string): string { return PII_PATTERNS.reduce((acc, { regex, mask }) => acc.replace(regex, mask), input); } export class GenAiRedactionSpanProcessor implements SpanProcessor { forceFlush(): Promise<void> { return Promise.resolve(); } shutdown(): Promise<void> { return Promise.resolve(); } onStart(span: Span): void {} onEnd(span: ReadableSpan): void { const attributes = (span as unknown as { attributes: Record<string, unknown> }).attributes; if (!attributes) return; const targetKeys = ['gen_ai.prompt', 'gen_ai.completion', 'gen_ai.system_prompt']; let redactedCount = 0; for (const key of targetKeys) { const val = attributes[key]; if (typeof val === 'string') { const cleaned = redactText(val); if (cleaned !== val) redactedCount++; attributes[key] = cleaned; } } if (redactedCount > 0) { attributes['gen_ai.redaction.applied'] = true; attributes['gen_ai.redaction.tokens_scrubbed'] = redactedCount; } } } const tracer = trace.getTracer('llm-service'); export async function executeLlmCall(userId: string, promptText: string): Promise<string> { return tracer.startActiveSpan('gen_ai.client.call', async (span) => { try { span.setAttribute('gen_ai.system', 'openai'); span.setAttribute('gen_ai.request.model', 'gpt-4o'); // Sanitized prior to attribute recording and guaranteed scrubbed via SpanProcessor span.setAttribute('gen_ai.prompt', redactText(promptText)); span.setAttribute('enduser.pseudo_id', Buffer.from(userId).toString('base64url')); const completion = await callProviderApi(promptText); span.setAttribute('gen_ai.completion', redactText(completion)); return completion; } finally { span.end(); } }); } async function callProviderApi(prompt: string): Promise<string> { return 'Mock response for ' + prompt; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry GenAI Semantic Convention Token Redaction Standard

OpenTelemetry GenAI Semantic Convention Token Redaction Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, pre-export sanitization and masking of PII, secrets, and raw vector payloads within OpenTelemetry GenAI semantic convention span attributes. Prohibits direct ingestion of unredacted prompt and completion telemetry into collectors, requiring custom span processors to scrub sensitive tokens before transport. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Exporting Unredacted Customer PII and Secrets in OpenTelemetry GenAI Span Attributes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Exporting Unredacted Customer PII and Secrets in OpenTelemetry GenAI Span Attributes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS register an in-pipeline sanitizing SpanProcessor before any OTLP BatchSpanProcessor or exporter is invoked.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.