Mandatory In-Memory PII Scrubbing SpanProcessor for OpenTelemetry GenAI Conventions

--- description: Mandatory In-Memory PII Scrubbing SpanProcessor for OpenTelemetry GenAI Conventions - Enforces deterministic, in-memory sanitization of OpenTelemetry GenAI semantic attributes (e.g., gen_ai.prompt, gen_ai.completion, and event payloads) prior to dispatching to OTLP batch processors. Prevents cross-region exfiltration of personally identifiable information (PII), protected health information (PHI), and credential secrets in AI observability pipelines. globs: src/telemetry/**/*.ts, src/instrumentation/**/*.ts, src/observability/**/*.ts, src/tracing/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opentelemetry-genai-pii-sanitization-guardrail" --- # Mandatory In-Memory PII Scrubbing SpanProcessor for OpenTelemetry GenAI Conventions > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Attaching Raw Prompts and Model Outputs to OTLP Exporters Without Pre-Export Sanitization ## System Constraints You are a Principal Observability and Compliance Architect enforcing OpenTelemetry GenAI Semantic Conventions with strict PII sanitization. You must reject any telemetry pipeline that passes unredacted LLM prompt/completion attributes directly to OTLP exporters. Require dedicated sanitizing SpanProcessors to intercept, scrub, and mask sensitive customer data in memory before network serialization. ## Strict Guidelines - Engineers MUST ALWAYS register a custom PII-sanitizing SpanProcessor upstream of any BatchSpanProcessor or OTLPExporter when instrumenting GenAI workflows. - Engineers MUST NOT populate standard OpenTelemetry GenAI attributes (`gen_ai.prompt`, `gen_ai.completion`, `gen_ai.system`) with raw customer context unless pre-scrubbed through an approved token redaction engine. - Span events capturing prompt-completion sequences (`gen_ai.content.prompt`, `gen_ai.content.completion`) MUST redact email addresses, national identity numbers, payment tokens, and bearer credentials prior to span recording. - Services MUST ensure semantic attributes containing user identities (`gen_ai.user.id` or `enduser.id`) use one-way salted HMAC hashes rather than plaintext database primary keys or email addresses. - Telemetry processors NEVER permit fallthrough of unmasked attributes upon sanitization exceptions; processing failures MUST fallback to complete attribute redaction (`[REDACTION_FAILURE]`). ## Bad vs Good Implementations ### In-Memory GenAI PII Sanitizing SpanProcessor Implementation #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly serializing prompt and completion text into span attributes without an intermediate scrubbing processor risks transmitting SSNs, emails, and credentials to third-party collectors or cross-region APM backends, violating GDPR, HIPAA, and corporate data governance policies. ```typescript import { NodeSDK } from '@opentelemetry/sdk-node'; import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'; import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'; import { trace } from '@opentelemetry/api'; // ANTI-PATTERN: Raw user prompt and completion are exported directly to cross-region OTLP collector export const tracer = trace.getTracer('genai-service'); export async function recordInference(prompt: string, completion: string, userId: string) { const span = tracer.startSpan('genai.inference'); span.setAttributes({ 'gen_ai.system': 'openai', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.prompt': prompt, // Leak: Raw PII (e.g. 'SSN: 000-12-3456, email: [email protected]') 'gen_ai.completion': completion, // Leak: Raw output with potential PII reflection 'enduser.id': userId // Leak: Plaintext user identifier }); span.end(); } export const sdk = new NodeSDK({ spanProcessor: new BatchSpanProcessor(new OTLPTraceExporter()), }); ``` #### Production Standard (Correct) > **Standard**: Implements an explicit OpenTelemetry SpanProcessor that intercepts OpenTelemetry GenAI semantic conventions ('gen_ai.prompt', 'gen_ai.completion', 'enduser.id') in memory. PII is scrubbed using precompiled patterns, end-user IDs are pseudonymized using HMAC-SHA256, and failure cases fallback safely without leaking raw customer attributes. ```typescript import { SpanProcessor, ReadableSpan, Span } from '@opentelemetry/sdk-trace-base'; import { Context } from '@opentelemetry/api'; import { createHmac } from 'crypto'; const PII_PATTERNS = [ { pattern: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g, mask: '[REDACTED_EMAIL]' }, { pattern: /\b\d{3}-\d{2}-\d{4}\b/g, mask: '[REDACTED_SSN]' }, { pattern: /Bearer\s+[A-Za-z0-9-_=.]+/gi, mask: 'Bearer [REDACTED_TOKEN]' } ]; const GENAI_TEXT_ATTRIBUTES = new Set([ 'gen_ai.prompt', 'gen_ai.completion', 'gen_ai.system' ]); function sanitizeText(input: unknown): string { if (typeof input !== 'string') return ''; try { return PII_PATTERNS.reduce((acc, { pattern, mask }) => acc.replace(pattern, mask), input); } catch { return '[REDACTION_FAILURE]'; } } export class GenAiPiiSanitizingSpanProcessor implements SpanProcessor { private hmacSecret = process.env.TELEMETRY_HMAC_SALT || 'default-salt'; onStart(span: Span, _parentContext: Context): void { // Intercept attributes at start if present this.sanitizeSpan(span); } onEnd(span: ReadableSpan): void { // Cast to mutable span internal attributes prior to export pipeline dispatch const mutableSpan = span as unknown as { attributes: Record<string, unknown> }; for (const [key, value] of Object.entries(mutableSpan.attributes)) { if (GENAI_TEXT_ATTRIBUTES.has(key)) { mutableSpan.attributes[key] = sanitizeText(value); } else if (key === 'enduser.id' && typeof value === 'string') { mutableSpan.attributes[key] = createHmac('sha256', this.hmacSecret).update(value).digest('hex'); } } } shutdown(): Promise<void> { return Promise.resolve(); } forceFlush(): Promise<void> { return Promise.resolve(); } private sanitizeSpan(span: Span): void { // Optional pre-scrubbing hooks for dynamic attribute modifications } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Mandatory In-Memory PII Scrubbing SpanProcessor for OpenTelemetry GenAI Conventions

Mandatory In-Memory PII Scrubbing SpanProcessor for OpenTelemetry GenAI Conventions defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic, in-memory sanitization of OpenTelemetry GenAI semantic attributes (e.g., gen_ai.prompt, gen_ai.completion, and event payloads) prior to dispatching to OTLP batch processors. Prevents cross-region exfiltration of personally identifiable information (PII), protected health information (PHI), and credential secrets in AI observability pipelines. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Attaching Raw Prompts and Model Outputs to OTLP Exporters Without Pre-Export Sanitization Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Attaching Raw Prompts and Model Outputs to OTLP Exporters Without Pre-Export Sanitization	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS register a custom PII-sanitizing SpanProcessor upstream of any BatchSpanProcessor or OTLPExporter when instrumenting GenAI workflows.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.