AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
securitycritical PriorityID: otel-genai-streaming-token-pii-redaction-standard

OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard

Mandates deterministic, sliding-window inline masking of personally identifiable information (PII) across Server-Sent Event (SSE) token chunks before attaching content to OpenTelemetry spans, events, or telemetry exporters. Prevents boundary-split PII leakage across chunk boundaries while maintaining ultra-low-latency streaming pipelines. Enforces zero raw token retention in telemetry buffers.

Prohibited Anti-Pattern: Attaching Unmasked Streaming Token Chunks Directly to OpenTelemetry Spans Without Multi-Chunk Boundary PII Redaction

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard - Mandates deterministic, sliding-window inline masking of personally identifiable information (PII) across Server-Sent Event (SSE) token chunks before attaching content to OpenTelemetry spans, events, or telemetry exporters. Prevents boundary-split PII leakage across chunk boundaries while maintaining ultra-low-latency streaming pipelines. Enforces zero raw token retention in telemetry buffers.
globs: src/telemetry/**/*.{ts,js}, src/ai/**/*.{ts,js}, src/llm/**/*.{ts,js}, src/streaming/**/*.{ts,js}, src/instrumentation/**/*.{ts,js}
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/otel-genai-streaming-token-pii-redaction-standard"
---

# OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard

> **Severity**: CRITICAL | **Category**: security
> **Anti-Pattern Warning**: Attaching Unmasked Streaming Token Chunks Directly to OpenTelemetry Spans Without Multi-Chunk Boundary PII Redaction

## System Constraints
You are a Principal Software Architect enforcing the OpenTelemetry GenAI Token Streaming PII Redaction Standard. You must reject any OpenTelemetry instrumentation that writes raw SSE/streaming GenAI chunks directly to span attributes (e.g., 'gen_ai.completion', 'gen_ai.content.completion') or span events without passing through an inline sliding-window PII mask. Ensure that pattern-matching spans boundary tokens to prevent split secrets (e.g., SSN split across SSE chunks) from reaching telemetry backends.

## Strict Guidelines
- Engineers MUST ALWAYS pass streaming token chunks through a bounded sliding-window PII redaction pipeline before recording span events or buffer aggregation.
- Engineers MUST NOT emit raw, unmasked LLM token fragments into OpenTelemetry span attributes, span events, metrics, or log records.
- Code SHOULD buffer a minimal sliding window (e.g., 32-64 characters) across token boundaries to detect PII (such as SSNs, credit cards, API keys) split across consecutive chunks.
- Services MUST ensure inline PII sanitization operates synchronously with the stream without introducing more than 5ms overhead per chunk.
- Components NEVER export or flush aggregated prompt/completion telemetry strings until full redaction validation passes across the entire assembled buffer.

## Bad vs Good Implementations
### Inline Sliding-Window PII Redaction for OpenTelemetry GenAI Streaming

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Raw token chunks are immediately recorded into OpenTelemetry span events without sanitization. If an SSN or email is emitted by the model, it is logged verbatim to collector traces. Furthermore, assembling chunks without cross-boundary scanning leaks PII split between consecutive chunks.
```typescript
import { Span } from '@opentelemetry/api';

export async function streamToTelemetryBad(
  stream: AsyncIterable<{ text: string }>,
  span: Span
): Promise<void> {
  let fullResponse = '';
  
  for await (const chunk of stream) {
    // VIOLATION: Directly attaching raw chunk to span events
    span.addEvent('gen_ai.content.completion', {
      'gen_ai.completion.token': chunk.text,
    });
    fullResponse += chunk.text;
  }
  
  // VIOLATION: Emitting unredacted aggregate to span attributes
  span.setAttribute('gen_ai.completion', fullResponse);
}
```

#### Production Standard (Correct)
> **Standard**: Implements a sliding-window redactor that buffers trailing characters across chunks to detect multi-token PII boundaries. Spans and trace events receive only sanitized strings, preventing PII transmission to collectors while preserving streaming behavior.
```typescript
import { Span } from '@opentelemetry/api';

interface Redactor {
  process(chunk: string, isLast: boolean): string;
}

class StreamingPiiRedactor implements Redactor {
  private windowBuffer = '';
  private static readonly PATTERNS: RegExp[] = [
    /\b\d{3}-\d{2}-\d{4}\b/g, // SSN
    /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, // Email
    /\b(?:\d[ -]*?){13,16}\b/g, // Credit Card
  ];
  private readonly overlapSize = 32;

  public process(chunk: string, isLast = false): string {
    this.windowBuffer += chunk;
    let sanitized = this.windowBuffer;

    for (const pattern of StreamingPiiRedactor.PATTERNS) {
      sanitized = sanitized.replace(pattern, '[REDACTED_PII]');
    }

    if (isLast) {
      this.windowBuffer = '';
      return sanitized;
    }

    if (sanitized.length > this.overlapSize) {
      const flushIndex = sanitized.length - this.overlapSize;
      const readyToEmit = sanitized.slice(0, flushIndex);
      this.windowBuffer = sanitized.slice(flushIndex);
      return readyToEmit;
    }

    return '';
  }
}

export async function streamToTelemetryGood(
  stream: AsyncIterable<{ text: string }>,
  span: Span
): Promise<void> {
  const redactor = new StreamingPiiRedactor();
  let redactedAccumulator = '';

  for await (const chunk of stream) {
    const safeChunk = redactor.process(chunk.text, false);
    if (safeChunk.length > 0) {
      span.addEvent('gen_ai.content.completion', {
        'gen_ai.completion.token': safeChunk,
      });
      redactedAccumulator += safeChunk;
    }
  }

  const finalChunk = redactor.process('', true);
  if (finalChunk.length > 0) {
    span.addEvent('gen_ai.content.completion', {
      'gen_ai.completion.token': finalChunk,
    });
    redactedAccumulator += finalChunk;
  }

  span.setAttribute('gen_ai.completion', redactedAccumulator);
}
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

OpenTelemetry Collector Telemetry Validator
Validates that OpenTelemetry span attributes and semantic conventions comply with GenAI telemetry security standards.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

OTel PII Trace Analyzer
CLI tool that scans trace JSON dumps and telemetry exports for unmasked PII tokens before release verification.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard

OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard defines an industrial-grade behavioral standard for security engineering. Mandates deterministic, sliding-window inline masking of personally identifiable information (PII) across Server-Sent Event (SSE) token chunks before attaching content to OpenTelemetry spans, events, or telemetry exporters. Prevents boundary-split PII leakage across chunk boundaries while maintaining ultra-low-latency streaming pipelines. Enforces zero raw token retention in telemetry buffers. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Attaching Unmasked Streaming Token Chunks Directly to OpenTelemetry Spans Without Multi-Chunk Boundary PII Redaction Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftAttaching Unmasked Streaming Token Chunks Directly to OpenTelemetry Spans Without Multi-Chunk Boundary PII RedactionCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS pass streaming token chunks through a bounded sliding-window PII redaction pipeline before recording span events or buffer aggregation.

Engineers MUST ALWAYS pass streaming token chunks through a bounded sliding-window PII redaction pipeline before recording span events or buffer aggregation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT emit raw, unmasked LLM token fragments into OpenTelemetry span attributes, span events, metrics, or log records.

Engineers MUST NOT emit raw, unmasked LLM token fragments into OpenTelemetry span attributes, span events, metrics, or log records.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Code SHOULD buffer a minimal sliding window (e.g., 32-64 characters) across token boundaries to detect PII (such as SSNs, credit cards, API keys) split across consecutive chunks.

Code SHOULD buffer a minimal sliding window (e.g., 32-64 characters) across token boundaries to detect PII (such as SSNs, credit cards, API keys) split across consecutive chunks.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Services MUST ensure inline PII sanitization operates synchronously with the stream without introducing more than 5ms overhead per chunk.

Services MUST ensure inline PII sanitization operates synchronously with the stream without introducing more than 5ms overhead per chunk.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Components NEVER export or flush aggregated prompt/completion telemetry strings until full redaction validation passes across the entire assembled buffer.

Components NEVER export or flush aggregated prompt/completion telemetry strings until full redaction validation passes across the entire assembled buffer.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is OpenTelemetry GenAI Token Streaming Inline PII Redaction Standard critical for AI coding assistants?

Mandates deterministic, sliding-window inline masking of personally identifiable information (PII) across Server-Sent Event (SSE) token chunks before attaching content to OpenTelemetry spans, events, or telemetry exporters. Prevents boundary-split PII leakage across chunk boundaries while maintaining ultra-low-latency streaming pipelines. Enforces zero raw token retention in telemetry buffers. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Attaching Unmasked Streaming Token Chunks Directly to OpenTelemetry Spans Without Multi-Chunk Boundary PII Redaction) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related SECURITY Rules

View all rules →
security
105

OpenFGA Graph Invariants: Cyclic Relation Prevention & Contextual Tuple Hardening

Enforces strict directed acyclic graph (DAG) topology validation during relationship mutation writes and mandates strict server-side schema verification for ephemeral contextual tuples in OpenFGA evaluations. Prohibits persistent storage of ephemeral request attributes and guards against recursive relation loops causing graph expansion denial-of-service.

security
81

WebAuthn Conditional UI Mediation, Nonce Atomicity, and Assertion Cryptographic Verification

Enforces zero-trust cryptographic validation for FIDO2/WebAuthn Passkey authentication with Conditional UI autofill. Requires atomic single-use challenge consumption to eliminate replay attacks, strict RP ID hash and origin binding, authenticatorData flag verification (User Presence and User Verification), and monotonic signature counter enforcement to detect authenticator cloning.

security
111

Enforce Sigstore Keyless OIDC Attestation and Admission Controller Verification

Enforces immutable supply-chain security using Sigstore Cosign keyless signing via ephemeral OIDC tokens issued by Fulcio and verified against the Rekor transparency log. Container workloads must be signed and attested at build time by digest, and admission controllers (such as Kyverno or Sigstore Policy Controller) must strictly validate the OIDC issuer and subject identity before allowing image execution.

1 Rule in Stack

OpenTelemetry