Prometheus & OpenTelemetry RED Observability Standard

--- description: Prometheus & OpenTelemetry RED Observability Standard - Guardrails for observability: implement RED metrics (Rate, Errors, Duration), prevent high-cardinality metric label explosions (never put user IDs or UUIDs in labels), and propagate W3C trace context. globs: **/*metrics*, **/*tracing*, **/*telemetry* alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/prometheus-grafana-metrics" --- # Prometheus & OpenTelemetry RED Observability Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: AI attaches dynamic user IDs or query parameters as Prometheus metric labels, causing out-of-memory crashes on Prometheus servers. ## System Constraints You are an observability engineer. Follow the RED method (Rate, Errors, Duration). Labels on metrics must have strictly bounded, low cardinality (e.g. http_status_code, method, route_pattern). NEVER put user_id, email, or order_uuid in Prometheus labels. Propagate traceparent headers across distributed HTTP requests. ## Strict Guidelines - Prometheus labels must be low-cardinality: use parameterized route patterns (/users/:id) instead of raw paths (/users/12345). - Measure HTTP request durations with a Prometheus Histogram using exponential or standard web buckets. - Inject and extract OpenTelemetry traceparent headers to correlate distributed spans with logs. ## Bad vs Good Implementations ### Low-Cardinality Route Metric vs Unbounded Cardinality Explosion #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Creating millions of time series will exhaust Prometheus memory within hours, causing monitoring blackout. ```typescript // CRITICAL DISASTER: Cardinality explosion crashes Prometheus httpRequestDuration .labels({ path: req.originalUrl, // Contains "/users/a9f1-432b..." (Millions of unique paths!) userId: req.user.id // Unbounded cardinality! }) .observe(duration); ``` #### Production Standard (Correct) > **Standard**: Keeps label cardinality strictly bounded (under 100 combinations), ensuring decades of stable metric storage. ```typescript // Production Standard: Normalized route pattern with bounded labels httpRequestDuration .labels({ method: req.method, route: req.route?.path || "unmatched", // Normalized: "/users/:id" status: String(res.statusCode) }) .observe(duration); ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Prometheus & OpenTelemetry RED Observability Standard

Prometheus & OpenTelemetry RED Observability Standard defines an industrial-grade behavioral standard for devops engineering. Guardrails for observability: implement RED metrics (Rate, Errors, Duration), prevent high-cardinality metric label explosions (never put user IDs or UUIDs in labels), and propagate W3C trace context. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: AI attaches dynamic user IDs or query parameters as Prometheus metric labels, causing out-of-memory crashes on Prometheus servers. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	AI attaches dynamic user IDs or query parameters as Prometheus metric labels, causing out-of-memory crashes on Prometheus servers.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Prometheus labels must be low-cardinality

Prometheus labels must be low-cardinality: use parameterized route patterns (/users/:id) instead of raw paths (/users/12345).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

Measure HTTP request durations with a Prometheus Histogram using exponential or standard web buckets.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

Inject and extract OpenTelemetry traceparent headers to correlate distributed spans with logs.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is Prometheus & OpenTelemetry RED Observability Standard critical for AI coding assistants?

Guardrails for observability: implement RED metrics (Rate, Errors, Duration), prevent high-cardinality metric label explosions (never put user IDs or UUIDs in labels), and propagate W3C trace context. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (AI attaches dynamic user IDs or query parameters as Prometheus metric labels, causing out-of-memory crashes on Prometheus servers.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.