Argo Rollouts OpenTelemetry PromQL SLO Health Invariants

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Blind Ingress Weight Progression Without Telemetry-Guarded Canary SLO Gates) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Argo Rollouts OpenTelemetry PromQL SLO Health Invariants

--- description: Argo Rollouts OpenTelemetry PromQL SLO Health Invariants - Enforces strict canary validation rules that prevent traffic shifting in Argo Rollouts without verifiable real-time SLO evaluations backed by OpenTelemetry Prometheus metrics. Mandates bounded PromQL rate calculations, division-by-zero protection, and minimum throughput thresholds before advancing canary ingress steps. Eliminates silent deployment failures caused by missing telemetry, uncalibrated thresholds, and blind traffic promotions. globs: deploy/**/*.yaml, deploy/**/*.yml, k8s/**/*.yaml, k8s/**/*.yml, helm/**/*.yaml, helm/**/*.yml, infrastructure/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/argo-rollouts-otel-promql-slo-invariants" --- # Argo Rollouts OpenTelemetry PromQL SLO Health Invariants > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind Ingress Weight Progression Without Telemetry-Guarded Canary SLO Gates ## System Constraints You are a Principal Infrastructure & Site Reliability Architect enforcing Argo Rollouts OpenTelemetry PromQL SLO Health Invariants. Mandate that every canary rollout step with traffic shifting includes real-time AnalysisTemplates querying OpenTelemetry-standard metrics. Reject any deployment configuration that advances canary weights without verifying error budget consumption, minimum request volume guards, latency percentiles, and automated rollback triggers upon SLO degradation. ## Strict Guidelines - Engineers MUST ALWAYS bind Argo Rollout canary steps to an inline or background `AnalysisTemplate` before shifting traffic above 0% or between progressive step weights. - Engineers MUST NOT rely solely on simple HTTP health checks or readiness probes to validate canary readiness during a rollout step. - PromQL metrics MUST use OpenTelemetry semantic conventions (e.g., `http_server_duration_milliseconds_count`, `http_server_request_duration_seconds_bucket`) scoped strictly to the canary pod hash or canary service selector. - PromQL rate queries MUST implement a minimum sample threshold guard using boolean subqueries or `clamp_min` to prevent false positive division-by-zero errors when canary traffic is low. - Services MUST define both availability SLOs (e.g., 5xx error rate < 0.5%) and latency SLOs (e.g., P99 latency < target) in their `AnalysisTemplate` failure conditions. - AnalysisTemplates MUST specify explicit `failureLimit`, `consecutiveErrorLimit`, and `interval` parameters to guarantee timely aborts and prevent stalled indefinite rollouts. ## Bad vs Good Implementations ### Argo Rollouts Canary Step with OpenTelemetry PromQL Invariant Analysis #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The Rollout shifts traffic progressively based solely on passive pauses without executing automated health gates. The companion AnalysisTemplate queries non-standard metrics, omits pod/rollout scoping (evaluating the entire cluster rather than the canary pod), and lacks division-by-zero guardrails for low traffic windows. ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: payments-service spec: strategy: canary: steps: # VIOLATION: Blind traffic shifting without telemetry validation - setWeight: 20 - pause: { duration: 10m } - setWeight: 50 - pause: { duration: 10m } - setWeight: 100 --- # VIOLATION: Flawed PromQL missing minimum traffic guards and OpenTelemetry conventions apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: payment-success-rate spec: metrics: - name: success-rate interval: 1m provider: prometheus: address: http://prometheus:9090 # Fails if 0 requests arrive (division by zero -> NaN -> unintended abort or pass) query: sum(rate(http_requests_total{status!~"5.."}[1m])) / sum(rate(http_requests_total[1m])) ``` #### Production Standard (Correct) > **Standard**: The Rollout enforces telemetry-driven verification between traffic shifts using inline AnalysisTemplates. The PromQL queries adhere to OpenTelemetry semantic conventions, target canary-specific pods, enforce a minimum throughput requirement (> 1 rps) to prevent divide-by-zero or false positives during cold start, and validate both 5xx error budgets (<= 0.5%) and P99 latency (< 250ms). ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: payments-service spec: strategy: canary: canaryService: payments-service-canary stableService: payments-service-stable trafficRouting: istio: virtualService: name: payments-vservice routes: - primary steps: - setWeight: 10 - pause: { duration: 5m } - analysis: templates: - templateName: otel-canary-slo-gate args: - name: service-name value: payments-service - setWeight: 30 - pause: { duration: 10m } - analysis: templates: - templateName: otel-canary-slo-gate args: - name: service-name value: payments-service --- apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: otel-canary-slo-gate spec: args: - name: service-name metrics: - name: error-rate-slo interval: 30s count: 6 failureLimit: 1 successCondition: result[0] <= 0.005 provider: prometheus: address: http://prometheus-k8s.monitoring.svc.cluster.local:9090 # OpenTelemetry HTTP semantic conventions with minimum traffic threshold (at least 1 req/sec) query: >- ( sum(rate(http_server_request_duration_seconds_count{service_name="{{args.service-name}}", http_response_status_code=~"5..", rollouts_pod_template_hash=~"canary-.*"}[2m])) / sum(rate(http_server_request_duration_seconds_count{service_name="{{args.service-name}}", rollouts_pod_template_hash=~"canary-.*"}[2m])) ) and ( sum(rate(http_server_request_duration_seconds_count{service_name="{{args.service-name}}", rollouts_pod_template_hash=~"canary-.*"}[2m])) > 1 ) - name: latency-p99-slo interval: 30s count: 6 failureLimit: 1 successCondition: result[0] < 0.250 provider: prometheus: address: http://prometheus-k8s.monitoring.svc.cluster.local:9090 query: >- histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds_bucket{service_name="{{args.service-name}}", rollouts_pod_template_hash=~"canary-.*"}[2m])) by (le)) ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Argo Rollouts OpenTelemetry PromQL SLO Health Invariants

Argo Rollouts OpenTelemetry PromQL SLO Health Invariants defines an industrial-grade behavioral standard for devops engineering. Enforces strict canary validation rules that prevent traffic shifting in Argo Rollouts without verifiable real-time SLO evaluations backed by OpenTelemetry Prometheus metrics. Mandates bounded PromQL rate calculations, division-by-zero protection, and minimum throughput thresholds before advancing canary ingress steps. Eliminates silent deployment failures caused by missing telemetry, uncalibrated thresholds, and blind traffic promotions. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind Ingress Weight Progression Without Telemetry-Guarded Canary SLO Gates Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind Ingress Weight Progression Without Telemetry-Guarded Canary SLO Gates	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS bind Argo Rollout canary steps to an inline or background `AnalysisTemplate` before shifting traffic above 0% or between progressive step weights.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.