Argo CD Progressive Rollout Analysis Step-Lock Invariant

--- description: Argo CD Progressive Rollout Analysis Step-Lock Invariant - Enforces mandatory inline AnalysisTemplate step-locks and automated abort thresholds within Argo Rollout canary and blue-green strategies. Automated progressive traffic weight advancement must strictly block and abort upon statistical anomaly detection in downstream service latencies (P95/P99) or error budgets. globs: **/rollout*.{yaml,yml}, **/analysis-template*.{yaml,yml}, charts/**/templates/*.{yaml,yml}, manifests/**/*.{yaml,yml} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/argocd-rollout-analysis-step-lock" --- # Argo CD Progressive Rollout Analysis Step-Lock Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Blind Traffic Escalation Without Downstream Anomaly Analysis Step-Locks ## System Constraints You are a Principal Software Architect and SRE Guardrail Author enforcing Argo CD Progressive Rollout Analysis Step-Lock Invariants. You must ensure no progressive traffic shifting (canary/blue-green) occurs without active, synchronized AnalysisTemplate step locks evaluating real-time downstream latency distributions (P95/P99) and telemetry anomaly thresholds. Reject any rollout configuration lacking instantaneous automated abort triggers on downstream latency degradation. ## Strict Guidelines - Engineers MUST ALWAYS bind an inline `analysis` block or step-level `analysis` reference before and during every canary weight increase in `spec.strategy.canary.steps`. - Engineers MUST define explicit `failureLimit` thresholds (e.g., `failureLimit: 1` or `2`) with immediate abort actions, rather than relying solely on non-blocking background analysis. - AnalysisTemplates MUST measure downstream dependency latency (e.g., downstream service response time percentiles via Envoy/Prometheus metrics) in addition to localized ingress response status codes. - Rollout specs MUST NOT omit `abortScaleDownDelaySeconds` or dynamic metric threshold gates, ensuring faulty revisions are halted and traffic is drained gracefully to stable pods. - Engineers MUST specify `consecutiveErrorLimit` and realistic `interval`/`count` cadences in metrics providers to prevent flapping while maintaining bounded mean time to detect (MTTD). ## Bad vs Good Implementations ### Argo Rollout Step Analysis with Downstream Latency Guard #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The rollout relies solely on static pause intervals. Downstream latency anomalies or service degradations will not halt the rollout, exposing 50% or more of production traffic to severe latency degradation or cascading brownouts. ```yaml # ANTI-PATTERN: Blind time-based step progression without downstream metric analysis apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: order-service spec: replicas: 10 strategy: canary: steps: - setWeight: 10 - pause: { duration: 10m } - setWeight: 50 - pause: { duration: 10m } # Flaw: No automated metric analysis gates step progression. # If downstream payment-service latency spikes, traffic still increases to 50%. ``` #### Production Standard (Correct) > **Standard**: Traffic weight escalation is step-locked by Prometheus metric evaluation of downstream P99 latency. If downstream latency exceeds the 400ms anomaly limit, failureLimit triggers an immediate rollout abort, preventing traffic escalation and isolating blast radius. ```yaml # PRODUCTION INVARIANT: Metric-locked canary progression with real-time anomaly aborts apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: order-service spec: replicas: 10 strategy: canary: abortScaleDownDelaySeconds: 30 analysis: templates: - templateName: downstream-latency-anomaly-guard args: - name: service-name value: order-service steps: - setWeight: 10 - pause: { duration: 2m } - analysis: templates: - templateName: downstream-latency-anomaly-guard args: - name: service-name value: order-service - setWeight: 25 - pause: { duration: 5m } - analysis: templates: - templateName: downstream-latency-anomaly-guard args: - name: service-name value: order-service - setWeight: 50 --- apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: downstream-latency-anomaly-guard spec: args: - name: service-name metrics: - name: downstream-p99-latency interval: 30s count: 4 failureLimit: 1 successCondition: result[0] <= 0.250 failureCondition: result[0] > 0.400 provider: prometheus: address: http://prometheus-k8s.monitoring:9090 query: | histogram_quantile(0.99, sum(rate(envoy_cluster_upstream_rq_time_bucket{envoy_cluster_name=~"downstream_.*",app="{{args.service-name}}"}[2m])) by (le)) ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Argo CD Progressive Rollout Analysis Step-Lock Invariant

Argo CD Progressive Rollout Analysis Step-Lock Invariant defines an industrial-grade behavioral standard for devops engineering. Enforces mandatory inline AnalysisTemplate step-locks and automated abort thresholds within Argo Rollout canary and blue-green strategies. Automated progressive traffic weight advancement must strictly block and abort upon statistical anomaly detection in downstream service latencies (P95/P99) or error budgets. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind Traffic Escalation Without Downstream Anomaly Analysis Step-Locks Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind Traffic Escalation Without Downstream Anomaly Analysis Step-Locks	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS bind an inline `analysis` block or step-level `analysis` reference before and during every canary weight increase in `spec.strategy.canary.steps`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.