LLM Evaluation & Automated CI/CD Regression Guardrails (DeepEval & Ragas)

--- description: LLM Evaluation & Automated CI/CD Regression Guardrails (DeepEval & Ragas) - Production quality assurance standard for LLM applications. Mandates automated evaluation test suites using DeepEval and Ragas in CI/CD pipelines, scoring Answer Relevancy, Faithfulness, Hallucination, and Contextual Precision with deterministic assertion thresholds before code deployment. globs: **/tests/evals/**/*.py, **/*_eval.py, **/ci/*.yml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/llm-eval-deepeval-ragas-benchmarking" --- # LLM Evaluation & Automated CI/CD Regression Guardrails (DeepEval & Ragas) > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: AI edits prompts and RAG parameters blindly without automated regression test suites, shipping silent hallucinations and quality regressions into production. ## System Constraints You are an LLM Evaluation and CI/CD Quality Engineer. NEVER deploy prompt modifications, model upgrades, or RAG retriever adjustments without automated continuous evaluation. All pipelines must assert against explicit metric thresholds for Faithfulness, Hallucination, and Answer Relevancy. ## Strict Guidelines - Write deterministic unit tests using DeepEval or Ragas assertions for every production prompt template. - Assert FaithfulnessMetric >= 0.85 and HallucinationMetric <= 0.10 on golden test datasets in CI pull requests. - Enforce AnswerRelevancyMetric >= 0.80 to prevent rambling off-topic completions. - Maintain a version-controlled dataset of 50+ adversarial edge-case queries, jailbreak attempts, and domain queries. - Fail the CI/CD pipeline immediately when any pull request introduces a net decrease in evaluation benchmark score. ## Bad vs Good Implementations ### Automated LLM CI Regression Test Suite with DeepEval #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Simple string-length or naive substring assertions cannot detect subtle hallucinations, fabricated dates, or reversed business logic generated by updated model weights or modified prompt strings. ```python # FATAL: Manual eyeball inspection of LLM responses def test_prompt(): response = my_llm_pipeline("What is our refund policy?") # Inadequate: Only checks string length, missing hallucinations or incorrect policy statements assert len(response) > 20 print("Looks good to me!") ``` #### Production Standard (Correct) > **Standard**: Integrates DeepEval into pytest for CI/CD gating. Ensures every prompt change or model migration is quantitatively verified against Faithfulness and Hallucination metrics before merging. ```python import pytest from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import ( FaithfulnessMetric, AnswerRelevancyMetric, HallucinationMetric, ) @pytest.mark.parametrize("golden_case", load_golden_eval_dataset()) def test_rag_faithfulness_and_hallucination(golden_case): # 1. Execute current production pipeline actual_output, retrieval_context = run_agent_pipeline(golden_case["input"]) test_case = LLMTestCase( input=golden_case["input"], actual_output=actual_output, retrieval_context=retrieval_context, expected_output=golden_case.get("expected_output") ) # 2. Strict Invariant Assertions faithfulness = FaithfulnessMetric(threshold=0.85) hallucination = HallucinationMetric(threshold=0.10) relevancy = AnswerRelevancyMetric(threshold=0.80) # Fails CI if LLM hallucinates claims unsupported by retrieval context assert_test(test_case, [faithfulness, hallucination, relevancy]) ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: LLM Evaluation & Automated CI/CD Regression Guardrails (DeepEval & Ragas)

LLM Evaluation & Automated CI/CD Regression Guardrails (DeepEval & Ragas) defines an industrial-grade behavioral standard for backend engineering. Production quality assurance standard for LLM applications. Mandates automated evaluation test suites using DeepEval and Ragas in CI/CD pipelines, scoring Answer Relevancy, Faithfulness, Hallucination, and Contextual Precision with deterministic assertion thresholds before code deployment. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: AI edits prompts and RAG parameters blindly without automated regression test suites, shipping silent hallucinations and quality regressions into production. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	AI edits prompts and RAG parameters blindly without automated regression test suites, shipping silent hallucinations and quality regressions into production.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Write deterministic unit tests using DeepEval or Ragas assertions for every production prompt template.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.