Enterprise RAG Pipeline: Semantic Chunking & Cross-Encoder Reranking

--- description: Enterprise RAG Pipeline: Semantic Chunking & Cross-Encoder Reranking - Production architecture for Retrieval-Augmented Generation (RAG). Replaces arbitrary character splitting with semantic Markdown header hierarchy chunking, hybrid vector + BM25 keyword search with Reciprocal Rank Fusion (RRF), and Cross-Encoder reranking (Cohere / BGE) to eliminate LLM hallucinations. globs: **/rag/**/*.py, **/retriever/**/*.py, **/pipeline/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/rag-semantic-chunking-reranking" --- # Enterprise RAG Pipeline: Semantic Chunking & Cross-Encoder Reranking > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: AI uses fixed chunk_size=500 character splitting cutting code blocks and sentences in half, passes raw top-k vector similarity scores, and prompts LLMs with fragmented noise. ## System Constraints You are an Enterprise RAG and Information Retrieval Specialist. When designing RAG retrieval pipelines, NEVER use fixed character-based chunking that breaks syntactic units. Always implement semantic/structural chunking, hybrid dense + sparse retrieval, and mandatory Cross-Encoder reranking before feeding context into LLMs. ## Strict Guidelines - Use structural/markdown-aware splitters (e.g. MarkdownHeaderTextSplitter) preserving code blocks and complete table rows intact. - Combine dense vector embeddings (e.g. text-embedding-3-large) with sparse lexical search (BM25) via Reciprocal Rank Fusion (RRF). - Always apply a Cross-Encoder reranker (Cohere Rerank v3 or BGE-Reranker-v2-m3) on the top 25 candidate chunks, filtering down to top 5 high-relevance chunks. - Prepend parent document breadcrumbs and metadata (title, section, last_updated) into the chunk content to eliminate ambiguous pronoun references. - Enforce relevance score thresholds (e.g. rerank_score >= 0.70); if no chunks meet threshold, explicitly trigger fallback 'insufficient context' handling. ## Bad vs Good Implementations ### Production RAG Retrieval with Semantic Chunking and Cross-Encoder Reranking #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Fixed character splitting slices sentences and code blocks mid-statement. Pure vector similarity searches frequently retrieve semantically related but factually irrelevant chunks, causing hallucination in generated answers. ```python # FATAL: Fixed character splitting with raw top-k vector lookup from langchain_text_splitters import CharacterTextSplitter # Broken: splits sentences and code blocks in half splitter = CharacterTextSplitter(chunk_size=500, chunk_overlap=50) chunks = splitter.split_text(raw_document) # Raw vector lookup without reranker - top matches are frequently irrelevant semantic duplicates results = vector_store.similarity_search(query, k=5) prompt = f"Context: {results}\nQuestion: {query}" ``` #### Production Standard (Correct) > **Standard**: Preserves structural context using markdown header hierarchies, retrieves candidate breadth via hybrid search, and scores deep relevance using a Cross-Encoder to guarantee only factually pertinent chunks reach the LLM. ```python from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter from sentence_transformers import CrossEncoder # 1. Structural Markdown Hierarchy Splitting headers_to_split_on = [("#", "h1"), ("##", "h2"), ("###", "h3")] markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on) header_splits = markdown_splitter.split_text(raw_document) text_splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=150) chunks = text_splitter.split_documents(header_splits) # 2. Hybrid Retrieval: Retrieve Top-20 candidates from Vector + BM25 candidates = hybrid_retriever.get_relevant_documents(query, limit=20) # 3. Cross-Encoder Second-Stage Reranker reranker = CrossEncoder("BAAI/bge-reranker-large") pairs = [[query, doc.page_content] for doc in candidates] scores = reranker.predict(pairs) # Rank and filter strictly by confidence threshold ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True) filtered_context = [doc.page_content for doc, score in ranked[:5] if score > 0.65] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Enterprise RAG Pipeline: Semantic Chunking & Cross-Encoder Reranking

Enterprise RAG Pipeline: Semantic Chunking & Cross-Encoder Reranking defines an industrial-grade behavioral standard for backend engineering. Production architecture for Retrieval-Augmented Generation (RAG). Replaces arbitrary character splitting with semantic Markdown header hierarchy chunking, hybrid vector + BM25 keyword search with Reciprocal Rank Fusion (RRF), and Cross-Encoder reranking (Cohere / BGE) to eliminate LLM hallucinations. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: AI uses fixed chunk_size=500 character splitting cutting code blocks and sentences in half, passes raw top-k vector similarity scores, and prompts LLMs with fragmented noise. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	AI uses fixed chunk_size=500 character splitting cutting code blocks and sentences in half, passes raw top-k vector similarity scores, and prompts LLMs with fragmented noise.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Use structural/markdown-aware splitters (e.g. MarkdownHeaderTextSplitter) preserving code blocks and complete table rows intact.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.