pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Deploying unquantized 32-bit floating-point HNSW vector indexes without memory caps, causing catastrophic PostgreSQL worker process OOM evictions under concurrency) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard

--- description: pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard - Enforces strict memory ceiling budgets and SIMD-aligned scalar quantization (halfvec/int8) for pgvector 0.8+ HNSW indexes to prevent unconstrained RAM exhaustion and kernel OOM killer termination during high-concurrency searches. Mandates deterministic index sizing validation, bounded build-time maintenance memory, and scoped runtime search constraints. globs: src/db/**/*.ts, src/migrations/**/*.ts, migrations/**/*.sql, src/**/*.service.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/pgvector-hnsw-quantization-memory-ceiling" --- # pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Deploying unquantized 32-bit floating-point HNSW vector indexes without memory caps, causing catastrophic PostgreSQL worker process OOM evictions under concurrency ## System Constraints You are a Principal Database Architect enforcing pgvector 0.8 HNSW index quantization and memory safety standards. High-dimensional vector similarity search without scalar quantization or dimension pruning rapidly causes severe page swapping and Linux OOM killer termination of PostgreSQL instances under concurrent load. You must enforce halfvec/scalar quantization, bounded index parameters (m, ef_construction), explicit maintenance_work_mem sizing, and bounded hnsw.ef_search runtime settings across all schema migrations, DDL scripts, and ORM query builders. ## Strict Guidelines - Engineers MUST ALWAYS cast high-dimensional embeddings (>768 dimensions) to halfvec (fp16) or enforce scalar quantization for HNSW indexing to cut memory footprint by at least 50% and maximize SIMD throughput. - Engineers MUST NOT execute CREATE INDEX ... USING hnsw without establishing deterministic maintenance_work_mem ceilings inside an isolated build transaction. - Code SHOULD enforce explicit HNSW graph parameters (m <= 24, ef_construction <= 128) calibrated to the available shared_buffers and system memory ceiling. - Services MUST bound the runtime hnsw.ef_search parameter (e.g., SET LOCAL hnsw.ef_search = 64) per transaction to avoid worker memory amplification and CPU starvation under high QPS. - Components NEVER execute unbounded vector similarity queries without verifyIndexUsage guards or fall back to brute-force sequential scans on multi-million row tables. ## Bad vs Good Implementations ### HNSW Index Creation and Query Execution with Memory Bounds #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Creating an HNSW index on 1536-dimension fp32 vectors with default settings consumes massive amounts of RAM (approx. 1.5-2 KB per vector in graph overhead + fp32 data). Without quantization and explicit maintenance_work_mem bounds, the index creation process can exhaust available database memory. Under high concurrency, unconstrained search parameters further risk worker crashes and page thrashing. ```typescript import { Pool } from 'pg'; export async function setupEmbeddingsTable(pool: Pool) { // Anti-pattern: fp32 unquantized HNSW index with default unbounded build memory // and default graph parameters on high-dimensional vectors (e.g., 1536 dims). // On large datasets (1M+ rows), this consumes >10GB RAM and triggers OOM kills. await pool.query(` CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS documents ( id UUID PRIMARY KEY, embedding vector(1536) ); -- DANGEROUS: Unbounded maintenance_work_mem, no halfvec/quantization, unconstrained m/ef CREATE INDEX IF NOT EXISTS idx_documents_embedding ON documents USING hnsw (embedding vector_cosine_ops); `); } export async function searchSimilar(pool: Pool, queryEmbedding: number[]) { // Anti-pattern: Default ef_search can monopolize CPU/RAM during concurrent bursts const res = await pool.query(` SELECT id, embedding <=> $1::vector AS distance FROM documents ORDER BY distance ASC LIMIT 10; `, [JSON.stringify(queryEmbedding)]); return res.rows; } ``` #### Production Standard (Correct) > **Standard**: By utilizing pgvector 0.8's native halfvec type, vector storage is halved, enabling SIMD AVX-512/NEON-accelerated distance comparisons with negligible recall degradation. Explicitly scoping maintenance_work_mem guards the index build phase against database OOM killer evictions, while capping m=16, ef_construction=64, and setting runtime hnsw.ef_search=64 guarantees deterministic CPU and memory ceilings during concurrent searches. ```typescript import { Pool, PoolClient } from 'pg'; const EMBEDDING_DIM = 1536; const MAX_SAFE_EF_SEARCH = 64; export async function setupEmbeddingsTableSafe(client: PoolClient): Promise<void> { // Invariant: Enforce halfvec (fp16) scalar quantization to reduce index size by 50%, // bound build memory ceiling locally, and cap HNSW graph parameters (m=16, ef_construction=64). await client.query('BEGIN'); try { await client.query('CREATE EXTENSION IF NOT EXISTS vector;'); await client.query(` CREATE TABLE IF NOT EXISTS documents ( id UUID PRIMARY KEY, embedding halfvec(${EMBEDDING_DIM}) ); `); // Bound index build memory ceiling to avoid OOM during iterative build await client.query("SET LOCAL maintenance_work_mem = '1GB';"); // Build quantized HNSW index using SIMD-optimized halfvec_cosine_ops with bounded m & ef_construction await client.query(` CREATE INDEX IF NOT EXISTS idx_documents_halfvec_embedding ON documents USING hnsw (embedding halfvec_cosine_ops) WITH (m = 16, ef_construction = 64); `); await client.query('COMMIT'); } catch (err) { await client.query('ROLLBACK'); throw err; } } export async function searchSimilarSafe(client: PoolClient, queryEmbedding: number[]): Promise<{ id: string; distance: number }[]> { // Invariant: Isolate search parameter hnsw.ef_search per session/transaction to strictly limit per-query memory and SIMD cycle budget await client.query('BEGIN'); try { await client.query(`SET LOCAL hnsw.ef_search = ${MAX_SAFE_EF_SEARCH};`); const res = await client.query<{ id: string; distance: number }>(` SELECT id, (embedding <=> $1::halfvec) AS distance FROM documents ORDER BY distance ASC LIMIT 10; `, [JSON.stringify(queryEmbedding)]); await client.query('COMMIT'); return res.rows; } catch (err) { await client.query('ROLLBACK'); throw err; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard

pgvector 0.8 HNSW Scalar Quantization & Memory Ceiling Standard defines an industrial-grade behavioral standard for backend engineering. Enforces strict memory ceiling budgets and SIMD-aligned scalar quantization (halfvec/int8) for pgvector 0.8+ HNSW indexes to prevent unconstrained RAM exhaustion and kernel OOM killer termination during high-concurrency searches. Mandates deterministic index sizing validation, bounded build-time maintenance memory, and scoped runtime search constraints. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Deploying unquantized 32-bit floating-point HNSW vector indexes without memory caps, causing catastrophic PostgreSQL worker process OOM evictions under concurrency Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Deploying unquantized 32-bit floating-point HNSW vector indexes without memory caps, causing catastrophic PostgreSQL worker process OOM evictions under concurrency	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS cast high-dimensional embeddings (>768 dimensions) to halfvec (fp16) or enforce scalar quantization for HNSW indexing to cut memory footprint by at least 50% and maximize SIMD throughput.

Engineers MUST ALWAYS cast high-dimensional embeddings (>768 dimensions) to halfvec (fp16) or enforce scalar quantization for HNSW indexing to cut memory footprint by at least 50% and maximize SIMD throughput.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.