HashiCorp Vault Ephemeral Database Credentials and Lease-Aware Connection Pool Lifecycle

--- description: HashiCorp Vault Ephemeral Database Credentials and Lease-Aware Connection Pool Lifecycle - Production services interacting with relational datastores must consume short-lived ephemeral credentials issued dynamically by HashiCorp Vault with zero persistent root or shared credentials in configuration. Application connection pools must actively monitor lease TTLs, schedule proactive renewals before expiry, and gracefully drain and refresh database connection pools upon credential rotation without service downtime. globs: src/**/database/**/*.{ts,js}, src/**/vault/**/*.{ts,js}, src/**/config/**/*.{ts,js}, infra/**/*.tf alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/vault-dynamic-db-credentials-rotation-guardrail" --- # HashiCorp Vault Ephemeral Database Credentials and Lease-Aware Connection Pool Lifecycle > **Severity**: CRITICAL | **Category**: security > **Anti-Pattern Warning**: Static Database Credentials and Non-Rotating Long-Lived Connection Pools Leading to Outages or Compromise ## System Constraints You are a Principal Security Architect enforcing HashiCorp Vault Dynamic Database Credentials Rotation. You must mandate zero static database credentials, strict lease lifetime adherence, proactive credential renewal at 60-70% TTL, graceful connection pool recreation upon credential revocation, and seamless failover to newly issued leases without dropping active client requests. ## Strict Guidelines - Engineers MUST NEVER configure static database usernames or passwords in environment variables, configuration files, Kubernetes ConfigMaps, or container secrets. - Services MUST request short-lived dynamic credentials through the Vault Database Secrets Engine API (v1/database/creds/:role) at application startup and rotation intervals. - Connection pool managers MUST actively track the Vault lease `lease_duration` and trigger renewal operations at 60-70% of lease elapsed time via `sys/leases/renew`. - Services MUST implement a dual-pool or graceful swap pattern to instantiate a new connection pool with fresh dynamic credentials and drain active connections before the previous lease expires or if renewal fails. - Engineers MUST configure Vault Database Roles with explicit `default_ttl` (max 1 hour) and `max_ttl` (max 24 hours), alongside automated root password rotation via `sys/leases/lookup` or automated Vault root rotation policies. - Database connection error handlers MUST capture authentication failure events (e.g., Postgres `28P01`) and trigger an immediate lease re-acquisition instead of unbounded reconnection attempts with expired credentials. ## Bad vs Good Implementations ### Dynamic Credential Lifecycle and Seamless Connection Pool Rotation with Vault #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Relies on persistent environment variables containing static database credentials. It lacks awareness of HashiCorp Vault ephemeral leases, does not handle credential rotation, and forces application downtime or security vulnerabilities when root or static credentials are leaked or manually rotated. ```typescript import { Pool } from 'pg'; // ANTI-PATTERN: Consuming long-lived/static DB credentials from environment // No lease tracking, no rotation handling, credentials remain persistent indefinitely export const dbPool = new Pool({ host: process.env.DB_HOST, port: Number(process.env.DB_PORT || 5432), user: process.env.DB_USER, password: process.env.DB_PASSWORD, database: process.env.DB_NAME, max: 20, idleTimeoutMillis: 30000, }); export async function executeQuery(queryText: string, params: unknown[]) { const client = await dbPool.connect(); try { return await client.query(queryText, params); } finally { client.release(); } } ``` #### Production Standard (Correct) > **Standard**: Fetches ephemeral database credentials directly from HashiCorp Vault, proactively renews leases at 60% duration, gracefully swaps active connection pools before credentials expire, and revokes prior leases while guaranteeing zero downtime for ongoing transactions. ```typescript import { Pool, PoolClient } from 'pg'; interface VaultDBCredentials { lease_id: string; lease_duration: number; renewable: boolean; data: { username: string; password: string; }; } export class DynamicDatabaseManager { private currentPool: Pool | null = null; private renewalTimeout: NodeJS.Timeout | null = null; private activeLeaseId: string | null = null; private isRotating = false; constructor( private readonly vaultClient: { request: (path: string, options?: any) => Promise<any> }, private readonly dbRole: string, private readonly host: string, private readonly databaseName: string ) {} public async initialize(): Promise<void> { await this.rotateCredentials(); } private async fetchDynamicCredentials(): Promise<VaultDBCredentials> { const response = await this.vaultClient.request(`/v1/database/creds/${this.dbRole}`); return response as VaultDBCredentials; } private async rotateCredentials(): Promise<void> { if (this.isRotating) return; this.isRotating = true; try { const creds = await this.fetchDynamicCredentials(); const nextPool = new Pool({ host: this.host, database: this.databaseName, user: creds.data.username, password: creds.data.password, max: 20, idleTimeoutMillis: 10000, }); // Verify connectivity before swapping const testClient = await nextPool.connect(); testClient.release(); const oldPool = this.currentPool; const oldLeaseId = this.activeLeaseId; this.currentPool = nextPool; this.activeLeaseId = creds.lease_id; // Schedule renewal at 60% of lease TTL const renewalDelayMs = creds.lease_duration * 1000 * 0.6; if (this.renewalTimeout) clearTimeout(this.renewalTimeout); this.renewalTimeout = setTimeout(() => this.renewOrRotate(creds.lease_id, creds.lease_duration), renewalDelayMs); // Gracefully terminate old pool connections without terminating current inflight work if (oldPool) { setTimeout(async () => { await oldPool.end(); if (oldLeaseId) { await this.vaultClient.request('/v1/sys/leases/revoke', { method: 'PUT', body: { lease_id: oldLeaseId }, }).catch(() => {}); } }, 15000); } } finally { this.isRotating = false; } } private async renewOrRotate(leaseId: string, duration: number): Promise<void> { try { const renewed = await this.vaultClient.request('/v1/sys/leases/renew', { method: 'PUT', body: { lease_id: leaseId, increment: duration }, }); const nextDelayMs = renewed.lease_duration * 1000 * 0.6; this.renewalTimeout = setTimeout(() => this.renewOrRotate(leaseId, renewed.lease_duration), nextDelayMs); } catch { // Fall back to complete rotation if lease renewal fails await this.rotateCredentials(); } } public async getClient(): Promise<PoolClient> { if (!this.currentPool) throw new Error('Database pool not initialized'); return this.currentPool.connect(); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: HashiCorp Vault Ephemeral Database Credentials and Lease-Aware Connection Pool Lifecycle

HashiCorp Vault Ephemeral Database Credentials and Lease-Aware Connection Pool Lifecycle defines an industrial-grade behavioral standard for security engineering. Production services interacting with relational datastores must consume short-lived ephemeral credentials issued dynamically by HashiCorp Vault with zero persistent root or shared credentials in configuration. Application connection pools must actively monitor lease TTLs, schedule proactive renewals before expiry, and gracefully drain and refresh database connection pools upon credential rotation without service downtime. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Static Database Credentials and Non-Rotating Long-Lived Connection Pools Leading to Outages or Compromise Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Static Database Credentials and Non-Rotating Long-Lived Connection Pools Leading to Outages or Compromise	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST NEVER configure static database usernames or passwords in environment variables, configuration files, Kubernetes ConfigMaps, or container secrets.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.