Redis 7.4 Field-Level Hash TTL Invalidation and Stream PEL Retention Guardrails

--- description: Redis 7.4 Field-Level Hash TTL Invalidation and Stream PEL Retention Guardrails - Mandates deterministic key lifecycle management using Redis 7.4 per-field hash expiration (HEXPIRE/HPEXPIRE) to eliminate key-space fragmentation and expensive multi-key invalidations. Enforces strict idempotent processing and proactive Pending Entries List (PEL) reclamation (XAUTOCLAIM) for Redis Streams to prevent unbounded memory bloat and silent message drops under volatile eviction policies. globs: src/**/*.ts, src/**/*.js, lib/**/*.ts, services/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/redis-7-4-hash-ttl-stream-pel-lifecycle" --- # Redis 7.4 Field-Level Hash TTL Invalidation and Stream PEL Retention Guardrails > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded Consumer Group PEL Growth and Whole-Key Invalidation with Stale Hash Fields ## System Constraints You are a Principal Backend Architect enforcing strict invariants for Redis 7.4+ data structures. You must mandate atomic per-field hash expiration (HEXPIRE/HPEXPIRE) for compound entity caches, eliminate lingering Pending Entries List (PEL) records in Redis Streams via automated XAUTOCLAIM dead-letter workflows, and safeguard cluster memory under maxmemory eviction pressures. ## Strict Guidelines - Engineers MUST ALWAYS use Redis 7.4 `HEXPIRE`, `HPEXPIRE`, `HEXPIREAT`, or `HPEXPIREAT` with explicit field lists for sub-entity TTLs rather than decomposing hashes into top-level flat string keys or clearing entire parent entities with `DEL`. - Engineers MUST NEVER assume that unacknowledged stream consumer group messages will self-evict; all stream consumer groups MUST implement background worker loops using `XAUTOCLAIM` with explicit `min-idle-time` thresholds to reclaim or Dead-Letter-Queue (DLQ) stuck entries from the Pending Entries List (PEL). - Engineers MUST ALWAYS execute `XACK` strictly after downstream side-effects succeed, verifying that the acknowledgment count equals expected processed IDs before moving offsets. - Services running under `volatile-lru`, `volatile-lfu`, or `volatile-ttl` MUST NOT rely solely on Redis key eviction for stream retention; stream lengths MUST be explicitly bounded at ingest time using `XADD ... MAXLEN ~ <limit>` to guarantee stream memory isolation. - Engineers MUST verify client library compatibility with Redis 7.4 RESP3 field-level TTL semantics; fallback pipelines or explicit Lua scripts MUST be used only when native `HEXPIRE` command support is absent in the client driver. ## Bad vs Good Implementations ### Hash Field Expiration and Stream Consumer Group PEL Auto-Claim Lifecycle #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Splitting compound entities into multiple flat keys introduces memory overhead and distributed consistency hazards. Furthermore, processing stream messages without an `XAUTOCLAIM` or PEL recovery routine means unacknowledged messages remain trapped in the Pending Entries List indefinitely when a consumer fails, leaking RAM and dropping business events. ```typescript import Redis from 'ioredis'; const redis = new Redis(); // ANTI-PATTERN: Decomposing entity attributes into separate keys just for TTL, // and failing to claim/prune unacknowledged messages in the consumer group PEL. export async function updateSessionCache(sessionId: string, authData: string, cartData: string) { // Creates key bloat, multi-roundtrip overhead, and risk of orphaned keys await redis.set(`session:${sessionId}:auth`, authData, 'EX', 900); await redis.set(`session:${sessionId}:cart`, cartData, 'EX', 3600); } export async function processStreamEvents(streamKey: string, group: string, consumer: string) { // Messages read here enter the consumer group PEL but are never claimed if a worker crashes const events = await redis.xreadgroup('GROUP', group, consumer, 'STREAMS', streamKey, '>'); if (!events) return; for (const [, entries] of events) { for (const [id, fields] of entries) { try { // Process event await handleEvent(fields); await redis.xack(streamKey, group, id); } catch (err) { // Error thrown: message remains permanently stuck in PEL without a DLQ or retry policy! console.error(`Failed processing ${id}`); } } } } declare function handleEvent(fields: string[]): Promise<void>; ``` #### Production Standard (Correct) > **Standard**: The fix preserves low key cardinality by keeping sessions in one Hash and utilizing Redis 7.4's native `HEXPIRE` to apply discrete field-level TTLs. Stream processing reliably cleans the Pending Entries List (PEL) via `XAUTOCLAIM`, bounds stream memory using `MAXLEN ~`, and safely routes poision-pill events to a Dead-Letter Queue (DLQ) after retry limits are exceeded. ```typescript import Redis from 'ioredis'; const redis = new Redis(); // PRODUCTION FIX: Use Redis 7.4 HEXPIRE for granular field TTLs within a single hash, // cap stream size with MAXLEN, and maintain an XAUTOCLAIM worker for stuck PEL items. export async function updateSessionCache(sessionId: string, authData: string, cartData: string) { const key = `session:${sessionId}`; const pipeline = redis.pipeline(); pipeline.hset(key, { auth: authData, cart: cartData }); // Redis 7.4 HEXPIRE syntax: HEXPIRE key seconds FIELDS numfields field [field ...] pipeline.sendCommand(new Redis.Command('HEXPIRE', [key, '900', 'FIELDS', '1', 'auth'])); pipeline.sendCommand(new Redis.Command('HEXPIRE', [key, '3600', 'FIELDS', '1', 'cart'])); await pipeline.exec(); } export async function processAndReclaimStream( streamKey: string, group: string, consumer: string, dlqKey: string ) { const MIN_IDLE_TIME_MS = 60000; const MAX_DELIVERY_ATTEMPTS = 5; // 1. Proactively claim stuck messages from dead consumers (PEL Maintenance) const claimResult = await redis.xautoclaim( streamKey, group, consumer, MIN_IDLE_TIME_MS, '0-0', 'COUNT', 50 ) as [string, [string, string[]][], string[]]; const [, claimedEntries] = claimResult; for (const [id, fields] of claimedEntries) { await processWithDLQ(streamKey, group, id, fields, dlqKey, MAX_DELIVERY_ATTEMPTS); } // 2. Consume fresh messages const response = await redis.xreadgroup('GROUP', group, consumer, 'COUNT', 50, 'STREAMS', streamKey, '>'); if (!response) return; for (const [, entries] of response) { for (const [id, fields] of entries) { await processWithDLQ(streamKey, group, id, fields, dlqKey, MAX_DELIVERY_ATTEMPTS); } } } async function processWithDLQ( stream: string, group: string, id: string, fields: string[], dlq: string, maxAttempts: number ) { try { await handleEvent(fields); const ackCount = await redis.xack(stream, group, id); if (ackCount !== 1) { console.warn(`Idempotency note: Message ${id} was already acknowledged or missing from PEL.`); } } catch (err) { // Inspect pending summary for delivery count const pendingInfo = await redis.xpending(stream, group, id, id, 1) as [string, string, number, number][]; const deliveryCount = pendingInfo[0]?.[3] ?? 1; if (deliveryCount >= maxAttempts) { await redis.pipeline() .xadd(dlq, 'MAXLEN', '~', 100000, '*', 'original_id', id, 'payload', JSON.stringify(fields)) .xack(stream, group, id) .exec(); } } } declare function handleEvent(fields: string[]): Promise<void>; ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Redis 7.4 Field-Level Hash TTL Invalidation and Stream PEL Retention Guardrails

Redis 7.4 Field-Level Hash TTL Invalidation and Stream PEL Retention Guardrails defines an industrial-grade behavioral standard for backend engineering. Mandates deterministic key lifecycle management using Redis 7.4 per-field hash expiration (HEXPIRE/HPEXPIRE) to eliminate key-space fragmentation and expensive multi-key invalidations. Enforces strict idempotent processing and proactive Pending Entries List (PEL) reclamation (XAUTOCLAIM) for Redis Streams to prevent unbounded memory bloat and silent message drops under volatile eviction policies. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Consumer Group PEL Growth and Whole-Key Invalidation with Stale Hash Fields Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Consumer Group PEL Growth and Whole-Key Invalidation with Stale Hash Fields	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS use Redis 7.4 `HEXPIRE`, `HPEXPIRE`, `HEXPIREAT`, or `HPEXPIREAT` with explicit field lists for sub-entity TTLs rather than decomposing hashes into top-level flat string keys or clearing entire parent entities with `DEL`.

Engineers MUST ALWAYS use Redis 7.4 `HEXPIRE`, `HPEXPIRE`, `HEXPIREAT`, or `HPEXPIREAT` with explicit field lists for sub-entity TTLs rather than decomposing hashes into top-level flat string keys or clearing entire parent entities with `DEL`.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.