Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard

--- description: Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard - Production message queue standard for Redis Streams. Mandates consumer group creation with explicit MKSTREAM, manual message acknowledgment (XACK), Pending Entries List (PEL) monitoring with XPENDING, and poison pill auto-recovery using XAUTOCLAIM. globs: **/queue/**/*.py, **/stream/**/*.ts, **/workers/**/*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/redis-stream-consumer-group-pel" --- # Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: AI reads Redis Streams using XREAD without consumer groups or forgets XACK, causing unacknowledged messages to accumulate indefinitely in memory. ## System Constraints You are a Redis Distributed Systems and Messaging Architect. When using Redis Streams for distributed task distribution, NEVER use simple XREAD without consumer groups. Always use XREADGROUP with explicit consumer identity, acknowledge processed messages via XACK, and run a background task reclaiming abandoned messages via XAUTOCLAIM. ## Strict Guidelines - MUST create consumer groups using XGROUP CREATE stream group $ MKSTREAM to avoid stream not found errors. - MUST acknowledge successfully processed messages explicitly using XACK stream group id. - MUST implement XAUTOCLAIM stream group consumer 60000 0-0 COUNT 100 to claim tasks from crashed workers. - MUST track retry delivery counts in PEL (Pending Entries List) and route poison messages to a Dead Letter Stream (DLQ) after 5 delivery attempts. - MUST configure stream length capping with XADD stream MAXLEN ~ 500000 to prevent unbounded Redis RAM growth. ## Bad vs Good Implementations ### Redis Stream Resilient Consumer with XACK and XAUTOCLAIM in Python #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: XREAD without consumer groups broadcasts messages to every worker instead of load-balancing, and provides zero delivery tracking or crash recovery mechanisms. ```python # FATAL: Unsafe stream reading without consumer group or acknowledgments import redis r = redis.Redis() while True: # Broken: No consumer group, multiple workers receive duplicate messages, no retry on crash! messages = r.xread({"events": "$"}, block=5000) for stream, entries in messages: for msg_id, data in entries: process(data) ``` #### Production Standard (Correct) > **Standard**: Uses consumer groups for load-balancing across worker instances. Implements XAUTOCLAIM to safely recover tasks from crashed nodes and calls XACK only upon successful completion. ```python import redis r = redis.Redis() # Ensure consumer group exists try: r.xgroup_create("events", "worker_group", id="$", mkstream=True) except redis.exceptions.ResponseError as e: if "BUSYGROUP" not in str(e): raise def worker_loop(consumer_name: str): while True: # 1. First reclaim abandoned tasks from dead workers (idle > 60s) reclaimed = r.xautoclaim("events", "worker_group", consumer_name, min_idle_time=60000, start_id="0-0", count=10) process_and_ack_batch(reclaimed[1]) # 2. Read new messages for this specific consumer messages = r.xreadgroup("worker_group", consumer_name, {"events": ">"}, count=10, block=2000) for stream, entries in messages: for msg_id, data in entries: try: handle_event(data) # 3. Explicitly ACK to remove from Pending Entries List r.xack("events", "worker_group", msg_id) except Exception: # Leaves in PEL for retry or dead-letter routing log.error("Failed processing event %s", msg_id) ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard

Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard defines an industrial-grade behavioral standard for backend engineering. Production message queue standard for Redis Streams. Mandates consumer group creation with explicit MKSTREAM, manual message acknowledgment (XACK), Pending Entries List (PEL) monitoring with XPENDING, and poison pill auto-recovery using XAUTOCLAIM. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: AI reads Redis Streams using XREAD without consumer groups or forgets XACK, causing unacknowledged messages to accumulate indefinitely in memory. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	AI reads Redis Streams using XREAD without consumer groups or forgets XACK, causing unacknowledged messages to accumulate indefinitely in memory.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

MUST create consumer groups using XGROUP CREATE stream group $ MKSTREAM to avoid stream not found errors.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.