Kafka Consumer Commit + Redis Streams PEL + ClickHouse Deduplication + Go errgroup

# Combined .cursorrules generated by AgentRules.org (https://agentrules.org)
# Specification Version: 2026.10-Industrial | Licensed under CC BY 4.0
# Stack: Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard, Redis Streams Consumer Groups, PEL & Poison Pill Recovery Standard, ClickHouse ReplacingMergeTree & FINAL Query Argument Optimization, Go errgroup Context-Aware Parallel Fan-Out & First-Error Abort Standard, Docker Compose v2 service_healthy Dependency Orchestration Standard
# Free for in-repo use. Attribution to agentrules.org required if cited or republished.

# ==============================================================================
# SECTION 1: APACHE KAFKA MANUAL OFFSET COMMIT DISCIPLINE & REBALANCE LISTENER STANDARD
# ==============================================================================
## Directive
You are an Apache Kafka Distributed Streaming Architect. NEVER call commitSync() synchronously on every message inside consumer poll loops; it blocks the consumer thread waiting for broker round-trips. Use commitAsync() in the loop with an offset callback, and reserve commitSync() for ConsumerRebalanceListener and graceful shutdown.

## Strict Guidelines
1. MUST disable enable.auto.commit = false to take control of processing guarantees.
2. MUST NOT call commitSync() inside per-record loops (destroys consumer network batching and throughput).
3. MUST use commitAsync(callback) inside polling batches, handling retry logic only when offsets monotonically increase.
4. MUST implement a ConsumerRebalanceListener that executes commitSync() during onPartitionsRevoked.
5. MUST call commitSync() in the application shutdown/finally block before closing the consumer.


# ==============================================================================
# SECTION 2: REDIS STREAMS CONSUMER GROUPS, PEL & POISON PILL RECOVERY STANDARD
# ==============================================================================
## Directive
You are a Redis Distributed Systems and Messaging Architect. When using Redis Streams for distributed task distribution, NEVER use simple XREAD without consumer groups. Always use XREADGROUP with explicit consumer identity, acknowledge processed messages via XACK, and run a background task reclaiming abandoned messages via XAUTOCLAIM.

## Strict Guidelines
1. MUST create consumer groups using XGROUP CREATE stream group $ MKSTREAM to avoid stream not found errors.
2. MUST acknowledge successfully processed messages explicitly using XACK stream group id.
3. MUST implement XAUTOCLAIM stream group consumer 60000 0-0 COUNT 100 to claim tasks from crashed workers.
4. MUST track retry delivery counts in PEL (Pending Entries List) and route poison messages to a Dead Letter Stream (DLQ) after 5 delivery attempts.
5. MUST configure stream length capping with XADD stream MAXLEN ~ 500000 to prevent unbounded Redis RAM growth.


# ==============================================================================
# SECTION 3: CLICKHOUSE REPLACINGMERGETREE & FINAL QUERY ARGUMENT OPTIMIZATION
# ==============================================================================
## Directive
You are a ClickHouse High-Performance Analytics Engineer. In ClickHouse, ReplacingMergeTree only guarantees background deduplication; duplicates can exist until background merge occurs. NEVER use raw SELECT * FROM table FINAL in production dashboards without partition bounds. Use argMax() or argMin() aggregations for instant sub-millisecond deduplication.

## Strict Guidelines
1. MUST define ReplacingMergeTree with an explicit numeric or timestamp version column: ReplacingMergeTree(ver).
2. MUST NOT execute SELECT ... FINAL on queries without restrictive WHERE partition filters (FINAL forces multi-threaded block merging across the entire dataset).
3. SHOULD use SELECT id, argMax(status, ver) as status FROM table GROUP BY id for 10x faster deduplicated reads over FINAL.
4. MUST order the table by primary lookup keys: ORDER BY (tenant_id, id).
5. SHOULD utilize OPTIMIZE TABLE table FINAL only during off-peak maintenance windows, never on continuous ingestion cycles.


# ==============================================================================
# SECTION 4: GO ERRGROUP CONTEXT-AWARE PARALLEL FAN-OUT & FIRST-ERROR ABORT STANDARD
# ==============================================================================
## Directive
You are a Principal Go Systems Architect. When executing parallel concurrent operations (such as multi-service fan-outs or batch validations), DO NOT use primitive sync.WaitGroup with manual error channels. Use errgroup.WithContext(ctx) to propagate cancellation to sibling tasks upon first error.

## Strict Guidelines
1. MUST initialize errgroup with g, ctx := errgroup.WithContext(parentCtx).
2. MUST respect ctx.Done() inside all spawned worker functions to abort work when a sibling encounters an error.
3. MUST cap concurrent goroutine spawning using g.SetLimit(maxWorkers) to prevent unbounded thread/memory explosion.
4. MUST NOT read or write shared maps or slice indexes concurrently without synchronization inside errgroup functions.
5. MUST verify that g.Wait() error is returned and checked before consuming aggregated result datasets.


# ==============================================================================
# SECTION 5: DOCKER COMPOSE V2 SERVICE_HEALTHY DEPENDENCY ORCHESTRATION STANDARD
# ==============================================================================
## Directive
You are a Container Orchestration Specialist. In Docker Compose v2, NEVER declare depends_on as a simple string list (depends_on: ['db']). Docker Compose starts containers in parallel; a container marked as 'started' does not mean PostgreSQL is ready to accept connections. Always define explicit healthcheck blocks and set condition: service_healthy.

## Strict Guidelines
1. MUST define an explicit healthcheck on database and cache services (e.g. pg_isready -U postgres).
2. MUST configure depends_on with condition: service_healthy on all consuming web and API services.
3. MUST set appropriate interval (5s), timeout (3s), and retries (5) on healthcheck definitions.
4. MUST NOT write arbitrary sleep loops inside application startup scripts as a substitute for healthchecks.
5. SHOULD use modern top-level compose.yaml naming convention (Compose Specification v2).