Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard | AgentRules.org

Rules Stacks Clients Auditor Pruner Self-Heal RFCs Benchmark CLI CI/CD Authors Submit

This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →

backendcritical PriorityID: kafka-consumer-commit-sync-anti-pattern

Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard

Production event streaming standard for Apache Kafka consumers. Mandates asynchronous offset commits (commitAsync) during high-throughput polling loops, coupled with synchronous commits (commitSync) on partition revocation listeners and shutdown hooks to prevent duplicate message replay and broker commit backpressure.

Prohibited Anti-Pattern: AI calls commitSync() synchronously after every single processed message, crushing consumer throughput from 50,000 msg/sec down to 100 msg/sec.

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard - Production event streaming standard for Apache Kafka consumers. Mandates asynchronous offset commits (commitAsync) during high-throughput polling loops, coupled with synchronous commits (commitSync) on partition revocation listeners and shutdown hooks to prevent duplicate message replay and broker commit backpressure.
globs: **/*kafka*.py, **/*consumer*.go, **/*Kafka*.java
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/kafka-consumer-commit-sync-anti-pattern"
---

# Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard

> **Severity**: CRITICAL | **Category**: backend
> **Anti-Pattern Warning**: AI calls commitSync() synchronously after every single processed message, crushing consumer throughput from 50,000 msg/sec down to 100 msg/sec.

## System Constraints
You are an Apache Kafka Distributed Streaming Architect. NEVER call commitSync() synchronously on every message inside consumer poll loops; it blocks the consumer thread waiting for broker round-trips. Use commitAsync() in the loop with an offset callback, and reserve commitSync() for ConsumerRebalanceListener and graceful shutdown.

## Strict Guidelines
- MUST disable enable.auto.commit = false to take control of processing guarantees.
- MUST NOT call commitSync() inside per-record loops (destroys consumer network batching and throughput).
- MUST use commitAsync(callback) inside polling batches, handling retry logic only when offsets monotonically increase.
- MUST implement a ConsumerRebalanceListener that executes commitSync() during onPartitionsRevoked.
- MUST call commitSync() in the application shutdown/finally block before closing the consumer.

## Bad vs Good Implementations
### Asynchronous Batch Commit with Rebalance Sync in Kafka

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Calling synchronous commit on every message forces the consumer thread to wait for a network round-trip to the Kafka coordinator broker for every item, collapsing throughput.
```python
# FATAL: Synchronous commit on every message throttles throughput by 500x
for message in consumer:
  process_record(message)
  # CRITICAL BOTTLENECK: Blocks 20ms on network round-trip for EVERY message!
  consumer.commit()
```

#### Production Standard (Correct)
> **Standard**: Processes records in batches and commits asynchronously without blocking the event loop. The finally block guarantees a final synchronous commit before consumer disconnection.
```python
# Resilient asynchronous batch commit pattern in Kafka
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'orders',
    enable_auto_commit=False,
    group_id='order_processors'
)

try:
    while not should_stop:
        records_batch = consumer.poll(timeout_ms=1000)
        for topic_partition, records in records_batch.items():
            for record in records:
                process_record(record)
        
        # Non-blocking async commit for batch
        consumer.commit_async()
finally:
    # Synchronous guaranteed flush before teardown
    try:
        consumer.commit()
    finally:
        consumer.close()
```

Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Kafka Cluster & Consumer Lag MCP

Monitors partition lag, rebalance events, and uncommitted offset windows.

Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

askills: Kafka Consumer Offset Tuner

Scaffolds high-throughput Kafka consumers with commitAsync batching and rebalance hooks.

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard

Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard defines an industrial-grade behavioral standard for backend engineering. Production event streaming standard for Apache Kafka consumers. Mandates asynchronous offset commits (commitAsync) during high-throughput polling loops, coupled with synchronous commits (commitSync) on partition revocation listeners and shutdown hooks to prevent duplicate message replay and broker commit backpressure. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: AI calls commitSync() synchronously after every single processed message, crushing consumer throughput from 50,000 msg/sec down to 100 msg/sec. Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	AI calls commitSync() synchronously after every single processed message, crushing consumer throughput from 50,000 msg/sec down to 100 msg/sec.	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

MUST disable enable.auto.commit = false to take control of processing guarantees.

MUST disable enable.auto.commit = false to take control of processing guarantees.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

2

MUST NOT call commitSync() inside per-record loops (destroys consumer network batching and throughput).

MUST NOT call commitSync() inside per-record loops (destroys consumer network batching and throughput).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

3

MUST use commitAsync(callback) inside polling batches, handling retry logic only when offsets monotonically increase.

MUST use commitAsync(callback) inside polling batches, handling retry logic only when offsets monotonically increase.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

4

MUST implement a ConsumerRebalanceListener that executes commitSync() during onPartitionsRevoked.

MUST implement a ConsumerRebalanceListener that executes commitSync() during onPartitionsRevoked.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

5

MUST call commitSync() in the application shutdown/finally block before closing the consumer.

MUST call commitSync() in the application shutdown/finally block before closing the consumer.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.

Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.

Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is Apache Kafka Manual Offset Commit Discipline & Rebalance Listener Standard critical for AI coding assistants?

Production event streaming standard for Apache Kafka consumers. Mandates asynchronous offset commits (commitAsync) during high-throughput polling loops, coupled with synchronous commits (commitSync) on partition revocation listeners and shutdown hooks to prevent duplicate message replay and broker commit backpressure. Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (AI calls commitSync() synchronously after every single processed message, crushing consumer throughput from 50,000 msg/sec down to 100 msg/sec.) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related BACKEND Rules

View all rules →

Fastify 5 Schema Compilers & Encapsulation Production Standard

Fastify relies on lexical encapsulation contexts to sandbox decorators, hooks, and compiled validation/serialization schemas without memory leaks or cross-tenant bleed. In Fastify 5, misuse of global compilers or failing to encapsulate plugin boundaries causes compiled `fast-json-stringify` serializers and Ajv validators to either crash on duplicate IDs, leak private data across sibling scopes, or bypass JIT compilation optimizations. Production services must enforce strict encapsulation hierarchies and type-safe schema compilation across every routing subtree.

Bun Embedded SQLite WAL Mode & Transaction Synchronization Standard

Production database standard for bun:sqlite. Mandates enabling Write-Ahead Logging (PRAGMA journal_mode = WAL), busy timeouts, and using db.transaction() for atomic operations without lock starvation.

Drizzle ORM Batch API & Interactive Transaction Isolation Standard

Production database standard for Drizzle ORM (PostgreSQL & MySQL). Mandates using db.batch() over HTTP drivers to prevent round-trip waterfalls, explicit db.transaction() with serializable/read-committed isolation, and strictly avoids unawaited query promises.

1 Rule in Stack

Apache