Kafka 3.9 KRaft Metadata Quorum & Exactly-Once Idempotent Producer Invariants

--- description: Kafka 3.9 KRaft Metadata Quorum & Exactly-Once Idempotent Producer Invariants - Enforces strict configuration and runtime transactional semantics for Kafka 3.9 ZooKeeper-less (KRaft) metadata quorums. Guarantees message deduplication via idempotent producer settings and prevents split-brain partition states during controller leader elections and cooperative sticky consumer group rebalances. globs: src/**/*.kafka.ts, src/**/kafka*.ts, src/**/producers/**/*.ts, src/**/consumers/**/*.ts, src/**/event-bus/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/kafka-3-9-kraft-idempotent-producer-standards" --- # Kafka 3.9 KRaft Metadata Quorum & Exactly-Once Idempotent Producer Invariants > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Non-idempotent producer configurations with blind retries and non-transactional commits causing data duplication during KRaft controller failover ## System Constraints You are a Principal Distributed Systems Engineer enforcing Kafka 3.9 KRaft and Idempotent Producer standards. Mandate zero data loss, strict idempotence, proper metadata quorum bootstrap handling, and transactional boundaries on message commit. ## Strict Guidelines - Engineers MUST ALWAYS enable 'enable.idempotence': true and set 'acks': 'all' (-1) on all Kafka producers to prevent message duplication and out-of-order writes during KRaft controller leader transitions. - Engineers MUST configure 'max.in.flight.requests.per.connection' to less than or equal to 5 when idempotence is enabled to preserve strict per-partition ordering. - Engineers MUST NOT rely on deprecated ZooKeeper configuration parameters (`zookeeper.connect`); all connection brokers must point to the unified KRaft broker/controller bootstrap servers. - Services MUST implement CooperativeStickyAssignor (`partition.assignment.strategy`: 'org.apache.kafka.clients.consumer.CooperativeStickyAssignor') to prevent stop-the-world rebalance storms in high-throughput consumer groups. - Transactional producers MUST wrap message batches and offset commits in explicit `beginTransaction()`, `sendOffsetsToTransaction()`, and `commitTransaction()` blocks, handling `ProducerFencedException` with immediate termination. ## Bad vs Good Implementations ### Configuring Idempotent Producer with KRaft Rebalance & Transactional Guardrails #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Configuring acks: 1 without enabling idempotent delivery guarantees duplicate records if network glitches occur or if a KRaft metadata election shifts partition leadership before replica synchronization. ```typescript import { Kafka } from 'kafkajs'; const kafka = new Kafka({ clientId: 'order-service', brokers: ['localhost:9092'], }); export const publishOrderEvents = async (orders: Array<{ id: string; payload: string }>) => { // Anti-Pattern: Missing idempotency and explicit ACKs const producer = kafka.producer(); await producer.connect(); for (const order of orders) { await producer.send({ topic: 'orders', messages: [{ key: order.id, value: order.payload }], acks: 1, // Broker ack without quorum replica confirmation; vulnerable to KRaft leader failover }); } }; ``` #### Production Standard (Correct) > **Standard**: Enforces Kafka 3.9 idempotent guarantees with transactional boundaries, transactionalId, acks: -1 (all ISR replicas), and bounded in-flight requests, preventing duplication and split-brain phenomena. ```typescript import { Kafka, Partitioners } from 'kafkajs'; const kafka = new Kafka({ clientId: 'order-service-producer', brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(','), retry: { initialRetryTime: 100, retries: 8, factor: 2, }, }); export const publishOrderEvents = async (orders: Array<{ id: string; payload: string }>) => { // Production Standard: Enforced idempotence, maxInFlight <= 5, and all acks const producer = kafka.producer({ idempotent: true, maxInFlightRequests: 5, createPartitioner: Partitioners.DefaultPartitioner, transactionalId: 'order-tx-producer-01', }); await producer.connect(); const transaction = await producer.transaction(); try { await transaction.send({ topic: 'orders', acks: -1, // Wait for all in-sync replicas messages: orders.map(order => ({ key: order.id, value: order.payload, headers: { 'kraft-quorum-sync': 'verified' }, })), }); await transaction.commit(); } catch (error) { await transaction.abort(); throw error; } finally { await producer.disconnect(); } }; ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Kafka 3.9 KRaft Metadata Quorum & Exactly-Once Idempotent Producer Invariants

Kafka 3.9 KRaft Metadata Quorum & Exactly-Once Idempotent Producer Invariants defines an industrial-grade behavioral standard for backend engineering. Enforces strict configuration and runtime transactional semantics for Kafka 3.9 ZooKeeper-less (KRaft) metadata quorums. Guarantees message deduplication via idempotent producer settings and prevents split-brain partition states during controller leader elections and cooperative sticky consumer group rebalances. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Non-idempotent producer configurations with blind retries and non-transactional commits causing data duplication during KRaft controller failover Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Non-idempotent producer configurations with blind retries and non-transactional commits causing data duplication during KRaft controller failover	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS enable 'enable.idempotence'

Engineers MUST ALWAYS enable 'enable.idempotence': true and set 'acks': 'all' (-1) on all Kafka producers to prevent message duplication and out-of-order writes during KRaft controller leader transitions.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.