Apache Kafka 4.0 KRaft Dynamic Quorum Reconfiguration & Epoch Guard Standard

--- description: Apache Kafka 4.0 KRaft Dynamic Quorum Reconfiguration & Epoch Guard Standard - Mandates strict quorum epoch verification, single-voter transitions, and replica catch-up preconditions during KRaft dynamic quorum reconfiguration (KIP-853) in Kafka 4.0. Prevents split-brain divergence, quorum loss, and metadata log truncation during controller cluster topology changes. globs: src/**/*.ts, services/kafka/**/*.ts, infra/controllers/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/kafka-kraft-quorum-reconfiguration-standard" --- # Apache Kafka 4.0 KRaft Dynamic Quorum Reconfiguration & Epoch Guard Standard > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unfenced Concurrent Controller Membership Mutations Without Epoch Fencing ## System Constraints You are a Principal Distributed Systems Architect enforcing the Apache Kafka 4.0 KRaft Controller Dynamic Quorum Reconfiguration Standard (KIP-853). You must strictly reject any code or automation that updates controller voter sets without leader epoch fencing, strict single-node transition serialization, and replica log-end offset validation. ## Strict Guidelines - Engineers MUST ALWAYS verify the active controller leader epoch and validate clusterId against the current metadata snapshot prior to submitting quorum reconfiguration RPCs. - Engineers MUST NOT execute concurrent or batch voter additions or removals; every membership change must serialize through committed VotersRecord states one voter at a time. - Code SHOULD verify prospective voter nodes have synchronized to the active metadata leader high watermark (zero replica lag) before promoting an observer to a voter. - Services MUST enforce that any voter removal satisfies the quorum invariant N_new >= floor(N_current / 2) + 1 live healthy voters to avert immediate quorum starvation. - Components NEVER re-register controller nodes using static IDs without validating the unique voterDirectoryId (Uuid) to prevent partition corruption from recycled disk volumes. ## Bad vs Good Implementations ### KRaft Quorum Reconfiguration with Epoch Fencing and Lag Guards #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The bad example issues an AddVoter request without verifying the active controller leader epoch, ignoring the unique voterDirectoryId, and omitting replica lag verification. If issued against a partitioned or stale controller, or before the target replica is fully caught up, this can permanently stall the metadata quorum or cause split-brain log divergence. ```typescript import { KafkaAdminClient } from './kafkaAdmin'; interface ReconfigureParams { controllerId: number; endpoint: string; } export async function promoteToVoter(admin: KafkaAdminClient, params: ReconfigureParams): Promise<void> { // ANTI-PATTERN: Blindly mutating KRaft quorum without checking cluster ID, // leader epoch, replica synchronization status, or voter directory ID. // Can trigger split-brain or metadata divergence in Kafka 4.0 KRaft clusters. await admin.sendAddVoterRequest({ voterId: params.controllerId, endpoint: params.endpoint }); } ``` #### Production Standard (Correct) > **Standard**: The good example validates the cluster ID, verifies that the target node is an active observer caught up to the High Watermark, binds the unique KIP-853 voterDirectoryId to avoid resurrected storage state, and fences the AddVoter request with the current leaderEpoch to guarantee serialized quorum consensus. ```typescript import { KafkaAdminClient, QuorumStatus, VoterRecordRequest } from './kafkaAdmin'; interface ReconfigureParams { clusterId: string; controllerId: number; voterDirectoryId: string; // KIP-853 UUID guard against re-used broker disk state endpoint: string; maxAllowedLagMs: number; } export async function promoteToVoter(admin: KafkaAdminClient, params: ReconfigureParams): Promise<void> { const quorum: QuorumStatus = await admin.describeQuorum(); if (quorum.clusterId !== params.clusterId) { throw new Error(`Cluster ID mismatch: expected ${params.clusterId}, received ${quorum.clusterId}`); } // Guard 1: Must target active leader and capture fencing epoch if (quorum.leaderId === null || quorum.leaderEpoch <= 0) { throw new Error('No stable KRaft leader detected; quorum reconfiguration aborted'); } // Guard 2: Target node must exist as an observer and be fully caught up to High Watermark const observer = quorum.observers.find(o => o.nodeId === params.controllerId); if (!observer) { throw new Error(`Node ${params.controllerId} is not an active observer in the metadata quorum`); } if (observer.replicaLagMs > params.maxAllowedLagMs || observer.logEndOffset < quorum.highWatermark) { throw new Error(`Node ${params.controllerId} lag exceeds threshold; cannot safely promote to voter`); } // Guard 3: Strict serialized transition using epoch-fenced payload const voterRequest: VoterRecordRequest = { clusterId: params.clusterId, voterId: params.controllerId, voterDirectoryId: params.voterDirectoryId, leaderEpoch: quorum.leaderEpoch, endpoint: params.endpoint }; await admin.addVoter(voterRequest); await admin.awaitQuorumCommit(params.controllerId, quorum.leaderEpoch); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Apache Kafka 4.0 KRaft Dynamic Quorum Reconfiguration & Epoch Guard Standard

Apache Kafka 4.0 KRaft Dynamic Quorum Reconfiguration & Epoch Guard Standard defines an industrial-grade behavioral standard for backend engineering. Mandates strict quorum epoch verification, single-voter transitions, and replica catch-up preconditions during KRaft dynamic quorum reconfiguration (KIP-853) in Kafka 4.0. Prevents split-brain divergence, quorum loss, and metadata log truncation during controller cluster topology changes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unfenced Concurrent Controller Membership Mutations Without Epoch Fencing Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unfenced Concurrent Controller Membership Mutations Without Epoch Fencing	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS verify the active controller leader epoch and validate clusterId against the current metadata snapshot prior to submitting quorum reconfiguration RPCs.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.