ClickHouse SharedMergeTree Lease Fencing & Split-Brain Prevention

--- description: ClickHouse SharedMergeTree Lease Fencing & Split-Brain Prevention - Enforces strict monotonic fencing tokens and Keeper ephemeral lease validation across stateless ClickHouse compute replicas operating on SharedMergeTree engines. Prevents partitioned or zombie worker nodes from writing uncoordinated mutations or orphan parts directly to shared object storage (S3/GCS). globs: infra/**/*.{ts,js,py}, src/clickhouse/**/*.{ts,js,go}, deploy/clickhouse/**/*.{yaml,json,ts}, operators/**/*.{ts,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/clickhouse-sharedmergetree-lease-fencing" --- # ClickHouse SharedMergeTree Lease Fencing & Split-Brain Prevention > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Unfenced Object Storage Mutations via Expired Keeper Leases ## System Constraints You are a Principal Database Infrastructure Engineer enforcing distributed lease fencing for ClickHouse SharedMergeTree topologies. You must eliminate split-brain data corruption and orphan object storage mutations by mandating monotonic fencing tokens, strict Keeper session renewal deadlines, and atomic multi-request commits. ## Strict Guidelines - Engineers MUST ALWAYS validate monotonic fencing tokens (Keeper epoch/znode cversion) inside multi-operation atomic transactions before finalizing object storage part commits. - Engineers MUST NOT allow compute replicas to execute mutations, background merges, or metadata updates without an active, non-expired ephemeral lease in ClickHouse Keeper. - Services MUST implement lease heartbeating with proactive task cancellation (aborting local uploads and disk writes) if the Keeper session renew timeout exceeds lease TTL minus safety leeway. - Code SHOULD assert fencing token monotonically strictly increases upon state transfer, rejecting any mutation payload carrying an obsolete generation token. - Components NEVER directly mutate or delete SharedMergeTree object storage prefixes outside of atomic Keeper multi-request commit protocols. ## Bad vs Good Implementations ### Keeper Lease Fencing for S3 Part Commit Coordination #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The worker performs S3 upload and creates the Keeper part node without passing or checking a fencing token. If the node experienced a GC pause or network hiccup, its ephemeral lease expired, another node claimed leadership, and this zombie node will corrupt table state by committing overlapping part metadata. ```typescript // BAD: Blind commit without verifying fencing token epoch or lease validity. // If a network partition occurs, this zombie worker writes to S3 and commits // outdated metadata over a superseded leader's state. export async function commitMergePart( keeperClient: KeeperClient, partName: string, s3Key: string ): Promise<void> { // Blind write: assumes lease acquired 30 seconds ago is still valid await uploadPartToS3(s3Key); // Commit directly without verifying version or epoch await keeperClient.create( `/clickhouse/tables/shared_db/events/parts/${partName}`, Buffer.from(JSON.stringify({ s3Key, state: 'committed' })) ); } ``` #### Production Standard (Correct) > **Standard**: The implementation verifies the fencing token prior to S3 operations and commits the part using an atomic Keeper MultiRequest that includes a `checkVersion` precondition on the lease node. If the lease expired or another replica claimed leadership during upload, the entire transaction is atomically rejected. ```typescript // GOOD: Epoch-based fencing token validation with atomic MultiRequest commit. // Aborts immediately if the ephemeral lease expired or generation was superseded. export async function commitMergePart( keeperClient: KeeperClient, partName: string, s3Key: string, leaseToken: { path: string; epoch: number } ): Promise<void> { // 1. Verify lease is still active and monotonic epoch has not drifted const leaseStat = await keeperClient.exists(leaseToken.path); if (!leaseStat || leaseStat.version !== leaseToken.epoch) { throw new Error(`[FENCE_VIOLATION] Lease lost or epoch mismatch: expected ${leaseToken.epoch}`); } // 2. Perform object write with cancellation token tied to lease heartbeat await uploadPartToS3(s3Key); // 3. Atomic MultiRequest: assert lease epoch has not changed while committing metadata const partPath = `/clickhouse/tables/shared_db/events/parts/${partName}`; const payload = Buffer.from(JSON.stringify({ s3Key, epoch: leaseToken.epoch, state: 'committed' })); await keeperClient.multi([ // Fencing check: fails entire transaction if epoch incremented or lease deleted keeperClient.ops.checkVersion(leaseToken.path, leaseToken.epoch), keeperClient.ops.create(partPath, payload) ]); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: ClickHouse SharedMergeTree Lease Fencing & Split-Brain Prevention

ClickHouse SharedMergeTree Lease Fencing & Split-Brain Prevention defines an industrial-grade behavioral standard for devops engineering. Enforces strict monotonic fencing tokens and Keeper ephemeral lease validation across stateless ClickHouse compute replicas operating on SharedMergeTree engines. Prevents partitioned or zombie worker nodes from writing uncoordinated mutations or orphan parts directly to shared object storage (S3/GCS). Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unfenced Object Storage Mutations via Expired Keeper Leases Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unfenced Object Storage Mutations via Expired Keeper Leases	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS validate monotonic fencing tokens (Keeper epoch/znode cversion) inside multi-operation atomic transactions before finalizing object storage part commits.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.