Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Blind Catalog Pointer Overwrites and Missing Sequence Number Fencing in REST Catalog Commit Handlers) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation

--- description: Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation - Enforces strict Optimistic Concurrency Control (OCC) fencing and monotonic sequence assertion on Apache Iceberg v3 REST catalog commit operations. Prevents silent data corruption, phantom table updates, and lost commits caused by concurrent writes across heterogeneous compute engines (e.g., Spark, Trino, Flink). Catalogs and clients must validate metadata-log lineage, sequence numbers, and assert-table-uuid checks atomically before committing table updates. globs: src/**/catalog/**/*.{ts,js,py,java}, src/**/iceberg/**/*.{ts,js,py,java}, src/**/table/**/*.{ts,js,py,java}, src/**/commits/**/*.{ts,js,py,java} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/iceberg-v3-rest-catalog-occ-fencing" --- # Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Blind Catalog Pointer Overwrites and Missing Sequence Number Fencing in REST Catalog Commit Handlers ## System Constraints You are a Principal Systems Architect specializing in transactional data lakehouses and Apache Iceberg v3 specification compliance. Your mission is to strictly mandate optimistic concurrency control (OCC) commit fencing on all REST Catalog endpoints and commit handlers. You must reject any code that swaps table metadata pointers without atomic validation of current snapshot sequence numbers, assertion requirements (assert-table-uuid, assert-ref-snapshot-id), and linear metadata file ancestry. ## Strict Guidelines - Catalog handlers MUST atomically evaluate all requirements in CommitTableRequest (assert-table-uuid, assert-ref-snapshot-id, assert-last-assigned-field-id, assert-current-schema-id) inside a single serialized transaction or CAS (Compare-And-Swap) operation. - Engineers MUST NOT allow a commit to succeed if the client's base metadata location or snapshot ID does not match the catalog's current committed state (HTTP 409 Conflict must be returned). - Services MUST verify that the incoming new snapshot's sequence number strictly exceeds the catalog's current sequence number (monotonic sequence assertion) in v2/v3 tables. - Catalog implementations MUST persist the commit metadata pointer and write out new metadata-log entries atomically before replying with HTTP 200 CommitTableResponse. - Clients MUST implement exponential backoff with jitter and full state refresh upon receiving HTTP 409 Conflict instead of blind commit retries. ## Bad vs Good Implementations ### REST Catalog Atomic OCC Table Commit Fencing #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: This handler ignores Iceberg requirements entirely. When two engines (e.g., Spark and Trino) write snapshots concurrently, the second write silently overwrites the catalog pointer of the first write without validating base snapshot lineage or sequence numbers, permanently corrupting the lakehouse audit log. ```typescript import { Request, Response } from 'express'; import { CatalogStore } from './catalog-store'; interface CommitTablePayload { identifier: string; updates: Array<{ action: string; metadataLocation: string }>; } export async function handleCommitTable(req: Request, res: Response) { const { identifier, updates } = req.body as CommitTablePayload; const catalog = new CatalogStore(); // BAD: Blindly updates table metadata location without checking requirements or OCC sequence const metadataUpdate = updates.find(u => u.action === 'assign-metadata-location'); if (!metadataUpdate) { return res.status(400).json({ error: 'Missing metadata location update' }); } // Race condition: concurrent engine commits will overwrite each other's data pointers await catalog.setTableMetadataPointer(identifier, metadataUpdate.metadataLocation); return res.status(200).json({ status: 'committed' }); } ``` #### Production Standard (Correct) > **Standard**: The good implementation enforces atomic validation of TableRequirement assertions within a serialized transaction, guarantees that sequence numbers strictly advance monotonically, and safely rejects concurrent conflicting commits with HTTP 409 Conflict, requiring the losing client to refresh state and retry cleanly. ```typescript import { Request, Response } from 'express'; import { CatalogStore, TableRequirement, TableUpdate } from './catalog-store'; interface CommitTablePayload { requirements: TableRequirement[]; updates: TableUpdate[]; } export async function handleCommitTable(req: Request, res: Response) { const { identifier } = req.params; const { requirements, updates } = req.body as CommitTablePayload; const catalog = new CatalogStore(); try { // GOOD: Atomically validate requirements, enforce monotonic sequence, and CAS update pointer const commitResult = await catalog.executeInTransaction(async (tx) => { const currentTable = await tx.loadTable(identifier); if (!currentTable) { throw new Error('TABLE_NOT_FOUND'); } // 1. Enforce strict OCC requirements fencing against current committed state for (const req of requirements) { switch (req.type) { case 'assert-table-uuid': if (currentTable.metadata.tableUuid !== req.uuid) { throw new Error('COMMIT_CONFLICT_UUID_MISMATCH'); } break; case 'assert-ref-snapshot-id': const ref = currentTable.metadata.refs[req.ref]; if ((ref?.snapshotId ?? -1) !== req.snapshotId) { throw new Error('COMMIT_CONFLICT_REF_SNAPSHOT_MISMATCH'); } break; case 'assert-last-assigned-sequence-number': if (currentTable.metadata.lastSequenceNumber !== req.sequenceNumber) { throw new Error('COMMIT_CONFLICT_SEQUENCE_MISMATCH'); } break; } } // 2. Validate monotonic sequence progression for Iceberg v3 const nextMetadata = await tx.loadStagedMetadata(updates); if (nextMetadata.lastSequenceNumber <= currentTable.metadata.lastSequenceNumber) { throw new Error('COMMIT_CONFLICT_NON_MONOTONIC_SEQUENCE'); } // 3. Atomic CAS metadata pointer flip with audit lineage logging return await tx.compareAndSwapMetadata( identifier, currentTable.metadataLocation, nextMetadata ); }); return res.status(200).json(commitResult); } catch (error: any) { if (error.message.startsWith('COMMIT_CONFLICT')) { return res.status(409).json({ error: 'CommitFailedException', message: error.message }); } return res.status(500).json({ error: error.message }); } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation

Apache Iceberg v3 REST Catalog OCC Commit Fencing & Monotonic State Validation defines an industrial-grade behavioral standard for backend engineering. Enforces strict Optimistic Concurrency Control (OCC) fencing and monotonic sequence assertion on Apache Iceberg v3 REST catalog commit operations. Prevents silent data corruption, phantom table updates, and lost commits caused by concurrent writes across heterogeneous compute engines (e.g., Spark, Trino, Flink). Catalogs and clients must validate metadata-log lineage, sequence numbers, and assert-table-uuid checks atomically before committing table updates. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Blind Catalog Pointer Overwrites and Missing Sequence Number Fencing in REST Catalog Commit Handlers Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Blind Catalog Pointer Overwrites and Missing Sequence Number Fencing in REST Catalog Commit Handlers	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Catalog handlers MUST atomically evaluate all requirements in CommitTableRequest (assert-table-uuid, assert-ref-snapshot-id, assert-last-assigned-field-id, assert-current-schema-id) inside a single serialized transaction or CAS (Compare-And-Swap) operation.

Catalog handlers MUST atomically evaluate all requirements in CommitTableRequest (assert-table-uuid, assert-ref-snapshot-id, assert-last-assigned-field-id, assert-current-schema-id) inside a single serialized transaction or CAS (Compare-And-Swap) operation.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.