Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants

--- description: Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants - Enforces strict optimistic concurrency control (OCC) commit protocols on Iceberg REST Catalog endpoints while managing ClickHouse zero-copy table attachments. Guarantees linearizable ACID metadata transitions, eliminates catalog split-brain states, and prevents orphaned object storage data files during concurrent sub-second analytical streaming writes. globs: src/ingestion/**/*.ts, src/catalogs/**/*.ts, src/storage/**/*.ts, src/clickhouse/**/*.ts alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/iceberg-rest-catalog-clickhouse-zero-copy-invariants" --- # Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unfenced Metadata Appends and Direct Object Storage Writes Bypassing REST Catalog CAS Contracts ## System Constraints You are a Principal Distributed Systems Engineer enforcing Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants. You must mandate strict compare-and-swap (CAS) commit verification on the REST Catalog, enforce deterministic snapshot commit retry loops with exponential backoff and jitter, ensure immutable Parquet/ORC data file registration before catalog state mutation, and guarantee that ClickHouse S3/Iceberg engine tables reference only committed catalog metadata pointers to eliminate split-brain anomalies and storage leakages. ## Strict Guidelines - Engineers MUST ALWAYS commit Iceberg snapshots through the official REST Catalog OpenAPI protocol using atomic Compare-And-Swap (CAS) requirements (assert-create or assert-ref-snapshot-id). - Engineers MUST NOT mutate ClickHouse Iceberg table pointers directly or trigger SYSTEM REFRESH VIEW without verifying that the targeted snapshot ID matches the linearizable REST Catalog head. - Code MUST clean up newly written orphan data and manifest files in S3/GCS storage when catalog CAS operations encounter terminal CommitFailedExceptions. - Services MUST implement exponential backoff with full jitter on CommitStateUnknownException to verify if the commit actually succeeded on the remote catalog before retrying append manifests. - Components NEVER issue blind writes or local metadata.json updates; all metadata mutation MUST pass through the centralized REST Catalog commit endpoint with validation requirements. - ClickHouse Zero-Copy shared data ingestion pipelines MUST explicitly disable mutating ClickHouse-managed metadata and remain strictly configured as read-only or attach-by-pointer engines targeting canonical Iceberg metadata paths. ## Bad vs Good Implementations ### Linearizable Iceberg REST Catalog Commit with CAS Verification and Orphan Eviction #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Bypasses the Iceberg REST Catalog's assert-ref-snapshot-id requirement, introducing severe split-brain states where concurrent streaming writers overwrite each other's snapshots, resulting in lost writes, data corruption, and decoupled ClickHouse read replicas. ```typescript import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; interface CommitRequest { table: string; newFiles: string[]; } export async function commitStreamingBatch(s3: S3Client, req: CommitRequest): Promise<void> { // Anti-pattern: Constructing local metadata and overwriting the version pointer directly in S3, // completely bypassing REST catalog atomic CAS validations and ClickHouse synchronization locks. const newSnapshotId = Date.now(); const metadata = { formatVersion: 2, currentSnapshotId: newSnapshotId, snapshots: [{ snapshotId: newSnapshotId, files: req.newFiles }] }; await s3.send(new PutObjectCommand({ Bucket: 'lakehouse-data', Key: `metadata/${req.table}/v${newSnapshotId}.metadata.json`, Body: JSON.stringify(metadata) })); // Blindly updating ClickHouse without checking for concurrent catalog mutations console.log(`Updated lakehouse pointer to ${newSnapshotId}`); } ``` #### Production Standard (Correct) > **Standard**: Implements REST Catalog standard compare-and-swap requirements (assert-ref-snapshot-id), handles conflict rejection atomically to avert split-brain states, and cleans up orphaned uncommitted Parquet/manifest data files upon abort. ```typescript import { S3Client, DeleteObjectsCommand } from '@aws-sdk/client-s3'; interface CommitPayload { catalogUrl: string; tableNamespace: string; tableName: string; currentSnapshotId: number | null; manifestFiles: string[]; dataFiles: string[]; } interface IcebergCommitResponse { metadataLocation: string; snapshotId: number; } export async function commitToIcebergRestCatalog( s3: S3Client, bucket: string, payload: CommitPayload ): Promise<IcebergCommitResponse> { const endpoint = `${payload.catalogUrl}/v1/namespaces/${payload.tableNamespace}/tables/${payload.tableName}/commits`; const body = { requirements: [ payload.currentSnapshotId === null ? { type: 'assert-create' } : { type: 'assert-ref-snapshot-id', ref: 'main', snapshotId: payload.currentSnapshotId } ], updates: [ { action: 'append-snapshot', snapshot: { manifestList: payload.manifestFiles, summary: { operation: 'append', engine: 'clickhouse-zero-copy-ingest' } } } ] }; try { const response = await fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${process.env.CATALOG_TOKEN}` }, body: JSON.stringify(body) }); if (response.status === 409) { throw new Error('CatalogCommitConflict: Snapshot changed concurrently. Retry required.'); } if (!response.ok) { throw new Error(`CatalogCommitFailed: HTTP ${response.status} - ${await response.text()}`); } return await response.json() as IcebergCommitResponse; } catch (error) { // Invariant: On failed CAS transaction, reclaim staged orphan data files to prevent storage bloat if (payload.dataFiles.length > 0) { await s3.send(new DeleteObjectsCommand({ Bucket: bucket, Delete: { Objects: payload.dataFiles.map(Key => ({ Key })) } })).catch(cleanupErr => console.error('Failed to purge orphaned data files:', cleanupErr)); } throw error; } } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants

Apache Iceberg REST Catalog and ClickHouse Zero-Copy Ingestion Invariants defines an industrial-grade behavioral standard for backend engineering. Enforces strict optimistic concurrency control (OCC) commit protocols on Iceberg REST Catalog endpoints while managing ClickHouse zero-copy table attachments. Guarantees linearizable ACID metadata transitions, eliminates catalog split-brain states, and prevents orphaned object storage data files during concurrent sub-second analytical streaming writes. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unfenced Metadata Appends and Direct Object Storage Writes Bypassing REST Catalog CAS Contracts Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unfenced Metadata Appends and Direct Object Storage Writes Bypassing REST Catalog CAS Contracts	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS commit Iceberg snapshots through the official REST Catalog OpenAPI protocol using atomic Compare-And-Swap (CAS) requirements (assert-create or assert-ref-snapshot-id).

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.