AgentRules.org
RulesStacksClientsAuditorPrunerSelf-HealRFCsBenchmarkCLICI/CDAuthorsSubmit
This specification is maintained by the AgentRules Working Group. Have enhancements or want to claim maintainership?Submit Specification →
backendcritical PriorityID: apache-iceberg-partition-evolution-compatibility-guardrail

Apache Iceberg Partition Evolution and Schema Invariant Guardrail

Enforces non-breaking schema and hidden partition evolution invariants on Apache Iceberg lakehouse tables to guarantee deterministic query engine scan planning. Mandates void transforms for retired partition fields, preserves source column field IDs, and strictly prevents physical partition rewrites or field ID reassignments that cause query pruning degradation across distributed engines (Trino, Spark, DuckDB).

Prohibited Anti-Pattern: Mutating existing partition field specs in-place or re-ordering schema field IDs during Iceberg partition updates

Export Ready-to-Drop Configuration

Select format below to copy code or download file directly.

---
description: Apache Iceberg Partition Evolution and Schema Invariant Guardrail - Enforces non-breaking schema and hidden partition evolution invariants on Apache Iceberg lakehouse tables to guarantee deterministic query engine scan planning. Mandates void transforms for retired partition fields, preserves source column field IDs, and strictly prevents physical partition rewrites or field ID reassignments that cause query pruning degradation across distributed engines (Trino, Spark, DuckDB).
globs: src/**/*.py, src/**/*.scala, src/**/*.java, src/**/*.ts, pipelines/**/*.py, lakehouse/**/*.py
alwaysApply: true
verified_by: "agentrules.org"
license: "CC-BY-4.0"
source: "https://agentrules.org/rules/apache-iceberg-partition-evolution-compatibility-guardrail"
---

# Apache Iceberg Partition Evolution and Schema Invariant Guardrail

> **Severity**: CRITICAL | **Category**: backend
> **Anti-Pattern Warning**: Mutating existing partition field specs in-place or re-ordering schema field IDs during Iceberg partition updates

## System Constraints
You are a Principal Engineer enforcing Apache Iceberg Partition Evolution Compatibility Guardrails. Mandate that all lakehouse partition and schema modifications preserve historical scan compatibility, follow Iceberg Spec v2 partition evolution invariants (retiring via void transforms without modifying prior spec IDs), maintain immutable column field-ids, and guarantee real-time query engine planning determinism.

## Strict Guidelines
- Engineers MUST ALWAYS use Iceberg Table API `UpdatePartitionSpec` or explicit SQL evolution (`ALTER TABLE ... ADD/DROP PARTITION FIELD`) rather than recreating the table metadata or modifying older PartitionSpec objects directly.
- Engineers MUST NOT drop a partition field by deleting its definition from metadata; dropped partition fields MUST be replaced with a `void` transform to preserve partition field-id monotonically incrementing continuity.
- Code SHOULD verify that source column field IDs remain unchanged during schema evolution so that historical manifest files and delete files retain 1:1 predicate pushdown alignment.
- Services MUST ensure any new partition transform (e.g., evolving from `month(ts)` to `day(ts)`) allows co-existence with older data layouts without requiring immediate synchronous partition rewrites.
- Components NEVER re-assign or recycle existing Iceberg `field-id` or `partition-field-id` values across evolving table schema versions.

## Bad vs Good Implementations
### Iceberg Table Partition Evolution (PyIceberg / Python Lakehouse Service)

#### Bad Antipattern (AI Common Hallucination)
> **Why it fails**: Direct removal without tracking the existing partition field ID causes historical manifests referencing the old field ID to fail query planning or trigger full-table scans. Modifying partition specs without explicit void transform preservation invalidates engine pushdown caches.
```python
# Destructive partition modification recreating metadata
def evolve_partitioning(table):
    # ANTI-PATTERN: Mutating partition spec in-place or resetting table properties
    # forces a full table rewrite, breaks historical manifest pruners, and desynchronizes field IDs
    schema = table.schema()
    new_spec = table.spec().copy()
    # Re-assigning or attempting to drop field by direct removal breaks Iceberg spec guarantees
    table.update_spec().remove_field("event_date").add_field("event_hour").commit()
```

#### Production Standard (Correct)
> **Standard**: Using the Iceberg transaction context to evolve partition fields ensures that retired partition columns are replaced by void transforms, preserves spec ID history (v0, v1, etc.), maintains field ID monotonic integrity, and allows query engines to push down predicates across mixed layout snapshots.
```python
# Non-breaking partition evolution adhering to Iceberg v2 invariants
def evolve_partitioning(table):
    # PRODUCTION FIX: Evolve partition spec via transaction preserving spec history
    # Replaces 'event_date' with void transform implicitly and assigns new monotonically increasing field ID
    with table.transaction() as tx:
        with tx.update_spec() as update:
            # Retires the existing partition field by assigning void transform
            update.remove_field("event_date")
            # Adds new partition transform without mutating underlying source column field-ids
            update.add_field("event_hour", transform="hour")
    # Deterministic query engines (Trino/Spark/DuckDB) can now prune both old and new data layouts
```
Recommended MCP Servers (mcp-serverlist.com)

Equip agent with system tools to verify code for this stack.

Iceberg REST Catalog MCP Inspector
Inspects Iceberg table metadata, verifies schema evolution diffs, and validates partition spec evolution invariants directly against REST catalogs.
Recommended Agent Skills (askills.dev)

Ready-to-use executable skill functions tailored to this stack.

PyIceberg Inspection CLI
CLI utility to inspect snapshot manifests, partition specs, and metadata JSON files for invariant verification.
Architectural Specification & Invariant Rationale

Architectural Invariant Design: Apache Iceberg Partition Evolution and Schema Invariant Guardrail

Apache Iceberg Partition Evolution and Schema Invariant Guardrail defines an industrial-grade behavioral standard for backend engineering. Enforces non-breaking schema and hidden partition evolution invariants on Apache Iceberg lakehouse tables to guarantee deterministic query engine scan planning. Mandates void transforms for retired partition fields, preserves source column field IDs, and strictly prevents physical partition rewrites or field ID reassignments that cause query pruning degradation across distributed engines (Trino, Spark, DuckDB). Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Mutating existing partition field specs in-place or re-ordering schema field IDs during Iceberg partition updates Without an explicit system instruction file, developers spend hours debugging silent failures.
Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure ModeAI Symptom (The Anti-Pattern)SeverityProduction Impact & Remedy
AI Hallucination & Deprecation DriftMutating existing partition field specs in-place or re-ordering schema field IDs during Iceberg partition updatesCritical
Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production.
Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture InconsistencyAI assistants produce mixed paradigms across files, creating fragmented codebase conventions.Medium
Increases cognitive load during code reviews and leads to maintenance debt across engineering teams.
Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.
Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

1

Engineers MUST ALWAYS use Iceberg Table API `UpdatePartitionSpec` or explicit SQL evolution (`ALTER TABLE ... ADD/DROP PARTITION FIELD`) rather than recreating the table metadata or modifying older PartitionSpec objects directly.

Engineers MUST ALWAYS use Iceberg Table API `UpdatePartitionSpec` or explicit SQL evolution (`ALTER TABLE ... ADD/DROP PARTITION FIELD`) rather than recreating the table metadata or modifying older PartitionSpec objects directly.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
2

Engineers MUST NOT drop a partition field by deleting its definition from metadata

Engineers MUST NOT drop a partition field by deleting its definition from metadata; dropped partition fields MUST be replaced with a `void` transform to preserve partition field-id monotonically incrementing continuity.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
3

Code SHOULD verify that source column field IDs remain unchanged during schema evolution so that historical manifest files and delete files retain 1

Code SHOULD verify that source column field IDs remain unchanged during schema evolution so that historical manifest files and delete files retain 1:1 predicate pushdown alignment.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
4

Services MUST ensure any new partition transform (e.g., evolving from `month(ts)` to `day(ts)`) allows co-existence with older data layouts without requiring immediate synchronous partition rewrites.

Services MUST ensure any new partition transform (e.g., evolving from `month(ts)` to `day(ts)`) allows co-existence with older data layouts without requiring immediate synchronous partition rewrites.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
5

Components NEVER re-assign or recycle existing Iceberg `field-id` or `partition-field-id` values across evolving table schema versions.

Components NEVER re-assign or recycle existing Iceberg `field-id` or `partition-field-id` values across evolving table schema versions.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.
Enforcement:Integrated into the Agent Pack rule file. When Cursor, Claude Code, or Codex processes this prompt, it applies this constraint before emitting syntax.
Technical Questions & Answers

Frequently Asked Technical Questions

Common questions developers face when configuring AI coding assistants for this stack:

Q:Why is Apache Iceberg Partition Evolution and Schema Invariant Guardrail critical for AI coding assistants?

Enforces non-breaking schema and hidden partition evolution invariants on Apache Iceberg lakehouse tables to guarantee deterministic query engine scan planning. Mandates void transforms for retired partition fields, preserves source column field IDs, and strictly prevents physical partition rewrites or field ID reassignments that cause query pruning degradation across distributed engines (Trino, Spark, DuckDB). Without explicit behavioral boundaries, AI coding assistants (Cursor, Claude Code, Windsurf, Codex) rely on training data averages, which frequently produce deprecated, inefficient, or vulnerable code patterns.

Q:How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Mutating existing partition field specs in-place or re-ordering schema field IDs during Iceberg partition updates) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q:Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q:Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Open Specification · CC BY 4.0·Free for Personal, Team & Commercial In-Repo Use

Citation & Open Source Attribution Guidelines

This specification is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You are completely free to adapt and share this rule in your workspace. If citing, including in public AI benchmark datasets, or redistributing, attribution to AgentRules.org is required.

Related BACKEND Rules

View all rules →
backend
42

Fastify 5 Schema Compilers & Encapsulation Production Standard

Fastify relies on lexical encapsulation contexts to sandbox decorators, hooks, and compiled validation/serialization schemas without memory leaks or cross-tenant bleed. In Fastify 5, misuse of global compilers or failing to encapsulate plugin boundaries causes compiled `fast-json-stringify` serializers and Ajv validators to either crash on duplicate IDs, leak private data across sibling scopes, or bypass JIT compilation optimizations. Production services must enforce strict encapsulation hierarchies and type-safe schema compilation across every routing subtree.

backend
68

Bun Embedded SQLite WAL Mode & Transaction Synchronization Standard

Production database standard for bun:sqlite. Mandates enabling Write-Ahead Logging (PRAGMA journal_mode = WAL), busy timeouts, and using db.transaction() for atomic operations without lock starvation.

backend
68

Drizzle ORM Batch API & Interactive Transaction Isolation Standard

Production database standard for Drizzle ORM (PostgreSQL & MySQL). Mandates using db.batch() over HTTP drivers to prevent round-trip waterfalls, explicit db.transaction() with serializable/read-committed isolation, and strictly avoids unawaited query promises.

1 Rule in Stack

Apache