The data lakehouse has gone from contested idea to default architecture in just a few years, and the table format you choose is the single most consequential decision in that stack. Apache Iceberg has emerged as the standard that most vendors are rallying around, but Apache Hudi and Delta Lake each have distinct strengths that matter depending on your workload. Let’s cut through the marketing and look at what actually differentiates these three formats at the architectural level.
Every table format solves the same fundamental problem: bringing ACID transactions, schema evolution, and time travel to object storage like S3, GCS, or Azure Data Lake. The differences are in how they organize metadata, handle concurrent writes, and manage row-level updates — and those differences have real operational consequences.
Apache Iceberg: The Metadata Tree
Iceberg organizes metadata hierarchically: each table snapshot points to a manifest list, which points to manifests, which point to data files. This tree structure means a query engine can prune files at multiple levels before reading any data — skipping entire partitions and file groups without scanning their metadata.
The defining feature is hidden partitioning. In traditional Hive-style partitioning, you manually decide which columns to partition by, and the physical directory structure mirrors that choice. Iceberg lets you define partition transforms in table metadata — truncate, bucket, or even temporal transforms like days(ts) — and the query engine handles the mapping automatically. A query filtering on WHERE event_date >= '2026-08-01' will hit the right partitions without you writing partition predicates. You can evolve the partition scheme later without rewriting data, which is impossible with Hive-style partitioning.
Iceberg requires a catalog (Nessie, Glue, REST, or Hive Metastore) to manage table state. This is a deployment dependency that Hudi and Delta don’t share, but it’s also what enables true multi-engine interoperability: Spark, Trino, Flink, DuckDB, and Snowflake can all read and write the same table through the catalog.
Delta Lake: The Transaction Log
Delta Lake takes a simpler approach: a single append-only transaction log (the _delta_log directory) records every change as a JSON commit. Each commit is an atomic set of actions — add file, remove file, update metadata. Readers load the latest checkpoint, replay any subsequent commits, and get a consistent snapshot.
This design is easy to reason about and doesn’t require a separate catalog — the log lives alongside the data. The trade-off is that as tables grow, the log can become a bottleneck for planning large queries, though Delta addresses this with checkpoint files that compact the log into Parquet format periodically.
Delta’s deep integration with Spark and Databricks is both its greatest strength and its primary limitation. If you’re an all-Spark shop, Delta is the path of least resistance. The Delta Engine and Photon runtime bring optimizations that aren’t available in the open-source version. But if you need multi-engine access — say, Trino for ad-hoc queries and Flink for streaming — Delta’s connector ecosystem is narrower than Iceberg’s.
Apache Hudi: The Timeline
Hudi was built for streaming-first workloads and it shows. Its timeline architecture tracks every action (commit, deltacommit, compaction, clean) as an instant, creating an ordered log of operations. This timeline is the foundation for Hudi’s standout feature: Merge-On-Read (MoR) tables.
In a MoR table, updates land in row-level log files and are merged with base Parquet files at read time. This avoids the write amplification of Copy-On-Write, where every update rewrites an entire file. Hudi also supports Copy-On-Write tables for read-heavy workloads, giving you a choice based on your access patterns.
Hudi’s indexing system is another differentiator. With over 8 index types — Bloom filter, HBase, Simple, Bucket, Record Level — Hudi can efficiently locate records for upserts without scanning the entire table. This makes it the strongest format for high-frequency update workloads, like CDC pipelines syncing operational database changes to the lakehouse.
Concurrency: OCC vs Non-Blocking
All three formats use optimistic concurrency control (OCC) as the baseline: a writer assumes its commit will succeed, and if a conflict is detected, it retries or fails. But Hudi goes further with Non-Blocking Concurrency Control (NBCC), which allows competing writers to commit simultaneously without failing each other. This is critical for multi-writer scenarios — say, a streaming ingestion job and a batch backfill running against the same table.
Iceberg and Delta both require external lock providers (like DynamoDB) for multi-writer setups, and competing writers will fail and retry. Hudi’s NBCC eliminates that failure mode, which matters for pipelines that need to stay running without manual intervention.
Working with Iceberg from Python
PyIceberg lets you create and manage Iceberg tables directly from Python without Spark:
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType, TimestampType, LongType
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform
# Connect to a REST catalog
catalog = load_catalog("default", **{
"type": "rest",
"uri": "http://rest-catalog:8181",
"warehouse": "s3://my-bucket/warehouse"
})
# Create a table with hidden partitioning
schema = Schema(
NestedField(1, "event_id", StringType(), required=True),
NestedField(2, "event_time", TimestampType(), required=True),
NestedField(3, "user_id", LongType()),
NestedField(4, "payload", StringType())
)
partition_spec = PartitionSpec(
PartitionField(
source_id=2,
field_id=1000,
transform=DayTransform(),
name="event_day"
)
)
table = catalog.create_table(
identifier="events.user_actions",
schema=schema,
partition_spec=partition_spec
)
# Append data from an Arrow table
import pyarrow as pa
batch = pa.table({
"event_id": pa.array(["evt-001", "evt-002"]),
"event_time": pa.array([1723000000, 1723000001]).cast(pa.timestamp("s")),
"user_id": pa.array([42, 99]),
"payload": pa.array(["click", "scroll"])
})
table.append(batch)
print("Rows:", len(table.scan().to_arrow()))The DayTransform in the partition spec means Iceberg automatically partitions by day, but queries can filter on event_time directly — no need to add a separate event_date column or write partition predicates. If you later decide to also bucket by user_id, you can evolve the spec without rewriting existing data.
Maintenance Operations
Every table format requires maintenance to keep query performance healthy as data accumulates. The key operations differ:
# Spark SQL maintenance for Iceberg tables
# Compact small files into larger ones
spark.sql("""
CALL system.rewrite_data_files(
table => 'events.user_actions',
options => map('target-size-bytes', '536870912')
)
""")
# Expire old snapshots to free storage
spark.sql("""
CALL system.expire_snapshots(
table => 'events.user_actions',
older_than => TIMESTAMP '2026-07-01 00:00:00',
retain_last => 10
)
""")
# Remove orphaned data files
spark.sql("""
CALL system.remove_orphan_files(
table => 'events.user_actions',
older_than => TIMESTAMP '2026-08-01 00:00:00'
)
""")Hudi requires more active tuning, particularly for compaction schedules on MoR tables. Delta’s maintenance is generally the simplest — OPTIMIZE and VACUUM commands handle file compaction and cleanup with sensible defaults. Iceberg sits in between: the operations are straightforward but you need to schedule them, either through Spark jobs or using tools like Apache Iceberg’s maintenance procedures.
Which Format Should You Pick?
The honest answer is that all three are production-ready and the gaps are narrowing. But the decision still matters today:
Choose Iceberg if you need multi-engine interoperability (Spark + Trino + Flink + DuckDB), hidden partitioning, or if your organization is investing in vendor-neutral standards. The catalog requirement adds a deployment component, but it’s what enables the broadest ecosystem compatibility. Snowflake, Dremio, and Google BigQuery all provide native Iceberg read support, and AWS Athena can both read and write Iceberg tables.
Choose Delta Lake if you’re primarily a Spark shop, especially if you’re already on Databricks. The integration is seamless, performance optimizations are strongest, and the operational model is simple. Databricks has also added UniForm — a feature that generates Iceberg-compatible metadata from Delta tables, acknowledging the interoperability standard while keeping the Delta runtime.
Choose Hudi if your workload is streaming-heavy with frequent upserts, CDC pipelines, or multiple concurrent writers. Hudi’s index-based upserts and non-blocking concurrency control solve real operational pain that the other two formats handle less gracefully. The DeltaStreamer ingestion tool also gives you managed CDC from Kafka and other sources without writing custom code.
One emerging option worth noting: Apache XTable (incubating) provides cross-format interoperability, letting you write in one format and read in another. If you’re worried about lock-in, XTable reduces the stakes of the initial choice — though it’s still early in its incubation and production adoption is limited.
The table format wars are effectively over in the sense that all three formats are viable, and the ecosystem is converging toward Iceberg as the interop standard. But “viable” doesn’t mean “interchangeable.” Your workload patterns — batch vs streaming, single-engine vs multi-engine, read-heavy vs write-heavy — should drive the decision, not vendor hype.