Tag: Apache Spark

All the articles with the tag "Apache Spark".

• 31 MIN READ•Aug 25, 2026

  DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg

  DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.

  Apache IcebergApache SparkDataFusion Comet
• 24 MIN READ•May 22, 2026

  Maintaining Apache Iceberg Tables: Compaction, Snapshot Expiration, and Orphan File Cleanup

  An in-depth guide to orchestrating maintenance operations on Apache Iceberg tables, covering bin-packing, sort-based, Z-Order compaction, snapshot expiration, and orphan file removal, with query acceleration details for the Dremio engine.

  Apache IcebergCompactionData Engineering
• 24 MIN READ•May 22, 2026

  Apache Iceberg with Spark: Create, MERGE, Upsert, and Evolve Tables End to End

  A comprehensive developer guide to configuring Apache Spark with Apache Iceberg, executing transactional writes, and managing schema evolution.

  apache sparkapache icebergdata engineering