Dremio Sabot Engine
The Dremio Sabot Engine is the underlying, high-performance distributed query execution engine that powers Dremio. Named after a “sabot” (a device that supports a projectile as it moves through a barrel, ensuring maximum velocity), the engine is designed to execute SQL queries against data lakes and cloud object storage at sub-second speeds.
Rather than processing data row-by-row using the traditional Volcano iterator model, the Sabot Engine processes batches of columnar data in memory, maximizing CPU instruction cache efficiency and enabling vectorized execution.
Architectural Pillars of the Sabot Engine
The Sabot Engine’s extreme performance is achieved through three core engineering designs:
1. Vectorized Memory Execution via Apache Arrow
The Sabot Engine uses Apache Arrow as its native, in-memory data representation format. When data is read from disk (for example, from a Parquet file), the engine deserializes it directly into Arrow memory buffers. Because Arrow stores data in structured columns, operations like filtering, projection, and aggregation can process entire arrays of values in a single operation. This matches the physical layout of modern CPU L1/L2 caches and registers.
2. Runtime Code Generation
To avoid interpretive overhead during query execution, the Sabot Engine compiles SQL expressions into optimized Java bytecode at query runtime. This code generation phase compiles specific SQL filters, projections, and mathematical operators directly into tight execution loops, eliminating conditional branching overhead and ensuring maximum execution speed.
3. Distributed Pipelined Execution
The Sabot Engine acts as a distributed system, coordinating query fragments across a cluster of executor nodes:
- Pipelining: Executor nodes stream intermediate query results to other nodes in the cluster using high-speed network connections without writing intermediate states to disk.
- Asynchronous I/O: The engine separates data loading threads from query processing threads. Executors fetch new data blocks in the background while the CPU processes existing memory buffers.
- Dynamic Load Balancing: If an executor node encounters a slowdown, the engine dynamically shifts query partition fragments to other available nodes to maintain execution speeds.
Comparison: Volcano Iterator vs. Vectorized Sabot
Traditional query engines process data using row-by-row iterators:
Row-by-Row Iterator:
Get Row 1 → Process Col A → Process Col B → Output Row 1
Get Row 2 → Process Col A → Process Col B → Output Row 2
(Causes high CPU instruction cache misses and virtual function call overheads)The Sabot Engine processes columns in parallel arrays:
Vectorized Column Array:
Fetch Array [Col A (Rows 1-1000)] → Apply filter in one loop
Fetch Array [Col B (Rows 1-1000)] → Apply projection in one loop
(Utilizes hardware SIMD instructions and achieves optimal CPU L1/L2 cache locality)Importance for Apache Iceberg and Parquet
Because Apache Iceberg tables store data in columnar formats (predominantly Apache Parquet), the Sabot Engine is exceptionally efficient at querying them. It reads Parquet column pages directly into Arrow memory buffers without performing expensive row reconstruction or format translation steps. This native columnar alignment from disk (Parquet) to memory (Arrow) to execution (Sabot) removes the serialization bottleneck that typically plagues data lakehouse queries.