There’s a clear shift happening in the data world: newer engines like DuckDB, Polars, and DataFusion are deliberately avoiding the JVM. This isn’t just an implementation choice—it reflects a deeper change in how performance is defined. Systems like Apache Spark were built around scaling out across clusters. But modern engines are optimizing for something else: how efficiently a single core can execute work.
That shift leads directly to hardware-aware execution. Native engines (C++/Rust) can fully exploit SIMD, operate on columnar memory, and avoid garbage collection entirely. The JVM, while great for general-purpose systems, introduces overhead exactly where analytical workloads are sensitive—GC pauses, object-heavy memory layouts, and limited vectorization. The result is a hard ceiling on per-core performance that scaling alone can’t fix.
What’s emerging isn’t “Spark vs new engines,” but a separation of roles:
- Spark for orchestration, distribution, and fault tolerance
- Native engines (like Velox or Apache Arrow) for execution
The interesting question now isn’t whether the JVM is good or bad—it’s whether execution efficiency is becoming more important than abstraction in modern data systems.