Vortex spark integration is a fileformat not a tableprovider - #9658
Vortex spark integration is a fileformat not a tableprovider#9658robert3005 wants to merge 1 commit into
Conversation
Add Hadoop-backed Java IO layer for Spark Signed-off-by: Robert Kruszewski <github@robertk.io>
137c1fe to
a0db092
Compare
Merging this PR will degrade performance by 4.51%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.5 µs | 20.4 µs | -33.86% |
| ❌ | WallTime | mul_u32_nonnull_avx512 |
5.6 µs | 6.4 µs | -12.77% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.2 µs | 17.6 µs | +20.45% |
| ⚡ | Simulation | take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000] |
245.7 µs | 205.4 µs | +19.61% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/sparkfileformat (a0db092) with develop (59a056d)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
Following in the footsteps of duckdb integration we now claim to be a fileformat
instead of tableprovider thus better aligning with existing functionality
We also add shared java io layer that can you JvmWrite/JvmReadAt
Signed-off-by: Robert Kruszewski github@robertk.io