[VL] Fix empty GPU broadcast joins and CudfVector host read crashes in the cuDF backend#12471
Open
ReemaAlzaid wants to merge 5 commits into
Open
[VL] Fix empty GPU broadcast joins and CudfVector host read crashes in the cuDF backend#12471ReemaAlzaid wants to merge 5 commits into
ReemaAlzaid wants to merge 5 commits into
Conversation
Add materializeVeloxRowVector() (cpp/velox/utils/CudfVectorUtils.h) and call it at every GPU->CPU boundary that reads RowVector children on the host: CudfVectorStream/RowVectorStream::next, VeloxColumnarBatch ensureFlattened/compose/select/toUnsafeRow, VeloxBatchResizer, JniHashTable nativeHashTableBuild, VeloxJniWrapper prune, VeloxColumnarBatchSerializer append/framedSerializeWithStats, HashTableBuilder::addInput. Eliminates the broadcast/shuffle childAt crash on GPU-resident CudfVector (0 host children). Also wires cudf.allowCpuFallback through VeloxBackend/GlutenConfig.
VeloxColumnarToRowConverter::convert read the raw (device-resident) CudfVector via getRowVector(); switch to getFlattenedRowVector() so it materializes to host first. This is the one host-read site the materialize set missed. It feeds both the broadcast build-side relation (UnsafeColumnarBuildSideRelation columnar->row) and result return, so the gap caused empty broadcast joins and VARCHAR-offset SIGSEGVs (e.g. q15).
For offloaded broadcast hash joins the build-side RDD fed Iterator.empty into the native plan and stashed the data in a prebuilt CPU hash table (VeloxBroadcastBuildSideCache -> OpaqueHashTable on the HashJoinNode). CudfHashJoin has no knowledge of OpaqueHashTable and builds from the build-side value stream, so every GPU broadcast join built from an empty stream and silently returned zero rows. When spark.gluten.sql.columnar.cudf=true, stream the deserialized broadcast batches instead (the same path shuffle joins use on GPU) and skip the CPU cache build. Hybrid stays correct: with no cached table the HashJoinNode carries no reusable table, so a CPU-fallback join builds from the stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Broadcast hash joins on the cuDF (GPU) backend were silently returning empty
results, and GPU batches were crashing whenever CPU code tried to read them.
This PR fixes both:
1. Broadcast joins built from an empty stream (
VeloxBroadcastBuildSideRDD.scala)For broadcast joins, Gluten puts the build-side data into a prebuilt CPU hash
table and feeds the plan an empty iterator. The CPU join uses that prebuilt
table but the GPU join (
CudfHashJoin) doesn't know it exists, so it builtits hash table from the empty iterator and every broadcast join returned 0 rows.
Fix: when cuDF is enabled, stream the broadcast batches into the plan, same as
shuffle joins already do.
2. GPU vectors crashing on host reads (new
CudfVectorUtils.h+ call sites)A
CudfVectoron the GPU and has no host side children, so host codereading or serializing it saw garbage (the
childAtcrashes). Added a smallhelper that copies the vector to host first, and applied it at every
host read site including the ColumnarToRow converter, which was causing a
JVM segfault in q15.
Needs a Velox build with the multi-column
hash_with_seedfixfacebookincubator/velox#18047
How was this patch tested?
gluten-it queries-comparevs vanilla Spark (2× NVIDIA L40S, Spark 3.5,pure GPU:
allowCpuFallback=false, broadcast joins enabled):(e.g. q21: 82s → 11s)
Remaining failures are pre-existing cuDF expression gaps
(
row_constructor_with_null,substring), not related to this change.cc: @marin-ma @zhouyuan