[VL] Fix SPARK-53322 in GlutenKeyGroupedPartitioningSuite for Spark 4.1#12469
Open
zml1206 wants to merge 1 commit into
Open
[VL] Fix SPARK-53322 in GlutenKeyGroupedPartitioningSuite for Spark 4.1#12469zml1206 wants to merge 1 commit into
zml1206 wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Spark 4.1-specific regression coverage for SPARK-53322 within Gluten’s KeyGroupedPartitioning test suite, focusing on how checkpointed scans interact with shuffle planning (especially around SMJ/SPJ scenarios) under the Velox backend test environment.
Changes:
- Added multiple
SPARK-53322regression tests covering aggregates and partitioned-join scenarios when inputs are checkpointed. - Introduced a helper to collect vanilla
ShuffleExchangeExecnodes from an executed plan (in addition to existing columnar-shuffle collectors). - Updated Spark 4.1 Velox test settings to exclude tests by the
SPARK-53322prefix.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| gluten-ut/spark41/src/test/scala/org/apache/spark/sql/connector/GlutenKeyGroupedPartitioningSuite.scala | Adds SPARK-53322 regression tests and a helper for collecting vanilla shuffles to validate exchange behavior with checkpointed scans. |
| gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala | Adjusts Velox Spark 4.1 suite exclusions to include the SPARK-53322 prefix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| .excludeByPrefix("SPARK-48012") | ||
| .excludeByPrefix("SPARK-44647") | ||
| .excludeByPrefix("SPARK-41471") | ||
| .excludeByPrefix("SPARK-53322") |
Contributor
Author
There was a problem hiding this comment.
The newly added testGluten uses "Gluten - " as a prefix.
Comment on lines
+1883
to
+1886
| withTempDir { | ||
| dir => | ||
| spark.sparkContext.setCheckpointDir(dir.getPath) | ||
| val itemsPartitions = Array(identity("id")) |
Comment on lines
+1899
to
+1902
| val shuffles = collectAllShuffles(df.queryExecution.executedPlan) | ||
| assert( | ||
| shuffles.isEmpty, | ||
| "should not contain shuffle when not grouping by partition values") |
| checkAnswer(df, Seq(Row(1, "aa", 41.0, 40.0), Row(3, "cc", 15.5, 25.5))) | ||
|
|
||
| // One shuffle for sort and two shuffles for join are expected. | ||
| assert(collectAllShuffles(df.queryExecution.executedPlan).length === 3) |
| checkAnswer(df, Seq(Row(1, "aa", 41.0, 40.0), Row(3, "cc", 15.5, 25.5))) | ||
|
|
||
| // One shuffle for sort and two shuffles for join are expected. | ||
| assert(collectAllShuffles(df.queryExecution.executedPlan).length === 3) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes
SPARK-53322coverage inGlutenKeyGroupedPartitioningSuitefor Spark 4.1 with Velox backend.Spark 4.1 adds several
SPARK-53322tests to verify that checkpointed scans are not incorrectly reused for Storage Partitioned Join planning. The upstream Spark tests assert shuffle counts using vanilla SparkShuffleExchangeExec, but Gluten may useColumnarShuffleExchangeExec, andKeyGroupedPartitioningshuffle may also fall back to vanilla Spark shuffle after GLUTEN-10992.This PR keeps the upstream
SPARK-53322tests excluded in Velox test settings, and adds Gluten-aware versions inGlutenKeyGroupedPartitioningSuite.The new tests verify:
For the mixed shuffle case, the test separately counts:
ColumnarShuffleExchangeExecfor Gluten columnar shuffleShuffleExchangeExecfor vanilla fallback shuffleThis is needed because
KeyGroupedPartitioningshuffle is not supported by Gluten native shuffle and should remain as vanilla Spark shuffle.