Spark 4.2: Validate snapshot when removing dangling deletes - #18061
Open
manuzhang wants to merge 2 commits into
Open
Spark 4.2: Validate snapshot when removing dangling deletes#18061manuzhang wants to merge 2 commits into
manuzhang wants to merge 2 commits into
Conversation
Pin dangling-delete discovery to one snapshot and validate that snapshot on every commit attempt so concurrent rollbacks cannot remove valid deletes. Generated-by: Codex
Use a fixed v2 table and position-delete file because these tests exercise snapshot concurrency rather than differences between table formats. Generated-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A rollback between dangling-delete discovery and commit can restore data files that still need the candidate deletes. The cleanup rewrite can then rebase onto the restored snapshot and remove valid deletes, making deleted rows visible again.
Pin all metadata scans to the planning snapshot and validate that it is still current inside every commit attempt, including retries. Cleanup fails with a
ValidationExceptionif the snapshot changes; tables without a snapshot remain a no-op. This change is limited to Spark 4.2 and uses the existing core validation API.Adds regression coverage for rollback before commit, rollback during a commit retry, snapshot changes during planning, and empty tables. The concurrency tests use a fixed v2 table and position-delete file. Related to #13120.
Validation:
TestRemoveDanglingDeleteAction: 18 tests, 17 passed and 1 expected skip.spotlessCheck.AI Disclosure