Repository navigation
Core: Reduce DeleteFileIndex memory for file-scoped position deletes - #18359
Open
raunaqmorarka wants to merge 2 commits into
Open
raunaqmorarka wants to merge 2 commits into
raunaqmorarka wants to merge 2 commits into
Conversation
A position delete whose file_path bounds name a single data file is matched by location, so the copy records the data file in referencedDataFile and drops column stats. JOL retained bytes per delete, 100k v2 position deletes, 150-char paths: | data files | before | after | |------------|----------|---------| | 100k (1:1) | 2122.6 B | 810.6 B | | 37k (2.7:1)| 1908.1 B | 717.0 B |
Position deletes of the same data file hold one location String. JOL retained bytes per delete, 100k v2 position deletes, 150-char paths, 2.7 deletes per data file: 717.0 B before, 596.1 B after.
raunaqmorarka
force-pushed
the
delete-file-index-memory
branch
from
October 8, 2026 03:23
90ee9c7 to
5d9dd27
Compare
findinpath
reviewed
Oct 8, 2026
Comment on lines
+518
to
+522
| private static DeleteFile copyWithMinStats(DeleteFile file) { | ||
| if (file.content() == FileContent.EQUALITY_DELETES) { | ||
| // copy with stats for better filtering against data file stats | ||
| return file.copyWithStats(Set.copyOf(file.equalityFieldIds())); | ||
| } |
Contributor
There was a problem hiding this comment.
see #18338 (comment)
This changes what DeleteFile.lowerBounds()/valueCounts()/nullValueCounts() return, not just how they're used for predicate matching.
cc @pvary
Contributor
Author
There was a problem hiding this comment.
These copies only reach callers through FileScanTask.deletes(). Metadata tables open manifests directly and still see full stats. On main the index already keeps only the file_path stats on these copies and drops the rest. For a file-scoped delete, this PR moves that one value from the file_path bounds into referencedDataFile. So ContentFileUtil.referencedDataFile() and isFileScoped() return the same result. Position deletes that span several data files keep their bounds.
findinpath
reviewed
Oct 8, 2026
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DeleteFileIndexcopies every v2 position delete with itsfile_pathlower and upper bounds, plus the column sizes, value counts and null counts for that column. For a file-scoped position delete the two bounds are the same path, which the index only uses to compute the referenced data file location. This PR copies such deletes without column stats and records the location inreferencedDataFile, the same shape a DV already has. Deletes that reference the same data file share one locationString. Position deletes that span multiple data files keep thefile_pathbounds as before.Motivation
A Trino coordinator heap dump showed
DeleteFileIndexholding 4.3M position delete entries for 1.57M data files at about 2.25 KB per entry, roughly 9.7 GB, and the coordinator ran out of memory while planning. Most of the retained size per entry was the twoByteBufferbounds and the stats maps around them, duplicated for every delete on the same data file.Results
JOL retained size of a
DeleteFileIndexbuilt from a v2 delete manifest with 100k position deletes and 150-char paths:Testing
Extended
DeleteFileIndexTestBasewith file-scoped, presetreferencedDataFile, and multi-file position delete cases that check the copied fields and stats.AI assistance (Claude Code) was used to draft the code, tests and the JOL measurement harness. I reviewed the logic and ran the tests and measurements myself.