Repository navigation
Align relative JSONL seeks using the actual read length - #682
Open
charan-rathore wants to merge 1 commit into
Open
charan-rathore wants to merge 1 commit into
charan-rathore wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The change
Relative JSONL seeking silently drops later records when a seekable file-like reader returns short reads.
_align_to_newlinecounts the requested block size instead of the amount actually read, then seeks well past EOF. For example, a three-byte-capped reader seeks to 28694 in a 220-byte file where the ordinary reader returns four later records.Use
len(cur)both in the accumulated distance and when subtracting the final chunk. Keep the existing seek position at the newline. Eighteen regression cases cover binary/text readers, positive/negative relative positions, and one-, three-, and five-unit read caps, comparing against ordinary reads of the same data.This overlaps the function edited by #633, but that PR addresses decoding when a relative seek splits UTF-8 and retains the requested-block-size arithmetic. #669 handles short reads in
reverse_iter_lines, a separate path. Reverse-mode behavior with short-reading streams is not claimed fixed here.Validation on Python 3.12/macOS:
git diff --checkpasses. The first sandboxed full run could not bind two loopback sockets; the unchanged suite passes with local socket access. The full tox/cross-platform matrix was not run.The backstory
This was found through an AI-assisted source audit, not a production incident. The audit followed the file-like read contract through relative seeking and found a silent-loss case rather than an exception. The useful distinction was between the requested chunk size and the cursor distance actually consumed; the change remains confined to that arithmetic.
AI assistance