Skip to content

Align relative JSONL seeks using the actual read length - #682

Open
charan-rathore wants to merge 1 commit into
mahmoud:masterfrom
charan-rathore:fix/jsonl-short-read-alignment-20261030
Open

charan-rathore wants to merge 1 commit into
mahmoud:masterfrom
charan-rathore:fix/jsonl-short-read-alignment-20261030

Conversation

@charan-rathore

Copy link
Copy Markdown

The change

Relative JSONL seeking silently drops later records when a seekable file-like reader returns short reads. _align_to_newline counts the requested block size instead of the amount actually read, then seeks well past EOF. For example, a three-byte-capped reader seeks to 28694 in a 220-byte file where the ordinary reader returns four later records.

Use len(cur) both in the accumulated distance and when subtracting the final chunk. Keep the existing seek position at the newline. Eighteen regression cases cover binary/text readers, positive/negative relative positions, and one-, three-, and five-unit read caps, comparing against ordinary reads of the same data.

This overlaps the function edited by #633, but that PR addresses decoding when a relative seek splits UTF-8 and retains the requested-block-size arithmetic. #669 handles short reads in reverse_iter_lines, a separate path. Reverse-mode behavior with short-reading streams is not claimed fixed here.

Validation on Python 3.12/macOS:

  • Baseline: 18 new regression failures, 12 existing JSONL tests pass.
  • Fixed JSONL module: 30 pass.
  • Full tests and module doctests: 793 pass, 22 subtests pass.
  • git diff --check passes. The first sandboxed full run could not bind two loopback sockets; the unchanged suite passes with local socket access. The full tox/cross-platform matrix was not run.

The backstory

This was found through an AI-assisted source audit, not a production incident. The audit followed the file-like read contract through relative seeking and found a silent-loss case rather than an exception. The useful distinction was between the requested chunk size and the cursor distance actually consumed; the change remains confined to that arithmetic.

AI assistance

  • Harness/tooling: OpenAI Codex, with Instinct used for an independent diagnosis/overlap check.
  • Model(s): GPT-6-based Codex; Instinct's exact model identifier is not exposed here.
  • Project-specific review prompt sent to Instinct, verbatim (broader orchestration instructions omitted):

I am reserving a distinct Boltons JSONLIterator short-read alignment defect: _align_to_newline adds requested blocksize rather than len(cur), so a seekable reader returning short chunks can seek far beyond EOF and silently drop later JSON records. Existing #633 covers UTF-8 decoding and #669 covers reverse_iter_lines short reads, not this offset accumulator. Please review this specific diagnosis for overlap/correctness while I reproduce it and test a two-line len(cur) fix; do not edit these files. This is source-audit discovery, not production usage.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant