Skip to content

feat(storage): precompute checksums by default for unbuffered resumable uploads - #6769

Open
xlai20 wants to merge 15 commits into
googleapis:mainfrom
xlai20:feature-precompute-checksums
Open

xlai20 wants to merge 15 commits into
googleapis:mainfrom
xlai20:feature-precompute-checksums

Conversation

@xlai20

@xlai20 xlai20 commented Sep 10, 2026

Copy link
Copy Markdown
Member

This PR enables automatic checksum precomputation for unbuffered resumable uploads and updates the public API to give callers explicit control over this behavior.

Part of #6461

Key Public API Changes

  • Deprecated API:
    • WriteObject::precompute_checksums(): Deprecated because checksums are now automatically computed and validated by default across all upload modes.
  • New Replacement API:
    • WriteObject::with_checksum_precomputation(bool): Added with a default value of true. Callers performing unbuffered resumable uploads can pass false to opt out and prioritize upload latency over upfront checksum validation.

Implementation Details

  • Unbuffered Resumable Precomputation: When precomputation is enabled (default) and an upfront CRC32C is not already known, the client computes the checksum over the seekable payload before initiating the resumable session, passes the precomputed hash in session creation metadata, and streams the upload without redundant double-hashing.
  • Version Bump: Bumped google-cloud-storage version to 1.19.0 across Cargo.toml, librarian.yaml, and README.md.
  • Tests: Added comprehensive unit tests covering builder defaults, wire hash verification, payload rewinding, error handling on seek/read failures, retry memoization (ensuring no re-computation across transient retries), and non-redundant integration test cases.

@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Sep 10, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces automatic checksum precomputation for unbuffered resumable uploads by default, while deprecating the manual precompute_checksums method. It also adds a with_checksum_precomputation builder method to allow disabling this feature for faster uploads. The review feedback highlights an inefficiency in the precomputation loop, where checksums are calculated twice redundantly, and suggests refactoring the stream handling to avoid this overhead.

Comment thread src/storage/src/storage/perform_upload/unbuffered.rs
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.98%. Comparing base (1cbc193) to head (4081f47).
⚠️ Report is 31 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #6769    +/-   ##
========================================
  Coverage   96.97%   96.98%            
========================================
  Files         325      325            
  Lines      106409   106666   +257     
========================================
+ Hits       103195   103450   +255     
- Misses       3214     3216     +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…mputation

Leverage ChecksummedSource's built-in hashing during the precomputation pass in send_unbuffered_resumable rather than manually hashing chunks in parallel. Add take_checksum() to ChecksummedSource to finalize and reset the hasher back to None, ensuring subsequent streaming during upload does not perform redundant on-the-fly checksumming.
…state reset into final_checksum and reset_checksum

Replace take_checksum with an explicit reset_checksum method on ChecksummedSource, allowing callers to inspect the calculated checksum via final_checksum and explicitly reset the internal hasher state when finished.
…t steps

Organize the precomputation block in send_unbuffered_resumable into 3 distinct logical sections: (1) precomputation of checksum, (2) update checksum in resource metadata, and (3) clean up and reset work.
@xlai20
xlai20 marked this pull request as ready for review September 10, 2026 10:19
@xlai20
xlai20 requested review from a team as code owners September 10, 2026 10:19
Comment thread src/storage/src/storage/checksum/details.rs
Comment thread src/storage/src/storage/checksum/details.rs
Comment thread src/storage/src/storage/write_object.rs Outdated
@xlai20
xlai20 requested a review from vsharonlynn September 15, 2026 03:22
Comment thread src/storage/src/storage/checksum/details.rs
Comment thread src/storage/src/storage/write_object.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants