Skip to content

Prefer CopyRequest for move API - #1182

Open
pmscosta wants to merge 3 commits into
masterfrom
refactor-move-api-copy-object
Open

pmscosta wants to merge 3 commits into
masterfrom
refactor-move-api-copy-object

Conversation

@pmscosta

@pmscosta pmscosta commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

We detected livelock issue with using the AWS S3 CRT client in our current move implementation.
It seemed to be caused by a CRT upload fed by a blocking InputStream that is itself created by another CRT download with the same client.
If they both land on the same event loop and the object is large (from our testing, above 16MB), the upload loops infinitely and stops the download from fetching data.

Does this change relate to existing issues or pull requests?

Follow-up on #1128

Does this change require an update to the documentation?

No.

How has this been tested?

Local ad-hoc testing. Verified this caused no deadlocks when moving files agains a MinIO backend (since the problem is a logic one with aws).

This also seems to provide more throughput. I didn't measure against a true S3 bucket, but against MinIO backend.

this branch:

scenario: server-copy; size: 512 MiB; runs: 5
copy 1: 0.498s, 1028.0 MiB/s
copy 2: 0.442s, 1159.3 MiB/s
copy 3: 0.312s, 1639.8 MiB/s
copy 4: 0.364s, 1408.4 MiB/s
copy 5: 0.313s, 1636.7 MiB/s
mean: 1374.4 MiB/s

main:

scenario: streamed-move; size: 512 MiB; runs: 5
copy 1: 2.980s, 171.8 MiB/s
copy 2: 2.899s, 176.6 MiB/s
copy 3: 2.799s, 182.9 MiB/s
copy 4: 2.738s, 187.0 MiB/s
copy 5: 2.762s, 185.3 MiB/s
mean: 180.7 MiB/s

We detected livelock issue with using the AWS S3 CRT client in our current move implementation.
It seemed to be caused by a CRT upload fed by a blocking InputStream that is itself created by another CRT download with the same client.
If they both land on the same _event loop_ and the object is large (from our testing, above 16MB), the upload loops infinitely and stops the download from fetching data.
@pmscosta
pmscosta requested a review from bdmendes September 9, 2026 12:57
@pmscosta pmscosta self-assigned this Sep 9, 2026
@pmscosta
pmscosta marked this pull request as ready for review September 9, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant