Repository navigation
Fix multipart upload abort on transient part retry (#499) - #526
Open
hanabanaka wants to merge 1 commit into
Open
hanabanaka wants to merge 1 commit into
hanabanaka wants to merge 1 commit into
Conversation
Each ciphertext part was wrapped in NoRetriesAsyncRequestBody, which throws on re-subscribe. When the SDK retried a part after a transient failure, the whole upload aborted. That guard is only needed for live AES-GCM cipher streams (re-running the cipher would reuse the key/IV). Parts are static ciphertext files on disk, so re-reading them is safe. Pass the file body straight through so the SDK can retry the part natively; the live-cipher call sites keep the guard. Adds UploadObjectObserverTest (no AWS; mocks S3AsyncClient) covering part-body re-subscription on retry and temp-file cleanup after upload.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: #499
Description of changes:
In the high-level multipart put path (
enableMultipartPutObject(true)), eachalready-encrypted ciphertext part was uploaded with its body wrapped in
NoRetriesAsyncRequestBody, which throws"Re-subscription is not supported!"on any second
subscribe(). When the async SDK retried a part after a transientnetwork failure, it re-subscribed to the body, the wrapper threw, and the entire
multipart upload aborted (issue #499).
That guard exists to stop re-subscription of live AES-GCM cipher streams,
where re-running the cipher would reuse the key/IV. It does not apply here: by
the time parts upload, the object has already been encrypted once and written to
temp ciphertext files on disk (
MultipartUploadObjectPipeline.putLocalObject),so each part is a static file. Re-reading it yields identical bytes with no
cipher re-run, and
uploadPartis idempotent peruploadId/partNumber.Fix: pass the file-based part body straight to the SDK in
UploadObjectObserver.onPartCreateso native per-part retry works. The threecall sites that wrap genuine live-cipher streams
(
MultipartUploadObjectPipeline,S3AsyncEncryptionClient) are untouched, sothe GCM protection stays intact.
Tests: adds
UploadObjectObserverTest(no AWS; mocksS3AsyncClient) coveringthe three branches of
onPartCreate— part-body re-subscription on retry (theregression guard), temp-file cleanup and delete-notification after upload, and
unwrapping an upload failure into
S3EncryptionClientException. There-subscription and unwrap tests are mutation-verified.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Check any applicable: