Repository navigation
Conversation
chrisburr
force-pushed
the
feat/sandbox-diracx-future-client
branch
from
October 19, 2026 11:38
f005ba1 to
c649b15
Compare
Accessing an S3 storage element through the S3Gateway was broken with S3
implementations which validate presigned requests:
- the gateway looked up urls.get("Fields") instead of the fields of each URL,
so the checksum metadata was never stored on upload
- boto does not add the Fields of generate_presigned_post to the policy, so
once passed S3 rejected them as "Extra input fields"
- exists() sent a GET to a URL signed for HEAD, which is rejected as Forbidden
S3Mock is pinned to 3.12.0, which does not support presigned POST uploads and does not validate credentials or signatures. Use SeaweedFS, as DiracX does, with real credentials for the S3 storage elements. This also allows uploads through the S3Gateway to be tested again.
…eClient When /DiracX/FutureClientEnabled/WorkloadManagement/SandboxStore is enabled, sandboxes are uploaded to and downloaded from the DiracX S3 backend using presigned URLs instead of being proxied through the SandboxStore service. Transfers are streamed to disk, retried with a jittered backoff and the checksum embedded in the PFN is verified after download. Sandboxes which are not stored in S3 are still downloaded through their storage element.
chrisburr
force-pushed
the
feat/sandbox-diracx-future-client
branch
from
October 19, 2026 12:13
c649b15 to
34f364d
Compare
Member
Author
|
Needs #8836 |
chrisburr
marked this pull request as ready for review
October 19, 2026 19:00
chrisburr
requested review from
andresailer,
atsareg,
chaen and
fstagni
as code owners
October 19, 2026 19:00
Contributor
|
A documentation line should be added somewhere to state that this future client is available -- prob in DiracX. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
FutureClient/SandboxStoreClientwhich uploads, downloads and looks up sandboxes directly against the DiracX S3 backend using presigned URLs, instead of proxying them through theSandboxStoreservice.To test this in CI, the S3Mock container is replaced by SeaweedFS (as used by DiracX). S3Mock 3.12.0 does not support presigned POST uploads and does not validate presigned requests, which was hiding several bugs in the S3Gateway that are also fixed here:
urls.get("Fields")instead of the fields of each URL, so the checksum was never stored on uploadexists()sent a GET to a URL signed for HEADBEGINRELEASENOTES
*WorkloadManagement
NEW: SandboxStoreClient can access the DiracX sandbox store directly when the SandboxStore FutureClient is enabled
*DataManagement
FIX: S3Gateway presigned URLs work with S3 implementations which validate them, and store the checksum on upload
ENDRELEASENOTES