fix: DurableFuture.anyOf blocking without deregistering the active thread - #730
Conversation
d2e3fc5 to
86b16e1
Compare
34c7858 to
16a32a4
Compare
| // Attach the callback before the caller decides whether to deregister. This ensures the combined future cannot | ||
| // settle between "is it done?" and "deregister the caller thread" without updating the wait state below. | ||
| var reactivationFuture = anyOfFuture.whenComplete((ignored, failure) -> { | ||
| synchronized (wait) { |
There was a problem hiding this comment.
This is the new lock logic that reviewers should pay extra attention to.
| var completionLock = | ||
| completionLockParent == null ? completionLockFor(operation) : completionLockFor(completionLockParent); | ||
| operationCompletionLocks.put(operation.getOperationId(), completionLock); |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_4hw3zgpptmxeatdkdawqqh6a3f
[P1] Publish the final child completion lock before registration
For map/parallel children, the preceding registeredOperations.put(...) makes the operation visible before these lines install its parent lock. A concurrent checkpoint can create and use a child-local lock, which is then replaced here. Consequently, get() may check and attach its callback under a different lock from completion; completion between those actions registers the still-active caller, and the following deregistration leaves it inactive, permitting premature suspension. Install the final lock before publishing the operation to registeredOperations (or make both publications atomic), and add a child-registration/checkpoint race test.
Codex AI reviewFound one high-severity thread-coordination race in child-operation registration. Reviewed commit |
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue Link, if available
#707
Description
Fixes
DurableFuture.anyOfsuspension handling.Previously
anyOfwaited onCompletableFuture.anyOf(...).join()directly from the durable context thread. If none of the candidate durable operations could settle in the current invocation, that thread stayed registered as active, so the execution manager never observed an idle execution and could not suspend.This change routes first-completion waits through SDK thread coordination:
DurableFuture.anyOfnow validates that at least one SDK-created durable future is provided, converts those futures toBaseDurableOperations, waits for the first operation to settle, and returns that operation's result.BaseDurableOperation.waitForFirstOperationCompletion(...)andExecutionManager.waitForFirstOperationCompletion(...)provide the multi-operation equivalent ofget(): the caller thread deregisters while waiting, is re-registered when the first operation settles successfully, and suspension/failure paths propagate through the existing execution error handling.ExecutionManager.waitForOperationCompletion(...)soget()andanyOf()share the same manager-owned coordination model.ExecutionManager, including the existing parent-lock behavior for map/parallel child operations, so checkpoint state publication and completion-future notification remain serialized with waiter registration.Demo/Screenshots
n/a
Checklist
Testing
Automated coverage was added/updated for the new first-completion path and the original suspension regression. GitHub checks for this commit are currently pending.
Unit Tests
BaseDurableOperationFirstCompletionTestcoverage for input validation, same-manager validation, step-thread rejection, already-settled operations, suspension while no operation can settle, caller reactivation after a later settlement, reactivation-before-return ordering, and suspension propagation.DurableFutureTestcoverage foranyOfempty/null input and non-SDK future validation.ExecutionManagerTestcoverage around manager-owned completion locks, checkpoint update atomicity, single-operation waiter reactivation, deferred suspension, and checkpoint delivery ordering.Integration Tests
DurableFutureAnyOfSuspensionTestwith local durable-runner coverage forfuture.get(),DurableFuture.allOf(...), andDurableFuture.anyOf(...)suspending instead of leaving the execution blocked.Manual E2E Test
DurableFuture.anyOf()and 2 async callback operationsScreenshots
Before and after screenshots below.
Note that the before has an invocation timeout immediately after both callbacks are submitted. This is because execution is not suspending and instead runs until timeout.
Both before and after show multiple invocation timeouts after the first callback resolves. This is a separate issue tracked under #734.
Before
After