Skip to content

fix: DurableFuture.anyOf blocking without deregistering the active thread - #730

Open
hln33 wants to merge 1 commit into
mainfrom
707-bug-durablefutureanyof-blocks-without-deregistering-the-active-thread-so-the-execution-never-suspends
Open

hln33 wants to merge 1 commit into
mainfrom
707-bug-durablefutureanyof-blocks-without-deregistering-the-active-thread-so-the-execution-never-suspends

Conversation

@hln33

@hln33 hln33 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

#707

Description

Fixes DurableFuture.anyOf suspension handling.

Previously anyOf waited on CompletableFuture.anyOf(...).join() directly from the durable context thread. If none of the candidate durable operations could settle in the current invocation, that thread stayed registered as active, so the execution manager never observed an idle execution and could not suspend.

This change routes first-completion waits through SDK thread coordination:

  • DurableFuture.anyOf now validates that at least one SDK-created durable future is provided, converts those futures to BaseDurableOperations, waits for the first operation to settle, and returns that operation's result.
  • BaseDurableOperation.waitForFirstOperationCompletion(...) and ExecutionManager.waitForFirstOperationCompletion(...) provide the multi-operation equivalent of get(): the caller thread deregisters while waiting, is re-registered when the first operation settles successfully, and suspension/failure paths propagate through the existing execution error handling.
  • Single-operation waiting was moved into ExecutionManager.waitForOperationCompletion(...) so get() and anyOf() share the same manager-owned coordination model.
  • Operation completion locking is now owned by ExecutionManager, including the existing parent-lock behavior for map/parallel child operations, so checkpoint state publication and completion-future notification remain serialized with waiter registration.
  • First-completion waits reject null/empty inputs, non-SDK futures, operations from different execution managers, and nested waits from step threads.

Demo/Screenshots

n/a

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Automated coverage was added/updated for the new first-completion path and the original suspension regression. GitHub checks for this commit are currently pending.

Unit Tests

  • Added BaseDurableOperationFirstCompletionTest coverage for input validation, same-manager validation, step-thread rejection, already-settled operations, suspension while no operation can settle, caller reactivation after a later settlement, reactivation-before-return ordering, and suspension propagation.
  • Added DurableFutureTest coverage for anyOf empty/null input and non-SDK future validation.
  • Updated ExecutionManagerTest coverage around manager-owned completion locks, checkpoint update atomicity, single-operation waiter reactivation, deferred suspension, and checkpoint delivery ordering.
  • Updated operation unit tests and shared mocks to use the manager-owned completion coordination.

Integration Tests

  • Added DurableFutureAnyOfSuspensionTest with local durable-runner coverage for future.get(), DurableFuture.allOf(...), and DurableFuture.anyOf(...) suspending instead of leaving the execution blocked.

Manual E2E Test

  • Deployed a real Lambda Durable Function that uses DurableFuture.anyOf() and 2 async callback operations
  • Verified that the function does not constantly invocation timeout when all callbacks are pending which was the prior behaviour
    • Since the function was not suspending, the lambda continued running until invocation timeout

Screenshots

Before and after screenshots below.

Note that the before has an invocation timeout immediately after both callbacks are submitted. This is because execution is not suspending and instead runs until timeout.

Both before and after show multiple invocation timeouts after the first callback resolves. This is a separate issue tracked under #734.

Before
Screenshot 2026-09-24 at 5 55 44 PM
After
Screenshot 2026-09-24 at 5 56 05 PM

@hln33
hln33 force-pushed the 707-bug-durablefutureanyof-blocks-without-deregistering-the-active-thread-so-the-execution-never-suspends branch from d2e3fc5 to 86b16e1 Compare September 23, 2026 16:48
@hln33 hln33 changed the title fix(sdk): deregister awaited thread in anyOf so execution suspends wip: fix DurableFuture.anyOf blocking without deregistering the active thread Sep 23, 2026
@hln33
hln33 force-pushed the 707-bug-durablefutureanyof-blocks-without-deregistering-the-active-thread-so-the-execution-never-suspends branch from 34c7858 to 16a32a4 Compare September 24, 2026 23:51
// Attach the callback before the caller decides whether to deregister. This ensures the combined future cannot
// settle between "is it done?" and "deregister the caller thread" without updating the wait state below.
var reactivationFuture = anyOfFuture.whenComplete((ignored, failure) -> {
synchronized (wait) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the new lock logic that reviewers should pay extra attention to.

@hln33
hln33 marked this pull request as ready for review September 25, 2026 00:23
@hln33
hln33 requested a review from a team September 25, 2026 00:23
@hln33
hln33 deployed to ai-pr-review-runtime September 25, 2026 00:23 — with GitHub Actions Active
@hln33
hln33 requested a review from zhongkechen September 25, 2026 00:23
@hln33
hln33 deployed to ai-pr-review-runtime September 25, 2026 00:31 — with GitHub Actions Active
Comment on lines +219 to +221
var completionLock =
completionLockParent == null ? completionLockFor(operation) : completionLockFor(completionLockParent);
operationCompletionLocks.put(operation.getOperationId(), completionLock);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review · Finding arf_v1_4hw3zgpptmxeatdkdawqqh6a3f

[P1] Publish the final child completion lock before registration

For map/parallel children, the preceding registeredOperations.put(...) makes the operation visible before these lines install its parent lock. A concurrent checkpoint can create and use a child-local lock, which is then replaced here. Consequently, get() may check and attach its callback under a different lock from completion; completion between those actions registers the still-active caller, and the following deregistration leaves it inactive, permitting premature suspension. Install the final lock before publishing the operation to registeredOperations (or make both publications atomic), and add a child-registration/checkpoint race test.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Found one high-severity thread-coordination race in child-operation registration.

Reviewed commit 16a32a4c34b03f42131862d999a1c6588c91dd17. Workflow run

@hln33 hln33 changed the title wip: fix DurableFuture.anyOf blocking without deregistering the active thread fix: DurableFuture.anyOf blocking without deregistering the active thread Sep 25, 2026

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 16a32a4c Deployed Sep 25, 2026 by hln33 via ai-pr-review / Codex review / Generate Codex review #1025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: DurableFuture.anyOf blocks without deregistering the active thread, so the execution never suspends

1 participant