Skip to content

fix: bound invocation lifecycle and cleanup - #733

Draft
zhongkechen wants to merge 16 commits into
feature/lmi-invocation-scopefrom
fix/lmi-invocation-lifecycle
Draft

zhongkechen wants to merge 16 commits into
feature/lmi-invocation-scopefrom
fix/lmi-invocation-lifecycle

Conversation

@zhongkechen

Copy link
Copy Markdown
Contributor

Issue Link, if available

Partial fix for #726, covering invocation deadlines, bounded shutdown, root-cleanup/PENDING ordering, and interruption-safe checkpoint cleanup. Stacked on #731. Shared fixed-executor starvation remains separate follow-up work.

Description

This turns the task ownership added by #731 into a bounded invocation lifecycle without changing durable replay identities or shutting down executors shared by other invocations.

  • Wait for the root outcome on the Lambda runtime thread with a deadline derived from Context.getRemainingTimeInMillis(), reserving cleanup and response headroom.
  • Enter DRAINING before response preparation, reject new durable operations/tasks, and hold PENDING, SUCCEEDED, or FAILED until the root wrapper and all invocation-owned task wrappers have actually exited.
  • Gracefully drain tasks first, then request cooperative cancellation through each task's underlying Future<?>. A task that ignores interruption cannot be forcibly stopped, but cleanup remains bounded and the closed manager rejects its later SDK operations/checkpoints.
  • Convert deadline or cleanup-budget exhaustion into a retryable invocation-level UnrecoverableDurableExecutionException; preserve the original cause and retain an original execution outcome as a suppressed exception when cleanup becomes the primary failure.
  • Keep checkpoint coordination alive while tasks drain, stop new checkpoint admission immediately before shutdown, flush already-admitted requests with the remaining budget, and close lifecycle state in finally even when draining or checkpoint shutdown fails.
  • Restore the Lambda runtime thread's interrupt flag after best-effort checkpoint shutdown.
  • Bound response-critical large-output checkpoint waits and ApiRequestDelayedBatcher shutdown; remove the previous unbounded operation Future.get() and batch-flush join() paths from invocation cleanup.
  • Fire onInvocationEnd only after task drain and checkpoint cleanup have completed.

The current internal budgets are 500 ms execution-deadline headroom, 100 ms response headroom, up to 5 seconds graceful task drain, up to 1 second cancellation drain, and a 30-second cleanup cap for local/null-context execution. A real Lambda context always constrains these caps further.

This PR targets feature/lmi-invocation-scope, the head branch of #731. It should be retargeted to main after #731 lands.

Testing

  • mvn -pl sdk spotless:check test — 1,178 tests passed, 3 opt-in LMI tests skipped.
  • mvn test — all 10 reactor modules passed, including 405 SDK integration tests, plugins, examples, conformance fixtures, and LMI fixture compilation; 31 cloud example tests remained disabled.
  • mvn -pl sdk -am -DskipTests javadoc:javadoc — passed; two pre-existing WaitForConditionOperation link warnings.
  • mvn -pl sdk '-Dtest=LmiLifecycleRegressionTest#pendingMustWaitForRootFinally+deadlineMustCancelAnInvocationOwnedTaskAndBoundCleanup' -Dtest.lmi.regressions.enabled=true test — both lifecycle regressions passed.
  • Full opt-in LmiLifecycleRegressionTest — lifecycle/deadline cases passed; the two fixed-executor parameterizations still failed as expected because executor separation is not part of this PR.
  • Added default unit coverage for stuck-root deadline cancellation, response-critical checkpoint timeout, interrupted drain cleanup/interrupt restoration, bounded batcher shutdown, and PENDING plus onInvocationEnd ordering after root finally.

Cloud LMI execution was not run locally because it requires the configured persistent capacity provider; #728 supplies that validation workflow.

Checklist

  • Preserved replay operation identity and existing public durable APIs.
  • Never shuts down the shared user executor.
  • Keeps the PR in Draft state for stacked review.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@zhongkechen
zhongkechen added this pull request to stack #732 September 25, 2026 17:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant