Preserve entity unlocks on late completion failures for .NET parity #169

Please preserve and finalize entity unlocks when a late worker failure makes an orchestration terminal, and handle permanent payload externalization failures without an endless abandon/redelivery loop. This is a parity follow-up to durabletask-dotnet PR 816, based on source inspection of current main (c713409eb9a4295884169c144df9221f9a42aef6).

Findings

The completion-size fallback drops entity releases.

• The normal task runtime releases critical sections in setCompleteInternal / releaseCriticalSection before producing its terminal completion action.
• The worker subsequently calls failOversizedOrchestratorResponse. When the response exceeds the completion bound, this function replaces the entire action list with one non-retriable Failed action. Existing SendEntityMessage.EntityUnlockSent actions are discarded.
• If a still-pending response is made terminal by this late fallback, releases for locks acquired in earlier history also need to be generated. The runtime's normal terminal cleanup has already been bypassed in that case.

Permanent payload errors are abandoned rather than completed safely.

• processOrchestration abandons the work item on every TransformOrchestratorResponse error, before reaching the completion-size fallback.
• Externalize can return the deterministic api.ErrLargePayloadTooLarge when the configured maximum is exceeded. Replaying the same response against the same cap repeats the abandonment and does not send its entity releases.

These are distinct paths: an externalization error currently abandons and does not send a Failed completion; an oversized aggregate response after externalization succeeds can send Failed while dropping unlocks. Both need to preserve the entity-cleanup invariant.

Regression recipes

1. Extend the existing TestFailOversizedOrchestratorResponse... coverage with a response containing two valid unlock actions and an oversized result or action batch. The current helper returns only Failed; it should retain the two unlocks before one Failed completion while discarding unrelated unpersistable work.
2. Replay a native orchestration that acquired two locks and then produced an oversized pending entity call or fan-out. When late worker handling makes it terminal, releases must be generated even if none were in the original response.
3. Use a fake large-payload store and input exceeding MaxPayloadBytes, and record completion/abandon calls. The permanent cap failure should produce a safe non-retriable completion with the releases instead of repeating abandonment indefinitely. Keep transient storage failures on the retry/abandon path.

The source paths establish the behavior above; these recipes are requested automated regression coverage, not claimed live DTS test runs.

Acceptance Criteria

• Late terminal response rewriting preserves already-emitted entity unlock actions and their owner, critical-section ID and destination.
• Any remaining acquired-lock obligations are finalized after the terminal decision and before transport submission, including when the original response was pending.
• Deterministic permanent payload errors such as ErrLargePayloadTooLarge produce a small non-retriable failure with safe entity cleanup rather than a permanent abandon/redelivery loop.
• Transient storage failures and cancellation retain retry/abandon semantics, and completion retries do not duplicate unlock actions.
• Regressions cover zero/partial/all pre-existing releases and demonstrate a second caller progressing after cleanup.
• Existing native-token and wrapped-reference idempotency remains intact.

Related

• .NET parity reference: PR 816, currently proposed rather than a released fix.
• Go already recognizes native tokens and wrapped references before raw-payload size checks in Externalize, and completion transport retries run after transformation. This report does not claim Go has the .NET token-reupload bug.