Skip to content

Fix Azure Blob payload externalization for continue-as-new carryover - #828

Merged
wangbill (YunchuWang) merged 1 commit into
mainfrom
yunchuwang-continue-as-new-payloads
Oct 30, 2026
Merged

wangbill (YunchuWang) merged 1 commit into
mainfrom
yunchuwang-continue-as-new-payloads

Conversation

@YunchuWang

Copy link
Copy Markdown
Member

Summary

What changed?

  • Traverse CompleteOrchestrationAction.CarryoverEvents through the existing history-event payload mapping and bounded upload runner.
  • Apply the existing UTF-8 externalization threshold and MaxPayloadBytes checks to native carryover payloads without changing event identity, order, names, or completion metadata.
  • Add 40 regression cases covering all 18 mapped history-event types, ASCII/multibyte boundaries, protobuf round-trip hydration, oversized responses, cancellation, bounded concurrency, drainage, and permanent/transient upload failures.
  • Add a release note to CHANGELOG.md.

Why is this change needed?

  • Completion carryover events were not collected for externalization, allowing their payloads to bypass both configured size limits and leave a continue-as-new response above the transport bound.
  • Oversized carryover payloads must use the existing non-retriable failed-completion conversion rather than remain inline or be abandoned indefinitely.

Issues / work items


Project checklist

  • Release notes are not required for the next release
    • Otherwise: Notes added to release_notes.md
  • Backport is not required
    • Otherwise: Backport tracked by issue/PR N/A (maintainer decision pending)
  • All required tests have been added/updated (unit tests, E2E tests)
  • Breaking change?
    • If yes:
      • Impact: N/A.
      • Migration guidance: N/A.

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No
  • Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
  • Yes, an AI agent generated most of this PR

If AI was used:

  • Tool(s): GitHub Copilot.
  • AI-assisted areas/files: The sidecar payload interceptor, its regression tests, and CHANGELOG.md.
  • What you changed after AI output: Agent iteration resolved a nullable boundary-test value and preserved unrelated changelog bytes. No human post-edit or verification is claimed.

AI verification (required if AI was used):

  • I understand the code and can explain it
  • I verified referenced APIs/types exist and are correct
  • I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
  • I reviewed concurrency/async behavior
  • I checked for unintended breaking or behavior changes

Human attestations and the backport decision are left for review.


Testing

Automated tests

  • Result: Passed; 99 focused tests, 0 failed, 0 skipped. The extension also builds for all four target frameworks.
dotnet restore .\test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configfile .\nuget.config --verbosity minimal
# Passed; performed after the initial --no-restore attempt reported missing assets (NETSDK1004).

dotnet test .\test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configuration Release --no-restore --filter "FullyQualifiedName~AzureBlobPayloadsSideCarInterceptorTests|FullyQualifiedName~LargePayloadTests|FullyQualifiedName~AutochunkTests" --logger "console;verbosity=minimal" --verbosity quiet
# Passed: 99, Failed: 0, Skipped: 0; includes all 40 new carryover cases.

dotnet build .\src\Extensions\AzureBlobPayloads\AzureBlobPayloads.csproj --configuration Release --no-restore --verbosity quiet
# Passed for netstandard2.0, net6.0, net8.0, and net10.0; 0 errors.
# Existing nullable, obsolete-API, and StyleCop warnings remain.

git --no-pager diff --check
# Passed on the complete patch before committing.

The focused reproduction was also run before the production fix:

dotnet test .\test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configuration Release --no-restore --filter FullyQualifiedName~ShrinksNativeCarryoverAndHydratesAfterProtobufRoundTrip --logger "console;verbosity=minimal" --verbosity quiet
# Failed as expected before the fix: expected 5 carryover uploads, observed 0.
# Passed after the fix as part of the 99-test run above.

The full solution and live Azure/DTS tests were not run.

Manual validation (only if runtime/behavior changed)

  • Environment (OS, .NET version, components): Windows, .NET SDK 10.0.403; in-memory payload store and the existing local gRPC integration harness.
  • Steps + observed results: Automated checks above, not manual cloud validation:
    1. Five native CarryoverEvents each contain exactly 1 MiB of serialized event input. Their original protobuf response exceeds 4 MiB; externalization uploads all five and reduces it below 4 MiB. Parsing the response from protobuf bytes and hydrating its events through the incoming worker-work-item path restores the complete original response, including event IDs, timestamps, order, names, completion token, version, tags, and continued-as-new status.
    2. ASCII, two-byte characters, and supplementary Unicode characters exercise inclusive threshold/maximum boundaries. A payload one byte above the maximum performs no upload and becomes a non-retriable failed completion.
    3. Gated carryover uploads verify the eight-operation bound, cancellation-token forwarding, stopping queued work, and drainage before return. Permanent 403 failures retain failed-completion conversion; transient 500 failures propagate without replacing the continued-as-new completion.
  • Evidence (optional): The original five-event regression failed before the traversal was added. No live Azure Blob Storage or DTS execution is claimed.

Notes for reviewers

  • No breaking change — public/protected APIs, protobuf fields, token format, serializer defaults, and orchestration replay logic are unchanged. The existing inbound history resolver restores externalized values before replay.
  • The production change is five lines and uses the same history-event mapping already used for rewind history. The shared concurrency runner and storage-failure conversion are unchanged.
  • Adjacent Fix payload retries and entity cleanup after storage failures #816 remains open. Its retry-idempotency, entity-cleanup, and worker-preparation changes are deliberately outside this PR.
  • Release notes are recorded in the existing CHANGELOG.md; this repository does not have the template's release_notes.md.

Traverse completion carryover history through the shared bounded payload externalization path. Cover UTF-8 limits, protobuf hydration, concurrency, cancellation, and payload failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 23:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the carryover payload externalization change.

No open issues 🎉

What changed in this PR

Fixes #819 by including native continue-as-new carryover payloads in Azure Blob externalization, reusing the SDK’s existing size checks and bounded upload handling.

Changes:

  • Processes completion carryover events through the existing history-event payload mapping.
  • Adds 40 regression cases covering payload mappings, UTF-8 size boundaries, round-trip hydration, concurrency, cancellation, and failure handling.
  • Adds an unreleased changelog entry; related #816 changes remain out of scope.
File Description
test/​Grpc.IntegrationTests/​AzureBlobPayloadsSideCarInterceptorTests.cs Adds carryover externalization and hydration regression coverage.
src/​Extensions/​AzureBlobPayloads/​Interceptors/​AzureBlobPayloadsSideCarInterceptor.cs Includes carryover history events in the bounded externalization runner.
CHANGELOG.md Documents the carryover payload fix.

Review effort: Balanced

@YunchuWang
wangbill (YunchuWang) merged commit 1fe77da into main Oct 30, 2026
11 checks passed
@YunchuWang
wangbill (YunchuWang) deleted the yunchuwang-continue-as-new-payloads branch October 30, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure Blob payload interceptor skips continue-as-new carryover payloads

3 participants