Skip to content

Reduce per-call SHA1 allocation in deterministic NewGuid() - #780

Open
Bernd Verst (berndverst) wants to merge 17 commits into
mainfrom
berndverst-reimagined-carnival
Open

Bernd Verst (berndverst) wants to merge 17 commits into
mainfrom
berndverst-reimagined-carnival

Conversation

@berndverst

@berndverst Bernd Verst (berndverst) commented Jul 24, 2026 •

Copy link
Copy Markdown
Member

Summary

What changed?

  • Reuse a lazily created SHA1 instance across deterministic NewGuid() calls only on internal worker-owned orchestration shims, and dispose it at the owning execution/cache boundary.
  • Keep all public DurableTaskShimFactory.CreateOrchestration overloads resource-neutral: they still create and dispose SHA1 per call, without imposing a hidden disposal obligation through the TaskOrchestration return type.
  • Track extended-session ownership by runner/cache generations. Taking a session transfers ownership before removal; stale eviction callbacks cannot dispose a resumed or reinserted shim. Shutdown, expiration, rejected reinsertion, and failed entry setup settle ownership exactly once.
  • Merge main at 936325b2. Resolve the test conflict by retaining both deterministic-GUID coverage and main's canceled external-event/continue-as-new regressions.
  • Add a warmed-up, thread-local allocation regression: 1,024 calls in managed-lifetime mode must allocate less than per-call mode and produce the same GUID sequence.

Why is this change needed?

  • Repeated SHA1 construction is unnecessary for GUID-heavy worker execution/replay. The namespace, SHA1 inputs, counter, byte order, version/variant bits, and output remain unchanged.
  • Value assessment: a real but workload-specific allocation improvement, not an established end-to-end throughput gain. The ownership infrastructure is substantially larger than the original hashing change, so this is the lowest-priority of the three performance PRs. A representative GUID-heavy workload benchmark should justify adopting the full lifecycle machinery; otherwise a narrower follow-up is preferable.
  • The optimization does not apply to external public-factory consumers. .NET Framework may still recreate native CAPI hash handles during reset; modern runtime gains should not be extrapolated to all supported hosts.

Issues / work items


Project checklist

  • Release notes are not required for the next release
    • Otherwise: Notes added to release_notes.md
  • Backport is not required
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • All required tests have been added/updated (unit tests, E2E tests)
  • Breaking change?
    • If yes:
      • Impact: N/A. No breaking change — shipped public signatures, valid orchestration replay output, GUID bytes, serialization, and wire fields remain unchanged. Cache initialization after disposal explicitly throws; disposal is idempotent.
      • Migration guidance: N/A. Existing public factory callers retain their prior resource-neutral behavior.

AI-assisted code disclosure (required)

Was an AI tool used? (select one)

  • No
  • Yes, AI helped write parts of this PR (e.g., GitHub Copilot)
  • Yes, an AI agent generated most of this PR

If AI was used:

  • Tool(s): GitHub Copilot; this refresh used GPT-6.1 Sol.
  • AI-assisted areas/files: Worker shim/factory, deterministic GUID implementation, extended-session ownership/cache, gRPC runners/processor, and related Worker tests.
  • What you changed after AI output: Retained the prior review-driven public/internal lifetime separation and generation checks; reconciled latest-main external-event tests; added an allocation-and-output equivalence regression. Earlier review suggestions that conflict with cache ownership transfer were not reapplied.

AI verification (required if AI was used):

  • I understand the code and can explain it
  • I verified referenced APIs/types exist and are correct
  • I reviewed edge cases/failure paths (timeouts, retries, cancellation, exceptions)
  • I reviewed concurrency/async behavior
  • I checked for unintended breaking or behavior changes

Testing

Automated tests

  • Result: Passed locally on the refreshed branch (Windows, .NET SDK 10.0.403).
  • dotnet restore Microsoft.DurableTask.sln --configfile nuget.config --verbosity quiet — passed. Required after the merge to refresh stale DurableTask.Core dependency assets.
  • dotnet test test\Worker\Core.Tests\Worker.Tests.csproj --configuration Release --no-restore --verbosity quiet — passed, 170/170, including allocation reduction and golden GUID/replay coverage.
  • dotnet test test\Worker\Grpc.Tests\Worker.Grpc.Tests.csproj --configuration Release --no-restore --verbosity quiet -p:WarningLevel=0 — passed, 218/218, including ownership, eviction, shutdown, and entity runner coverage.
  • dotnet test test\Grpc.IntegrationTests\Grpc.IntegrationTests.csproj --configuration Release --no-restore --filter "FullyQualifiedName~OrchestrationPatterns|FullyQualifiedName~TracingIntegrationTests" --verbosity quiet -p:WarningLevel=0 — passed, 54/54.
  • dotnet build src\Worker\Grpc\Worker.Grpc.csproj --configuration Release --no-restore --verbosity quiet -p:WarningLevel=0 — passed for all configured target frameworks.
  • The initial no-restore attempt encountered stale 3.8.0 versus 3.11.0 DurableTask.Core assets; the repository-configured restore resolved it. No dependency manifest workaround was added.
  • Full solution test execution, live DTS workloads, .NET Framework runtime tests, and an end-to-end performance benchmark were not run locally. Current GitHub checks are authoritative for CI.

Manual validation (only if runtime/behavior changed)

  • Environment (OS, .NET version, components): Windows; .NET SDK 10.0.403/runtime 10.0.14; in-process sidecar.
  • Steps + observed results:
    1. Compared warmed-up managed-lifetime and public/per-call GUID allocation paths across the same 1,024-call sequence; the allocation regression passed and all GUID outputs matched.
    2. Exercised runner/cache generation transfer, expiration, concurrent shutdown, rejected reinsertion, and disposal through automated regressions; passed.
    3. Ran orchestration-pattern and tracing integration coverage after the main merge; passed. No separate live-service/manual workload was run.
  • Evidence (optional): Exact local commands/results above; refreshed merge commit c068aec9.

Notes for reviewers

  • Prefer Perf: avoid repeated orchestration-history scans for tracing #788 and Perf: avoid repeated protobuf sizing during orchestration response chunking #784 ahead of this PR on benefit-to-maintenance-cost grounds. This PR is worth retaining, but adoption should be motivated by measured GUID-heavy workloads rather than a blanket performance claim.
  • MemoryCache.Clear() schedules eviction callbacks asynchronously; cache disposal does not promise that every callback has finished before returning.
  • Intentional prior review decisions remain intact: unconditional shim using is unsafe where ownership transfers to the extended-session cache, and retaining the wrapper reference preserves GetStatus behavior while its owned SHA1 is cleared.
  • No public API additions/removals, protobuf edits, generator interface changes, or serialization-default changes are introduced by this PR.

TaskOrchestrationContextWrapper.NewGuid() previously created and disposed a
new SHA1 instance on every call. Since a single wrapper instance is used for
the duration of one orchestration execution, and orchestrator code within
that execution runs sequentially (never concurrently), the SHA1 instance can
safely be cached on the wrapper and reused across calls via
HashAlgorithm.Initialize(), avoiding a per-call allocation.

The hashed name, DNS namespace bytes, byte-swap order, and RFC 4122
version/variant bit handling are all unchanged, so generated GUIDs are
byte-for-byte identical to before this change for the same inputs.

Adds regression tests asserting:
- stable, golden-value deterministic GUIDs for fixed inputs
- distinct GUIDs across repeated calls with the same instance/timestamp
- identical GUID sequences across two independent wrapper instances given
  identical inputs (simulated replay)
- the cached SHA1 instance is reused (not reallocated) across calls

Fixes #778

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI lite review requested due to automatic review settings July 24, 2026 22:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes deterministic GUID generation in the Worker shim by avoiding per-call SHA1 allocations inside TaskOrchestrationContextWrapper.NewGuid(), while adding regression tests intended to protect replay compatibility and validate the optimization.

Changes:

  • Cache and reuse a SHA1 instance in TaskOrchestrationContextWrapper.NewGuid() via Initialize() to reset state between calls.
  • Add golden-value determinism tests and replay-sequence tests for NewGuid().
  • Extend the test TestOrchestrationContext helper to allow pinning InstanceId and CurrentUtcDateTime inputs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
src/Worker/Core/Shims/TaskOrchestrationContextWrapper.cs Reuses a cached SHA1 instance for deterministic GUID hashing to remove per-call allocation/disposal.
test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs Adds deterministic GUID regression tests and enhances the test context helper to enable stable inputs.

Address blocking review feedback on PR #780 (issue #778):

1. Cached SHA1 instance is now disposed. TaskOrchestrationContextWrapper
   implements IDisposable, and TaskOrchestrationShim disposes the previous
   wrapper right before replacing it with a new one on each replay/decision
   task, so the cached hash algorithm no longer accumulates undisposed
   instances. The base TaskOrchestration type has no disposal hook, so a
   documented CA1001 suppression covers the final instance per orchestration
   execution's lifetime, which is left for the GC/finalizer.

2. Corrected the code comment to honestly scope the optimization per TFM,
   verified against reference source: on .NET Framework,
   SHA1CryptoServiceProvider.Initialize() disposes/recreates its native CAPI
   handle every call, so native-handle churn is not eliminated there, only
   managed-side allocation. On modern .NET (net5.0+) on Windows, the
   CNG-based implementation can reuse and reset its native handle, so this
   caching also avoids native allocation on those runtimes.

Added regression tests verifying Dispose() releases the cached SHA1
instance, is idempotent/safe to call without a prior NewGuid() call, and
that GUID generation remains byte-identical after a mid-sequence dispose.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 24, 2026 23:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:428

  • Using DateTime.Parse(..., null, ...) makes this test depend on the current thread culture. Use CultureInfo.InvariantCulture so the golden-value regression remains stable across environments.
            DateTime.Parse("2023-05-06T07:08:09.1234567Z", null, DateTimeStyles.RoundtripKind));

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:446

  • Using DateTime.Parse(..., null, ...) makes this test depend on the current thread culture. Use CultureInfo.InvariantCulture so the regression remains stable across environments.
            DateTime.Parse("2024-01-01T00:00:00.0000000Z", null, DateTimeStyles.RoundtripKind));

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:468

  • Using DateTime.Parse(..., null, ...) makes this test depend on the current thread culture. Use CultureInfo.InvariantCulture so the replay regression remains stable across environments.
        DateTime timestamp = DateTime.Parse("2022-11-11T11:11:11.1111111Z", null, DateTimeStyles.RoundtripKind);

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:576

  • Using DateTime.Parse(..., null, ...) makes this test depend on the current thread culture. Use CultureInfo.InvariantCulture so the golden-value regression remains stable across environments.
            DateTime.Parse("2023-05-06T07:08:09.1234567Z", null, DateTimeStyles.RoundtripKind));

TaskOrchestrationShim.Execute() runs exactly once per shim instance in
every real call site; extended-session resumption reuses the cached
executor via TaskOrchestrationExecutor.ExecuteNewEvents() and never
calls Execute() again. This meant the round-2 fix (disposing the
*previous* wrapper inside Execute() before replacing it) never
actually freed the *final* cached SHA1 for normal execution or for
extended sessions, leaking a native hash handle per orchestration
instance until GC/finalization.

This introduces deterministic ownership-based disposal instead:

- TaskOrchestrationShim now implements IDisposable directly (the
  defensive dispose-previous-wrapper logic in Execute() is kept as a
  safety net but documented as effectively unreachable for current
  callers).
- GrpcDurableTaskWorker's processor wraps shim construction/execution
  in try/finally and disposes the shim once its single work item
  completes.
- GrpcOrchestrationRunner disposes the shim immediately when it is not
  handed off to the extended-session cache (completed on first
  execution, or the request isn't part of an extended session), and
  registers a PostEvictionCallbackRegistration on cached entries so the
  shim is disposed exactly once whenever the cache entry is evicted for
  any reason (explicit Remove, sliding-expiration timeout, capacity
  eviction, or cache disposal) -- never while the orchestration may
  still resume.

No generated GUID bytes, algorithm, namespace, endianness, version
bits, target frameworks, or public API are changed.

Tests added:
- TaskOrchestrationShimTests: Dispose() forwards to and actually
  releases the wrapper's cached SHA1 (ObjectDisposedException on
  reuse), is a no-op when Execute() was never called, and is
  idempotent across repeated calls.
- GrpcOrchestrationRunnerTests: extended-session cache eviction via
  explicit removal and via sliding-expiration timeout both dispose the
  cached shim's SHA1, using reflection (no InternalsVisibleTo from this
  assembly) to reach the private wrapperContext/cachedHashAlgorithm
  fields via the public ExtendedSessionState.TaskOrchestration
  property.

Verified: Worker.Tests 145/145 passed, Worker.Grpc.Tests 139/139
passed (including new eviction tests).

Fixes #778

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 00:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Addresses three remaining Medium lifecycle issues from round-4 review of PR #780:

1. ExtendedSessionsCache.Dispose() now calls MemoryCache.Clear() before
   Dispose(). Confirmed against the pinned Microsoft.Extensions.Caching.Memory
   8.0.1 source that MemoryCache.Dispose() alone does NOT invoke post-eviction
   callbacks for entries still present in the cache -- only Clear(), Remove(),
   and the internal capacity/expiration removal paths do. Without this fix,
   any extended session still cached at worker shutdown would leak its
   cached shim (and SHA1 instance).

2. GrpcOrchestrationRunner's direct-execution path now wraps
   TaskOrchestrationExecutor construction/Execute()/cache Set() in
   try/finally, using a transferredShimToCache flag to gate disposal. If
   Execute() (or the subsequent Set() call) throws before the shim's
   ownership is transferred to the extended-sessions cache, the shim is now
   still deterministically disposed in finally instead of leaking.

3. The round-3 eviction-disposal tests asserted disposal immediately after
   triggering an eviction/removal, but MemoryCache dispatches post-eviction
   callbacks via Task.Factory.StartNew (i.e. asynchronously, on a background
   thread), so this was a latent race. Added a bounded polling helper
   (WaitUntilDisposedAsync) used by both existing tests, and added a new
   regression test (ExtendedSessionsCache_Dispose_DisposesCachedShimResources)
   proving that disposing the cache itself while an extended session is still
   pending disposes the cached shim's resources.

Verified:
- Worker.Tests: 145/145 passed
- Worker.Grpc.Tests: 140/140 passed (139 + 1 new)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 00:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (6)

src/Worker/Core/Shims/TaskOrchestrationContextWrapper.cs:42

  • The comment implies the cached SHA1 is disposed primarily when the shim replaces the wrapper in a subsequent Execute call, but the actual primary ownership boundary is TaskOrchestrationShim.Dispose() (and cache eviction) once the shim is no longer usable. Updating this comment will prevent confusion about when disposal happens.
    // This instance is disposed via Dispose() (see TaskOrchestrationShim, which disposes the previous
    // wrapper before replacing it with a new one on the next replay/decision task).

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:405

  • These tests parse a fixed, culture-invariant ISO-8601 timestamp but pass a null IFormatProvider, which makes the parse behavior depend on CurrentCulture. Using CultureInfo.InvariantCulture keeps the test independent of machine culture settings.
            DateTime.Parse("2023-05-06T07:08:09.1234567Z", null, DateTimeStyles.RoundtripKind));

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:428

  • These tests parse a fixed, culture-invariant ISO-8601 timestamp but pass a null IFormatProvider, which makes the parse behavior depend on CurrentCulture. Using CultureInfo.InvariantCulture keeps the test independent of machine culture settings.
            DateTime.Parse("2023-05-06T07:08:09.1234567Z", null, DateTimeStyles.RoundtripKind));

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:446

  • These tests parse a fixed, culture-invariant ISO-8601 timestamp but pass a null IFormatProvider, which makes the parse behavior depend on CurrentCulture. Using CultureInfo.InvariantCulture keeps the test independent of machine culture settings.
            DateTime.Parse("2024-01-01T00:00:00.0000000Z", null, DateTimeStyles.RoundtripKind));

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:468

  • This test parses a fixed, culture-invariant ISO-8601 timestamp but passes a null IFormatProvider, which makes the parse behavior depend on CurrentCulture. Using CultureInfo.InvariantCulture keeps the test independent of machine culture settings.
        DateTime timestamp = DateTime.Parse("2022-11-11T11:11:11.1111111Z", null, DateTimeStyles.RoundtripKind);

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:576

  • These tests parse a fixed, culture-invariant ISO-8601 timestamp but pass a null IFormatProvider, which makes the parse behavior depend on CurrentCulture. Using CultureInfo.InvariantCulture keeps the test independent of machine culture settings.
            DateTime.Parse("2023-05-06T07:08:09.1234567Z", null, DateTimeStyles.RoundtripKind));

Round-5 fix: MemoryCache.Clear() (added in round 4 to force eviction
callbacks to run before shutdown) throws ObjectDisposedException if
called on an already-disposed MemoryCache. Since Dispose() was not
guarded against repeat calls, invoking it twice (e.g. duplicate
shutdown hooks, or concurrent shutdown paths) would throw instead of
being a safe no-op.

- Add an `int disposed` field guarded via
  `Interlocked.Exchange(ref this.disposed, 1) != 0` at the top of
  Dispose(), making it both idempotent (repeat calls short-circuit)
  and thread-safe (concurrent calls only let one caller proceed).
- Clarify the disposal comment: Clear() only guarantees eviction
  callbacks are scheduled (via Task.Factory.StartNew) before Dispose()
  returns, not that they have completed.
- Add ExtendedSessionsCache_Dispose_CalledMultipleTimes_DoesNotThrow
  (sequential double-dispose) and
  ExtendedSessionsCache_Dispose_CalledConcurrently_DoesNotThrow
  (8-thread concurrent dispose) regression tests.

Verified: targeted GrpcOrchestrationRunnerTests 17/17, full
Worker.Grpc.Tests 142/142, full Worker.Tests 145/145.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 00:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

test/Worker/Grpc.Tests/GrpcOrchestrationRunnerTests.cs:499

  • This comment says eviction/removal "should synchronously invoke" the eviction callback, but the test’s earlier note (and the helper name) correctly explain that MemoryCache dispatches post-eviction callbacks asynchronously. The wording here is misleading and could cause future contributors to reintroduce racy assertions.
        // Now set the extended session flag to false for this instance, which removes/evicts the cache
        // entry and should synchronously invoke the eviction callback that disposes the cached shim.
        orchestratorRequest.Properties.Clear();

…review)

- Replace the round-5 Interlocked.Exchange disposed flag with a shared
  syncRoot lock guarding both Dispose() and GetOrInitializeCache(). This
  closes a race where Dispose() could observe an uninitialized cache field,
  mark itself disposed, and return -- while a concurrent
  GetOrInitializeCache() call then lazily constructed a brand-new
  MemoryCache that would never be disposed (since disposed was already
  permanently true), permanently leaking it.
- GetOrInitializeCache() now throws ObjectDisposedException immediately if
  called after Dispose(), instead of possibly returning an already-disposed
  MemoryCache or racing to create an unreachable one.
- Dispose() clears the extendedSessions field to null under the lock before
  tearing down the captured local reference outside the lock (avoids holding
  the lock during Clear()/Dispose(), which is safe since the eviction
  callback never calls back into ExtendedSessionsCache).
- Add ExtendedSessionsCache_DisposeRaceWithGetOrInitializeCache_NeverLeaksCache,
  racing Dispose() against GetOrInitializeCache() across many iterations and
  asserting the returned cache (if any) is provably disposed, and that a
  repeated Dispose() call remains a safe no-op.
- Fix a stale comment in
  ExternallyEndedExtendedSession_Evicted_DisposesCachedShimResources
  claiming the eviction callback fires synchronously; it is dispatched
  asynchronously and the test already awaits WaitUntilDisposedAsync.
- Use CultureInfo.InvariantCulture instead of null for the DateTime.Parse
  format provider in the NewGuid() golden-value regression tests, removing
  any dependency on the test runner's current culture.

Verified: targeted GrpcOrchestrationRunnerTests 18/18, full
Worker.Grpc.Tests 143/143, full Worker.Tests 145/145 (golden GUID values
unchanged after the culture fix).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 00:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

- Add two deterministic (non-racy) tests asserting GetOrInitializeCache()
  throws ObjectDisposedException after Dispose(), covering both the
  never-initialized and already-initialized cache scenarios.
- Rewrite the Dispose()/GetOrInitializeCache() race stress test to use a
  Barrier(2) so both competing tasks start racing at (approximately) the
  same instant, instead of relying on Task.Run queue-order bias which let
  the 'init wins' ordering dominate and never genuinely exercise the
  'dispose wins' path. Track and assert that both orderings actually
  occurred across the 50 iterations.
- Add a new deterministic ExtendedSessionsCache_Dispose_DisposesCachedEntryExactlyOnce
  test using a CountingDisposable spy wired up with the same
  PostEvictionCallbackRegistration pattern GrpcOrchestrationRunner uses for
  real cached shims, proving exact-once disposal of cached content tied to
  eviction (not just that the owning MemoryCache object becomes unusable).
  This is intentionally separate from the racing stress test: racing a
  Set() call concurrently against Dispose()'s internal Clear()/Dispose()
  split introduces a narrow window where an entry added in between is
  never evicted by that cache instance -- a test-harness artifact, not a
  production scenario.

No production code changes in this round; NewGuid determinism, cache
handoff exception cleanup, and eviction/shutdown lifecycle already passed
final review.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 01:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

src/Worker/Core/Shims/TaskOrchestrationContextWrapper.cs:464

  • HashAlgorithm.Hash is nullable; if it ever returns null, Array.Copy will throw a NullReferenceException. Since this code requires a hash after TransformFinalBlock, add an explicit null-check (or null-forgiving with justification) to make the contract explicit and avoid nullable warnings.
        byte[] hashByteArray = hashAlgorithm.Hash;

test/Worker/Core.Tests/Shims/TaskOrchestrationContextWrapperTests.cs:708

  • This nested test context type is private and not designed for inheritance. To follow the repo guideline of sealing private non-base classes, consider making it sealed.
    class TestOrchestrationContext : OrchestrationContext

The Dispose()/GetOrInitializeCache() race stress test asserted that both
the 'init wins' and 'dispose wins' orderings occurred at least once across
50 Barrier-coordinated iterations. A Barrier release does not guarantee
fair or varying thread-pool scheduling, so correct, race-free code could
legitimately let the same side win every iteration -- making this
assertion's pass/fail outcome probabilistic and CI-flaky rather than a
genuine correctness check.

Remove the win-count tracking and both-orderings-required assertions.
The test still asserts the invariants that must hold under every possible
interleaving: a cache handed back by GetOrInitializeCache() before a
concurrent Dispose() completes is provably disposed (TryGetValue throws
ObjectDisposedException), and repeated Dispose() calls remain a safe,
idempotent no-op.

Also give each Barrier.SignalAndWait() a bounded timeout instead of
waiting indefinitely, so a hung/stalled participant surfaces as a test
failure (TimeoutException) rather than hanging the test run.

The deterministic post-dispose contract tests
(GetOrInitializeCache_AfterDispose*) and the deterministic exact-once
disposal test (ExtendedSessionsCache_Dispose_DisposesCachedEntryExactlyOnce)
are unchanged and continue to directly catch the original leak.

No production code changes in this round.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9b8e1d4-b96d-41ec-adeb-a691aac6fee1
Copilot AI review requested due to automatic review settings July 25, 2026 01:15
Copilot AI review requested due to automatic review settings July 27, 2026 18:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c
Copilot AI review requested due to automatic review settings July 28, 2026 00:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

@halspang halspang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it adds a lot of additional complexity (especially around extended sessions). All that complexity seems to stem from being able to properly dispose the cached SHA1?

Given the code has to initialize the SHA1 anyway, are we really saving anything here? At the very least the complexity is now much worse.

Also, since this was already deterministic, I'm not sure why we need two paths with it cached and not? Does it actually impact an in-progress orchestration to suddenly cache its SHA?

@berndverst

Copy link
Copy Markdown
Member Author

Quantitative performance impact

I compared live head e396934 with its merge base using identical NewGuid() hash inputs. The measurements below are BenchmarkDotNet means from Release/.NET 8.0.29 on Windows x64 (AMD EPYC 7763). They isolate GUID generation and cache ownership; they are not fleet-wide orchestration averages.

Case Scenario GUID CPU time, base -> managed reuse Managed allocation GUID-stage throughput
Worst / no amortization 1 GUID in one managed lifetime 1.046 -> 1.072 us (+2.5% observed, likely close to benchmark noise) 816 -> 816 B (no gain) -2.4% observed
Average-style representative 10 GUIDs in one replay/lifetime 10.485 -> 7.086 us (-32.4%) 8,160 -> 7,008 B (-1,152 B / -14.1%) +48.0%
Best measured reuse 100 GUIDs in one replay or extended session 104.571 -> 68.429 us (-34.6%) 81,601 -> 68,929 B (-12,672 B / -15.5%) +52.8%

The allocation relationship is deterministic in this benchmark: reuse avoids 128 B for every SHA1 instance after the first, so the GUID-only saving is (N - 1) * 128 B.

Extended-session bookkeeping

The stronger ownership model is a fixed investment that enables safe reuse. An extracted success-path benchmark measured:

  • Base cache hit: 129 ns, 0 B
  • Head take/remove/reinsert ownership cycle: 2.015 us, 664 B
  • Incremental investment: 1.886 us and 664 B per incomplete resume

Combining those measurements, rather than treating the cache work as a standalone regression:

Managed lifetime Approximate total including one ownership cycle
1 GUID 1.175 -> 3.087 us; allocation +664 B
10 GUIDs 10.614 -> 9.101 us (-14.3%); 8,160 -> 7,672 B (-488 B / -6.0%)
100 GUIDs 104.700 -> 70.444 us (-32.7%); 81,601 -> 69,593 B (-12,008 B / -14.7%)

At the measured marginal saving, about six reused calls after the first (seven total GUIDs) amortize one ownership cycle for both CPU and allocation. Lifetimes below that point can be a net loss; high-GUID replay and extended-session workloads receive the material gain.

Memory and scope

  • One cached SHA1 remains live for the managed shim lifetime instead of being created/disposed per call. The measured managed object cost is about 128 B, plus unmeasured native provider state.
  • The ownership state adds two long fields (16 B of field payload) per extended-session state.
  • Fewer transient SHA1 objects reduce Gen 0 allocation and disposal frequency; GC-pause effects were not measured.
  • The public CreateOrchestration path intentionally remains resource-neutral and is not a beneficiary of reuse. It retained 816 B/call, and its CPU differences were within roughly +/-2.5% in these runs.

These are isolated, environment-specific measurements. Whole-worker latency/throughput gains will be smaller in proportion to how much time a workload spends generating deterministic GUIDs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c
Copilot AI review requested due to automatic review settings August 4, 2026 19:40
@berndverst

Copy link
Copy Markdown
Member Author

halspang The complexity is indeed ownership/disposal rather than determinism. The two paths are intentional because the public factory returns TaskOrchestration with no disposal contract; retaining SHA1 there would create a hidden leak, while the internal worker path has an explicit shim/cache lifetime. Both paths generate byte-identical GUIDs, so switching an in-progress orchestration to managed reuse does not change replay output. The measured break-even including one extended-session ownership cycle is about seven GUID calls: 10 calls measured ~14% less combined CPU and ~6% less allocation, while one call is a net loss. I kept reuse scoped to worker-owned lifetimes for that reason. The detailed benchmark is in the quantitative comment above.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Worker/Core/ExtendedSessionsCache.cs:68

  • ExtendedSessionsCache.Dispose() calls MemoryCache.Clear() unconditionally. Because GetOrInitializeCache() exposes the underlying MemoryCache publicly, a consumer can dispose it directly (your own code already handles this possibility elsewhere). If that happens, Clear() will throw ObjectDisposedException, causing ExtendedSessionsCache.Dispose() to throw during shutdown/cleanup even though Dispose() should be best-effort and non-throwing.
        cacheToDispose?.Clear();
        cacheToDispose?.Dispose();

Copilot AI review requested due to automatic review settings August 4, 2026 19:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/Worker/Core/ExtendedSessionsCache.cs:68

  • ExtendedSessionsCache.Dispose() calls MemoryCache.Clear() and Dispose() directly. Because GetOrInitializeCache() returns the underlying MemoryCache publicly, external callers can dispose it independently; in that case, Clear() (and potentially Dispose()) can throw ObjectDisposedException, causing ExtendedSessionsCache.Dispose() itself to throw. Dispose implementations should be resilient and avoid throwing when the inner cache is already disposed.
        cacheToDispose?.Clear();
        cacheToDispose?.Dispose();

Use conditional SHA1 ownership for per-call disposal and document that the extended-session wrapper retains ownership of its returned cache.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01dc5b7c-c742-45b7-ae8c-5cda9bcfbd2c
Copilot AI review requested due to automatic review settings August 4, 2026 20:25
@berndverst

Copy link
Copy Markdown
Member Author

I also evaluated the suppressed suggestion to swallow ObjectDisposedException when the publicly returned inner MemoryCache has already been disposed. I did not apply that catch: MemoryCache.Dispose() does not run eviction callbacks, and after disposal its entries cannot be cleared or enumerated, so swallowing the later Clear() failure would hide leaked cache-owned orchestration/SHA1 state. Commit 9daaa65 instead documents the existing ownership contract explicitly: the returned cache remains owned by ExtendedSessionsCache and callers must not dispose it. Preserving the exception is safer than reporting successful cleanup that could not occur.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Preserve canceled-event and deterministic GUID regressions; verify managed-lifetime allocation savings without changing GUID sequences.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings November 1, 2026 09:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The generation-based ownership protocol spans concurrent runner handoff, asynchronous cache eviction callbacks, shutdown, and direct cache access, so remaining interleavings and exactly-once disposal behavior require maintainer/runtime validation.

No open issues 🎉

Review effort: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance: reduce per-call SHA1 allocation for deterministic orchestration GUIDs

3 participants