Skip to content

WriteAheadLog (6.x): make a committed write durable across a graceful restart with a batched FlushInterval #284

Closed

Description

@msallin

DotNext.Net.Cluster 6.3.0, .NET 10. RaftCluster (TCP) with a hand-assigned WriteAheadLog as AuditTrail over a SimpleStateMachine, one cluster per pod. Membership and restart leadership are already solved with a persistent configuration storage (per #207); this question is purely about WAL durability.

I expose a synchronous storage API, so a write must be durable before it returns, or at minimum survive a graceful (SIGTERM) restart. Completion is detected with a waiter completed inside SimpleStateMachine.ApplyAsync: I await ReplicateAsync(...) then await the waiter, so a write returns once the entry is applied.

  • With FlushInterval = TimeSpan.Zero (flush on every commit in the background) this is reliable: a node restarted from the same WAL recovers everything via InitializeAsync. But it fsyncs on every commit.
  • With a positive FlushInterval (to batch flushes for throughput), a graceful shutdown loses the un-snapshotted tail. A single committed entry written then immediately followed by DisposeAsync is not recovered after a restart, even when I call FlushAsync() before disposing the cluster and the log. A longer run (e.g. 200 writes) is recovered, because the periodic flusher and snapshots cover it.

Questions:

  1. Is there a supported way to make a committed write durable on return — awaiting the flush, not just the apply — so the caller knows it is on stable storage?
  2. With a hand-assigned WriteAheadLog, what is the correct flush-on-graceful-shutdown sequence so a positive FlushInterval does not lose the most recent committed-but-unflushed entries? Is FlushAsync() the right call, and where must it run relative to RaftCluster.StopAsync() and disposal?
  3. Or is FlushInterval = TimeSpan.Zero the only mode that guarantees a committed entry survives a restart, with any positive interval being best-effort (up to one interval lost on a stop)?

Goal: one cluster per pod that can batch flushes for throughput yet never lose a committed write across a graceful restart. Happy to provide a minimal repro.

Activity

  1. changed the title [-]Single-member cluster: resuming leadership on restart with WriteAheadLog + in-memory configuration[/-] [+]WriteAheadLog (6.x): durable commit / flush-on-shutdown so a batched FlushInterval survives a graceful restart (membership part resolved via #207)[/+] on Jun 15, 2026
  2. changed the title [-]WriteAheadLog (6.x): durable commit / flush-on-shutdown so a batched FlushInterval survives a graceful restart (membership part resolved via #207)[/-] [+]WriteAheadLog (6.x): make a committed write durable across a graceful restart with a batched FlushInterval[/+] on Jun 15, 2026
  3. locked and limited conversation to collaborators on Jun 22, 2026
  4. converted this issue into a discussion #285 on Jun 22, 2026
  5. moved this from Opened to Closed in Clusteron Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Lib:ClusterDotNext.Net.Cluster library

    Type

    No type

    Projects

    • Status
      Closed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions