Repository navigation
fix(sync): terminate sync websockets that stop answering pings - #78
Open
gmaclennan wants to merge 2 commits into
Open
gmaclennan wants to merge 2 commits into
gmaclennan wants to merge 2 commits into
Conversation
The server never detected a dead websocket: no ws ping/pong and no secret-stream timeout, so a phone that dropped off the network stayed connected until TCP retransmits gave up. Ping each /sync client on an interval and terminate it when no pong arrives within the timeout. Both are configurable via SYNC_HEARTBEAT_INTERVAL_MS and SYNC_HEARTBEAT_TIMEOUT_MS (defaults 15 s / 30 s).
This was referenced Nov 5, 2026
gmaclennan
marked this pull request as ready for review
November 5, 2026 12:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a phone drops off the network mid-sync (wifi to mobile switch, walking out of range), the cloud keeps that connection counted as a live peer for several minutes. The phone's next connection then races the stale one in core's per-device peer state, and the late close of the old connection can wipe the new one, which is what partners see as "Waiting for devices" with the archive showing as connected.
The cause is that nothing on the server ever probes the link. The
/syncroute insrc/routes.jshands the socket to the replicator with no ws ping/pong, and hypercore's secret-stream keepalive has no timeout, so a socket whose peer vanished without a FIN stays open until TCP retransmits give up.This adds a per-connection heartbeat: the server pings each sync client every
SYNC_HEARTBEAT_INTERVAL_MS(default 15 s) and terminates the socket if neither a pong nor any inbound data has arrived withinSYNC_HEARTBEAT_TIMEOUT_MS(default 30 s). Inbound replication data counts as proof of life so that a phone on a slow link, still sending data while the server's ping waits behind a large outgoing sync, is not cut off. One interval timer per connection, cleared on close or when it fires the terminate; a socket that closed before the heartbeat started is noticed on the next tick. Termination lands between timeout and timeout plus one interval, so up to 45 s with the defaults. The timeout must be greater than the interval, and the server refuses to start otherwise. The same values can be passed assyncHeartbeatIntervalMs/syncHeartbeatTimeoutMsapp options, which the test uses with short values.For reviewers: are 15 s / 30 s the right defaults for the field, given phones on poor links may take a while to answer a ping?
Related changes: digidem/comapeo-core#1349 removes the race itself, on both the cloud and the phone, so a late close no longer wipes the newer connection's state. This heartbeat is still worth having: it shrinks the window while phones are on older core, and dead sockets stop holding peer slots.
Note that digidem/comapeo-mobile#2213 leaves phone websockets open after the user leaves the Exchange screen, so this heartbeat then depends on the phone answering ws pings, or on secret-stream keepalive counting as inbound data; worth checking on a real device before the defaults are settled.