Skip to content

fix(sync): terminate sync websockets that stop answering pings - #78

Open
gmaclennan wants to merge 2 commits into
mainfrom
fix/sync-ws-heartbeat
Open

gmaclennan wants to merge 2 commits into
mainfrom
fix/sync-ws-heartbeat

Conversation

@gmaclennan

@gmaclennan gmaclennan commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

When a phone drops off the network mid-sync (wifi to mobile switch, walking out of range), the cloud keeps that connection counted as a live peer for several minutes. The phone's next connection then races the stale one in core's per-device peer state, and the late close of the old connection can wipe the new one, which is what partners see as "Waiting for devices" with the archive showing as connected.

The cause is that nothing on the server ever probes the link. The /sync route in src/routes.js hands the socket to the replicator with no ws ping/pong, and hypercore's secret-stream keepalive has no timeout, so a socket whose peer vanished without a FIN stays open until TCP retransmits give up.

This adds a per-connection heartbeat: the server pings each sync client every SYNC_HEARTBEAT_INTERVAL_MS (default 15 s) and terminates the socket if neither a pong nor any inbound data has arrived within SYNC_HEARTBEAT_TIMEOUT_MS (default 30 s). Inbound replication data counts as proof of life so that a phone on a slow link, still sending data while the server's ping waits behind a large outgoing sync, is not cut off. One interval timer per connection, cleared on close or when it fires the terminate; a socket that closed before the heartbeat started is noticed on the next tick. Termination lands between timeout and timeout plus one interval, so up to 45 s with the defaults. The timeout must be greater than the interval, and the server refuses to start otherwise. The same values can be passed as syncHeartbeatIntervalMs / syncHeartbeatTimeoutMs app options, which the test uses with short values.

For reviewers: are 15 s / 30 s the right defaults for the field, given phones on poor links may take a while to answer a ping?

Related changes: digidem/comapeo-core#1349 removes the race itself, on both the cloud and the phone, so a late close no longer wipes the newer connection's state. This heartbeat is still worth having: it shrinks the window while phones are on older core, and dead sockets stop holding peer slots.

Note that digidem/comapeo-mobile#2213 leaves phone websockets open after the user leaves the Exchange screen, so this heartbeat then depends on the phone answering ws pings, or on secret-stream keepalive counting as inbound data; worth checking on a real device before the defaults are settled.

The server never detected a dead websocket: no ws ping/pong and no
secret-stream timeout, so a phone that dropped off the network stayed
connected until TCP retransmits gave up. Ping each /sync client on an
interval and terminate it when no pong arrives within the timeout.
Both are configurable via SYNC_HEARTBEAT_INTERVAL_MS and
SYNC_HEARTBEAT_TIMEOUT_MS (defaults 15 s / 30 s).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant