Skip to content

fix(sync): handle replication stream errors on the sync websocket - #77

Open
gmaclennan wants to merge 3 commits into
mainfrom
fix/sync-ws-unhandled-rejection
Open

gmaclennan wants to merge 3 commits into
mainfrom
fix/sync-ws-unhandled-rejection

Conversation

@gmaclennan

@gmaclennan gmaclennan commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

When a sync client sends a bad frame instead of a noise handshake (or the handshake fails for any other reason), the whole cloud process exits with code 1. Every other websocket connection and any in-flight REST download is dropped with it. An internal repro hit this repeatedly.

The /sync/:projectPublicId handler in src/routes.js discards the promise returned by wsCoreReplicator in src/ws-core-replicator.js, which is the node:stream/promises pipeline promise. When the replication stream errors that promise rejects with nobody listening, and node 20 treats an unhandled rejection as fatal.

This change catches the rejection in the route, logs it on the request logger at warn and terminates the offending socket if it is still open. It also adds a process-level unhandledRejection handler in src/server.js that logs at error and keeps the process running, so a future miss shows up in logs instead of taking the server down. The new test keeps sending garbage frames on the sync websocket until the server closes the socket, then checks that no unhandled rejection fired and that the server still answers GET /info. It has to keep sending because the handler awaits getProject before attaching the pipeline, and a frame that arrives before then is dropped; that early-frame gap is pre-existing and worth a separate issue.

One question for reviewers: is warn the right level for a per-connection replication failure? Garbage from one client is not a server fault, so info may be enough, but warn makes repeated failures from a misbehaving client easier to spot.

The pipeline promise returned by wsCoreReplicator was discarded, so a
replication stream error (for example a failed noise handshake from a
client sending a bad frame) became an unhandled rejection and node
exited the whole process. Log it on the request logger and terminate the
offending socket if it is still open. Also log unhandled rejections at
process level in server.js as defence in depth so a future miss is
visible without taking the server down.
Comment thread src/routes.js Resolved
@gmaclennan
gmaclennan marked this pull request as ready for review November 5, 2026 12:24
@gmaclennan
gmaclennan requested review from RangerMauve and a balanced review from Copilot November 5, 2026 14:46
@gmaclennan
gmaclennan removed the request for review from Copilot November 5, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud process exits on a bad replication frame, dropping every connection

1 participant