[zyansheep]

What changed

Hub (packages/hub/src/uploads.ts, server.ts)

• Uploads go straight to <data>/uploads/<id> as they arrive, never whole into memory. A JSON record <id>.json sits beside each file: { name, mime, size, at }. The folder stays 0700 and the files 0600.
• New resumable routes, following the same pattern as history imports:
  • POST /api/uploads/start { name, mime, size }
  • PUT /api/uploads/:id?offset=N takes at most UPLOAD_CHUNK (2 MiB). A wrong offset gets 409 with { received }.
  • GET /api/uploads/:id and DELETE /api/uploads/:id
  • received is the file's size on disk. A piece cut off midway still leaves good bytes, and the next piece starts after them.
• The old POST /api/uploads (whole body in one request) still works, for older cached pages. It now streams to disk and is refused with 413 as soon as it passes the cap, even with no Content-Length. Nothing is kept when the upload is too large, empty, cut off, or shorter than its Content-Length.
• Uploads survive a restart. At startup the hub keeps every upload used in the last hour, whole or half-made. "Used" means made, written to, or handed to a send. Everything else in the folder goes: older uploads, a stream a crash cut off before its record, a half-written record.
• Files are still deleted once sent. A failed send keeps them for a retry, and the hourly sweep is unchanged.
• Where queued sends live today: they are not in the hub (delivery.ts is notifications). They are in the app's IndexedDB outbox, with their files. Before this change the hub emptied the uploads folder at every start, so after a restart:
  • a queued send got 410 and uploaded every file again;
  • a send that a remote adapter was still reading lost its file (dither-accounts keeps running while the hub restarts).

Web (upload.ts, ui/ComposerFiles.tsx, state.ts, composer edits in Timeline.tsx)

• Each file starts uploading as soon as it is attached, in pieces sent with XHR so progress can be shown. Each file chip has a progress bar and a cancel (×). Cancelling, or leaving the conversation, deletes what the hub has of that file.

• After no answer or a 5xx, the app retries with backoff (1 s, 2 s, 4 s, 8 s, 15 s, 15 s), each time asking how much the hub has. After that it gives up as a TypeError, and the outbox waits for the hub as it does for any send. The upload ids are saved with the send in IndexedDB as soon as the hub gives them, so a reload carries on where the upload stopped.

• Sending while files are still uploading queues the send (the existing outbox flow) and frees the composer. The pending copy shows "Uploading…" with each file's percentage.

  • A failed upload shows the reason and a Retry button, and Send waits until it is retried or removed.
  • If it failed only because the hub was unreachable, it shows "Waiting for the hub" instead. It retries when the hub is back and doesn't hold up a send, so offline sends work as before.

• No file is dropped silently any more. A refused file stays in the row with its reason and is never uploaded:

  • Over Discord’s 10 MB limit (this is 12 MB)
  • Over the hub’s 25 MB limit
  • Empty file
  • Cannot be read…
  • Discord takes 10 files at a time

  Before this, files over the limit got a toast that didn't name the file, and files over the count were cut off by .slice(0, limit.files) without a word.

• The attach button's tooltip names the network's limit.

Protocol: adds UPLOAD_CHUNK and UploadState. Both are additions only; accounts-sync copies them over.

Why resumable pieces rather than one streamed request

The case to plan for is a phone on a flaky connection. With pieces, a dropped connection loses at most the 2 MiB in flight, and the upload carries on from the hub's received. With one stream it would start the whole 25 MB again. Pieces also let a hub restart in the middle of an upload carry on from where it stopped. The cost is one extra round trip per 2 MiB.

Why MAX_UPLOAD stays at 25 MiB

The hub no longer holds files in memory, but the adapters still read each file whole to send it. Discord's adapter base64-encodes the file into the command it hands its client. Signal, WhatsApp, Telegram and Google Messages also take MAX_UPLOAD as their limit in ATTACHMENT_LIMITS. Raising it safely needs each network's real limit checked and the adapters changed to stream (dither-accounts). The reasoning is in docs/uploads.md.

How it was verified

• Hub: packages/hub/test/uploads.test.ts (14 tests) covers:

  • streaming over the cap, which stops reading early;
  • a dropped connection leaving no partial file;
  • a body shorter than its Content-Length;
  • pieces, offsets, and a piece cut off then resumed;
  • a restart keeping whole and half-made uploads while records, strays and old uploads are swept;
  • the keep window counted from last use;
  • name cleaning.

• server.test.ts "sending files" has 5 new tests over HTTP:

  • a body with no Content-Length cut off at the cap;
  • a connection aborted midway;
  • the piece routes with 409, 410 and 404;
  • refused sizes and an oversized piece;
  • a hub restarted between upload and send.

  A mutation check (not deleting partial files) fails 5 of these tests.

• Web: packages/web/test/upload.test.ts (9 tests) runs against a fake hub:

  • progress;
  • resuming after drops, with backoff;
  • resuming after a reload by id;
  • beginning again after a 404, and a refusal shown at once;
  • giving up as offline;
  • an unreadable file;
  • cancel and retry on a job;
  • every refusal reason.

• scripts/verify-uploads.ts: all 21 checks pass in headless Chromium against its own demo hub on a fresh data dir. The script owns the hub, so restarting it ends nothing else.

  • The over-limit, empty and unreadable (chmod 000) files are each shown with their reason, and only the good file reaches the hub.
  • With uploads throttled to 200 KB/s, progress is seen (0→16%). Cancelling midway leaves nothing on the hub.
  • A message is sent while a 3 MB file is uploading. The pending copy says "Uploading…".
  • The hub is SIGTERMed with 819,200 bytes on disk and restarted. The upload carries on at offset=819200, with no file uploaded again (no start or POST after the restart). The message goes out once, with both files, and the demo replies.
  • The hub's uploads folder is empty afterwards.

• verify-sends (17/17) and verify-reply (9/9) still pass, the first because the outbox send path changed.

• bun run typecheck passes. In bun run test, every package passes except 4 tests:

  • two hub timing tests: inbox-plan "in a few milliseconds" and learn "fits twenty thousand examples quickly". These are CPU timing thresholds; the machine's load average was about 27 on 16 cores, and neither test touches this code.
  • two WhatsApp adapter tests, which failed once under that load and pass when run alone.

• The composer was checked by eye in screenshots at desktop and at 390 px.

What I could not verify

• A real phone, or the Android WebView, on a flaky network. Drops were simulated by a hub restart and by the fake transport.
• Sending on a real network. Only the demo adapter sent anything. I didn't exercise a remote dither-accounts adapter reading a file across a hub restart.
• Carrying on after a page reload in a real browser. Only the unit test (fake transport) covers it.
• The "Waiting for the hub" chip and its retry when the hub comes back, in a browser.
• A dropped folder probably shows as "Empty file" or "Cannot be read" depending on the browser. Not tested.

🤖 Generated with Claude Code