Skip to content

Read quiet writer directories less often, and slow down in the background - #101

Merged
shovon merged 4 commits into
mainfrom
worktree-issue-52
Oct 27, 2026
Merged

shovon merged 4 commits into
mainfrom
worktree-issue-52

Conversation

@shovon

@shovon shovon commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Closes #52.

What changed

Writer directories pile up, one per session that writes, and almost all of them have ended. The 1 s poll read every one of them every second, so its cost grew with how long a project had been in use. Now:

  • The ledger tiers directories. A directory is read on every scan while it is active. One where nothing has changed for quietAfterMs is read every quietEveryMs, and never skipped for good. A new directory is read at once. "Changed" means the segment list, or the size of the segment being read, differs from the last read; on first sight the newest segment's modification time stands in, and a directory with no segment yet (sync clients create the folder before its files) counts as just changed. Neither orders or gates anything, so a wrong guess costs reads, never correctness. The cadence is timed on a monotonic clock, so a wall clock set back cannot stop the reads.

  • Cadence, pace() and refresh(). openOrCreate(path, cadence) replaces pollMs. pace changes the cadence from the next scan, and refresh scans now, reading every directory. Both openOrCreate and pace throw a TypeError on a cadence that is not one, the old pollMs number included, and openOrCreate does so before writing anything. Scans queue and never overlap; refreshes asked for before one starts share it.

  • The app paces by focus. setFocused in app/src/main/ledgers.ts, wired to browser-window-focus and browser-window-blur in index.ts. createLedgers takes its two paces, defaulting to these:

    Scan every Quiet after Quiet read every
    Focused 1 s 5 min 30 s
    Background 5 s 5 min 2 min

    Gaining focus also refreshes the open project's ledger, so what comes on screen is never behind by the background pace. The other projects kept open this session are only paced, not read in full on every focus.

  • Spec. "Noticing change" in packages/ledger/multi-writer.md now has the cadence, what counts as a change and why size is enough, how old a quiet directory's status can be, the app's pace, and why there is still no fs.watch.

  • app/CLAUDE.md. A shell module that gets a unit test stays loose, with the test beside it (ledgers.ts, ledgers.test.ts), so it no longer reads as breaking the folder-per-tested-module rule.

Measurement

One scan on a local disk (APFS), median of 200, every directory but one aged an hour:

Directories Every directory Quiet ones skipped
10 0.74 ms 0.10 ms
100 6.6 ms 0.13 ms
300 21 ms 0.22 ms
1,000 76 ms 0.53 ms

Not measured inside a sync client's folder.

Testing

  • Nine new ledger tests: a quiet directory is skipped, read by refresh, then active again; a quiet one is still read every quietEveryMs; a directory written just before open is active from the start; a directory first seen empty is active, so its first segment is read at once; a wall clock set back 10 minutes does not stop quiet reads; a bad cadence is refused before anything is written; refreshes asked for before one starts share it; pace changes the cadence; nothing scans after close.
  • Four new tests in app/src/main/ledgers.test.ts, each writing from a second session: gaining focus reads the project on screen at once; it reads only that one, not other open projects; the background pace holds until focus returns; a ledger opened in the background starts at the background pace.
  • Each fix's test was checked against the code broken on purpose: with the refresh on focus off, the empty-first-sight rule reverted, or the cadence timed on Date.now(), the matching test fails.
  • npm test (all workspaces), tsc --noEmit and eslint pass.

Review

A review of this PR found eight issues. Six are fixed in the last commit: the empty first sight, the wall clock, refreshing every project on focus, unvalidated cadences, the app/CLAUDE.md rule conflict and the missing test. Two needed no code change and are now explained in the spec. Change is told by size only, but the key decides how often a directory is read, never what a read folds, so a same-size fill-in (such as an online-only placeholder) is folded on the next read either way. A quiet directory's status can be up to quietEveryMs old, because keeping it current would mean listing every directory every scan, which gives back half the saving.

In the running app

These runs predate the review fixes. None of the fixes changes the cases timed below, but they were not rerun afterwards.

The dev app ran with a scratch profile against fresh scratch projects. A driver wrote blocks from another session (openWriter, and once the real pbnotes append) and timed each from the write being on disk to the block arriving in the window. "Quiet" directories were seeded with segments dated an hour back before the app first saw them. Two back-to-back runs, with nobody at the machine:

Case Expected Run 1 Run 2
Focused, active directory ≤ ~1 s 502, 679, 438 ms 31, 274, 502 ms
Background, active directory ≤ ~5 s 4.0, 4.1, 4.0 s 4.0, 4.7, 4.2 s
Written in the background, then focus at once 22 ms after focus 24 ms after focus
Focused, quiet directory ≤ 30 s 30.0 s 30.0 s
Background, quiet directory ≤ 2 min 120.1 s 120.1 s
pbnotes append, focused ≤ ~1 s 380 ms 373 ms

The quiet cases sit on their limits because each write came right after a full read, which is the worst case. In both runs every focus and blur the app heard was one the driver made, and a 250 ms timer in main was never more than 100 ms late. A log of every directory read matched the rules: active directories every 1 s focused and 5 s in the background, quiet ones on focus, 30 s later, then every 120 s in the background.

One unexplained miss. In an earlier run, a focused write to a quiet directory had not arrived after 45 s. It was folded later, and the next quiet write, in the background, arrived after 9.6 s rather than up to 2 min. It has not come back since: three full runs and two focused-only checks gave 27.6 to 30.0 s, and the same sequence in plain Node gave 30.0 s. That run logged neither directory reads nor focus timestamps, and a later run was spoiled by a stray blur from desktop activity, so a focus change the driver did not make is the likely cause, but it is not confirmed.

🤖 Generated with Claude Code

shovon and others added 4 commits October 27, 2026 11:27
…ound (#52)

A scan now reads a writer directory on every pass only while it is active.
One where nothing has changed for 5 minutes is read every 30 s, and a new
one is read at once. At 1,000 directories a quiet tick drops from 76 ms to
0.5 ms on a local disk.

The ledger takes a Cadence and gains pace() and refresh(); scans queue and
never overlap. The app paces every ledger by window focus: 1 s while
focused, 5 s (quiet every 2 min) in the background, and a full read when a
window gains focus. The numbers are under "Noticing change" in
multi-writer.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
createLedgers takes its two paces, defaulting to the app's, so a test can
use one that never scans by itself. The tests write from a second session
and check that gaining focus reads at once, that the background pace holds
until then, and that a ledger opened in the background starts at it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A directory first seen empty, as sync clients create folders before
  their files, now counts as just changed instead of quiet, so a new
  session's first segment is read on the next scan.
- The cadence is timed on performance.now(), so a wall clock set back
  cannot stop quiet directories being read.
- Gaining focus refreshes only the project on screen; refreshes asked for
  before one starts share it.
- openOrCreate and pace refuse a cadence that is not one, before anything
  is written.
- The spec says why size is enough to tell a change, and how old the
  status of a quiet directory can be. app/CLAUDE.md says a tested shell
  module stays loose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shovon
shovon merged commit a7322ba into main Oct 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decide how often the ledger looks for changes

1 participant