Skip to content

engine: an idle floor costs a stat per job; a busy floor folder no longer rebuilds the manager index - #2547

Closed
santoshkumarradha wants to merge 2 commits into
app/v3from
fix/idle-cpu-2
Closed

santoshkumarradha wants to merge 2 commits into
app/v3from
fix/idle-cpu-2

Conversation

@santoshkumarradha

Copy link
Copy Markdown
Member

What

An idle codeaf app on the owner's Mac (371 factory jobs, ~100 chats) still sat at ~50–140% CPU after #2544/#2545. Two causes, both fixed here:

  1. The manager index never settled. engine: an idle job chat and an unchanged vault cost nothing per look #2545's per-floor index trusted the floor folder's mtime, but the same folder holds sources.json, serve.json, runners.json, worker.json, which a live engine rewrites every 1–2 s (observed on the Mac). So every job chat's 100 ms person-Stop look rebuilt the index from a full decode of the floor: 44% of the steady-state profile. The index now answers until this process writes a job document (Store.Written) or 1 s passes; a recheck is a folder listing plus a stat per job (Store.Managers), decoding only changed documents. The job's own document is still stated every look, so a Stop is seen on the next look; another process's change to which conversation manages a job is seen within 1 s.
  2. Every floor read decoded every job. settleJobs (each factory request) and lease.Desk.Pull (each worker poll) listed and decoded the whole floor. The store now keeps each decoded document beside its file identity (fstat dev/inode, size, mtime; every write renames a fresh file in) per floor folder, and hands out deep copies (internal/factory/store/doccache.go). List no longer re-reads each document unless a tier lookup could have written it.

Also: CODEAF_DIAGNOSTIC_PROFILE_AFTER (0..3600 s) delays the diagnostic profile window so a steady state can be profiled; a process that ends inside the delay leaves no file.

Knowledge refineLearning/compile (15% of the first minute) is one-time startup work and is absent from the steady-state profile; left alone.

Proof

Counter tests, deterministic clock, fail on origin/app/v3, pass here:

  • TestAnUnchangedFloorIsListedWithoutReadingAJob: 50 looks at 60 unchanged jobs, 3,000 decodes → 0.
  • TestABusyFloorFolderReadsNoJobs: 100 manager looks in a folder rewritten every look, 6,000 decodes → 0.
  • TestABusyFloorFolderCostsTheManagerWatchNothing: ≤ 1 floor check per second of looks; another process's move seen after the recheck; a Stop seen on the next look.
  • TestEveryChangeToAJobIsSeenOnTheNextRead (store write, in-place edit dated back, same-size same-time rename, removal) and TestEveryDocumentTypeCanBeCopiedDeeply (the copy shares nothing; no document type hides unexported fields).

make pr-ready BASE=origin/app/v3 green.

Owner's Mac (blackmac), idle app, CPU over 10 s windows:

1 min 2 min 3 min 5 min
32747af4d 142% 121% 59% 53%
this PR 28% 45% 21% 14%

At 7–9 min this PR holds ~20% (12–30%).

Not done (next lane)

Target was <10%. The remaining ~20% is mostly the desktop's own reads (ListAgents walking every conversation's meta/index/owner files ≈ 23% of the remainder, OpenCollection 8%, KnowledgeSnapshot 6%, ListInbox 6%), the knowledge pass every 2 s (12%: transcript inventory stats + sqlite counts), and chat publication (6%). These need a stat-validated world cache in internal/session or a slower desktop poll, which is a separate change. Profiles: ~/work/work-wave/prof/idle2/{before,v1}.

santoshkumarradha and others added 2 commits November 9, 2026 16:22
…o longer rebuilds the manager index

The factory store keeps each decoded job beside the identity of the file it
came from and hands out deep copies, so settleJobs, worker pulls, List and
Get decode only a document that changed. The person-Stop manager index no
longer trusts the floor folder's clock, which a live engine moves every
second or two; it is checked against the floor at most once a second, or at
once after this process writes a job. CODEAF_DIAGNOSTIC_PROFILE_AFTER delays
the profile window so a steady state can be measured.

50 looks at 60 unchanged jobs: 3,000 decodes -> 0.
100 manager looks in a busy folder: 6,000 decodes -> 0, <= 1 floor check/s.
Owner's Mac, idle app at 1/2/3/5 min: 142/121/59/53% -> 28/45/21/14%.

Assisted-by: CodeAF
Co-Authored-By: CodeAF <267109073+agentfield-bot@users.noreply.github.com>
Assisted-by: CodeAF
Co-Authored-By: CodeAF <267109073+agentfield-bot@users.noreply.github.com>
@santoshkumarradha

Copy link
Copy Markdown
Member Author

Closing as superseded: #2478 (f508fec) landed on app/v3 with the same fixes in its own form — per-file stop checks in cmd/codeaf/factory_person_stop.go (replacing factory_manager_index.go), a cached document read in internal/factory/store, and a delayed profile window (CODEAF_DIAGNOSTIC_PROFILE_DELAY). Merging this would duplicate them. Blackmac now runs the app/v3 head instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant