Repository navigation
engine: an idle floor costs a stat per job; a busy floor folder no longer rebuilds the manager index - #2547
Closed
santoshkumarradha wants to merge 2 commits into
Closed
engine: an idle floor costs a stat per job; a busy floor folder no longer rebuilds the manager index#2547santoshkumarradha wants to merge 2 commits into
santoshkumarradha wants to merge 2 commits into
Conversation
…o longer rebuilds the manager index The factory store keeps each decoded job beside the identity of the file it came from and hands out deep copies, so settleJobs, worker pulls, List and Get decode only a document that changed. The person-Stop manager index no longer trusts the floor folder's clock, which a live engine moves every second or two; it is checked against the floor at most once a second, or at once after this process writes a job. CODEAF_DIAGNOSTIC_PROFILE_AFTER delays the profile window so a steady state can be measured. 50 looks at 60 unchanged jobs: 3,000 decodes -> 0. 100 manager looks in a busy folder: 6,000 decodes -> 0, <= 1 floor check/s. Owner's Mac, idle app at 1/2/3/5 min: 142/121/59/53% -> 28/45/21/14%. Assisted-by: CodeAF Co-Authored-By: CodeAF <267109073+agentfield-bot@users.noreply.github.com>
Assisted-by: CodeAF Co-Authored-By: CodeAF <267109073+agentfield-bot@users.noreply.github.com>
Member
Author
|
Closing as superseded: #2478 (f508fec) landed on app/v3 with the same fixes in its own form — per-file stop checks in cmd/codeaf/factory_person_stop.go (replacing factory_manager_index.go), a cached document read in internal/factory/store, and a delayed profile window (CODEAF_DIAGNOSTIC_PROFILE_DELAY). Merging this would duplicate them. Blackmac now runs the app/v3 head instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An idle
codeaf appon the owner's Mac (371 factory jobs, ~100 chats) still sat at ~50–140% CPU after #2544/#2545. Two causes, both fixed here:sources.json,serve.json,runners.json,worker.json, which a live engine rewrites every 1–2 s (observed on the Mac). So every job chat's 100 ms person-Stop look rebuilt the index from a full decode of the floor: 44% of the steady-state profile. The index now answers until this process writes a job document (Store.Written) or 1 s passes; a recheck is a folder listing plus astatper job (Store.Managers), decoding only changed documents. The job's own document is stillstated every look, so a Stop is seen on the next look; another process's change to which conversation manages a job is seen within 1 s.settleJobs(each factory request) andlease.Desk.Pull(each worker poll) listed and decoded the whole floor. The store now keeps each decoded document beside its file identity (fstatdev/inode, size, mtime; every write renames a fresh file in) per floor folder, and hands out deep copies (internal/factory/store/doccache.go).Listno longer re-reads each document unless a tier lookup could have written it.Also:
CODEAF_DIAGNOSTIC_PROFILE_AFTER(0..3600 s) delays the diagnostic profile window so a steady state can be profiled; a process that ends inside the delay leaves no file.Knowledge
refineLearning/compile(15% of the first minute) is one-time startup work and is absent from the steady-state profile; left alone.Proof
Counter tests, deterministic clock, fail on
origin/app/v3, pass here:TestAnUnchangedFloorIsListedWithoutReadingAJob: 50 looks at 60 unchanged jobs, 3,000 decodes → 0.TestABusyFloorFolderReadsNoJobs: 100 manager looks in a folder rewritten every look, 6,000 decodes → 0.TestABusyFloorFolderCostsTheManagerWatchNothing: ≤ 1 floor check per second of looks; another process's move seen after the recheck; a Stop seen on the next look.TestEveryChangeToAJobIsSeenOnTheNextRead(store write, in-place edit dated back, same-size same-time rename, removal) andTestEveryDocumentTypeCanBeCopiedDeeply(the copy shares nothing; no document type hides unexported fields).make pr-ready BASE=origin/app/v3green.Owner's Mac (blackmac), idle app, CPU over 10 s windows:
32747af4dAt 7–9 min this PR holds ~20% (12–30%).
Not done (next lane)
Target was <10%. The remaining ~20% is mostly the desktop's own reads (
ListAgentswalking every conversation's meta/index/owner files ≈ 23% of the remainder,OpenCollection8%,KnowledgeSnapshot6%,ListInbox6%), the knowledge pass every 2 s (12%: transcript inventory stats + sqlite counts), and chat publication (6%). These need a stat-validated world cache ininternal/sessionor a slower desktop poll, which is a separate change. Profiles:~/work/work-wave/prof/idle2/{before,v1}.