Skip to content

feat: event stream (hgs watch) and the desktop consumer (event data plane M3+M4) - #369

Open
ufna wants to merge 7 commits into
mainfrom
work-ba0ed9fd-stream
Open

ufna wants to merge 7 commits into
mainfrom
work-ba0ed9fd-stream

Conversation

@ufna

@ufna ufna commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Event data plane M4 (epic zerus-604d, task zerus-604d.4). Stacked on #368 (M3), which is stacked on #367 (M1). The M4 change is the last commit.

What changes

  • EventStream (tray/src/EventStream.{h,cpp}) runs one hgs [@host] watch --json per computer whose CLI prints Event stream ABI: watch-v1. The child, NDJSON framing, JSON parsing and the per-computer catalog and Activity store live on the stream's own thread. The GUI thread receives the same BoxState a listing poll delivers and inspect-shaped Activity increments (summary + new events + cursor, or replace_events after a gap), so onLocalReady/onPeerReady/acceptInspection are unchanged.
  • TrayAgent owns the EventStreams hub (local plus every known peer). Periodic listing polls skip a computer while its stream is live. Write-triggered refreshes still poll once.
  • SessionsWindow follows the selected session's Activity through its computer's live stream. Only the Processes section is still polled, at its existing interval (2.5 s on the tab, 30 s otherwise). A run whose subscription ends (subscription_ended/session_changed) goes back to polling.
  • Cadence: Activity within 50 ms. Listings are applied at most every 2 s, because applying one still rebuilds on the GUI thread (about 150 ms on the Mac) until M2. An unchanged listing is re-applied every 10 s, because automatic approvals need a peer listed within 30 s and 3 minutes mean stale.
  • Fallback: a missing marker means Unsupported, which polls and rechecks after 10 minutes. A failed or 30 s silent stream restarts after 5 s, 15 s, 60 s and then every 5 minutes, polling meanwhile. Frames prepared just before a failure are dropped. ZERUS_EVENT_STREAM=0 or dataPlane/eventStream=false opts out.
  • Documented in docs/reference.md (Event stream) and the design spec.

Verification (arch)

  • test_eventstream (8 cases, 3 consecutive passing runs): listing order and host, a peer keeps its alias, Activity increments with cursor, deltas with removals, no GUI-thread parse, Unsupported without the marker, an ended stream fails and never goes Live afterwards, an unchanged listing is re-applied.
  • Full Qt suite with ZERUS_GUI_THREAD_CHECKS=fatal: 84/84 suites, 3008 test functions, 0 failed, 23 skipped (platform-only). test_source_*: 26 OK; privacy guard: 0 findings.
  • End to end against the real hgs watch from feat(cli): hgs watch --json event stream (event data plane M3) #368 on arch, following a busy session (raw data on the review page):
    • stream live in 327 ms; first Activity about 170 ms after follow;
    • event lag (receive minus event at) p50 0.20 s, p90 0.28 s, max 0.38 s;
    • today's 2.5 s inspect polling of the same session: p50 2.34 s, p90 2.70 s.

Peers need the new CLI on both sides. Until a peer has it, the desktop keeps polling that peer.

🤖 Generated with Claude Code

ufna and others added 6 commits October 27, 2026 16:41
Owner direction 2026-10-27: data parsing and similar processing run outside
the GUI thread. First milestone of the event data plane (zerus-604d.1).

- ProcessRunner passes every result through one serial preparation thread,
  hgs-prepare, keeping completion order. run(request, owner, prepare, apply)
  and post(owner, prepare, apply) shape values there; the owner's thread only
  applies them.
- Every HgsClient reply, the connector panels, the speech client, the update
  feed and startup fleet discovery are parsed in prepare steps. The task
  parse pool and the 8 MiB worktree snapshot re-parse per lookup are gone.
- JsonParse is the tray's only JSON parser and reports a parse on the GUI
  thread; every Qt suite runs with ZERUS_GUI_THREAD_CHECKS=fatal.
  check-source.py rejects QJsonDocument::fromJson elsewhere and limits
  JsonParse::local() to a listed, shrink-only set of small local values
  (owner: "согласен, оставляем исключения до M2").
- SettingsJson parses QSettings JSON values once per change, warms the large
  keys at startup and records what writers stored, so per-poll read-mark
  reads no longer re-parse settings.
- IdentityHash remembers per-row SHA-256 identities, 3.9 % of the Mac GUI
  thread in a 2026-10-27 sample.

Design: docs/superpowers/specs/2026-10-27-event-data-plane-design.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Event data plane M3 (zerus-604d.3). `hgs [@host] watch --json` is a
long-lived NDJSON stream: hello, a bounded catalog snapshot of the live
listing (rows exactly as ls --json --local --no-archives, keyed run:<run_id>),
deltas with removal reasons, catalog.meta, and Activity subscriptions over
stdin whose cursor is the events.sqlite3 seq that inspect --after takes.
Heartbeats, gap/snapshot resync, a 4 MiB coalescing writer queue and 1 MiB
frames bound it; EOF on stdin ends it. `hgs -h` prints the marker
`Event stream ABI: watch-v1`; @host runs it over one SSH connection with
keepalives, guarded so an older peer never reads it as a launch.

Change detection uses no new dependency: a 250 ms pulse stats the listing's
inputs and polls PRAGMA data_version on a held journal connection, with a 5 s
safety listing for changes no file shows. Volatile observation fields are
masked, so deltas carry real changes only. The serde frame types live in
crates/watch-protocol (no Loro or Qt) for the desktop, the Tauri client and
later the connector. The inspection's journal query is shared, not forked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Event data plane M4 (zerus-604d.4). EventStream runs one
`hgs [@host] watch --json` per computer whose CLI prints the watch-v1
marker. The child, NDJSON framing, JSON parsing and the computer's catalog
and Activity store live on the stream's own thread; the GUI thread receives
the same BoxState a listing poll delivers and inspect-shaped Activity
increments, so the existing apply paths stay unchanged.

While a computer's stream is live, the tray skips its periodic listing polls
and the Sessions window follows the selected session's Activity through it,
polling only the Processes section at its own interval. Listings are applied
at most every 2 s until M2 makes applying one cheap; Activity within 50 ms.
A failed or silent stream restarts after 5 s, 15 s, 60 s and then every
5 minutes, and polling continues meanwhile. A computer without the marker
keeps polling. ZERUS_EVENT_STREAM=0 or dataPlane/eventStream=false opts out.

On arch, following a busy session: event lag p50 0.20 s, p90 0.28 s against
2.34 s and 2.70 s for today's 2.5 s inspect polling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A loaded macOS runner produced one heartbeat in the fixed 180 ms the test
slept (PR #369, CLI unit job). Both writer tests now wait for the expected
frames with a 5 s deadline instead of sleeping a fixed time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zerus-604d.6 and zerus-604d.7. The watcher re-derives only a session whose
binding changed or whose journal rows arrived, through the listing's own row
steps under the same lock, and lists all rows again when anything else
changes and every 5 s. A 5 s probe covers what no file shows: tmux sessions
that start, end, are renamed, attached or detached, exited agent processes,
ended background work, sessions without a hook event yet, and DeepSeek and
OpenCode hosts.

A listing captured each live pane several times: once for the effort footer
and once per eligible terminal chooser (usage limit, resume, Kimi cache hint,
approval), 62 tmux capture-pane calls on arch for 35 panes. Inside the
read-only status observation those readers now share one capture per pane;
answers, resizes and key sends still capture afresh.

On arch: a listing dropped from about 0.25 s to 0.21 s CPU, and the watcher
from 12.9 % to 6.1 % of a core over 60 s. With a 15 s whole listing it
measured 4.3 %, but a chooser without a hook would show up to 15 s late;
that tradeoff waits for the owner. ls and inspect output are unchanged
(105/105 stable rows and 831/831 stable inspect fields compared).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ufna ufna changed the title feat(tray): follow each computer through its event stream (event data plane M4) feat: event stream (hgs watch) and the desktop consumer (event data plane M3+M4) Oct 27, 2026
@ufna

ufna commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Update: #367 (M1) is merged. This PR now carries M3 (#368's commit) plus three more commits:

  • test(watch): wait for writer output with a deadline: feat(cli): hgs watch --json event stream (event data plane M3) #368/feat: event stream (hgs watch) and the desktop consumer (event data plane M3+M4) #369 CI failed on macOS because the heartbeat writer test slept a fixed 180 ms and a loaded runner produced one heartbeat. Both writer tests now wait with a 5 s deadline.
  • perf(cli): refresh only changed rows in hgs watch (zerus-604d.6/.7): per-row refresh under the listing's lock, a 5 s probe for what no file shows, and a whole listing every 5 s. One shared capture per pane inside the status observation (62 → 35 capture-pane per listing on arch). Watcher CPU on arch: 12.9 % → 6.1 % of a core. ls/inspect output is unchanged. Rust: 1244 passed. test_watch: 11/11. The subagent also ran 514 focused Python tests (0 failures).
  • The M4 desktop consumer, unchanged.

#368's macOS desktop failure was TestTasksView::taskDocumentAndDiscussionUseActivityComposition timing out on a fake-CLI round trip. #367 passed with identical tray code; watching whether it recurs here.

🤖 Generated with Claude Code

The Windows fake CLI runs Python scripts; a /bin/sh fake never answered there,
so the stream failed instead of reporting Unsupported (#369 Windows job).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant