Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions docs/SURFACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -989,8 +989,10 @@ preload, or with an already replaced Promise or Array iterator intrinsic, also
admits new roots in serial mode. Generated arrays such as `Array.from` use serial
mode. Branches may not mutate shared values, await timers or other promises,
or call `f.done`; completion belongs at the end of
the root body. The check is conservative: an unrecognized source form uses
serial mode even when its author knows it would be safe. In parallel mode,
the root body. TypeScript annotations on the body parameter, awaited values,
and branch return types are allowed when the type-stripped body loaded by Node
matches the certified source exactly. The check is conservative: an unrecognized
source form uses serial mode even when its author knows it would be safe. In parallel mode,
each deterministic child uses its own journal connection because the
daemon handles requests on one connection in order and `run.start` returns
only after its command exits. An authored body admits at most eight
Expand Down Expand Up @@ -1034,10 +1036,12 @@ failure and capacity rules.
resume in serial compatibility mode, preserving their `kind-N` step ids
and shared connection. Completed child effects remain bound to their original
admission keys, including follow-ups after a `Promise.all` join.
New parallel roots carry a distinct root kind. A 2.0.50 or 2.0.51 CLI refuses one on
rollback, leaving it resumable by this or a newer CLI rather than replaying
parallel follow-ups with old `run-N` identities. New serial roots retain the
original root kind and serial IDs. An active pinned parallel root resumed in a
New 2.0.53 parallel roots carry the `parallel.v3` root kind. A 2.0.52 or older
CLI refuses one before dispatch on rollback, leaving it resumable by a 2.0.53
or newer CLI. The 2.0.53 CLI can still resume earlier `parallel.v2` roots.
Cloud must keep a 2.0.53-capable artifact available for every active
`parallel.v3` root; rollback cannot resume those roots on 2.0.52. New serial
roots retain the original root kind and serial IDs. An active pinned parallel root resumed in a
process with a Node preload or altered Promise or Array iterator intrinsic is
refused before dispatch and remains resumable in a clean process. A completed
root reports its journaled result without dispatching new work.
Expand Down
105 changes: 105 additions & 0 deletions evidence/hosted-parallel-2-0-53/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Hosted parallel `f.run` repair (2026-10-18)

## Cause

The 2.0.52 standalone CLI is compiled with Bun. Bun reformats the authored
body's `Function.prototype.toString()` result: it changes whitespace and quote
style and can combine adjacent `const` declarations. Parallel admission
compared that string byte-for-byte with the original `.flow.ts` body, so a
source-faithful hosted flow was pinned to `executionMode: serial`. The npm
Node CLI kept the original function text and admitted the same flow as parallel.

After source admission was repaired, the standalone result verifier exposed a
second blocker. It required the numeric suffixes of all child IDs to be a
single `1..N` sequence. Parallel IDs use an ordinal **per causal predecessor**,
so three valid dependent runs can all end in `-1`. The verifier rejected their
durable success as `authored runtime result has no matching durable completion`.

## Repair and safety checks

- Bun admits parallel only from a closed, parsed, hash-pinned source module.
Its in-memory body never executes: the embedded Node runner loads the pinned
source. The Node CLI retains exact function/source identity comparison.
- Before any parallel child starts, the Node runner independently verifies the
pinned source hash, closed module grammar, and Node-loaded body grammar.
The executor still checks Promise and Array intrinsics before execution.
- For a parallel result, the standalone verifier compares every claimed child
against the root's durable `authored-steps` index, requires no unclaimed or
unfinished child, and then reads every child journal and completion marker.
Serial roots retain their global ordinal check.
- Set `FLOWS_PARALLEL_DEBUG=1` on the CLI to log why a new root selected serial
mode (`source_graph`, `extensions`, `shared_budget`, `runtime_intrinsics`, or
`source_certificate`). The reason is deliberately not stored in root
metadata: changing a 2.0.52 root's admission spec hash could disrupt an
in-flight resume after upgrade.

## Source reproduction

The exact hosted-reported shape is [hosted-repro.flow.ts](hosted-repro.flow.ts).
These commands ran from `/tmp/hv52`, which contained that source and the
published 2.0.52 Surface package:

```sh
bun -e "import handle from './par.flow.ts'; import {getFlowDefinition} from '@relayflows/surface/runtime'; console.log(Function.prototype.toString.call(getFlowDefinition(handle).body));"
node --experimental-strip-types --input-type=module -e "import handle from './par.flow.ts'; import {getFlowDefinition} from '@relayflows/surface/runtime'; console.log(Function.prototype.toString.call(getFlowDefinition(handle).body));"
```

Captured complete outputs: [Bun](logs/bun-function-source.txt),
[Node](logs/node-function-source.txt), and their [literal diff](logs/function-source-diff.txt).
The old byte-for-byte body comparison fails on that diff; the parser unit test
also asserts that the Node path still refuses a mismatched in-memory body.

## Final-build checks

Commands ran from `packages/sdk` at the final source tree. Each link is the
complete captured stdout/stderr for the adjacent literal command (all exited 0):

| Command | Captured output |
| --- | --- |
| `npm run typecheck` | [typecheck.txt](logs/typecheck.txt) |
| `npm run build` | [build.txt](logs/build.txt) |
| `npm run typecheck:tests` | [typecheck-tests.txt](logs/typecheck-tests.txt) |
| `./node_modules/.bin/vitest run tests/authored-root.test.ts tests/authored-parallel-body.test.ts tests/authored-parallel-eligibility.test.ts tests/authored-node-result.test.ts --pool=forks --maxWorkers=2 --minWorkers=1` | [focused-tests.txt](logs/focused-tests.txt) |
| `PATH=/tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin:$PATH RELAYFLOWD_BIN=/tmp/hv52/node_modules/@relayflows/runtime-darwin-arm64/bin/relayflowd /tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin/node ./node_modules/vitest/vitest.mjs run tests/authored-node-runtime.test.ts -t 'runs a certified Promise.all group' --pool=forks --maxWorkers=1 --minWorkers=1` | [standalone-parallel.txt](logs/standalone-parallel.txt) |
| `PATH=/tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin:$PATH RELAYFLOWD_BIN=/tmp/hv52/node_modules/@relayflows/runtime-darwin-arm64/bin/relayflowd /tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin/node ./node_modules/vitest/vitest.mjs run tests/authored-node-runtime.test.ts -t 'awaits agent plus three run steps' --pool=forks --maxWorkers=1 --minWorkers=1` | [standalone-serial.txt](logs/standalone-serial.txt) |

The parallel live test builds the standalone CLI, checks parallel root metadata,
proves all three route starts precede the first route end, and checks the
dependents and join. The serial case includes a completed-root resume with no
repeated effects. The verifier unit test accepts repeated causal `-1` suffixes
and refuses mismatched or extra root-index records.

## Manual standalone journal

The final standalone binary was built with this command from the repo root;
its complete output is [standalone-build.txt](logs/standalone-build.txt):

```sh
node scripts/build-standalone-cli.mjs bun-darwin-arm64 /tmp/ms-flows-parallel-hosted/flows
```

The CLI commands below ran from a fixture with the published 2.0.52 Surface
package, the copied [source](hosted-repro.flow.ts), published 2.0.52 daemon,
and Node 22.16.0 embedded runner. The inspector commands ran from the repo root:

```sh
FLOWS_AUTHORED_NODE=/tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin/node RELAYFLOWD_BIN=/tmp/hv52/node_modules/@relayflows/runtime-darwin-arm64/bin/relayflowd /tmp/ms-flows-parallel-hosted/flows run par.flow.ts --data-dir /tmp/ms-flows-parallel-hosted/data6 --json --no-observer-link --input '{}'
python3 evidence/hosted-parallel-2-0-53/inspect-journal.py /tmp/ms-flows-parallel-hosted/data6 01M57SMB30E9AD784XVP5T2X9S
FLOWS_AUTHORED_NODE=/tmp/ms-flows-parallel-node22/node_modules/node-bin-darwin-arm64/bin/node RELAYFLOWD_BIN=/tmp/hv52/node_modules/@relayflows/runtime-darwin-arm64/bin/relayflowd /tmp/ms-flows-parallel-hosted/flows resume 01M57SMB30E9AD784XVP5T2X9S --data-dir /tmp/ms-flows-parallel-hosted/data6 --json --no-observer-link
python3 evidence/hosted-parallel-2-0-53/inspect-journal.py /tmp/ms-flows-parallel-hosted/data6 01M57SMB30E9AD784XVP5T2X9S
```

Captured outputs: [run](logs/manual-run.txt),
[journal](logs/manual-journal.txt), [resume](logs/manual-resume.txt), and
[journal after resume](logs/manual-journal-after-resume.txt). The two journal
outputs are identical:

```text
root executionMode: parallel
run-1: started=1792336867426 ended=1792336869451
run-2: started=1792336867424 ended=1792336869450
run-3: started=1792336867422 ended=1792336869450
route start spread ms: 4
all routes overlapped: True
run journals: 9 attempt starts: 9
```
9 changes: 9 additions & 0 deletions evidence/hosted-parallel-2-0-53/hosted-repro.flow.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { flow } from '@relayflows/surface';
export default flow('hv52-parallel', async f => {
const a = async () => { await f.run('echo A-start $(date +%s%3N); sleep 2; echo A-end $(date +%s%3N)'); await f.run('echo a-dep $(date +%s%3N)'); };
const b = async () => { await f.run('echo B-start $(date +%s%3N); sleep 2; echo B-end $(date +%s%3N)'); await f.run('echo b-dep $(date +%s%3N)'); };
const c = async () => { await f.run('echo C-start $(date +%s%3N); sleep 2; echo C-end $(date +%s%3N)'); await f.run('echo c-dep $(date +%s%3N)'); };
await Promise.all([a(), b(), c()]);
await f.run('echo join $(date +%s%3N)');
f.done('success');
});
45 changes: 45 additions & 0 deletions evidence/hosted-parallel-2-0-53/inspect-journal.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
"""Read one local standalone reproduction's SQLite journals."""

import json
import sqlite3
import sys
from pathlib import Path


data = Path(sys.argv[1])
root_id = sys.argv[2]
runs = sorted((data / "runs").glob("*.sqlite3"))
root = sqlite3.connect(data / "runs" / f"{root_id}.sqlite3")
root_spawn = next(json.loads(payload) for kind, payload in root.execute(
"select entry_type, payload from entries where entry_type='run.spawned'"
) if kind == "run.spawned")
metadata = json.loads(root_spawn["spec"]["steps"][0]["instruction"])
print("root executionMode:", metadata["executionMode"])

attempts = 0
routes = []
for path in runs:
entries = sqlite3.connect(path).execute(
"select entry_type, at_ms, payload from entries order by seq"
).fetchall()
attempts += sum(kind == "step.attempt.started" for kind, _, _ in entries)
spawned = next((json.loads(payload) for kind, _, payload in entries
if kind == "run.spawned"), None)
if spawned is None:
continue
step = spawned["spec"]["steps"][0]
command = step.get("command", "")
if not any(f"{name}-start" in command for name in "ABC"):
continue
started = next(at for kind, at, _ in entries if kind == "step.attempt.started")
ended = next(at for kind, at, _ in entries if kind == "step.completed")
routes.append((step["id"], started, ended))

for step_id, started, ended in sorted(routes):
print(f"{step_id}: started={started} ended={ended}")
print("route start spread ms:", max(start for _, start, _ in routes)
- min(start for _, start, _ in routes))
print("all routes overlapped:", max(start for _, start, _ in routes)
< min(end for _, _, end in routes))
print("run journals:", len(runs), "attempt starts:", attempts)
4 changes: 4 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/build.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

> @relayflows/sdk@2.0.52 build
> tsc && node scripts/make-cli-executable.mjs

15 changes: 15 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/bun-function-source.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
async (f) => {
const a = async () => {
await f.run("echo A-start $(date +%s%3N); sleep 2; echo A-end $(date +%s%3N)");
await f.run("echo a-dep $(date +%s%3N)");
}, b = async () => {
await f.run("echo B-start $(date +%s%3N); sleep 2; echo B-end $(date +%s%3N)");
await f.run("echo b-dep $(date +%s%3N)");
}, c = async () => {
await f.run("echo C-start $(date +%s%3N); sleep 2; echo C-end $(date +%s%3N)");
await f.run("echo c-dep $(date +%s%3N)");
};
await Promise.all([a(), b(), c()]);
await f.run("echo join $(date +%s%3N)");
f.done("success");
}
15 changes: 15 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/focused-tests.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@

RUN v2.1.9 /Volumes/Paris Drive/AgentWorkforce/swarm/ms-flows-parallel/hosted-parallel/packages/sdk

✓ tests/authored-node-result.test.ts (46 tests) 12ms
✓ tests/authored-parallel-eligibility.test.ts (11 tests) 20ms
(node:19316) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it.
(Use `node --trace-warnings ...` to show where the warning was created)
✓ tests/authored-root.test.ts (42 tests) 753ms
✓ tests/authored-parallel-body.test.ts (5 tests) 9ms

Test Files 4 passed (4)
Tests 104 passed (104)
Start at 17:19:59
Duration 4.26s (transform 1.04s, setup 40ms, collect 4.30s, tests 794ms, environment 0ms, prepare 260ms)

24 changes: 24 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/function-source-diff.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
--- evidence/hosted-parallel-2-0-53/logs/node-function-source.txt 2026-10-18 17:23:27
+++ evidence/hosted-parallel-2-0-53/logs/bun-function-source.txt 2026-10-18 17:23:24
@@ -1,8 +1,15 @@
-async f => {
- const a = async () => { await f.run('echo A-start $(date +%s%3N); sleep 2; echo A-end $(date +%s%3N)'); await f.run('echo a-dep $(date +%s%3N)'); };
- const b = async () => { await f.run('echo B-start $(date +%s%3N); sleep 2; echo B-end $(date +%s%3N)'); await f.run('echo b-dep $(date +%s%3N)'); };
- const c = async () => { await f.run('echo C-start $(date +%s%3N); sleep 2; echo C-end $(date +%s%3N)'); await f.run('echo c-dep $(date +%s%3N)'); };
+async (f) => {
+ const a = async () => {
+ await f.run("echo A-start $(date +%s%3N); sleep 2; echo A-end $(date +%s%3N)");
+ await f.run("echo a-dep $(date +%s%3N)");
+ }, b = async () => {
+ await f.run("echo B-start $(date +%s%3N); sleep 2; echo B-end $(date +%s%3N)");
+ await f.run("echo b-dep $(date +%s%3N)");
+ }, c = async () => {
+ await f.run("echo C-start $(date +%s%3N); sleep 2; echo C-end $(date +%s%3N)");
+ await f.run("echo c-dep $(date +%s%3N)");
+ };
await Promise.all([a(), b(), c()]);
- await f.run('echo join $(date +%s%3N)');
- f.done('success');
+ await f.run("echo join $(date +%s%3N)");
+ f.done("success");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
root executionMode: parallel
run-1: started=1792336867426 ended=1792336869451
run-2: started=1792336867424 ended=1792336869450
run-3: started=1792336867422 ended=1792336869450
route start spread ms: 4
all routes overlapped: True
run journals: 9 attempt starts: 9
7 changes: 7 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/manual-journal.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
root executionMode: parallel
run-1: started=1792336867426 ended=1792336869451
run-2: started=1792336867424 ended=1792336869450
run-3: started=1792336867422 ended=1792336869450
route start spread ms: 4
all routes overlapped: True
run journals: 9 attempt starts: 9
1 change: 1 addition & 0 deletions evidence/hosted-parallel-2-0-53/logs/manual-resume.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"ok":true,"command":"resume","resolutions":[],"diagnostics":[],"rootRunId":"01M57SMB30E9AD784XVP5T2X9S","runId":"01M57SMB30E9AD784XVP5T2X9S","socketPath":"/Users/khaliqgant/.agent-tmp/relayflowd-9fb8f3c4039e.sock","completedSteps":8,"status":"completed","completionReason":"success","subscriptions":[]}
1 change: 1 addition & 0 deletions evidence/hosted-parallel-2-0-53/logs/manual-run.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"ok":true,"command":"run","resolutions":[],"diagnostics":[],"path":"par.flow.ts","rootRunId":"01M57SMB30E9AD784XVP5T2X9S","runId":"01M57SMB30E9AD784XVP5T2X9S","socketPath":"/Users/khaliqgant/.agent-tmp/relayflowd-9fb8f3c4039e.sock","completedSteps":8,"status":"completed","completionReason":"success","subscriptions":[]}
8 changes: 8 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/node-function-source.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
async f => {
const a = async () => { await f.run('echo A-start $(date +%s%3N); sleep 2; echo A-end $(date +%s%3N)'); await f.run('echo a-dep $(date +%s%3N)'); };
const b = async () => { await f.run('echo B-start $(date +%s%3N); sleep 2; echo B-end $(date +%s%3N)'); await f.run('echo b-dep $(date +%s%3N)'); };
const c = async () => { await f.run('echo C-start $(date +%s%3N); sleep 2; echo C-end $(date +%s%3N)'); await f.run('echo c-dep $(date +%s%3N)'); };
await Promise.all([a(), b(), c()]);
await f.run('echo join $(date +%s%3N)');
f.done('success');
}
19 changes: 19 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/review1-focused-tests.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@

RUN v2.1.9 /Volumes/Paris Drive/AgentWorkforce/swarm/ms-flows-parallel/hosted-parallel/packages/sdk

✓ tests/authored-node-result.test.ts (46 tests) 10ms
(node:52258) ExperimentalWarning: stripTypeScriptTypes is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
✓ tests/authored-parallel-eligibility.test.ts (12 tests) 62ms
(node:52248) ExperimentalWarning: stripTypeScriptTypes is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:52248) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it.
✓ tests/authored-root.test.ts (42 tests) 493ms
(node:52265) ExperimentalWarning: SQLite is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
✓ tests/authored-status-detail.test.ts (23 tests) 212ms

Test Files 4 passed (4)
Tests 123 passed (123)
Start at 19:54:21
Duration 2.33s (transform 452ms, setup 23ms, collect 2.07s, tests 778ms, environment 0ms, prepare 148ms)
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

> @relayflows/sdk@2.0.52 typecheck:tests
> tsc -p tsconfig.tests.json
3 changes: 3 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/review1-typecheck.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

> @relayflows/sdk@2.0.52 typecheck
> tsc --noEmit && tsc -p tsconfig.type-tests.json
6 changes: 6 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/standalone-build.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Bundled 796 modules in 778ms

authored-node.mjs 12.19 MB (entry point)

[376ms] bundle 768 modules
[3.933s] compile /tmp/ms-flows-parallel-hosted/flows
11 changes: 11 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/standalone-parallel.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

RUN v2.1.9 /Volumes/Paris Drive/AgentWorkforce/swarm/ms-flows-parallel/hosted-parallel/packages/sdk

✓ tests/authored-node-runtime.test.ts (18 tests | 17 skipped) 10524ms
✓ Bun 1.4.x standalone → native Node authored lifecycle > runs a certified Promise.all group in parallel through the embedded Node runner 5735ms

Test Files 1 passed (1)
Tests 1 passed | 17 skipped (18)
Start at 17:20:08
Duration 12.32s (transform 759ms, setup 13ms, collect 1.65s, tests 10.52s, environment 0ms, prepare 37ms)

11 changes: 11 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/standalone-serial.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

RUN v2.1.9 /Volumes/Paris Drive/AgentWorkforce/swarm/ms-flows-parallel/hosted-parallel/packages/sdk

✓ tests/authored-node-runtime.test.ts (18 tests | 17 skipped) 11006ms
✓ Bun 1.4.x standalone → native Node authored lifecycle > awaits agent plus three run steps and resumes without repeating effects 5383ms

Test Files 1 passed (1)
Tests 1 passed | 17 skipped (18)
Start at 17:20:24
Duration 13.53s (transform 905ms, setup 12ms, collect 2.34s, tests 11.01s, environment 0ms, prepare 46ms)

4 changes: 4 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/typecheck-tests.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

> @relayflows/sdk@2.0.52 typecheck:tests
> tsc -p tsconfig.tests.json

4 changes: 4 additions & 0 deletions evidence/hosted-parallel-2-0-53/logs/typecheck.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

> @relayflows/sdk@2.0.52 typecheck
> tsc --noEmit && tsc -p tsconfig.type-tests.json

Loading
Loading