test(machines,tooling): check every engine with one conformance runner

Trax.Api.StateMachine/AGENTS.md

@@ -15,7 +15,9 @@ This file is the entry point. It routes; it does not restate the rules.
 | `src/server/`, `src/react.ts` | The server-authoritative controller, and `@traxsharp/state-machine/react` (`useServerMachine`, `useRememberedMachine`). |
 | `src/rules/generateTypes.ts` | `generateContextTypes` and `generateMachineFactory`, which turn an IR into a twin. |
 | `tools/` | The node entrypoints the `trax machine` commands shell out to (`generate-twin.mjs`, `generate-corpus.mjs`), and `bundle-cli.mjs`, which bundles them with the engine compiled in for Trax.Cli's package. |
+| `tools/conformance/` | The conformance runner (`run.mjs`), which sends every case of `machines/` and `conformance/` to one engine over the testee protocol in `conformance/PROTOCOL.md`, the TypeScript testee, and the IR schema validator. |
 | `machines/` | The shared fixtures both engines drive: each machine's IR, its generated twin and its differential corpus. |
+| `conformance/` | Engine-neutral tables any engine replays: the IR's JSON Schema (`ir.schema.json`, normative), the canonical-wire vectors, the refusal-code registry, the strict-input cases (written from the CLR by the C# `StrictInputGoldenTests`). A vector goes here, not in one engine's tests. |
 | `docs/design/` | The design notes the codegen was built from. Historical: the published docs are the rule. |
 
 ## The rules, and where they are written
@@ -25,7 +27,7 @@ This file is the entry point. It routes; it does not restate the rules.
   here, so the twin defers to the server on it. See [the codegen pipeline](https://traxsharp.net/docs/statemachine/codegen-pipeline)
   and [the IR format](https://traxsharp.net/docs/sdk-reference/statemachine-api/ir-format).
 - **The two engines agree case for case.** The differential corpus under `machines/<machine>/` is enumerated here
-  and replayed by the C# engine; only result codes are contract, messages may differ. The wire is canonical JSON
+  and replayed by the C# engine, and the conformance runner sends the whole kit to each engine, the C# one included; only result codes are contract, messages may differ. The wire is canonical JSON
   (RFC 8785) in both ([canonical wire](https://traxsharp.net/docs/statemachine/canonical-wire)).
 - **Every operation is total.** The engine and the clients answer a refusal or a transport failure as a value
   (`problem`, `TRANSPORT_ERROR`), never by throwing.
@@ -45,5 +47,5 @@ npm test          # vitest: conformance, totality, serialization, migration, cod
 npm run typecheck
 ```
 
-The Trax.Effect conformance tests and the Trax.Cli byte-parity tests read `machines/`, which is why CI lists this
-folder upstream of both.
+The Trax.Effect conformance tests read `machines/` and `conformance/`, and the Trax.Cli byte-parity tests read
+`machines/`, which is why CI lists this folder upstream of both.

Trax.Api.StateMachine/conformance/PROTOCOL.md

@@ -0,0 +1,91 @@
+# The conformance runner and its testee protocol
+
+`tools/conformance/run.mjs` turns every case in `machines/` and `conformance/` into requests, sends them to one
+engine, the **testee**, and compares what it answers with what the case expects. An engine in any language joins by
+implementing the testee side of this protocol over its public API, not over its internals, so a bug in how a binding
+marshals a value is caught too. Only codes and wire bytes are compared; messages are never sent.
+
+## Running it
+
+```bash
+node tools/conformance/run.mjs --testee node tools/conformance/testee-ts.mjs   # spawn a testee command
+node tools/conformance/run.mjs --stdio                                          # be spawned by one
+```
+
+With `--testee`, the runner starts the command and talks to it over the command's stdin and stdout. With `--stdio`,
+the roles on the pipes are reversed: the runner writes requests on its own stdout and reads answers on its stdin, so a
+test in another language (the C# `ConformanceRunnerTests`) can start the runner and answer from inside its own
+process. Either way the runner writes its report on stderr and exits 0 when every case passed, 1 when any failed.
+
+`--only <prefix>` runs the cases whose name starts with the prefix.
+
+## The protocol
+
+One JSON object per line, UTF-8, in both directions. The runner sends a request and waits for its answer before
+sending the next, so a testee may answer synchronously. Every request has an `id` and an `op`; the answer carries the
+same `id`. A testee answers an `op` it does not implement with `{ "id", "unsupported": true }`, which fails that case.
+
+**`load`** gives the testee a machine to serve under `machine`, as its IR. A testee that builds machines from an IR
+builds it; one that only has machines compiled in checks that the IR is the one its machine exports, and answers
+`error` when it is not.
+
+```json
+{ "id": 1, "op": "load", "machine": "turnstile", "ir": { ... } }
+{ "id": 1, "loaded": true }
+{ "id": 1, "error": "the turnstile this engine has exports a different IR" }
+```
+
+**`advance`**, **`outcome`** and **`timer`** apply a trigger to a snapshot through the entry point that owns it: a
+client's trigger through advance, an invoked train's outcome (`Fetching.done`) through the outcome path, a state's
+timer (`Review.after`) through the timer path. `snapshot` is an object (`machine`, `version`, `state`, `context`);
+`input` is absent or `null` when the trigger takes none. The answer is the canonical wire after the step, or the
+refusal code.
+
+```json
+{ "id": 2, "op": "advance", "machine": "turnstile", "snapshot": { ... }, "trigger": "Coin", "input": { "coin": "quarter" } }
+{ "id": 2, "outcome": "transitioned", "wire": "{\"machine\":\"turnstile\",...}" }
+{ "id": 2, "outcome": "rejected", "reason": "GUARD_FAILED" }
+```
+
+**`rehydrate`** reads stored text as a snapshot of `machine`. The answer is the snapshot's canonical wire, or the
+refusal code.
+
+```json
+{ "id": 3, "op": "rehydrate", "machine": "turnstile", "text": "{\"machine\":\"turnstile\",...}" }
+{ "id": 3, "result": "ok", "wire": "..." }
+{ "id": 3, "result": "error", "code": "MALFORMED" }
+```
+
+**`serialize`** writes a snapshot object of `machine` as canonical wire, without validating it.
+
+```json
+{ "id": 4, "op": "serialize", "machine": "checkout", "snapshot": { ... } }
+{ "id": 4, "wire": "..." }
+```
+
+**`canonical`** writes the snapshot `{ "machine": "m", "version": 1, "state": "S", "context": <context> }` as
+canonical wire, for the vectors in `canonical-wire.json`. The context is given as JSON text in `context`, or, for a
+string no JSON parser carries (a lone surrogate), as UTF-16 code units: `stringUnits` is the context
+`{ "s": <string> }`, `keyUnits` the context `{ <string>: 1 }`.
+
+```json
+{ "id": 5, "op": "canonical", "context": "{\"n\":1e21}" }
+{ "id": 5, "wire": "{\"machine\":\"m\",\"version\":1,\"state\":\"S\",\"context\":{\"n\":1e+21}}" }
+```
+
+**`end`** is the last request. The testee answers `{ "id" }` and exits.
+
+## What the runner sends
+
+| Source | Requests |
+| --- | --- |
+| `machines/<m>/<m>.ir.json` | `load` |
+| `machines/<m>/differential.json` | one `advance`, `outcome` or `timer` per case, by the trigger's kind in the IR |
+| `machines/<m>/fixtures/advance/*.json` | `advance` |
+| `machines/<m>/fixtures/rehydrate/*.json` | `rehydrate` |
+| `machines/<m>/wire-handoff.json` | `serialize` |
+| `conformance/strict-input/` | `load`, then one `advance` per case |
+| `conformance/canonical-wire.json` | `canonical` |
+
+`machines/turnstile/migration.json` is not sent: it migrates from a v1 to a v2 machine that has no IR, so each engine
+replays it in its own tests.

Trax.Api.StateMachine/conformance/canonical-wire.json

@@ -0,0 +1,37 @@
+{
+  "_comment": "The canonical wire (RFC 8785) of a snapshot's context: object keys sorted by UTF-16 code unit, numbers as ECMAScript Number::toString writes them, strings as JSON.stringify escapes them. Each case's context is parsed and serialized inside the envelope {\"machine\":\"m\",\"version\":1,\"state\":\"S\",\"context\":...}; 'wire' is the context part of the result. A context that a JSON parser cannot carry is given as UTF-16 code units instead: 'stringUnits' is the context {\"s\":<the string>}, 'keyUnits' the context {<the string>:1}. Every engine reads this file; add a vector here, not in one engine's tests.",
+  "envelope": {"machine":"m","version":1,"state":"S"},
+  "cases": [
+    {"name":"number 0.1","context":"{\"n\":0.1}","wire":"{\"n\":0.1}"},
+    {"name":"number 1e21","context":"{\"n\":1e21}","wire":"{\"n\":1e+21}"},
+    {"name":"number 1e20","context":"{\"n\":1e20}","wire":"{\"n\":100000000000000000000}"},
+    {"name":"number 1e-7","context":"{\"n\":1e-7}","wire":"{\"n\":1e-7}"},
+    {"name":"number 1e-6","context":"{\"n\":1e-6}","wire":"{\"n\":0.000001}"},
+    {"name":"number 5e-324","context":"{\"n\":5e-324}","wire":"{\"n\":5e-324}"},
+    {"name":"number 1.7976931348623157e308","context":"{\"n\":1.7976931348623157e308}","wire":"{\"n\":1.7976931348623157e+308}"},
+    {"name":"number 1.2345678901234568e20","context":"{\"n\":1.2345678901234568e20}","wire":"{\"n\":123456789012345680000}"},
+    {"name":"number 100","context":"{\"n\":100}","wire":"{\"n\":100}"},
+    {"name":"number 100.0","context":"{\"n\":100.0}","wire":"{\"n\":100}"},
+    {"name":"number -1.5","context":"{\"n\":-1.5}","wire":"{\"n\":-1.5}"},
+    {"name":"number 2.5","context":"{\"n\":2.5}","wire":"{\"n\":2.5}"},
+    {"name":"number 1","context":"{\"n\":1}","wire":"{\"n\":1}"},
+    {"name":"number -0","context":"{\"n\":-0}","wire":"{\"n\":0}"},
+    {"name":"number 3.4028235E+38","context":"{\"n\":3.4028235E+38}","wire":"{\"n\":3.4028235e+38}"},
+    {"name":"string: non-ASCII stays literal","context":"{\"s\":\"café\"}","wire":"{\"s\":\"café\"}"},
+    {"name":"string: an astral character stays literal","context":"{\"s\":\"😀\"}","wire":"{\"s\":\"😀\"}"},
+    {"name":"string: a slash is not escaped","context":"{\"s\":\"a/b\"}","wire":"{\"s\":\"a/b\"}"},
+    {"name":"string: a quote is escaped","context":"{\"s\":\"q\\\"x\"}","wire":"{\"s\":\"q\\\"x\"}"},
+    {"name":"string: a backslash is escaped","context":"{\"s\":\"b\\\\s\"}","wire":"{\"s\":\"b\\\\s\"}"},
+    {"name":"string: controls use short escapes, else lowercase hex","context":"{\"s\":\"\\u0001\\t\\n\\u001f\"}","wire":"{\"s\":\"\\u0001\\t\\n\\u001f\"}"},
+    {"name":"string: backspace, form feed and carriage return use short escapes","context":"{\"s\":\"\\b\\f\\r\"}","wire":"{\"s\":\"\\b\\f\\r\"}"},
+    {"name":"keys sort by UTF-16 code unit, integer-like ones too","context":"{\"b\":1,\"10\":\"x\",\"2\":\"y\"}","wire":"{\"10\":\"x\",\"2\":\"y\",\"b\":1}"},
+    {"name":"nested keys sort the same way","context":"{\"o\":{\"9\":1,\"10\":2,\"a\":3}}","wire":"{\"o\":{\"10\":2,\"9\":1,\"a\":3}}"},
+    {"name":"a key named __proto__ is kept","context":"{\"__proto__\":{\"x\":1},\"a\":2}","wire":"{\"__proto__\":{\"x\":1},\"a\":2}"},
+    {"name":"string: a lone high surrogate is escaped","stringUnits":[55296],"wire":"{\"s\":\"\\ud800\"}"},
+    {"name":"string: a lone low surrogate is escaped","stringUnits":[57343],"wire":"{\"s\":\"\\udfff\"}"},
+    {"name":"string: a high surrogate at the end is escaped","stringUnits":[97,55357],"wire":"{\"s\":\"a\\ud83d\"}"},
+    {"name":"string: a reversed pair is escaped","stringUnits":[56832,55357],"wire":"{\"s\":\"\\ude00\\ud83d\"}"},
+    {"name":"string: a lone high surrogate before a pair is escaped","stringUnits":[55357,55357,56832],"wire":"{\"s\":\"\\ud83d😀\"}"},
+    {"name":"key: a lone surrogate is escaped","keyUnits":[56319],"wire":"{\"\\udbff\":1}"}
+  ]
+}