Standardised test reporting

[ScottGuymer]

It would be great to output a report in some kind of standardised test reporting format like junit

Describe the solution you'd like
An option to output a test report that is compatible with other standard testing tools. For example to be able to take the results and put a summary on a github PR.

[AmirK-S]

A data point from driving this suite programmatically.

Measured 2026-09-05 against @modelcontextprotocol/conformance@0.2.0-alpha.11, source read at 7160fc8f.

Confirming the gap: server has no --json, --format, --reporter or --junit. --verbose prints JSON.stringify(checks, null, 2) per scenario, interleaved with === Running scenario: ... === lines, so it is not parseable as a stream. tier-check --output json is clean, but it requires --repo or --sdk and queries the GitHub API, so it does not cover a plain server run. action.yml declares inputs and no outputs, and relies on the exit code alone.

Two things complicated the file route:

1. Nothing is written without -o. README.md line 99 says results are saved to results/server-<scenario>-<timestamp>/, while src/runner/server.ts:44 gates the write on if (outputDir) and src/index.ts:545 declares -o, --output-dir <path> with no default. feat: make results directory output optional #107 suggests this is intended, so I read the README line as stale.

2. A scenario that throws creates its directory (src/runner/server.ts:44-47, before scenario.run at :101) but never writes checks.json (:105). The synthetic FAILURE built at src/index.ts:682-697 stays in memory. One 50 scenario run left 50 directories and 49 files, so counting directories looks complete while aggregating checks.json undercounts.

We now read checks.json against requirements/<revision>.yaml and treat a required scenario with no file as crashed.

If a full JUnit reporter is too much for a first cut, a --format json on server writing one file for the whole run, including scenarios that threw, would cover both points above, and JUnit could be generated from it. The exit code cannot help here: 1 covers a conformance regression, a bad flag and an unreachable server alike.

This analysis was done with the help of an AI assistant and verified by running the suite; this comment was drafted with AI assistance.

Reproduction harness: https://github.com/AmirK-S/mcp-migrate-delta