[zpzjzj]

Summary

Add skill-up compare <result.json> <result.json>... to review two or more completed evaluations without running an Agent or Judge. Export HTML, Markdown, text, or JSON. HTML extends the existing report with a cross-file overview and interactive selection of any two results; the original single-run and with/without-Skill report output remains unchanged.

skill-up compare alpha/result.json beta/result.json gamma/result.json --format html --output comparison.html

Related issues

Closes #175. Related to #349.

Changes

• Align cases by stable case ID and preserve the union of case sets. Compare each file's canonical primary case result: prefer with_skill, otherwise a non-baseline configuration, falling back to without_skill only when nothing else is available. Document this rule and expose configuration identity.
• Show all-run metrics and a case status matrix with readable scrolling columns and a pinned first column. Default labels preserve input paths verbatim; numeric source IDs track input order and explicit labels remain optional.
• Allow arbitrary Reference/Compared selections, including reversed pairs. Reuse the original case viewer inside Comparison for aligned prompts, full responses/turns, grades, evidence, and explicit absent-case states. Switching back to Run details restores each file's internal with/without binding and feedback.
• Reuse original report styles, sections, navigation, and case rendering. Align selectors and a concise Compare button; collapse detailed help, color explanations, and successful CI gates. In comparison mode, show Benchmark only when the selected run contains benchmark data. Legacy Benchmark behavior is unchanged.
• Color pass rates, statuses, and changes without losing text/signs: green for improvement, red for regression, blue for descriptive changes, gray for unchanged/unknown/absent. Token and time reductions are green, increases red.
• Preserve missing measurements as unknown/null. Report metadata and recorded prompt differences without regrading or inferring missing provenance. Text/JSON/Markdown and CI gates compare later inputs to the first; interactive HTML pair selection does not change gate results. Save the complete report before returning gate failure and protect every source artifact from overwriting.

Test plan

• make verify passes, including full lint and commit-hook checks.
• Comparison, report, and CLI race tests pass, covering multiple inputs, arbitrary pair semantics, primary selection and fallback, coverage/transitions, unknown metrics, directional colors, gates, label/path preservation, and source protection.
• Browser checks cover B→C and C→B, added/removed cases, original response/grade/turn rendering, feedback isolation, with/without restoration, and unchanged legacy output after whitespace normalization.
• Six-input desktop/mobile checks cover horizontal scrolling, pinned columns, long paths, dropdown spacing, wrapped controls, fifth/sixth-run selection, actual equal-height/bottom-aligned Compare controls, collapsed help, and Benchmark visibility changing with recorded data.
• Full make test: all packages pass except the existing internal/runtime test TestNoneRuntime_ExecStaleLockDoesNotHangNextCommand (none_test.go:491, stale child reports ALIVE). Reproduced from an unmodified archive of base commit 99894b3.

Notes for reviewers

The pairwise comparison core builds on the unmerged implementation in #183. HTML pre-renders ordered pair summaries with the Go comparison engine, so summary size grows quadratically with input count; original-run details and canonical primary-case snapshots are embedded once per input. Cross-file case details are read-only; normal comparison-mode run feedback is isolated per run and exported locally.

Run IDs are positional within an invocation. Run-level total tokens include all configurations; input/output tokens and pass rates use primary cases. Aggregate deltas include each run's full case set; only shared case IDs are classified as fixed or regressed. Missing suite, Skill revision, or judge/rubric provenance remains unknown.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-29T03:49:36.429178Z	555c508	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.