test(cli): narrow config diff live pin (CLI-2294) - #6437
Conversation
There was a problem hiding this comment.
🤖 AI Review
The PR improves the live test’s handling of legitimate auth drift, but three minor test-quality concerns are confirmed: spinner residue can hide the first matching stdout line, the baseline-family regex can drift from the registry, and the scope assertion does not prove auth fields reached classification. The stderr-normalization concern is refuted because the asserted diagnostics are emitted as unstyled raw strings. Codex reported no findings; all four Claude findings were adjudicated.
Findings
| Severity | Location | Category | Sources | Claim |
|---|---|---|---|---|
| 🟡 MINOR | apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:21 |
test-coverage |
claude | The start-anchored auth-noise filter can miss the first diff entry when non-newline-terminated spinner residue precedes it in captured stdout. |
| 🟡 MINOR | apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:23 |
maintainability |
claude | The hardcoded subtree regex can drift from the registry baselines it is intended to pin and can also classify future legitimate changes within those subtrees as baseline noise. |
| 🟡 MINOR | apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:15 |
test-coverage |
claude | The scope assertion proves that the raw response contains an auth block but does not prove that auth fields survived projection and classification. |
Refuted findings (kept for transparency, not posted as review comments)
apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:14(consistency): Matching raw stderr without ANSI normalization makes the assertions fragile to styled diagnostics or spinner output on stderr.
Refuted: The current code provides concrete counter-evidence: both diagnostics are deliberately emitted as raw, unstyled strings, while the text-mode spinner writes to stdout. A hypothetical future rendering change is not a defect in these assertions today.
Stats
Claude findings: 4 · Codex findings: 0 · Confirmed: 3 · Refuted: 1 · Uncertain: 0
Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run
This review runs once per PR. A maintainer can request another with a /ai-review comment.
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@ffdaf2ce103742115a397dfe81feb4248f4523adPreview package for commit |
TL;DR
fixes the
config difflive test failing on every CI run since it landed, which was caused by asserting a freshsupabase initconfig has no auth drift against a fresh project whose platform defaults legitimately differ from the local template.now fixed by:
pinning only the auth paths the registry declares baselines for, requiring the auth block in the comparison scope so the pin cannot pass vacuously, and stripping the spinner's ANSI from stdout before matching...
ref: