Skip to content

test(cli): narrow config diff live pin (CLI-2294) - #6437

Open
7ttp wants to merge 2 commits into
developfrom
7ttp/config-diff-live-platform-defaults
Open

test(cli): narrow config diff live pin (CLI-2294)#6437
7ttp wants to merge 2 commits into
developfrom
7ttp/config-diff-live-platform-defaults

Conversation

@7ttp

@7ttp 7ttp commented Sep 2, 2026

Copy link
Copy Markdown
Member

TL;DR

fixes the config diff live test failing on every CI run since it landed, which was caused by asserting a fresh supabase init config has no auth drift against a fresh project whose platform defaults legitimately differ from the local template.

now fixed by:
pinning only the auth paths the registry declares baselines for, requiring the auth block in the comparison scope so the pin cannot pass vacuously, and stripping the spinner's ANSI from stdout before matching...

ref:

@7ttp 7ttp self-assigned this Sep 2, 2026
@7ttp
7ttp requested a review from a team as a code owner September 2, 2026 14:57

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

The PR improves the live test’s handling of legitimate auth drift, but three minor test-quality concerns are confirmed: spinner residue can hide the first matching stdout line, the baseline-family regex can drift from the registry, and the scope assertion does not prove auth fields reached classification. The stderr-normalization concern is refuted because the asserted diagnostics are emitted as unstyled raw strings. Codex reported no findings; all four Claude findings were adjudicated.

Findings

Severity Location Category Sources Claim
🟡 MINOR apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:21 test-coverage claude The start-anchored auth-noise filter can miss the first diff entry when non-newline-terminated spinner residue precedes it in captured stdout.
🟡 MINOR apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:23 maintainability claude The hardcoded subtree regex can drift from the registry baselines it is intended to pin and can also classify future legitimate changes within those subtrees as baseline noise.
🟡 MINOR apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:15 test-coverage claude The scope assertion proves that the raw response contains an auth block but does not prove that auth fields survived projection and classification.
Refuted findings (kept for transparency, not posted as review comments)
  • apps/cli/src/legacy/commands/config/diff/diff.live.test.ts:14 (consistency): Matching raw stderr without ANSI normalization makes the assertions fragile to styled diagnostics or spinner output on stderr.
    Refuted: The current code provides concrete counter-evidence: both diagnostics are deliberately emitted as raw, unstyled strings, while the text-mode spinner writes to stdout. A hypothetical future rendering change is not a defect in these assertions today.

Stats

Claude findings: 4 · Codex findings: 0 · Confirmed: 3 · Refuted: 1 · Uncertain: 0


Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread apps/cli/src/legacy/commands/config/diff/diff.live.test.ts Outdated
Comment thread apps/cli/src/legacy/commands/config/diff/diff.live.test.ts Outdated
Comment thread apps/cli/src/legacy/commands/config/diff/diff.live.test.ts Outdated
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@ffdaf2ce103742115a397dfe81feb4248f4523ad

Preview package for commit ffdaf2c.

@7ttp 7ttp changed the title test(cli): narrow config diff live pin test(cli): narrow config diff live pin (CLI-2294) Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant