Repository navigation
test: derive slow tests' timeouts from the work they do - #33
Merged
Merged
Conversation
The suites that visit every generated model gave each test vitest's default timeout, which a machine under heavy load exceeds while the work is unchanged. Each test now takes a timeout of its own work count (candidate combinations, observed tuples or engine queries) times a named per-operation budget, so load alone cannot fail it and a search that stops terminating still does.
Each agreement and round-trip property made `RUNS` runs under vitest's default timeout, which a loaded machine exceeds while the work is unchanged. The timeout is now the property runs the test makes times a named per-run budget.
…meouts from their work The whole-space cancellation test, the condition monotonicity test and the trilean evaluation budget test ran under vitest's default timeout, which a loaded machine exceeds while the work is unchanged. Each now takes its searches, evaluations or tree nodes times a named budget.
Mearman
marked this pull request as ready for review
October 29, 2026 19:52
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.0.3 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #32
The unit tests that visit every generated model, or make
RUNSproperty runs, ran under vitest's default timeout. On a loaded machine they time out with unchanged work, which fails the pre-push hook. Each now takes a timeout of its own work count times a named per-operation budget, in the way the rank test already did. No global timeout changed, nothing skipped or retried.test/budget.tsholdsbudgetFor(operations, perOperationMs), which returns the{ timeout }options and refuses a non-positive result (a timeout of 0 disables vitest's), pluscandidateCount(the product of the options each dimension offers, constraints ignored) andsubsetCount.CANDIDATE_BUDGET_MSΣ (count + 1)over the models (unchanged); its constant is raised, see belowRANK_OPERATION_BUDGET_MSOBSERVATION_BUDGET_MSqueryBoundQUERY_BUDGET_MSCANDIDATE_BUDGET_MSin eachparseModelproperty testsRUNSper property, one or two propertiesRUN_BUDGET_MSSPACEtimes the five searches it makesCOMBINATION_BUDGET_MSEVALUATION_BUDGET_MSNODE_BUDGET_MSEach constant is at least two hundred times the CPU time per operation of the heaviest test that uses it on an idle machine, so only a loop that stops terminating fails. The existing rank budget was about sixteen times, not the forty its comment claimed, so I raised it to the same margin.
Not covered: the README example imports (
readme.unit.test.ts), the first test ofenumerate.unit.test.tsand the site suites. Their cost is module loading or DOM rendering of a few nodes, not a work count to derive; none exceeded the default timeout in the full run the pre-push hook made.Every commit type releases a patch version in this repo, so the
test:commits cut a patch release with unchanged package contents.