Repository navigation
Improve integration test performance - #1300
Merged
Merged
Conversation
isc-dchui
requested review from
isc-cborbonm,
isc-egabhart,
isc-eneil,
isc-jili,
isc-jlechtne,
isc-kiyer and
isc-tleavitt
as code owners
November 2, 2026 21:59
isc-kiyer
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Resolves #1195
Resolves #1251
This cuts integration test time by about a third, based on a MONLBL profile of the full suite and per-class timings. Most of the time went to work the tests didn't need: scanning unrelated modules, downloading large Python packages, and querying the public community registry. It also makes every test clean up the files it creates in the repo.
CI timings
Each CI job runs the full test suite twice: in "Test and Verify", and again in "Clean install current version and test new one." The table compares the average of the two main runs just before this branch (a765da5, d3815fa) with the branch run (d5c57f2). The averages cover the four images of each flavor.
The slowest job, which bounds the workflow when nothing is queued, went from 42.8 min to 28.7 min. Runner timings vary by up to about 30% between runs of the same job on main, so read these as approximate.
Where the time came from, measured locally per class against an unprofiled full run:
repo -fsrootrepo -fsrootWhat changed
Repo roots. A
repo -fsrepo scans every module.xml under its root when it is created. FileCopy and PublishModuleWithDeps rooted theirs at all of_data/, which holds 217 modules. Each is now rooted at its own fixture folder, and FileCopy no longer leaves its repo configured for later classes.Python fixtures. The packages in
requirements.txtand the committed wheels were much larger than the tests need:package-with-python-depsjust-reqsjust-wheelsEach fixture still covers what its test checks: a package with a dependency of its own, a requirements.txt-only module, and a
<PythonWheel>resource.Community registry. Seven classes ended with
repo -reset-defaults, which re-enablespm.community.intersystems.com. The verify namespace keeps its repos between classes and between runs, so every later dependency lookup also queried the internet:The cleanups now only delete repos, and BuildDependencyGraph clears repos before creating its own. This also removes the suite's dependency on reaching the internet registry.
Files left in the repo. Run one class at a time, every integration test class and the unit suite now leave the working tree as they found it. The ones that didn't:
module-lock.jsonfiles, which are inputs for the "overwrite an existing lock file" cases, so it now saves and restores them_data/varresolver/module.xmlwhenever the test failed before its own cleanup. The module is now generated outside the repolocalize/HTTPStatuses.xml, which Activate exports into the module root-export-python-depsdownloads into<module root>/wheels. The committedpackage-with-python-deps/wheels/was leftover output from those downloads, so it is removedThe
.gitignorefiles that hid the Resources and ProcessPythonWheel output are removed, so a future leak shows up ingit status.Product code.
zpm "..."call even though those never show it.LoadNewModuleran a full%XML.Readerpass over module.xml twice, once for<Defaults>and once for the version checks.Testing
Checklist
mainbranch rebased or merged.zpm test -only) and integration tests (zpm verify -only) pass.🤖 Generated with Claude Code