Skip to content

Improve integration test performance - #1300

Merged
isc-dchui merged 3 commits into
mainfrom
improve-integration-tests
Nov 3, 2026
Merged

isc-dchui merged 3 commits into
mainfrom
improve-integration-tests

Conversation

@isc-dchui

@isc-dchui isc-dchui commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Resolves #1195
Resolves #1251

This cuts integration test time by about a third, based on a MONLBL profile of the full suite and per-class timings. Most of the time went to work the tests didn't need: scanning unrelated modules, downloading large Python packages, and querying the public community registry. It also makes every test clean up the files it creates in the repo.

CI timings

Each CI job runs the full test suite twice: in "Test and Verify", and again in "Clean install current version and test new one." The table compares the average of the two main runs just before this branch (a765da5, d3815fa) with the branch run (d5c57f2). The averages cover the four images of each flavor.

main this branch saved
IRIS, Test and Verify 557s 392s 165s (30%)
IRIS, Clean install and test 611s 440s 171s (28%)
IRIS, whole job 24.9 min 19.4 min 5.5 min (22%)
IRIS for Health, Test and Verify 945s 576s 370s (39%)
IRIS for Health, Clean install and test 1002s 624s 378s (38%)
IRIS for Health, whole job 39.4 min 26.7 min 12.8 min (32%)

The slowest job, which bounds the workflow when nothing is queued, went from 42.8 min to 28.7 min. Runner timings vary by up to about 30% between runs of the same job on main, so read these as approximate.

Where the time came from, measured locally per class against an unprofiled full run:

Change Class Before After
Smaller Python fixtures ProcessPythonWheel 187s 44s
Narrower repo -fs root FileCopy 93s 20s
Narrower repo -fs root PublishModuleWithDeps 77s 4s
Clear repos before building graphs BuildDependencyGraph 66s 13s
Stop re-enabling the community registry InstallModule 36s 26s

What changed

Repo roots. A repo -fs repo scans every module.xml under its root when it is created. FileCopy and PublishModuleWithDeps rooted theirs at all of _data/, which holds 217 modules. Each is now rooted at its own fixture folder, and FileCopy no longer leaves its repo configured for later classes.

Python fixtures. The packages in requirements.txt and the committed wheels were much larger than the tests need:

Fixture Before After
package-with-python-deps lune + ansible: 11 wheels, 53MB for ansible alone lune + jinja2, which pulls in markupsafe
just-reqs docx2md, which pulls in lxml and pillow markdown-it-py, which pulls in mdurl
just-wheels numpy wheel (16MB) tomli wheel (15KB)

Each fixture still covers what its test checks: a package with a dependency of its own, a requirements.txt-only module, and a <PythonWheel> resource.

Community registry. Seven classes ended with repo -reset-defaults, which re-enables pm.community.intersystems.com. The verify namespace keeps its repos between classes and between runs, so every later dependency lookup also queried the internet:

any class ─ reset-defaults ─> community registry ON, into the next run too
   ├── BuildDependencyGraph   ~2.5s per graph build (0.02s without it),
   │                          paying for the registry left on by the previous run
   ├── InstallModule          +9s
   └── ... every class that resolves modules by name

The cleanups now only delete repos, and BuildDependencyGraph clears repos before creating its own. This also removes the suite's dependency on reaching the internet registry.

Files left in the repo. Run one class at a time, every integration test class and the unit suite now leave the working tree as they found it. The ones that didn't:

Class Left behind
LockFile rewrote 8 committed module-lock.json files, which are inputs for the "overwrite an existing lock file" cases, so it now saves and restores them
Module _data/varresolver/module.xml whenever the test failed before its own cleanup. The module is now generated outside the repo
Resources localize/HTTPStatuses.xml, which Activate exports into the module root
ProcessPythonWheel the wheel build output in the fixture's package source, and the wheels -export-python-deps downloads into <module root>/wheels. The committed package-with-python-deps/wheels/ was leftover output from those downloads, so it is removed

The .gitignore files that hid the Resources and ProcessPythonWheel output are removed, so a future leak shows up in git status.

Product code.

  • The shell welcome message lists modules in every namespace, and it was built on every one-shot zpm "..." call even though those never show it.
  • LoadNewModule ran a full %XML.Reader pass over module.xml twice, once for <Defaults> and once for the version checks.

Testing

  • Existing tests pass

Checklist

  • This branch has the latest changes from the main branch rebased or merged.
  • [N/A] Changelog entry added.
  • Unit (zpm test -only) and integration tests (zpm verify -only) pass.
  • Style matches the style guide in the contributing guide.
  • Documentation has been/will be updated
    • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
  • Pull request correctly renders in the "Preview" tab.

🤖 Generated with Claude Code

@isc-dchui
isc-dchui merged commit 7b407e9 into main Nov 3, 2026
21 of 24 checks passed
@isc-dchui
isc-dchui deleted the improve-integration-tests branch November 3, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some IPM tests leave modified files Improve integration test performance

2 participants