Repository navigation
Run each test assembly in a test host process of its own - #45
Merged
Merged
Conversation
motus run and the visual runner loaded test assemblies into their own process, so a suite saw the runner's copies of anything both depend on (SkiaSharp 2.88 rather than the app's 3.x), the runner's install folder as AppContext.BaseDirectory, and the user's directory as the current one. WebApplicationFactory reads the app's .deps.json and manifest from those folders, so a suite that hosts an ASP.NET Core app could not run at all, though it passes under dotnet test. Each test assembly now runs in a child process started the way dotnet test starts its test host: dotnet exec of Motus.Runner.dll with the assembly's own .runtimeconfig.json and .deps.json, with its output folder as base and current directory. Discovery and each test run go over a named pipe; reporters, retries, quarantine, sharding and flaky history stay in the runner and work on the reports that come back. The visual runner starts one per run and gets the screencast, the timeline and step debugging from it over the same pipe. A collectible AssemblyLoadContext per assembly was not enough even inside a child: a test that carries newer Microsoft.Extensions packages than the installed ASP.NET Core would load those into its own context while ASP.NET Core bound the framework's, splitting their types. Only the host's own dependency resolution unifies them the way dotnet test does. A test that ends its process now fails with the exit code instead of taking the runner down, and the tests of that assembly after it fail saying they did not run. Fixes #27 Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
Contributor
Author
|
Design notes and the tests run, to go with the description above.
What changedEach test assembly now runs in a test host process of its own, started the way
The runner and the test host talk over a named pipe, one JSON message per line (
Tests built by hand from reflection metadata (the existing runner unit tests do this) still run in-process, through the same Startup cost. On this machine a test host adds about 0.35 s per assembly: A test host that crashes. Its stdout and stderr are the runner's own, so whatever the process wrote before it ended (a stack overflow report, say) is on the console. The runner sees the pipe close and waits briefly for the exit code. The test that was running fails with A suite built for an older .NET. Alternatives consideredA collectible Keeping the runner's Motus for the tests. Making the tests run on Behavior changes
No public library API changes. Docs: the Tests
Run locally on macOS:
Firefox wasn't run. Nothing here touches the protocol layer, and the test host is browser-agnostic. Left for later
|
The fixture suite that TestHostIsolationTests runs was built only through a project reference from Motus.Cli.Tests. A project outside the solution is built in its default configuration, so the Release build in CI produced it under bin/Debug while the test looked under bin/Release. In the solution it builds in the solution's configuration, and its own tests run under dotnet test, which shows the suite passes there too. Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
SummarySummary
CoverageMotus - 81.6%
Motus.Abstractions - 87.7%
Motus.Analyzers - 91%
Motus.Cli - 82.1%
Motus.Codegen - 89.8%
Motus.Mcp - 91.9%
Motus.Mcp.Http - 98.6%
Motus.Recorder - 77.8%
Motus.Testing - 75.4%
Motus.Testing.MSTest - 79%
Motus.Testing.MSTest.v4 - 22.8%
Motus.Testing.xUnit - 63.6%
|
Contributor
Author
|
Follow-up to the notes above: the first CI run failed only WebAppSuite_RunsWithItsOwnDependenciesAndFolder, because the fixture outside the solution was built as Debug in CI's Release build. 28eff8c adds both fixture projects to Motus.sln under tests, so they build in the solution's configuration and their own four tests now also run under dotnet test. CI is green on 28eff8c across all three .NET legs and both integration jobs. |
The test host runs a suite on the Motus its project references, but it named Motus types directly: the test report carried accessibility, performance and coverage records, and the host called the sinks, the test method name context and the lost browser checks. A suite on an older published package lacks some of those, so on 1.0.10 the host crashed reading its first message and no test ran, and on 1.0.11 to 1.0.14 every test failed with a TypeLoadException. What a run collects now crosses the pipe as JSON that the runner reads into its own types, and every call into Motus goes through a guard that makes it from a method of its own and, when the suite's Motus cannot satisfy it, says so once and goes on without it. The visual runner's browser hooks are compiled up front for the same reason. A suite on 1.0.1, the oldest published package, now runs with its own results; a fixture pinned to it keeps that true. An assembly whose test host could not start, load it or list its tests was reported on standard error and skipped, and the run exited 0 when nothing else failed, so a broken run looked green. It now fails the run. The visual runner fails that assembly's tests and runs the others, where it used to stop and leave them as they were. A fresh test host reads each data source again, and a row that names itself differently on each read (a Guid or a time in it) was then not found. Each test case now has an identity apart from its name: its class, its method with the parameter types, and for a data row, which data source and which row of it. Runs ask for tests by it, a row the source no longer returns fails saying so, and sharding sorts by it so that separate shard processes agree. Overloads, which share a name, no longer collapse into one test run twice. Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
Contributor
Author
|
58a4446 fixes the review findings. Older Motus packages. The test host runs on the suite's own Motus. I ran the reviewer's suites and a 1.0.1 suite against the runner before the fix. These members of
Now:
Results:
The new
Its own six tests also run under Assemblies that fail to load now fail the run.
Stable test identity.
Runs (macOS, net8.0 and net10.0):
|
motus run made each reporter's per-test callbacks from whichever worker finished the test, and the JUnit, TRX and HTML reporters append to plain lists. On main, tests that finished one after another hid that; with the test host, every test finishes on a pool thread after a pipe round trip, and a quick suite under --workers auto lost most of its results from the JUnit and TRX files. The runner now makes those callbacks one at a time. The visual runner looked a test up by its id before its name. After a rebuild that removed a [DataRow], the old row's id belonged to the row that moved up, which then ran and reported under the removed row's name. A test is now found by its name when exactly one case has it, by its id otherwise, and a [DataRow], whose name never changes, is never taken from its id under a different name; that case fails saying the rows changed. Overloaded test methods without data rows now get their parameter types in their names, so reporters and results can tell them apart. Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
Contributor
Author
|
79d08e3 fixes the re-review findings. Reporter results lost under several workers (HIGH). I checked the other per-test paths:
Wrong data row after a rebuild (medium). Overload names (low). Overloaded test methods without data rows get their parameter types in the name: Runs (macOS, net8.0 and net10.0):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
motus run and the visual runner loaded each test assembly into their own process, so a suite got the runner's copy of anything both depend on (SkiaSharp 2.88 instead of the app's 3.x) and the runner's install folder as AppContext.BaseDirectory. WebApplicationFactory reads the app's .deps.json from the base directory and its manifest from the current directory, so a suite that hosts an ASP.NET Core app could not run at all, though it passes under dotnet test.
Each test assembly now runs in a test host process of its own, started the way dotnet test starts its test host: dotnet exec of Motus.Runner.dll with the suite's own .runtimeconfig.json and .deps.json, with the suite's output folder as base and current directory. motus.config.json is still read from the directory motus run was started in, so the runner and its test hosts see the same settings. Discovery, the assembly, class and test lifecycle, and the accessibility, performance and coverage sinks run in the test host, using the execution code the two runners already shared, which is now one copy of discovery and lifecycle instead of two. Scheduling, retries, quarantine, sharding, filters, reporters, flaky history and enforcement stay in the runner and work on the report each test run sends back over a named pipe. The visual runner starts a test host per run and gets the screencast, the timeline and step debugging from it over the same pipe. A suite built for an older .NET than the runner rolls forward to the newest installed one, since Motus.Runner.dll cannot load on an older runtime.
Because the suite runs on its own Motus, which can be older than the runner's, the runner never names Motus types across the pipe (what a run collects travels as JSON) and makes every call into Motus through a guard that, when the suite's Motus lacks something, warns once and goes on without that extra. Suites on every published package back to 1.0.1 run with their own results. Each test case also has an identity apart from its display name (class, method with parameter types, and for a data row its data source and row), so a fresh test host finds a [DynamicData] row whose name changes on every read, sharding agrees across processes, and overloaded tests each run.
A collectible AssemblyLoadContext per assembly was rejected, even inside a child process: besides leaving the base directory alone, it splits types whenever a suite carries newer Microsoft.Extensions packages than the installed ASP.NET Core (as Mvc.Testing does), because ASP.NET Core in the default context binds the framework's copies while the suite's context loads its own. Only the host's own resolution from the suite's .deps.json unifies them the way dotnet test does.
Behavior changes: tests run in a separate process per assembly, on the dependency versions, frameworks and Motus version their project references rather than motus's own, and motus run warns when that Motus version differs from its own. The current directory for tests is the assembly's output folder, so relative paths used by tests and relative failure.tracePath and failure.screenshotPath settings now resolve there; failure traces from motus run land under the output folder's test-results/traces unless an absolute path is set. A test that exits or crashes its process now fails with the exit code, along with the tests of that assembly still to run, instead of ending motus run. An assembly named on the command line that cannot be loaded, or whose test host cannot start or list its tests, now makes motus run exit 1 (it used to be skipped with a message and exit 0), and that includes a non-test file picked up by a shell glob such as bin//.dll; the visual runner fails that assembly's tests and runs the others. On an older Motus package, accessibility reports, performance metrics, coverage or retry of lost browsers that the package cannot support are left out with a warning. Shards are assigned by test identity rather than display name, so shard membership differs from 1.1. Overloaded test methods without data rows are named with their parameter types, such as Class.Method(Int32). Reporter callbacks are made one at a time, so every result reaches the JUnit, TRX and HTML files under several workers. The visual runner starts a fresh test host for each run, so statics do not carry over and a rebuilt assembly is picked up. Each test assembly adds about a third of a second of startup.
Fixes #27
Signed-off-by: Torrey Betts torrey.betts@gmail.com