Repository navigation
Give MSTest tests under motus run a TestContext, data rows and MSTest's cleanup order - #40
Merged
Merged
Conversation
…'s cleanup order motus run and the visual runner call tests by reflection, and their MSTest support stopped short of what a suite written for dotnet test expects. No TestContext was ever set, so a test that used one threw, the initializers were handed null, and MotusTestBase read a null outcome as a failure and saved a trace for every test that passed. [DataRow] and [DynamicData] were ignored, so data-driven tests failed with a parameter count mismatch, and cleanup ran base class first, taking the browser away before a derived cleanup that needed it. Both runners now run each test case through one shared invoker. It builds a TestContext from the test assembly's own MSTest by generating a subclass at run time, because the runners don't reference MSTest and the type has moved between assemblies and gained abstract members across major versions. That covers MSTest 3 and 4, including TestContext.Current and TestDisplayName where they exist, and a version it can't subclass runs without a context as before. CurrentTestOutcome is the test's own outcome by the time cleanup runs, so a passing MotusTestBase test keeps no trace. Discovery expands every MSTest data source into one test per row, named the way MSTest names it, and recognizes attributes derived from [TestClass] and [TestMethod], such as [DataTestMethod] and Motus's own [MotusTestClass]. Cleanup and NUnit teardown now run derived class first, and every cleanup runs even when an earlier one throws. The visual runner also runs [ClassInitialize] and [ClassCleanup], which it skipped, and a quarantine list can name a data-driven test's method to cover every row. Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
SummarySummary
CoverageMotus - 80.7%
Motus.Abstractions - 87.7%
Motus.Analyzers - 91%
Motus.Cli - 82.2%
Motus.Codegen - 89.8%
Motus.Mcp - 91.9%
Motus.Mcp.Http - 98.6%
Motus.Recorder - 77.8%
Motus.Testing - 75.4%
Motus.Testing.MSTest - 79%
Motus.Testing.MSTest.v4 - 22.8%
Motus.Testing.xUnit - 63.6%
|
TorreyBetts
added a commit
that referenced
this pull request
Oct 24, 2026
Brings in #31, #32, #33, #35, #36, #37, #38 and #40. Everything merged cleanly; the frame navigation handlers in Page.Events.cs sit next to the execution context tracking from #33 without touching it, and same-document navigations still leave execution contexts alone. Signed-off-by: Torrey Betts <torrey.betts@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
motus runand the visual runner call tests by reflection rather than through the MSTest adapter, and their MSTest support stopped short of what a suite written fordotnet testexpects:TestContextwas ever set. A test that used one threw aNullReferenceException,[AssemblyInitialize]and[ClassInitialize]were handednull, andMotusTestBaseread the missing outcome as a failure, so every passing test saved a failure trace (Under motus run and the visual runner, MSTest tests that pass still save failure traces #10).[DataRow]and[DynamicData]were ignored. The method was called once with no arguments and failed with a parameter count mismatch, and[DataTestMethod]methods weren't discovered at all.[TestCleanup]ran base class first, so a derived cleanup ran after the browser it needed had gone. NUnit's[TearDown]had the same problem.What changed
Both runners now run each test case through one shared invoker in
Motus.Runner/Services/Execution, whichmotus runreaches throughInternalsVisibleTo. The two copies of the lifecycle code are gone.TestContext. The runners don't reference MSTest, the test assembly brings its own, and
TestContexthas moved between assemblies and gained abstract members from one major version to the next (MSTest 4 keeps it inMSTest.TestFramework.Extensions.dlland addsDisplayMessage,TestDisplayName,TestData,TestExceptionand, from 4.1, a staticTestContext.Current). So the subclass is generated at run time withReflection.Emit, once perTestContexttype: it overrides whatever that version declares abstract, plusCurrentTestOutcome, and forwards each call to the runner. The property keys it fills are read from MSTest's own label fields, and the optional members (TestDisplayName,TestData,TestException,CancellationTokenSource,Current) are set by reflection when the version has them. If a version can't be subclassed this way, the test runs without a context, as before, rather than failing.The test gets its context through its
TestContextproperty, or through a constructor that takes one (MSTest 3.6 and later), before[TestInitialize].[AssemblyInitialize],[ClassInitialize]and the matching cleanups get one when they take it.CurrentTestOutcomeisInProgresswhile the test is set up and run, thenPassed,FailedorInconclusiveby the time[TestCleanup]runs, which is whatMotusTestBasechecks before keeping a trace. #10 is fixed entirely from the runner side;Motus.Testing.MSTestis untouched.I considered a compiled subclass per major version picked by reflection, but each would need its own MSTest reference, and a runner build would still break on the next version that adds an abstract member. Generating it covers every version the runner meets.
Data rows. Discovery asks every attribute that implements MSTest's
ITestDataSourcefor its rows and their display names, so[DataRow],[DynamicData]and custom sources all work, and a row is named the way MSTest names it:SignInTests.SignsIn (owner)on MSTest 3,SignInTests.SignsIn ("owner")on MSTest 4. A display name that leaves out the method name gets it added, identical rows get a#2suffix so results don't collide, and a row'sIgnoreMessage(MSTest 3.8+) skips just that row. Arguments are bound with optional parameters andparamsarrays filled in. A data source that throws is reported and leaves a single case that fails saying its parameters weren't supplied. The visual runner's explorer shows each row's name instead of repeating the method name.Discovery by base type. A test class or method attribute counts if its type or any base type is one of the known names, which picks up
[DataTestMethod], customTestMethodAttributesubclasses, and Motus's own[MotusTestClass]and[MotusTestMethod]. xUnit[Theory]derives from[Fact]and is still left out, since nothing supplies its data.Lifecycle order.
[TestInitialize]and NUnit[SetUp]still run base class first.[TestCleanup]and NUnit[TearDown]now run derived class first and base class last, which is what both frameworks do. xUnit has no inherited lifecycle attributes, only the constructor andDispose, so nothing changes there. Each cleanup runs even if an earlier one threw, so a throwing derived cleanup can't stopMotusTestBasefrom returning its browser context to the fixture.Also:
[ClassInitialize]/[OneTimeSetUp]before a class's first test and[ClassCleanup]/[OneTimeTearDown]at the end of the batch. Before, it skipped them, whilemotus runran them.docs/reference/cli.mdgets a "How tests run" note saying what the runners support. TheTestContextrow indocs/guides/testing-frameworks.mdand the quarantine list format indocs/guides/flaky-tests-and-quarantine.mdare updated.Behavior changes
motus runand the visual runner get aTestContext, andMotusTestBasekeeps failure traces only for tests that fail. Before, every test saved one when tracing was on.[DataRow]and[DynamicData]tests run once per row, under names likeClass.Method (args).--filter, shard assignment, flaky history and reporters all see these per-row names.[TestCleanup]and[TearDown]run derived class first, and a throwing cleanup no longer skips the rest.[MotusTestClass]and methods marked[DataTestMethod]or[MotusTestMethod]are now discovered. Most ofMotus.Samplesuses[MotusTestClass], so the CI step that runsmotus runon the samples now runs 51 tests where it ran almost none. Locally all 48 that aren't ignored pass.[ClassInitialize]and[ClassCleanup].TestContext.WriteLineoutput goes to the console.AddResultFileis accepted and ignored, since the runners have no attachments.Tests
New tests fail on
origin/main(10 of the 11 inTestRunnerMSTestTests; the eleventh guards the cleanup-throws case that the reordering would otherwise break):TestRunnerMSTestTests(Motus.Cli.Tests):TestContextbefore[TestInitialize], the outcome seen in cleanup for passing, failing and inconclusive tests, constructor injection,[ClassInitialize]and[AssemblyInitialize]contexts, MSTest and NUnit cleanup order, a throwing derived cleanup, data row discovery and execution ([DataRow],[DataTestMethod],[DynamicData]), and quarantine by method.FailureTraceIntegrationTests(Motus.Cli.Tests, Integration): runs a passing and a failingMotusTestBasetest throughTestRunnerwith failure tracing on, and checks only the failing one leaves a trace. With the context disabled it fails with "a passing test should leave no failure trace". Failure tracing is read from configuration once per process, so the test assembly switches it on in a module initializer.TestExecutionServiceTests(Motus.Tests): the same outcome, ordering, data row and[ClassInitialize]checks through the visual runner, plus discovery.TestCaseExecutionTests(Motus.Tests): argument binding, row naming, unreadable data sources, and the generated context, including one built from MSTest 4.0.1'sTestContextloaded into its own load context. Motus.Tests downloads that package withPackageDownload, without referencing it.Run locally on macOS:
dotnet build Motus.sln: 0 errors.TestCategory!=Integration, Samples excluded) on net8.0 and net10.0: all pass (Motus.Tests 1077, Motus.Cli.Tests 320 with 2 skipped, Mcp 366, Recorder 105, Codegen 59, Analyzers 29, xUnit 8).FailureTraceIntegrationTestson net8.0 and net10.0, and the Motus.Cli.Tests Integration category on net8.0 (10 passed), in Chrome.motus run(net10.0) onMotus.Samples: 48 passed, 0 failed, 3 ignored.motus runon small suites built against MSTest 3.8.3, 4.0.1 and 4.5.1 with the repro from motus run's MSTest support lacks TestContext and [DataRow], and runs a base class's [TestCleanup] before the derived class's #28: contexts in every test and initializer, rows named and run, derived cleanup first, constructor injection, and on 4.5.1TestContext.CurrentandTestDisplayName.Firefox wasn't run; nothing here touches the protocol layer.
Fixes #28
Fixes #10