Validate all public samples in CI

[Amaury Levé (Evangelink)]

Summary

• Make dotnet test work for the modern UWP MTP sample while preserving its real packaged AppContainer validation path.
• Build standalone public sample projects that were previously omitted by solution-only discovery.
• Add a classified execution matrix so CI runs tests or smoke-checks every public sample.
• Fix Windows application VSTest recipe generation and isolate stale development package registrations between runs.

Validation

• Passed: eng\build-samples.ps1 -Configuration Release -TreatWarningsAsErrors -BinaryLogDirectory Q:\dev\testfx\artifacts\log\ReleaseSamplesBuild
• Passed: eng\test-samples.ps1 -Configuration Release -BinaryLogDirectory Q:\dev\testfx\artifacts\log\ReleaseSamplesTestFinal — 28/28 sample invocations validated.
• Passed: dotnet test -bl:{{}} from samples\public\UwpMtpApp — 1 passed, 2 inconclusive, 0 failed.

[Copilot]

Copilot review overview

🟡 Changes recommended

CI gating, change classification, Visual Studio discovery, and MSBuild property override issues remain unresolved.

Review effort: Balanced
Findings: 4

Open (4)

• Honor SkipTests when running sample tests · New
• Classify sample-validation script changes · New
• Preserve required UWP and test tooling filters · New
• Guard packaged path default against imported properties · New

What changed in this PR

Adds comprehensive CI validation for all public samples, including UWP packaged and command-line execution paths.

Changes:

• Adds a classified 28-invocation sample test matrix.
• Builds standalone projects omitted from solutions.
• Enables dual packaged/code-only UWP validation.

Confidence at a glance

🟡 Build, CI, and sample execution

Four actionable CI and MSBuild issues require resolution.

File	Description
samples/​public/​UwpMtpApp/​UwpMtpApp.csproj	Configures packaged and code-only builds.
samples/​public/​UwpMtpApp/​UnitTests.cs	Adds host and dispatcher-aware tests.
samples/​public/​UwpMtpApp/​README.md	Documents both execution paths.
samples/​public/​UwpMtpApp/​MainPage.cs	Supports operation without compiled XAML.
samples/​public/​UwpMtpApp/​App.cs	Makes packaged activation conditional.
samples/​public/​mstest-runner/​runner_vs_vstest/​10C_100M/​10C100M.csproj	Suppresses intentional constant assertions.
samples/​public/​mstest-runner/​runner_vs_vstest/​100C_100M/​100C100M.csproj	Suppresses intentional constant assertions.
samples/​public/​mstest-runner/​runner_vs_vstest/​1000C_100M/​1000C100M.csproj	Suppresses intentional constant assertions.
samples/​public/​Directory.Build.targets	Adjusts Windows VSTest recipe generation.
eng/​test-samples.ps1	Adds the sample execution matrix.
eng/​build-samples.ps1	Builds projects omitted from solutions.
azure-pipelines.yml	Runs sample tests in Windows CI.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

The sample test step ignores SkipTests, and Visual Studio discovery can select an installation without required UWP tooling.

Review effort: Balanced
Findings: 5

Open (5)

• Retain UWP requirements when selecting Visual Studio · New
• Guard packaged path default against imported properties
• Preserve required UWP and test tooling filters
• Classify sample-validation script changes
• Honor SkipTests when running sample tests

[Copilot]

Copilot review overview

🟡 Changes recommended

The packaged UWP path now tests a local dispatcher attribute instead of MSTest’s public UITestMethodAttribute.

Review effort: Balanced
Findings: 1

Open (1)

• Local UITestMethodAttribute shadows MSTest attribute in packaged builds · New

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation consistently covers public sample discovery, execution, and Windows application-model validation without unresolved defects.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Local UITestMethodAttribute shadows MSTest attribute in packaged builds

[Amaury Levé (Evangelink)]

/azp run microsoft.testfx

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[github-actions]

Final test-triage resolution — Azure Pipelines build 1620910

This supersedes any earlier preliminary comment for this build. Re-verified immediately before posting: PR #11689 head 56e3dcf1674a and merge commit b3f1bbf59ec4 match the expected values for this analysis.

Status: Clearing — isolated, non-reproducing timeout; no issue warranted.

Signal: Microsoft.Testing.Platform.ServerMode.Client.Sources.UnitTests.MtpServerProcessTests.ShutdownPreservesNaturalExitCodeAfterDisposingProcess failed once, on the Windows Release / net8.0 leg only, with:

Microsoft.Testing.Platform.ServerMode.Client.MtpServerConnectionClosedException: The Microsoft.Testing.Platform application '...\ConnectsThenExits.cmd' did not connect back within 10s.

Matrix: The same test passed on Windows Debug net8.0 (518ms) and net462 (610ms, 699ms) in this same build, and was (expectedly) skipped on Linux via [OSCondition]. Only the Windows Release net8.0 leg hit the 10-second ConnectionTimeout — a single affected cell out of four executed legs, with no retry/attempt record (no CTRF retryAttempts, so this is not evidence of flakiness per se, just a single observed failure).

History: Across the 5 most-recent completed builds in history.json (back to 20261001.35), this test passed every time it ran, with durations of 643ms–1436ms — nowhere near the 10s timeout. history.json is flagged incomplete (artifact-download-failed, download-budget-exhausted), so absence of a prior failure is not conclusive, but no recurrence is visible in the available window.

Issue search: No open or closed issue references this test or this exact timeout message. A superficially related closed issue (#11631) covers a different test (StartAsyncTimeoutKillsProcessAndReleasesListener) and a different race (message-selection ordering at a 300ms timeout), not applicable here.

Assessment: This looks like a one-off, agent-load-sensitive timing issue: the test spawns a real child process and powershell client over TCP loopback with a generous 10s allowance, and that allowance was still exceeded on this single Windows Release leg while an equivalent net462 run on the same job succeeded. No deterministic code defect is indicated, and the durable recurrence threshold (two independent builds/commits, or a proven fail-then-pass recovery) is not met.

Next step if this recurs: If this signature reappears in a future main/scheduled build or another PR, treat it as flaky and consider widening the timeout margin or adding diagnostic logging around process-connect latency before escalating to an issue.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 73.5 AIC · ⌖ 5.33 AIC · ⊞ 9.6K