Skip to content

Add the desktop-test harness, package-owned desktop testing guidance, and focused test runs - #4389

Merged
Jason Morse (JasonVMo) merged 9 commits into
mainfrom
user/jasonvmo/replace-appium
Oct 23, 2026
Merged

Jason Morse (JasonVMo) merged 9 commits into
mainfrom
user/jasonvmo/replace-appium

Conversation

@JasonVMo

@JasonVMo Jason Morse (JasonVMo) commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Platforms Impacted

  • iOS
  • macOS
  • win32 (Office)
  • windows
  • android

Description of changes

Replaces the Storybook-specific desktop lifecycle with an application-neutral desktop-test harness on top of the
existing desktop-driver, and keeps Storybook as an optional integration. The PR also:

  • qualifies the refactor on Win32 and macOS;
  • moves desktop-testing knowledge into package-owned agent guidance;
  • lets yarn test run a subset of a package's tests.

Legacy desktop Appium and all mobile lanes are unchanged. They stay until native semantic parity and multi-day
reliability qualification are complete (#4345).

Summary

  • @fluentui-react-native/desktop-test (new):
    • one configuration schema for app targets and suites;
    • native lifecycle and process ownership for installed apps, RN Test App, and REX hosts;
    • standalone WDIO/Jasmine execution;
    • one authenticated reporting collector with live diagnostics, durable logs, and JSON, JUnit, console, and GitHub
      Actions output.
  • Storybook (storybook-desktop, apps/storybook): uses the shared lifecycle and reporting, while keeping
    isolated CSF callback workers, authenticated readiness, exact app leases, and platform exclusions.
  • desktop-driver:
    • binds native visibility and focus to existing WDIO browsers;
    • strengthens Windows attach identity;
    • forwards trusted launch timeouts and unpackaged working directories;
    • exposes UI Automation required-form state, reported as unsupported on other providers;
    • reads native checkbox and switch state as checked state, without changing selected semantics or legacy Appium
      behavior.
  • FluentTester and E2E: apps/fluent-tester, apps/win32, and apps/E2E declare canonical desktop targets and
    suites in desktop.config.mts without importing test tooling into the app. E2E page objects gain native desktop
    adapters.
  • macOS fixes (31098f4):
    • the reporting root is canonicalized (/var to /private/var);
    • the harness no longer sends stop frames to producers whose Unix sockets have half-closed;
    • Storybook requests the renderer-scoped target ID;
    • RN Test App launches the exact built product, and fails closed on a bundle-identifier, renderer, or embedded-bundle
      mismatch;
    • retried attempts are labeled by ordinal;
    • FluentTester buttons scroll until their click point is unobscured;
    • the RNmacOS Yarn patch now exposes accessible Fabric <Text> paragraphs to accessibility.
  • Package-owned guidance (a27723f):
    • desktop-driver, desktop-test, storybook-desktop, and storybook-desktop-runtime each get an agent
      AGENTS.md, a user README.md, and an indexed references/ directory.
    • New rules R-PKGDOC-001 to R-PKGDOC-008 are enforced by yarn check-agent-instructions.
    • New task yarn check-package-contents, part of buildci, checks actual yarn pack inventories. files
      allowlists keep build caches out of the four packages.
    • New skills desktop-testing and maintain-desktop-testing. agentic-storybook-development now covers only the
      app toolchain.
    • driver-next-plan.md and desktop-driver/PLAN.md are removed; Infrastructure: Complete Desktop Driver production release qualification #4345 tracks the remaining qualification.
  • Focused test runs (e33dae0): fluentui-scripts jest, behind yarn test and yarn update-snapshots, used to
    pass every discovered test file to Jest, so yarn test popover ran the whole suite.
    • It now accepts path selectors, -t, --findRelatedTests, -o, and --changedSince.
    • Platform-suffixed tests still run only on their own platform.
    • New rule R-VALID-004 has agents iterate on focused runs, then run yarn clean-lage-cache and yarn lage buildci
      once as final acceptance.
  • Code scanning and CI (3c40367):
    • CodeQL js/polynomial-redos: the duplicated secret-redaction regex was quadratic on a keyword followed by many
      spaces (7.3 s on 200 KB). One shared linear-time pattern now gives identical output (0.2 ms), with a regression
      test.
    • CodeQL js/bad-code-sanitization: desktop-test contract fixtures spliced JSON.stringify'd paths into executed
      Node source. The source is now constant, and values arrive through process.argv or import.meta.url.
    • check-agent-instructions checked agent-map paths on the file system. The generated, gitignored
      storybook.requires.ts therefore passed locally but failed the clean CI checkout. It now checks against Git and
      exempts sources declared policy: generated-….
    • New reference references/agents/repository/code-scanning.md (R-SCAN-001 to R-SCAN-003) is cited from the root
      safeguards and the review and desktop maintenance skills.

Findings worth reviewing

  • Runner behavior, 77 packages: with fluentui-scripts jest, a bare argument that used to do nothing now narrows
    the run, and one that matches no test file exits 1. Unselected runs are unchanged, except for a one-line tip printed
    locally (not under CI or TF_BUILD) in packages with 10 or more test files. No pipeline or package script passes
    positional arguments to the runner.
  • RNmacOS patch: the Fabric <Text> accessibility fix belongs upstream in react-native-macos. Until then, the Yarn
    patch is required for accessible Fabric paragraphs.
  • Infrastructure: Complete Desktop Driver production release qualification #4345: its description links packages/agentic/desktop-driver/PLAN.md as the remaining-work inventory, which this
    PR removes. Its acceptance criteria still list the work; update the description when this merges.
  • knip.config.ts: a narrow Windows portability fix for backslash-separated workspace keys, found during Windows
    validation.
  • Windows baseline: two unmatched Windows baseline patterns fail closed rather than silently reducing coverage.

Deferred and follow-ups

  • Infrastructure: Complete Desktop Driver production release qualification #4345:

    • equivalent-build old and new comparisons;
    • complete assertion mapping;
    • FluentTester runs on Windows Fabric;
    • the remaining Windows Storybook renderers;
    • live GitHub Actions presentation;
    • the 100-run, 14-day reliability gate.

    Retiring desktop Appium waits on these.

  • Extend package-owned agent guidance to agentic and framework-base packages #4388: extend package-owned guidance to the remaining packages/agentic/* packages and framework-base.

  • Harness cleanup: a failed readiness hook can leave "owned reporting producers did not finish their shutdown
    handshake" noise. A crashed session leaves the per-user interactive lock behind, and the harness correctly refuses to
    run until it is removed.

  • Dead scripts: 23 packages declare a start-test script that calls a nonexistent fluentui-scripts jest-watch.

Verification

PR head 3c40367, macOS, October 22, 2026

Check Result
yarn clean-lage-cache, then yarn lage buildci Passed: 175 tasks, 0 failed
yarn lage test-links Passed: 88 tasks
yarn change:check Passed
yarn check-package-contents Passed: four inventories
yarn check-agent-instructions with the generated storybook.requires.ts removed, as in CI Passed
packages/agentic/desktop-test: yarn test Passed: 25 tests, including the new redaction tests
apps/storybook: yarn storybook smoke --macos --mode stories-and-tests (Fabric), at d683ca8 Passed: 199 stories rendered and 81 story tests passed (280/280), 0 skipped, 0 retries

Focused test runner, macOS

  • yarn workspace @fluentui-react-native/scripts test: passed, 49/49, including selection unit tests that load the
    installed Jest's option table.
  • In packages/agentic/components:
    • yarn test popover selected 2 of 66 files, 19 tests, in 2.5 s; the unfiltered yarn test still ran 66 suites and
      561 tests in 5.3 s.
    • yarn update-snapshots button selected 4 files with no snapshot changes.
    • A typo exited 1.
    • --findRelatedTests on a shared primitive selected 16 files; a missing source path exited 1.
    • With a temporary uncommitted edit, -o and --changedSince=HEAD selected the 5 related tests.
    • --changedSince=<commit>^ selected 25 tests from committed history.
  • Multi-platform and react-platform packages:
    • In packages/components/Button, yarn test ColorTokens.test.win32 ran only the win32 platform group.
    • packages/theming/default-theme (five platforms) passed.
    • packages/agentic/desktop-test (react platform) passed.

macOS qualification at 31098f4, October 22, 2026 (after merging main at 60121de)

No skips or retries:

Lane Fabric Paper
Storybook lifecycle 280/280 280/280
FluentTester pilot 2/2 2/2
FluentTester legacy (62 tests) 62/62 62/62
  • Package suites:
    • desktop-test contracts: 16/16;
    • storybook-desktop: 175 passed, 2 skipped;
    • opt-in macOS desktop-driver tests: 68 passed, 2 skipped.
  • buildci: the first run hit a Jest worker SIGSEGV in the unchanged design package, whose tests had all passed. The
    rerun passed.

Win32 at 238db96, October 21, 2026 (before merging main)

  • yarn workspace @fluentui-react-native/e2e-testing desktop-test --target win32 test --suite behavioral-pilot: passed
    37/37. These are the unchanged Button, Checkbox, Switch, directional FocusZone, and Callout assertions.
  • With STORYBOOK_SMOKE_STORY=components-button--default,
    yarn workspace @fluentui-react-native/agentic-components-storybook desktop-test --target win32 test --suite stories:
    passed 154/154, which is 152 navigations plus both native Button callbacks.
  • With FURN_NATIVE_DRIVER_TEST=1, yarn workspace @fluentui-react-native/desktop-driver test --runInBand: 68 passed;
    2 macOS-only tests skipped.
  • Package suites:
    • harness: 15 passed;
    • Storybook: 175 passed;
    • E2E adapters: 4 passed;
    • test:appium: 3 discovery and configuration tests passed; this is not legacy native execution.
  • After yarn clean-lage-cache, yarn lage buildci --concurrency 4: passed.

Not run

  • Windows Fabric and Win32 were not rerun after merging main (Add Popover, Menu, MenuButton, and Dropdown for macOS, Windows, and Win32 #4386) or after the macOS fixes. The Win32 evidence
    above is from 238db96.
  • FluentTester on Windows Fabric, and the remaining Windows Storybook renderers.
  • The Paper Storybook smoke at the PR head. Since Paper passed 280/280 at 31098f4, only documentation, the Jest
    runner, and the four packages' files allowlists have changed, and none of these depend on the renderer. The macOS
    Paper Storybook CI job runs on this PR.
  • The focused runner on Windows: backslash selectors are covered by unit tests only, and the --runInBand path under
    TF_BUILD is unexercised.
  • Live GitHub Actions presentation and the reliability gate.
  • VoiceOver and Narrator, including the newly exposed Fabric <Text> paragraphs.
  • Live model routing for the new evaluation fixtures, which check-agent-instructions validates structurally only.

Pull request checklist

This PR has considered (when applicable):

  • Automated Tests
  • Documentation and examples
  • Keyboard Accessibility
  • Voiceover
  • Internationalization and Right-to-left Layouts

…-appium

# Conflicts:
#	apps/storybook/AGENTS.md
Fix macOS-only harness defects found while running the neutral desktop-test
harness and Storybook integration on macOS:

- Canonicalize the reporting root (/var -> /private/var) and its contract
  fixture root.
- Skip stop frames to producers whose Unix sockets already half-closed.
- Request the renderer-scoped Storybook target the embedded driver registers.
- Launch the exact built RN Test App product on macOS, failing closed on a
  bundle identifier, renderer, or embedded bundle mismatch.
- Label retried attempts by ordinal.

Scroll FluentTester buttons until their click point is unobscured, and let
specs that share a page find it already selected. Extend the RNmacOS patch so
accessible Fabric <Text> paragraphs are exposed to accessibility on macOS.

macOS Fabric and Paper both pass Storybook 280/280 and the FluentTester pilot
and legacy (62 tests) suites with no skips or retries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The package PLAN.md was removed, but the macOS and Windows helper READMEs
still linked to it, which fails the link check. Point the remaining
qualification work at its open tracking issue and drop the two prose
mentions of the plan.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread packages/agentic/desktop-test/src/reporting/collector.ts Fixed
Comment thread packages/agentic/desktop-test/src/reporting/session.ts Fixed
Comment thread packages/agentic/desktop-test/src/contracts.contract.cjs Fixed
Comment thread packages/agentic/desktop-test/src/contracts.contract.cjs Fixed
Comment thread packages/agentic/desktop-test/src/contracts.contract.cjs Fixed
Comment thread packages/agentic/desktop-test/src/contracts.contract.cjs Fixed
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Bundle size report

Tree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.

Scenario Modules-Mac (Δ) Modules-Win (Δ) Size-Mac (Δ) Size-Win (Δ)
components-button 64 (+2) 65 (+2) 50.63k (+687b) 50.55k (+687b)
components-catalog 214 (+55) 215 (+55) 205.49k (+50.22k) 205.41k (+50.22k)
design-theme-state 34 (+1) 35 (+1) 30.15k (+42b) 30.08k (+42b)
design-themed-root 38 (+1) 39 (+1) 32.12k (+42b) 32.04k (+42b)
design-color-lib 6 (+0) 6 (+0) 6.33k (+0b) 6.33k (+0b)
design-theme-state-legacy-source 35 (+1) 36 (+1) 31.81k (+42b) 31.73k (+42b)
design-legacy-theme-from-flex 58 (+1) 59 (+1) 97.77k (+42b) 97.63k (+42b)

The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail.


View bundle-size workflow run

…ules

check-agent-instructions verified agent-map sources on the file system,
so the gitignored, prebuild-generated storybook.requires.ts passed locally
and failed in a clean CI checkout. Check sources against the Git-aware
file list and exempt entries whose policy declares them generated.

Address the code-scanning alerts on #4389:
- js/polynomial-redos: the duplicated secret-assignment redaction regex let
  two whitespace runs compete across an optional quote, which is quadratic
  on a keyword followed by many spaces. Share one linear-time pattern with
  identical output, and add a regression test with adversarial input.
- js/bad-code-sanitization: contract fixtures spliced JSON.stringify'd
  paths into executed Node source. Keep the source constant and pass
  values through process.argv or import.meta.url.

Add references/agents/repository/code-scanning.md (R-SCAN-001 to 003),
cite it from the root safeguards, the review skill, and the desktop
maintenance skill, document generated agent-map sources, and add a
development evaluation case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JasonVMo
Jason Morse (JasonVMo) merged commit d690e1c into main Oct 23, 2026
31 of 33 checks passed
@JasonVMo
Jason Morse (JasonVMo) deleted the user/jasonvmo/replace-appium branch October 23, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants