From 703edb36e9250d3b6730dfe2690afa5ebbca7c2d Mon Sep 17 00:00:00 2001 From: Jason Morse Date: Wed, 21 Oct 2026 16:07:18 -0700 Subject: [PATCH 1/8] add driver refactoring plan --- driver-next-plan.md | 991 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 991 insertions(+) create mode 100644 driver-next-plan.md diff --git a/driver-next-plan.md b/driver-next-plan.md new file mode 100644 index 0000000000..f19f45df1f --- /dev/null +++ b/driver-next-plan.md @@ -0,0 +1,991 @@ +# General-purpose desktop testing: plan of record + +**Plan of record: Option 1, the neutral harness + Storybook integration.** +**Architecture accepted by the owner on October 21, 2026.** +Keep `desktop-driver` independent, add `desktop-test` for configuration and lifecycle, +and retain `storybook-desktop` and `storybook-desktop-runtime` as optional Storybook +packages. Run standalone specs through the normal WebdriverIO testrunner; preserve +the existing isolated story-callback runner. + +This makes FluentTester and REX ordinary consumers of the same infrastructure as +Storybook, without making ordinary applications pretend to be Storybook. +The harness also owns end-to-end reporting: delegated services and test engines +produce diagnostics/results, and one supervisor collects, persists, and presents +them. GitHub Actions grouping is a renderer of that reporting flow, not something +each child process manages independently. + +## Scope and evidence + +This is the accepted architecture and delivery plan, not an implementation or a +native qualification report. New configuration examples, interfaces, and commands +below describe intended APIs; they do not exist yet. Outstanding implementation +gates do not reopen the selected package architecture. + +The review covers the checkout at `e2092f9df3fd4fafc141c696ca0e49e690d69cd1`. +The legacy package is actually `apps/E2E`, with uppercase `E2E`. + +Two independent investigations were followed by reciprocal reviews: + +- Claude Opus 5.5, `claude-opus-5.5`, maximum reasoning, `long_context`. +- GPT-6 Astra, `gpt-6-astra`, maximum reasoning, `long_context`. + +`long_context` was the maximum context tier exposed by the agent tool. Both models +read the other's complete initial report before reviewing it. Their initial +recommendations differed on package boundaries; both ultimately recommended the +four-package architecture. Section 11 records the substantive corrections. The +complete reports and cross-reviews are retained as local session artifacts, not +dependencies of this document. Follow-up reporting research inspected the current +command runner, combined server, native diagnostic transport, Windows smoke +scripts, test workers, CI, and upstream reporting documentation; see section 6. + +**Evidence labels:** **Observed** means source-inspected behavior; **Inferred** +means a conclusion from that evidence; **Proposed** means a future design; +**Unqualified** means native behavior not demonstrated by this planning exercise. + +### Success criteria + +1. The selected macOS, Windows, and Win32 suites in `apps/E2E` can run against the + existing tester applications without Appium/WinAppDriver on those endpoints. +2. Storybook and non-Storybook applications share one configuration schema and + lifecycle implementation. A generic app needs no Storybook installation. +3. Standalone WDIO specs, page objects, Jasmine hooks, and grouped suites remain + supported. Migration does not require converting them into CSF stories. +4. Existing Storybook traversal, callback isolation, readiness authentication, + failure evidence, and platform-specific behavior remain intact. +5. Launch/attach identity, cancellation, application ownership, and native input + safety remain explicit. Skips are not counted as demonstrated parity. +6. iOS/Android E2E tooling remains operational and outside this migration. +7. Delegated-service diagnostics are visible while services run, without requiring + `--verbose` or waiting for shutdown. Test output is correlated, grouped, and + retained; required-service failures cannot leave the parent apparently healthy. +8. Terminal output, GitHub Actions groups/annotations/summary, JSON results, and + JUnit describe the same run, including skips, retries, incomplete work, and + infrastructure failures. + +## 1. What actually needs to change + +### The automation engine is already mostly generic + +**Observed:** a driver session can launch or attach to an application and enumerate +its windows without a story catalog or application-runtime hello. +`DesktopTarget` makes its story orchestrator optional, and story endpoints reject +targets without one. + +Sources: +`packages/agentic/desktop-driver/src/host/types.ts:134-184`; +`packages/agentic/desktop-driver/src/server/SessionManager.ts:78-141`; +`packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts:309-319`. + +However, **this is a generic API, not a finished generic production command**. +The existing production composition of `NativeDesktopHost` and the WebDriver +listener lives inside the Storybook server. `desktop-driver serve` currently uses +the fake target. + +Sources: +`packages/agentic/storybook-desktop/config/server.cjs:67-94`; +`packages/agentic/desktop-driver/README.md:100-104`. + +The missing piece is therefore a reusable native supervisor and test integration, +not a replacement native driver. + +### Storybook currently owns too much composition + +**Observed:** the config loader searches `storybook.config.*` and requires a +`DesktopStorybookConfig` class instance. The schema mixes story packages, addons, +and test IDs with app identity, native commands, platform settings, and WDIO +settings. `driver()` allocates Storybook, Metro, and WebDriver resources together. +The server requires the Storybook channel before embedding the native driver. + +Sources: +`packages/agentic/storybook-desktop/src/cli/loadConfig.ts:7-22`; +`packages/agentic/storybook-desktop/src/config/makeDesktopStorybookConfig.ts:74-128`; +`packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts:143-185`; +`packages/agentic/storybook-desktop/config/server.cjs:12-24,47-94`. + +The manifest also mixes generic application/helper leases with story digests, +runtime nonce, test-ID prefix, and channel/Metro ports. + +Source: +`packages/agentic/storybook-desktop/src/driver/driverManifest.ts:12-34,58-83`. + +**Proposed extraction seam:** generic target resolution, supervisor, resource +ownership, and runner connection below; Storybook discovery, channel, manifests, +navigation, and callback execution above. + +### The runtime is genuinely a Storybook runtime + +**Observed:** its bridge imports Storybook preview APIs, the app factory wraps +`getStorybookUI`, and Metro integration installs Storybook and runtime identity. + +Sources: +`packages/agentic/storybook-desktop-runtime/src/DesktopDriverBridge.tsx:1-54`; +`packages/agentic/storybook-desktop-runtime/src/createDesktopStorybookApp.tsx:1-49`; +`packages/agentic/storybook-desktop-runtime/config/metro.cjs:1-95`. + +Do not rename that package into a mandatory general-purpose runtime. An installed +native application can use native identity/window readiness, optionally followed +by a stable accessibility marker. The native helper's own authenticated handshake +still applies; it is not an application-side Storybook handshake. + +### There are two different test contracts + +| Existing consumer | Execution contract | Preserve during migration | +| --- | --- | --- | +| `apps/E2E` | WDIO local runner, Jasmine, globals/hooks/retries, grouped spec arrays, page navigation | Normal WDIO spec execution and grouping | +| Storybook callbacks | Extracted inline callbacks, source-digest validation, supervisor-created session, correlated navigation/remount, isolated `node:test` worker | Existing per-case isolation and explicit callback context | + +Sources: +`apps/E2E/wdio.conf.win32.js:15-69`; +`apps/E2E/tsconfig.json:5`; +`packages/agentic/storybook-desktop/src/testing/runWdioTests.ts:71-80,98-198`; +`packages/agentic/storybook-desktop/src/testing/worker.ts:20-69`; +`packages/agentic/storybook-desktop/config/transform-story-tests.cjs:59-81,175-213`. + +Do not make replacing the automation transport depend on replacing Jasmine, +emulating its globals inside `node:test`, or rewriting all tests as stories. + +## 2. Architecture decision + +| Approach considered | Package structure | Primary advantage | Primary cost | Decision | +| --- | --- | --- | --- | --- | +| **1. Neutral harness + Storybook integration** | Driver + new generic harness + existing Storybook tooling + optional Storybook runtime | Real dependency/release boundaries; supports interactive and test clients | One additional package and a small integration contract | **Accepted; implement this** | +| **2. Generalized all-in-one harness** | Driver + renamed/generalized tooling package with lazy Storybook subpaths + optional runtime | Fewest release units and easiest initial extraction | Storybook-only tooling dependencies and releases remain shared | Not selected | +| **3. WDIO-launcher-centric** | Driver + WDIO desktop service/harness + Storybook adapter + optional runtime | Standard runner ecosystem becomes the organizing model | Story callback semantics and interactive lifecycle need adaptation | Not selected | + +The accepted design uses the single config format, native-only application +support, and explicit platform/ownership contracts below. The other approaches +are retained only as decision history, not parallel implementation tracks. + +### Selected design: app-neutral harness plus separate Storybook integration + +**Package dependency direction:** + +```text +apps/E2E ---------------------> desktop-test ------> desktop-driver + ^ + | +apps/storybook ---> storybook-desktop + | + +---------> storybook-desktop-runtime + +storybook-desktop may also consume desktop-driver's public extension contracts. +desktop-test never imports Storybook, React, or React Native. +desktop-driver never imports desktop-test or any application framework. +``` + +`desktop-test` is the composition/lifecycle layer. `storybook-desktop` supplies an +explicitly imported integration to it; it does not run a competing supervisor. +There is no auto-discovered plugin registry and no package-name-based loading. + +The normal WDIO launcher runs standalone suites. The Storybook integration keeps +the CSF executor, but obtains resources and session lifetime management through +the same harness. + +**Decision rationale:** both an application-neutral lifecycle and an application-neutral +installation boundary become enforceable. Generic consumers do not install the +Storybook-only Babel/regexpu transforms. Storybook changes do not automatically +become generic harness releases. The existing package already declares these +transform dependencies, so a lazy subpath alone does not isolate installation. + +Source: +`packages/agentic/storybook-desktop/package.json:59-68`. + +**Risks:** maintaining integration contracts; accidentally creating duplicate +lifecycle engines; extraction changes to listener/channel startup and shutdown. +Mitigate with one resource owner and consumer-level contracts. Keep a supervised +channel child initially rather than moving everything in-process at once. + +**Migration shape:** extract the generic supervisor; prove a runtime-free app; +add standalone WDIO/Jasmine integration; reconnect Storybook through the optional +integration; migrate legacy suites by endpoint. + +### Alternative considered: generalized all-in-one harness + +Not selected: renaming `storybook-desktop` to `desktop-test` and placing Storybook +behind lazy subpaths would reduce package contracts, but retain its installation +dependencies and release coupling. Forming module boundaries inside the current +package is an acceptable extraction step, not the final architecture. + +### Alternative considered: WDIO-launcher-centric orchestration + +Not selected: making the WDIO launcher the owner of all infrastructure would fit +standalone E2E, but require extra adaptation for interactive server use and CSF +isolation. The selected harness may generate WDIO configuration and use its +reporter hooks; it does not make every operation a WDIO job or require another +authored config format. + +## 3. Package responsibilities + +| Package | Owns | Must not own | +| --- | --- | --- | +| `desktop-driver` | W3C remote end; target registration contracts; native helpers; `DesktopHost`; capabilities; clients; native assertions/artifacts; low-level WDIO command compatibility | App build commands, Metro, Storybook dependencies, Jasmine configuration | +| **New `desktop-test`** | Canonical config/loading; target and instance resolution; app/host adapters; helpers/listener lifecycle; launch/attach leases; readiness coordination; cancellation; WDIO launcher/worker service; cleanup; reporting protocol/collector, journals, result normalization, and console/Actions/JSON/JUnit renderers | Story discovery, CSF extraction, preview navigation, Storybook UI | +| `storybook-desktop` | Story config extension; discovery and digests; callback extraction/stripping; channel adapter/orchestrator; traversal; story executor; Storybook-specific diagnostic/result adapters into the harness | Another helper/server/app supervisor or independent top-level reporter | +| `storybook-desktop-runtime` | Storybook shell/theme/chrome; preview bridge; Storybook-specific Metro/Babel integration | Mandatory runtime for generic apps or ownership of host processes | + +Keep low-level WDIO native-command adaptation in `desktop-driver/wdio`; put WDIO +launcher/framework/lifecycle integration in `desktop-test`. This avoids pulling +the full WDIO runner or Jasmine into the driver. + +Keep CLI/config entry points free of peer-dependent installation assumptions. +Resolve suite frameworks from their owning workspace, and preserve the existing +requirement that CLI binaries have a stable physical installation location +(`packages/agentic/storybook-desktop/README.md:13-19`). + +Initially retain dependency-free story protocol hooks in the driver if moving +them would churn qualified native boundaries. Only register/use story-specific +client commands for targets advertising the integration. Renaming every +`preview`/`story` internal field is not a prerequisite for generic targets. + +The integration contract should be deliberately small: validated configuration, +optional service/resource descriptors, readiness/navigation behavior, and test +execution using core-provided session/resource scopes. The integration must not +independently spawn helpers, select unregistered apps, or own teardown. + +The imported integration is trusted local code. Resolved target descriptions, +capabilities, manifests, run results, and agent output remain serializable data. +Do not serialize executable adapter objects or introduce unrestricted callbacks +into remote capabilities. + +Reporting belongs inside `desktop-test`, not a fifth package or a new general +telemetry service. Driver/native layers retain narrow diagnostic callbacks and +typed results; they do not acquire GitHub Actions, WDIO-runner, or Storybook +dependencies. Section 6 defines the producer/collector/renderer boundary. + +## 4. One configuration format, with optional Storybook + +### Format and ownership + +**Proposed name:** `desktop.config.mts`, with the same schema for every consumer +and equivalent supported TypeScript/JavaScript extensions. + +- One format does **not** mean one repository-global file. +- App-root files own app lifecycle and identity exceptions. +- The `apps/E2E` file owns test selection, frameworks, hooks, retries, and readiness + expectations. It references app targets; app configs never import E2E runners. +- Target references use this same schema, not a second app-config format. +- Permit suite-local inline targets for unrelated installed applications that + have no app-owned config. `app.json` is not universally required. +- Generated native identity, instance manifests, and WDIO configs are outputs, + not additional user-authored formats. + +Use structural/schema validation rather than `instanceof`. Reject unknown core +fields and delegate extension-field validation to the explicitly imported +integration. Resolve paths relative to their defining config/module, not cwd. +Reject reference cycles and ambiguous target overrides. + +Keep these axes separate: + +| Axis | Examples | Rule | +| --- | --- | --- | +| Endpoint | `macos`, `windows`, `win32` | Test-facing identity; Win32 is not an alias for the Windows Fabric endpoint | +| Host OS | macOS, Windows | Both Windows endpoints use the Windows helper | +| Host adapter | Native installed/executable, RN Test App, REX, external attach | Determines supported lifecycle stages, not test framework | +| Renderer | Fabric/Paper, when applicable | Optional for non-RN apps; actual generated/running mode must match | +| Application integration | None, Storybook | Optional; not inferred from test runner | +| Test source | WDIO specs, CSF callbacks, existing static story plans | Determines execution/navigation contract | +| Readiness | Verified native window/marker, optional authenticated integration | `/status` alone does not prove the app is ready | + +### Illustrative configs + +These sketches describe the proposed API, not code that works today. + +```ts +// apps/storybook/desktop.config.mts +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; +import { storybook } from '@fluentui-react-native/storybook-desktop/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { host: { kind: 'react-native-test-app' } }, + windows: { host: { kind: 'react-native-test-app', renderer: 'fabric' } }, + win32: { host: { kind: 'rex' } }, + }, + integration: storybook({ + storyPackages: [ + '@fluentui-react-native/components', + '@fluentui-react-native/native-core', + ], + // Carry over current per-endpoint patterns, exclusions, and smoke checks. + }), + suites: { + stories: { kind: 'csf-wdio' }, + }, +}); +``` + +The imported Storybook factory supplies the integration explicitly; the core does +not look up a package by name. Existing platform exclusions remain exactly as +declared in `apps/storybook/storybook.config.mts`. + +The macOS renderer remains selectable per run, with Fabric as the existing +Storybook default and a Paper lane. A renderer switch requires preparation; +renderer must participate in instance/lease validation. Do not fix Fabric in +config in a way that makes the Paper lane impossible. + +```ts +// apps/fluent-tester/desktop.config.mts +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { host: { kind: 'react-native-test-app' } }, + windows: { host: { kind: 'react-native-test-app' } }, + }, +}); + +// apps/win32/desktop.config.mts uses the same schema: +// targets: { win32: { host: { kind: 'rex' } } } +// Preserve the canonical development/attach window identity, not a new title. +``` + +```ts +// apps/E2E/desktop.config.mts +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; +import { BASE_TESTPAGE } from './src/common/consts.ts'; +import { legacyDesktopSpecs } from './desktop-specs.mts'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { + use: { config: '../fluent-tester/desktop.config.mts', target: 'macos' }, + }, + windows: { + use: { config: '../fluent-tester/desktop.config.mts', target: 'windows' }, + }, + win32: { + use: { config: '../win32/desktop.config.mts', target: 'win32' }, + }, + }, + suites: { + legacy: { + kind: 'wdio', + framework: 'jasmine', + specs: legacyDesktopSpecs, + readiness: { kind: 'native', testId: BASE_TESTPAGE }, + hooks: './src/common/desktopHooks.ts', + maxInstances: 1, + specFileRetries: 1, + bail: 1, + }, + }, +}); +``` + +`desktop-specs.mts` is ordinary shared selection data, not another config format. +It preserves the current macOS selection, the Windows explicit allowlist, and +the Win32 `.win.ts` selection, including grouped arrays. Do not broaden Windows +to all matching files. Diagnose unmatched legacy globs rather than silently +reducing coverage. + +Sources: +`apps/E2E/wdio.conf.macos.js:9`; +`apps/E2E/wdio.conf.windows.js:11-25`; +`apps/E2E/wdio.conf.win32.js:17`. + +An unrelated, uninstrumented app can instead define an inline target: + +```ts +targets: { + macos: { + host: { + kind: 'installed', + application: { bundleIdentifier: 'com.example.DesktopApp' }, + ownership: 'external', + }, + readiness: { kind: 'native', testId: 'main-window-ready' }, + }, +} +``` + +That target has no `integration`, Storybook runtime, React Native manifest, +Metro service, renderer, or story-navigation requirement. Omitted lifecycle +stages do not run; explicitly requesting an unsupported stage fails clearly. + +### Identity and defaults + +Resolve identity once, then derive commands, registration, leases, runtime +identity, and runner capabilities from the same immutable target description. + +- RN Test App: use app metadata and generated build identity; do not duplicate + bundle IDs, Appx identity, component names, or display names across configs. +- REX: resolve executable, component, arguments, bundle root, cwd, and window + title once. Development launch and test attach consume the same identity. +- Native/external: accept trusted explicit identities; do not invent an + `app.json` requirement for non-RN apps. +- Use canonical app root + endpoint + renderer/profile for instance identity; + a suite config referencing another app must not change that app's identity. +- Separate stable installation identity, isolated instance identity, and unique + run nonce. Do not rewrite the identity of externally installed apps. +- Keep `app.json` as the owner of the existing Storybook test-ID prefix. +- Paths, executable selection, cwd, helper selection, and allowed launch modes + are server-owned configuration, never unrestricted WDIO capabilities. + +Preserve source-loadable config entry points before a repository build, because +Storybook's generation step loads config before compiled outputs exist. + +### Reporting configuration + +Reporting is another section of the same schema, not a separate reporter config +or per-service environment convention: + +```ts +reporting: { + console: 'auto', // GitHub Actions when GITHUB_ACTIONS=true; plain otherwise. + artifactsRoot: './artifacts/desktop', + junit: true, + summary: 'auto', // Bounded Actions summary when GITHUB_STEP_SUMMARY is present. +} +``` + +The versioned event journal and canonical run result are always retained. +`console` also accepts explicit `plain` and `github-actions` modes. Structured +CLI output mode takes precedence: stdout contains only the requested JSON result +or event stream, while human diagnostics go to stderr without Actions framing. +`--verbose` increases raw informational log detail; it must not determine whether +errors, service failures, or test outcomes are visible. + +The invoking config owns run-level reporting. Referenced app configs contribute +targets, not additional collectors/renderers or competing reporting overrides; +delegated commands inherit the invocation's reporting context. + +## 5. Lifecycle, sessions, and readiness + +### Exactly one owner per resource + +| Resource | Owner | +| --- | --- | +| Helper resolution, verified artifact, W3C listener, target registry | Generic supervisor | +| Optional Metro/channel/build processes | Generic supervisor, from selected adapter/integration descriptors | +| Normal WDIO browser session | WDIO worker; supervisor tracks association and removes orphans | +| Story callback session | Supervisor session scope; callback worker attaches to that session | +| Supervisor-launched app | Supervisor lease; driver sessions attach | +| Session-launched app | Driver session; session deletion closes it | +| External attached app | External owner; ordinary cleanup preserves it | + +**Initial application-lifetime policy:** retain the +existing supervisor-owned app and attached-session model for Storybook. Use +session-owned launch for standalone suites when the adapter supports it, or +supervisor launch plus verified attach when host tooling must launch the app. +Do not add driver-managed service-lifetime retention as part of this extraction. +That can be a later feature with its own crash/relaunch/cleanup contract. + +A worker must not call `remote()` again when WDIO already supplied `browser`. +CSF attachment likewise must not create a second competing session. Preserve +grouped spec lifecycle rather than assuming a session always corresponds to one +physical spec file. + +**Observed:** one active session per target is enforced within a service, and +physical input is serialized. That does not prevent two independent supervisors +or whole tests from stealing focus from one another. + +Sources: +`packages/agentic/desktop-driver/src/server/SessionManager.ts:84-88,201-208`. + +Default to one active interactive workload per desktop. Reject conflicting +claims on the same physical application across target aliases/supervisors. +Use separate desktops/machines for parallel physical-input qualification. + +### Native readiness is not Storybook readiness + +For ordinary applications, verify native identity and a window, then any +configured accessibility marker. Make launch/readiness timeouts configurable; +the current 120-second session startup deadline is shorter than the legacy +180-second Win32 allowance. + +Sources: +`packages/agentic/desktop-driver/src/server/SessionManager.ts:93-121`; +`apps/E2E/wdio.conf.win32.js:10-12`. + +For Storybook, preserve nonce-authenticated hello, initial preview readiness, +correlated navigation/generation, digest validation, and native story-root +verification. Do not downgrade an instrumented target to a window-only check +when its authenticated readiness fails. + +Sources: +`packages/agentic/storybook-desktop/src/driver/StorybookChannelOrchestrator.ts:174-240`; +`packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts:326-390`. + +Expose an optional readiness interface, but defer a reusable non-Storybook +runtime package until a real second instrumented app needs one. + +### Cancellation and evidence + +On interruption: stop scheduling; abort and settle in-flight work; stop workers +within a deadline; release input; delete recorded sessions; stop only owned apps +and services; dispose helpers. Preserve the original failure and report cleanup +or evidence failures separately. + +Keep artifacts under a confined root with unique +run/endpoint/renderer/worker/spec/case/attempt namespaces. Retain merged command +logs, native tree/window/source evidence, screenshots where supported, helper +identity, capabilities, skip reasons, test results, and cleanup outcome. +Lease nonces and authentication material must not become report content. + +Do not collapse the current child process into the CLI without a disposal +contract: the existing channel bootstrap owns keepalive/signal behavior and +calls `process.exit()`. + +Source: +`packages/agentic/storybook-desktop/config/server.cjs:24,96-102`. + +## 6. Reporting and delegated output + +### Research findings: why output can disappear from the combined view + +The owner's reported symptom is consistent with the following source-inspected +gaps. This research did not reproduce a specific lost-output incident or prove +that every hidden diagnostic originates from the same path. + +| Observed path | Current behavior | Consequence | +| --- | --- | --- | +| `NodeDesktopCommandRunner` | Sends child stdout/stderr to one retained file; replays on failure or in verbose mode when reporting/stopping the command | A live service can write useful output that never reaches the terminal until shutdown. Verbose is replay, not live forwarding. | +| Combined Storybook server | Forwards native stderr to its own stderr; channel and WebDriver startup messages use its stdout | Forwarding one process hop is insufficient when the outer runner captures that stream. | +| `driver()` lifecycle | Starts Metro separately, then awaits the server command; Metro completion is not raced against the server for ongoing supervision | A delegated service's failure report and the parent's actual lifetime are not one coherent health/result contract. | +| Windows/Win32 smoke wrappers | `Start-Process` redirects delegated commands to separate `.out.log`/`.err.log` files | The generic command runner cannot automatically surface those grandchildren's diagnostic files; file retention alone is not console forwarding. | +| Existing grouping | Per-runner `reportQueue` serializes textual `BEGIN`/`END` replay | Good local ordering, but neither Actions `group` commands nor a hierarchy shared across nested processes/runners. | +| Test workers and aggregate reports | Worker result files carry pass/skip/failure; console reporter output is a separate captured subprocess stream; aggregate status is based on presence of failures | Rendering, structured outcomes, and lifecycle are separate flows. All-skipped work can still yield a top-level `passed` label. | + +Source anchors: + +- `packages/agentic/storybook-desktop/src/cli/commandRunner.ts:76-145,175-198` + and its `commandRunner.test.ts:29-116` explicitly exercise success suppression, + replay-on-stop, and non-interleaved replay within one runner. +- `packages/agentic/storybook-desktop/config/server.cjs:18-31,67-99`; + `packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts:143-185,379-409`. +- `packages/agentic/storybook-desktop/config/smoke-windows.ps1:32-45`; + `packages/agentic/storybook-desktop/config/smoke-win32.ps1:41-54`. +- `packages/agentic/storybook-desktop/src/testing/worker.ts:20-67`; + `packages/agentic/storybook-desktop/src/testing/runWdioTests.ts:151-174,221-229,268-307`; + `packages/agentic/storybook-desktop/src/cli/smokeTests.ts:130-150,222-226`. + +Preserve what already works: durable command logs, bounded-memory file replay, +failure causes, explicit worker results, evidence error reporting, and one-runner +replay serialization. Replace the fragmented ownership, not just the log prefix. + +### A single reporting flow + +```text +supervisor lifecycle ---------+ +Node child services -- IPC ---+ +WDIO reporter adapter -------+--> desktop-test collector --> events.jsonl +CSF worker/result adapter ----+ | run.json +Storybook integration --------+ | junit.xml +native diagnostic callbacks -+ +--> one presentation queue +third-party stdout/stderr ----+ | + (durable per-source logs) plain / Actions / JSON +``` + +The supervisor is the sole console/Actions owner. Services and integrations emit +events or provide raw streams; they never each start their own Actions groups, +append competing summaries, or print a second top-level result. + +Use a small versioned envelope containing `schemaVersion`, `runId`, `eventId`, +`sourceId`, `sourceSequence`, `scopeId`/`parentScopeId`, timestamp, kind, and payload. The +collector adds an arrival sequence and correlation with endpoint, renderer, +service, spec/story/test, worker, and attempt as applicable. Do not correlate +events by parsing titles, choosing the most recently active test, or assuming +timestamps establish a total order across processes. + +Give propagated failures a stable `failureId` so wrapper/context events can +reference the same cause without duplicating annotations or counting failures +twice. Redact known secrets and lease/authentication material before persistence +or presentation; do not dump complete environments or capabilities into logs. + +Start with lifecycle/scope start/end, service ready/failed, diagnostic, output, +test result, artifact, and run-finished events. Validate envelopes and reject +malformed or incompatible structured producers explicitly. Preserve exception +name/message/stack/cause and aggregate children, with redaction, rather than +reducing delegated failures to an exit-code-only message. + +For owned Node children, use dedicated IPC and a terminal result/flush handshake; +do not multiplex reporting JSON into ordinary stdout. Third-party tools keep +stdout/stderr capture adapters. Native helper stdout remains its framed protocol; +route its diagnostic callback instead of treating protocol bytes as text. +The existing native transport already separates these channels: +`packages/agentic/desktop-driver/src/hosts/native/NativeHostProcess.ts:75-85`. + +During migration, a delegated CLI inherits reporting context and acts as an event +producer rather than constructing another default console renderer. PowerShell +wrappers must either return service descriptors for core-owned launch or bridge +their redirected files and terminal results into the collector. Merely printing +the path to a grandchild log does not satisfy the contract. + +### Visibility, health, and completion guarantees + +1. Emit service startup/readiness, warnings, typed errors, test start/end, and + artifact locations promptly, without waiting for service termination. + Render unstructured service stderr promptly with source/stream labels, but do + not infer that all stderr is fatal. Bulk informational output may stay in + retained logs or appear under verbose replay. +2. Observe required services for their entire owned lifetime. Unexpected exit, + an explicit fatal event, or loss of readiness/control required by active work + fails the owning phase, aborts dependent tests, and starts owned cleanup. + Do not wait indefinitely for the outer combined server to exit. +3. Finalization distinguishes process exit, stream closure, event delivery, and + sink flush. Await pipe/IPC drain and queued writes with a bounded deadline + before final output and normal exit. Prefer `process.exitCode` to abrupt + `process.exit()`. Distinguish expected stop from a crash. +4. Retain logs while they are produced, not just after a successful finish. + Stream/spool rather than buffering the full run in memory. Bound disk/queue + use and make truncation or transport loss explicit; never silently discard + fatal events or terminal test results. +5. De-duplicate forwarded events using producer identity/sequence and diagnose + missing terminal events. On worker crash or hard parent termination, retain + the available journal and mark affected work incomplete. Do not promise + exactly-once delivery through an uncatchable process kill. +6. Required journal/result/JUnit write failures are infrastructure failures. + Optional presentation failures, such as an Actions summary upload, are + reported explicitly with a fallback location. Neither may replace the + original test failure or masquerade as a passing run. + +The existing command runner resolves `completed` on `exit` +(`commandRunner.ts:193-198`). Node documents that child stdio may still be open +then; `close` occurs after process termination and stdio closure. This matters +when introducing piped/IPC aggregation. It is not, by itself, proof that today's +shared-file-descriptor implementation truncated a particular log. + +### Test results are data, not parsed reporter output + +Keep each test engine, but adapt both to the same normalized run result: + +- WDIO uses its supported reporter lifecycle hooks for suites, hooks, tests, + skips, runner failure, and retries. A reporter module is referenced in the + generated runner configuration; do not pass non-serializable reporter + constructors through the WDIO `Launcher` options. Hooks enqueue synchronously; + pending transport writes participate in `isSynchronised` and the supervisor's + bounded final drain. +- CSF uses its existing worker result contract plus supervisor outcomes for + navigation, timeout, crash, evidence, and cleanup. Preserve authenticated story + readiness and digest checks. Node `spec`/`tap`/`dot` text remains diagnostic + output, not the source of test status. +- Static story plans adapt their existing structured step/test results. Attach + console output to a test only when the engine provides that association; + otherwise retain spec/service scope rather than guessing. + +Keep stable test identity separate from attempt identity. Record pass, failure, +skip/unsupported reason, cancellation, not-run work, retry history, duration, +infrastructure errors, evidence errors, and artifact references. A retry that +passes must still expose the earlier failed attempt and flaky classification. + +Generate `run.json`, console totals, JUnit, and Actions summary from this one +result model. Empty selection is an error. An all-skipped or unsupported cohort +is not `passed`; required-cohort policy must fail or mark it incomplete, while +an explicitly optional cohort may be reported as skipped. Missing worker results, +setup/hook failures, service crashes, and cleanup failures must be representable +even when no test body ran. JUnit must include those infrastructure errors rather +than inventing a successful testcase. + +### GitHub Actions presentation + +Auto-select the Actions renderer only for `GITHUB_ACTIONS=true`, with explicit +override available. Generic `CI=true` is not enough to print GitHub commands. + +- Use actual `::group::`/`::endgroup::` commands, not textual `BEGIN`/`END`. + Default to one completed spec/group for WDIO and one story/callback worker + group for CSF, labeled with endpoint, renderer, source, and attempt. +- Emit short live progress/diagnostics outside groups; replay each completed + group's captured output through one serialized writer. A long-lived Metro or + channel service must not hold a group open for the entire run. Do not nest + groups across concurrent producers or let unrelated service output appear + under another test's group. +- Close groups in `finally` and during bounded failure/cancellation flush. + Emit one attributed annotation per failure identity, with verified + repository-relative file/line when known; preserve the original assertion or + startup failure plus distinct cleanup/evidence failures without duplicating + it at every wrapper layer. +- Escape annotation properties and data. Protect raw child output from being + interpreted as runner workflow commands; when replaying raw batches, use the + documented random-token `stop-commands`/resume mechanism under the same writer + lock, then issue trusted annotations/groups outside that protected interval. + Do not rely on a cosmetic prefix as command isolation. +- Write a bounded `GITHUB_STEP_SUMMARY` section with revision/target/renderer, + selected/executed/result counts, unsupported reasons, retries/flakes, + infrastructure outcome, and artifact references. Do not paste full logs. + GitHub caps each step summary at 1 MiB and displays at most 20 per job. +- Keep `if: always()` artifact upload for journals, canonical results, JUnit, + raw service/command logs, and evidence. Use unique run/endpoint/renderer/shard + artifact names. A filesystem path is not a clickable Actions artifact URL; + the upload step can add the real artifact URL to a later summary. + +Current Storybook jobs already upload platform and command logs with `always()`; +extend that behavior rather than replacing it: +`.github/workflows/pr.yml:237-245,463-474,555-564`. +Rendering annotations or JUnit does not itself publish a GitHub check or alter +job status; the harness still returns the correct exit status, and any additional +check publisher is an explicit workflow choice, not a new token requirement. + +### Reuse and implementation boundary + +The repository already uses `@rnx-kit/reporter` in +`scripts/src/utils/getReporter.ts:1-13` and declares it in +`scripts/package.json:68`; `yarn.lock:9609-9611` resolves version 0.2.0. +Its installed README documents task hierarchies, timing/error events through +Node diagnostics channels, custom writers, and merged outputs. + +Reuse those in-process logging/task/writer facilities where they satisfy the +contract, behind `desktop-test`'s reporting adapter; declare a direct dependency +when implementing rather than importing the scripts workspace. Do not assume +process-local diagnostics channels propagate across child processes or provide +the run/result/IPC protocol. Keep the latter owned and tested by the harness. +No new telemetry broker, reporter plugin registry, or standalone reporting +package is required. + +### Reporting acceptance gates + +Use deterministic fake services/workers before native qualification: + +| Fixture | Required result | +| --- | --- | +| Long-running service writes a diagnostic after readiness | Source-attributed diagnostic becomes visible before stop, without verbose | +| Metro/helper/channel fails while outer server remains alive | Parent fails the correct phase, aborts dependent work, and reports the original cause | +| Direct child and grandchild interleave stdout/stderr, partial UTF-8, and late output | Durable captures are complete; source/stream metadata is preserved; only collector arrival order is claimed across streams | +| Two producers finish together | Each Actions group is contiguous/balanced; unrelated service output is not attributed to a test | +| Child prints workflow-command-shaped text | No unauthorized annotation/group/runner command is executed | +| Worker fails, times out, crashes, or omits a result; setup hook fails | Same non-passing outcome/cause appears in console, JSON, JUnit, and summary | +| All tests skip, capability is unsupported, or selection is empty | No success-shaped `passed` aggregate; required/optional policy is explicit | +| Retry passes after failure | Both attempts and flaky status survive; totals do not double-count final tests | +| Slow sink, write failure, cancellation, or abrupt service exit | Bounded completion, explicit reporting failure/incomplete state, retained evidence, no leaked owned resources | +| Structured CLI mode under Actions | Machine stdout remains parseable and free of progress, ANSI, or group commands | + +Then exercise the same adapters on macOS, Windows, and Win32, including the +transitional PowerShell path. Test the emitted Actions command stream locally +and confirm actual grouping/summary/artifact behavior in an Actions lane. +This is a phase-1/2 prerequisite, not reporting polish deferred until retirement. + +### External reporting references + +Research checked October 21, 2026: + +- [GitHub Actions workflow commands](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands): + groups, annotations, command isolation, summaries, and step-summary limits. +- [Node child process `close`](https://nodejs.org/api/child_process.html#event-close) + and [`exit`](https://nodejs.org/api/child_process.html#event-exit): + process termination versus stdio closure. +- [WebdriverIO custom reporters](https://webdriver.io/docs/customreporter/): + lifecycle events, synchronous event callbacks, `isSynchronised` for pending + reporter work, and `Launcher` reporter serialization constraints. Implement + against the repository's pinned WDIO version, not unverified newer hooks. + +## 7. Platform/application boundaries + +| Endpoint | Existing legacy app | Required treatment | +| --- | --- | --- | +| macOS | `apps/fluent-tester`, RN Test App | Verify actual renderer/build identity; do not infer runtime mode from a Pod entry | +| Windows | `apps/fluent-tester`, RN Test App | Verify generated Fabric setting and packaged identity; do not presume UWP from old docs | +| Win32 | `apps/win32`, prebuilt REX/Paper | Windows helper with `win32` endpoint; preserve cwd/args/title and bundle-driven launch; native app build remains unsupported | +| Storybook macOS | `apps/storybook` | Preserve Fabric and Paper lanes and prep-on-renderer-switch | +| Storybook Windows/Win32 | `apps/storybook` | Preserve distinct identities, platform exclusions, host-specific chrome and launch behavior | + +**Important reciprocal-review correction:** omitting `--use-fabric` does not +establish that FluentTester Windows is UWP/Paper. The pinned RN Test App 5.3.3 +generator defaults to Fabric for RNW 0.80+, and the resolved RNW is 0.81.35. +Fresh generation is therefore **inferred** to be Fabric. Inspect generated +`UseFabric` and the actual running app before qualification; no separate renderer +migration is justified by the omitted flag alone. + +Sources: +`yarn.lock:21897-21899,21947`; +installed `react-native-test-app/windows/project.mjs:481-493` and +`windows/app.mjs:358-385`; +`.github/workflows/pr.yml:376-392`. +Installed dependency paths are version-specific evidence, not permanent repo +paths. + +Generic Windows attach also needs stronger identity than a title alone. Current +unleased attach matches a unique exact title across processes; old E2E attaches +by HWND. Prefer a verified owner lease or executable/AUMID-qualified identity; +neither title-only selection nor "first ReactApp process" is a safe generic +replacement. Validate PID reuse, duplicate titles, and packaged ownership. + +Sources: +`packages/agentic/desktop-driver/native/windows/src/application.cpp:965-985`; +`apps/E2E/wdio.conf.windows.js:36`; +`apps/E2E/launchE2EWindows.ps1:14-27,69-70`. + +## 8. Compatibility work: package extraction alone will not migrate E2E + +The WebDriver wire protocol is not an Appium semantic compatibility guarantee. +Inventory actual selected tests and their command paths before changing them. + +| Area | Observed mismatch | Recommended handling | +| --- | --- | --- | +| **WDIO visibility/focus** | With WDIO 9.24 and current desktop capabilities, `isDisplayed` takes the browser JavaScript path; `isFocused` also executes JavaScript. The server rejects `execute`. `waitForDisplayed` is used during initial page readiness. | Central native-command compatibility installation for `remote`, `attach`, and WDIO-managed workers. Route visibility/focus to supported native queries. Test waits and assertion consumers. Do not spoof Appium identity or enable arbitrary execution. | +| Selectors | macOS hooks use XCUI XPath; driver supports accessibility ID, role/tag, accessible name/link, and text, not XPath | Replace accidental platform-specific hooks with supported selectors/window behavior. Implement a new selector only when required by real test intent. | +| Attributes/state | Legacy names such as `Name`, `ControlType`, `HasKeyboardFocus`, and `Toggle.ToggleState` differ from normalized properties and types; unknown attributes return `null` | Migrate a shared page-object adapter with explicit mappings. Add missing justified native state contracts; preserve provider-specific accessibility assertions rather than normalizing them away. | +| Checked vs selected | Legacy mode switch uses `isSelected`; UIA toggle checkedness is distinct from SelectionItem state | Change toggle assertions to `checked`. Do not globally redefine `selected` as checkedness. | +| Keyboard | `keys.join()` introduces commas; typed units press/release modifiers rather than preserving a chord; disabled-element send-keys may fail focus validation | Replace chords with explicit W3C key actions and releases; test cancellation/repetition/Unicode. Decide and document element-send-keys modifier semantics separately. | +| Window sizing | Legacy setup calls `maximizeWindow`; driver routes currently support handles/switch/close/rect, not maximize | Implement and qualify maximize only if needed, or explicitly change setup intent. Setting a rectangle is not automatically maximize. | +| Popups/focus | Search is current-window scoped; legacy page objects often assume root-wide lookup or treat extra windows as assertion failures | Preserve explicit window scope; update tests to identify popup ownership and restore focus. Measure before adding optional app-wide lookup. | +| Desktop scrolling | Windows navigation can reach a helper whose default branch emits Android UiScrollable selectors | Separate desktop navigation/scrolling from mobile helpers. Keep mobile paths and dependencies. | +| Runner grouping | Legacy hooks, globals, retries, Jasmine matchers, and suite state differ from story callbacks | Preserve WDIO/Jasmine and grouped arrays; change transport and semantics incrementally. | +| REX startup | Legacy config supplies working directory; native application descriptor lacks it | Add trusted adapter/server-owned cwd or supervisor launch with an exact attach lease; qualify actual argument/cwd requirements. | + +Key source anchors: + +- Visibility: `packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.ts:113-148`; + `packages/agentic/desktop-driver/src/protocol/capabilities.ts:96-106`; + `packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts:740-741`; + `apps/E2E/src/common/BasePage.ts:172-186,298-300`. + Both reviewers also inspected installed WDIO 9.24's `isDisplayed`/`isFocused` + implementations and native/mobile session detection. +- Selectors/window routes: + `apps/E2E/wdio.conf.macos.js:115-132`; + `packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts:449-496,649-705`. +- State: + `apps/E2E/src/common/consts.ts:32-43,77-84`; + `apps/E2E/src/common/BasePage.ts:100-126`; + `packages/agentic/desktop-driver/native/windows/src/automation.cpp:586-625`. +- Input: + `apps/E2E/src/common/BasePage.ts:200-202`; + `packages/agentic/desktop-driver/native/windows/src/input.cpp:447-466`; + `packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Input.swift:189-203`. +- Windows/mobile and popup assumptions: + `apps/E2E/src/common/BasePage.ts:79-89,211-294,386-411`; + `apps/E2E/wdio.conf.win32.js:166-178`. +- Runner/REX: + `apps/E2E/src/ButtonV1/specs/ButtonV1.spec.win.ts:5-24,56-103`; + `apps/E2E/wdio.conf.win32.js:4-7,25-28`; + `packages/agentic/desktop-driver/src/native/types.ts:58-66`. + +Not every mismatch requires a new driver feature. Prioritize the selected pilot's +real requirements; retain explicit unsupported results rather than invented +values or broad compatibility shims. + +## 9. Phased delivery and gates + +| Phase | Owning surface | Deliverable | Exit gate | +| --- | --- | --- | --- | +| **0. Baseline and probes** | E2E + native driver + current supervisor | Inventory exact selected cases, commands, state assertions, grouping, existing failures, renderer, host identity, input policy, and every delegated output path | Recorded baseline and blocker list per endpoint; confirm Windows generated renderer and capture a reporting-gap fixture | +| **1. Generic boundary and reporting spine** | `desktop-test` + driver | Canonical schema/resolver, controlled native target composition, adapters, ownership/readiness, real `serve`, event collector/journal, live diagnostics, one renderer and flush contract | Plain app works without Storybook; fake-service tests prove child/grandchild forwarding, required-service failure, bounded shutdown, and isolated Actions commands/groups | +| **2. WDIO compatibility/pilot** | Driver `/wdio` + harness + E2E | WDIO/Jasmine launcher and reporter adapter, visibility fixes, shared page-object adaptation, representative specs, canonical JSON/JUnit/summary | Pilot passes with required capabilities; grouped output/result parity and retry/skip/hook/crash contracts pass; no blocked sessions/input/app leaks | +| **3. Storybook as consumer** | Storybook tooling/runtime/app | Same schema; optional integration; one supervisor and collector; existing traversal/CSF lifecycle and normalized results | macOS Fabric/Paper, Windows, Win32 behavior and live diagnostics preserved, including nested PowerShell paths; no weakened readiness or exclusions | +| **4. Comparative migration** | E2E apps + CI | Old/new lanes on equivalent app builds, explicit test mapping, semantic comparisons, Actions summaries and always-uploaded artifacts | Each endpoint meets parity/reliability gates; real Actions grouping, annotations, summary, and artifact references verified | +| **5. Retirement** | E2E + CI + docs | Remove obsolete desktop transport configs/scripts/dependencies | No remaining desktop consumer depends on old transport; mobile still works | + +Recommended pilot: Win32 Button plus Checkbox/Switch state, a keyboard/focus case, +a scrolling case, and a popup/window case. Then establish macOS and Windows parity +with their actual selected cohorts. Do not infer macOS coverage from a richer +Win32 suite or count a navigation-only case as a behavioral assertion. + +For each baseline unit `(endpoint, spec/group, test title)`, record migrated, +rewritten-with-mapping, or explicitly waived status. Compare assertion intent, +not just counts. A waiver requires a reviewed reason; reduced functionality or +missing capabilities must not silently become success. + +Use the existing driver qualification target: **at least 100 representative +executions per endpoint over 14 days, at least 99% infrastructure success, and +zero process/input leaks**, alongside semantic parity. A smaller pilot is not a +replacement for that gate. + +Source: `packages/agentic/desktop-driver/PLAN.md:112-115`. + +### Validation and CI ownership + +| Layer | Required evidence | +| --- | --- | +| Driver | W3C/fake-host contracts; native WDIO visibility/focus/waits; unsupported state; stale elements; target reservation and cancellation | +| Native helpers | Matching-OS builds/handshakes; verified artifacts; launch/attach identity; input/state/capture; crash and orphan recovery | +| Generic harness | No-Storybook consumer installation/import test; same-schema references; config-relative paths; optional stages; session/worker cleanup; external-app preservation | +| WDIO integration | Real Jasmine/global/hook/group/retry behavior; no duplicate sessions; worker crash/timeout/interrupt; reporters and artifact failure paths | +| Storybook integration/runtime | Existing extraction/digests/static plans; authenticated readiness and stale-message rejection; navigation/remount; worker failure/timeout contracts; renderer switching | +| Reporting | Section 6 fixtures; cross-process attribution; live service diagnostics; required-service health; complete drain/flush; result parity across sinks; balanced Actions groups; safe raw replay; incomplete/all-skipped handling | +| App lanes | Existing FluentTester macOS/Windows and REX Win32 builds plus selected legacy assertions; separate Storybook platform/renderer regressions | +| Interactive qualification | Physical input, focus, popup windows, wheel/DPI, capture/occlusion, cross-supervisor contention, repeated reliability and zero leaks | + +Reuse `.github/actions/setup-desktop-driver`; its workspace input is already +general enough for non-Storybook consumers. Keep explicit helper build/doctor and +prebuilt-only resolution during test execution. Do not compile native helpers +during package installation or download them opportunistically. + +Source: +`.github/actions/setup-desktop-driver/action.yml:36-53,102-120,134-160`. + +**Hosted-input gate:** current Storybook hosted lanes disable physical click on +macOS and keyboard/click/wheel on Windows. These lanes cannot qualify equivalent +legacy physical-input behavior under that policy. Recommend retaining hosted +build/traversal/non-input checks and qualifying input on managed interactive +desktops. Any decision to enable hosted physical input needs an explicit policy +and evidence; accessibility activation is not proof of physical clicking. + +Source: `.github/workflows/pr.yml:214-220,432-438,544-549`. + +During implementation, run each changed workspace's declared checks, matching-OS +native qualification where applicable, then repository-required +`yarn clean-lage-cache` followed by `yarn lage buildci`. JavaScript contracts and +bundles do not substitute for native app evidence. + +### Retirement details + +Remove desktop `wdio.conf.{macos,windows,win32}.js`, redundant desktop launch +scripts, WinAppDriver setup, and desktop-specific Appium driver dependencies only +after the corresponding endpoint gates pass. + +Keep the Appium service and mobile driver/config dependencies needed by iOS and +Android. Preserve `@fluentui-react-native/e2e-testing` constant exports consumed +by `tester-core`; removing desktop automation must not break test IDs imported +by application pages. + +Sources: +`apps/E2E/wdio.conf.android.js:40-46`; +`apps/E2E/wdio.conf.ios.js:38-44`; +`apps/tester-core/package.json:45`. + +Also update package exports/dependencies, root TS project references, changesets, +CI paths and names, app scripts, Metro/Babel adapters, ignores, documentation, +agent maps, and scoped guidance in the same implementation sequence. There is no +need to combine a wholesale directory relocation with the architectural split. + +## 10. Remaining implementation decisions and risks + +| Decision/risk | Plan/default | Evidence needed to change it | +| --- | --- | --- | +| Package architecture | **Settled: Option 1**, neutral harness plus separate Storybook integration | A new explicit owner decision; not an implicit implementation simplification | +| App lifetime across story sessions | Existing supervisor lease + attach | A separately designed driver keepalive contract with crash/relaunch/teardown coverage | +| Element-send-keys modifiers | Migrate chords to actions now; specify sticky-modifier support or explicit rejection separately | W3C/native conformance cases; no silent press/release mismatch | +| Popup lookup | Explicit identified-window switching | Measured need for opt-in app-wide scope without ambiguous lookup or stale references | +| Unknown attributes/provider state | Keep protocol behavior explicit; migrate supported semantic mappings and report unsupported state | A justified namespaced provider-state or new normalized-property contract | +| FluentTester Windows renderer | Verify freshly generated `UseFabric`; do not assume UWP | Generated project plus actual launched runtime/build evidence | +| Windows attach identity | Exact verified lease or strong executable/AUMID identity | Duplicate-title, PID-reuse, package-collision, and external-app preservation tests | +| Generic runtime | None initially | A real second instrumented consumer needing authenticated JS/render milestones | +| Physical input on hosted CI | Do not count disabled-input lanes as qualification | Owner-approved policy plus repeatable native input evidence | +| Reporting | One collector/renderer in the harness, typed producer adapters, durable results and Actions groups | Implementation fixtures close transport/flush details; logging-library choice must preserve the reporting contract | + +The architecture decision is closed. The remaining rows are implementation +contracts or qualification gates, not competing package plans. + +## 11. Research provenance and reciprocal-review reconciliation + +| Topic | Initial disagreement or omission | Reconciled plan | +| --- | --- | --- | +| Packaging | Opus preferred three packages; Astra preferred four | Both now prefer four because lazy subpaths do not isolate installed dependencies/releases | +| WDIO native behavior | Astra initially missed visibility/focus JavaScript execution | Both verified it; command compatibility is an early migration blocker | +| Windows renderer | Opus inferred UWP from omitted flag; Astra initially treated it as unknown | Both checked the pinned generator and retracted the UWP inference; verify fresh generated/running mode | +| Config ownership | Astra's initial app configs imported E2E suite definitions | Test-to-app dependency direction; same-format app targets referenced by the suite config | +| Runtime extraction | Astra floated a generic runtime entry point | Defer it; ordinary native apps need no runtime | +| Selected vs checked | Opus proposed a driver fallback | Withdrawn; adapt toggle tests to checked semantics | +| Windows scrolling | Opus initially considered mobile branches unreachable on desktop | Corrected: Windows can enter Android-specific scrolling; separate the paths | +| WDIO-centric alternative | Opus's initial alternative assumed two authored config formats | Strengthened Option 3 preserves the single canonical format with generated WDIO config | +| App keepalive | Driver-retained service lifetime vs supervisor ownership | Preserve current leases initially; no incidental driver lifetime redesign | +| Qualification | One initial proposal suggested a smaller run window | Existing 100-run/14-day gate remains authoritative | + +**Provenance:** local artifacts `opus-driver-initial.md`, +`astra-driver-initial.md`, `opus-driver-cross-review.md`, and +`astra-driver-cross-review.md`. This plan incorporates both complete reciprocal +reviews, not just their displayed summaries. + +**Decision update:** the owner accepted Option 1 on October 21, 2026, and requested +formal reporting research. Section 6 is follow-up source/documentation research, +not a claim that both original models re-reviewed the new reporting design. + +**Validation status:** repository source, relevant installed dependency +implementations, tests, CI definitions, and upstream reporting documentation +were inspected. Application launches, builds, native tests, reporting fixtures, +and parity/reliability runs were **not run**; no implementation, +incident-reproduction, or native-qualification claim is made. From 238db96af1014ef6a9d035628447ec24817271dc Mon Sep 17 00:00:00 2001 From: Jason Morse Date: Wed, 21 Oct 2026 21:56:50 -0700 Subject: [PATCH 2/8] refactor driver and validate on windows and win32 --- .changeset/neutral-desktop-harness.md | 26 + .github/workflows/pr.yml | 16 +- .gitignore | 1 + apps/E2E/README.md | 54 ++ apps/E2E/desktop-adapters.test.cjs | 86 +++ apps/E2E/desktop-specs.mts | 20 + apps/E2E/desktop.config.mts | 51 ++ apps/E2E/package.json | 4 + .../Callout/pages/CalloutPageObject.win.ts | 2 +- .../pages/CheckboxLegacyPageObject.ts | 2 +- .../CheckboxV1/pages/CheckboxV1PageObject.ts | 2 +- .../FocusZone/pages/FocusZonePageObject.ts | 6 +- apps/E2E/src/Switch/pages/SwitchPageObject.ts | 2 +- apps/E2E/src/common/BasePage.ts | 65 +- apps/E2E/src/common/desktopHooks.ts | 44 ++ apps/E2E/src/common/nativeDesktop.ts | 111 ++++ apps/E2E/tsconfig.json | 6 + apps/E2E/wdio.conf.macos.js | 3 +- apps/E2E/wdio.conf.win32.js | 3 +- apps/E2E/wdio.conf.windows.js | 21 +- apps/fluent-tester/desktop.config.mts | 10 + apps/fluent-tester/package.json | 2 + apps/fluent-tester/tsconfig.json | 3 + apps/storybook/AGENTS.md | 23 +- apps/storybook/README.md | 14 +- apps/storybook/agent-map.yaml | 12 +- apps/storybook/desktop.config.mts | 69 +++ apps/storybook/package.json | 2 + apps/storybook/src/main.ts | 4 +- apps/storybook/storybook.config.mts | 64 -- apps/storybook/tsconfig.json | 5 +- apps/win32/desktop.config.mts | 13 + apps/win32/package.json | 2 + apps/win32/tsconfig.json | 3 + driver-next-plan.md | 288 +++++---- knip.config.ts | 2 +- packages/agentic/desktop-driver/README.md | 5 + .../agentic/desktop-driver/native/PROTOCOL.md | 13 + .../desktop-driver/native/macos/README.md | 7 + .../DesktopDriverHost/Application.swift | 42 +- .../desktop-driver/native/windows/README.md | 13 +- .../native/windows/src/application.cpp | 50 +- .../native/windows/src/automation.cpp | 3 + .../native/windows/src/automation.h | 1 + .../native/windows/src/selftest.cpp | 30 + .../references/native-helpers.md | 5 + .../desktop-driver/references/protocol.md | 15 + .../desktop-driver/references/service.md | 43 +- .../references/test-integration.md | 13 +- .../desktop-driver/src/authoring/results.ts | 2 +- .../src/authoring/storyTests.test.ts | 16 + .../src/authoring/storyTests.ts | 3 + .../agentic/desktop-driver/src/host/types.ts | 6 +- .../src/hosts/fake/FakeDesktopHost.ts | 13 +- .../src/hosts/native/NativeDesktopHost.ts | 9 +- .../src/native/nativeDriver.test.ts | 21 + .../desktop-driver/src/native/nativeDriver.ts | 47 +- .../desktop-driver/src/native/packageRoot.cjs | 2 + .../desktop-driver/src/native/types.ts | 1 + .../src/protocol/capabilities.ts | 3 +- .../src/runner/StoryTestRunner.test.ts | 10 +- .../src/runner/StoryTestRunner.ts | 9 +- .../src/server/SessionManager.ts | 7 +- .../src/server/TargetRegistry.ts | 9 + .../src/server/createDesktopDriverServer.ts | 6 + .../src/server/desktopDriver.test.ts | 43 +- .../src/wdio/DesktopWebdriver.test.ts | 11 + .../src/wdio/DesktopWebdriver.ts | 61 +- .../agentic/desktop-driver/src/wdio/index.ts | 7 +- .../src/wdio/wdioRunner.contract.cjs | 7 +- packages/agentic/desktop-test/README.md | 172 +++++ packages/agentic/desktop-test/config/cli.cjs | 8 + .../agentic/desktop-test/config/launcher.cjs | 36 ++ .../agentic/desktop-test/config/service.cjs | 10 + packages/agentic/desktop-test/jest.config.cjs | 9 + packages/agentic/desktop-test/package.json | 74 +++ packages/agentic/desktop-test/src/adapters.ts | 152 +++++ packages/agentic/desktop-test/src/cli.ts | 112 ++++ packages/agentic/desktop-test/src/commands.ts | 325 ++++++++++ packages/agentic/desktop-test/src/config.ts | 402 ++++++++++++ .../desktop-test/src/contracts.contract.cjs | 586 ++++++++++++++++++ .../desktop-test/src/contracts.test.ts | 50 ++ packages/agentic/desktop-test/src/index.ts | 7 + .../agentic/desktop-test/src/integration.ts | 22 + .../agentic/desktop-test/src/ownedService.ts | 71 +++ .../desktop-test/src/reporting/collector.ts | 544 ++++++++++++++++ .../desktop-test/src/reporting/index.ts | 19 + .../desktop-test/src/reporting/protocol.ts | 255 ++++++++ .../desktop-test/src/reporting/results.ts | 166 +++++ .../desktop-test/src/reporting/session.ts | 131 ++++ .../desktop-test/src/reporting/transport.ts | 363 +++++++++++ packages/agentic/desktop-test/src/services.ts | 115 ++++ .../agentic/desktop-test/src/supervisor.ts | 264 ++++++++ .../desktop-test/src/wdio/configuration.ts | 130 ++++ .../agentic/desktop-test/src/wdio/identity.ts | 8 + .../agentic/desktop-test/src/wdio/index.ts | 3 + .../agentic/desktop-test/src/wdio/reporter.ts | 161 +++++ packages/agentic/desktop-test/src/wdio/run.ts | 83 +++ packages/agentic/desktop-test/tsconfig.json | 12 + packages/agentic/storybook-desktop/README.md | 148 +++-- .../config/owned-services.ps1 | 78 +++ .../storybook-desktop/config/server.cjs | 195 +++--- .../storybook-desktop/config/smoke-win32.ps1 | 52 +- .../config/smoke-windows.ps1 | 50 +- .../config/storybook-control.cjs | 88 ++- .../agentic/storybook-desktop/jest.config.cjs | 1 + .../agentic/storybook-desktop/package.json | 2 +- .../src/cli/DesktopStorybookCli.test.ts | 28 +- .../src/cli/DesktopStorybookCli.ts | 163 +++-- .../storybook-desktop/src/cli/README.md | 12 +- .../src/cli/commandRunner.test.ts | 143 ++--- .../src/cli/commandRunner.ts | 271 +------- .../src/cli/createDesktopStorybookCommand.ts | 61 +- .../storybook-desktop/src/cli/loadConfig.ts | 38 +- .../src/cli/smokeTests.test.ts | 8 +- .../storybook-desktop/src/cli/smokeTests.ts | 78 ++- .../storybook-desktop/src/config/index.ts | 2 + .../src/config/integration.ts | 55 ++ .../src/config/makeDesktopStorybookConfig.ts | 31 +- .../src/config/sourceConfig.test.ts | 40 +- .../src/driver/driverManifest.ts | 5 +- .../src/driver/serverIntegration.test.ts | 142 ++++- .../src/testing/runWdioTests.ts | 145 ++++- .../src/testing/wdioRunner.contract.cjs | 28 +- .../storybook-desktop/src/testing/worker.ts | 32 +- .../agentic/storybook-desktop/tsconfig.json | 3 + tsconfig.json | 3 + yarn.lock | 29 +- 128 files changed, 6731 insertions(+), 1064 deletions(-) create mode 100644 .changeset/neutral-desktop-harness.md create mode 100644 apps/E2E/desktop-adapters.test.cjs create mode 100644 apps/E2E/desktop-specs.mts create mode 100644 apps/E2E/desktop.config.mts create mode 100644 apps/E2E/src/common/desktopHooks.ts create mode 100644 apps/E2E/src/common/nativeDesktop.ts create mode 100644 apps/fluent-tester/desktop.config.mts create mode 100644 apps/storybook/desktop.config.mts delete mode 100644 apps/storybook/storybook.config.mts create mode 100644 apps/win32/desktop.config.mts create mode 100644 packages/agentic/desktop-driver/src/native/packageRoot.cjs create mode 100644 packages/agentic/desktop-test/README.md create mode 100644 packages/agentic/desktop-test/config/cli.cjs create mode 100644 packages/agentic/desktop-test/config/launcher.cjs create mode 100644 packages/agentic/desktop-test/config/service.cjs create mode 100644 packages/agentic/desktop-test/jest.config.cjs create mode 100644 packages/agentic/desktop-test/package.json create mode 100644 packages/agentic/desktop-test/src/adapters.ts create mode 100644 packages/agentic/desktop-test/src/cli.ts create mode 100644 packages/agentic/desktop-test/src/commands.ts create mode 100644 packages/agentic/desktop-test/src/config.ts create mode 100644 packages/agentic/desktop-test/src/contracts.contract.cjs create mode 100644 packages/agentic/desktop-test/src/contracts.test.ts create mode 100644 packages/agentic/desktop-test/src/index.ts create mode 100644 packages/agentic/desktop-test/src/integration.ts create mode 100644 packages/agentic/desktop-test/src/ownedService.ts create mode 100644 packages/agentic/desktop-test/src/reporting/collector.ts create mode 100644 packages/agentic/desktop-test/src/reporting/index.ts create mode 100644 packages/agentic/desktop-test/src/reporting/protocol.ts create mode 100644 packages/agentic/desktop-test/src/reporting/results.ts create mode 100644 packages/agentic/desktop-test/src/reporting/session.ts create mode 100644 packages/agentic/desktop-test/src/reporting/transport.ts create mode 100644 packages/agentic/desktop-test/src/services.ts create mode 100644 packages/agentic/desktop-test/src/supervisor.ts create mode 100644 packages/agentic/desktop-test/src/wdio/configuration.ts create mode 100644 packages/agentic/desktop-test/src/wdio/identity.ts create mode 100644 packages/agentic/desktop-test/src/wdio/index.ts create mode 100644 packages/agentic/desktop-test/src/wdio/reporter.ts create mode 100644 packages/agentic/desktop-test/src/wdio/run.ts create mode 100644 packages/agentic/desktop-test/tsconfig.json create mode 100644 packages/agentic/storybook-desktop/config/owned-services.ps1 create mode 100644 packages/agentic/storybook-desktop/src/config/integration.ts diff --git a/.changeset/neutral-desktop-harness.md b/.changeset/neutral-desktop-harness.md new file mode 100644 index 0000000000..b047894a16 --- /dev/null +++ b/.changeset/neutral-desktop-harness.md @@ -0,0 +1,26 @@ +--- +'@fluentui-react-native/desktop-test': minor +'@fluentui-react-native/desktop-driver': minor +'@fluentui-react-native/storybook-desktop': minor +'@fluentui-react-native/tester': patch +--- + +Add an application-neutral desktop-test harness with canonical app/suite +configuration, native lifecycle ownership, standalone WDIO/Jasmine execution, +and shared event, JSON, JUnit, console, and GitHub Actions reporting. + +Make Storybook an optional integration while preserving isolated callbacks, +authenticated readiness, platform exclusions, and owned PowerShell services. +Bind native visibility/focus to existing WDIO browsers, retain delayed-render +waits, strengthen Windows attach identity, and forward trusted launch timeouts +and unpackaged working directories. + +Expose authoritative UI Automation required-form state, preserving unsupported +results on other providers. Adapt native checkbox/switch reads to checked state +without changing selected semantics or legacy Appium behavior. + +Expose the FluentTester application's canonical desktop targets for referenced +standalone suites without importing test or Storybook tooling into the app. + +Legacy desktop Appium and mobile lanes remain until native semantic parity and +multi-day reliability qualification are complete. diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index f479f0396f..e3b37a0eab 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -238,10 +238,10 @@ jobs: if: ${{ always() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: Storybook_macos_Dump + name: Storybook_macos_fabric_${{ github.run_id }}_${{ github.run_attempt }} path: | apps/storybook/artifacts/macos - apps/storybook/artifacts/storybook-commands + apps/storybook/artifacts/desktop ${{ steps.desktop-driver.outputs.doctor-path }} macos-paper-storybook: @@ -289,10 +289,10 @@ jobs: if: ${{ always() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: Storybook_macos_Paper_Dump + name: Storybook_macos_paper_${{ github.run_id }}_${{ github.run_attempt }} path: | apps/storybook/artifacts/macos - apps/storybook/artifacts/storybook-commands + apps/storybook/artifacts/desktop ${{ steps.desktop-driver.outputs.doctor-path }} ios: @@ -464,10 +464,10 @@ jobs: if: ${{ always() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: Storybook_windows_Dump + name: Storybook_windows_fabric_${{ github.run_id }}_${{ github.run_attempt }} path: | apps/storybook/artifacts/windows - apps/storybook/artifacts/storybook-commands + apps/storybook/artifacts/desktop ${{ steps.desktop-driver.outputs.doctor-path }} ${{ steps.desktop-driver.outputs.cache-root }}/v1/artifacts/**/build.log ${{ steps.desktop-driver.outputs.cache-root }}/v1/diagnostics/** @@ -556,10 +556,10 @@ jobs: if: ${{ always() }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: Storybook_win32_Dump + name: Storybook_win32_paper_${{ github.run_id }}_${{ github.run_attempt }} path: | apps/storybook/artifacts/win32 - apps/storybook/artifacts/storybook-commands + apps/storybook/artifacts/desktop ${{ steps.desktop-driver.outputs.doctor-path }} ${{ steps.desktop-driver.outputs.cache-root }}/v1/artifacts/**/build.log ${{ steps.desktop-driver.outputs.cache-root }}/v1/diagnostics/** diff --git a/.gitignore b/.gitignore index d0ff6b78de..48a56a740a 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ yarn-error.log* **/errorShots/ **/allure-results/ **/allure-report/ +**/artifacts/desktop/ # Runtime data pids diff --git a/apps/E2E/README.md b/apps/E2E/README.md index fde98703f8..c901113e25 100644 --- a/apps/E2E/README.md +++ b/apps/E2E/README.md @@ -1,5 +1,59 @@ # E2E Testing Overview +## Native desktop migration + +`desktop.config.mts` adds the application-neutral +[`desktop-test`](../../packages/agentic/desktop-test/README.md) harness alongside +the existing Appium runners. It references canonical app targets in +`apps/fluent-tester` and `apps/win32`; applications do not import E2E suites or +install Storybook. + +From the repository root, after the normal repository build: + +```sh +yarn workspace @fluentui-react-native/tester-win32 desktop-test --target win32 build-driver +yarn workspace @fluentui-react-native/tester-win32 desktop-test --target win32 bundle +yarn workspace @fluentui-react-native/e2e-testing desktop-test --target win32 test --suite pilot --list +yarn workspace @fluentui-react-native/e2e-testing desktop-test --target win32 test --suite pilot +``` + +The pilot deliberately starts small: Pressable navigation on Windows/Win32 and +TextLegacy on macOS. It is not the behavioral parity gate. `legacy` retains the +exact grouped selection shared with the old configs in `desktop-specs.mts`: +31 macOS files, 40 Win32 files, and the existing Windows allowlist. That allowlist +matches ten files and still contains two unmatched patterns +(`TabsLegacy/specs/*.windows.ts` and `Tokens/specs/*.win.ts`). Native discovery +fails on those patterns rather than silently omitting them. + +`--suite behavioral-pilot` selects the unchanged Win32 Button, Checkbox, +Switch, directional FocusZone, and Callout specs in one group. An interactive +Win32 run on October 22, 2026 (UTC) passes all 37 original assertions with no +skips, retries, or infrastructure failures, including keyboard callbacks, +checked/required state, focus navigation, scrolling-dependent page navigation, +and popup opening. This probe is not the complete legacy baseline or an +old/new comparative run. It does not enable restricted input or waive +unsupported assertions. + +Jasmine globals, hooks, matchers, retries, and grouped specs remain WDIO-owned. +The shared page adapter maps only established native attributes, uses checked +state for toggles, explicit key actions for chords, and native wheel input for +desktop scrolling. Required state maps to UI Automation's authoritative +`IsRequiredForForm` property; providers without it fail explicitly. `HelpText` +and toggle-pattern availability remain unmapped. Native setup retains initial +window bounds: it does **not** emulate the old maximize hook. Popup lookup +remains scoped to the selected window. + +Reports, original failures, and target-scoped evidence live under +`artifacts/desktop/`. Native helpers must be built/provisioned explicitly. +Windows additionally requires a built, registered RN Test App with a verified +generated renderer; macOS requires its own built application and permissions. + +Keep all legacy desktop and mobile lanes until per-test assertion mapping, +equivalent-build comparisons, and at least 100 representative runs per endpoint +over 14 days demonstrate at least 99% infrastructure success and zero process +or input leaks. Hosted input-disabled checks do not qualify physical input. +The remaining sections describe the retained Appium workflow. + **There is currently an Appium issue with NPM v8.19.2. Please use an older version of NPM** ## Win32/UWP Prerequisites diff --git a/apps/E2E/desktop-adapters.test.cjs b/apps/E2E/desktop-adapters.test.cjs new file mode 100644 index 0000000000..b5785534b7 --- /dev/null +++ b/apps/E2E/desktop-adapters.test.cjs @@ -0,0 +1,86 @@ +const assert = require('node:assert/strict'); +const { afterEach, test } = require('node:test'); +const { BasePage } = require('./lib/common/BasePage.js'); +const { Keys } = require('./lib/common/consts.js'); +const { installLegacyDesktopAttributes, sendNativeDesktopKeys } = require('./lib/common/nativeDesktop.js'); + +afterEach(() => { + delete globalThis.browser; +}); + +test('native checkbox and switch state never falls back to selected', async () => { + const page = new BasePage(); + const element = { elementId: 'toggle', isSelected: () => assert.fail('checked is not selected') }; + for (const value of [false, true, 'mixed', null, undefined]) { + globalThis.browser = { + capabilities: { browserName: 'furn-native-desktop' }, + async getElementProperty(id, property) { + assert.equal(id, 'toggle'); + assert.equal(property, 'checked'); + return value; + }, + }; + if (typeof value === 'boolean') assert.equal(await page.isChecked(element), value); + else await assert.rejects(page.isChecked(element), /native boolean checked state/); + } +}); + +test('legacy runners retain their selected-based toggle behavior', async () => { + globalThis.browser = { capabilities: { browserName: 'legacy-appium' } }; + const page = new BasePage(); + for (const value of [false, true]) assert.equal(await page.isChecked({ isSelected: async () => value }), value); +}); + +test('legacy required-form attributes use the authoritative native required property', async () => { + let attribute; + globalThis.browser = { + overwriteCommand(name, callback, element) { + assert.equal(name, 'getAttribute'); + assert.equal(element, true); + attribute = callback; + }, + }; + installLegacyDesktopAttributes(); + for (const value of [false, true, null, undefined, 'False']) { + const element = { + elementId: 'field', + async getElementProperty(id, property) { + assert.equal(id, 'field'); + assert.equal(property, 'required'); + return value; + }, + }; + const result = attribute.call(element, undefined, 'IsRequiredForForm'); + if (typeof value === 'boolean') assert.equal(await result, value ? 'True' : 'False'); + else await assert.rejects(result, /required-form state is unavailable/); + } +}); + +test('native key chords release held input even when dispatch fails', async () => { + const actions = [ + { type: 'keyDown', value: Keys.CONTROL }, + { type: 'keyDown', value: Keys.ENTER }, + { type: 'keyUp', value: Keys.ENTER }, + { type: 'keyUp', value: Keys.CONTROL }, + ]; + for (const fail of [false, true]) { + let released = false; + globalThis.browser = { + async elementSendKeys(id, value) { + assert.equal(id, 'button'); + assert.equal(value, ''); + }, + async performActions(sources) { + assert.deepEqual(sources, [{ type: 'key', id: 'tester-keyboard', actions }]); + if (fail) throw new Error('input dispatch failed'); + }, + async releaseActions() { + released = true; + }, + }; + const dispatch = sendNativeDesktopKeys({ elementId: 'button' }, [Keys.CONTROL, Keys.ENTER, Keys.NULL]); + if (fail) await assert.rejects(dispatch, /input dispatch failed/); + else await dispatch; + assert.equal(released, true); + } +}); diff --git a/apps/E2E/desktop-specs.mts b/apps/E2E/desktop-specs.mts new file mode 100644 index 0000000000..b23c4efdb6 --- /dev/null +++ b/apps/E2E/desktop-specs.mts @@ -0,0 +1,20 @@ +export const legacyDesktopSpecs = { + macos: [['src/**/specs/*.macos.ts']], + windows: [ + [ + 'src/ActivityIndicator/specs/*.win.ts', + 'src/ButtonLegacy/specs/*.win.ts', + 'src/CheckboxLegacy/specs/*.win.ts', + 'src/LinkLegacy/specs/*.win.ts', + 'src/PersonaCoin/specs/*.win.ts', + 'src/Pressable/specs/*.win.ts', + 'src/Separator/specs/*.win.ts', + 'src/TabsLegacy/specs/*.windows.ts', + 'src/TextLegacy/specs/*.win.ts', + 'src/TextV1/specs/*.win.ts', + 'src/Theme/specs/*.win.ts', + 'src/Tokens/specs/*.win.ts', + ], + ], + win32: [['../E2E/src/**/specs/*.win.ts']], +} as const; diff --git a/apps/E2E/desktop.config.mts b/apps/E2E/desktop.config.mts new file mode 100644 index 0000000000..d98b5dd441 --- /dev/null +++ b/apps/E2E/desktop.config.mts @@ -0,0 +1,51 @@ +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; +import { BASE_TESTPAGE } from './src/common/consts.ts'; +import { legacyDesktopSpecs } from './desktop-specs.mts'; + +const runner = { + kind: 'wdio', + framework: 'jasmine', + readiness: { kind: 'native', testId: BASE_TESTPAGE, timeoutMs: 180_000 }, + hooks: './src/common/desktopHooks.ts', + maxInstances: 1, + specFileRetries: 1, + bail: 1, +} as const; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { use: { config: '../fluent-tester/desktop.config.mts', target: 'macos' } }, + windows: { use: { config: '../fluent-tester/desktop.config.mts', target: 'windows' } }, + win32: { use: { config: '../win32/desktop.config.mts', target: 'win32' } }, + }, + suites: { + legacy: { ...runner, specs: legacyDesktopSpecs }, + pilot: { + ...runner, + specFileRetries: 0, + specs: { + macos: [['src/TextLegacy/specs/*.macos.ts']], + windows: [['src/Pressable/specs/*.win.ts']], + win32: [['src/Pressable/specs/*.win.ts']], + }, + }, + 'behavioral-pilot': { + ...runner, + specFileRetries: 0, + specs: { + win32: [ + [ + 'src/ButtonV1/specs/*.win.ts', + 'src/CheckboxV1/specs/*.win.ts', + 'src/Switch/specs/*.win.ts', + 'src/FocusZone/specs/FocusZone_Directional.spec.win.ts', + 'src/Callout/specs/*.win.ts', + ], + ], + }, + }, + }, + reporting: { artifactsRoot: './artifacts/desktop' }, +}); diff --git a/apps/E2E/package.json b/apps/E2E/package.json index 76847f9322..5f960e6cab 100644 --- a/apps/E2E/package.json +++ b/apps/E2E/package.json @@ -26,6 +26,7 @@ }, "scripts": { "build": "tsc -b", + "desktop-test": "desktop-test", "e2eprep:android": "cross-env APPIUM_HOME=.appium yarn exec appium driver install uiautomator2", "e2eprep:ios": "cross-env APPIUM_HOME=.appium yarn exec appium driver install xcuitest", "e2eprep:macos": "cross-env APPIUM_HOME=.appium yarn exec appium driver install mac2", @@ -37,6 +38,7 @@ "e2etest:win32": "cross-env APPIUM_HOME=. wdio run wdio.conf.win32.js", "e2etest:windows": "rimraf errorShots reports && cross-env APPIUM_HOME=. wdio run wdio.conf.windows.js", "lint": "fluentui-scripts lint", + "test": "node --test desktop-adapters.test.cjs", "test:appium": "cross-env APPIUM_HOME=. node --test appium.test.cjs" }, "dependencies": { @@ -45,6 +47,8 @@ "devDependencies": { "@babel/core": "catalog:", "@babel/runtime": "catalog:", + "@fluentui-react-native/desktop-driver": "workspace:*", + "@fluentui-react-native/desktop-test": "workspace:*", "@fluentui-react-native/focus-zone": "workspace:*", "@fluentui-react-native/scripts": "workspace:*", "@office-iss/react-native-win32": "^0.81.0", diff --git a/apps/E2E/src/Callout/pages/CalloutPageObject.win.ts b/apps/E2E/src/Callout/pages/CalloutPageObject.win.ts index 862eb665e9..1a6f78ea4b 100644 --- a/apps/E2E/src/Callout/pages/CalloutPageObject.win.ts +++ b/apps/E2E/src/Callout/pages/CalloutPageObject.win.ts @@ -23,7 +23,7 @@ class CalloutPageObject extends BasePage { async closeCallout(): Promise { // all we have to do is click outside the callout - this._testPage.click(); + await this._testPage.click(); } /*****************************************/ diff --git a/apps/E2E/src/CheckboxLegacy/pages/CheckboxLegacyPageObject.ts b/apps/E2E/src/CheckboxLegacy/pages/CheckboxLegacyPageObject.ts index 416b2ad14e..175ce9b18e 100644 --- a/apps/E2E/src/CheckboxLegacy/pages/CheckboxLegacyPageObject.ts +++ b/apps/E2E/src/CheckboxLegacy/pages/CheckboxLegacyPageObject.ts @@ -18,7 +18,7 @@ class CheckboxLegacyPageObject extends BasePage { // for native windows, .isSelected() always returns false. this is a workaround return (await checkbox.getAttribute(Attribute.ToggleState)) === AttributeValue.on; } else { - return await checkbox.isSelected(); + return this.isChecked(checkbox); } } diff --git a/apps/E2E/src/CheckboxV1/pages/CheckboxV1PageObject.ts b/apps/E2E/src/CheckboxV1/pages/CheckboxV1PageObject.ts index 3228d27dff..3d2e1f9740 100644 --- a/apps/E2E/src/CheckboxV1/pages/CheckboxV1PageObject.ts +++ b/apps/E2E/src/CheckboxV1/pages/CheckboxV1PageObject.ts @@ -12,7 +12,7 @@ class CheckboxV1PageObject extends BasePage { /**************** UI Element Interaction Methods ******************/ /******************************************************************/ async isCheckboxChecked(): Promise { - return await (await this._primaryComponent).isSelected(); + return this.isChecked(this._primaryComponent); } /* Waits for the checkbox to be checked or unchecked if the new state is true or false. Returns true if the diff --git a/apps/E2E/src/FocusZone/pages/FocusZonePageObject.ts b/apps/E2E/src/FocusZone/pages/FocusZonePageObject.ts index 20be39d436..27954bcce0 100644 --- a/apps/E2E/src/FocusZone/pages/FocusZonePageObject.ts +++ b/apps/E2E/src/FocusZone/pages/FocusZonePageObject.ts @@ -65,15 +65,15 @@ class FocusZonePageObject extends BasePage { break; default: } - const switchValue = await switchElement.isSelected(); + const switchValue = await this.isChecked(switchElement); if (switchValue !== arg) { await switchElement.click(); } // Wait until the switch correctly changes states - await browser.waitUntil(async () => (await switchElement.isSelected()) == arg, { + await browser.waitUntil(async () => (await this.isChecked(switchElement)) == arg, { timeout: 15000, - timeoutMsg: `Attempted to switch the ${option} to ${arg}, but it remained at ${await switchElement.isSelected()}`, + timeoutMsg: `Attempted to switch the ${option} to ${arg}, but it remained at ${await this.isChecked(switchElement)}`, }); } diff --git a/apps/E2E/src/Switch/pages/SwitchPageObject.ts b/apps/E2E/src/Switch/pages/SwitchPageObject.ts index 2b03898b35..82c4727ea2 100644 --- a/apps/E2E/src/Switch/pages/SwitchPageObject.ts +++ b/apps/E2E/src/Switch/pages/SwitchPageObject.ts @@ -17,7 +17,7 @@ class SwitchPageObject extends BasePage { } async isSwitchChecked(): Promise { - return await (await this._primaryComponent).isSelected(); + return this.isChecked(this._primaryComponent); } async waitForSwitchStateChange(newState: boolean, errorMsg: string): Promise { diff --git a/apps/E2E/src/common/BasePage.ts b/apps/E2E/src/common/BasePage.ts index 5c9507d378..15f3b5596f 100644 --- a/apps/E2E/src/common/BasePage.ts +++ b/apps/E2E/src/common/BasePage.ts @@ -11,7 +11,8 @@ import { ROOT_VIEW, TESTPAGE_BUTTONS_SCROLLVIEWER, TESTPAGE_CONTENT_SCROLLVIEWER, -} from './consts'; +} from './consts.js'; +import { isNativeDesktop, scrollNativeDesktopElement, sendNativeDesktopKeys } from './nativeDesktop.js'; const DUMMY_CHAR = ''; // The E2ETEST_PLATFORM environment variable should be set in the beforeSession hook in the wdio.conf file for the respective platform @@ -78,11 +79,13 @@ export abstract class BasePage { */ async navigateToPageAndLoadTests(): Promise { // Desktop platforms automatically scroll to a page's navigation button - this extra step is purely for mobile platforms. - if (this.platform === 'android' || this.platform === 'ios' || this.platform === 'windows') { + if (isNativeDesktop()) { + await scrollNativeDesktopElement(this._pageButton, this._testPageButtonScrollViewer); + } else if (this.platform === 'android' || this.platform === 'ios' || this.platform === 'windows') { await this.mobileScrollToComponentButton(); } - if (this.platform === 'windows') { + if (this.platform === 'windows' && !isNativeDesktop()) { const pageButton = await this._pageButton; // sendKeys will move focus to the button, so we dont have to worry about the button being offscreen in the scrollable area. await this.sendKeys(pageButton, [Keys.ENTER]); @@ -104,25 +107,32 @@ export abstract class BasePage { await e2eSwitch.waitForDisplayed({ timeoutMsg: 'E2E Test Switch should be visible to enable it' }); await e2eSwitch.waitForEnabled({ timeoutMsg: 'The E2E Switch should be enabled before we interact with it' }); - switch (this.platform) { - // Usually, we use .isSelected() to see if a control (our switch) is checked true or false, but the process is - // different on android because .isSelected() doesn't function as expected on the platform. - case 'android': - if ((await e2eSwitch.getAttribute(AndroidAttribute.Checked)) === 'false') { - await e2eSwitch.click(); - await e2eSwitch.waitUntil(async () => (await e2eSwitch.getAttribute(AndroidAttribute.Checked)) === 'true', { - timeoutMsg: 'Clicked the E2E Mode Switch, but it failed to toggle.', - }); - } - break; - default: - if (!(await e2eSwitch.isSelected())) { - await e2eSwitch.click(); - await e2eSwitch.waitUntil(async () => await e2eSwitch.isSelected(), { - timeoutMsg: 'Clicked the E2E Mode Switch, but it failed to toggle.', - }); - } - } + if (isNativeDesktop()) { + const checked = () => this.isChecked(e2eSwitch); + if (!(await checked())) { + await e2eSwitch.click(); + await browser.waitUntil(checked, { timeoutMsg: 'Clicked the native E2E switch, but checked state did not become true.' }); + } + } else + switch (this.platform) { + // Usually, we use .isSelected() to see if a control (our switch) is checked true or false, but the process is + // different on android because .isSelected() doesn't function as expected on the platform. + case 'android': + if ((await e2eSwitch.getAttribute(AndroidAttribute.Checked)) === 'false') { + await e2eSwitch.click(); + await e2eSwitch.waitUntil(async () => (await e2eSwitch.getAttribute(AndroidAttribute.Checked)) === 'true', { + timeoutMsg: 'Clicked the E2E Mode Switch, but it failed to toggle.', + }); + } + break; + default: + if (!(await e2eSwitch.isSelected())) { + await e2eSwitch.click(); + await e2eSwitch.waitUntil(async () => await e2eSwitch.isSelected(), { + timeoutMsg: 'Clicked the E2E Mode Switch, but it failed to toggle.', + }); + } + } return await this._e2eSection.waitForDisplayed({ timeoutMsg: 'Pressed E2E Mode Switch, but E2E Test Sections failed to display before the timeout.', @@ -190,6 +200,13 @@ export abstract class BasePage { return element.click(); } + async isChecked(element: ChainablePromiseElement): Promise { + if (!isNativeDesktop()) return element.isSelected(); + const value: unknown = await browser.getElementProperty(await element.elementId, 'checked'); + if (typeof value !== 'boolean') throw new Error('The control did not expose a native boolean checked state.'); + return value; + } + /** Given a WebdriverIO element promise, send the passed in list of keys as keyboard inputs. Use this across all PageObject methods and test specs. * * Common examples: @@ -198,6 +215,7 @@ export abstract class BasePage { * - Escape out of a menu: MenuPageObject.sendKeys(MenuPageObject.item1, [KEY_ESCAPE]) */ async sendKeys(element: ChainablePromiseElement, keys: Keys[]) { + if (isNativeDesktop()) return sendNativeDesktopKeys(element, keys); return element.addValue(keys.join()); } @@ -209,6 +227,7 @@ export abstract class BasePage { /* Scrolls until the desired test page's button is displayed. We use the scroll viewer UI element as the point to start scrolling. * We use a negative number as the Y-coordinate because that enables us to scroll downwards */ async mobileScrollToComponentButton() { + if (isNativeDesktop()) return scrollNativeDesktopElement(this._pageButton, this._testPageButtonScrollViewer); if (this.platform !== 'android' && (await this._pageButton.isDisplayed())) { return; } @@ -302,6 +321,7 @@ export abstract class BasePage { /* Scrolls to the specified or primary UI test element until it is displayed. */ async scrollToTestElement(component?: ChainablePromiseElement) { const ComponentToScrollTo = component ? component : this._primaryComponent; + if (isNativeDesktop()) return scrollNativeDesktopElement(ComponentToScrollTo, By(TESTPAGE_CONTENT_SCROLLVIEWER)); if (await ComponentToScrollTo.isDisplayed()) { return; } @@ -331,6 +351,7 @@ export abstract class BasePage { /* Scrolls to the specified or primary UI test element until it is displayed. */ async mobileScrollToTestElement(componentIdentifier?: string): Promise { const componentToScrollTo = componentIdentifier ?? this._primaryComponentName; + if (isNativeDesktop()) return scrollNativeDesktopElement(By(componentToScrollTo), By(TESTPAGE_CONTENT_SCROLLVIEWER)); if (await By(componentToScrollTo).isDisplayed()) { return; } diff --git a/apps/E2E/src/common/desktopHooks.ts b/apps/E2E/src/common/desktopHooks.ts new file mode 100644 index 0000000000..055bd09442 --- /dev/null +++ b/apps/E2E/src/common/desktopHooks.ts @@ -0,0 +1,44 @@ +import path from 'node:path'; +import { createHash } from 'node:crypto'; +import { createDesktopDriverClient, DesktopSessionClient } from '@fluentui-react-native/desktop-driver'; +import { ArtifactManager, captureDesktopFailureEvidence } from '@fluentui-react-native/desktop-driver/artifacts'; +import { createDesktopReportSession } from '@fluentui-react-native/desktop-test/reporting'; +import type { DesktopReportSession } from '@fluentui-react-native/desktop-test/reporting'; +import { installLegacyDesktopAttributes } from './nativeDesktop'; + +let reporting: DesktopReportSession | undefined; + +export default { + async before() { + reporting = await createDesktopReportSession({ artifactsRoot: path.resolve('artifacts', 'desktop'), kind: 'lifecycle' }); + installLegacyDesktopAttributes(); + // Native cohorts use the app's initial bounds and native scrolling, not an emulated maximize command. + const { width, height } = await browser.getWindowRect(); + if (width <= 0 || height <= 0) throw new Error('The native tester window has no usable bounds.'); + }, + async afterTest(test: { fullTitle?: string; title: string }, _context: unknown, outcome: { passed: boolean }) { + if (outcome.passed || !reporting) return; + const reporter = reporting.reporter; + const name = createHash('sha256') + .update(JSON.stringify([test.fullTitle ?? test.title, Date.now()])) + .digest('hex') + .slice(0, 24); + const root = reporter.artifactPath(path.join('evidence', name)); + const url = `${browser.options.protocol}://${browser.options.hostname}:${browser.options.port}${browser.options.path}`; + const session = new DesktopSessionClient( + createDesktopDriverClient({ url }), + browser.sessionId, + Object.fromEntries(Object.entries(browser.capabilities)), + ); + const evidence = await captureDesktopFailureEvidence(session, new ArtifactManager(root), 'failure'); + for (const artifact of evidence.artifacts) + reporter.emit({ + kind: 'artifact', + payload: { path: path.relative(reporter.root, path.join(root, artifact.path)), description: artifact.kind }, + }); + for (const error of evidence.errors) reporter.fail(new Error(error), 'evidence'); + }, + async afterSession() { + await reporting?.close(); + }, +}; diff --git a/apps/E2E/src/common/nativeDesktop.ts b/apps/E2E/src/common/nativeDesktop.ts new file mode 100644 index 0000000000..675b0bf1d6 --- /dev/null +++ b/apps/E2E/src/common/nativeDesktop.ts @@ -0,0 +1,111 @@ +import { Keys } from './consts.js'; + +export function isNativeDesktop(): boolean { + return browser.capabilities.browserName === 'furn-native-desktop'; +} + +export async function sendNativeDesktopKeys(element: ChainablePromiseElement, keys: readonly Keys[]): Promise { + const target = await element; + await browser.elementSendKeys(await target.elementId, ''); + const actions: { type: 'keyDown' | 'keyUp'; value: string }[] = []; + const held = new Set(); + for (const value of keys) { + if (value === Keys.NULL) { + for (const modifier of held) actions.push({ type: 'keyUp', value: modifier }); + held.clear(); + } else if ([Keys.SHIFT, Keys.CONTROL, Keys.ALT].some((modifier) => modifier === value)) { + if (!held.has(value)) actions.push({ type: 'keyDown', value }); + held.add(value); + } else actions.push({ type: 'keyDown', value }, { type: 'keyUp', value }); + } + for (const value of held) actions.push({ type: 'keyUp', value }); + let failure: unknown; + try { + await browser.performActions([{ type: 'key', id: 'tester-keyboard', actions }]); + } catch (error) { + failure = error; + } + try { + await browser.releaseActions(); + } catch (error) { + throw new AggregateError(failure === undefined ? [error] : [failure, error], 'Desktop keyboard input or release failed.'); + } + if (failure !== undefined) throw failure; +} + +export async function scrollNativeDesktopElement(element: ChainablePromiseElement, scroller: ChainablePromiseElement): Promise { + if (await element.isDisplayed()) return; + if (!browser.capabilities['furn:features']?.wheel) + throw new Error('This desktop does not support native wheel input; navigation cannot be qualified.'); + const origin = { 'element-6066-11e4-a52e-4f735466cecf': await scroller.elementId }; + await browser.waitUntil( + async () => { + await browser.performActions([ + { type: 'wheel', id: 'tester-scroll', actions: [{ type: 'scroll', origin, x: 0, y: 0, deltaX: 0, deltaY: 300, duration: 100 }] }, + ]); + return element.isDisplayed(); + }, + { timeout: 30_000, timeoutMsg: 'The native scroll container did not expose the selected tester control.' }, + ); +} + +export function installLegacyDesktopAttributes(): void { + const properties: Readonly> = { + AutomationId: 'automationId', + Name: 'name', + ControlType: 'role', + IsEnabled: 'enabled', + HasKeyboardFocus: 'focused', + IsRequiredForForm: 'required', + 'Toggle.ToggleState': 'checked', + 'ExpandCollapse.ExpandCollapseState': 'expanded', + }; + const roles: Readonly> = { + button: 'Button', + group: 'Group', + checkbox: 'CheckBox', + image: 'Image', + link: 'HyperLink', + menu: 'Menu', + menuitem: 'MenuItem', + radio: 'RadioButton', + list: 'List', + tablist: 'Tab', + tab: 'TabItem', + text: 'Text', + }; + browser.overwriteCommand( + 'getAttribute', + async function (original, name) { + const property = properties[name]; + if (!property) { + if (['HelpText', 'IsTogglePatternAvailable'].includes(name)) { + throw new Error( + `Legacy attribute "${name}" has no qualified native mapping. It must not be reported as false or silently skipped.`, + ); + } + return original(name); + } + const value: unknown = await this.getElementProperty(this.elementId, property); + if (name === 'ControlType') { + if (typeof value !== 'string' || !roles[value]) + throw new Error(`Native role "${String(value)}" has no legacy ControlType mapping.`); + return `ControlType.${roles[value]}`; + } + if (name === 'Toggle.ToggleState') { + if (value === 'mixed') return '2'; + if (typeof value !== 'boolean') throw new Error('Native checked state is unavailable.'); + return value ? '1' : '0'; + } + if (name === 'ExpandCollapse.ExpandCollapseState') { + if (typeof value !== 'boolean') throw new Error('Native expanded state is unavailable.'); + return value ? 'Expanded' : 'Collapsed'; + } + if (name === 'IsRequiredForForm' && typeof value !== 'boolean') throw new Error('Native required-form state is unavailable.'); + if (typeof value === 'boolean') return value ? 'True' : 'False'; + if (value === null || typeof value === 'string') return value; + throw new Error(`Native property "${property}" returned an invalid value.`); + }, + true, + ); +} diff --git a/apps/E2E/tsconfig.json b/apps/E2E/tsconfig.json index 55864b6fde..7bd00ba3fb 100644 --- a/apps/E2E/tsconfig.json +++ b/apps/E2E/tsconfig.json @@ -9,6 +9,12 @@ }, "include": ["src"], "references": [ + { + "path": "../../packages/agentic/desktop-test/tsconfig.json" + }, + { + "path": "../../packages/agentic/desktop-driver/tsconfig.json" + }, { "path": "../../packages/shim/focus-zone/tsconfig.json" }, diff --git a/apps/E2E/wdio.conf.macos.js b/apps/E2E/wdio.conf.macos.js index 67e61c5a8b..943720e48e 100644 --- a/apps/E2E/wdio.conf.macos.js +++ b/apps/E2E/wdio.conf.macos.js @@ -1,4 +1,5 @@ const fs = require('fs'); +const { legacyDesktopSpecs } = require('./desktop-specs.mts'); const defaultWaitForTimeout = 20000; const defaultConnectionRetryTimeout = 60000; @@ -6,7 +7,7 @@ const jasmineDefaultTimeout = 60000; // 60 seconds for Jasmine test timeout exports.config = { runner: 'local', // Where should your test be launched - specs: [['src/**/specs/*.macos.ts']], + specs: legacyDesktopSpecs.macos, exclude: [/* 'path/to/excluded/files' */], maxInstances: 30, diff --git a/apps/E2E/wdio.conf.win32.js b/apps/E2E/wdio.conf.win32.js index c2f180a404..b8a27e81f2 100644 --- a/apps/E2E/wdio.conf.win32.js +++ b/apps/E2E/wdio.conf.win32.js @@ -1,5 +1,6 @@ const path = require('path'); const fs = require('fs'); +const { legacyDesktopSpecs } = require('./desktop-specs.mts'); const appPath = path.resolve(path.dirname(require.resolve('@office-iss/rex-win32/rex-win32.js')), 'ReactTest.exe'); const appArgs = @@ -14,7 +15,7 @@ const jasmineDefaultTimeout = 60000; // 60 seconds for Jasmine test timeout exports.config = { runner: 'local', // Where should your test be launched - specs: [['../E2E/src/**/specs/*.win.ts']], + specs: legacyDesktopSpecs.win32, exclude: [], capabilities: [ diff --git a/apps/E2E/wdio.conf.windows.js b/apps/E2E/wdio.conf.windows.js index 997c2a24d6..57de27f124 100644 --- a/apps/E2E/wdio.conf.windows.js +++ b/apps/E2E/wdio.conf.windows.js @@ -1,4 +1,5 @@ const fs = require('fs'); +const { legacyDesktopSpecs } = require('./desktop-specs.mts'); const defaultWaitForTimeout = 20000; const defaultConnectionRetryTimeout = 60000; @@ -6,24 +7,8 @@ const jasmineDefaultTimeout = 60000; // 60 seconds for Jasmine test timeout exports.config = { runner: 'local', - /* UWP controls are a subset of the Win32 controls. Only some work on our UWP test app, - so we must specify which ones we want to test here. */ - specs: [ - [ - 'src/ActivityIndicator/specs/*.win.ts', - 'src/ButtonLegacy/specs/*.win.ts', - 'src/CheckboxLegacy/specs/*.win.ts', // See spec file for more information - 'src/LinkLegacy/specs/*.win.ts', - 'src/PersonaCoin/specs/*.win.ts', - 'src/Pressable/specs/*.win.ts', - 'src/Separator/specs/*.win.ts', - 'src/TabsLegacy/specs/*.windows.ts', // See spec file for more information - 'src/TextLegacy/specs/*.win.ts', - 'src/TextV1/specs/*.win.ts', - 'src/Theme/specs/*.win.ts', - 'src/Tokens/specs/*.win.ts', - ], - ], + // Preserve the existing allowlist, including unmatched patterns that block native parity. + specs: legacyDesktopSpecs.windows, exclude: [/* 'path/to/excluded/files' */], maxInstances: 1, diff --git a/apps/fluent-tester/desktop.config.mts b/apps/fluent-tester/desktop.config.mts new file mode 100644 index 0000000000..a8b7c59dd4 --- /dev/null +++ b/apps/fluent-tester/desktop.config.mts @@ -0,0 +1,10 @@ +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { host: { kind: 'react-native-test-app', renderer: 'fabric' }, launchTimeoutMs: 180_000 }, + windows: { host: { kind: 'react-native-test-app', renderer: 'fabric' }, launchTimeoutMs: 180_000 }, + }, +}); diff --git a/apps/fluent-tester/package.json b/apps/fluent-tester/package.json index fc4bf900da..ed277ea12a 100644 --- a/apps/fluent-tester/package.json +++ b/apps/fluent-tester/package.json @@ -30,6 +30,7 @@ "bundle:macos": "rnx-cli bundle --dev false --platform macos", "bundle:windows": "rnx-cli bundle --dev false --platform windows", "clean": "fluentui-scripts clean", + "desktop-test": "desktop-test", "format": "fluentui-scripts format", "ios": "rnx-cli run --platform ios", "lint": "fluentui-scripts lint", @@ -51,6 +52,7 @@ "@babel/core": "catalog:", "@babel/runtime": "catalog:", "@fluentui-react-native/callout": "workspace:*", + "@fluentui-react-native/desktop-test": "workspace:*", "@fluentui-react-native/experimental-appearance-additions": "workspace:*", "@fluentui-react-native/experimental-avatar": "workspace:*", "@fluentui-react-native/experimental-checkbox": "workspace:*", diff --git a/apps/fluent-tester/tsconfig.json b/apps/fluent-tester/tsconfig.json index ae103d0427..a66e1911e5 100644 --- a/apps/fluent-tester/tsconfig.json +++ b/apps/fluent-tester/tsconfig.json @@ -13,6 +13,9 @@ { "path": "../../packages/shim/callout/tsconfig.json" }, + { + "path": "../../packages/agentic/desktop-test/tsconfig.json" + }, { "path": "../../packages/experimental/AppearanceAdditions/tsconfig.json" }, diff --git a/apps/storybook/AGENTS.md b/apps/storybook/AGENTS.md index ab6f495d0e..245d901450 100644 --- a/apps/storybook/AGENTS.md +++ b/apps/storybook/AGENTS.md @@ -3,7 +3,7 @@ These instructions apply to `apps/storybook` and its descendants. Read [`agent-map.yaml`](agent-map.yaml) first for the compact architecture, lookup, and interaction map. Read this file, -`README.md`, `package.json`, and `storybook.config.mts` before changing the Storybook application or its native projects. +`README.md`, `package.json`, and `desktop.config.mts` before changing the Storybook application or its native projects. ## Agent-efficient discovery and interaction @@ -12,7 +12,7 @@ Read [`agent-map.yaml`](agent-map.yaml) first for the compact architecture, look - Query and run stories through `yarn desktop-driver` against the listener started by `yarn storybook driver --`. - Use screenshots only for visual evidence. Use WebSocket events and accessibility bounds for lookup and interaction. -- Treat `src/storybook.requires.ts` as generated output; use `storybook.config.mts` for story package discovery and +- Treat `src/storybook.requires.ts` as generated output; use `desktop.config.mts` for story package discovery and `src/main.ts` only as the shared config adapter. - Keep `agent-map.yaml` synchronized when stable paths, scripts, services, or interaction contracts change. Do not list individual stories there; the runtime index is authoritative. @@ -26,7 +26,7 @@ Read [`agent-map.yaml`](agent-map.yaml) first for the compact architecture, look - Run `yarn` from the repository root only after dependency manifests change or when a declared command fails because a dependency is missing. - Preserve unrelated manifest and lockfile edits already present in the worktree. -- Keep only native command exceptions and ownership-specific smoke settings in `storybook.config.mts`. Standard macOS +- Keep only native command exceptions and ownership-specific smoke settings in `desktop.config.mts`. Standard macOS and Windows prep, bundle, build, and run commands come from the shared config, derive identity from `app.json`, and route through `rnx-cli`. Use `yarn storybook --`; do not add platform aliases or app-local lifecycle scripts. @@ -50,7 +50,7 @@ capture limits. From this directory: | Runtime trees and crash reproduction | `yarn desktop-driver agent describe ...` and `yarn storybook --verbose smoke --` | For `agent` commands, use `--platform macos` on macOS and `--platform windows` for both Windows and Win32. Inspect -retained command logs under `artifacts/storybook-commands` and platform artifacts rather than redirecting output to +retained command logs under `artifacts/desktop//commands` and platform artifacts rather than redirecting output to arbitrary locations. ## Desktop Driver workflow @@ -72,15 +72,16 @@ arbitrary locations. - Run executable tests with `yarn storybook test --` and optionally `--test `. Callbacks receive the target endpoint as `platform`; each named case gets an isolated worker/session and fresh preview. Defaults belong in - `storybook.config.mts` under `wdio`; the shared Babel config removes + `desktop.config.mts` under the Storybook integration's `wdio`; the shared Babel config removes callbacks from native bundles. `stories-and-tests` smoke mode runs configured callbacks together with static plans, grouped by story. Always navigate to the requested story and await correlated readiness before executing tests. - Keep `storybook-desktop.generated`, reports, trees, screenshots, and run manifests ignored. Never patch generated runtime identity or story manifests. -- Shared command logs merge stdout/stderr in write order under - `artifacts/storybook-commands`. Failed commands replay the complete log; - `--verbose` also replays successful commands. Preserve labeled stderr +- Shared command logs retain stdout/stderr attribution in collector arrival order under + `artifacts/desktop//commands`; cross-stream producer write order is not guaranteed. + Stderr and test progress are live; failed commands replay the complete log. + `--verbose` also streams informational output. Preserve labeled stderr diagnostics and nested failure causes rather than replacing them with exit-code-only summaries. - Treat the exact-platform and portable-plan digests as contracts. A dynamic or @@ -128,7 +129,7 @@ arbitrary locations. stages. Use `yarn storybook smoke --windows --mode stories` for the package-owned generation, channel server, native build and registration, Metro launch, full indexed-story traversal, and ownership-safe cleanup. Use `--mode stories-and-tests` to run the component-authored desktop-e2e plans after the complete traversal. -- Keep the Windows story-pattern overrides in `storybook.config.mts`; the +- Keep the Windows story-pattern overrides in `desktop.config.mts`; the current Accordion Fabric stories still fail-fast the RNW 0.81 host during traversal. Native-core Callout and FocusZone stories run on Windows. - WinAppDriver screenshots are not a reliable capture path for WinAppSDK Composition content. After selecting a story @@ -151,7 +152,7 @@ arbitrary locations. - Run `yarn storybook bundle --win32` before `yarn storybook run --win32`. The bundle is the native dependency source for the prebuilt REX host. - Keep package discovery and platform-specific story inclusion in - `storybook.config.mts`; keep `src/main.ts` as the shared config adapter. + `desktop.config.mts`; keep `src/main.ts` as the shared config adapter. - Keep shared Storybook source platform-neutral. Win32-specific source belongs in `.win32.ts` or `.win32.tsx` files, and Metro platform resolution belongs in `metro.config.js`. @@ -173,7 +174,7 @@ arbitrary locations. - Keep macOS and Windows on upstream LiteUI. Replacing it with the reduced Win32 chrome would regress addon controls and responsive behavior while increasing local maintenance. -- Keep the Win32 story-pattern override in `storybook.config.mts`. It intentionally excludes ListItem and Accordion +- Keep the Win32 story-pattern override in `desktop.config.mts`. It intentionally excludes ListItem and Accordion because their Paper implementations fail-fast crash REX 0.81.1. - Keep the Win32 window title distinct from the Windows Fabric title so automation never attaches to the wrong endpoint. diff --git a/apps/storybook/README.md b/apps/storybook/README.md index 48b7b02549..76f3999d41 100644 --- a/apps/storybook/README.md +++ b/apps/storybook/README.md @@ -7,7 +7,8 @@ stories. FocusZone stories remain agentic primitive stories and use native-core' `legacy` entrypoint, as does the Win32 Callout portal. Native-core owns the shared macOS pod and Windows Fabric library for both components. -The reusable desktop CLI and configuration live in +The application-neutral lifecycle and reporting live in +`packages/agentic/desktop-test`. The optional Storybook CLI and configuration live in `packages/agentic/storybook-desktop`, with peer-dependent React Native implementation in `packages/agentic/storybook-desktop-runtime`. The runtime runs Storybook in **liteMode**, which mocks out the heavy default on-device UI @@ -38,13 +39,14 @@ discover the case names, or use `--test ` to filter them. With `yarn storybook test --macos --story 'components-button--*'`. The shared runner extracts the callbacks for Node, injects WebdriverIO and native assertions, and leaves test dependencies out of Metro. Optional runner -defaults belong under `wdio` in `storybook.config.mts`; no separate test config +defaults belong under the Storybook integration's `wdio` in `desktop.config.mts`; no separate test config is needed. See the [authoring contract](../../packages/agentic/storybook-desktop/README.md#executable-tests-inside-stories). `stories-and-tests` smoke mode also runs these callbacks, grouped with each story's static plans. Both paths select the correct page before executing tests. -Commands retain merged stdout/stderr logs under `artifacts/storybook-commands`. -Successful commands emit short summaries; failures replay their complete log +Commands retain source-attributed stdout/stderr logs under `artifacts/desktop//commands`, +alongside the canonical event journal, JSON result, and JUnit report. +Stderr and test progress are live; failures replay their complete log and print contextual story/test diagnostics. Use `yarn storybook --verbose smoke --macos --mode stories-and-tests` to include successful-command output as well. No shell redirection is required. @@ -54,7 +56,7 @@ successful-command output as well. No shell redirection is required. ``` storybook/ src/ Storybook adapters, generated requires, and shared-runtime integration - storybook.config.mts App-owned package discovery, platform patterns, and native CLI settings + desktop.config.mts Canonical targets and optional Storybook package discovery/platform settings index.js AppRegistry entry app.json react-native-test-app manifest metro.config.js rnx-kit metro config wrapped with withStorybook (liteMode) @@ -357,7 +359,7 @@ Run it alongside `yarn start` and `yarn storybook run --macos|--windows`. The on ## Writing stories Follow the package-level story authoring instructions in `../../packages/agentic/components/AGENTS.md`. Add a -`*.stories.tsx` file next to its component; package discovery belongs in `storybook.config.mts`, adapted by `src/main.ts`. +`*.stories.tsx` file next to its component; package discovery belongs in `desktop.config.mts`, adapted by `src/main.ts`. See `../../packages/agentic/components/src/components/button/button.stories.tsx` for the canonical higher-order component example. Prefer named `wdio` functions with stable `testID` selectors and injected `platform` context, as demonstrated by Button. All catalog tests, including diff --git a/apps/storybook/agent-map.yaml b/apps/storybook/agent-map.yaml index c70de0a9f6..44cecb118f 100644 --- a/apps/storybook/agent-map.yaml +++ b/apps/storybook/agent-map.yaml @@ -2,13 +2,14 @@ version: 1 purpose: Agent-optimized map of stable Storybook structure and runtime discovery paths. sources: - config: storybook.config.mts + config: desktop.config.mts config_adapter: src/main.ts app_entry: index.js app_shell: src/StorybookApp.tsx theme_host: ../../packages/agentic/storybook-desktop-runtime/src/StorybookTheme.tsx theme_palette: ../../packages/agentic/storybook-desktop-runtime/src/useStorybookTheme.ts shared_cli: ../../packages/agentic/storybook-desktop + shared_harness: ../../packages/agentic/desktop-test shared_runtime: ../../packages/agentic/storybook-desktop-runtime generated_requires: path: src/storybook.requires.ts @@ -31,7 +32,7 @@ services: command: yarn storybook test -- discovery: yarn storybook test -- --list authoring: top-level wdio callbacks or named callback collections in CSF3 stories - config: storybook.config.mts wdio settings + config: desktop.config.mts Storybook integration wdio settings lifecycle: attached-session-per-test-with-isolated-node-worker context_platform: macos-or-windows-or-win32-target-endpoint test_filter: --test @@ -48,7 +49,7 @@ agent_commands: lookup_order: - Generate the exact platform manifest; never guess ID slugging. - Read this map for stable paths, services, and commands. - - Read storybook.config.mts for story-package and platform changes. + - Read desktop.config.mts for story-package and platform changes. - Read a matched story file for component-specific behavior. - Read generated requires only to diagnose generation output. @@ -97,8 +98,9 @@ validation: lifecycle: yarn storybook smoke -- --mode stories-and-tests output_hygiene: - - Commands retain merged stdout/stderr logs under artifacts/storybook-commands; failures replay grouped output. - - Use --verbose to replay successful-command logs; files update continuously while commands run. + - Canonical reports and source-attributed command logs live under artifacts/desktop/. + - Stderr and test progress are live; --verbose also streams informational output. + - Logs preserve collector arrival order across streams; failures replay grouped output. - Use structured CLI output instead of logging raw channel events. - Search large native build logs for errors and final status instead of reading them in full. - Keep warning suppression exact; console diagnostics remain available even when LogBox hides a warning. diff --git a/apps/storybook/desktop.config.mts b/apps/storybook/desktop.config.mts new file mode 100644 index 0000000000..673ea7ac2b --- /dev/null +++ b/apps/storybook/desktop.config.mts @@ -0,0 +1,69 @@ +import { + createWindowsSmokeOptions, + createWin32RunCommand, + createWin32SmokeCommand, + storybook, +} from '@fluentui-react-native/storybook-desktop/config'; +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +import appManifest from './app.json' with { type: 'json' }; + +const win32Host = { + component: 'AgenticStorybook', + windowTitle: 'Agentic Components Storybook (Win32)', +} as const; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + macos: { host: { kind: 'react-native-test-app', renderer: 'fabric' } }, + windows: { host: { kind: 'react-native-test-app', renderer: 'fabric' } }, + win32: { host: { kind: 'rex', renderer: 'paper', ...win32Host }, commands: { run: createWin32RunCommand(win32Host) } }, + }, + reporting: { artifactsRoot: './artifacts/desktop' }, + suites: { stories: { kind: 'csf-wdio' } }, + integration: storybook({ + storyPackages: [ + [ + '@fluentui-react-native/components', + { + platformSettings: { + windows: { + storyPatterns: ['src/primitives/**/*.stories.?(ts|tsx)', 'src/components/!(accordion)/**/*.stories.?(ts|tsx)'], + }, + win32: { + storyPatterns: ['src/primitives/**/*.stories.?(ts|tsx)', 'src/components/!(accordion|list-item)/**/*.stories.?(ts|tsx)'], + }, + }, + }, + ], + [ + '@fluentui-react-native/native-core', + { + platformSettings: { + win32: { + storyPatterns: ['src/legacy/**/*.stories.?(ts|tsx)'], + }, + }, + }, + ], + ], + platformOptions: { + windows: { + smoke: createWindowsSmokeOptions({ + windowTitle: 'Agentic Components Storybook', + }), + }, + win32: { + smoke: { + command: createWin32SmokeCommand({ + ...win32Host, + testIDPrefix: appManifest.storybook.testIDPrefix, + requiredStoryIds: ['native-callout--default', 'native-callout--placement', 'native-callout--window-commands'], + }), + }, + }, + }, + }), +}); diff --git a/apps/storybook/package.json b/apps/storybook/package.json index 13f134e41c..4db10644f4 100644 --- a/apps/storybook/package.json +++ b/apps/storybook/package.json @@ -15,6 +15,7 @@ "storybook": "storybook-desktop", "storybook-server": "storybook-desktop server", "desktop-driver": "desktop-driver", + "desktop-test": "desktop-test", "prebuild": "sb-rn-get-stories --config-path src", "lint": "fluentui-scripts lint", "format": "fluentui-scripts format" @@ -35,6 +36,7 @@ "devDependencies": { "@babel/core": "catalog:", "@fluentui-react-native/desktop-driver": "workspace:*", + "@fluentui-react-native/desktop-test": "workspace:*", "@fluentui-react-native/scripts": "workspace:*", "@office-iss/rex-win32": "0.81.1", "@react-native-community/cli": "^20.0.0", diff --git a/apps/storybook/src/main.ts b/apps/storybook/src/main.ts index 0f7aa21534..921cb4ab04 100644 --- a/apps/storybook/src/main.ts +++ b/apps/storybook/src/main.ts @@ -1,7 +1,7 @@ import type { DesktopReactNativeStorybookConfig } from '@fluentui-react-native/storybook-desktop/config'; -import config from '../storybook.config.mts'; +import config from '../desktop.config.mts'; -const storybookConfig: DesktopReactNativeStorybookConfig = config.getStorybookConfig(); +const storybookConfig: DesktopReactNativeStorybookConfig = config.integration.configure(config.projectRoot).getStorybookConfig(); export default storybookConfig; diff --git a/apps/storybook/storybook.config.mts b/apps/storybook/storybook.config.mts deleted file mode 100644 index a4b78b9e01..0000000000 --- a/apps/storybook/storybook.config.mts +++ /dev/null @@ -1,64 +0,0 @@ -import { - createWindowsSmokeOptions, - createWin32RunCommand, - createWin32SmokeCommand, - makeDesktopStorybookConfig, -} from '@fluentui-react-native/storybook-desktop/config'; - -import appManifest from './app.json' with { type: 'json' }; - -const win32Host = { - component: 'AgenticStorybook', - windowTitle: 'Agentic Components Storybook (Win32)', -} as const; - -export default makeDesktopStorybookConfig({ - projectRoot: new URL('.', import.meta.url), - storyPackages: [ - [ - '@fluentui-react-native/components', - { - platformSettings: { - windows: { - storyPatterns: ['src/primitives/**/*.stories.?(ts|tsx)', 'src/components/!(accordion)/**/*.stories.?(ts|tsx)'], - }, - win32: { - storyPatterns: ['src/primitives/**/*.stories.?(ts|tsx)', 'src/components/!(accordion|list-item)/**/*.stories.?(ts|tsx)'], - }, - }, - }, - ], - [ - '@fluentui-react-native/native-core', - { - platformSettings: { - win32: { - storyPatterns: ['src/legacy/**/*.stories.?(ts|tsx)'], - }, - }, - }, - ], - ], - platformOptions: { - windows: { - smoke: createWindowsSmokeOptions({ - windowTitle: 'Agentic Components Storybook', - }), - }, - win32: { - nativeDriver: { - application: { - windowTitle: win32Host.windowTitle, - }, - }, - run: createWin32RunCommand(win32Host), - smoke: { - command: createWin32SmokeCommand({ - ...win32Host, - testIDPrefix: appManifest.storybook.testIDPrefix, - requiredStoryIds: ['native-callout--default', 'native-callout--placement', 'native-callout--window-commands'], - }), - }, - }, - }, -}); diff --git a/apps/storybook/tsconfig.json b/apps/storybook/tsconfig.json index 3ce850c19e..7898b14c32 100644 --- a/apps/storybook/tsconfig.json +++ b/apps/storybook/tsconfig.json @@ -12,7 +12,7 @@ "composite": true, "tsBuildInfoFile": ".cache/tsconfig.tsbuildinfo" }, - "include": ["index.js", "src", "app.json", "storybook.config.mts"], + "include": ["index.js", "src", "app.json", "desktop.config.mts"], "exclude": ["node_modules", "dist", ".cache"], "references": [ { @@ -21,6 +21,9 @@ { "path": "../../packages/agentic/desktop-driver/tsconfig.json" }, + { + "path": "../../packages/agentic/desktop-test/tsconfig.json" + }, { "path": "../../packages/agentic/native-core/tsconfig.json" }, diff --git a/apps/win32/desktop.config.mts b/apps/win32/desktop.config.mts new file mode 100644 index 0000000000..5e079b91ad --- /dev/null +++ b/apps/win32/desktop.config.mts @@ -0,0 +1,13 @@ +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + win32: { + host: { kind: 'rex', renderer: 'paper', component: 'FluentTester', windowTitle: 'FluentUI Tester' }, + launchTimeoutMs: 180_000, + commands: { run: { command: 'yarn', args: ['run-win32'] } }, + }, + }, +}); diff --git a/apps/win32/package.json b/apps/win32/package.json index 1bccb80f13..af8f8d78b3 100644 --- a/apps/win32/package.json +++ b/apps/win32/package.json @@ -22,6 +22,7 @@ "bundle": "rnx-cli bundle --dev false", "bundle-dev": "rnx-cli bundle", "clean": "fluentui-scripts clean", + "desktop-test": "desktop-test", "format": "fluentui-scripts format", "lint": "fluentui-scripts lint", "run-win32": "rex-win32 --bundle index.win32 --component FluentTester --basePath ./dist --useDirectDebugger --windowTitle \"FluentUI Tester\" --pluginProps --debugBundlePath index --jsEngine v8", @@ -41,6 +42,7 @@ "devDependencies": { "@babel/core": "catalog:", "@babel/runtime": "catalog:", + "@fluentui-react-native/desktop-test": "workspace:*", "@fluentui-react-native/scripts": "workspace:*", "@office-iss/rex-win32": "0.81.1", "@react-native-community/cli": "^20.0.0", diff --git a/apps/win32/tsconfig.json b/apps/win32/tsconfig.json index 931e1d606d..6035768d57 100644 --- a/apps/win32/tsconfig.json +++ b/apps/win32/tsconfig.json @@ -1,6 +1,9 @@ { "files": [], "references": [ + { + "path": "../../packages/agentic/desktop-test/tsconfig.json" + }, { "path": "../../scripts/tsconfig.json" }, diff --git a/driver-next-plan.md b/driver-next-plan.md index f19f45df1f..2bb5095921 100644 --- a/driver-next-plan.md +++ b/driver-next-plan.md @@ -16,10 +16,61 @@ each child process manages independently. ## Scope and evidence -This is the accepted architecture and delivery plan, not an implementation or a -native qualification report. New configuration examples, interfaces, and commands -below describe intended APIs; they do not exist yet. Outstanding implementation -gates do not reopen the selected package architecture. +This is the accepted architecture and delivery plan. The implementation update +below supersedes the original planning-time statements about missing APIs; +the remaining sections retain their research-time evidence and proposed +contracts. Current API usage belongs in the +[desktop-test guide](packages/agentic/desktop-test/README.md). +Outstanding qualification gates do not reopen the selected architecture. + +### Implementation update (October 22, 2026, UTC) + +**Implemented:** the neutral harness, structural same-schema target references, +native adapters, owned services/workers, authenticated reporting collector, +normal WDIO/Jasmine integration, native visibility/focus waits, and optional +Storybook integration. FluentTester, REX, Storybook, and E2E now have canonical +configs. Existing desktop grouped spec selections, Storybook exclusions, and +mobile lanes are preserved. The Windows baseline still fails closed on its two +unmatched patterns rather than claiming reduced coverage. + +**Observed:** all 15 harness contracts, all 175 Storybook package tests, and +the Windows native build/handshake/self-test contracts pass. The real Win32 +Pressable navigation pilot passes both existing tests. The behavioral cohort +now passes all 37 unchanged Button, Checkbox, Switch, directional FocusZone, +and Callout tests, including keyboard callbacks, checked/required state, +scrolling-dependent navigation, and popup opening. Its canonical report is +`apps/E2E/artifacts/desktop/6d08d3d3-84b8-4621-af17-5ddaf04a4875/run.json`. +The Win32 Storybook lifecycle also passes through the neutral CLI, including +its owned PowerShell path and inherited reporting: 152 story navigations and +both native Button callbacks pass. Its canonical report is +`apps/storybook/artifacts/desktop/35036f1a-94c4-49f7-ac62-52a0cbb195c3/run.json`. +Both runs have zero skipped tests, retries, infrastructure, reporting, or +cleanup failures. The verified Windows helper artifact is +`aa1852e48380e8bc9a6327debeba2e94305eae780c9b365d5b6261702ddd8185`. + +An earlier Storybook run explicitly skipped its pointer/screenshot/focus +callback when those capabilities were unavailable. The later interactive +pass does not attribute the availability change to the adapter fixes; no +input policy was relaxed. These are implementation and limited app evidence, +not complete behavioral parity. + +**Repository validation:** package checks, publishing, changesets, and all +changed Markdown links pass. After `yarn clean-lage-cache`, the final aggregate +passes with `yarn lage buildci --concurrency 4`. Initial runs exposed missing +app project references, the owner-approved Windows Knip workspace-path fix, +and a 30-second CLI-contract timeout at default aggregate concurrency. The +bounded rerun preserves all assertions and deadlines. Validation used a short +task-local temporary directory; the earlier default-TEMP inline-worker exit +remains undiagnosed. The separate opt-in Windows driver run passes 68 tests +with two macOS-only tests skipped. The aggregate Markdown-link task remains +blocked by Windows zx shell quoting; direct checks of all changed files pass. + +**Still open:** per-test assertion mapping and equivalent-build old/new comparisons; actual +FluentTester Windows and macOS renderer/runtime qualification; macOS native +compilation; all Storybook platform/renderer regressions; and real Actions +grouping/summary/upload verification. The 100-run/14-day reliability gate and +desktop Appium retirement cannot be completed by this implementation session. +No lane is retired and no capability skip is counted as a native pass. The review covers the checkout at `e2092f9df3fd4fafc141c696ca0e49e690d69cd1`. The legacy package is actually `apps/E2E`, with uppercase `E2E`. @@ -129,9 +180,9 @@ still applies; it is not an application-side Storybook handshake. ### There are two different test contracts -| Existing consumer | Execution contract | Preserve during migration | -| --- | --- | --- | -| `apps/E2E` | WDIO local runner, Jasmine, globals/hooks/retries, grouped spec arrays, page navigation | Normal WDIO spec execution and grouping | +| Existing consumer | Execution contract | Preserve during migration | +| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| `apps/E2E` | WDIO local runner, Jasmine, globals/hooks/retries, grouped spec arrays, page navigation | Normal WDIO spec execution and grouping | | Storybook callbacks | Extracted inline callbacks, source-digest validation, supervisor-created session, correlated navigation/remount, isolated `node:test` worker | Existing per-case isolation and explicit callback context | Sources: @@ -146,11 +197,11 @@ emulating its globals inside `node:test`, or rewriting all tests as stories. ## 2. Architecture decision -| Approach considered | Package structure | Primary advantage | Primary cost | Decision | -| --- | --- | --- | --- | --- | -| **1. Neutral harness + Storybook integration** | Driver + new generic harness + existing Storybook tooling + optional Storybook runtime | Real dependency/release boundaries; supports interactive and test clients | One additional package and a small integration contract | **Accepted; implement this** | -| **2. Generalized all-in-one harness** | Driver + renamed/generalized tooling package with lazy Storybook subpaths + optional runtime | Fewest release units and easiest initial extraction | Storybook-only tooling dependencies and releases remain shared | Not selected | -| **3. WDIO-launcher-centric** | Driver + WDIO desktop service/harness + Storybook adapter + optional runtime | Standard runner ecosystem becomes the organizing model | Story callback semantics and interactive lifecycle need adaptation | Not selected | +| Approach considered | Package structure | Primary advantage | Primary cost | Decision | +| ---------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------ | ---------------------------- | +| **1. Neutral harness + Storybook integration** | Driver + new generic harness + existing Storybook tooling + optional Storybook runtime | Real dependency/release boundaries; supports interactive and test clients | One additional package and a small integration contract | **Accepted; implement this** | +| **2. Generalized all-in-one harness** | Driver + renamed/generalized tooling package with lazy Storybook subpaths + optional runtime | Fewest release units and easiest initial extraction | Storybook-only tooling dependencies and releases remain shared | Not selected | +| **3. WDIO-launcher-centric** | Driver + WDIO desktop service/harness + Storybook adapter + optional runtime | Standard runner ecosystem becomes the organizing model | Story callback semantics and interactive lifecycle need adaptation | Not selected | The accepted design uses the single config format, native-only application support, and explicit platform/ownership contracts below. The other approaches @@ -216,12 +267,12 @@ authored config format. ## 3. Package responsibilities -| Package | Owns | Must not own | -| --- | --- | --- | -| `desktop-driver` | W3C remote end; target registration contracts; native helpers; `DesktopHost`; capabilities; clients; native assertions/artifacts; low-level WDIO command compatibility | App build commands, Metro, Storybook dependencies, Jasmine configuration | -| **New `desktop-test`** | Canonical config/loading; target and instance resolution; app/host adapters; helpers/listener lifecycle; launch/attach leases; readiness coordination; cancellation; WDIO launcher/worker service; cleanup; reporting protocol/collector, journals, result normalization, and console/Actions/JSON/JUnit renderers | Story discovery, CSF extraction, preview navigation, Storybook UI | -| `storybook-desktop` | Story config extension; discovery and digests; callback extraction/stripping; channel adapter/orchestrator; traversal; story executor; Storybook-specific diagnostic/result adapters into the harness | Another helper/server/app supervisor or independent top-level reporter | -| `storybook-desktop-runtime` | Storybook shell/theme/chrome; preview bridge; Storybook-specific Metro/Babel integration | Mandatory runtime for generic apps or ownership of host processes | +| Package | Owns | Must not own | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | +| `desktop-driver` | W3C remote end; target registration contracts; native helpers; `DesktopHost`; capabilities; clients; native assertions/artifacts; low-level WDIO command compatibility | App build commands, Metro, Storybook dependencies, Jasmine configuration | +| **New `desktop-test`** | Canonical config/loading; target and instance resolution; app/host adapters; helpers/listener lifecycle; launch/attach leases; readiness coordination; cancellation; WDIO launcher/worker service; cleanup; reporting protocol/collector, journals, result normalization, and console/Actions/JSON/JUnit renderers | Story discovery, CSF extraction, preview navigation, Storybook UI | +| `storybook-desktop` | Story config extension; discovery and digests; callback extraction/stripping; channel adapter/orchestrator; traversal; story executor; Storybook-specific diagnostic/result adapters into the harness | Another helper/server/app supervisor or independent top-level reporter | +| `storybook-desktop-runtime` | Storybook shell/theme/chrome; preview bridge; Storybook-specific Metro/Babel integration | Mandatory runtime for generic apps or ownership of host processes | Keep low-level WDIO native-command adaptation in `desktop-driver/wdio`; put WDIO launcher/framework/lifecycle integration in `desktop-test`. This avoids pulling @@ -276,15 +327,15 @@ Reject reference cycles and ambiguous target overrides. Keep these axes separate: -| Axis | Examples | Rule | -| --- | --- | --- | -| Endpoint | `macos`, `windows`, `win32` | Test-facing identity; Win32 is not an alias for the Windows Fabric endpoint | -| Host OS | macOS, Windows | Both Windows endpoints use the Windows helper | -| Host adapter | Native installed/executable, RN Test App, REX, external attach | Determines supported lifecycle stages, not test framework | -| Renderer | Fabric/Paper, when applicable | Optional for non-RN apps; actual generated/running mode must match | -| Application integration | None, Storybook | Optional; not inferred from test runner | -| Test source | WDIO specs, CSF callbacks, existing static story plans | Determines execution/navigation contract | -| Readiness | Verified native window/marker, optional authenticated integration | `/status` alone does not prove the app is ready | +| Axis | Examples | Rule | +| ----------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------- | +| Endpoint | `macos`, `windows`, `win32` | Test-facing identity; Win32 is not an alias for the Windows Fabric endpoint | +| Host OS | macOS, Windows | Both Windows endpoints use the Windows helper | +| Host adapter | Native installed/executable, RN Test App, REX, external attach | Determines supported lifecycle stages, not test framework | +| Renderer | Fabric/Paper, when applicable | Optional for non-RN apps; actual generated/running mode must match | +| Application integration | None, Storybook | Optional; not inferred from test runner | +| Test source | WDIO specs, CSF callbacks, existing static story plans | Determines execution/navigation contract | +| Readiness | Verified native window/marker, optional authenticated integration | `/status` alone does not prove the app is ready | ### Illustrative configs @@ -304,10 +355,7 @@ export default defineDesktopConfig({ win32: { host: { kind: 'rex' } }, }, integration: storybook({ - storyPackages: [ - '@fluentui-react-native/components', - '@fluentui-react-native/native-core', - ], + storyPackages: ['@fluentui-react-native/components', '@fluentui-react-native/native-core'], // Carry over current per-endpoint patterns, exclusions, and smoke checks. }), suites: { @@ -459,15 +507,15 @@ delegated commands inherit the invocation's reporting context. ### Exactly one owner per resource -| Resource | Owner | -| --- | --- | -| Helper resolution, verified artifact, W3C listener, target registry | Generic supervisor | -| Optional Metro/channel/build processes | Generic supervisor, from selected adapter/integration descriptors | -| Normal WDIO browser session | WDIO worker; supervisor tracks association and removes orphans | -| Story callback session | Supervisor session scope; callback worker attaches to that session | -| Supervisor-launched app | Supervisor lease; driver sessions attach | -| Session-launched app | Driver session; session deletion closes it | -| External attached app | External owner; ordinary cleanup preserves it | +| Resource | Owner | +| ------------------------------------------------------------------- | ------------------------------------------------------------------ | +| Helper resolution, verified artifact, W3C listener, target registry | Generic supervisor | +| Optional Metro/channel/build processes | Generic supervisor, from selected adapter/integration descriptors | +| Normal WDIO browser session | WDIO worker; supervisor tracks association and removes orphans | +| Story callback session | Supervisor session scope; callback worker attaches to that session | +| Supervisor-launched app | Supervisor lease; driver sessions attach | +| Session-launched app | Driver session; session deletion closes it | +| External attached app | External owner; ordinary cleanup preserves it | **Initial application-lifetime policy:** retain the existing supervisor-owned app and attached-session model for Storybook. Use @@ -543,14 +591,14 @@ The owner's reported symptom is consistent with the following source-inspected gaps. This research did not reproduce a specific lost-output incident or prove that every hidden diagnostic originates from the same path. -| Observed path | Current behavior | Consequence | -| --- | --- | --- | -| `NodeDesktopCommandRunner` | Sends child stdout/stderr to one retained file; replays on failure or in verbose mode when reporting/stopping the command | A live service can write useful output that never reaches the terminal until shutdown. Verbose is replay, not live forwarding. | -| Combined Storybook server | Forwards native stderr to its own stderr; channel and WebDriver startup messages use its stdout | Forwarding one process hop is insufficient when the outer runner captures that stream. | -| `driver()` lifecycle | Starts Metro separately, then awaits the server command; Metro completion is not raced against the server for ongoing supervision | A delegated service's failure report and the parent's actual lifetime are not one coherent health/result contract. | -| Windows/Win32 smoke wrappers | `Start-Process` redirects delegated commands to separate `.out.log`/`.err.log` files | The generic command runner cannot automatically surface those grandchildren's diagnostic files; file retention alone is not console forwarding. | -| Existing grouping | Per-runner `reportQueue` serializes textual `BEGIN`/`END` replay | Good local ordering, but neither Actions `group` commands nor a hierarchy shared across nested processes/runners. | -| Test workers and aggregate reports | Worker result files carry pass/skip/failure; console reporter output is a separate captured subprocess stream; aggregate status is based on presence of failures | Rendering, structured outcomes, and lifecycle are separate flows. All-skipped work can still yield a top-level `passed` label. | +| Observed path | Current behavior | Consequence | +| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| `NodeDesktopCommandRunner` | Sends child stdout/stderr to one retained file; replays on failure or in verbose mode when reporting/stopping the command | A live service can write useful output that never reaches the terminal until shutdown. Verbose is replay, not live forwarding. | +| Combined Storybook server | Forwards native stderr to its own stderr; channel and WebDriver startup messages use its stdout | Forwarding one process hop is insufficient when the outer runner captures that stream. | +| `driver()` lifecycle | Starts Metro separately, then awaits the server command; Metro completion is not raced against the server for ongoing supervision | A delegated service's failure report and the parent's actual lifetime are not one coherent health/result contract. | +| Windows/Win32 smoke wrappers | `Start-Process` redirects delegated commands to separate `.out.log`/`.err.log` files | The generic command runner cannot automatically surface those grandchildren's diagnostic files; file retention alone is not console forwarding. | +| Existing grouping | Per-runner `reportQueue` serializes textual `BEGIN`/`END` replay | Good local ordering, but neither Actions `group` commands nor a hierarchy shared across nested processes/runners. | +| Test workers and aggregate reports | Worker result files carry pass/skip/failure; console reporter output is a separate captured subprocess stream; aggregate status is based on presence of failures | Rendering, structured outcomes, and lifecycle are separate flows. All-skipped work can still yield a top-level `passed` label. | Source anchors: @@ -741,18 +789,18 @@ package is required. Use deterministic fake services/workers before native qualification: -| Fixture | Required result | -| --- | --- | -| Long-running service writes a diagnostic after readiness | Source-attributed diagnostic becomes visible before stop, without verbose | -| Metro/helper/channel fails while outer server remains alive | Parent fails the correct phase, aborts dependent work, and reports the original cause | +| Fixture | Required result | +| ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | +| Long-running service writes a diagnostic after readiness | Source-attributed diagnostic becomes visible before stop, without verbose | +| Metro/helper/channel fails while outer server remains alive | Parent fails the correct phase, aborts dependent work, and reports the original cause | | Direct child and grandchild interleave stdout/stderr, partial UTF-8, and late output | Durable captures are complete; source/stream metadata is preserved; only collector arrival order is claimed across streams | -| Two producers finish together | Each Actions group is contiguous/balanced; unrelated service output is not attributed to a test | -| Child prints workflow-command-shaped text | No unauthorized annotation/group/runner command is executed | -| Worker fails, times out, crashes, or omits a result; setup hook fails | Same non-passing outcome/cause appears in console, JSON, JUnit, and summary | -| All tests skip, capability is unsupported, or selection is empty | No success-shaped `passed` aggregate; required/optional policy is explicit | -| Retry passes after failure | Both attempts and flaky status survive; totals do not double-count final tests | -| Slow sink, write failure, cancellation, or abrupt service exit | Bounded completion, explicit reporting failure/incomplete state, retained evidence, no leaked owned resources | -| Structured CLI mode under Actions | Machine stdout remains parseable and free of progress, ANSI, or group commands | +| Two producers finish together | Each Actions group is contiguous/balanced; unrelated service output is not attributed to a test | +| Child prints workflow-command-shaped text | No unauthorized annotation/group/runner command is executed | +| Worker fails, times out, crashes, or omits a result; setup hook fails | Same non-passing outcome/cause appears in console, JSON, JUnit, and summary | +| All tests skip, capability is unsupported, or selection is empty | No success-shaped `passed` aggregate; required/optional policy is explicit | +| Retry passes after failure | Both attempts and flaky status survive; totals do not double-count final tests | +| Slow sink, write failure, cancellation, or abrupt service exit | Bounded completion, explicit reporting failure/incomplete state, retained evidence, no leaked owned resources | +| Structured CLI mode under Actions | Machine stdout remains parseable and free of progress, ANSI, or group commands | Then exercise the same adapters on macOS, Windows, and Win32, including the transitional PowerShell path. Test the emitted Actions command stream locally @@ -775,13 +823,13 @@ Research checked October 21, 2026: ## 7. Platform/application boundaries -| Endpoint | Existing legacy app | Required treatment | -| --- | --- | --- | -| macOS | `apps/fluent-tester`, RN Test App | Verify actual renderer/build identity; do not infer runtime mode from a Pod entry | -| Windows | `apps/fluent-tester`, RN Test App | Verify generated Fabric setting and packaged identity; do not presume UWP from old docs | -| Win32 | `apps/win32`, prebuilt REX/Paper | Windows helper with `win32` endpoint; preserve cwd/args/title and bundle-driven launch; native app build remains unsupported | -| Storybook macOS | `apps/storybook` | Preserve Fabric and Paper lanes and prep-on-renderer-switch | -| Storybook Windows/Win32 | `apps/storybook` | Preserve distinct identities, platform exclusions, host-specific chrome and launch behavior | +| Endpoint | Existing legacy app | Required treatment | +| ----------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| macOS | `apps/fluent-tester`, RN Test App | Verify actual renderer/build identity; do not infer runtime mode from a Pod entry | +| Windows | `apps/fluent-tester`, RN Test App | Verify generated Fabric setting and packaged identity; do not presume UWP from old docs | +| Win32 | `apps/win32`, prebuilt REX/Paper | Windows helper with `win32` endpoint; preserve cwd/args/title and bundle-driven launch; native app build remains unsupported | +| Storybook macOS | `apps/storybook` | Preserve Fabric and Paper lanes and prep-on-renderer-switch | +| Storybook Windows/Win32 | `apps/storybook` | Preserve distinct identities, platform exclusions, host-specific chrome and launch behavior | **Important reciprocal-review correction:** omitting `--use-fabric` does not establish that FluentTester Windows is UWP/Paper. The pinned RN Test App 5.3.3 @@ -814,18 +862,18 @@ Sources: The WebDriver wire protocol is not an Appium semantic compatibility guarantee. Inventory actual selected tests and their command paths before changing them. -| Area | Observed mismatch | Recommended handling | -| --- | --- | --- | +| Area | Observed mismatch | Recommended handling | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **WDIO visibility/focus** | With WDIO 9.24 and current desktop capabilities, `isDisplayed` takes the browser JavaScript path; `isFocused` also executes JavaScript. The server rejects `execute`. `waitForDisplayed` is used during initial page readiness. | Central native-command compatibility installation for `remote`, `attach`, and WDIO-managed workers. Route visibility/focus to supported native queries. Test waits and assertion consumers. Do not spoof Appium identity or enable arbitrary execution. | -| Selectors | macOS hooks use XCUI XPath; driver supports accessibility ID, role/tag, accessible name/link, and text, not XPath | Replace accidental platform-specific hooks with supported selectors/window behavior. Implement a new selector only when required by real test intent. | -| Attributes/state | Legacy names such as `Name`, `ControlType`, `HasKeyboardFocus`, and `Toggle.ToggleState` differ from normalized properties and types; unknown attributes return `null` | Migrate a shared page-object adapter with explicit mappings. Add missing justified native state contracts; preserve provider-specific accessibility assertions rather than normalizing them away. | -| Checked vs selected | Legacy mode switch uses `isSelected`; UIA toggle checkedness is distinct from SelectionItem state | Change toggle assertions to `checked`. Do not globally redefine `selected` as checkedness. | -| Keyboard | `keys.join()` introduces commas; typed units press/release modifiers rather than preserving a chord; disabled-element send-keys may fail focus validation | Replace chords with explicit W3C key actions and releases; test cancellation/repetition/Unicode. Decide and document element-send-keys modifier semantics separately. | -| Window sizing | Legacy setup calls `maximizeWindow`; driver routes currently support handles/switch/close/rect, not maximize | Implement and qualify maximize only if needed, or explicitly change setup intent. Setting a rectangle is not automatically maximize. | -| Popups/focus | Search is current-window scoped; legacy page objects often assume root-wide lookup or treat extra windows as assertion failures | Preserve explicit window scope; update tests to identify popup ownership and restore focus. Measure before adding optional app-wide lookup. | -| Desktop scrolling | Windows navigation can reach a helper whose default branch emits Android UiScrollable selectors | Separate desktop navigation/scrolling from mobile helpers. Keep mobile paths and dependencies. | -| Runner grouping | Legacy hooks, globals, retries, Jasmine matchers, and suite state differ from story callbacks | Preserve WDIO/Jasmine and grouped arrays; change transport and semantics incrementally. | -| REX startup | Legacy config supplies working directory; native application descriptor lacks it | Add trusted adapter/server-owned cwd or supervisor launch with an exact attach lease; qualify actual argument/cwd requirements. | +| Selectors | macOS hooks use XCUI XPath; driver supports accessibility ID, role/tag, accessible name/link, and text, not XPath | Replace accidental platform-specific hooks with supported selectors/window behavior. Implement a new selector only when required by real test intent. | +| Attributes/state | Legacy names such as `Name`, `ControlType`, `HasKeyboardFocus`, and `Toggle.ToggleState` differ from normalized properties and types; unknown attributes return `null` | Migrate a shared page-object adapter with explicit mappings. Add missing justified native state contracts; preserve provider-specific accessibility assertions rather than normalizing them away. | +| Checked vs selected | Legacy mode switch uses `isSelected`; UIA toggle checkedness is distinct from SelectionItem state | Change toggle assertions to `checked`. Do not globally redefine `selected` as checkedness. | +| Keyboard | `keys.join()` introduces commas; typed units press/release modifiers rather than preserving a chord; disabled-element send-keys may fail focus validation | Replace chords with explicit W3C key actions and releases; test cancellation/repetition/Unicode. Decide and document element-send-keys modifier semantics separately. | +| Window sizing | Legacy setup calls `maximizeWindow`; driver routes currently support handles/switch/close/rect, not maximize | Implement and qualify maximize only if needed, or explicitly change setup intent. Setting a rectangle is not automatically maximize. | +| Popups/focus | Search is current-window scoped; legacy page objects often assume root-wide lookup or treat extra windows as assertion failures | Preserve explicit window scope; update tests to identify popup ownership and restore focus. Measure before adding optional app-wide lookup. | +| Desktop scrolling | Windows navigation can reach a helper whose default branch emits Android UiScrollable selectors | Separate desktop navigation/scrolling from mobile helpers. Keep mobile paths and dependencies. | +| Runner grouping | Legacy hooks, globals, retries, Jasmine matchers, and suite state differ from story callbacks | Preserve WDIO/Jasmine and grouped arrays; change transport and semantics incrementally. | +| REX startup | Legacy config supplies working directory; native application descriptor lacks it | Add trusted adapter/server-owned cwd or supervisor launch with an exact attach lease; qualify actual argument/cwd requirements. | Key source anchors: @@ -860,14 +908,14 @@ values or broad compatibility shims. ## 9. Phased delivery and gates -| Phase | Owning surface | Deliverable | Exit gate | -| --- | --- | --- | --- | -| **0. Baseline and probes** | E2E + native driver + current supervisor | Inventory exact selected cases, commands, state assertions, grouping, existing failures, renderer, host identity, input policy, and every delegated output path | Recorded baseline and blocker list per endpoint; confirm Windows generated renderer and capture a reporting-gap fixture | -| **1. Generic boundary and reporting spine** | `desktop-test` + driver | Canonical schema/resolver, controlled native target composition, adapters, ownership/readiness, real `serve`, event collector/journal, live diagnostics, one renderer and flush contract | Plain app works without Storybook; fake-service tests prove child/grandchild forwarding, required-service failure, bounded shutdown, and isolated Actions commands/groups | -| **2. WDIO compatibility/pilot** | Driver `/wdio` + harness + E2E | WDIO/Jasmine launcher and reporter adapter, visibility fixes, shared page-object adaptation, representative specs, canonical JSON/JUnit/summary | Pilot passes with required capabilities; grouped output/result parity and retry/skip/hook/crash contracts pass; no blocked sessions/input/app leaks | -| **3. Storybook as consumer** | Storybook tooling/runtime/app | Same schema; optional integration; one supervisor and collector; existing traversal/CSF lifecycle and normalized results | macOS Fabric/Paper, Windows, Win32 behavior and live diagnostics preserved, including nested PowerShell paths; no weakened readiness or exclusions | -| **4. Comparative migration** | E2E apps + CI | Old/new lanes on equivalent app builds, explicit test mapping, semantic comparisons, Actions summaries and always-uploaded artifacts | Each endpoint meets parity/reliability gates; real Actions grouping, annotations, summary, and artifact references verified | -| **5. Retirement** | E2E + CI + docs | Remove obsolete desktop transport configs/scripts/dependencies | No remaining desktop consumer depends on old transport; mobile still works | +| Phase | Owning surface | Deliverable | Exit gate | +| ------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **0. Baseline and probes** | E2E + native driver + current supervisor | Inventory exact selected cases, commands, state assertions, grouping, existing failures, renderer, host identity, input policy, and every delegated output path | Recorded baseline and blocker list per endpoint; confirm Windows generated renderer and capture a reporting-gap fixture | +| **1. Generic boundary and reporting spine** | `desktop-test` + driver | Canonical schema/resolver, controlled native target composition, adapters, ownership/readiness, real `serve`, event collector/journal, live diagnostics, one renderer and flush contract | Plain app works without Storybook; fake-service tests prove child/grandchild forwarding, required-service failure, bounded shutdown, and isolated Actions commands/groups | +| **2. WDIO compatibility/pilot** | Driver `/wdio` + harness + E2E | WDIO/Jasmine launcher and reporter adapter, visibility fixes, shared page-object adaptation, representative specs, canonical JSON/JUnit/summary | Pilot passes with required capabilities; grouped output/result parity and retry/skip/hook/crash contracts pass; no blocked sessions/input/app leaks | +| **3. Storybook as consumer** | Storybook tooling/runtime/app | Same schema; optional integration; one supervisor and collector; existing traversal/CSF lifecycle and normalized results | macOS Fabric/Paper, Windows, Win32 behavior and live diagnostics preserved, including nested PowerShell paths; no weakened readiness or exclusions | +| **4. Comparative migration** | E2E apps + CI | Old/new lanes on equivalent app builds, explicit test mapping, semantic comparisons, Actions summaries and always-uploaded artifacts | Each endpoint meets parity/reliability gates; real Actions grouping, annotations, summary, and artifact references verified | +| **5. Retirement** | E2E + CI + docs | Remove obsolete desktop transport configs/scripts/dependencies | No remaining desktop consumer depends on old transport; mobile still works | Recommended pilot: Win32 Button plus Checkbox/Switch state, a keyboard/focus case, a scrolling case, and a popup/window case. Then establish macOS and Windows parity @@ -888,16 +936,16 @@ Source: `packages/agentic/desktop-driver/PLAN.md:112-115`. ### Validation and CI ownership -| Layer | Required evidence | -| --- | --- | -| Driver | W3C/fake-host contracts; native WDIO visibility/focus/waits; unsupported state; stale elements; target reservation and cancellation | -| Native helpers | Matching-OS builds/handshakes; verified artifacts; launch/attach identity; input/state/capture; crash and orphan recovery | -| Generic harness | No-Storybook consumer installation/import test; same-schema references; config-relative paths; optional stages; session/worker cleanup; external-app preservation | -| WDIO integration | Real Jasmine/global/hook/group/retry behavior; no duplicate sessions; worker crash/timeout/interrupt; reporters and artifact failure paths | -| Storybook integration/runtime | Existing extraction/digests/static plans; authenticated readiness and stale-message rejection; navigation/remount; worker failure/timeout contracts; renderer switching | -| Reporting | Section 6 fixtures; cross-process attribution; live service diagnostics; required-service health; complete drain/flush; result parity across sinks; balanced Actions groups; safe raw replay; incomplete/all-skipped handling | -| App lanes | Existing FluentTester macOS/Windows and REX Win32 builds plus selected legacy assertions; separate Storybook platform/renderer regressions | -| Interactive qualification | Physical input, focus, popup windows, wheel/DPI, capture/occlusion, cross-supervisor contention, repeated reliability and zero leaks | +| Layer | Required evidence | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Driver | W3C/fake-host contracts; native WDIO visibility/focus/waits; unsupported state; stale elements; target reservation and cancellation | +| Native helpers | Matching-OS builds/handshakes; verified artifacts; launch/attach identity; input/state/capture; crash and orphan recovery | +| Generic harness | No-Storybook consumer installation/import test; same-schema references; config-relative paths; optional stages; session/worker cleanup; external-app preservation | +| WDIO integration | Real Jasmine/global/hook/group/retry behavior; no duplicate sessions; worker crash/timeout/interrupt; reporters and artifact failure paths | +| Storybook integration/runtime | Existing extraction/digests/static plans; authenticated readiness and stale-message rejection; navigation/remount; worker failure/timeout contracts; renderer switching | +| Reporting | Section 6 fixtures; cross-process attribution; live service diagnostics; required-service health; complete drain/flush; result parity across sinks; balanced Actions groups; safe raw replay; incomplete/all-skipped handling | +| App lanes | Existing FluentTester macOS/Windows and REX Win32 builds plus selected legacy assertions; separate Storybook platform/renderer regressions | +| Interactive qualification | Physical input, focus, popup windows, wheel/DPI, capture/occlusion, cross-supervisor contention, repeated reliability and zero leaks | Reuse `.github/actions/setup-desktop-driver`; its workspace input is already general enough for non-Storybook consumers. Keep explicit helper build/doctor and @@ -944,36 +992,36 @@ need to combine a wholesale directory relocation with the architectural split. ## 10. Remaining implementation decisions and risks -| Decision/risk | Plan/default | Evidence needed to change it | -| --- | --- | --- | -| Package architecture | **Settled: Option 1**, neutral harness plus separate Storybook integration | A new explicit owner decision; not an implicit implementation simplification | -| App lifetime across story sessions | Existing supervisor lease + attach | A separately designed driver keepalive contract with crash/relaunch/teardown coverage | -| Element-send-keys modifiers | Migrate chords to actions now; specify sticky-modifier support or explicit rejection separately | W3C/native conformance cases; no silent press/release mismatch | -| Popup lookup | Explicit identified-window switching | Measured need for opt-in app-wide scope without ambiguous lookup or stale references | -| Unknown attributes/provider state | Keep protocol behavior explicit; migrate supported semantic mappings and report unsupported state | A justified namespaced provider-state or new normalized-property contract | -| FluentTester Windows renderer | Verify freshly generated `UseFabric`; do not assume UWP | Generated project plus actual launched runtime/build evidence | -| Windows attach identity | Exact verified lease or strong executable/AUMID identity | Duplicate-title, PID-reuse, package-collision, and external-app preservation tests | -| Generic runtime | None initially | A real second instrumented consumer needing authenticated JS/render milestones | -| Physical input on hosted CI | Do not count disabled-input lanes as qualification | Owner-approved policy plus repeatable native input evidence | -| Reporting | One collector/renderer in the harness, typed producer adapters, durable results and Actions groups | Implementation fixtures close transport/flush details; logging-library choice must preserve the reporting contract | +| Decision/risk | Plan/default | Evidence needed to change it | +| ---------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | +| Package architecture | **Settled: Option 1**, neutral harness plus separate Storybook integration | A new explicit owner decision; not an implicit implementation simplification | +| App lifetime across story sessions | Existing supervisor lease + attach | A separately designed driver keepalive contract with crash/relaunch/teardown coverage | +| Element-send-keys modifiers | Migrate chords to actions now; specify sticky-modifier support or explicit rejection separately | W3C/native conformance cases; no silent press/release mismatch | +| Popup lookup | Explicit identified-window switching | Measured need for opt-in app-wide scope without ambiguous lookup or stale references | +| Unknown attributes/provider state | Keep protocol behavior explicit; migrate supported semantic mappings and report unsupported state | A justified namespaced provider-state or new normalized-property contract | +| FluentTester Windows renderer | Verify freshly generated `UseFabric`; do not assume UWP | Generated project plus actual launched runtime/build evidence | +| Windows attach identity | Exact verified lease or strong executable/AUMID identity | Duplicate-title, PID-reuse, package-collision, and external-app preservation tests | +| Generic runtime | None initially | A real second instrumented consumer needing authenticated JS/render milestones | +| Physical input on hosted CI | Do not count disabled-input lanes as qualification | Owner-approved policy plus repeatable native input evidence | +| Reporting | One collector/renderer in the harness, typed producer adapters, durable results and Actions groups | Implementation fixtures close transport/flush details; logging-library choice must preserve the reporting contract | The architecture decision is closed. The remaining rows are implementation contracts or qualification gates, not competing package plans. ## 11. Research provenance and reciprocal-review reconciliation -| Topic | Initial disagreement or omission | Reconciled plan | -| --- | --- | --- | -| Packaging | Opus preferred three packages; Astra preferred four | Both now prefer four because lazy subpaths do not isolate installed dependencies/releases | -| WDIO native behavior | Astra initially missed visibility/focus JavaScript execution | Both verified it; command compatibility is an early migration blocker | -| Windows renderer | Opus inferred UWP from omitted flag; Astra initially treated it as unknown | Both checked the pinned generator and retracted the UWP inference; verify fresh generated/running mode | -| Config ownership | Astra's initial app configs imported E2E suite definitions | Test-to-app dependency direction; same-format app targets referenced by the suite config | -| Runtime extraction | Astra floated a generic runtime entry point | Defer it; ordinary native apps need no runtime | -| Selected vs checked | Opus proposed a driver fallback | Withdrawn; adapt toggle tests to checked semantics | -| Windows scrolling | Opus initially considered mobile branches unreachable on desktop | Corrected: Windows can enter Android-specific scrolling; separate the paths | -| WDIO-centric alternative | Opus's initial alternative assumed two authored config formats | Strengthened Option 3 preserves the single canonical format with generated WDIO config | -| App keepalive | Driver-retained service lifetime vs supervisor ownership | Preserve current leases initially; no incidental driver lifetime redesign | -| Qualification | One initial proposal suggested a smaller run window | Existing 100-run/14-day gate remains authoritative | +| Topic | Initial disagreement or omission | Reconciled plan | +| ------------------------ | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| Packaging | Opus preferred three packages; Astra preferred four | Both now prefer four because lazy subpaths do not isolate installed dependencies/releases | +| WDIO native behavior | Astra initially missed visibility/focus JavaScript execution | Both verified it; command compatibility is an early migration blocker | +| Windows renderer | Opus inferred UWP from omitted flag; Astra initially treated it as unknown | Both checked the pinned generator and retracted the UWP inference; verify fresh generated/running mode | +| Config ownership | Astra's initial app configs imported E2E suite definitions | Test-to-app dependency direction; same-format app targets referenced by the suite config | +| Runtime extraction | Astra floated a generic runtime entry point | Defer it; ordinary native apps need no runtime | +| Selected vs checked | Opus proposed a driver fallback | Withdrawn; adapt toggle tests to checked semantics | +| Windows scrolling | Opus initially considered mobile branches unreachable on desktop | Corrected: Windows can enter Android-specific scrolling; separate the paths | +| WDIO-centric alternative | Opus's initial alternative assumed two authored config formats | Strengthened Option 3 preserves the single canonical format with generated WDIO config | +| App keepalive | Driver-retained service lifetime vs supervisor ownership | Preserve current leases initially; no incidental driver lifetime redesign | +| Qualification | One initial proposal suggested a smaller run window | Existing 100-run/14-day gate remains authoritative | **Provenance:** local artifacts `opus-driver-initial.md`, `astra-driver-initial.md`, `opus-driver-cross-review.md`, and @@ -984,7 +1032,7 @@ reviews, not just their displayed summaries. formal reporting research. Section 6 is follow-up source/documentation research, not a claim that both original models re-reviewed the new reporting design. -**Validation status:** repository source, relevant installed dependency +**Planning-time validation status:** repository source, relevant installed dependency implementations, tests, CI definitions, and upstream reporting documentation were inspected. Application launches, builds, native tests, reporting fixtures, and parity/reliability runs were **not run**; no implementation, diff --git a/knip.config.ts b/knip.config.ts index 95934217b7..b48dabd8a1 100644 --- a/knip.config.ts +++ b/knip.config.ts @@ -14,7 +14,7 @@ const config = async (): Promise => { // now fill in the details for those const workspaceDetails = Object.fromEntries( workspaceDirs.map((dir: string) => { - const relativePath = path.posix.normalize(path.relative(process.cwd(), dir)); + const relativePath = path.posix.normalize(path.relative(process.cwd(), dir).split(path.sep).join('/')); if (relativePath === '.') { // skip the root dir return [relativePath, {}]; diff --git a/packages/agentic/desktop-driver/README.md b/packages/agentic/desktop-driver/README.md index cb4023485f..34e20e4980 100644 --- a/packages/agentic/desktop-driver/README.md +++ b/packages/agentic/desktop-driver/README.md @@ -103,6 +103,11 @@ lifecycle. The standalone `desktop-driver serve` command intentionally exposes a fake target for protocol and integration tests; it is not a native service shortcut. +For a production native listener, application lifecycle, and standalone WDIO +suites, use the optional +[`desktop-test` harness](../desktop-test/README.md). It composes this driver +without requiring Storybook or React Native in ordinary applications. + See [Getting started](references/getting-started.md) for installation, toolchains, first builds, and diagnostic commands. diff --git a/packages/agentic/desktop-driver/native/PROTOCOL.md b/packages/agentic/desktop-driver/native/PROTOCOL.md index 267fdfb7d0..a2c05448f5 100644 --- a/packages/agentic/desktop-driver/native/PROTOCOL.md +++ b/packages/agentic/desktop-driver/native/PROTOCOL.md @@ -106,6 +106,19 @@ Parameters are command-specific serializable data already validated by the Node boundary. Public WebDriver element UUIDs are resolved before dispatch; only helper-private opaque IDs cross FDR1. +The `launch` request includes `launchTimeoutMs` when explicitly configured on +the registered target. Both native providers validate its positive bounded +integer value, otherwise use their 90-second launch default, and honor the +cancellation token while discovering the application/window. Windows +application descriptors may include an existing absolute `workingDirectory` +for unpackaged process launch; packaged activation rejects that override. +These fields are trusted supervisor configuration, not WebDriver capabilities. + +Element snapshots can additionally include `required` as a supported Boolean +or an unsupported value with its reason. Older or non-supporting providers may +omit it; the Node boundary treats absence as unsupported. This is additive +snapshot metadata and does not change the FDR1 framing or wire version. + Request and cancellation correlation IDs are limited to 256 UTF-8 bytes. Helpers ignore invalid IDs before queuing work or echoing a response. diff --git a/packages/agentic/desktop-driver/native/macos/README.md b/packages/agentic/desktop-driver/native/macos/README.md index be6df555b0..779ac1ddac 100644 --- a/packages/agentic/desktop-driver/native/macos/README.md +++ b/packages/agentic/desktop-driver/native/macos/README.md @@ -169,6 +169,13 @@ timestamps allow only the bounded precision required by `NSRunningApplication.launchDate`. Zero or ambiguous application/window matches fail. +Launch honors an explicitly configured `launchTimeoutMs`, or uses the native +90-second default within the server's 120-second default deadline. If LaunchServices +completes after timeout or cancellation, the late completion closes only a +new application causally owned by that request, never a preexisting app. +This changed timeout path still needs matching-macOS build and runtime +qualification; Windows tests do not validate it. + Leases record whether the app was launched or attached. Session deletion preserves attached apps and closes only an exact process owned by the helper. Opaque helper IDs represent windows and AX elements; native references never diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Application.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Application.swift index a917c63983..0975d4dadd 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Application.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Application.swift @@ -88,6 +88,7 @@ private final class LaunchResultBox: @unchecked Sendable { let lock = NSLock() var application: NSRunningApplication? var error: Error? + var abandoned = false } struct AXWindowAttributeDiagnostic { @@ -306,6 +307,16 @@ final class ApplicationManager { func launch(_ params: JSONObject, token: CancellationToken) throws -> ApplicationLeaseRecord { let application = try requireObject(params["application"], "The launch request is missing its application descriptor.") + var launchTimeout = 90_000 + if let value = params["launchTimeoutMs"] { + guard let number = value as? NSNumber, !isBoolean(number), + number.doubleValue.isFinite, number.doubleValue >= 1, number.doubleValue <= 2_147_483_647, + number.doubleValue == Double(number.intValue) + else { + try fail(ErrorCode.invalidParams, "The registered launch timeout must be a positive bounded integer.") + } + launchTimeout = number.intValue + } try failIfSessionUnavailable("Launching the application") let bundleIdentifier = optionalString(application, "bundleIdentifier") let executablePath = optionalString(application, "executablePath") @@ -335,7 +346,9 @@ final class ApplicationManager { "The selected application bundle does not have the expected bundle identifier \"\(bundleIdentifier)\"." ) } - running = try launchApplication(at: appURL, arguments: arguments, token: token) + running = try launchApplication( + at: appURL, arguments: arguments, preexisting: preexisting, timeoutMilliseconds: launchTimeout, token: token + ) if preexisting.contains(running.processIdentifier) { try fail( ErrorCode.launchFailed, @@ -370,7 +383,7 @@ final class ApplicationManager { _ = try waitForUniqueWindow( processID: running.processIdentifier, exactTitle: windowTitle, - timeoutMilliseconds: 90_000, + timeoutMilliseconds: launchTimeout, token: token, code: ErrorCode.launchFailed ) @@ -1013,6 +1026,8 @@ final class ApplicationManager { private func launchApplication( at url: URL, arguments: [String], + preexisting: Set, + timeoutMilliseconds: Int, token: CancellationToken ) throws -> NSRunningApplication { let configuration = NSWorkspace.OpenConfiguration() @@ -1026,11 +1041,30 @@ final class ApplicationManager { result.lock.lock() result.application = application result.error = error + let abandoned = result.abandoned result.lock.unlock() + if abandoned, let application, !preexisting.contains(application.processIdentifier) { + application.terminate() + } semaphore.signal() } - while semaphore.wait(timeout: .now() + .milliseconds(50)) == .timedOut { - try token.throwIfCancelled() + let deadline = DispatchTime.now() + .milliseconds(timeoutMilliseconds) + do { + while semaphore.wait(timeout: .now() + .milliseconds(50)) == .timedOut { + try token.throwIfCancelled() + if DispatchTime.now() >= deadline { + try fail(ErrorCode.launchFailed, "Launch Services exceeded the registered launch timeout.") + } + } + } catch { + result.lock.lock() + result.abandoned = true + let application = result.application + result.lock.unlock() + if let application, !preexisting.contains(application.processIdentifier) { + application.terminate() + } + throw error } result.lock.lock() let application = result.application diff --git a/packages/agentic/desktop-driver/native/windows/README.md b/packages/agentic/desktop-driver/native/windows/README.md index c0ac52e49c..a3b56ddb28 100644 --- a/packages/agentic/desktop-driver/native/windows/README.md +++ b/packages/agentic/desktop-driver/native/windows/README.md @@ -62,8 +62,17 @@ Launch requires: - an exact `windowTitle`; - optional fixed, server-owned arguments. -Attach accepts an owner-generated nonce-bound lease, or an exact title only -when it resolves to one live process/window. Zero or multiple matches fail. +Unpackaged launch also accepts a server-owned `workingDirectory`, which must +be an existing absolute directory. Packaged activation rejects that override. +The helper validates and honors the server-owned `launchTimeoutMs`. + +Snapshots expose required-form state from cached `UIA_IsRequiredForFormPropertyId` +with provider defaults disabled. A missing, unsupported, or non-Boolean native +value remains unsupported rather than becoming `false`. + +Attach accepts an owner-generated nonce-bound lease, or an exact title +qualified by the canonical executable or AUMID. Title-only attach, zero +matches, and multiple qualified matches fail. Leases record ownership, PID, process start, executable, and the primary window. Attached apps survive WebDriver deletion; launched apps are closed only by their exact recorded process identity. diff --git a/packages/agentic/desktop-driver/native/windows/src/application.cpp b/packages/agentic/desktop-driver/native/windows/src/application.cpp index 90045141ae..5674a7e41c 100644 --- a/packages/agentic/desktop-driver/native/windows/src/application.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/application.cpp @@ -5,7 +5,9 @@ #include #include +#include #include +#include #include #include #include @@ -696,9 +698,24 @@ const ApplicationLease& ApplicationManager::Launch(const json::Value& params, co if (application == nullptr || !application->IsObject()) { Fail(kErrorInvalidParams, "The launch request is missing its application descriptor."); } + const json::Value* timeout = params.Find("launchTimeoutMs"); + const double configuredTimeout = params.NumberField("launchTimeoutMs", kLaunchTimeoutMs); + if ((timeout != nullptr && !timeout->IsNumber()) || !std::isfinite(configuredTimeout) || + configuredTimeout < 1 || configuredTimeout > 2147483647 || std::floor(configuredTimeout) != configuredTimeout) { + Fail(kErrorInvalidParams, "The registered launch timeout must be a positive bounded integer."); + } const std::wstring aumid = application->WideField("aumid"); const std::wstring configuredExecutablePath = application->WideField("executablePath"); const std::wstring windowTitle = application->WideField("windowTitle"); + const std::wstring workingDirectory = application->WideField("workingDirectory"); + if (!workingDirectory.empty() && + (!std::filesystem::path(workingDirectory).is_absolute() || + !std::filesystem::is_directory(workingDirectory))) { + Fail(kErrorInvalidParams, "The application workingDirectory must name an existing absolute directory."); + } + if (!workingDirectory.empty() && !aumid.empty()) { + Fail(kErrorInvalidParams, "Packaged activation does not support a workingDirectory override."); + } if (aumid.empty() && configuredExecutablePath.empty()) { Fail(kErrorInvalidParams, "The application descriptor needs an \"aumid\" or an \"executablePath\"."); } @@ -772,7 +789,8 @@ const ApplicationLease& ApplicationManager::Launch(const json::Value& params, co STARTUPINFOW startup{}; startup.cb = sizeof(startup); PROCESS_INFORMATION information{}; - if (CreateProcessW(executablePath.c_str(), commandLine.data(), nullptr, nullptr, FALSE, 0, nullptr, nullptr, + if (CreateProcessW(executablePath.c_str(), commandLine.data(), nullptr, nullptr, FALSE, 0, nullptr, + workingDirectory.empty() ? nullptr : workingDirectory.c_str(), &startup, &information) == FALSE) { FailLastError(kErrorLaunchFailed, "Starting the unpackaged application", GetLastError()); } @@ -797,7 +815,7 @@ const ApplicationLease& ApplicationManager::Launch(const json::Value& params, co // Packaged activation can report a short-lived broker. Only a new, // identity-matched process with a non-preexisting window can become owned. - const Deadline deadline(kLaunchTimeoutMs); + const Deadline deadline(static_cast(configuredTimeout)); std::optional graceDeadline; HWND matched = nullptr; DWORD matchedProcessId = 0; @@ -966,10 +984,29 @@ const ApplicationLease& ApplicationManager::Attach(const json::Value& params, co Fail(kErrorInvalidParams, "Attaching without a lease requires an exact \"windowTitle\" in the application descriptor."); } + const std::wstring aumid = application->WideField("aumid"); + const std::wstring configuredExecutablePath = application->WideField("executablePath"); + if (aumid.empty() && configuredExecutablePath.empty()) { + Fail(kErrorInvalidParams, "Attaching without a lease requires an executablePath or aumid, not a window title alone."); + } + const std::wstring executablePath = + aumid.empty() ? CanonicalizeExecutablePath(configuredExecutablePath) : std::wstring(); std::vector matches; + winrt::handle matchedProcess; + DWORD matchedProcessId = 0; for (const HWND window : FindWindowsWithTitle(windowTitle)) { token.ThrowIfCancelled(); - matches.push_back(window); + DWORD candidateId = 0; + GetWindowThreadProcessId(window, &candidateId); + winrt::handle candidate(OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION | SYNCHRONIZE, FALSE, candidateId)); + if (candidate && IsProcessAlive(candidate.get()) && + ProcessMatchesLaunchIdentity(candidate.get(), aumid, executablePath)) { + if (matches.empty()) { + matchedProcessId = candidateId; + matchedProcess = std::move(candidate); + } + matches.push_back(window); + } } if (matches.empty()) { Fail(kErrorAttachFailed, "No window titled \"" + ToUtf8(windowTitle) + "\" is currently open."); @@ -981,8 +1018,11 @@ const ApplicationLease& ApplicationManager::Attach(const json::Value& params, co } DWORD processId = 0; GetWindowThreadProcessId(matches.front(), &processId); - winrt::handle process = OpenProcessForIdentity(processId, kErrorAttachFailed); - return CreateLease("attached", processId, process.detach(), matches.front(), windowTitle, params); + if (processId != matchedProcessId || !IsProcessAlive(matchedProcess.get()) || + !ProcessMatchesLaunchIdentity(matchedProcess.get(), aumid, executablePath)) { + Fail(kErrorAttachFailed, "The application identity changed while attaching."); + } + return CreateLease("attached", processId, matchedProcess.detach(), matches.front(), windowTitle, params); } std::vector ApplicationManager::Windows(const std::string& leaseId, const CancellationToken& token) { diff --git a/packages/agentic/desktop-driver/native/windows/src/automation.cpp b/packages/agentic/desktop-driver/native/windows/src/automation.cpp index a3444cef6f..dea2cc0e95 100644 --- a/packages/agentic/desktop-driver/native/windows/src/automation.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/automation.cpp @@ -272,6 +272,7 @@ json::Value SerializeSnapshot(const ElementSnapshot& snapshot) { value.Set("enabled", SerializeSupported(snapshot.enabled)); value.Set("expanded", SerializeSupported(snapshot.expanded)); value.Set("focused", SerializeSupported(snapshot.focused)); + value.Set("required", SerializeSupported(snapshot.required)); if (!snapshot.name.empty()) { value.Set("name", json::Value::String(snapshot.name)); } @@ -353,6 +354,7 @@ winrt::com_ptr Automation::CreateCacheRequest(TreeSco UIA_ControlTypePropertyId, UIA_BoundingRectanglePropertyId, UIA_IsEnabledPropertyId, + UIA_IsRequiredForFormPropertyId, UIA_HasKeyboardFocusPropertyId, UIA_IsKeyboardFocusablePropertyId, UIA_IsOffscreenPropertyId, @@ -523,6 +525,7 @@ ElementSnapshot Automation::ReadSnapshot(IUIAutomationElement* element, const st snapshot.enabled = readBool(UIA_IsEnabledPropertyId, "The provider does not report an enabled state."); snapshot.focused = readBool(UIA_HasKeyboardFocusPropertyId, "The provider does not report keyboard focus."); + snapshot.required = readBool(UIA_IsRequiredForFormPropertyId, "The provider does not report required-form state."); const SupportedBool focusable = readBool(UIA_IsKeyboardFocusablePropertyId, "The provider does not report keyboard focusability."); snapshot.keyboardFocusable = focusable.supported && focusable.value; diff --git a/packages/agentic/desktop-driver/native/windows/src/automation.h b/packages/agentic/desktop-driver/native/windows/src/automation.h index f1bcee7a45..da179004f6 100644 --- a/packages/agentic/desktop-driver/native/windows/src/automation.h +++ b/packages/agentic/desktop-driver/native/windows/src/automation.h @@ -52,6 +52,7 @@ struct ElementSnapshot { RECT physicalRect{}; SupportedBool enabled; SupportedBool focused; + SupportedBool required{false, false, "The provider does not report required-form state."}; SupportedBool visible; SupportedBool expanded; SupportedBool selected; diff --git a/packages/agentic/desktop-driver/native/windows/src/selftest.cpp b/packages/agentic/desktop-driver/native/windows/src/selftest.cpp index 7b89736444..cba73a533c 100644 --- a/packages/agentic/desktop-driver/native/windows/src/selftest.cpp +++ b/packages/agentic/desktop-driver/native/windows/src/selftest.cpp @@ -263,6 +263,15 @@ void TestSnapshotShape() { "mixed checked state serializes as a string"); Check(serialized.find(R"("selected":{"supported":false,"reason":"no selection pattern"})") != std::string::npos, "unsupported state keeps its reason"); + Check(serialized.find(R"("required":{"supported":false,"reason":)") != std::string::npos, + "unknown required-form state is unsupported, not false"); + for (const bool required : {false, true}) { + snapshot.required = SupportedBool{true, required, {}}; + const std::string state = required ? R"("required":{"supported":true,"value":true})" + : R"("required":{"supported":true,"value":false})"; + Check(SerializeSnapshot(snapshot).Serialize().find(state) != std::string::npos, + required ? "required-form true is retained" : "required-form false is retained"); + } Check(serialized.find(R"("text")") == std::string::npos, "absent optional text is omitted"); Check(serialized.find(R"("rect":{"height":40,"width":120,"x":10,"y":20})") != std::string::npos, "rect serializes logical geometry"); @@ -321,6 +330,27 @@ void TestLeaseIdentity() { Check(TryParseIso8601("2026-08-31T23:59:59.0000000+02:00", offset), "offset timestamps parse"); Check(FileTimeTicks(offset) == FileTimeTicks(parsed) - 1234567, "offset timestamps normalize to UTC"); Check(!TryParseIso8601("not-a-timestamp", parsed), "invalid timestamps are rejected"); + + ApplicationManager applications; + CancellationToken token; + for (const double timeout : {0.0, -1.0, 0.5, 2147483648.0}) { + json::Value request = json::Value::Parse(R"({"application":{"executablePath":"unused.exe","windowTitle":"self-test"}})"); + request.Set("launchTimeoutMs", json::Value::Number(timeout)); + bool rejected = false; + try { + applications.Launch(request, token); + } catch (const HelperError& error) { + rejected = std::string(error.code()) == kErrorInvalidParams; + } + Check(rejected, "invalid launch deadlines are rejected before application side effects"); + } + bool titleOnlyRejected = false; + try { + applications.Attach(json::Value::Parse(R"({"application":{"windowTitle":"self-test"}})"), token); + } catch (const HelperError& error) { + titleOnlyRejected = std::string(error.code()) == kErrorInvalidParams; + } + Check(titleOnlyRejected, "unleased title-only attach is rejected"); } void TestHello() { diff --git a/packages/agentic/desktop-driver/references/native-helpers.md b/packages/agentic/desktop-driver/references/native-helpers.md index 0a4917e527..3fa0b15304 100644 --- a/packages/agentic/desktop-driver/references/native-helpers.md +++ b/packages/agentic/desktop-driver/references/native-helpers.md @@ -100,6 +100,11 @@ Build the helper explicitly, run `doctor` against that cache, then set `FURN_DESKTOP_DRIVER_BUILD_POLICY=never` before application preparation or smoke so later phases cannot compile implicitly. +Source resolution is anchored in the driver's own package, not the caller's +working directory or direct dependencies. Windows staging paths use a short +unique directory name to stay within MSBuild's path-length limits; immutable +artifact identities and the shared cache layout are unchanged. + Hosted runners that cannot provide authoritative global input may set `FURN_DESKTOP_DRIVER_DISABLED_INPUT_FEATURES=keyboard,physicalClick,wheel`. The Node host masks those capabilities and rejects direct use while leaving diff --git a/packages/agentic/desktop-driver/references/protocol.md b/packages/agentic/desktop-driver/references/protocol.md index 973aa84dc6..64148a42e9 100644 --- a/packages/agentic/desktop-driver/references/protocol.md +++ b/packages/agentic/desktop-driver/references/protocol.md @@ -19,6 +19,8 @@ V1 platform names are `macos` and `windows`. Endpoint names are `macos`, - A target records whether the application was launched or attached. - Deleting an attached session preserves the application. - Cleanup uses exact process identity, never process-name matching. +- Renderer is optional for non-React-Native targets. +- A registered target may restrict `launchModes`; clients cannot override that policy. Supported extension capabilities: @@ -77,6 +79,13 @@ Standard implicit, page-load, and script timeout values are accepted by the session contract. Native and Storybook operations use driver-owned bounded deadlines rather than repurposing page-load semantics. +`DesktopTarget.launchTimeoutMs` is a positive bounded integer. The server's +deadline defaults to 120 seconds; an explicitly configured value is also +forwarded to the native launch request. Otherwise both native providers retain +their 90-second launch default within that server deadline. +Clients cannot override it through capabilities. A timed-out launch must +cancel and dispose causally owned processes before the target can be reused. + Every host operation receives an `AbortSignal`. On timeout, the server aborts and drains native work before input release, teardown, or a later command. A provider may never apply late input after timeout cleanup. @@ -107,6 +116,12 @@ Portable lookup strategies: CSS and XPath are not redefined for non-DOM trees. +The `required` property and expectation report required-form state. Windows +and Win32 read UI Automation's `IsRequiredForForm` Boolean without provider +defaults. A provider that does not expose this state, including the current +macOS provider, returns `unsupported operation`, not `false`. Tree output +includes `states.required` only when the provider supplies a value. + ## Errors | Condition | WebDriver error | diff --git a/packages/agentic/desktop-driver/references/service.md b/packages/agentic/desktop-driver/references/service.md index 2c59226735..792fc94228 100644 --- a/packages/agentic/desktop-driver/references/service.md +++ b/packages/agentic/desktop-driver/references/service.md @@ -4,6 +4,11 @@ The Desktop Driver service is an embeddable, loopback-only W3C remote end. Integrators register controlled targets before listening; clients can select only those targets. +For application lifecycle commands and normal WDIO suites, prefer the +[`desktop-test` harness](../../desktop-test/README.md). Its real `serve` +composition supports applications without Storybook. The low-level APIs below +remain available for other controlled supervisors. + ## Embed a native target ```ts @@ -62,15 +67,17 @@ releases input, disposes hosts, and reports cleanup failures. Each `DesktopTarget` declares: -| Field | Meaning | -| ------------------- | ------------------------------------------------------- | -| `id` | Stable service-local identity selected by `furn:target` | -| `platformName` | `macos` or `windows` | -| `endpoint` | `macos`, `windows`, or `win32` | -| `renderer` | `fabric` or `paper` | -| `host` | A fake or native `DesktopHost` implementation | -| `storyOrchestrator` | Optional Storybook selection/reset adapter | -| `storyRootTestId` | Optional native marker verified after story readiness | +| Field | Meaning | +| ------------------- | -------------------------------------------------------------------------- | +| `id` | Stable service-local identity selected by `furn:target` | +| `platformName` | `macos` or `windows` | +| `endpoint` | `macos`, `windows`, or `win32` | +| `renderer` | Optional `fabric` or `paper`; omit for non-RN apps | +| `launchModes` | Allowed client launch modes; an external app can restrict this to `attach` | +| `launchTimeoutMs` | Server-owned, positive bounded launch/attach deadline | +| `host` | A fake or native `DesktopHost` implementation | +| `storyOrchestrator` | Optional Storybook selection/reset adapter | +| `storyRootTestId` | Optional native marker verified after story readiness | Do not derive a target from incoming capabilities. Executable paths, app arguments, environment variables, helper paths, lease paths, and artifact roots @@ -87,13 +94,14 @@ are server-owned configuration. | `aumid` | Windows packaged-app launch identity | | `executablePath` | Controlled unpackaged launch path | | `arguments` | Fixed server-owned launch arguments | +| `workingDirectory` | Fixed unpackaged Windows process working directory | | `windowTitle` | Exact expected window title, required by the Windows helper | | `leasePath` and `leaseNonce` | Exact owner-generated attach lease; do not accept from a client | Windows launch needs either `aumid` or `executablePath` plus an exact -`windowTitle`. Attach without an owner-generated lease also requires a title -that resolves to exactly one live window. Ambiguity fails instead of selecting -the first result. +`windowTitle`. Attach without an owner-generated lease requires the title +**and** matching executable or AUMID, resolving to exactly one live window. +Title-only attach and ambiguity fail instead of selecting the first result. For macOS descriptor and launch behavior, see [the macOS native provider](../native/macos/README.md). Storybook supplies a @@ -109,7 +117,8 @@ Clients request one of two modes: - `attach` resolves an already running application from the controlled descriptor or owner-generated lease and records it as external. -The default is `launch`. Deleting an attached session preserves the app. +The default is `launch`; a target may restrict the allowed modes. +Deleting an attached session preserves the app. Deleting a launched session may close the exact owned process. Cleanup never kills by process name. @@ -136,15 +145,19 @@ try { ``` The returned browser supports standard WebdriverIO APIs backed by the -implemented W3C subset plus these commands when a Storybook orchestrator is +implemented W3C subset. Visibility and focus use native queries, including +re-querying initially absent elements during waits; they never execute DOM +JavaScript. `installDesktopWebdriverCommands(browser, url)` installs the same +binding on a WDIO-managed browser without opening another session. +The following story commands are registered only when a Storybook orchestrator is registered: - `desktopListStories()`; - `desktopOpenStory(storyId, runId?)`; - `desktopResetStory(storyId, runId?)`; -- `desktopExpect(expectation)`; - `desktopRunStoryTests(options?)`. +`desktopExpect(expectation)` is available for all native targets. `clickMode` can be `physical`, `accessibility`, or `auto`. The server returns the negotiated features; it never claims an unsupported input mode. diff --git a/packages/agentic/desktop-driver/references/test-integration.md b/packages/agentic/desktop-driver/references/test-integration.md index 4d9c174dea..141d0685f6 100644 --- a/packages/agentic/desktop-driver/references/test-integration.md +++ b/packages/agentic/desktop-driver/references/test-integration.md @@ -10,6 +10,13 @@ The same W3C service and `DesktopHost` contract underpin all three. ## WebdriverIO tests +Use [`desktop-test`](../../desktop-test/README.md) for an application-owned +configuration and normal WDIO/Jasmine suites. It preserves TypeScript hooks, +Jasmine matchers, grouped spec arrays, retries, and native browser ownership. +`installDesktopWebdriverCommands(browser, url)` binds an existing WDIO-managed +browser, including native `isDisplayed`, `isFocused`, and delayed-render waits. +No Appium capability spoofing or second session is required. + Use `connectDesktopWebdriver` when a test owns service connection and session lifecycle: @@ -43,9 +50,10 @@ For Storybook targets, the connection registers: - `desktopListStories()`; - `desktopOpenStory(storyId, runId?)`; - `desktopResetStory(storyId, runId?)`; -- `desktopExpect(expectation)`; - `desktopRunStoryTests(options?)`. +`desktopExpect(expectation)` is available without Storybook. + The runner resets the preview for each test, filters and shards a stable `storyId/testId` ordering, checks required capabilities, and classifies assertion, timeout, cancellation, skip, and infrastructure outcomes. @@ -170,6 +178,9 @@ desktop-driver stories run \ Selectors can filter by story glob, test glob, tag, and deterministic shard. The CLI emits JSON and exits nonzero when the run status is not `passed`. +An all-skipped static run is `incomplete`, not a passing run. The neutral +harness's explicitly optional cohorts may finish skipped without claiming +that the underlying capabilities were qualified. ## Evidence diff --git a/packages/agentic/desktop-driver/src/authoring/results.ts b/packages/agentic/desktop-driver/src/authoring/results.ts index 1d15435e6a..b77440308e 100644 --- a/packages/agentic/desktop-driver/src/authoring/results.ts +++ b/packages/agentic/desktop-driver/src/authoring/results.ts @@ -8,7 +8,7 @@ export type DesktopArtifact = { export type DesktopStepStatus = 'failed' | 'passed' | 'skipped'; export type DesktopTestStatus = 'cancelled' | 'failed' | 'infrastructure-error' | 'passed' | 'skipped' | 'timed-out'; -export type DesktopRunStatus = 'failed' | 'passed'; +export type DesktopRunStatus = 'failed' | 'passed' | 'incomplete'; export type DesktopStoryStepResult = { artifacts: readonly DesktopArtifact[]; diff --git a/packages/agentic/desktop-driver/src/authoring/storyTests.test.ts b/packages/agentic/desktop-driver/src/authoring/storyTests.test.ts index 63da1367f8..7c7b44d1cd 100644 --- a/packages/agentic/desktop-driver/src/authoring/storyTests.test.ts +++ b/packages/agentic/desktop-driver/src/authoring/storyTests.test.ts @@ -1,6 +1,22 @@ import { validateDesktopStoryTests } from './storyTests.js'; describe('validateDesktopStoryTests', () => { + test('requires boolean values for required-form assertions', () => { + for (const value of [true, false]) { + const plan = { + version: 1, + tests: [{ id: 'required', steps: [{ expect: { state: 'required', target: { testId: 'field' }, value } }] }], + }; + expect(validateDesktopStoryTests(plan)).toBe(plan); + } + expect(() => + validateDesktopStoryTests({ + version: 1, + tests: [{ id: 'required', steps: [{ expect: { state: 'required', target: { testId: 'field' }, value: 'false' } }] }], + }), + ).toThrow('must be a boolean'); + }); + test('accepts a serializable versioned plan', () => { const plan = { version: 1, diff --git a/packages/agentic/desktop-driver/src/authoring/storyTests.ts b/packages/agentic/desktop-driver/src/authoring/storyTests.ts index dd97aa0b32..db56ef9f8b 100644 --- a/packages/agentic/desktop-driver/src/authoring/storyTests.ts +++ b/packages/agentic/desktop-driver/src/authoring/storyTests.ts @@ -26,6 +26,7 @@ export type DesktopStoryState = | 'expanded' | 'exists' | 'focused' + | 'required' | 'role' | 'selected' | 'text' @@ -224,6 +225,7 @@ function validateExpectation(value: unknown, source: string): void { 'expanded', 'exists', 'focused', + 'required', 'role', 'selected', 'text', @@ -259,6 +261,7 @@ function validateExpectation(value: unknown, source: string): void { expectation.state === 'expanded' || expectation.state === 'exists' || expectation.state === 'focused' || + expectation.state === 'required' || expectation.state === 'selected') && expectation.value !== undefined && typeof expectation.value !== 'boolean' && diff --git a/packages/agentic/desktop-driver/src/host/types.ts b/packages/agentic/desktop-driver/src/host/types.ts index b3693b5e67..1773f53005 100644 --- a/packages/agentic/desktop-driver/src/host/types.ts +++ b/packages/agentic/desktop-driver/src/host/types.ts @@ -78,6 +78,7 @@ export type NativeElementSnapshot = { name?: string; parentId?: string; rect: Rect; + required?: SupportedValue; role: string; scope: NativeElementScope; selected: SupportedValue; @@ -123,6 +124,7 @@ export type DesktopTreeNode = { enabled?: boolean; expanded?: boolean; focused?: boolean; + required?: boolean; selected?: boolean; visible?: boolean; }; @@ -136,7 +138,9 @@ export type DesktopTarget = { host: DesktopHost; id: string; platformName: DesktopPlatformName; - renderer: DesktopRenderer; + renderer?: DesktopRenderer; + launchModes?: readonly ('attach' | 'launch')[]; + launchTimeoutMs?: number; storyRootTestId?: string; storyOrchestrator?: StoryOrchestrator; }; diff --git a/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts b/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts index ce5978c9b1..c1b9ecd4bb 100644 --- a/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts +++ b/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts @@ -19,11 +19,15 @@ import type { import { HostStaleError, HostUnsupportedError } from '../../protocol/errors.js'; import type { DesktopClickMode, DesktopEndpoint, DesktopPlatformName } from '../../protocol/types.js'; -export type FakeDesktopElement = Omit & { +export type FakeDesktopElement = Omit< + NativeElementSnapshot, + 'checked' | 'enabled' | 'expanded' | 'focused' | 'required' | 'selected' | 'visible' +> & { checked?: boolean | 'mixed'; enabled?: boolean; expanded?: boolean; focused?: boolean; + required?: boolean; selected?: boolean; visible?: boolean; }; @@ -108,7 +112,7 @@ export class FakeDesktopHost implements DesktopHost { setElementStateUnsupported( elementId: string, - state: 'checked' | 'enabled' | 'expanded' | 'focused' | 'selected' | 'visible', + state: 'checked' | 'enabled' | 'expanded' | 'focused' | 'required' | 'selected' | 'visible', reason: string, ): void { const update = (element: NativeElementSnapshot) => { @@ -125,6 +129,9 @@ export class FakeDesktopHost implements DesktopHost { case 'focused': element.focused = { supported: false, reason }; break; + case 'required': + element.required = { supported: false, reason }; + break; case 'selected': element.selected = { supported: false, reason }; break; @@ -482,6 +489,7 @@ function toSnapshot(element: FakeDesktopElement): NativeElementSnapshot { enabled: { supported: true, value: element.enabled ?? true }, expanded: { supported: true, value: element.expanded ?? false }, focused: { supported: true, value: element.focused ?? false }, + required: element.required === undefined ? undefined : { supported: true, value: element.required }, selected: { supported: true, value: element.selected ?? false }, visible: { supported: true, value: element.visible ?? true }, }; @@ -495,6 +503,7 @@ function cloneElement(element: NativeElementSnapshot): NativeElementSnapshot { enabled: { ...element.enabled }, expanded: { ...element.expanded }, focused: { ...element.focused }, + required: element.required ? { ...element.required } : undefined, selected: { ...element.selected }, visible: { ...element.visible }, }; diff --git a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts index 856c191de6..d943790aa9 100644 --- a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts +++ b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts @@ -30,6 +30,7 @@ export type NativeDesktopHostOptions = { disabledInputFeatures?: readonly NativeDisabledInputFeature[]; endpoint: DesktopEndpoint; onStderr?: (message: string) => void; + onFailure?: (error: Error) => void; }; export type NativeDisabledInputFeature = 'keyboard' | 'physicalClick' | 'wheel'; @@ -43,6 +44,8 @@ export class NativeDesktopHost implements DesktopHost { private readonly artifact: NativeDriverArtifact; private readonly disabledInputFeatures: ReadonlySet; private readonly onStderr?: (message: string) => void; + private readonly onFailure?: (error: Error) => void; + private disposing = false; private readonly listeners = new Set<(event: DesktopHostEvent) => void>(); private readonly depressedButtons = new Set(); private readonly depressedKeys = new Set(); @@ -51,7 +54,7 @@ export class NativeDesktopHost implements DesktopHost { private failure?: Error; private processPromise?: Promise; - constructor({ application, artifact, disabledInputFeatures, endpoint, onStderr }: NativeDesktopHostOptions) { + constructor({ application, artifact, disabledInputFeatures, endpoint, onStderr, onFailure }: NativeDesktopHostOptions) { if (!artifact.endpoints.includes(endpoint)) { throw new TypeError(`Native helper ${artifact.provider} does not support endpoint "${endpoint}".`); } @@ -60,6 +63,7 @@ export class NativeDesktopHost implements DesktopHost { this.disabledInputFeatures = resolveDisabledInputFeatures(disabledInputFeatures); this.endpoint = endpoint; this.onStderr = onStderr; + this.onFailure = onFailure; } async probe(signal?: AbortSignal): Promise { @@ -222,6 +226,7 @@ export class NativeDesktopHost implements DesktopHost { } async dispose(signal?: AbortSignal): Promise { + this.disposing = true; const process = await this.processPromise; await process?.dispose(signal); } @@ -230,6 +235,7 @@ export class NativeDesktopHost implements DesktopHost { return { application: this.application, endpoint: target.endpoint, + launchTimeoutMs: target.launchTimeoutMs, renderer: target.renderer, storyRootTestId: target.storyRootTestId, targetId: target.id, @@ -272,6 +278,7 @@ export class NativeDesktopHost implements DesktopHost { process.on('exit', (error) => { this.failure = error; this.processPromise = undefined; + if (!this.disposing) this.onFailure?.(error); }); return process; })); diff --git a/packages/agentic/desktop-driver/src/native/nativeDriver.test.ts b/packages/agentic/desktop-driver/src/native/nativeDriver.test.ts index 36696f1b6c..a4b1c78ff8 100644 --- a/packages/agentic/desktop-driver/src/native/nativeDriver.test.ts +++ b/packages/agentic/desktop-driver/src/native/nativeDriver.test.ts @@ -23,6 +23,27 @@ const signedMacosTest = process.platform === 'darwin' && macosSigningIdentity && process.env.FURN_NATIVE_DRIVER_TEST === '1' ? test : test.skip; describe('native driver build and resolution', () => { + test('locates its published sources independently of the caller working directory', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'furn-native-caller-')); + try { + const anchor = path.resolve(__dirname, '../../lib/native/packageRoot.cjs'); + const child = spawnSync( + process.execPath, + ['-e', `process.stdout.write(require(${JSON.stringify(anchor)}).nativeDriverPackageRoot)`], + { + cwd: root, + encoding: 'utf8', + timeout: 10_000, + }, + ); + expect(child.status).toBe(0); + expect(child.stdout).toBe(path.resolve(__dirname, '../..')); + expect(fs.existsSync(path.join(child.stdout, 'native', 'protocol.json'))).toBe(true); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); + macosTest('builds, signs, reuses, and handshakes with the macOS helper', async () => { const cacheRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'furn-desktop-driver-macos-native-')); const emitWarning = jest.spyOn(process, 'emitWarning').mockImplementation(() => undefined); diff --git a/packages/agentic/desktop-driver/src/native/nativeDriver.ts b/packages/agentic/desktop-driver/src/native/nativeDriver.ts index 3f6c1c6b91..e457c88cfc 100644 --- a/packages/agentic/desktop-driver/src/native/nativeDriver.ts +++ b/packages/agentic/desktop-driver/src/native/nativeDriver.ts @@ -1,7 +1,6 @@ import { spawn } from 'node:child_process'; import { randomUUID } from 'node:crypto'; import fs from 'node:fs'; -import { createRequire } from 'node:module'; import path from 'node:path'; import { @@ -17,6 +16,7 @@ import { } from './filesystem.js'; import { nativeDriverWireProtocol } from './constants.js'; import { NativeDriverError } from './NativeDriverError.js'; +import { nativeDriverPackageRoot } from './packageRoot.cjs'; import type { NativeDriverArchitecture, NativeDriverArtifact, @@ -121,8 +121,10 @@ export async function buildNativeDesktopDriver(options: NativeDriverBuildOptions } } throwIfAborted(options.signal); - const stagingRoot = path.join(context.cacheRoot, 'v1', 'staging', `${buildFingerprint}-${process.pid}-${randomUUID()}`); - fs.mkdirSync(stagingRoot, { recursive: true }); + const stagingParent = path.join(context.cacheRoot, 'v1', 'staging'); + fs.mkdirSync(stagingParent, { recursive: true }); + // MSBuild's tracking files still use MAX_PATH even on long-path-enabled Windows hosts. + const stagingRoot = fs.mkdtempSync(path.join(stagingParent, `${shortKey(buildFingerprint)}-`)); try { const buildId = buildFingerprint; const built = await build(buildId, stagingRoot); @@ -317,7 +319,7 @@ async function resolveBuildContext(options: NativeDriverBuildOptions): Promise(manifestPath); - if (manifest.name === '@fluentui-react-native/desktop-driver') { - return current; - } - } - const parent = path.dirname(current); - if (parent === current) { - return undefined; - } - current = parent; - } - return undefined; -} diff --git a/packages/agentic/desktop-driver/src/native/packageRoot.cjs b/packages/agentic/desktop-driver/src/native/packageRoot.cjs new file mode 100644 index 0000000000..702a0107aa --- /dev/null +++ b/packages/agentic/desktop-driver/src/native/packageRoot.cjs @@ -0,0 +1,2 @@ +// A CommonJS anchor works in both the published ESM library and the source Jest host. +exports.nativeDriverPackageRoot = require('node:path').resolve(__dirname, '..', '..'); diff --git a/packages/agentic/desktop-driver/src/native/types.ts b/packages/agentic/desktop-driver/src/native/types.ts index 429095f372..eb5b8ba8a4 100644 --- a/packages/agentic/desktop-driver/src/native/types.ts +++ b/packages/agentic/desktop-driver/src/native/types.ts @@ -62,6 +62,7 @@ export type NativeDesktopApplicationDescriptor = { leaseNonce?: string; leasePath?: string; windowTitle?: string; + workingDirectory?: string; }; export type NativeHostHello = { diff --git a/packages/agentic/desktop-driver/src/protocol/capabilities.ts b/packages/agentic/desktop-driver/src/protocol/capabilities.ts index 74a74cfadc..5e8eee1716 100644 --- a/packages/agentic/desktop-driver/src/protocol/capabilities.ts +++ b/packages/agentic/desktop-driver/src/protocol/capabilities.ts @@ -101,7 +101,8 @@ export function createReturnedCapabilities( 'furn:clickMode': matched.clickMode, 'furn:endpoint': matched.target.endpoint, 'furn:features': host.features, - 'furn:renderer': matched.target.renderer, + ...(matched.target.renderer ? { 'furn:renderer': matched.target.renderer } : {}), + 'furn:storybook': Boolean(matched.target.storyOrchestrator), 'furn:target': matched.target.id, }; } diff --git a/packages/agentic/desktop-driver/src/runner/StoryTestRunner.test.ts b/packages/agentic/desktop-driver/src/runner/StoryTestRunner.test.ts index cf57ef5034..f198b6c189 100644 --- a/packages/agentic/desktop-driver/src/runner/StoryTestRunner.test.ts +++ b/packages/agentic/desktop-driver/src/runner/StoryTestRunner.test.ts @@ -177,18 +177,18 @@ describe('runDesktopStoryTests', () => { expect(second).toHaveLength(2); }); - test('does not turn an unsupported runtime property into a passing skip', async () => { + test.each(['checked', 'required'] as const)('does not turn unsupported %s state into a passing skip', async (state) => { const manifest = makeManifest({ version: 1, tests: [ { - id: 'unsupported-checked', - steps: [{ expect: { state: 'checked', target: { testId: 'button-primary' }, value: true } }], + id: `unsupported-${state}`, + steps: [{ expect: { state, target: { testId: 'button-primary' }, value: true } }], }, ], }); const harness = await createDesktopDriverStoryHarness(manifest); - harness.host.setElementStateUnsupported('button', 'checked', 'not exposed by this platform'); + harness.host.setElementStateUnsupported('button', state, 'not exposed by this platform'); try { const client = createDesktopDriverClient({ url: harness.server.url }); const session = await client.newSession({ @@ -315,8 +315,8 @@ describe('runDesktopStoryTests', () => { }); expect(result).toMatchObject({ - status: 'passed', tests: [{ skipReason: 'Unsupported capabilities: focus', status: 'skipped', steps: [] }], + status: 'incomplete', }); await session.delete(); } finally { diff --git a/packages/agentic/desktop-driver/src/runner/StoryTestRunner.ts b/packages/agentic/desktop-driver/src/runner/StoryTestRunner.ts index afc460a853..b475e39132 100644 --- a/packages/agentic/desktop-driver/src/runner/StoryTestRunner.ts +++ b/packages/agentic/desktop-driver/src/runner/StoryTestRunner.ts @@ -98,7 +98,11 @@ export async function runDesktopStoryTests({ runId, schemaVersion: 1, startedAt: startedAt.toISOString(), - status: results.every(({ status }) => status === 'passed' || status === 'skipped') ? 'passed' : 'failed', + status: results.every(({ status }) => status === 'skipped') + ? 'incomplete' + : results.every(({ status }) => status === 'passed' || status === 'skipped') + ? 'passed' + : 'failed', targetId, tests: results, }; @@ -363,6 +367,9 @@ export async function assertDesktopExpectation(session: DesktopSessionClient, ex case 'focused': actual = await element.getProperty('focused'); break; + case 'required': + actual = await element.getProperty('required'); + break; case 'role': actual = await element.getTagName(); break; diff --git a/packages/agentic/desktop-driver/src/server/SessionManager.ts b/packages/agentic/desktop-driver/src/server/SessionManager.ts index 0bb936e296..60311f5966 100644 --- a/packages/agentic/desktop-driver/src/server/SessionManager.ts +++ b/packages/agentic/desktop-driver/src/server/SessionManager.ts @@ -81,6 +81,9 @@ export class SessionManager { clickMode: DesktopClickMode, launchMode: 'attach' | 'launch', ): Promise { + if (target.launchModes && !target.launchModes.includes(launchMode)) { + throw new WebDriverError('session not created', `Target "${target.id}" does not allow ${launchMode} sessions.`); + } if (this.sessionsByTarget.has(target.id) || this.reservedTargets.has(target.id)) { throw new WebDriverError('session not created', `Target "${target.id}" already has an active session.`); } @@ -95,7 +98,7 @@ export class SessionManager { leasePromise = launchMode === 'attach' ? target.host.attach(target, signal) : target.host.launch(target, signal); return leasePromise; }, - 120_000, + target.launchTimeoutMs ?? 120_000, `${launchMode === 'attach' ? 'Attaching to' : 'Launching'} target "${target.id}"`, ); const activeLease = lease; @@ -114,7 +117,7 @@ export class SessionManager { clickMode, currentWindowId: windows[0].id, desktopTimeouts: { - appLaunch: 120_000, + appLaunch: target.launchTimeoutMs ?? 120_000, nativeCommand: 10_000, stableLayout: 1_000, storyRender: 30_000, diff --git a/packages/agentic/desktop-driver/src/server/TargetRegistry.ts b/packages/agentic/desktop-driver/src/server/TargetRegistry.ts index c289e71bbb..5bc87fcf1b 100644 --- a/packages/agentic/desktop-driver/src/server/TargetRegistry.ts +++ b/packages/agentic/desktop-driver/src/server/TargetRegistry.ts @@ -16,6 +16,15 @@ export class TargetRegistry { if (this.targets.has(target.id)) { throw new Error(`Desktop target "${target.id}" is already registered.`); } + if ( + target.launchTimeoutMs !== undefined && + (!Number.isSafeInteger(target.launchTimeoutMs) || target.launchTimeoutMs < 1 || target.launchTimeoutMs > 2_147_483_647) + ) { + throw new TypeError('Target launchTimeoutMs must be a positive bounded integer.'); + } + if (target.launchModes && (!target.launchModes.length || target.launchModes.some((mode) => mode !== 'attach' && mode !== 'launch'))) { + throw new TypeError('Target launchModes must contain attach and/or launch.'); + } this.targets.set(target.id, target); } diff --git a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts index a42830a4ad..d3d7b66801 100644 --- a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts +++ b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts @@ -260,6 +260,7 @@ async function route(context: RouteContext, sessions: SessionManager, targets: T ...(snapshot.enabled.supported ? { enabled: snapshot.enabled.value } : {}), ...(snapshot.expanded.supported ? { expanded: snapshot.expanded.value } : {}), ...(snapshot.focused.supported ? { focused: snapshot.focused.value } : {}), + ...(snapshot.required?.supported ? { required: snapshot.required.value } : {}), ...(snapshot.selected.supported ? { selected: snapshot.selected.value } : {}), ...(snapshot.visible.supported ? { visible: snapshot.visible.value } : {}), }, @@ -697,6 +698,11 @@ function getElementValue(snapshot: NativeElementSnapshot, name: string): unknown return requireSupported(snapshot.checked, 'checked'); case 'expanded': return requireSupported(snapshot.expanded, 'expanded'); + case 'required': + return requireSupported( + snapshot.required ?? { supported: false, reason: 'The provider does not expose required-form state.' }, + 'required', + ); case 'selected': return requireSupported(snapshot.selected, 'selected'); default: diff --git a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts index b686bdd1e1..901044a694 100644 --- a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts +++ b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts @@ -4,11 +4,52 @@ import { createDesktopDriverClient } from '../client/DesktopDriverClient.js'; import type { FakeDesktopHost } from '../hosts/fake/FakeDesktopHost.js'; import { webElementIdentifier } from '../protocol/constants.js'; import { HostWebDriverError } from '../protocol/errors.js'; -import { runDesktopStoryTests } from '../runner/StoryTestRunner.js'; +import { assertDesktopExpectation, runDesktopStoryTests } from '../runner/StoryTestRunner.js'; import { createDesktopDriverTestHarness } from '../testing/protocolHarness.js'; import type { DesktopStoryManifest, StoryOrchestrator, StoryReadyResult } from '../storybook.js'; describe('Desktop Driver W3C remote end', () => { + test.each([true, false, undefined])('preserves required-form state %s without inventing unsupported values', async (required) => { + const harness = await createDesktopDriverTestHarness({ + windows: [ + { + id: 'main', + title: 'Required-state fixture', + elements: [ + { + id: 'field', + automationId: 'field', + role: 'checkbox', + scope: 'application', + windowId: 'main', + rect: { x: 0, y: 0, width: 100, height: 40 }, + required, + }, + ], + }, + ], + }); + try { + const client = createDesktopDriverClient({ url: harness.server.url }); + const session = await client.newSession({ + alwaysMatch: { platformName: 'windows', 'furn:target': harness.target.id }, + }); + const element = await session.findElement('accessibility id', 'field'); + const tree = await session.getTree(); + if (required === undefined) { + await expect(element.getProperty('required')).rejects.toMatchObject({ code: 'unsupported operation' }); + expect(tree[0].states).not.toHaveProperty('required'); + } else { + expect(await element.getProperty('required')).toBe(required); + expect(tree[0].states.required).toBe(required); + await assertDesktopExpectation(session, { state: 'required', target: { testId: 'field' }, value: required }); + } + await session.delete(); + } finally { + await harness.close(); + } + }); + test('serves status and a complete fake-host session through raw HTTP', async () => { const harness = await createDesktopDriverTestHarness(); try { diff --git a/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.test.ts b/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.test.ts index 1ee524231a..1e30b2974c 100644 --- a/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.test.ts +++ b/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.test.ts @@ -37,18 +37,29 @@ describe('sanctioned WebdriverIO API', () => { schemaVersion: 1, }; const harness = await createDesktopDriverStoryHarness(manifest); + const find = harness.host.find.bind(harness.host); + let delayedLookups = 0; + harness.host.find = async (root, selector) => { + if (selector.value === 'button-delayed') { + if (++delayedLookups < 3) return []; + return find(root, { ...selector, value: 'button-primary' }); + } + return find(root, selector); + }; const artifactsRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'desktop-driver-wdio-')); try { const response = await runContract(harness.server.url, harness.target.id, artifactsRoot); expect(response).toMatchObject({ attachedWaitOptions: { waitforInterval: 100, waitforTimeout: 5000 }, attachedWaitPolls: 3, + delayedDisplayed: true, manifestEntries: 1, result: { status: 'passed', tests: [{ status: 'passed', storyId: 'components-button--default', testId: 'click' }], }, }); + expect(delayedLookups).toBeGreaterThanOrEqual(3); expect(fs.existsSync(path.join(artifactsRoot, 'run.json'))).toBe(true); } finally { await harness.close(); diff --git a/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.ts b/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.ts index 850a4015db..32520503fb 100644 --- a/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.ts +++ b/packages/agentic/desktop-driver/src/wdio/DesktopWebdriver.ts @@ -18,6 +18,7 @@ declare global { 'furn:endpoint'?: 'macos' | 'win32' | 'windows'; 'furn:features'?: Readonly; 'furn:launchMode'?: 'attach' | 'launch'; + 'furn:storybook'?: boolean; 'furn:target'?: string; } @@ -125,7 +126,7 @@ export async function connectDesktopWebdriver(options: DesktopWebdriverOptions): capabilities, logLevel: options.logLevel ?? 'silent', }); - return bindDesktopCommands(browser, options.url); + return installDesktopWebdriverCommands(browser, options.url); } // WebdriverIO applies these defaults in remote() but not attach(). Without an interval, waitUntil and the waitFor* @@ -140,7 +141,7 @@ export async function attachDesktopWebdriver(options: DesktopWebdriverAttachment sessionId: options.sessionId, options: { logLevel: 'silent', ...attachedWaitDefaults }, }); - return bindDesktopCommands(browser, options.url); + return installDesktopWebdriverCommands(browser, options.url); } function connectionOptions(address: string) { @@ -156,16 +157,58 @@ function connectionOptions(address: string) { }; } -function bindDesktopCommands(browser: WebdriverIO.Browser, url: string): DesktopWebdriverSession { +const installedBrowsers = new WeakSet(); +const installedStoryBrowsers = new WeakSet(); +const boundSessions = new WeakMap(); + +/** Bind a WDIO-managed browser without creating a competing remote session. */ +export function installDesktopWebdriverCommands(browser: WebdriverIO.Browser, url: string): DesktopWebdriverSession { + if (browser.capabilities.browserName !== 'furn-native-desktop') { + throw new TypeError('Native desktop compatibility requires a Desktop Driver session.'); + } const client = createDesktopDriverClient({ url }); const session = new DesktopSessionClient(client, browser.sessionId, Object.fromEntries(Object.entries(browser.capabilities))); const desktop = new DesktopWebdriverSession(browser, session); - - browser.addCommand('desktopListStories', () => desktop.listStories()); - browser.addCommand('desktopOpenStory', (storyId: string, runId?: string) => desktop.openStory(storyId, runId)); - browser.addCommand('desktopResetStory', (storyId: string, runId?: string) => desktop.resetStory(storyId, runId)); - browser.addCommand('desktopExpect', (expectation: DesktopStoryExpectation) => desktop.expect(expectation)); - browser.addCommand('desktopRunStoryTests', (runOptions?: DesktopWebdriverRunOptions) => desktop.runStoryTests(runOptions)); + boundSessions.set(browser, desktop); + + if (!installedBrowsers.has(browser)) { + browser.overwriteCommand( + 'isDisplayed', + async function (this: WebdriverIO.Element) { + const id = await nativeElementId(this); + return id ? this.isElementDisplayed(id) : false; + }, + true, + ); + browser.overwriteCommand( + 'isFocused', + async function (this: WebdriverIO.Element) { + const id = await nativeElementId(this); + if (!id) return false; + const focused = await this.getElementProperty(id, 'focused'); + if (typeof focused !== 'boolean') throw new Error('Desktop Driver did not report native focus as a boolean.'); + return focused; + }, + true, + ); + browser.addCommand('desktopExpect', (expectation: DesktopStoryExpectation) => boundSessions.get(browser)!.expect(expectation)); + installedBrowsers.add(browser); + } + if (browser.capabilities['furn:storybook'] && !installedStoryBrowsers.has(browser)) { + browser.addCommand('desktopListStories', () => boundSessions.get(browser)!.listStories()); + browser.addCommand('desktopOpenStory', (storyId: string, runId?: string) => boundSessions.get(browser)!.openStory(storyId, runId)); + browser.addCommand('desktopResetStory', (storyId: string, runId?: string) => boundSessions.get(browser)!.resetStory(storyId, runId)); + browser.addCommand('desktopRunStoryTests', (runOptions?: DesktopWebdriverRunOptions) => + boundSessions.get(browser)!.runStoryTests(runOptions), + ); + installedStoryBrowsers.add(browser); + } return desktop; } + +async function nativeElementId(element: WebdriverIO.Element): Promise { + // Like WDIO's native visibility path, retry the selector when the first lookup preceded the native render. + if (!element.elementId) element.elementId = (await element.parent.$(element.selector).getElement()).elementId; + return element.elementId; +} diff --git a/packages/agentic/desktop-driver/src/wdio/index.ts b/packages/agentic/desktop-driver/src/wdio/index.ts index db6ac1b5a0..bfd35c484f 100644 --- a/packages/agentic/desktop-driver/src/wdio/index.ts +++ b/packages/agentic/desktop-driver/src/wdio/index.ts @@ -1,2 +1,7 @@ -export { attachDesktopWebdriver, connectDesktopWebdriver, DesktopWebdriverSession } from './DesktopWebdriver.js'; +export { + attachDesktopWebdriver, + connectDesktopWebdriver, + DesktopWebdriverSession, + installDesktopWebdriverCommands, +} from './DesktopWebdriver.js'; export type { DesktopWebdriverAttachment, DesktopWebdriverOptions, DesktopWebdriverRunOptions } from './DesktopWebdriver.js'; diff --git a/packages/agentic/desktop-driver/src/wdio/wdioRunner.contract.cjs b/packages/agentic/desktop-driver/src/wdio/wdioRunner.contract.cjs index 1379e751b2..d5bc5496d5 100644 --- a/packages/agentic/desktop-driver/src/wdio/wdioRunner.contract.cjs +++ b/packages/agentic/desktop-driver/src/wdio/wdioRunner.contract.cjs @@ -25,6 +25,9 @@ async function main() { waitforInterval: worker.browser.options.waitforInterval, waitforTimeout: worker.browser.options.waitforTimeout, }; + const delayed = await worker.browser.$('~button-delayed'); + await delayed.waitForDisplayed({ timeout: 2000 }); + const delayedDisplayed = await delayed.isDisplayed(); const manifest = await worker.browser.desktopListStories(); await desktop.browser.desktopExpect({ state: 'enabled', @@ -32,7 +35,9 @@ async function main() { value: true, }); const result = await desktop.browser.desktopRunStoryTests({ artifactsRoot }); - process.stdout.write(JSON.stringify({ attachedWaitOptions, attachedWaitPolls, manifestEntries: manifest.entries.length, result })); + process.stdout.write( + JSON.stringify({ attachedWaitOptions, attachedWaitPolls, delayedDisplayed, manifestEntries: manifest.entries.length, result }), + ); } finally { await desktop.delete(); } diff --git a/packages/agentic/desktop-test/README.md b/packages/agentic/desktop-test/README.md new file mode 100644 index 0000000000..042326817e --- /dev/null +++ b/packages/agentic/desktop-test/README.md @@ -0,0 +1,172 @@ +# Native desktop test harness + +`@fluentui-react-native/desktop-test` owns application-neutral configuration, +native service composition, process lifetimes, and reporting. It depends on +[`desktop-driver`](../desktop-driver/README.md), not Storybook, React, or React +Native. Storybook is an explicitly imported +[integration](../storybook-desktop/README.md); ordinary applications do not +install it. + +## Application configuration + +Put a `desktop.config.mts` in the application workspace: + +```ts +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + projectRoot: new URL('.', import.meta.url), + targets: { + windows: { + host: { + kind: 'installed', + application: { + executablePath: 'C:\\apps\\Sample.exe', + windowTitle: 'Sample', + }, + }, + launchTimeoutMs: 180_000, + readiness: { kind: 'native', testId: 'application-ready' }, + }, + }, +}); +``` + +The source-loadable `/config` entry works before the repository build. `.ts`, +`.mjs`, `.js`, and `.cjs` configs are also supported. Validation is structural, +rejects unknown fields, and does not require a class instance. + +Target names default to their endpoint (`macos`, `windows`, or `win32`); custom +names must supply `endpoint`. Renderer is optional for ordinary native apps. +`host.ownership: 'external'` restricts sessions to attach and preserves the app; +the default permits owned launch only. Windows identity requires an exact title +and an executable, AUMID, or verified owner lease, never a title alone. + +| Host adapter | Behavior | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `installed` | Use the trusted application descriptor; build/prep/bundle/run stages require explicit commands. | +| `react-native-test-app` | Derive native commands and identity from `app.json`. Windows verifies the generated renderer and registered package identity. | +| `rex` | Win32 prebuilt host with declared `component` and `windowTitle`; use the existing bundle, executable, working directory, and fixed arguments. Native build is unsupported. | + +`commands` overrides individual `prep`, `bundle`, `build`, or `run` stages. +Each value is `{ command, args?, cwd?, env? }`, an ordered array of commands, or +`false` to reject the stage. No command string is parsed as an argument list. +Required `services` declare an `id`, command, and optional loopback `readyUrl`. +Service exits are failures even after successful readiness. + +The default project root is the config's directory. Application paths, +command working directories, helper paths, and artifacts resolve against their +owning project root. Target references and hook files resolve against their +declaring config's directory; spec globs resolve against the suite's project. + +## Standalone WDIO suites + +The suite workspace references the application's same-schema target instead +of duplicating its launch settings: + +```ts +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +export default defineDesktopConfig({ + schemaVersion: 1, + targets: { + windows: { use: { config: '../sample/desktop.config.mts', target: 'windows' } }, + }, + suites: { + regression: { + kind: 'wdio', + framework: 'jasmine', + specs: [['specs/startup.spec.ts', 'specs/controls.spec.ts']], + hooks: './hooks.ts', + specFileRetries: 1, + maxInstances: 1, + }, + }, +}); +``` + +References preserve the app's canonical target identity and paths; overrides +and reference cycles are rejected. Nested spec arrays remain one WDIO worker. +Every glob must match: an empty pattern is an error, not reduced coverage. +Specs can also be an endpoint-keyed map. + +Install `@wdio/cli`, `@wdio/local-runner`, and the selected +`@wdio/-framework` in the suite workspace. The harness does not install +frameworks into applications. Jasmine, Mocha, and Cucumber are accepted framework +names; deterministic integration coverage currently exercises Jasmine. +WDIO owns the session, globals, hooks, TypeScript loader, grouped execution, +retries, and framework assertions. Hooks export a WDIO hook object. The native +command binding uses that existing browser; it never creates a second session. +The generated config uses `.mts` so WDIO enables its TypeScript loader before +loading hooks. Frameworks retain their normal names: an absolute Jasmine +module path causes WDIO to replace Jasmine's assertion API. + +Expose `"desktop-test": "desktop-test"` in workspace scripts, then run: + +```sh +yarn desktop-test --target windows build-driver +yarn desktop-test --target windows test --suite regression --list +yarn desktop-test --target windows test --suite regression +yarn desktop-test --target windows serve --json +``` + +`--list` resolves the exact groups without launching anything. `serve` starts +the real native WebDriver service and required services, reports its actual +URL/target, and waits for an owned stop; clients create sessions explicitly. +`--json` keeps stdout machine-readable and diagnostics on stderr. +`prep`, `bundle`, `build`, and `run` expose supported adapter stages. +macOS RN Test App targets accept `--paper`; regenerate with `prep` after changing +renderer. Native tests use verified prebuilt helpers only: run `build-driver` +explicitly first, or provision a compatible helper/cache through +[`nativeDriver` settings](../desktop-driver/references/native-helpers.md). + +## Ownership and reporting + +One supervisor owns services, workers, the WebDriver listener, and its app +sessions. Nested CLIs and the PowerShell service bridge inherit an authenticated +reporting connection; they do not create competing collectors or consoles. +Required-service failure aborts dependent work. Cleanup drains producers and +stdio, closes sessions, releases input, stops exact owned process trees in +reverse service order, and preserves external apps. + +Interactive runs acquire a per-user claim at +`/furn-desktop-workload-/interactive.lock`. +Concurrent workloads fail closed. After a crash, inspect the recorded PID/run +and verify that the owner and its resources have stopped before manually +removing that exact stale claim. There is no automatic stale-lock recovery. + +Each root run writes `artifacts/desktop//` by default: + +| Artifact | Contents | +| -------------- | ---------------------------------------------------------------------------------------- | +| `events.jsonl` | Versioned, source-attributed journal with producer sequence and collector arrival order. | +| `run.json` | Canonical tests, attempts, selected groups, failures, evidence paths, and final status. | +| `junit.xml` | The same outcomes, including infrastructure errors and incomplete work. | +| `commands/` | Bounded, redacted stdout/stderr logs with stream attribution. | + +Stderr and test progress are live without `--verbose`; verbose also streams +informational output. Failed commands replay contiguous logs. GitHub Actions +adds balanced groups, deduplicated failure annotations, and an optional step +summary; raw child output cannot issue workflow commands. Artifact references +are filesystem paths, not uploaded URLs. Archive the entire run directory with +an `always()` upload step. + +Original errors, causes, retry attempts, flaky status, skips, unsupported +capabilities, and unfinished selected groups remain in the result. A zero +worker exit cannot override failed assertions or incomplete work. Empty or +all-skipped required cohorts do not pass; `optional: true` permits a fully +skipped cohort to exit successfully without labeling it passed. + +## Qualification + +Package contracts and a passing Win32 behavioral pilot are not complete native +qualification. See the [E2E migration boundary](../../../apps/E2E/README.md#native-desktop-migration) +for preserved baseline groups and unsupported semantics. Legacy desktop Appium +lanes remain until equivalent-build semantic parity and the driver's +100-execution/14-day reliability gates pass on each endpoint. Mobile lanes are +outside this migration. + +Use the package's `format`, `lint`, `build`, and `test` scripts for deterministic +checks. Matching-OS native builds and real application/input evidence are +separate requirements. diff --git a/packages/agentic/desktop-test/config/cli.cjs b/packages/agentic/desktop-test/config/cli.cjs new file mode 100644 index 0000000000..b94cdd9f21 --- /dev/null +++ b/packages/agentic/desktop-test/config/cli.cjs @@ -0,0 +1,8 @@ +#!/usr/bin/env node + +import('../lib/cli.js') + .then(({ runDesktopTestCli }) => runDesktopTestCli()) + .catch((error) => { + console.error(error); + process.exitCode = 1; + }); diff --git a/packages/agentic/desktop-test/config/launcher.cjs b/packages/agentic/desktop-test/config/launcher.cjs new file mode 100644 index 0000000000..b428a59d8b --- /dev/null +++ b/packages/agentic/desktop-test/config/launcher.cjs @@ -0,0 +1,36 @@ +async function main() { + const [launcherUrl, configPath] = process.argv.slice(2); + const { randomUUID } = require('node:crypto'); + const { connectReportProducer } = await import('../lib/reporting/transport.js'); + const { DesktopReportedError, serializeReportError } = await import('../lib/reporting/protocol.js'); + let launcher; + let stopped = false; + const producer = await connectReportProducer(`wdio-launcher-${randomUUID()}`, undefined, async () => { + stopped = true; + await launcher?.runner?.shutdown(); + }); + try { + const { Launcher } = await import(launcherUrl); + launcher = new Launcher(configPath); + await launcher.initialize(); + if (stopped) { + process.exitCode = 1; + return; + } + process.exitCode = (await launcher.run()) ?? 1; + } catch (error) { + const failure = + error instanceof DesktopReportedError + ? error.failure + : { failureId: randomUUID(), kind: 'infrastructure', error: serializeReportError(error) }; + producer.emit({ kind: 'service-failed', payload: { failure } }); + process.exitCode = 1; + } finally { + await producer.finish(process.exitCode === 0 ? 'passed' : 'failed'); + } +} + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/packages/agentic/desktop-test/config/service.cjs b/packages/agentic/desktop-test/config/service.cjs new file mode 100644 index 0000000000..54ae6bb76d --- /dev/null +++ b/packages/agentic/desktop-test/config/service.cjs @@ -0,0 +1,10 @@ +#!/usr/bin/env node +import('../lib/ownedService.js') + .then(({ runOwnedDesktopService }) => { + if (process.argv.length !== 3) throw new Error('Expected one generated desktop service descriptor path.'); + return runOwnedDesktopService(process.argv[2]); + }) + .catch((error) => { + process.stderr.write(`${error.stack ?? error}\n`); + process.exitCode = 1; + }); diff --git a/packages/agentic/desktop-test/jest.config.cjs b/packages/agentic/desktop-test/jest.config.cjs new file mode 100644 index 0000000000..d6535664e2 --- /dev/null +++ b/packages/agentic/desktop-test/jest.config.cjs @@ -0,0 +1,9 @@ +const config = require('@fluentui-react-native/scripts/jest-config'); + +module.exports = { + ...config, + moduleNameMapper: { + ...config.moduleNameMapper, + '^(\\.{1,2}/.*)\\.js$': '$1', + }, +}; diff --git a/packages/agentic/desktop-test/package.json b/packages/agentic/desktop-test/package.json new file mode 100644 index 0000000000..ec95f5c79e --- /dev/null +++ b/packages/agentic/desktop-test/package.json @@ -0,0 +1,74 @@ +{ + "name": "@fluentui-react-native/desktop-test", + "version": "0.1.0", + "description": "Application-neutral lifecycle and reporting for native desktop tests", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/microsoft/fluentui-react-native.git", + "directory": "packages/agentic/desktop-test" + }, + "bin": "./config/cli.cjs", + "type": "module", + "main": "lib/index.js", + "module": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "import": "./lib/index.js", + "default": "./src/index.ts" + }, + "./config": { + "types": "./lib/config.d.ts", + "default": "./src/config.ts" + }, + "./reporting": { + "types": "./lib/reporting/index.d.ts", + "import": "./lib/reporting/index.js", + "default": "./src/reporting/index.ts" + }, + "./commands": { + "types": "./lib/commands.d.ts", + "import": "./lib/commands.js", + "default": "./src/commands.ts" + }, + "./wdio": { + "types": "./lib/wdio/index.d.ts", + "import": "./lib/wdio/index.js", + "default": "./src/wdio/index.ts" + }, + "./service": "./config/service.cjs", + "./package.json": "./package.json" + }, + "scripts": { + "build": "tsc -b", + "clean": "fluentui-scripts clean", + "format": "fluentui-scripts format", + "lint": "fluentui-scripts lint", + "test": "fluentui-scripts jest" + }, + "dependencies": { + "@fluentui-react-native/desktop-driver": "workspace:*", + "@rnx-kit/reporter": "catalog:", + "@wdio/reporter": "9.24.0", + "commander": "^14.0.2", + "import-meta-resolve": "4.2.0" + }, + "devDependencies": { + "@fluentui-react-native/scripts": "workspace:*", + "@wdio/cli": "catalog:", + "@wdio/jasmine-framework": "catalog:", + "@wdio/local-runner": "catalog:", + "@wdio/runner": "catalog:", + "expect-webdriverio": "catalog:", + "webdriverio": "catalog:" + }, + "furn": { + "jestPlatform": "react" + }, + "rnx-kit": { + "kitType": "library", + "extends": "@fluentui-react-native/scripts/kit-config" + } +} diff --git a/packages/agentic/desktop-test/src/adapters.ts b/packages/agentic/desktop-test/src/adapters.ts new file mode 100644 index 0000000000..0062735562 --- /dev/null +++ b/packages/agentic/desktop-test/src/adapters.ts @@ -0,0 +1,152 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { createRequire } from 'node:module'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import type { NativeDesktopApplicationDescriptor } from '@fluentui-react-native/desktop-driver'; +import type { DesktopCommand, DesktopStage, ResolvedDesktopTarget } from './config.js'; + +export async function resolveDesktopApplication( + target: ResolvedDesktopTarget, + signal?: AbortSignal, +): Promise { + const configured = target.host.application ?? {}; + let application: NativeDesktopApplicationDescriptor; + if (target.host.kind === 'installed') { + application = { ...configured }; + } else if (target.host.kind === 'rex') { + const require = createRequire(path.join(target.projectRoot, 'package.json')); + const directory = path.dirname(require.resolve('@office-iss/rex-win32/rex-win32.js')); + const bundleRoot = target.host.bundleRoot ?? path.join(target.projectRoot, 'dist'); + if (!target.host.component || !target.host.windowTitle) + throw new Error('REX targets require their canonical component and windowTitle.'); + if (!fs.existsSync(path.join(bundleRoot, 'index.win32.bundle'))) + throw new Error(`REX bundle is missing in ${bundleRoot}; run the bundle stage first.`); + application = { + executablePath: path.join(directory, 'ReactTest.exe'), + workingDirectory: directory, + arguments: [ + 'basePath', + bundleRoot, + 'plugin', + 'defaultplugin', + 'bundle', + 'index.win32', + 'component', + target.host.component, + 'windowTitle', + target.host.windowTitle, + 'jsEngine', + 'v8', + ], + windowTitle: target.host.windowTitle, + ...configured, + }; + } else { + const app = readAppManifest(target); + const component = app.components?.[0]; + const windowTitle = component?.displayName ?? app.displayName ?? app.name; + if (!windowTitle) throw new Error('RN Test App manifest does not define its display identity.'); + if (target.endpoint === 'macos') { + application = { bundleIdentifier: app.macos?.bundleIdentifier ?? 'com.microsoft.ReactTestApp', windowTitle, ...configured }; + } else { + const propertiesPath = path.join(target.projectRoot, 'windows', 'ExperimentalFeatures.props'); + const properties = fs.readFileSync(propertiesPath, 'utf8'); + const useFabric = /\s*(true|false)\s*<\/UseFabric>/i.exec(properties)?.[1]?.toLowerCase(); + if (!useFabric) throw new Error(`Generated Windows renderer is not explicit in ${propertiesPath}; regenerate the project.`); + const actualRenderer = useFabric === 'true' ? 'fabric' : 'paper'; + if (!target.host.renderer || target.host.renderer !== actualRenderer) { + throw new Error( + `Configured Windows renderer ${target.host.renderer ?? '(not specified)'} does not match generated ${actualRenderer}.`, + ); + } + application = { windowTitle, ...configured }; + if (!application.aumid && !application.executablePath && !application.leasePath) { + const manifestPath = path.join(target.projectRoot, 'windows', 'ReactApp.Package', 'bin', 'x64', 'Debug', 'AppxManifest.xml'); + if (!fs.existsSync(manifestPath)) + throw new Error( + `Built Windows manifest missing at ${manifestPath}; build/register the app or configure its verified application identity.`, + ); + const manifest = fs.readFileSync(manifestPath, 'utf8'); + const name = /]*\bName="([^"]+)"/.exec(manifest)?.[1]; + const appId = /]*\bId="([^"]+)"/.exec(manifest)?.[1]; + if (!name || !appId || !/^[\w.-]+$/.test(name) || !/^[\w.-]+$/.test(appId)) + throw new Error('Generated Windows manifest has an invalid or ambiguous application identity.'); + const { stdout } = await promisify(execFile)( + 'powershell.exe', + [ + '-NoProfile', + '-NonInteractive', + '-Command', + `$ErrorActionPreference='Stop'; $p=@(Get-AppxPackage -Name '${name}'); if ($p.Count -ne 1) { throw 'Expected one registered app package; install the generated application first.' }; $p[0].PackageFamilyName`, + ], + { timeout: 10_000, signal, maxBuffer: 64 * 1024, windowsHide: true }, + ); + const family = stdout.trim(); + if (!/^[\w.-]+_[\w]+$/.test(family)) throw new Error('Could not verify the registered Windows package family.'); + application.aumid = `${family}!${appId}`; + } + } + } + if (target.endpoint === 'macos') { + if (!application.bundleIdentifier && !application.executablePath && !application.leasePath) + throw new Error('macOS application identity is missing.'); + if (application.workingDirectory) throw new Error('workingDirectory is currently supported only by the Windows helper.'); + } else { + if (!application.windowTitle) throw new Error('Windows applications require an exact window title.'); + if (!application.aumid && !application.executablePath && !application.leasePath) { + throw new Error('A title alone is not an application identity; configure an executable, AUMID, or verified owner lease.'); + } + } + return Object.freeze(application); +} + +export function desktopStageCommands(target: ResolvedDesktopTarget, stage: DesktopStage): readonly DesktopCommand[] { + const override = target.commands?.[stage]; + if (override === false) throw new Error(`The ${stage} stage is unsupported for ${target.endpoint}.`); + if (override !== undefined) return Array.isArray(override) ? override : [override as DesktopCommand]; + if (target.host.kind === 'installed') throw new Error(`Installed applications do not support the ${stage} stage.`); + if (target.host.kind === 'rex') { + if (stage === 'build') throw new Error('REX uses a prebuilt host; native build is unsupported.'); + if (stage === 'prep') return []; + if (stage === 'run') + throw new Error('Use a driver session to launch REX with recorded ownership, or declare an explicit development run command.'); + return [{ command: 'yarn', args: ['bundle'] }]; + } + const app = readAppManifest(target); + if (!app.name) throw new Error('RN Test App manifest must define name for native commands.'); + if (stage === 'prep') + return target.endpoint === 'windows' + ? [{ command: 'yarn', args: ['exec', 'install-windows-test-app', ...(target.host.renderer === 'paper' ? [] : ['--use-fabric'])] }] + : [{ command: 'pod', args: ['install'], cwd: 'macos' }]; + if (stage === 'bundle') + return [{ command: 'yarn', args: ['exec', 'rnx-cli', 'bundle', '--dev', 'false', '--platform', target.endpoint] }]; + return [ + { + command: 'yarn', + args: [ + 'exec', + 'rnx-cli', + stage, + '--platform', + target.endpoint, + ...(target.endpoint === 'windows' + ? ['--solution', path.join('windows', `${app.name}.sln`)] + : ['--workspace', path.join('macos', `${app.name}.xcworkspace`), '--scheme', app.name]), + ], + }, + ]; +} + +type AppManifest = { + name?: string; + displayName?: string; + components?: { appKey?: string; displayName?: string }[]; + macos?: { bundleIdentifier?: string }; +}; +function readAppManifest(target: ResolvedDesktopTarget): AppManifest { + const filename = target.host.appConfigPath ?? path.join(target.projectRoot, 'app.json'); + const value: unknown = JSON.parse(fs.readFileSync(filename, 'utf8')); + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError(`Invalid app manifest ${filename}.`); + return value as AppManifest; +} diff --git a/packages/agentic/desktop-test/src/cli.ts b/packages/agentic/desktop-test/src/cli.ts new file mode 100644 index 0000000000..bb746ed6e3 --- /dev/null +++ b/packages/agentic/desktop-test/src/cli.ts @@ -0,0 +1,112 @@ +import { Command } from 'commander'; +import { buildNativeDesktopDriver } from '@fluentui-react-native/desktop-driver'; +import { desktopStageCommands } from './adapters.js'; +import { loadDesktopConfig, resolveDesktopSpecs, selectDesktopRenderer } from './config.js'; +import type { DesktopStage } from './config.js'; +import { createDesktopReportSession } from './reporting/session.js'; +import { DesktopStopRequested } from './services.js'; +import { DesktopSupervisor } from './supervisor.js'; +import { runDesktopWdioSuite } from './wdio/run.js'; + +export async function runDesktopTestCli(argv = process.argv): Promise { + const program = new Command('desktop-test') + .description('Application-neutral native desktop lifecycle and WDIO suites') + .option('--config ', 'desktop.config file') + .requiredOption('--target ', 'registered target name') + .option('--json', 'keep stdout machine-readable') + .option('--paper', 'select macOS Paper (run prep after switching renderers)') + .option('--verbose', 'show informational service output'); + for (const stage of ['prep', 'bundle', 'build', 'run'] as const) program.command(stage).action(() => execute(stage)); + program.command('build-driver').action(() => execute('build-driver')); + program.command('serve').action(() => execute('serve')); + program + .command('test') + .requiredOption('--suite ', 'configured suite') + .option('--list', 'resolve exact spec groups without launching') + .action((options: { suite: string; list?: boolean }) => execute('test', options)); + await program.parseAsync(argv); + + async function execute( + action: DesktopStage | 'build-driver' | 'serve' | 'test', + test?: { suite: string; list?: boolean }, + ): Promise { + const options = program.opts<{ config?: string; target: string; json?: boolean; paper?: boolean; verbose?: boolean }>(); + const config = await loadDesktopConfig(options.config); + let target = config.targets[options.target]; + if (!target) throw new Error(`Unknown target "${options.target}".`); + if (options.paper) target = selectDesktopRenderer(target, 'paper'); + const suite = test ? config.config.suites?.[test.suite] : undefined; + if (test && !suite) throw new Error(`Unknown suite "${test.suite}".`); + if (suite?.kind === 'wdio') { + const specs = resolveDesktopSpecs(suite, target.endpoint, config.projectRoot); + if (test?.list) { + process.stdout.write(JSON.stringify({ endpoint: target.endpoint, specs }, null, 2) + '\n'); + return; + } + } else if (test?.list) throw new Error('--list currently requires a standalone WDIO suite.'); + const session = await createDesktopReportSession({ + ...config.reporting, + kind: action === 'test' ? 'tests' : 'lifecycle', + optional: suite?.kind === 'wdio' && suite.optional, + context: { endpoint: target.endpoint, renderer: target.host.renderer }, + structured: options.json, + verbose: options.verbose, + onFatal: (error) => owner.services.fail(error), + }); + const { reporter } = session; + const owner = new DesktopSupervisor({ target, reporter, reportSession: session, integration: config.config.integration }); + session.onStop?.(() => owner.services.requestStop()); + const interrupt = () => + action === 'serve' ? owner.services.requestStop() : owner.services.fail(new Error('Desktop run interrupted.')); + process.once('SIGINT', interrupt); + process.once('SIGTERM', interrupt); + let failure: unknown; + try { + if (action === 'build-driver') { + const artifact = await buildNativeDesktopDriver({ + platform: target.endpoint, + cacheRoot: target.nativeDriver?.cacheRoot, + signal: owner.services.controller.signal, + }); + reporter.emit({ + kind: 'diagnostic', + payload: { level: 'info', message: `Verified helper ${artifact.artifactId} at ${artifact.executablePath}` }, + }); + } else if (config.config.integration?.run && suite?.kind !== 'wdio') { + await owner.open(['test', 'serve', 'run'].includes(action)); + if (action === 'test' || action === 'serve') await owner.startConfiguredServices(); + await owner.services.guard(config.config.integration.run({ config, target, supervisor: owner, action, suite: test?.suite })); + } else if (action !== 'test' && action !== 'serve') { + await owner.open(action === 'run'); + for (const command of desktopStageCommands(target, action)) { + await owner.services.guard(owner.runner.run(owner.prepare(command, `${action} ${target.endpoint}`))); + } + } else { + await owner.start(); + if (action === 'serve') { + if (options.json) + process.stdout.write(JSON.stringify({ url: owner.server.url, targetId: target.id, runId: reporter.runId }) + '\n'); + await owner.services.guard(new Promise(() => undefined)); + } else if (suite?.kind === 'wdio') await runDesktopWdioSuite(owner, config, test!.suite); + else throw new Error('The imported integration does not implement this suite.'); + } + } catch (error) { + if (!(error instanceof DesktopStopRequested)) { + failure = error; + reporter.fail(error); + } + } finally { + process.off('SIGINT', interrupt); + process.off('SIGTERM', interrupt); + try { + await owner.close(); + } catch (error) { + reporter.fail(error, 'cleanup'); + failure = failure === undefined ? error : new AggregateError([failure, error], 'Desktop execution and cleanup failed.'); + } + } + process.exitCode = await session.close(failure); + if (options.json && action !== 'serve') + process.stdout.write(JSON.stringify(session.result ?? { runId: reporter.runId, delegated: true, exitCode: process.exitCode }) + '\n'); + } +} diff --git a/packages/agentic/desktop-test/src/commands.ts b/packages/agentic/desktop-test/src/commands.ts new file mode 100644 index 0000000000..fdbc0ac0e5 --- /dev/null +++ b/packages/agentic/desktop-test/src/commands.ts @@ -0,0 +1,325 @@ +import { spawn } from 'node:child_process'; +import fs from 'node:fs'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { StringDecoder } from 'node:string_decoder'; + +import type { DesktopCommand } from './config.js'; +import { DesktopRunReporter } from './reporting/collector.js'; +import type { DesktopReportWriter, ReportOutput } from './reporting/collector.js'; +import type { EventContext } from './reporting/protocol.js'; +import { DESKTOP_REPORT_CONTEXT, DESKTOP_REPORT_SCOPE, requestReportStop } from './reporting/transport.js'; + +export type PreparedDesktopCommand = DesktopCommand & { + args: readonly string[]; + cwd: string; + env: Readonly>; + label?: string; + category?: 'service' | 'worker' | 'phase'; + context?: EventContext; +}; +export type RunningDesktopCommand = { + completed: Promise; + logPath?: string; + readonly failure?: unknown; + stop(options?: { failed?: boolean }): Promise; +}; +export interface DesktopCommandRunner { + run(command: PreparedDesktopCommand): Promise; + start(command: PreparedDesktopCommand): RunningDesktopCommand; +} +export type NodeDesktopCommandRunnerOptions = { + reporter?: DesktopReportWriter; + environment?: Readonly>; + logDirectory?: string; + output?: ReportOutput; + errorOutput?: ReportOutput; + verbose?: boolean; + shutdownTimeoutMs?: number; + maxLogBytes?: number; +}; + +export class NodeDesktopCommandRunner implements DesktopCommandRunner { + private reporterValue?: DesktopReportWriter; + private ownedReporter?: DesktopRunReporter; + private readonly active = new Set(); + private readonly options: NodeDesktopCommandRunnerOptions; + constructor(options: NodeDesktopCommandRunnerOptions = {}) { + this.options = options; + this.reporterValue = options.reporter; + } + get reporter(): DesktopReportWriter | undefined { + return this.reporterValue; + } + + async run(command: PreparedDesktopCommand): Promise { + const running = this.start({ ...command, category: command.category ?? 'phase' }); + let failure: unknown; + try { + const code = await running.completed; + if (code !== 0) + throw ( + running.failure ?? new Error(`Command "${command.label ?? command.command}" exited with code ${code}. Log: ${running.logPath}`) + ); + } catch (error) { + failure = error; + } + try { + await running.stop({ failed: failure !== undefined }); + } catch (error) { + throw new AggregateError(failure === undefined ? [error] : [failure, error], 'Command execution or cleanup failed.'); + } + if (failure !== undefined) throw failure; + } + + start(command: PreparedDesktopCommand): RunningDesktopCommand { + const reporter = + (this.reporterValue ??= + this.ownedReporter = + new DesktopRunReporter({ + artifactsRoot: this.options.logDirectory ?? path.join(command.cwd, 'artifacts', 'desktop-commands'), + kind: 'lifecycle', + output: this.options.output, + errorOutput: this.options.errorOutput, + verbose: this.options.verbose, + })); + for (const [key, value] of Object.entries({ ...process.env, ...command.env, ...this.options.environment })) { + if (/nonce|token|password|secret|authorization/i.test(key) && value) reporter.addSecret(value); + } + const id = `command-${randomUUID()}`; + const emit = reporter.source(id); + const relativeLog = `commands/${id}.log`; + const logPath = reporter.artifactPath(relativeLog); + fs.mkdirSync(path.dirname(logPath), { recursive: true }); + fs.writeFileSync(logPath, reporter.redact(`$ ${[command.command, ...command.args].map((arg) => JSON.stringify(arg)).join(' ')}\n`), { + flag: 'wx', + mode: 0o600, + }); + emit( + { kind: 'scope-start', payload: { label: command.label ?? command.command, category: command.category ?? 'service' } }, + id, + command.context, + ); + emit({ kind: 'artifact', payload: { path: relativeLog, description: command.label ?? command.command } }, id, command.context); + const child = spawn(command.command, [...command.args], { + cwd: command.cwd, + env: { ...process.env, ...command.env, ...this.options.environment, [DESKTOP_REPORT_SCOPE]: id }, + detached: process.platform !== 'win32', + shell: windowsShell(command.command), + stdio: ['ignore', 'pipe', 'pipe'], + windowsHide: true, + }); + let stopped = false; + let closed = false; + let failed: unknown; + let exitFailure: Error | undefined; + let stopPromise: Promise | undefined; + let closeDeadline: ReturnType | undefined; + let logBytes = fs.statSync(logPath).size; + const capture = (stream: 'stdout' | 'stderr') => { + const decoder = new StringDecoder('utf8'); + let buffered = ''; + const record = (text: string) => { + const redacted = reporter.redact(text); + logBytes += Buffer.byteLength(redacted); + if (logBytes > (this.options.maxLogBytes ?? 64 * 1024 * 1024)) throw new Error(`Command log exceeded its size limit: ${logPath}`); + // Stream attribution is retained; ordering is the collector's arrival order, not a claim about producer write order. + fs.appendFileSync(logPath, `[${stream}] ${redacted}`, { mode: 0o600 }); + emit({ kind: 'output', payload: { stream, text: redacted } }, id, command.context); + }; + const consume = (text: string, flush: boolean) => { + buffered += text; + let newline: number; + while ((newline = buffered.indexOf('\n')) >= 0) { + record(buffered.slice(0, newline + 1)); + buffered = buffered.slice(newline + 1); + } + if (Buffer.byteLength(buffered) > 1024 * 1024) + throw new Error('Unterminated child output line exceeded 1 MiB; refusing silent truncation.'); + if (flush && buffered) { + record(buffered + '\n'); + buffered = ''; + } + }; + const onFailure = (error: unknown) => { + failed ??= error; + reporter.fail(error, 'reporting', id); + child.kill(); + }; + child[stream].on('data', (chunk: Buffer) => { + try { + consume(decoder.write(chunk), false); + } catch (error) { + onFailure(error); + } + }); + child[stream].on('end', () => { + try { + consume(decoder.end(), true); + } catch (error) { + onFailure(error); + } + }); + child[stream].on('error', onFailure); + }; + capture('stdout'); + capture('stderr'); + const completion = new Promise((resolve, reject) => { + child.once('error', (error) => { + failed ??= error; + }); + child.once('exit', () => { + closeDeadline = setTimeout(() => { + failed ??= new Error( + `Child exited but inherited output streams did not close within the drain deadline: ${command.label ?? command.command}`, + ); + child.stdout.destroy(); + child.stderr.destroy(); + }, this.options.shutdownTimeoutMs ?? 10_000); + }); + child.once('close', (code, signal) => { + clearTimeout(closeDeadline); + closed = true; + if (code !== 0 && !stopped) + exitFailure = new Error(`Command "${command.label ?? command.command}" exited with code ${code ?? signal}. Log: ${logPath}`); + if (failed !== undefined) reject(failed); + else resolve(code ?? (signal ? 1 : 0)); + }); + }); + // Owners observe completed; this branch only makes unexpected failures visible immediately. + void completion.then( + (code) => { + if (code !== 0 && !stopped && command.category !== 'worker') reporter.fail(exitFailure, 'infrastructure', id); + }, + (error: unknown) => reporter.fail(error, 'infrastructure', id), + ); + const running: RunningDesktopCommand = { + completed: completion, + logPath, + get failure() { + return failed ?? exitFailure; + }, + stop: (options = {}) => + (stopPromise ??= (async () => { + stopped = true; + let cleanupError: unknown; + const wasRunning = !closed; + try { + const encoded = + this.options.environment?.[DESKTOP_REPORT_CONTEXT] ?? + command.env[DESKTOP_REPORT_CONTEXT] ?? + process.env[DESKTOP_REPORT_CONTEXT]; + const controlled = await requestReportStop(id, encoded, this.options.shutdownTimeoutMs ?? 10_000); + if (controlled && !closed) + await deadline(completion, this.options.shutdownTimeoutMs ?? 10_000, 'Gracefully stopping owned Node services'); + } catch (error) { + cleanupError = error; + reporter.fail(error, 'cleanup', id); + } + if (wasRunning && child.pid) { + if (!closed) { + try { + await stopProcessTree(child.pid, completion, this.options.shutdownTimeoutMs ?? 10_000); + } catch (error) { + cleanupError ??= error; + reporter.fail(error, 'cleanup', id); + } + } + } + let code = 1; + try { + code = await deadline(completion, this.options.shutdownTimeoutMs ?? 10_000, 'Draining command output'); + } catch (error) { + failed ??= error; + } + const status = + options.failed || failed !== undefined || cleanupError !== undefined || (!wasRunning && code !== 0) ? 'failed' : 'passed'; + emit({ kind: 'scope-end', payload: { status, logPath: relativeLog } }, id, command.context); + emit({ kind: 'producer-finished', payload: { status } }, id, command.context); + this.active.delete(running); + if (cleanupError !== undefined) throw cleanupError; + })()), + }; + this.active.add(running); + return running; + } + + async close(): Promise { + const results = await Promise.allSettled([...this.active].map((command) => command.stop())); + await this.ownedReporter?.finish(); + const failures = results.flatMap((result) => (result.status === 'rejected' ? [result.reason as unknown] : [])); + if (failures.length) throw new AggregateError(failures, 'Owned commands did not stop cleanly.'); + } +} + +function windowsShell(command: string): boolean { + return ( + process.platform === 'win32' && + !['.exe', '.com'].includes(path.extname(command).toLowerCase()) && + !['node', 'powershell', 'pwsh'].includes(path.basename(command).toLowerCase()) + ); +} + +async function stopProcessTree(pid: number, completed: Promise, timeoutMs: number): Promise { + if (process.platform === 'win32') { + const killer = spawn('taskkill.exe', ['/PID', String(pid), '/T', '/F'], { stdio: ['ignore', 'pipe', 'pipe'], windowsHide: true }); + let diagnostic = ''; + killer.stdout.on('data', (data: Buffer) => { + diagnostic += data.toString(); + }); + killer.stderr.on('data', (data: Buffer) => { + diagnostic += data.toString(); + }); + const code = await deadline( + new Promise((resolve, reject) => { + killer.once('error', reject); + killer.once('close', resolve); + }), + timeoutMs, + `Stopping owned process tree ${pid}`, + ); + if (code !== 0) { + try { + process.kill(pid, 0); + } catch (error) { + if (isMissingProcess(error)) return; + throw error; + } + throw new Error(`Could not stop owned process tree ${pid}: ${diagnostic}`); + } + } else { + try { + process.kill(-pid, 'SIGTERM'); + } catch (error) { + if (!isMissingProcess(error)) throw error; + } + try { + await deadline(completed, timeoutMs, `Stopping owned process group ${pid}`); + } catch (error) { + try { + process.kill(-pid, 'SIGKILL'); + } catch (killError) { + if (!isMissingProcess(killError)) throw new AggregateError([error, killError], 'Owned process group did not stop.'); + } + } + } + await deadline(completed, timeoutMs, `Waiting for owned process tree ${pid}`); +} + +function isMissingProcess(error: unknown): boolean { + return error instanceof Error && 'code' in error && error.code === 'ESRCH'; +} + +export async function deadline(promise: Promise, timeoutMs: number, label: string): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error(`${label} exceeded ${timeoutMs}ms.`)), timeoutMs); + }), + ]); + } finally { + clearTimeout(timer); + } +} diff --git a/packages/agentic/desktop-test/src/config.ts b/packages/agentic/desktop-test/src/config.ts new file mode 100644 index 0000000000..18826ce2f8 --- /dev/null +++ b/packages/agentic/desktop-test/src/config.ts @@ -0,0 +1,402 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { createHash } from 'node:crypto'; + +import type { DesktopEndpoint, DesktopRenderer, NativeDesktopApplicationDescriptor } from '@fluentui-react-native/desktop-driver'; +import type { DesktopIntegration } from './integration.ts'; + +export type DesktopCommand = { + command: string; + args?: readonly string[]; + cwd?: string; + env?: Readonly>; +}; + +export type DesktopReadiness = { kind: 'native'; testId?: string; timeoutMs?: number }; +export type DesktopStage = 'prep' | 'bundle' | 'build' | 'run'; +export type DesktopHostConfig = { + kind: 'installed' | 'react-native-test-app' | 'rex'; + application?: NativeDesktopApplicationDescriptor; + ownership?: 'session' | 'external'; + renderer?: DesktopRenderer; + appConfigPath?: string; + component?: string; + windowTitle?: string; + bundleRoot?: string; +}; + +export type DesktopTargetConfig = { + endpoint?: DesktopEndpoint; + host: DesktopHostConfig; + readiness?: DesktopReadiness; + commands?: Partial>; + services?: readonly { id: string; command: DesktopCommand; readyUrl?: string }[]; + launchTimeoutMs?: number; + nativeDriver?: { cacheRoot?: string; helperPath?: string; installRoot?: string }; +}; + +export type DesktopTargetReference = { use: { config: string; target: string } }; +export type DesktopSpecs = readonly (string | readonly string[])[]; +export type DesktopWdioSuite = { + kind: 'wdio'; + framework: 'jasmine' | 'mocha' | 'cucumber'; + specs: DesktopSpecs | Partial>; + hooks?: string; + readiness?: DesktopReadiness; + maxInstances?: 1; + specFileRetries?: number; + bail?: number; + timeoutMs?: number; + optional?: boolean; +}; + +export type DesktopReportingConfig = { + artifactsRoot?: string; + console?: 'auto' | 'plain' | 'github-actions'; + junit?: boolean; + summary?: 'auto' | false; +}; + +export type DesktopConfig = { + schemaVersion: 1; + projectRoot?: string | URL; + targets: Readonly>; + suites?: Readonly>; + integration?: DesktopIntegration; + reporting?: DesktopReportingConfig; +}; + +export type ResolvedDesktopTarget = DesktopTargetConfig & { + endpoint: DesktopEndpoint; + id: string; + instanceId: string; + projectRoot: string; + configPath: string; +}; + +export type ResolvedDesktopConfig = { + config: DesktopConfig; + configPath: string; + projectRoot: string; + targets: Readonly>; + reporting: DesktopReportingConfig & { artifactsRoot: string }; +}; + +const configNames = ['desktop.config.mts', 'desktop.config.ts', 'desktop.config.mjs', 'desktop.config.js', 'desktop.config.cjs']; +const endpoints = ['macos', 'windows', 'win32'] as const; +const stages = ['prep', 'bundle', 'build', 'run'] as const; + +/** A structural, source-loadable entry point; config generation does not require a repository build. */ +export function defineDesktopConfig(config: T): T { + validateDesktopConfig(config); + return config; +} + +export function validateDesktopConfig(value: unknown): asserts value is DesktopConfig { + const config = object(value, 'desktop config'); + keys(config, ['schemaVersion', 'projectRoot', 'targets', 'suites', 'integration', 'reporting'], 'desktop config'); + if (config.schemaVersion !== 1) throw new TypeError('Desktop config schemaVersion must be 1.'); + if (config.projectRoot !== undefined && !(config.projectRoot instanceof URL)) text(config.projectRoot, 'projectRoot'); + const targets = object(config.targets, 'targets'); + if (Object.keys(targets).length === 0) throw new TypeError('Desktop config must declare at least one target.'); + for (const [id, entry] of Object.entries(targets)) { + text(id, 'target id'); + const target = object(entry, `target ${id}`); + if ('use' in target) { + keys(target, ['use'], `referenced target ${id} (overrides are ambiguous)`); + const reference = object(target.use, `${id}.use`); + keys(reference, ['config', 'target'], `${id}.use`); + text(reference.config, `${id}.use.config`); + text(reference.target, `${id}.use.target`); + continue; + } + keys(target, ['endpoint', 'host', 'readiness', 'commands', 'services', 'launchTimeoutMs', 'nativeDriver'], `target ${id}`); + oneOf(target.endpoint ?? id, endpoints, `${id}.endpoint`); + const host = object(target.host, `${id}.host`); + keys(host, ['kind', 'application', 'ownership', 'renderer', 'appConfigPath', 'component', 'windowTitle', 'bundleRoot'], `${id}.host`); + oneOf(host.kind, ['installed', 'react-native-test-app', 'rex'], `${id}.host.kind`); + optionalEnum(host.ownership, ['session', 'external'], `${id}.host.ownership`); + optionalEnum(host.renderer, ['fabric', 'paper'], `${id}.host.renderer`); + for (const field of ['appConfigPath', 'component', 'windowTitle', 'bundleRoot']) { + if (host[field] !== undefined) text(host[field], `${id}.host.${field}`); + } + if (host.kind === 'rex' && (target.endpoint ?? id) !== 'win32') throw new TypeError('REX requires the win32 endpoint.'); + if (host.kind === 'react-native-test-app' && (target.endpoint ?? id) === 'win32') { + throw new TypeError('Win32 uses REX, not react-native-test-app.'); + } + if (host.application !== undefined) validateApplication(host.application, `${id}.host.application`); + if (host.kind === 'installed' && host.application === undefined) + throw new TypeError(`${id}: installed hosts require application identity.`); + if (target.readiness !== undefined) validateReadiness(target.readiness); + positive(target.launchTimeoutMs, `${id}.launchTimeoutMs`); + if (target.nativeDriver !== undefined) { + const driver = object(target.nativeDriver, `${id}.nativeDriver`); + keys(driver, ['cacheRoot', 'helperPath', 'installRoot'], `${id}.nativeDriver`); + for (const [field, setting] of Object.entries(driver)) text(setting, `${id}.nativeDriver.${field}`); + } + if (target.commands !== undefined) { + const commands = object(target.commands, `${id}.commands`); + keys(commands, stages, `${id}.commands`); + for (const command of Object.values(commands)) { + if (command !== false) for (const item of Array.isArray(command) ? command : [command]) validateCommand(item); + } + } + if (target.services !== undefined) { + if (!Array.isArray(target.services)) throw new TypeError(`${id}.services must be an array.`); + const ids = new Set(); + for (const entry of target.services) { + const service = object(entry, `${id}.service`); + keys(service, ['id', 'command', 'readyUrl'], `${id}.service`); + text(service.id, 'service id'); + if (ids.has(service.id)) throw new TypeError(`Duplicate service ${service.id}.`); + ids.add(service.id); + validateCommand(service.command); + if (service.readyUrl !== undefined) { + text(service.readyUrl, 'readyUrl'); + const url = new URL(service.readyUrl); + if (!['http:', 'https:'].includes(url.protocol) || !['127.0.0.1', 'localhost', '[::1]'].includes(url.hostname)) { + throw new TypeError('Service readiness URLs must be loopback HTTP URLs.'); + } + } + } + } + } + if (config.suites !== undefined) { + for (const [id, entry] of Object.entries(object(config.suites, 'suites'))) { + const suite = object(entry, `suite ${id}`); + oneOf(suite.kind, ['wdio', 'csf-wdio'], `${id}.kind`); + if (suite.kind === 'csf-wdio') { + keys(suite, ['kind'], `suite ${id}`); + if (!config.integration) throw new TypeError('CSF suites require an explicitly imported integration.'); + continue; + } + keys( + suite, + ['kind', 'framework', 'specs', 'hooks', 'readiness', 'maxInstances', 'specFileRetries', 'bail', 'timeoutMs', 'optional'], + `suite ${id}`, + ); + oneOf(suite.framework, ['jasmine', 'mocha', 'cucumber'], `${id}.framework`); + if (Array.isArray(suite.specs)) validateSpecs(suite.specs); + else { + const specs = object(suite.specs, `${id}.specs`); + keys(specs, endpoints, `${id}.specs`); + for (const selection of Object.values(specs)) validateSpecs(selection); + } + if (suite.maxInstances !== undefined && suite.maxInstances !== 1) throw new TypeError('Desktop suites require maxInstances: 1.'); + for (const field of ['specFileRetries', 'bail']) positive(suite[field], `${id}.${field}`, true); + positive(suite.timeoutMs, `${id}.timeoutMs`); + if (suite.hooks !== undefined) text(suite.hooks, `${id}.hooks`); + if (suite.readiness !== undefined) validateReadiness(suite.readiness); + if (suite.optional !== undefined && typeof suite.optional !== 'boolean') throw new TypeError(`${id}.optional must be boolean.`); + } + } + if (config.reporting !== undefined) { + const reporting = object(config.reporting, 'reporting'); + keys(reporting, ['artifactsRoot', 'console', 'junit', 'summary'], 'reporting'); + if (reporting.artifactsRoot !== undefined) text(reporting.artifactsRoot, 'reporting.artifactsRoot'); + optionalEnum(reporting.console, ['auto', 'plain', 'github-actions'], 'reporting.console'); + if (reporting.junit !== undefined && typeof reporting.junit !== 'boolean') throw new TypeError('reporting.junit must be boolean.'); + if (reporting.summary !== undefined && reporting.summary !== false && reporting.summary !== 'auto') { + throw new TypeError('reporting.summary must be auto or false.'); + } + } + if (config.integration !== undefined) { + const integration = object(config.integration, 'integration'); + text(integration.name, 'integration.name'); + if (typeof integration.validate !== 'function') throw new TypeError('An integration must expose validate().'); + integration.validate(); + } +} + +export async function loadDesktopConfig(configPath?: string, cwd = process.cwd()): Promise { + const filename = configPath + ? path.resolve(cwd, configPath) + : configNames.map((name) => path.resolve(cwd, name)).find((candidate) => fs.existsSync(candidate)); + if (!filename) throw new Error(`No ${configNames.join(', ')} found in ${cwd}.`); + const cache = new Map>(); + const read = (filename: string) => { + const file = fs.realpathSync(filename); + let pending = cache.get(file); + if (!pending) { + pending = (async () => { + const loaded = (await import(pathToFileURL(file).href)) as { default?: unknown; config?: unknown }; + const config = loaded.default ?? loaded.config; + validateDesktopConfig(config); + const root = fs.realpathSync( + config.projectRoot instanceof URL + ? fileURLToPath(config.projectRoot) + : path.resolve(path.dirname(file), config.projectRoot ?? '.'), + ); + return { config, root, file }; + })(); + cache.set(file, pending); + } + return pending; + }; + const resolve = async (filename: string, id: string, ancestry: readonly string[]): Promise => { + const { config, root, file } = await read(filename); + const key = `${file}#${id}`; + if (ancestry.includes(key)) throw new Error(`Desktop target reference cycle: ${[...ancestry, key].join(' -> ')}`); + const target = config.targets[id]; + if (!target) throw new Error(`Unknown desktop target "${id}" in ${file}.`); + if ('use' in target) return resolve(path.resolve(path.dirname(file), target.use.config), target.use.target, [...ancestry, key]); + const endpoint = target.endpoint ?? (id as DesktopEndpoint); + const host = { + ...target.host, + ...(target.host.appConfigPath ? { appConfigPath: path.resolve(root, target.host.appConfigPath) } : {}), + ...(target.host.bundleRoot ? { bundleRoot: path.resolve(root, target.host.bundleRoot) } : {}), + ...(target.host.application + ? { + application: { + ...target.host.application, + ...resolvePaths(root, target.host.application, ['executablePath', 'leasePath', 'workingDirectory']), + }, + } + : {}), + }; + const instanceId = targetInstanceId(root, endpoint, host.renderer); + return Object.freeze({ + ...target, + endpoint, + host: Object.freeze(host), + id: `${endpoint}-${instanceId}`, + instanceId, + projectRoot: root, + configPath: file, + nativeDriver: target.nativeDriver + ? { ...target.nativeDriver, ...resolvePaths(root, target.nativeDriver, ['cacheRoot', 'helperPath', 'installRoot']) } + : undefined, + }); + }; + const { config, root, file } = await read(filename); + const targets = Object.fromEntries(await Promise.all(Object.keys(config.targets).map(async (id) => [id, await resolve(file, id, [])]))); + return { + config, + configPath: file, + projectRoot: root, + targets: Object.freeze(targets), + reporting: { ...config.reporting, artifactsRoot: path.resolve(root, config.reporting?.artifactsRoot ?? 'artifacts/desktop') }, + }; +} + +export function selectDesktopRenderer(target: ResolvedDesktopTarget, renderer: DesktopRenderer): ResolvedDesktopTarget { + if (target.endpoint !== 'macos' || target.host.kind !== 'react-native-test-app') + throw new Error('Renderer switching requires a macOS RN Test App target; regenerate with prep after switching.'); + const instanceId = targetInstanceId(target.projectRoot, target.endpoint, renderer); + return Object.freeze({ + ...target, + host: Object.freeze({ ...target.host, renderer }), + instanceId, + id: `${target.endpoint}-${instanceId}`, + }); +} + +function targetInstanceId(root: string, endpoint: DesktopEndpoint, renderer?: DesktopRenderer): string { + return createHash('sha256') + .update(`${root}\0${endpoint}\0${renderer ?? 'native'}`) + .digest('hex') + .slice(0, 16); +} + +export function resolveDesktopSpecs(suite: DesktopWdioSuite, endpoint: DesktopEndpoint, root: string): (string | string[])[] { + const specs: DesktopSpecs | undefined = Array.isArray(suite.specs) + ? suite.specs + : (suite.specs as Partial>)[endpoint]; + if (!specs?.length) throw new Error(`No specs selected for ${endpoint}.`); + const resolve = (pattern: string) => { + const matches = [...fs.globSync(pattern, { cwd: root })].sort().map((file) => path.resolve(root, file)); + if (matches.length === 0) throw new Error(`Unmatched ${endpoint} spec pattern "${pattern}" in ${root}; refusing reduced coverage.`); + return matches; + }; + return specs.flatMap((entry) => (typeof entry === 'string' ? resolve(entry) : [entry.flatMap(resolve)])); +} + +function resolvePaths(root: string, settings: object, fields: readonly string[]): Record { + return Object.fromEntries( + Object.entries(settings) + .filter(([key, value]) => fields.includes(key) && typeof value === 'string') + .map(([key, value]) => [key, path.resolve(root, value)]), + ); +} + +function validateApplication(value: unknown, context: string): void { + const application = object(value, context); + keys( + application, + ['arguments', 'aumid', 'bundleIdentifier', 'executablePath', 'leaseNonce', 'leasePath', 'windowTitle', 'workingDirectory'], + context, + ); + for (const [field, setting] of Object.entries(application)) { + if (field === 'arguments') { + if (!Array.isArray(setting) || !setting.every((item) => typeof item === 'string')) + throw new TypeError(`${context}.arguments must be strings.`); + } else text(setting, `${context}.${field}`); + } + if (Boolean(application.leasePath) !== Boolean(application.leaseNonce)) + throw new TypeError(`${context}: leasePath and leaseNonce must be paired.`); +} + +function validateReadiness(value: unknown): void { + const readiness = object(value, 'readiness'); + keys(readiness, ['kind', 'testId', 'timeoutMs'], 'readiness'); + oneOf(readiness.kind, ['native'], 'readiness.kind'); + if (readiness.testId !== undefined) text(readiness.testId, 'readiness.testId'); + positive(readiness.timeoutMs, 'readiness.timeoutMs'); +} + +export function validateCommand(value: unknown): asserts value is DesktopCommand { + const command = object(value, 'command'); + keys(command, ['command', 'args', 'cwd', 'env'], 'command'); + text(command.command, 'command.command'); + if (command.cwd !== undefined) text(command.cwd, 'command.cwd'); + if (command.args !== undefined && (!Array.isArray(command.args) || !command.args.every((arg) => typeof arg === 'string'))) { + throw new TypeError('command.args must be strings.'); + } + if (command.env !== undefined) + for (const [key, value] of Object.entries(object(command.env, 'command.env'))) { + if (typeof value !== 'string') throw new TypeError(`command.env.${key} must be a string.`); + } +} + +function validateSpecs(value: unknown): void { + if (!Array.isArray(value) || value.length === 0) + throw new TypeError('specs must be a nonempty array (nested arrays preserve worker grouping).'); + for (const entry of value) { + if (Array.isArray(entry)) { + if (!entry.length) throw new TypeError('A spec group cannot be empty.'); + for (const pattern of entry) text(pattern, 'spec pattern'); + } else text(entry, 'spec pattern'); + } +} + +function object(value: unknown, context: string): Record { + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError(`${context} must be an object.`); + return value as Record; +} + +function keys(value: Record, allowed: readonly string[], context: string): void { + for (const key of Object.keys(value)) if (!allowed.includes(key)) throw new TypeError(`Unknown ${context} field "${key}".`); +} + +function text(value: unknown, context: string): asserts value is string { + if (typeof value !== 'string' || !value.trim() || value.includes('\0')) + throw new TypeError(`${context} must be a nonempty string without NUL.`); +} + +function oneOf(value: unknown, allowed: readonly string[], context: string): void { + if (typeof value !== 'string' || !allowed.includes(value)) throw new TypeError(`${context} must be one of ${allowed.join(', ')}.`); +} + +function optionalEnum(value: unknown, allowed: readonly string[], context: string): void { + if (value !== undefined) oneOf(value, allowed, context); +} + +function positive(value: unknown, context: string, zero = false): void { + if ( + value !== undefined && + (typeof value !== 'number' || !Number.isSafeInteger(value) || value < (zero ? 0 : 1) || value > 2_147_483_647) + ) { + throw new TypeError(`${context} must be an integer between ${zero ? 0 : 1} and 2147483647.`); + } +} diff --git a/packages/agentic/desktop-test/src/contracts.contract.cjs b/packages/agentic/desktop-test/src/contracts.contract.cjs new file mode 100644 index 0000000000..ccff35d9ac --- /dev/null +++ b/packages/agentic/desktop-test/src/contracts.contract.cjs @@ -0,0 +1,586 @@ +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const os = require('node:os'); +const { pathToFileURL } = require('node:url'); +const { Writable } = require('node:stream'); + +const mode = process.argv[2]; +const packageRoot = path.resolve(__dirname, '..'); +const lib = (filename) => pathToFileURL(path.join(packageRoot, 'lib', filename)).href; +const capture = () => { + let text = ''; + return { + write(chunk) { + text += chunk.toString(); + return true; + }, + text: () => text, + }; +}; +const waitFor = async (condition) => { + const start = Date.now(); + while (!condition()) { + assert(Date.now() - start < 5000, 'fixture deadline exceeded'); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +}; + +async function main() { + const { DesktopRunReporter } = await import(lib('reporting/collector.js')); + const { NodeDesktopCommandRunner } = await import(lib('commands.js')); + const { DesktopServiceScope } = await import(lib('services.js')); + const { loadDesktopConfig, defineDesktopConfig, resolveDesktopSpecs } = await import(lib('config.js')); + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'desktop-harness-contract-')); + const out = capture(); + const err = capture(); + const reporter = (options = {}) => + new DesktopRunReporter({ artifactsRoot: root, output: out, errorOutput: err, summary: false, ...options }); + const command = (source, label = 'fixture') => ({ command: process.execPath, args: ['-e', source], cwd: root, env: {}, label }); + try { + if (mode === 'config') { + const source = await import(pathToFileURL(path.join(packageRoot, 'src', 'config.ts')).href); + assert.equal(typeof source.defineDesktopConfig, 'function', 'config must be source-loadable'); + const app = { + schemaVersion: 1, + targets: { win32: { host: { kind: 'installed', application: { executablePath: 'app.exe', windowTitle: 'Fixture' } } } }, + }; + assert.throws(() => defineDesktopConfig({ ...app, unrecognized: true }), /Unknown/); + assert.throws( + () => defineDesktopConfig({ ...app, targets: { win32: { use: { config: 'app.mjs', target: 'win32' }, host: {} } } }), + /ambiguous/, + ); + fs.mkdirSync(path.join(root, 'app')); + fs.mkdirSync(path.join(root, 'suite')); + fs.writeFileSync(path.join(root, 'app', 'desktop.config.mjs'), `export default ${JSON.stringify(app)};`); + const suite = { + schemaVersion: 1, + targets: { alias: { use: { config: '../app/desktop.config.mjs', target: 'win32' } } }, + reporting: { artifactsRoot: 'reports' }, + }; + fs.writeFileSync(path.join(root, 'suite', 'desktop.config.mjs'), `export default ${JSON.stringify(suite)};`); + const direct = await loadDesktopConfig(path.join(root, 'app', 'desktop.config.mjs')); + const referred = await loadDesktopConfig(path.join(root, 'suite', 'desktop.config.mjs')); + assert.equal(direct.targets.win32.id, referred.targets.alias.id); + assert.equal(referred.targets.alias.host.application.executablePath, path.join(root, 'app', 'app.exe')); + assert.equal(referred.reporting.artifactsRoot, path.join(root, 'suite', 'reports')); + fs.writeFileSync( + path.join(root, 'cycle.mjs'), + 'export default {schemaVersion:1, targets:{win32:{use:{config:"cycle.mjs",target:"win32"}}}}', + ); + await assert.rejects(loadDesktopConfig(path.join(root, 'cycle.mjs')), /cycle/); + fs.writeFileSync(path.join(root, 'one.spec.mjs'), ''); + fs.writeFileSync(path.join(root, 'two.spec.mjs'), ''); + assert.deepEqual(resolveDesktopSpecs({ kind: 'wdio', framework: 'jasmine', specs: [['*.spec.mjs']] }, 'win32', root), [ + [path.join(root, 'one.spec.mjs'), path.join(root, 'two.spec.mjs')], + ]); + assert.throws( + () => resolveDesktopSpecs({ kind: 'wdio', framework: 'jasmine', specs: ['missing.*'] }, 'windows', root), + /refusing reduced coverage/, + ); + } else if (mode === 'reporting') { + const run = reporter({ secrets: ['secret-value'], console: 'github-actions' }); + run.emit({ kind: 'output', payload: { stream: 'stderr', text: '::error::untrusted secret-value\n' } }); + const error = run.failure(new Error('original secret-value', { cause: new Error('root cause') }), 'test'); + const title = 'retries \u{1f680} <&\u0001'; + run.emit({ + kind: 'test-result', + payload: { testId: 'retry', title, attempt: 0, status: 'failed', durationMs: 2, failure: error }, + }); + run.emit({ kind: 'test-result', payload: { testId: 'retry', title, attempt: 1, status: 'passed', durationMs: 3 } }); + const result = await run.finish(); + assert.equal(result.status, 'passed'); + assert.equal(result.counts.flaky, 1); + assert.equal(result.counts.selected, 1); + assert.equal(result.counts.retries, 1); + const journal = fs.readFileSync(path.join(run.root, 'events.jsonl'), 'utf8'); + assert(!journal.includes('secret-value')); + assert(journal.includes('root cause')); + assert(err.text().includes('::stop-commands::')); + assert.equal((err.text().match(/::error title=/g) || []).length, 1); + const junit = fs.readFileSync(path.join(run.root, 'junit.xml'), 'utf8'); + assert(junit.includes('attempt')); + assert(junit.includes('\u{1f680} <&')); + assert(!junit.includes('\u0001')); + const partialStack = reporter({ console: 'github-actions' }); + partialStack.emit({ + kind: 'test-result', + payload: { + testId: 'original-message', + title: 'original message', + attempt: 0, + status: 'failed', + durationMs: 1, + failure: partialStack.failure({ name: 'TypeError', message: 'original assertion message', stack: ' at fixture:1:2' }, 'test'), + }, + }); + await partialStack.finish(); + assert(err.text().includes('TypeError: original assertion message\n at fixture:1:2')); + assert(err.text().includes('TypeError: original assertion message%0A at fixture:1:2')); + for (const optional of [false, true]) { + const skipped = reporter({ optional }); + skipped.emit({ kind: 'work-selected', payload: { workId: 'unsupported', label: 'unsupported group' } }); + skipped.emit( + { kind: 'scope-start', payload: { label: 'unsupported group', category: 'worker' } }, + 'unsupported', + undefined, + 'unsupported', + ); + skipped.emit( + { + kind: 'test-result', + payload: { + testId: 'skip', + title: 'skip', + attempt: 0, + status: 'unsupported', + durationMs: 0, + reason: 'Physical input unavailable.', + }, + }, + 'unsupported', + ); + skipped.emit({ kind: 'scope-end', payload: { status: 'unsupported' } }, 'unsupported'); + const skippedResult = await skipped.finish(); + assert.equal(skippedResult.work[0].status, 'unsupported'); + assert.equal(skippedResult.status, optional ? 'skipped' : 'incomplete'); + } + assert.equal((await reporter().finish()).status, 'incomplete'); + const omitted = reporter(); + omitted.emit({ kind: 'work-selected', payload: { workId: 'omitted', label: 'selected group' } }); + omitted.emit({ + kind: 'test-result', + payload: { testId: 'other', title: 'completed test', attempt: 0, status: 'passed', durationMs: 1 }, + }); + const omittedResult = await omitted.finish(); + assert.equal(omittedResult.status, 'incomplete'); + assert.equal(omittedResult.work[0].status, 'not-run'); + assert(fs.readFileSync(path.join(omitted.root, 'junit.xml'), 'utf8').includes('selected group')); + const missing = reporter(); + missing.emit({ kind: 'test-start', payload: { testId: 'crash', title: 'crashed', attempt: 0 } }); + assert.equal((await missing.finish()).status, 'incomplete'); + const malformed = reporter({ kind: 'lifecycle' }); + assert.throws(() => malformed.accept({ schemaVersion: 2 }), /Unsupported|Unknown/); + const source = malformed.source('child'); + source({ kind: 'producer-finished', payload: { status: 'passed' } }); + assert.equal((await malformed.finish()).status, 'passed'); + const stuck = new Writable({ + write(_chunk, _encoding, _callback) { + // Simulate a sink that never acknowledges its write. + }, + }); + const slow = reporter({ kind: 'lifecycle', output: stuck, flushTimeoutMs: 20 }); + slow.emit({ kind: 'diagnostic', payload: { level: 'info', message: 'slow sink' } }); + assert.equal((await slow.finish()).status, 'failed'); + stuck.destroy(); + const failedWrite = reporter({ kind: 'lifecycle' }); + const rename = fs.renameSync; + let fail = true; + fs.renameSync = (...args) => { + if (fail && args[1] === path.join(failedWrite.root, 'junit.xml')) { + fail = false; + throw new Error('disk fixture'); + } + return rename(...args); + }; + try { + assert.equal((await failedWrite.finish()).status, 'failed'); + } finally { + fs.renameSync = rename; + } + const summaryPath = path.join(root, 'summary.md'); + const priorSummary = process.env.GITHUB_STEP_SUMMARY; + process.env.GITHUB_STEP_SUMMARY = summaryPath; + const late = reporter({ + kind: 'lifecycle', + summary: 'auto', + output: { + write(text) { + if (text.includes('[desktop] PASSED:')) throw new Error('late terminal fixture'); + return true; + }, + }, + }); + try { + const final = await late.finish(); + assert.equal(final.status, 'failed'); + assert.equal(JSON.parse(fs.readFileSync(path.join(late.root, 'run.json'), 'utf8')).status, final.status); + const events = fs + .readFileSync(path.join(late.root, 'events.jsonl'), 'utf8') + .trim() + .split('\n') + .map((line) => JSON.parse(line)); + assert.equal(events.at(-1).payload.status, final.status); + assert(fs.readFileSync(summaryPath, 'utf8').includes('failed')); + assert(fs.readFileSync(path.join(late.root, 'junit.xml'), 'utf8').includes('late terminal fixture')); + } finally { + if (priorSummary === undefined) delete process.env.GITHUB_STEP_SUMMARY; + else process.env.GITHUB_STEP_SUMMARY = priorSummary; + } + } else if (mode === 'commands') { + const run = reporter({ kind: 'lifecycle', console: 'github-actions' }); + const runner = new NodeDesktopCommandRunner({ reporter: run }); + const running = runner.start( + command( + "process.stdout.write('ready\\n'); setTimeout(()=>process.stderr.write('live diagnostic\\n'),30); setInterval(()=>{},1000);", + ), + ); + try { + await waitFor(() => err.text().includes('live diagnostic')); + assert(!run.resultSnapshot, 'diagnostic must arrive before shutdown'); + } finally { + await running.stop(); + } + await Promise.all( + ['one', 'two'].map((label) => + runner.run({ + ...command( + `process.stdout.write('::warning::${label}\\n'); process.stdout.write(Buffer.from([0xe2])); setTimeout(()=>process.stdout.write(Buffer.from([0x82,0xac,10])),10);`, + label, + ), + category: 'worker', + }), + ), + ); + const result = await run.finish(); + assert.equal(result.status, 'passed'); + const logs = fs + .readdirSync(path.join(run.root, 'commands')) + .map((file) => fs.readFileSync(path.join(run.root, 'commands', file), 'utf8')) + .join(''); + assert(logs.includes('\u20ac')); + assert(logs.includes('[stderr] live diagnostic')); + assert.equal((out.text().match(/::group::/g) || []).length, 2); + assert.equal((out.text().match(/::endgroup::/g) || []).length, 2); + const groupOne = out + .text() + .slice(out.text().indexOf('::group::one'), out.text().indexOf('::endgroup::', out.text().indexOf('::group::one'))); + assert(groupOne.includes('::stop-commands::')); + assert(!groupOne.includes('::warning::two')); + } else if (mode === 'transport') { + const { startReportTransport } = await import(lib('reporting/transport.js')); + const run = reporter({ kind: 'lifecycle' }); + const transport = await startReportTransport(run); + const runner = new NodeDesktopCommandRunner({ reporter: run, environment: transport.environment }); + const producer = (id) => + `const {connectReportProducer}=await import(${JSON.stringify(lib('reporting/transport.js'))}); const p=await connectReportProducer(${JSON.stringify(id)}); p.emit({kind:'diagnostic',payload:{level:'warning',message:'hello ${id}'}}); await p.finish();`; + const child = `const {spawn}=require('node:child_process'); (async()=>{ ${producer('parent')} const child=spawn(process.execPath,['--input-type=module','-e',${JSON.stringify(producer('grandchild'))}],{stdio:'inherit'}); child.on('close',code=>process.exitCode=code); })().catch(error=>{console.error(error);process.exitCode=1;});`; + try { + await runner.run(command(child)); + } finally { + await transport.close(); + } + assert.equal((await run.finish()).status, 'passed'); + assert(err.text().includes('[grandchild]')); + const events = fs.readFileSync(path.join(run.root, 'events.jsonl'), 'utf8'); + assert(events.includes('"sourceId":"grandchild"')); + assert(!events.includes(JSON.parse(Object.values(transport.environment)[0]).nonce)); + const delegatedRun = reporter({ kind: 'lifecycle' }); + const delegatedTransport = await startReportTransport(delegatedRun); + const delegatedRunner = new NodeDesktopCommandRunner({ reporter: delegatedRun, environment: delegatedTransport.environment }); + const filename = path.join(root, 'delegated.mjs'); + fs.writeFileSync( + filename, + ` + import {createDesktopReportSession} from ${JSON.stringify(lib('reporting/session.js'))}; + const report = await createDesktopReportSession({artifactsRoot:'.',kind:'lifecycle'}); + report.reporter.addSecret('child-only-secret'); + const nested = report.reporter.source('nested'); + nested({kind:'diagnostic',payload:{level:'warning',message:'child-only-secret'}}); + nested({kind:'work-selected',payload:{workId:'delegated-case',label:'delegated case'}}); + nested({kind:'scope-start',payload:{label:'delegated case',category:'worker'}},'delegated-case',undefined,'delegated-case'); + nested({kind:'test-result',payload:{testId:'delegated-case',title:'delegated case',attempt:0,status:'passed',durationMs:1}},'delegated-case'); + nested({kind:'scope-end',payload:{status:'passed'}},'delegated-case'); + nested({kind:'producer-finished',payload:{status:'passed'}}); + await report.reporter.assertSuccessful(['delegated-case']); + await (await import('node:assert/strict')).rejects(report.reporter.assertSuccessful(['missing-case']), /unknown or empty/); + await report.close();`, + ); + try { + await delegatedRunner.run({ command: process.execPath, args: [filename], cwd: root, env: {} }); + } finally { + await delegatedTransport.close(); + } + assert.equal((await delegatedRun.finish()).status, 'passed'); + assert(!fs.readFileSync(path.join(delegatedRun.root, 'events.jsonl'), 'utf8').includes('child-only-secret')); + } else if (mode === 'delegated-cli') { + const { startReportTransport } = await import(lib('reporting/transport.js')); + const run = reporter({ kind: 'lifecycle' }); + const transport = await startReportTransport(run); + const runner = new NodeDesktopCommandRunner({ reporter: run, environment: transport.environment }); + const configPath = path.join(root, 'desktop.config.mjs'); + fs.writeFileSync( + configPath, + `export default ${JSON.stringify({ + schemaVersion: 1, + targets: { + win32: { + host: { kind: 'installed', application: { executablePath: 'not-launched.exe' } }, + commands: { prep: { command: process.execPath, args: ['-e', "process.stderr.write('nested CLI diagnostic\\n')"] } }, + }, + }, + reporting: { artifactsRoot: path.join(root, 'competing-collector') }, + })}`, + ); + try { + await runner.run({ + command: process.execPath, + args: [path.join(packageRoot, 'config', 'cli.cjs'), '--config', configPath, '--target', 'win32', '--json', 'prep'], + cwd: root, + env: { GITHUB_ACTIONS: 'true' }, + }); + } finally { + await runner.close(); + await transport.close(); + } + assert(!fs.existsSync(path.join(root, 'competing-collector'))); + assert(err.text().includes('nested CLI diagnostic')); + const result = await run.finish(); + assert.equal(result.status, 'passed', JSON.stringify(result)); + const events = fs + .readFileSync(path.join(run.root, 'events.jsonl'), 'utf8') + .trim() + .split('\n') + .map((line) => JSON.parse(line)); + const machine = events.find( + (event) => event.kind === 'output' && event.payload.stream === 'stdout' && event.payload.text.includes('"delegated"'), + ); + assert.deepEqual(JSON.parse(machine.payload.text), { runId: run.runId, delegated: true, exitCode: 0 }); + assert.equal((out.text().match(/\[desktop\] PASSED:/g) || []).length, 1); + } else if (mode === 'owned-service') { + const { startReportTransport } = await import(lib('reporting/transport.js')); + const run = reporter({ kind: 'lifecycle' }); + const transport = await startReportTransport(run); + const runner = new NodeDesktopCommandRunner({ reporter: run, environment: transport.environment, shutdownTimeoutMs: 10_000 }); + const descriptor = path.join(root, 'service.json'); + fs.writeFileSync( + descriptor, + JSON.stringify({ + command: { + command: process.execPath, + args: ['-e', "setTimeout(()=>process.stderr.write('grandchild live diagnostic\\n'),30);setInterval(()=>{},1000)"], + cwd: root, + }, + stopPath: path.join(root, 'service.stop'), + label: 'owned fixture', + }), + ); + const running = runner.start({ + command: process.execPath, + args: [path.join(packageRoot, 'config', 'service.cjs'), descriptor], + cwd: root, + env: {}, + }); + try { + await waitFor(() => err.text().includes('grandchild live diagnostic')); + } finally { + await running.stop(); + await transport.close(); + } + assert.equal(await running.completed, 0); + const result = await run.finish(); + assert.equal(result.status, 'passed', JSON.stringify(result)); + assert.equal((out.text().match(/\[desktop\] PASSED:/g) || []).length, 1, 'only the parent renders the run'); + const abandoned = reporter({ kind: 'lifecycle' }); + const abandonedTransport = await startReportTransport(abandoned); + const abandonedRunner = new NodeDesktopCommandRunner({ reporter: abandoned, environment: abandonedTransport.environment }); + const marker = path.join(root, 'owned-pid.json'); + const orphanDescriptor = path.join(root, 'orphan-service.json'); + fs.writeFileSync( + orphanDescriptor, + JSON.stringify({ + command: { + command: process.execPath, + args: [ + '-e', + `require('node:fs').writeFileSync(${JSON.stringify(marker)},JSON.stringify(process.pid));setInterval(()=>{},1000);`, + ], + cwd: root, + }, + stopPath: path.join(root, 'orphan-service.stop'), + label: 'orphan fixture', + }), + ); + const parent = abandonedRunner.start( + command(` + require('node:child_process').spawn(process.execPath, [${JSON.stringify(path.join(packageRoot, 'config', 'service.cjs'))}, ${JSON.stringify(orphanDescriptor)}], {stdio:'ignore',detached:true,windowsHide:true}); + setInterval(() => { if(require('node:fs').existsSync(${JSON.stringify(marker)})) process.exit(9); }, 50);`), + ); + try { + assert.equal(await parent.completed, 9); + } finally { + await parent.stop(); + await abandonedRunner.close(); + await abandonedTransport.close(); + } + assert.throws(() => process.kill(JSON.parse(fs.readFileSync(marker, 'utf8')), 0), /ESRCH|no such process/i); + const abandonedResult = await abandoned.finish(); + assert.equal(abandonedResult.status, 'failed'); + assert(!abandonedResult.failures.some(({ kind }) => kind === 'cleanup' || kind === 'reporting'), JSON.stringify(abandonedResult)); + } else if (mode === 'services') { + const run = reporter({ kind: 'lifecycle' }); + const runner = new NodeDesktopCommandRunner({ reporter: run }); + const services = new DesktopServiceScope(runner, run); + services.start('outer', command('setInterval(()=>{},1000);')); + services.start('metro', command("process.stderr.write('original metro failure\\n'); setTimeout(()=>process.exitCode=7,20);")); + try { + await assert.rejects(services.guard(new Promise(() => undefined)), /exited.*code 7/); + } finally { + await services.close(); + } + const result = await run.finish(); + assert.equal(result.status, 'failed'); + assert(err.text().includes('original metro failure')); + } else { + const { DesktopSupervisor } = await import(lib('supervisor.js')); + const { runDesktopWdioSuite } = await import(lib('wdio/run.js')); + const { FakeDesktopHost } = await import('@fluentui-react-native/desktop-driver/testing'); + const run = reporter({ context: { endpoint: 'win32', renderer: 'paper' } }); + const fixture = path.join(root, mode === 'typescript' ? 'spec.ts' : 'spec.mjs'); + const second = path.join(root, 'second.mjs'); + const marker = path.join(root, 'attempt'); + let source; + if (mode === 'typescript') { + fs.writeFileSync(path.join(root, 'helper.ts'), 'export const marker: string = "typescript";'); + fs.writeFileSync( + path.join(root, 'hooks.ts'), + "import { marker } from './helper'; export default { before() { globalThis.hookMarker = marker; } };", + ); + source = + "import { marker } from './helper'; describe('TypeScript fixture',()=>{it('preserves source imports in hooks and specs',()=>{const expected: string = marker;expect(globalThis.hookMarker).toBe(expected);});});"; + } else if (mode === 'retry') { + source = `import fs from 'node:fs'; describe('retry fixture',()=>{it('retains failed attempt',()=>{if(!fs.existsSync(${JSON.stringify(marker)})){fs.writeFileSync(${JSON.stringify(marker)},'attempt');throw new Error('first attempt');}});});`; + } else if (mode === 'hook-failure') { + source = + "describe('hook fixture',()=>{beforeAll(()=>{throw new Error('original setup failure');});it('does not execute',()=>{throw new Error('body must not run');});});"; + } else if (mode === 'crash') { + source = 'process.exit(17);'; + } else if (mode === 'bail') { + source = "describe('bail fixture',()=>{it('fails first',()=>{throw new Error('original bail failure');});});"; + fs.writeFileSync( + second, + "describe('omitted group',()=>{it('must not execute',()=>{throw new Error('bail did not stop scheduling');});});", + ); + } else { + source = + "describe('native WDIO fixture',()=>{beforeAll(()=>{globalThis.groupedMarker='preserved';});it('uses native display and focus without execute',async()=>{const e=await browser.$('~button-primary');await e.waitForDisplayed();await expect(await e.isDisplayed()).withContext('Jasmine matchers remain available').toBeTrue();await e.click();await expect(await e.isFocused()).toBeTrue();await expect(typeof browser.desktopOpenStory).toBe('undefined');});xit('explicit skip',()=>{});});"; + fs.writeFileSync( + second, + "describe('second grouped file',()=>{it('preserves Jasmine globals and grouping',()=>{expect(globalThis.groupedMarker).toBe('preserved');});});", + ); + } + fs.writeFileSync(fixture, source); + const target = { + endpoint: 'win32', + host: { kind: 'installed', application: { windowTitle: 'fake' }, renderer: 'paper' }, + id: 'plain-app', + instanceId: 'plain', + projectRoot: packageRoot, + configPath: path.join(root, 'desktop.config.mjs'), + }; + const host = new FakeDesktopHost({ endpoint: 'win32' }); + let launches = 0; + const launch = host.launch.bind(host); + host.launch = (...args) => { + launches++; + return launch(...args); + }; + const supervisor = new DesktopSupervisor({ + target, + reporter: run, + lockDirectory: path.join(root, 'locks'), + hostFactory: async () => host, + }); + const config = { + config: { + schemaVersion: 1, + targets: { win32: target }, + suites: { + fixture: { + kind: 'wdio', + framework: 'jasmine', + specs: mode === 'bail' ? [fixture, second] : [[fixture, ...(mode === 'wdio' ? [second] : [])]], + specFileRetries: mode === 'retry' ? 1 : 0, + }, + }, + }, + projectRoot: packageRoot, + configPath: target.configPath, + targets: { win32: target }, + reporting: { artifactsRoot: root }, + }; + if (mode === 'config-hook-failure') + config.config.suites.fixture.readiness = { kind: 'native', testId: 'missing-readiness-marker', timeoutMs: 50 }; + if (mode === 'typescript') config.config.suites.fixture.hooks = path.join(root, 'hooks.ts'); + if (mode === 'config-load-failure') { + config.config.suites.fixture.hooks = path.join(root, 'hooks.mts'); + fs.writeFileSync( + config.config.suites.fixture.hooks, + "throw new Error('original config import failure', { cause: new Error('original config cause') });", + ); + } + if (mode === 'bail') { + config.config.suites.fixture.specs = [fixture, second]; + config.config.suites.fixture.bail = 1; + } + let failure; + try { + await supervisor.start(); + const duplicate = new DesktopSupervisor({ + target, + reporter: run, + lockDirectory: path.join(root, 'locks'), + hostFactory: async () => host, + }); + await assert.rejects(duplicate.start(), /Another desktop workload/); + try { + await runDesktopWdioSuite(supervisor, config, 'fixture'); + } catch (error) { + failure = error; + run.fail(error); + } + } finally { + await supervisor.close(); + } + const result = await run.finish(); + if (['hook-failure', 'config-hook-failure', 'config-load-failure', 'crash', 'bail'].includes(mode)) { + assert(failure, `hook failure must fail launcher\n${JSON.stringify(result)}\n${err.text()}\n${out.text()}`); + assert.equal(result.status, 'failed'); + assert(!out.text().includes('PASSED WDIO fixture'), out.text()); + if (mode !== 'crash') + assert(!result.failures.some(({ kind }) => kind === 'cleanup' || kind === 'reporting'), JSON.stringify(result)); + if (mode === 'hook-failure') assert(JSON.stringify(result).includes('original setup failure'), JSON.stringify(result)); + if (mode === 'config-hook-failure') assert(JSON.stringify(result).includes('missing-readiness-marker'), JSON.stringify(result)); + if (mode === 'config-load-failure') { + assert(JSON.stringify(result).includes('original config import failure'), JSON.stringify(result)); + assert(JSON.stringify(result).includes('original config cause'), JSON.stringify(result)); + assert.equal(launches, 0); + assert(result.work.every(({ status }) => status === 'not-run')); + } + if (mode === 'bail') + assert( + result.work.some(({ status }) => status === 'not-run'), + JSON.stringify(result), + ); + } else { + assert.equal(failure, undefined, `${failure?.stack}\n${err.text()}\n${out.text()}`); + assert.equal(result.status, 'passed', JSON.stringify(result)); + assert.equal(launches, mode === 'retry' ? 2 : 1, 'WDIO must own exactly one session per grouped attempt'); + assert.equal(result.counts.flaky, mode === 'retry' ? 1 : 0, JSON.stringify(result)); + if (mode === 'wdio') { + assert.equal(result.counts.passed, 2); + assert.equal(result.counts.skipped, 1); + } + } + assert(!fs.existsSync(path.join(root, 'locks', 'interactive.lock'))); + } + process.stdout.write(JSON.stringify({ mode, passed: true })); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +} + +main().catch((error) => { + process.stderr.write(error.stack ?? String(error)); + process.exitCode = 1; +}); diff --git a/packages/agentic/desktop-test/src/contracts.test.ts b/packages/agentic/desktop-test/src/contracts.test.ts new file mode 100644 index 0000000000..8b333f8a26 --- /dev/null +++ b/packages/agentic/desktop-test/src/contracts.test.ts @@ -0,0 +1,50 @@ +import { spawn } from 'node:child_process'; +import path from 'node:path'; + +describe('neutral desktop harness contracts', () => { + test.each([ + 'config', + 'reporting', + 'commands', + 'transport', + 'delegated-cli', + 'services', + 'owned-service', + 'wdio', + 'typescript', + 'retry', + 'hook-failure', + 'config-hook-failure', + 'config-load-failure', + 'crash', + 'bail', + ])( + '%s contract', + async (mode) => { + const result = await runContract(mode); + expect(result).toEqual({ mode, passed: true }); + }, + 120_000, + ); +}); + +function runContract(mode: string): Promise { + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, [path.join(__dirname, 'contracts.contract.cjs'), mode], { stdio: ['ignore', 'pipe', 'pipe'] }); + const stdout: Buffer[] = []; + const stderr: Buffer[] = []; + child.stdout.on('data', (chunk: Buffer) => stdout.push(chunk)); + child.stderr.on('data', (chunk: Buffer) => stderr.push(chunk)); + child.once('error', reject); + child.once('close', (code) => { + if (code !== 0) reject(new Error(Buffer.concat(stderr).toString('utf8') || Buffer.concat(stdout).toString('utf8'))); + else { + try { + resolve(JSON.parse(Buffer.concat(stdout).toString('utf8'))); + } catch (error) { + reject(error); + } + } + }); + }); +} diff --git a/packages/agentic/desktop-test/src/index.ts b/packages/agentic/desktop-test/src/index.ts new file mode 100644 index 0000000000..158e3c23bb --- /dev/null +++ b/packages/agentic/desktop-test/src/index.ts @@ -0,0 +1,7 @@ +export { claimDesktopWorkload, DesktopSupervisor, startNativeDesktopService } from './supervisor.js'; +export type { DesktopSupervisorOptions } from './supervisor.js'; +export { DesktopServiceScope, DesktopStopRequested } from './services.js'; +export { defineDesktopConfig, loadDesktopConfig, resolveDesktopSpecs, validateDesktopConfig } from './config.js'; +export type { DesktopConfig, DesktopTargetConfig, DesktopWdioSuite, ResolvedDesktopConfig, ResolvedDesktopTarget } from './config.js'; +export type { DesktopIntegration } from './integration.js'; +export { desktopStageCommands, resolveDesktopApplication } from './adapters.js'; diff --git a/packages/agentic/desktop-test/src/integration.ts b/packages/agentic/desktop-test/src/integration.ts new file mode 100644 index 0000000000..0dce56433b --- /dev/null +++ b/packages/agentic/desktop-test/src/integration.ts @@ -0,0 +1,22 @@ +import type { DesktopHost, DesktopTarget } from '@fluentui-react-native/desktop-driver'; +import type { DesktopStage, ResolvedDesktopConfig, ResolvedDesktopTarget } from './config.js'; +import type { DesktopSupervisor } from './supervisor.js'; +import type { DesktopReportWriter } from './reporting/collector.js'; + +/** Trusted, explicitly imported local code; never included in capabilities or persisted manifests. */ +export type DesktopIntegration = { + name: string; + validate(): void; + createTarget?(context: { + target: ResolvedDesktopTarget; + host: DesktopHost; + reporter: DesktopReportWriter; + }): Promise>; + run?(context: { + config: ResolvedDesktopConfig; + target: ResolvedDesktopTarget; + supervisor: DesktopSupervisor; + action: DesktopStage | 'test' | 'serve'; + suite?: string; + }): Promise; +}; diff --git a/packages/agentic/desktop-test/src/ownedService.ts b/packages/agentic/desktop-test/src/ownedService.ts new file mode 100644 index 0000000000..e5f1d39587 --- /dev/null +++ b/packages/agentic/desktop-test/src/ownedService.ts @@ -0,0 +1,71 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { validateCommand } from './config.js'; +import { NodeDesktopCommandRunner } from './commands.js'; +import { createDesktopReportSession } from './reporting/session.js'; + +/** Bridges a generated PowerShell service descriptor into the parent's resource/reporting scope. */ +export async function runOwnedDesktopService(descriptorPath: string): Promise { + const value: unknown = JSON.parse(fs.readFileSync(descriptorPath, 'utf8')); + if ( + !value || + typeof value !== 'object' || + !('command' in value) || + !('stopPath' in value) || + !('label' in value) || + typeof value.stopPath !== 'string' || + !path.isAbsolute(value.stopPath) || + typeof value.label !== 'string' || + !value.label + ) { + throw new TypeError('Invalid owned desktop service descriptor.'); + } + validateCommand(value.command); + const command = value.command; + const cwd = path.resolve(command.cwd ?? process.cwd()); + const stopPath = value.stopPath; + const report = await createDesktopReportSession({ artifactsRoot: path.join(cwd, 'artifacts', 'desktop'), kind: 'lifecycle' }); + const runner = new NodeDesktopCommandRunner({ reporter: report.reporter, environment: report.environment }); + let stop!: () => void; + const stopped = new Promise((resolve) => { + stop = resolve; + }); + const interval = setInterval(() => { + if (fs.existsSync(stopPath)) stop(); + }, 100); + report.onStop?.(stop); + process.once('SIGINT', stop); + process.once('SIGTERM', stop); + let failure: unknown; + try { + const running = runner.start({ + ...command, + cwd, + args: command.args ?? [], + env: command.env ?? {}, + category: 'service', + label: value.label, + context: { service: value.label }, + }); + await Promise.race([ + stopped, + running.completed.then((code) => { + throw running.failure ?? new Error(`Required service "${value.label}" exited unexpectedly with code ${code}.`); + }), + ]); + } catch (error) { + failure = error; + report.reporter.emit({ kind: 'service-failed', payload: { failure: report.reporter.failure(error) } }, value.label); + } finally { + clearInterval(interval); + process.off('SIGINT', stop); + process.off('SIGTERM', stop); + try { + await runner.close(); + } catch (error) { + report.reporter.fail(error, 'cleanup'); + failure = failure === undefined ? error : new AggregateError([failure, error], 'Service execution and cleanup failed.'); + } + process.exitCode = await report.close(failure); + } +} diff --git a/packages/agentic/desktop-test/src/reporting/collector.ts b/packages/agentic/desktop-test/src/reporting/collector.ts new file mode 100644 index 0000000000..dfaa761417 --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/collector.ts @@ -0,0 +1,544 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { Writable } from 'node:stream'; +import { createReporter } from '@rnx-kit/reporter'; + +import { createEventEmitter, DesktopReportedError, sanitizeDesktopReport, serializeReportError, validateDesktopEvent } from './protocol.js'; +import type { + CollectedDesktopEvent, + DesktopEvent, + EventContext, + EventEmitter, + FailureKind, + ReportError, + ReportFailure, +} from './protocol.js'; +import { renderJUnit, renderRunSummary, summarizeDesktopRun } from './results.js'; +import type { DesktopRunResult, DesktopTestResult, DesktopWorkResult } from './results.js'; +import type { DesktopReportingConfig } from '../config.js'; + +export type ReportOutput = Pick; +export type DesktopRunReporterOptions = DesktopReportingConfig & { + artifactsRoot: string; + context?: EventContext; + kind?: 'tests' | 'lifecycle'; + optional?: boolean; + output?: ReportOutput; + errorOutput?: ReportOutput; + structured?: boolean; + verbose?: boolean; + secrets?: readonly string[]; + flushTimeoutMs?: number; + maxJournalBytes?: number; +}; +export type DesktopReportWriter = Pick< + DesktopRunReporter, + 'runId' | 'root' | 'emit' | 'source' | 'addSecret' | 'redact' | 'failure' | 'fail' | 'artifactPath' | 'flush' +> & { + assertSuccessful(workIds?: readonly string[], optional?: boolean): void | Promise; +}; + +export class DesktopRunReporter { + readonly runId: string = randomUUID(); + readonly root: string; + readonly emit: EventEmitter; + private readonly startedAt = new Date(); + private readonly sources = new Map(); + private readonly scopes = new Map< + string, + { label: string; category: string; finished: boolean; context?: EventContext; workId?: string } + >(); + private readonly work = new Map(); + private readonly tests = new Map(); + private readonly testWork = new Map(); + private readonly failures = new Map(); + private readonly failureIds = new WeakMap(); + private readonly annotations = new Set(); + private readonly presentedFailures = new Set(); + private readonly secrets = new Set(); + private readonly artifacts = new Map(); + private readonly options: DesktopRunReporterOptions; + private readonly actions: boolean; + private readonly output: ReportOutput; + private readonly errors: ReportOutput; + private readonly logger: ReturnType; + private queue = Promise.resolve(); + private arrivalSequence = 0; + private journalBytes = 0; + private pendingBytes = 0; + private finishing?: Promise; + private finished = false; + private journalFailed = false; + private presentationFailed = false; + private terminal?: DesktopRunResult; + + constructor(options: DesktopRunReporterOptions) { + this.options = options; + this.root = path.join(path.resolve(options.artifactsRoot), this.runId); + fs.mkdirSync(this.root, { recursive: true, mode: 0o700 }); + this.actions = + !options.structured && + (options.console === 'github-actions' || ((!options.console || options.console === 'auto') && process.env.GITHUB_ACTIONS === 'true')); + this.output = options.structured ? (options.errorOutput ?? process.stderr) : (options.output ?? process.stdout); + this.errors = options.errorOutput ?? process.stderr; + for (const secret of options.secrets ?? []) this.addSecret(secret); + this.emit = createEventEmitter(this.runId, 'supervisor', (event) => this.accept(event)); + this.logger = createReporter({ + name: 'desktop-test', + prefix: {}, + output: { + log: (message) => this.enqueue(() => this.writeRaw(this.output, message + '\n'), Buffer.byteLength(message)), + warn: (message) => this.enqueue(() => this.writeRaw(this.errors, message + '\n'), Buffer.byteLength(message)), + error: (message) => this.enqueue(() => this.writeRaw(this.errors, message + '\n'), Buffer.byteLength(message)), + }, + }); + fs.writeFileSync(path.join(this.root, 'events.jsonl'), '', { flag: 'wx', mode: 0o600 }); + this.artifacts.set('events.jsonl', 'Versioned event journal'); + this.artifacts.set('run.json', 'Canonical run result'); + if (options.junit !== false) this.artifacts.set('junit.xml', 'JUnit result'); + } + + source(sourceId: string): EventEmitter { + if (sourceId === 'supervisor' || this.sources.has(sourceId)) throw new Error(`Report source "${sourceId}" is already registered.`); + this.sources.set(sourceId, { sequence: 0, finished: false }); + return createEventEmitter(this.runId, sourceId, (event) => this.accept(event)); + } + + addSecret(secret: string): void { + if (secret.length > 0) this.secrets.add(secret); + } + + redact(text: string): string { + let result = text; + for (const secret of [...this.secrets].sort((a, b) => b.length - a.length)) result = result.replaceAll(secret, '[REDACTED]'); + return result.replace( + /((?:leaseNonce|bridgeNonce|authorization|password|access[_-]?token)\s*["']?\s*[:=]\s*["']?)[^\s"',;]+/gi, + '$1[REDACTED]', + ); + } + + failure(error: unknown, kind: FailureKind = 'infrastructure'): ReportFailure { + if (error instanceof DesktopReportedError) return this.sanitize(error.failure); + let failureId = typeof error === 'object' && error !== null ? this.failureIds.get(error) : undefined; + if (!failureId && error instanceof Error && error.cause instanceof DesktopReportedError) return this.sanitize(error.cause.failure); + if (!failureId && error instanceof Error && error.cause && typeof error.cause === 'object') + failureId = this.failureIds.get(error.cause); + if (!failureId) { + failureId = randomUUID(); + if (typeof error === 'object' && error !== null) this.failureIds.set(error, failureId); + } + const recorded = this.failures.get(failureId); + if (recorded) return recorded; + return this.sanitize({ failureId, kind, error: serializeReportError(error) }); + } + + fail(error: unknown, kind: FailureKind = 'infrastructure', scopeId = 'run'): ReportFailure { + if (error instanceof AggregateError && error.errors.length) { + const children: unknown[] = error.errors; + return children.map((child) => this.fail(child, kind, scopeId))[0]; + } + if (error instanceof Error && error.cause instanceof AggregateError) return this.fail(error.cause, kind, scopeId); + const failure = this.failure(error, kind); + if (this.failures.has(failure.failureId)) return failure; + this.failures.set(failure.failureId, failure); + try { + this.emit( + { kind: 'diagnostic', payload: { level: 'error', message: failure.error.message || failure.error.name, failure } }, + scopeId, + ); + } catch (reportingError) { + const reportingFailure = this.failure(reportingError, 'reporting'); + this.failures.set(reportingFailure.failureId, reportingFailure); + this.logger.error( + `Desktop reporting failed: ${reportingFailure.error.message}. Original failure: ${failure.error.message}. Reports: ${this.root}`, + ); + } + return failure; + } + + assertSuccessful(workIds?: readonly string[], optional = false): void { + let result = this.result(); + if (workIds) { + if (!workIds.length || workIds.some((id) => !this.work.has(id))) throw new Error('Cannot verify unknown or empty selected work.'); + const selected = new Set(workIds); + result = summarizeDesktopRun({ + ...result, + kind: 'tests', + optional, + tests: result.tests.filter(({ testId }) => selected.has(this.testWork.get(testId) ?? '')), + work: result.work.filter(({ workId }) => selected.has(workId)), + }); + } + if (result.status === 'passed' || result.status === 'skipped') return; + const failure = + result.failures.find(({ kind }) => kind !== 'test') ?? + result.tests.find(({ status }) => status === 'failed')?.attempts.at(-1)?.failure; + if (failure) throw new DesktopReportedError(failure); + throw new Error(`Desktop run is ${result.status}; selected work did not produce complete executable results.`); + } + + artifactPath(relative: string): string { + return desktopArtifactPath(this.root, relative); + } + + async flush(): Promise { + await this.queue; + } + + accept(value: unknown): void { + if (this.finished) throw new Error('Cannot report after run finalization.'); + validateDesktopEvent(value); + if (value.runId !== this.runId) throw new Error('Report producer belongs to another run.'); + const source = this.sources.get(value.sourceId) ?? { sequence: 0, finished: false }; + if (value.sourceSequence <= source.sequence) return; + if (source.finished) throw new Error(`Producer ${value.sourceId} emitted after its terminal event.`); + if (value.sourceSequence !== source.sequence + 1) + throw new Error( + `Missing reporting events from ${value.sourceId}: expected ${source.sequence + 1}, received ${value.sourceSequence}.`, + ); + source.sequence = value.sourceSequence; + source.finished = value.kind === 'producer-finished'; + this.sources.set(value.sourceId, source); + const event: CollectedDesktopEvent = { ...this.sanitize(value), arrivalSequence: ++this.arrivalSequence }; + if (event.kind === 'artifact') this.artifactPath(event.payload.path); + if (event.kind === 'scope-end' && event.payload.logPath) this.artifactPath(event.payload.logPath); + const line = JSON.stringify(event) + '\n'; + if (!this.journalFailed) { + try { + this.journalBytes += Buffer.byteLength(line); + if (this.journalBytes > (this.options.maxJournalBytes ?? 64 * 1024 * 1024)) + throw new Error('Desktop event journal exceeded its size limit; run is incomplete.'); + fs.appendFileSync(path.join(this.root, 'events.jsonl'), line, { mode: 0o600 }); + } catch (error) { + this.journalFailed = true; + const failure = this.failure(error, 'reporting'); + this.failures.set(failure.failureId, failure); + throw error; + } + } + this.reduce(event); + this.present(event); + } + + private reduce(event: CollectedDesktopEvent): void { + if ('failure' in event.payload && event.payload.failure) this.failures.set(event.payload.failure.failureId, event.payload.failure); + if (event.kind === 'work-selected') { + if (this.work.has(event.payload.workId)) throw new Error(`Work ${event.payload.workId} was selected twice.`); + this.work.set(event.payload.workId, { ...event.payload, status: 'not-run', context: event.context }); + } else if (event.kind === 'scope-start') { + if (this.scopes.has(event.scopeId)) throw new Error(`Scope ${event.scopeId} was started twice.`); + const work = event.parentScopeId ? this.work.get(event.parentScopeId) : undefined; + if (work) work.status = 'incomplete'; + this.scopes.set(event.scopeId, { ...event.payload, finished: false, context: event.context, workId: work?.workId }); + } else if (event.kind === 'scope-end') { + const scope = this.scopes.get(event.scopeId); + if (!scope || scope.finished) throw new Error(`Scope ${event.scopeId} has no outstanding start.`); + scope.finished = true; + if (scope.workId) this.work.get(scope.workId)!.status = event.payload.status; + } else if (event.kind === 'test-start' || event.kind === 'test-result') { + const p = event.payload; + const workId = this.scopes.get(event.scopeId)?.workId ?? event.parentScopeId; + if (workId && this.work.has(workId)) this.testWork.set(p.testId, workId); + const test = this.tests.get(p.testId) ?? { testId: p.testId, title: p.title, status: 'incomplete', flaky: false, attempts: [] }; + const index = test.attempts.findIndex(({ attempt }) => attempt === p.attempt); + const result = + event.kind === 'test-result' + ? { ...event.payload, context: event.context } + : { + attempt: p.attempt, + status: 'incomplete' as const, + durationMs: 0, + reason: 'Worker did not report a terminal result.', + context: event.context, + }; + if (index < 0) test.attempts.push(result); + else { + if (test.attempts[index].status !== 'incomplete') + throw new Error(`Duplicate terminal result for ${p.testId}, attempt ${p.attempt}.`); + test.attempts[index] = result; + } + test.attempts.sort((a, b) => a.attempt - b.attempt); + test.status = test.attempts[test.attempts.length - 1].status; + test.flaky = test.status === 'passed' && test.attempts.some(({ status }) => status === 'failed'); + this.tests.set(p.testId, test); + } else if (event.kind === 'artifact') this.artifacts.set(event.payload.path, event.payload.description); + } + + private present(event: DesktopEvent): void { + const prefix = `[${event.sourceId}${event.context?.endpoint ? ` ${event.context.endpoint}` : ''}]`; + const failure = 'failure' in event.payload ? event.payload.failure : undefined; + const firstFailure = !failure || !this.presentedFailures.has(failure.failureId); + if (failure) this.presentedFailures.add(failure.failureId); + switch (event.kind) { + case 'scope-start': + this.logger.log(`${prefix} START ${event.payload.label}`); + break; + case 'scope-end': { + const scope = this.scopes.get(event.scopeId)!; + if (event.payload.logPath && (event.payload.status === 'failed' || this.options.verbose || scope.category === 'worker')) { + this.enqueue(() => this.replay(scope.label, event.payload.logPath!, event.payload.status === 'failed'), 0); + } + this.logger.log(`${prefix} ${event.payload.status.toUpperCase()} ${scope.label}`); + break; + } + case 'service-ready': + this.logger.log(`${prefix} READY ${event.payload.message}`); + break; + case 'output': + if (event.payload.stream === 'stderr' || this.options.verbose) + this.logger.warn(`${prefix} ${event.payload.stream}: ${event.payload.text.replace(/\n$/, '')}`); + break; + case 'diagnostic': + if (!firstFailure) break; + if (event.payload.level === 'info') this.logger.log(`${prefix} ${event.payload.message}`); + else this.logger.warn(`${prefix} ${event.payload.level}: ${event.payload.message}`); + break; + case 'service-failed': + if (firstFailure) this.logger.error(`${prefix} FAILED ${event.payload.failure.error.message}`); + break; + case 'test-start': + this.logger.log(`${prefix} TEST ${event.payload.title} (attempt ${event.payload.attempt + 1})`); + break; + case 'test-result': + this.logger.log( + `${prefix} ${event.payload.status.toUpperCase()} ${event.payload.title}${event.payload.reason ? `: ${event.payload.reason}` : ''}`, + ); + if (failure && firstFailure) this.logger.error(`${prefix} ${errorDiagnostic(failure.error)}`); + break; + case 'artifact': + this.logger.log(`${prefix} artifact: ${path.join(this.root, event.payload.path)}`); + break; + } + if ('failure' in event.payload && event.payload.failure) { + const failure = event.payload.failure; + if (this.actions && !this.annotations.has(failure.failureId)) { + this.annotations.add(failure.failureId); + this.enqueue( + () => + this.write( + this.errors, + `::error title=${escapeProperty(`${event.sourceId}: ${failure.kind}`)}::${escapeData(errorDiagnostic(failure.error))}\n`, + ), + 0, + ); + } + } + } + + private sanitize(value: T): T { + return sanitizeDesktopReport(value, (text) => this.redact(text)); + } + + private enqueue(action: () => Promise, bytes: number): void { + if (this.presentationFailed) return; + this.pendingBytes += bytes; + if (this.pendingBytes > 4 * 1024 * 1024) { + this.presentationFailed = true; + const failure = this.failure(new Error('Presentation queue exceeded 4 MiB; retained journal is authoritative.'), 'reporting'); + this.failures.set(failure.failureId, failure); + return; + } + this.queue = this.queue + .then(() => (this.presentationFailed ? undefined : action())) + .catch((error: unknown) => { + this.presentationFailed = true; + const failure = this.failure(error, 'reporting'); + this.failures.set(failure.failureId, failure); + }) + .finally(() => { + this.pendingBytes -= bytes; + }); + } + + private async write(output: ReportOutput, text: string): Promise { + if (!(output instanceof Writable)) { + output.write(text); + return; + } + await new Promise((resolve, reject) => { + const timer = setTimeout(() => settle(new Error('Timed out flushing desktop report output.')), this.options.flushTimeoutMs ?? 5000); + const onError = (error: Error) => settle(error); + const settle = (error?: Error | null) => { + clearTimeout(timer); + output.off('error', onError); + if (error) reject(error); + else resolve(); + }; + output.once('error', onError); + output.write(text, settle); + }); + } + + private async writeRaw(output: ReportOutput, text: string): Promise { + if (!this.actions) { + await this.write(output, text); + return; + } + const token = randomUUID(); + await this.write(output, `::stop-commands::${token}\n`); + try { + await this.write(output, text.endsWith('\n') ? text : text + '\n'); + } finally { + await this.write(output, `::${token}::\n`); + } + } + + private async replay(label: string, relative: string, failed: boolean): Promise { + const output = failed ? this.errors : this.output; + await this.write(output, this.actions ? `::group::${escapeData(label)}\n` : `[desktop] BEGIN ${label}\n`); + const token = randomUUID(); + try { + if (this.actions) await this.write(output, `::stop-commands::${token}\n`); + try { + for await (const chunk of fs.createReadStream(this.artifactPath(relative), { encoding: 'utf8' })) + await this.write(output, this.redact(chunk)); + await this.write(output, '\n'); + } finally { + if (this.actions) await this.write(output, `::${token}::\n`); + } + } finally { + await this.write(output, this.actions ? '::endgroup::\n' : `[desktop] END ${label}\n`); + } + } + + finish(): Promise { + this.finishing ??= this.finalize(); + return this.finishing; + } + + private result(): DesktopRunResult { + return summarizeDesktopRun({ + schemaVersion: 1, + runId: this.runId, + kind: this.options.kind ?? 'tests', + startedAt: this.startedAt.toISOString(), + finishedAt: new Date().toISOString(), + durationMs: Date.now() - this.startedAt.getTime(), + context: this.options.context, + optional: this.options.optional, + incomplete: + [...this.scopes.values()].some(({ finished }) => !finished) || + [...this.sources.entries()].some(([id, source]) => id !== 'supervisor' && !source.finished), + tests: [...this.tests.values()], + work: [...this.work.values()], + failures: [...this.failures.values()], + artifacts: [...this.artifacts].map(([path, description]) => ({ path, description })), + }); + } + + private persist(result: DesktopRunResult): void { + const write = (name: string, data: string) => { + const temporary = this.artifactPath(`${name}.tmp`); + fs.writeFileSync(temporary, data, { mode: 0o600 }); + fs.renameSync(temporary, this.artifactPath(name)); + }; + if (this.options.junit !== false) write('junit.xml', renderJUnit(result)); + write('run.json', JSON.stringify(result, null, 2) + '\n'); + } + + private async finalize(): Promise { + await this.queue; + const persist = () => { + let result = this.result(); + try { + this.persist(result); + } catch (error) { + const failure = this.failure(error, 'reporting'); + this.failures.set(failure.failureId, failure); + result = this.result(); + try { + this.persist(result); + } catch (retryError) { + throw new AggregateError( + [error, retryError, ...result.failures.map((failure) => new DesktopReportedError(failure))], + `Cannot persist required desktop reports under ${this.root}.`, + ); + } + } + return result; + }; + let result = persist(); + const c = result.counts; + this.logger.log( + `[desktop] ${result.status.toUpperCase()}: ${c.selected} selected, ${c.executed} executed, ${c.passed} passed, ${c.failed} failed, ${c.skipped} skipped, ${c.unsupported} unsupported, ${c.incomplete} incomplete, ${c.retries} retries, ${c.flaky} flaky; ${result.work.filter(({ status }) => status === 'not-run').length} groups not run; ${path.join(this.root, 'run.json')}`, + ); + await this.queue; + result = persist(); + if (this.presentationFailed) { + try { + await this.writeRaw( + this.errors, + `[desktop] FAILED: presentation could not be flushed; authoritative result: ${path.join(this.root, 'run.json')}\n`, + ); + } catch (error) { + const failure = this.failure(error, 'reporting'); + this.failures.set(failure.failureId, failure); + result = persist(); + } + } + if (this.options.summary !== false && process.env.GITHUB_STEP_SUMMARY && !this.options.structured) { + try { + const summary = renderRunSummary(result).slice(0, 64 * 1024); + const size = fs.existsSync(process.env.GITHUB_STEP_SUMMARY) ? fs.statSync(process.env.GITHUB_STEP_SUMMARY).size : 0; + if (size + Buffer.byteLength(summary) > 1024 * 1024) throw new Error('GitHub step summary would exceed 1 MiB.'); + fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary); + } catch (error) { + this.emit({ + kind: 'diagnostic', + payload: { + level: 'warning', + message: `Optional Actions summary failed: ${String(error)}; result retained at ${path.join(this.root, 'run.json')}.`, + }, + }); + } + } + await this.queue; + result = persist(); + try { + this.emit({ kind: 'run-finished', payload: { status: result.status, resultPath: 'run.json' } }); + } catch (error) { + const failure = this.failure(error, 'reporting'); + this.failures.set(failure.failureId, failure); + result = persist(); + } + this.finished = true; + this.terminal = result; + return result; + } + + get resultSnapshot(): DesktopRunResult | undefined { + return this.terminal; + } +} + +function errorDiagnostic(error: ReportError): string { + const message = `${error.name}: ${error.message}`; + return error.stack ? (error.stack.includes(error.message) ? error.stack : `${message}\n${error.stack}`) : message; +} +function escapeData(text: string): string { + return text.replaceAll('%', '%25').replaceAll('\r', '%0D').replaceAll('\n', '%0A'); +} +function escapeProperty(text: string): string { + return escapeData(text).replaceAll(':', '%3A').replaceAll(',', '%2C'); +} + +export function desktopArtifactPath(directory: string, relative: string): string { + const resolved = path.resolve(directory, relative); + if ( + resolved === directory || + path.relative(directory, resolved).startsWith('..') || + path.isAbsolute(path.relative(directory, resolved)) + ) { + throw new Error(`Artifact must be confined beneath the run root: ${relative}`); + } + let existing = path.dirname(resolved); + while (!fs.existsSync(existing)) existing = path.dirname(existing); + const actual = fs.realpathSync(existing); + const root = fs.realpathSync(directory); + if (actual !== root && (path.relative(root, actual).startsWith('..') || path.isAbsolute(path.relative(root, actual)))) { + throw new Error('Artifact directory escapes the run root through a symlink.'); + } + if (fs.existsSync(resolved) && fs.lstatSync(resolved).isSymbolicLink()) throw new Error('Artifact destination is a symlink.'); + return resolved; +} diff --git a/packages/agentic/desktop-test/src/reporting/index.ts b/packages/agentic/desktop-test/src/reporting/index.ts new file mode 100644 index 0000000000..d820061327 --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/index.ts @@ -0,0 +1,19 @@ +export { DesktopRunReporter } from './collector.js'; +export type { DesktopReportWriter, DesktopRunReporterOptions, ReportOutput } from './collector.js'; +export { createDesktopReportSession } from './session.js'; +export type { DesktopReportSession } from './session.js'; +export { createEventEmitter, DesktopReportedError, serializeReportError, validateDesktopEvent } from './protocol.js'; +export type { + DesktopEvent, + EventContext, + EventData, + EventEmitter, + FailureKind, + ReportError, + ReportFailure, + TestStatus, +} from './protocol.js'; +export { desktopRunExitCode, renderJUnit, renderRunSummary, summarizeDesktopRun } from './results.js'; +export type { DesktopRunResult, DesktopTestAttempt, DesktopTestResult } from './results.js'; +export { connectReportProducer, DESKTOP_REPORT_CONTEXT, startReportTransport } from './transport.js'; +export type { ReportProducer, ReportTransport, ReportTransportContext } from './transport.js'; diff --git a/packages/agentic/desktop-test/src/reporting/protocol.ts b/packages/agentic/desktop-test/src/reporting/protocol.ts new file mode 100644 index 0000000000..51285abf1b --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/protocol.ts @@ -0,0 +1,255 @@ +import { randomUUID } from 'node:crypto'; + +export type FailureKind = 'test' | 'infrastructure' | 'cleanup' | 'evidence' | 'reporting'; +export type ReportError = { name: string; message: string; stack?: string; cause?: ReportError; errors?: ReportError[] }; +export type ReportFailure = { failureId: string; kind: FailureKind; error: ReportError }; +export class DesktopReportedError extends Error { + readonly failure: ReportFailure; + + constructor(failure: unknown) { + validateFailure(failure); + super(failure.error.message); + this.name = failure.error.name; + this.stack = failure.error.stack; + this.failure = failure; + } +} +export type TestStatus = 'passed' | 'failed' | 'skipped' | 'unsupported' | 'cancelled' | 'not-run' | 'incomplete'; +export type EventContext = { + endpoint?: 'macos' | 'windows' | 'win32'; + renderer?: 'fabric' | 'paper'; + service?: string; + worker?: string; + spec?: string; + story?: string; + test?: string; + attempt?: number; +}; +export type EventData = + | { kind: 'work-selected'; payload: { workId: string; label: string } } + | { kind: 'scope-start'; payload: { label: string; category: 'service' | 'worker' | 'phase' } } + | { kind: 'scope-end'; payload: { status: 'passed' | 'failed' | 'cancelled' | 'skipped' | 'unsupported'; logPath?: string } } + | { kind: 'service-ready'; payload: { message: string } } + | { kind: 'service-failed'; payload: { failure: ReportFailure } } + | { kind: 'diagnostic'; payload: { level: 'info' | 'warning' | 'error'; message: string; failure?: ReportFailure } } + | { kind: 'output'; payload: { stream: 'stdout' | 'stderr'; text: string } } + | { kind: 'test-start'; payload: { testId: string; title: string; attempt: number } } + | { + kind: 'test-result'; + payload: { + testId: string; + title: string; + attempt: number; + status: TestStatus; + durationMs: number; + reason?: string; + failure?: ReportFailure; + }; + } + | { kind: 'artifact'; payload: { path: string; description: string } } + | { kind: 'producer-finished'; payload: { status: 'passed' | 'failed' } } + | { kind: 'run-finished'; payload: { status: TestStatus; resultPath: string } }; + +export type DesktopEvent = { + schemaVersion: 1; + runId: string; + eventId: string; + sourceId: string; + sourceSequence: number; + scopeId: string; + parentScopeId?: string; + timestamp: string; + context?: EventContext; +} & EventData; + +export type CollectedDesktopEvent = DesktopEvent & { arrivalSequence: number }; +export type EventEmitter = (event: EventData, scopeId?: string, context?: EventContext, parentScopeId?: string) => void; +export const reportStatuses: readonly TestStatus[] = ['passed', 'failed', 'skipped', 'unsupported', 'cancelled', 'not-run', 'incomplete']; +const payloadFields = { + 'work-selected': ['workId', 'label'], + 'scope-start': ['label', 'category'], + 'scope-end': ['status', 'logPath'], + 'service-ready': ['message'], + 'service-failed': ['failure'], + diagnostic: ['level', 'message', 'failure'], + output: ['stream', 'text'], + 'test-start': ['testId', 'title', 'attempt'], + 'test-result': ['testId', 'title', 'attempt', 'status', 'durationMs', 'reason', 'failure'], + artifact: ['path', 'description'], + 'producer-finished': ['status'], + 'run-finished': ['status', 'resultPath'], +} as const; + +export function createEventEmitter(runId: string, sourceId: string, receive: (event: DesktopEvent) => void): EventEmitter { + let sequence = 0; + return (event, scopeId = sourceId, context, parentScopeId) => + receive({ + ...event, + schemaVersion: 1, + runId, + eventId: randomUUID(), + sourceId, + sourceSequence: ++sequence, + scopeId, + parentScopeId, + timestamp: new Date().toISOString(), + context, + }); +} + +export function validateDesktopEvent(value: unknown): asserts value is DesktopEvent { + const event = record(value, 'report event'); + exactKeys(event, [ + 'schemaVersion', + 'runId', + 'eventId', + 'sourceId', + 'sourceSequence', + 'scopeId', + 'parentScopeId', + 'timestamp', + 'context', + 'kind', + 'payload', + ]); + if (event.schemaVersion !== 1) throw new TypeError('Unsupported desktop reporting schemaVersion.'); + for (const key of ['runId', 'eventId', 'sourceId', 'scopeId', 'timestamp']) requireText(event[key], key); + if (event.parentScopeId !== undefined) requireText(event.parentScopeId, 'parentScopeId'); + if (!Number.isFinite(Date.parse(String(event.timestamp)))) throw new TypeError('Invalid reporting timestamp.'); + integer(event.sourceSequence, 'sourceSequence', 1); + if (typeof event.kind !== 'string' || !Object.hasOwn(payloadFields, event.kind)) + throw new TypeError('Unknown desktop report event kind.'); + const kind = event.kind as keyof typeof payloadFields; + const payload = record(event.payload, 'event payload'); + exactKeys(payload, payloadFields[kind]); + if (event.context !== undefined) { + const context = record(event.context, 'event context'); + exactKeys(context, ['endpoint', 'renderer', 'service', 'worker', 'spec', 'story', 'test', 'attempt']); + for (const [key, value] of Object.entries(context)) { + if (value === undefined) continue; + if (key === 'attempt') integer(value, key, 0); + else requireText(value, key); + } + if (context.endpoint !== undefined) member(context.endpoint, ['macos', 'windows', 'win32']); + if (context.renderer !== undefined) member(context.renderer, ['fabric', 'paper']); + } + switch (kind) { + case 'work-selected': + requireText(payload.workId, 'workId'); + requireText(payload.label, 'label'); + break; + case 'scope-start': + requireText(payload.label, 'label'); + member(payload.category, ['service', 'worker', 'phase']); + break; + case 'scope-end': + if (payload.logPath !== undefined) requireText(payload.logPath, 'logPath'); + member(payload.status, ['passed', 'failed', 'cancelled', 'skipped', 'unsupported']); + break; + case 'producer-finished': + member(payload.status, ['passed', 'failed']); + break; + case 'service-ready': + requireText(payload.message, 'message'); + break; + case 'service-failed': + validateFailure(payload.failure); + break; + case 'diagnostic': + member(payload.level, ['info', 'warning', 'error']); + requireText(payload.message, 'message'); + if (payload.failure !== undefined) validateFailure(payload.failure); + break; + case 'output': + member(payload.stream, ['stdout', 'stderr']); + if (typeof payload.text !== 'string') throw new TypeError('output.text must be a string.'); + break; + case 'test-start': + case 'test-result': + requireText(payload.testId, 'testId'); + requireText(payload.title, 'title'); + integer(payload.attempt, 'attempt', 0); + if (kind === 'test-result') { + member(payload.status, reportStatuses); + if (typeof payload.durationMs !== 'number' || !Number.isFinite(payload.durationMs) || payload.durationMs < 0) + throw new TypeError('Invalid test duration.'); + if (payload.reason !== undefined) requireText(payload.reason, 'reason'); + if (['skipped', 'unsupported', 'cancelled', 'not-run', 'incomplete'].includes(String(payload.status)) && !payload.reason) { + throw new TypeError('Non-executed test results require a reason.'); + } + if (payload.status === 'failed' && !payload.failure) throw new TypeError('Failed test results require their original failure.'); + if (payload.failure !== undefined) validateFailure(payload.failure); + } + break; + case 'artifact': + requireText(payload.path, 'artifact path'); + requireText(payload.description, 'artifact description'); + break; + case 'run-finished': + member(payload.status, reportStatuses); + requireText(payload.resultPath, 'resultPath'); + break; + } +} + +export function serializeReportError(error: unknown, seen = new Set()): ReportError { + if (seen.has(error)) return { name: 'Error', message: '[circular error]' }; + seen.add(error); + if (!error || typeof error !== 'object') return { name: 'Error', message: String(error) }; + return { + name: 'name' in error && typeof error.name === 'string' && error.name ? error.name : 'Error', + message: 'message' in error && typeof error.message === 'string' ? error.message : String(error), + ...('stack' in error && typeof error.stack === 'string' ? { stack: error.stack } : {}), + ...('cause' in error && error.cause !== undefined ? { cause: serializeReportError(error.cause, seen) } : {}), + ...('errors' in error && Array.isArray(error.errors) + ? { errors: error.errors.map((child: unknown) => serializeReportError(child, seen)) } + : {}), + }; +} + +export function sanitizeDesktopReport(value: T, redact: (text: string) => string): T { + return JSON.parse( + JSON.stringify(value, (key, item: unknown) => { + if (/nonce|password|secret|authorization|access[_-]?token|cookie/i.test(key)) return '[REDACTED]'; + return typeof item === 'string' ? redact(item) : item; + }), + ) as T; +} + +function validateFailure(value: unknown): asserts value is ReportFailure { + const failure = record(value, 'failure'); + exactKeys(failure, ['failureId', 'kind', 'error']); + requireText(failure.failureId, 'failureId'); + member(failure.kind, ['test', 'infrastructure', 'cleanup', 'evidence', 'reporting']); + const checkError = (value: unknown, depth: number): void => { + if (depth > 20) throw new TypeError('Report error cause depth exceeds 20.'); + const error = record(value, 'error'); + exactKeys(error, ['name', 'message', 'stack', 'cause', 'errors']); + requireText(error.name, 'error.name'); + if (typeof error.message !== 'string') throw new TypeError('error.message must be text.'); + if (error.stack !== undefined && typeof error.stack !== 'string') throw new TypeError('error.stack must be text.'); + if (error.cause !== undefined) checkError(error.cause, depth + 1); + if (error.errors !== undefined) { + if (!Array.isArray(error.errors)) throw new TypeError('error.errors must be an array.'); + for (const child of error.errors) checkError(child, depth + 1); + } + }; + checkError(failure.error, 0); +} + +function record(value: unknown, label: string): Record { + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError(`${label} must be an object.`); + return value as Record; +} +function requireText(value: unknown, name: string): void { + if (typeof value !== 'string' || !value.length) throw new TypeError(`${name} must be nonempty text.`); +} +function integer(value: unknown, name: string, minimum: number): void { + if (typeof value !== 'number' || !Number.isSafeInteger(value) || value < minimum) throw new TypeError(`Invalid ${name}.`); +} +function member(value: unknown, values: readonly string[]): void { + if (typeof value !== 'string' || !values.includes(value)) throw new TypeError(`Expected one of: ${values.join(', ')}.`); +} +function exactKeys(value: Record, allowed: readonly string[]): void { + for (const key of Object.keys(value)) if (!allowed.includes(key)) throw new TypeError(`Unknown reporting field "${key}".`); +} diff --git a/packages/agentic/desktop-test/src/reporting/results.ts b/packages/agentic/desktop-test/src/reporting/results.ts new file mode 100644 index 0000000000..78af092707 --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/results.ts @@ -0,0 +1,166 @@ +import type { EventContext, ReportFailure, TestStatus } from './protocol.js'; + +export type DesktopTestAttempt = { + attempt: number; + status: TestStatus; + durationMs: number; + reason?: string; + failure?: ReportFailure; + context?: EventContext; +}; +export type DesktopTestResult = { + testId: string; + title: string; + status: TestStatus; + flaky: boolean; + attempts: DesktopTestAttempt[]; +}; +export type DesktopWorkResult = { + workId: string; + label: string; + status: TestStatus; + context?: EventContext; +}; +export type DesktopRunResult = { + schemaVersion: 1; + runId: string; + kind: 'tests' | 'lifecycle'; + status: TestStatus; + startedAt: string; + finishedAt: string; + durationMs: number; + context?: EventContext; + counts: { + selected: number; + executed: number; + passed: number; + failed: number; + skipped: number; + unsupported: number; + incomplete: number; + retries: number; + flaky: number; + }; + tests: DesktopTestResult[]; + work: DesktopWorkResult[]; + failures: ReportFailure[]; + artifacts: { path: string; description: string }[]; +}; + +export function summarizeDesktopRun( + input: Omit & { optional?: boolean; incomplete?: boolean }, +): DesktopRunResult { + const counts = { + selected: input.tests.length, + executed: 0, + passed: 0, + failed: 0, + skipped: 0, + unsupported: 0, + incomplete: 0, + retries: 0, + flaky: 0, + }; + for (const test of input.tests) { + counts.retries += Math.max(0, test.attempts.length - 1); + if (test.flaky) counts.flaky++; + if (test.attempts.some(({ status }) => status === 'passed' || status === 'failed')) counts.executed++; + if (['cancelled', 'not-run', 'incomplete'].includes(test.status)) counts.incomplete++; + else if (test.status === 'passed') counts.passed++; + else if (test.status === 'failed') counts.failed++; + else if (test.status === 'skipped') counts.skipped++; + else if (test.status === 'unsupported') counts.unsupported++; + } + const infrastructureFailed = input.failures.some(({ kind }) => kind !== 'test') || input.work.some(({ status }) => status === 'failed'); + const unfinishedWork = input.work.some(({ status }) => ['not-run', 'incomplete', 'cancelled'].includes(status)); + const noExecution = input.kind === 'tests' && counts.executed === 0; + const status = + infrastructureFailed || counts.failed > 0 + ? 'failed' + : input.incomplete || unfinishedWork || counts.incomplete > 0 || (input.kind === 'tests' && counts.selected === 0) + ? 'incomplete' + : noExecution + ? input.optional + ? 'skipped' + : 'incomplete' + : 'passed'; + const { optional: _optional, incomplete: _incomplete, ...result } = input; + return { ...result, status, counts }; +} + +export function desktopRunExitCode(result: DesktopRunResult): number { + return result.status === 'passed' || result.status === 'skipped' ? 0 : 1; +} + +export function renderJUnit(result: DesktopRunResult): string { + const infrastructure = result.failures.filter(({ kind }) => kind !== 'test'); + const unfinishedWork = result.work.filter(({ status }) => ['not-run', 'incomplete', 'cancelled'].includes(status)); + const incomplete = result.status === 'incomplete' && result.counts.incomplete === 0; + const cases = result.tests.map((test) => { + const latest = test.attempts[test.attempts.length - 1]; + const history = JSON.stringify(test.attempts); + const body = + test.status === 'failed' + ? `${xml(JSON.stringify(latest.failure))}` + : test.status === 'passed' + ? '' + : ['skipped', 'unsupported'].includes(test.status) + ? `` + : ``; + return `${body}${xml(history)}`; + }); + for (const failure of infrastructure) { + cases.push( + `${xml(JSON.stringify(failure.error))}`, + ); + } + for (const work of unfinishedWork) + cases.push( + ``, + ); + if (incomplete) + cases.push( + '', + ); + return `\n${cases.join('')}\n`; +} + +export function renderRunSummary(result: DesktopRunResult): string { + const c = result.counts; + const reasons = [ + ...new Set(result.tests.flatMap((test) => test.attempts.flatMap((attempt) => (attempt.reason ? [attempt.reason] : [])))), + ]; + return ( + `## Desktop tests: ${result.status}\n\n` + + `Run: \`${result.runId}\`; endpoint: ${result.context?.endpoint ?? 'native'}; renderer: ${result.context?.renderer ?? 'not applicable'}.\n\n` + + `Selected ${c.selected}; executed ${c.executed}; passed ${c.passed}; failed ${c.failed}; skipped ${c.skipped}; unsupported ${c.unsupported}; incomplete ${c.incomplete}; retries ${c.retries}; flaky ${c.flaky}.\n\n` + + `Selected groups ${result.work.length}; not run ${result.work.filter(({ status }) => status === 'not-run').length}; incomplete ${result.work.filter(({ status }) => status === 'incomplete').length}.\n\n` + + `Infrastructure/evidence/cleanup failures: ${result.failures.filter(({ kind }) => kind !== 'test').length}.\n\n` + + (reasons.length + ? `Reasons:\n${reasons + .slice(0, 50) + .map((reason) => `- ${markdown(reason).slice(0, 1000)}`) + .join('\n')}\n\n` + : '') + + `Artifacts (filesystem paths, not upload URLs):\n${result.artifacts + .slice(0, 50) + .map((artifact) => `- \`${markdown(artifact.path)}\``) + .join('\n')}\n` + ); +} + +function xml(value: string): string { + return ( + value + // eslint-disable-next-line no-control-regex -- XML 1.0 permits only these control characters. + .replaceAll(/[^\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd\u{10000}-\u{10ffff}]/gu, '') + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll("'", ''') + ); +} +function markdown(value: string): string { + return value.replaceAll(/[\r\n`<>]/g, ' '); +} diff --git a/packages/agentic/desktop-test/src/reporting/session.ts b/packages/agentic/desktop-test/src/reporting/session.ts new file mode 100644 index 0000000000..a57d2711f4 --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/session.ts @@ -0,0 +1,131 @@ +import { randomUUID } from 'node:crypto'; +import { DesktopRunReporter, desktopArtifactPath } from './collector.js'; +import type { DesktopReportWriter, DesktopRunReporterOptions } from './collector.js'; +import { DesktopReportedError, sanitizeDesktopReport, serializeReportError } from './protocol.js'; +import type { EventEmitter, FailureKind, ReportFailure } from './protocol.js'; +import { assertReportedWork, connectReportProducer, DESKTOP_REPORT_CONTEXT, startReportTransport } from './transport.js'; +import { desktopRunExitCode } from './results.js'; +import type { DesktopRunResult } from './results.js'; + +export type DesktopReportSession = { + reporter: DesktopReportWriter; + delegated: boolean; + environment: Record; + readonly result?: DesktopRunResult; + onStop?(callback: () => void | Promise): void; + close(error?: unknown): Promise; +}; + +/** Delegated CLIs inherit the collector rather than opening a competing console or result directory. */ +export async function createDesktopReportSession( + options: DesktopRunReporterOptions & { onFatal?: (error: Error) => void }, +): Promise { + const encoded = process.env[DESKTOP_REPORT_CONTEXT]; + if (!encoded) { + const reporter = new DesktopRunReporter(options); + const transport = await startReportTransport(reporter, options.onFatal); + return { + reporter, + delegated: false, + environment: transport.environment, + get result() { + return reporter.resultSnapshot; + }, + async close(error) { + if (error !== undefined) reporter.fail(error); + try { + await transport.close(); + } catch (failure) { + reporter.fail(failure, 'cleanup'); + } + return desktopRunExitCode(await reporter.finish()); + }, + }; + } + let stop: (() => void | Promise) | undefined; + let stopping = false; + const producer = await connectReportProducer(`delegated-${randomUUID()}`, encoded, async () => { + stopping = true; + await stop?.(); + }); + // connectReportProducer has already validated this inherited context. + const context = JSON.parse(encoded) as { runId: string; artifactsRoot: string; nonce: string }; + const secrets = new Set([context.nonce]); + const failures = new WeakMap(); + const redact = (text: string) => { + for (const secret of [...secrets].sort((a, b) => b.length - a.length)) text = text.replaceAll(secret, '[REDACTED]'); + return text.replace( + /((?:leaseNonce|bridgeNonce|authorization|password|access[_-]?token)\s*["']?\s*[:=]\s*["']?)[^\s"',;]+/gi, + '$1[REDACTED]', + ); + }; + const protect = + (emit: EventEmitter): EventEmitter => + (event, scope, context, parent) => + emit(sanitizeDesktopReport(event, redact), scope, sanitizeDesktopReport(context ?? {}, redact), parent); + const reporter: DesktopReportWriter = { + runId: context.runId, + root: context.artifactsRoot, + emit: protect(producer.emit), + source: (id) => protect(producer.source(id)), + addSecret(secret) { + if (secret) secrets.add(secret); + }, + redact, + artifactPath(relative) { + return desktopArtifactPath(context.artifactsRoot, relative); + }, + flush: () => producer.flush(), + async assertSuccessful(workIds, optional = false) { + if (!workIds) throw new Error('Delegated result checks require their exact selected work.'); + await producer.flush(); + await assertReportedWork(workIds, optional, encoded); + }, + failure(error, kind: FailureKind = 'infrastructure') { + if (error instanceof DesktopReportedError) return sanitizeDesktopReport(error.failure, redact); + if (error instanceof Error && error.cause instanceof DesktopReportedError) return sanitizeDesktopReport(error.cause.failure, redact); + const object = error !== null && typeof error === 'object' ? error : undefined; + const prior = + (object && failures.get(object)) || + (error instanceof Error && error.cause && typeof error.cause === 'object' ? failures.get(error.cause) : undefined); + if (prior) return prior; + const failure = { + failureId: randomUUID(), + kind, + error: JSON.parse(redact(JSON.stringify(serializeReportError(error)))) as ReportFailure['error'], + }; + if (object) failures.set(object, failure); + return failure; + }, + fail(error, kind, scopeId) { + if (error instanceof AggregateError && error.errors.length) { + const children: unknown[] = error.errors; + return children.map((child) => reporter.fail(child, kind, scopeId))[0]; + } + if (error instanceof Error && error.cause instanceof AggregateError) return reporter.fail(error.cause, kind, scopeId); + const failure = reporter.failure(error, kind); + producer.emit( + { kind: 'diagnostic', payload: { level: 'error', message: failure.error.message || failure.error.name, failure } }, + scopeId, + ); + return failure; + }, + }; + return { + reporter, + delegated: true, + environment: { [DESKTOP_REPORT_CONTEXT]: encoded }, + onStop(callback) { + stop = callback; + if (stopping) + void Promise.resolve() + .then(callback) + .catch((error: unknown) => reporter.fail(error, 'cleanup')); + }, + async close(error) { + if (error !== undefined) reporter.fail(error); + await producer.finish(error === undefined ? 'passed' : 'failed'); + return error === undefined ? 0 : 1; + }, + }; +} diff --git a/packages/agentic/desktop-test/src/reporting/transport.ts b/packages/agentic/desktop-test/src/reporting/transport.ts new file mode 100644 index 0000000000..0af657a0d0 --- /dev/null +++ b/packages/agentic/desktop-test/src/reporting/transport.ts @@ -0,0 +1,363 @@ +import net from 'node:net'; +import os from 'node:os'; +import path from 'node:path'; +import { randomUUID, timingSafeEqual } from 'node:crypto'; +import { once } from 'node:events'; + +import { createEventEmitter, DesktopReportedError, validateDesktopEvent } from './protocol.js'; +import type { DesktopEvent, EventEmitter } from './protocol.js'; +import type { DesktopRunReporter } from './collector.js'; + +export const DESKTOP_REPORT_CONTEXT = 'FURN_DESKTOP_REPORT_CONTEXT'; +export const DESKTOP_REPORT_SCOPE = 'FURN_DESKTOP_REPORT_SCOPE'; +export type ReportTransportContext = { schemaVersion: 1; address: string; nonce: string; runId: string; artifactsRoot: string }; +export type ReportTransport = { environment: Record; close(): Promise }; +const maxFrame = 1024 * 1024; + +/** A dedicated local IPC channel, also usable by WDIO grandchildren without mixing JSON into stdout. */ +export async function startReportTransport(reporter: DesktopRunReporter, onFatal?: (error: Error) => void): Promise { + const nonce = randomUUID(); + reporter.addSecret(nonce); + const address = + process.platform === 'win32' + ? `\\\\.\\pipe\\furn-desktop-${reporter.runId}` + : path.join(os.tmpdir(), `furn-desktop-${reporter.runId}.sock`); + const context: ReportTransportContext = { schemaVersion: 1, address, nonce, runId: reporter.runId, artifactsRoot: reporter.root }; + const sockets = new Set(); + const controls = new Map(); + let closing = false; + const server = net.createServer((socket) => { + sockets.add(socket); + socket.setEncoding('utf8'); + let buffered = ''; + let sourceId: string | undefined; + let finished = false; + let reported = false; + const fail = (error: Error) => { + if (!reported) { + reported = true; + reporter.fail(error, 'reporting', sourceId ?? 'transport'); + onFatal?.(error); + } + socket.destroy(); + }; + socket.on('data', (chunk: string) => { + buffered += chunk; + if (Buffer.byteLength(buffered) > maxFrame) { + fail(new Error('Desktop reporting IPC frame exceeded 1 MiB.')); + return; + } + let newline: number; + while ((newline = buffered.indexOf('\n')) >= 0) { + const line = buffered.slice(0, newline); + buffered = buffered.slice(newline + 1); + try { + const frame: unknown = JSON.parse(line); + if (!sourceId) { + if (isContext(frame) && 'type' in frame && frame.type === 'assert') { + if (frame.runId !== context.runId || !sameSecret(frame.nonce, nonce)) throw new Error('Invalid desktop result request.'); + if ( + !('workIds' in frame) || + !Array.isArray(frame.workIds) || + !frame.workIds.every((id: unknown) => typeof id === 'string') || + !('optional' in frame) || + typeof frame.optional !== 'boolean' + ) + throw new Error('Invalid selected-work result request.'); + try { + reporter.assertSuccessful(frame.workIds, frame.optional); + socket.end('{"passed":true}\n'); + } catch (error) { + socket.end(JSON.stringify({ failure: reporter.failure(error) }) + '\n'); + } + return; + } + if (isContext(frame) && 'type' in frame && frame.type === 'stop' && 'scopeId' in frame && typeof frame.scopeId === 'string') { + if (frame.runId !== context.runId || !sameSecret(frame.nonce, nonce)) throw new Error('Invalid desktop stop request.'); + if ( + !('timeoutMs' in frame) || + typeof frame.timeoutMs !== 'number' || + !Number.isSafeInteger(frame.timeoutMs) || + frame.timeoutMs <= 0 || + frame.timeoutMs > 120_000 + ) + throw new Error('Invalid desktop stop deadline.'); + const children = [...controls].filter(([child, owner]) => owner === frame.scopeId && !child.destroyed); + const terminal = children.map(([child]) => new Promise((resolve) => child.once('close', resolve))); + const timer = setTimeout(() => { + if (!socket.destroyed && !socket.writableEnded) + socket.end( + JSON.stringify({ + failure: reporter.failure(new Error('Owned reporting producers did not finish their shutdown handshake.'), 'cleanup'), + }) + '\n', + ); + }, frame.timeoutMs); + socket.once('close', () => clearTimeout(timer)); + void Promise.all(terminal).then(() => { + clearTimeout(timer); + if (!socket.destroyed && !socket.writableEnded) socket.end(JSON.stringify({ controlled: children.length }) + '\n'); + }); + for (const [child] of children) child.write('{"stop":true}\n'); + return; + } + if (!isHello(frame) || frame.runId !== context.runId || !sameSecret(frame.nonce, nonce)) + throw new Error('Invalid desktop reporting IPC handshake.'); + sourceId = frame.sourceId; + if (frame.canStop === true && typeof frame.ownerScopeId === 'string') controls.set(socket, frame.ownerScopeId); + reporter.source(sourceId); + socket.write('{"ready":true}\n'); + } else { + validateDesktopEvent(frame); + if (frame.sourceId !== sourceId && !frame.sourceId.startsWith(`${sourceId}/`)) + throw new Error('Reporting source changed after its handshake.'); + reporter.accept(frame); + if (frame.sourceId === sourceId) finished = frame.kind === 'producer-finished'; + socket.write(JSON.stringify({ acknowledged: frame.eventId }) + '\n'); + if (frame.kind === 'service-failed') onFatal?.(new DesktopReportedError(frame.payload.failure)); + } + } catch (error) { + fail(error instanceof Error ? error : new Error(String(error))); + return; + } + } + }); + socket.on('error', fail); + socket.on('close', () => { + sockets.delete(socket); + controls.delete(socket); + if (sourceId && !finished && !reported) { + const error = new Error(`Reporting producer ${sourceId} disconnected without its terminal result/flush handshake.`); + reporter.fail(error, 'reporting', sourceId); + if (!closing) onFatal?.(error); + } + }); + }); + server.listen(address); + await once(server, 'listening'); + return { + environment: { [DESKTOP_REPORT_CONTEXT]: JSON.stringify(context) }, + async close() { + closing = true; + const closed = new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); + const timer = setTimeout(() => { + for (const socket of sockets) socket.destroy(); + }, 5000); + try { + await closed; + } finally { + clearTimeout(timer); + } + }, + }; +} + +export type ReportProducer = { + emit: EventEmitter; + source(id: string): EventEmitter; + flush(): Promise; + finish(status?: 'passed' | 'failed'): Promise; +}; + +export async function connectReportProducer( + sourceId: string, + encoded = process.env[DESKTOP_REPORT_CONTEXT], + onStop?: () => void | Promise, +): Promise { + if (!encoded) throw new Error('No inherited desktop reporting context.'); + const context: unknown = JSON.parse(encoded); + if (!isContext(context)) throw new TypeError('Invalid inherited desktop reporting context.'); + const socket = net.createConnection(context.address); + socket.setEncoding('utf8'); + const acknowledgements = new Map }>(); + let buffer = ''; + let readyResolve: () => void; + let readyReject: (error: Error) => void; + const ready = new Promise((resolve, reject) => { + readyResolve = resolve; + readyReject = reject; + }); + const readyTimer = setTimeout(() => fail(new Error('Reporting IPC handshake timed out.')), 5000); + let transportError: Error | undefined; + let pending = Promise.resolve(); + let queuedBytes = 0; + let ended = false; + function fail(error: Error) { + transportError ??= error; + clearTimeout(readyTimer); + readyReject(error); + for (const item of acknowledgements.values()) { + clearTimeout(item.timer); + item.reject(error); + } + acknowledgements.clear(); + socket.destroy(); + } + socket.on('error', fail); + socket.on('close', () => { + if (!ended || acknowledgements.size) fail(new Error('Reporting IPC closed before its flush completed.')); + }); + socket.on('data', (chunk: string) => { + buffer += chunk; + if (Buffer.byteLength(buffer) > maxFrame) { + fail(new Error('Invalid oversized reporting acknowledgement.')); + return; + } + let newline: number; + while ((newline = buffer.indexOf('\n')) >= 0) { + const line = buffer.slice(0, newline); + buffer = buffer.slice(newline + 1); + try { + const reply: unknown = JSON.parse(line); + if (isRecord(reply) && reply.stop === true && onStop) { + void Promise.resolve() + .then(onStop) + .catch((error: unknown) => fail(error instanceof Error ? error : new Error(String(error)))); + } else if (isRecord(reply) && reply.ready === true) { + clearTimeout(readyTimer); + readyResolve(); + } else if (isRecord(reply) && typeof reply.acknowledged === 'string') { + const item = acknowledgements.get(reply.acknowledged); + if (!item) throw new Error('Unexpected reporting acknowledgement.'); + clearTimeout(item.timer); + acknowledgements.delete(reply.acknowledged); + item.resolve(); + } else throw new Error('Invalid reporting acknowledgement.'); + } catch (error) { + fail(error instanceof Error ? error : new Error(String(error))); + } + } + }); + socket.on('connect', () => + socket.write( + JSON.stringify({ ...context, sourceId, type: 'hello', canStop: !!onStop, ownerScopeId: process.env[DESKTOP_REPORT_SCOPE] }) + '\n', + ), + ); + await ready; + const send = (event: DesktopEvent) => { + if (ended) throw new Error('Reporting producer has already finished.'); + const line = JSON.stringify(event) + '\n'; + const bytes = Buffer.byteLength(line); + if (bytes > maxFrame || queuedBytes + bytes > 4 * maxFrame) throw new Error('Reporting producer exceeded its bounded transport queue.'); + queuedBytes += bytes; + pending = pending + .then(async () => { + if (transportError) throw transportError; + await new Promise((resolve, reject) => { + const timer = setTimeout(() => fail(new Error('Timed out awaiting a persisted reporting acknowledgement.')), 5000); + acknowledgements.set(event.eventId, { resolve, reject, timer }); + socket.write(line); + }); + }) + .finally(() => { + queuedBytes -= bytes; + }); + // The owner observes the rejection through flush()/finish(); prevent an unhandled rejection between synchronous hooks. + void pending.catch((error: unknown) => { + transportError ??= error instanceof Error ? error : new Error(String(error)); + }); + }; + const emit = createEventEmitter(context.runId, sourceId, send); + const sources = new Set(); + return { + emit, + source(id) { + if (!id || sources.has(id)) throw new Error(`Duplicate or empty delegated reporting source ${id}.`); + sources.add(id); + return createEventEmitter(context.runId, `${sourceId}/${id}`, send); + }, + async flush() { + await pending; + if (transportError) throw transportError; + }, + async finish(status = 'passed') { + if (ended) { + await pending; + return; + } + emit({ kind: 'producer-finished', payload: { status } }); + ended = true; + try { + await pending; + } finally { + socket.end(); + } + }, + }; +} + +function isRecord(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} +function isContext(value: unknown): value is ReportTransportContext { + return ( + isRecord(value) && + value.schemaVersion === 1 && + typeof value.address === 'string' && + typeof value.nonce === 'string' && + typeof value.runId === 'string' && + typeof value.artifactsRoot === 'string' + ); +} +function isHello(value: unknown): value is ReportTransportContext & { sourceId: string; canStop?: boolean; ownerScopeId?: string } { + return isContext(value) && 'sourceId' in value && typeof value.sourceId === 'string' && value.sourceId.length > 0; +} + +export async function requestReportStop(scopeId: string, encoded?: string, timeoutMs = 10_000): Promise { + if (!encoded) return 0; + const value = await requestReportControl({ type: 'stop', scopeId, timeoutMs }, encoded, timeoutMs + 1000); + if (isRecord(value) && value.failure) throw new DesktopReportedError(value.failure); + if (!isRecord(value) || typeof value.controlled !== 'number' || !Number.isSafeInteger(value.controlled) || value.controlled < 0) + throw new Error('Invalid desktop stop acknowledgement.'); + return value.controlled; +} + +export async function assertReportedWork(workIds: readonly string[], optional: boolean, encoded: string): Promise { + const value = await requestReportControl({ type: 'assert', workIds, optional }, encoded); + if (isRecord(value) && value.passed === true) return; + if (isRecord(value) && value.failure) throw new DesktopReportedError(value.failure); + throw new Error('Invalid desktop selected-work result.'); +} + +async function requestReportControl(request: object, encoded: string, timeoutMs = 5000): Promise { + const context: unknown = JSON.parse(encoded); + if (!isContext(context)) throw new TypeError('Invalid desktop reporting control context.'); + const socket = net.createConnection(context.address); + socket.setEncoding('utf8'); + try { + return await new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error('Desktop reporting control request timed out.')), timeoutMs); + let buffer = ''; + const settle = (error?: Error, result?: unknown) => { + clearTimeout(timer); + if (error) reject(error); + else resolve(result); + }; + socket.once('error', (error) => settle(error)); + socket.once('end', () => { + if (!buffer.includes('\n')) settle(new Error('Desktop control acknowledgement was not delivered.')); + }); + socket.on('connect', () => socket.write(JSON.stringify({ ...context, ...request }) + '\n')); + socket.on('data', (chunk: string) => { + buffer += chunk; + if (Buffer.byteLength(buffer) > maxFrame) { + settle(new Error('Invalid oversized desktop control acknowledgement.')); + return; + } + if (!buffer.includes('\n')) return; + try { + const value: unknown = JSON.parse(buffer.slice(0, buffer.indexOf('\n'))); + settle(undefined, value); + } catch (error) { + settle(error instanceof Error ? error : new Error(String(error))); + } + }); + }); + } finally { + socket.destroy(); + } +} +function sameSecret(actual: string, expected: string): boolean { + const a = Buffer.from(actual); + const b = Buffer.from(expected); + return a.length === b.length && timingSafeEqual(a, b); +} diff --git a/packages/agentic/desktop-test/src/services.ts b/packages/agentic/desktop-test/src/services.ts new file mode 100644 index 0000000000..b452912f34 --- /dev/null +++ b/packages/agentic/desktop-test/src/services.ts @@ -0,0 +1,115 @@ +import type { DesktopCommandRunner, PreparedDesktopCommand, RunningDesktopCommand } from './commands.js'; +import { deadline } from './commands.js'; +import type { DesktopReportWriter } from './reporting/collector.js'; + +export class DesktopStopRequested extends Error { + constructor() { + super('Owned desktop service stop requested.'); + this.name = 'DesktopStopRequested'; + } +} + +/** Shared resource scope for native applications and integrations, including the transitional channel child. */ +export class DesktopServiceScope { + readonly controller = new AbortController(); + private readonly running: { id: string; command: RunningDesktopCommand }[] = []; + private readonly failed: Promise; + private rejectFailure!: (error: Error) => void; + private closing = false; + private closePromise?: Promise; + private readonly runner: DesktopCommandRunner; + private readonly reporter?: DesktopReportWriter; + + constructor(runner: DesktopCommandRunner, reporter?: DesktopReportWriter) { + this.runner = runner; + this.reporter = reporter; + this.failed = new Promise((_resolve, reject) => { + this.rejectFailure = reject; + }); + // A failure may arrive between guarded phases; retain it without an unhandled rejection. + void this.failed.catch(() => undefined); + } + + fail(error: Error): void { + if (this.closing || this.controller.signal.aborted) return; + this.reporter?.fail(error); + this.controller.abort(error); + this.rejectFailure(error); + } + + requestStop(): void { + this.closing = true; + const reason = new DesktopStopRequested(); + this.controller.abort(reason); + this.rejectFailure(reason); + } + + start(id: string, command: PreparedDesktopCommand): RunningDesktopCommand { + this.controller.signal.throwIfAborted(); + if (this.closing || this.running.some((service) => service.id === id)) + throw new Error(`Cannot start duplicate or closing service ${id}.`); + const running = this.runner.start({ ...command, category: 'service', context: { ...command.context, service: id } }); + this.running.push({ id, command: running }); + void running.completed.then( + (code) => { + if (!this.closing) + this.fail( + running.failure instanceof Error + ? running.failure + : new Error(`Required service "${id}" exited unexpectedly with code ${code}. Log: ${running.logPath ?? 'unavailable'}`), + ); + }, + (error: unknown) => this.fail(error instanceof Error ? error : new Error(String(error))), + ); + return running; + } + + async guard(work: Promise): Promise { + if (this.controller.signal.aborted) void work.catch(() => undefined); + this.controller.signal.throwIfAborted(); + return Promise.race([work, this.failed]); + } + + async ready(id: string, url: string, timeoutMs = 120_000, fetcher = globalThis.fetch): Promise { + const timeout = AbortSignal.timeout(timeoutMs); + const signal = AbortSignal.any([this.controller.signal, timeout]); + await this.guard( + (async () => { + let lastError: unknown; + while (!signal.aborted) { + try { + const response = await fetcher(url, { signal: AbortSignal.any([signal, AbortSignal.timeout(2000)]), redirect: 'error' }); + await response.body?.cancel(); + if (response.ok) { + this.reporter?.emit({ kind: 'service-ready', payload: { message: id } }, id, { service: id }); + return; + } + lastError = new Error(`Readiness returned HTTP ${response.status}.`); + } catch (error) { + lastError = error; + } + await new Promise((resolve) => setTimeout(resolve, 100)); + } + throw new Error(`Service "${id}" did not become ready within ${timeoutMs}ms.`, { cause: lastError ?? signal.reason }); + })(), + ); + } + + close(): Promise { + return (this.closePromise ??= (async () => { + this.closing = true; + const failed = this.controller.signal.aborted && !(this.controller.signal.reason instanceof DesktopStopRequested); + this.controller.abort(new Error('Desktop service scope is closing.')); + const failures: unknown[] = []; + for (const { id, command } of [...this.running].reverse()) { + try { + await deadline(command.stop({ failed }), 45_000, `Stopping ${id}`); + } catch (error) { + failures.push(error); + this.reporter?.fail(error, 'cleanup', id); + } + } + if (failures.length) throw new AggregateError(failures, 'Owned desktop services did not shut down cleanly.'); + })()); + } +} diff --git a/packages/agentic/desktop-test/src/supervisor.ts b/packages/agentic/desktop-test/src/supervisor.ts new file mode 100644 index 0000000000..e56d6a70c4 --- /dev/null +++ b/packages/agentic/desktop-test/src/supervisor.ts @@ -0,0 +1,264 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { createHash } from 'node:crypto'; + +import { createDesktopDriverServer, NativeDesktopHost, resolveNativeDesktopDriver } from '@fluentui-react-native/desktop-driver'; +import type { DesktopDriverServer, DesktopHost, DesktopTarget, NativeDesktopHostOptions } from '@fluentui-react-native/desktop-driver'; +import type { DesktopCommand, ResolvedDesktopTarget } from './config.js'; +import type { DesktopIntegration } from './integration.js'; +import { NodeDesktopCommandRunner, deadline } from './commands.js'; +import type { PreparedDesktopCommand, RunningDesktopCommand } from './commands.js'; +import { DesktopServiceScope } from './services.js'; +import { resolveDesktopApplication } from './adapters.js'; +import { DesktopRunReporter } from './reporting/collector.js'; +import type { DesktopReportWriter } from './reporting/collector.js'; +import type { DesktopReportSession } from './reporting/session.js'; +import { startReportTransport } from './reporting/transport.js'; +import type { ReportTransport } from './reporting/transport.js'; + +export type DesktopSupervisorOptions = { + target: ResolvedDesktopTarget; + reporter: DesktopReportWriter; + reportSession?: DesktopReportSession; + integration?: DesktopIntegration; + hostFactory?: (target: ResolvedDesktopTarget) => Promise; + lockDirectory?: string; +}; + +export class DesktopSupervisor { + readonly reporter: DesktopReportWriter; + readonly target: ResolvedDesktopTarget; + readonly runner: NodeDesktopCommandRunner; + readonly services: DesktopServiceScope; + private serverValue?: DesktopDriverServer; + private transport?: ReportTransport; + private opened = false; + private releaseClaim?: () => void; + private closePromise?: Promise; + private readonly environment: Record = {}; + private readonly workers = new Set(); + private readonly options: DesktopSupervisorOptions; + + constructor(options: DesktopSupervisorOptions) { + this.options = options; + this.target = options.target; + this.reporter = options.reporter; + Object.assign(this.environment, options.reportSession?.environment); + this.runner = new NodeDesktopCommandRunner({ reporter: this.reporter, environment: this.environment }); + this.services = new DesktopServiceScope(this.runner, this.reporter); + } + + get server(): DesktopDriverServer { + if (!this.serverValue) throw new Error('Desktop supervisor has not started.'); + return this.serverValue; + } + + async open(interactive = true): Promise { + if (this.closePromise) throw new Error('Desktop supervisor is closing.'); + if (this.opened) return; + if (interactive && !this.options.reportSession?.delegated) + this.releaseClaim = claimDesktopWorkload(this.reporter.runId, this.options.lockDirectory); + try { + if (!this.options.reportSession) { + if (!(this.reporter instanceof DesktopRunReporter)) throw new Error('A delegated supervisor requires its inherited reportSession.'); + this.transport = await startReportTransport(this.reporter, (error) => this.services.fail(error)); + Object.assign(this.environment, this.transport.environment); + } + this.opened = true; + } catch (error) { + this.releaseClaim?.(); + this.releaseClaim = undefined; + throw error; + } + } + + async start(): Promise { + if (this.serverValue || this.closePromise) throw new Error('Desktop supervisor may only be started once.'); + await this.open(); + try { + await this.startConfiguredServices(); + const host = this.options.hostFactory ? await this.options.hostFactory(this.target) : await this.createHost(); + try { + const extension = await this.options.integration?.createTarget?.({ host, target: this.target, reporter: this.reporter }); + this.serverValue = await createDesktopDriverServer({ + targets: [ + { + endpoint: this.target.endpoint, + id: this.target.id, + platformName: this.target.endpoint === 'macos' ? 'macos' : 'windows', + renderer: this.target.host.renderer, + launchTimeoutMs: this.target.launchTimeoutMs, + launchModes: this.target.host.ownership === 'external' ? ['attach'] : ['launch'], + host, + ...extension, + }, + ], + }); + await host.probe(this.services.controller.signal); + } catch (error) { + if (!this.serverValue) { + try { + await host.dispose(); + } catch (cleanup) { + throw new AggregateError([error, cleanup], 'Desktop target startup and disposal failed.'); + } + } + throw error; + } + this.reporter.emit( + { kind: 'service-ready', payload: { message: `WebDriver ${this.server.url}; target ${this.target.id}` } }, + 'webdriver', + { endpoint: this.target.endpoint, renderer: this.target.host.renderer }, + ); + } catch (error) { + this.reporter.fail(error); + try { + await this.close(); + } catch (cleanup) { + throw new AggregateError([error, cleanup], 'Desktop supervisor startup and cleanup failed.'); + } + throw error; + } + } + + async startConfiguredServices(): Promise { + for (const service of this.target.services ?? []) { + this.services.start(service.id, this.prepare(service.command, service.id)); + if (service.readyUrl) await this.services.ready(service.id, service.readyUrl, this.target.launchTimeoutMs); + } + } + + private async createHost(): Promise { + const requiredOS = this.target.endpoint === 'macos' ? 'darwin' : 'win32'; + if (process.platform !== requiredOS) + throw new Error(`Endpoint ${this.target.endpoint} requires ${requiredOS}; native helpers cannot run on ${process.platform}.`); + const application = await resolveDesktopApplication(this.target, this.services.controller.signal); + if (application.leaseNonce) this.reporter.addSecret(application.leaseNonce); + const artifact = await resolveNativeDesktopDriver({ + ...this.target.nativeDriver, + platform: this.target.endpoint, + buildPolicy: 'never', + signal: this.services.controller.signal, + }); + return new NativeDesktopHost({ + application, + artifact, + endpoint: this.target.endpoint, + onStderr: (text) => + this.reporter.emit({ kind: 'output', payload: { stream: 'stderr', text } }, 'native-helper', { + service: 'native-helper', + endpoint: this.target.endpoint, + }), + onFailure: (error) => this.services.fail(error), + }); + } + + prepare(command: DesktopCommand, label?: string): PreparedDesktopCommand { + return { + ...command, + args: command.args ?? [], + cwd: path.resolve(this.target.projectRoot, command.cwd ?? '.'), + env: { + ...(this.target.endpoint === 'macos' && this.target.host.renderer + ? { RCT_NEW_ARCH_ENABLED: this.target.host.renderer === 'paper' ? '0' : '1' } + : {}), + ...command.env, + }, + label, + context: { endpoint: this.target.endpoint, renderer: this.target.host.renderer }, + }; + } + + async worker(command: PreparedDesktopCommand, timeoutMs: number, verify?: (code: number) => Promise): Promise { + const worker = this.runner.start({ ...command, category: 'worker' }); + this.workers.add(worker); + let failure: unknown; + let code = 1; + try { + code = await this.services.guard(deadline(worker.completed, timeoutMs, command.label ?? 'Desktop worker')); + await verify?.(code); + } catch (error) { + failure = error; + } + try { + await worker.stop({ failed: failure !== undefined || code !== 0 }); + } catch (error) { + throw new AggregateError(failure === undefined ? [error] : [failure, error], 'Desktop worker and cleanup failed.'); + } finally { + this.workers.delete(worker); + } + if (failure !== undefined) throw failure; + return code; + } + + close(): Promise { + return (this.closePromise ??= (async () => { + const failures: unknown[] = []; + const close = async (label: string, operation: () => Promise) => { + try { + await operation(); + } catch (error) { + failures.push(error); + this.reporter.fail(error, 'cleanup', label); + } + }; + for (const worker of this.workers) await close('worker', () => worker.stop({ failed: true })); + this.workers.clear(); + if (this.serverValue) await close('webdriver', () => deadline(this.serverValue!.close(), 30_000, 'Closing desktop sessions')); + await close('services', () => this.services.close()); + await close('commands', () => this.runner.close()); + if (this.transport) await close('reporting', () => this.transport!.close()); + if (this.releaseClaim) { + try { + this.releaseClaim(); + } catch (error) { + failures.push(error); + this.reporter.fail(error, 'cleanup', 'desktop-claim'); + } + } + if (failures.length) throw new AggregateError(failures, 'Desktop supervisor cleanup failed.'); + })()); + } +} + +/** Storybook's channel child uses the same native composition without acquiring a second application owner. */ +export async function startNativeDesktopService( + options: NativeDesktopHostOptions & { + target: Omit; + port?: number; + host?: string; + }, +): Promise { + const host = new NativeDesktopHost(options); + try { + return await createDesktopDriverServer({ host: options.host, port: options.port, targets: [{ ...options.target, host }] }); + } catch (error) { + try { + await host.dispose(); + } catch (cleanup) { + throw new AggregateError([error, cleanup], 'Native listener startup and disposal failed.'); + } + throw error; + } +} + +export function claimDesktopWorkload(runId: string, directory?: string): () => void { + const user = createHash('sha256').update(os.homedir()).digest('hex').slice(0, 16); + const root = directory ?? path.join(os.tmpdir(), `furn-desktop-workload-${user}`); + fs.mkdirSync(root, { recursive: true, mode: 0o700 }); + const file = path.join(root, 'interactive.lock'); + const contents = JSON.stringify({ schemaVersion: 1, processId: process.pid, runId }); + try { + fs.writeFileSync(file, contents, { flag: 'wx', mode: 0o600 }); + } catch (error) { + throw new Error( + `Another desktop workload holds ${file}. Do not run parallel physical-input suites. After a crash, verify the recorded owner has exited before removing its stale claim.`, + { cause: error }, + ); + } + return () => { + if (fs.readFileSync(file, 'utf8') !== contents) throw new Error('Desktop claim ownership changed; refusing to remove another owner.'); + fs.unlinkSync(file); + }; +} diff --git a/packages/agentic/desktop-test/src/wdio/configuration.ts b/packages/agentic/desktop-test/src/wdio/configuration.ts new file mode 100644 index 0000000000..506294ad4f --- /dev/null +++ b/packages/agentic/desktop-test/src/wdio/configuration.ts @@ -0,0 +1,130 @@ +import { pathToFileURL } from 'node:url'; +import { randomUUID } from 'node:crypto'; +import { installDesktopWebdriverCommands } from '@fluentui-react-native/desktop-driver/wdio'; +import type { DesktopEndpoint, DesktopRenderer } from '@fluentui-react-native/desktop-driver'; +import type { DesktopReadiness } from '../config.js'; +import { connectReportProducer } from '../reporting/transport.js'; +import { DesktopReportedError, serializeReportError } from '../reporting/protocol.js'; + +export type DesktopWdioSettings = { + url: string; + targetId: string; + endpoint: DesktopEndpoint; + renderer?: DesktopRenderer; + ownership: 'session' | 'external'; + specs: (string | string[])[]; + framework: string; + reporter: string; + hooks?: string; + rootDir: string; + specFileRetries: number; + bail: number; + timeoutMs: number; + launchTimeoutMs: number; + readiness?: DesktopReadiness; +}; + +type Hook = (...args: unknown[]) => unknown; +const hookNames = [ + 'onPrepare', + 'onComplete', + 'beforeSession', + 'afterSession', + 'before', + 'after', + 'beforeSuite', + 'afterSuite', + 'beforeTest', + 'afterTest', + 'beforeHook', + 'afterHook', + 'beforeCommand', + 'afterCommand', +] as const; + +export async function createDesktopWdioConfiguration(settings: DesktopWdioSettings) { + let hooks: Partial> = {}; + if (settings.hooks) { + const loaded: unknown = await import(pathToFileURL(settings.hooks).href); + const value = loaded && typeof loaded === 'object' && 'default' in loaded ? loaded.default : loaded; + if (!value || typeof value !== 'object' || Array.isArray(value)) throw new TypeError('Desktop hooks must export a hook object.'); + for (const [key, callback] of Object.entries(value)) { + if (!hookNames.some((name) => name === key) || typeof callback !== 'function') + throw new TypeError(`Unsupported desktop hook "${key}".`); + } + hooks = value; + } + const address = new URL(settings.url); + const readiness = settings.readiness; + const configuration = { + ...hooks, + runner: 'local', + framework: settings.framework, + rootDir: settings.rootDir, + specs: settings.specs, + maxInstances: 1, + bail: settings.bail, + specFileRetries: settings.specFileRetries, + logLevel: 'error', + hostname: address.hostname, + port: Number(address.port), + path: address.pathname, + protocol: address.protocol.slice(0, -1), + connectionRetryCount: 0, + connectionRetryTimeout: Math.min(settings.launchTimeoutMs + 15_000, 2_147_483_647), + waitforTimeout: 20_000, + waitforInterval: 100, + reporterSyncTimeout: 10_000, + jasmineOpts: { defaultTimeoutInterval: settings.timeoutMs }, + reporters: [ + [ + settings.reporter, + { specFileRetries: settings.specFileRetries, desktopContext: { endpoint: settings.endpoint, renderer: settings.renderer } }, + ], + ], + capabilities: [ + { + browserName: 'furn-native-desktop', + platformName: settings.endpoint === 'macos' ? 'macos' : 'windows', + 'furn:target': settings.targetId, + 'furn:launchMode': settings.ownership === 'external' ? 'attach' : 'launch', + 'furn:clickMode': 'auto', + }, + ], + beforeSession: async (...args: unknown[]) => { + process.env.E2ETEST_PLATFORM = settings.endpoint; + await hooks.beforeSession?.(...args); + }, + before: async (capabilities: unknown, specs: unknown, browser: WebdriverIO.Browser) => { + installDesktopWebdriverCommands(browser, settings.url); + if ((await browser.getWindowHandles()).length === 0) throw new Error('Native application did not expose a window.'); + if (readiness?.testId) { + await browser.$(`~${readiness.testId}`).waitForDisplayed({ + timeout: readiness.timeoutMs ?? settings.launchTimeoutMs, + timeoutMsg: `Native readiness marker "${readiness.testId}" was not displayed.`, + }); + } + await hooks.before?.(capabilities, specs, browser); + }, + }; + for (const name of hookNames) { + const hook = configuration[name] as Hook | undefined; + if (!hook) continue; + const guarded: Hook = async (...args) => { + try { + return await hook(...args); + } catch (error) { + const failure = { failureId: randomUUID(), kind: 'infrastructure' as const, error: serializeReportError(error) }; + const producer = await connectReportProducer(`wdio-hook-${randomUUID()}`); + producer.emit({ kind: 'service-failed', payload: { failure } }, `hook:${name}`, { + endpoint: settings.endpoint, + renderer: settings.renderer, + }); + await producer.finish('failed'); + throw new DesktopReportedError(failure); + } + }; + Object.assign(configuration, { [name]: guarded }); + } + return configuration; +} diff --git a/packages/agentic/desktop-test/src/wdio/identity.ts b/packages/agentic/desktop-test/src/wdio/identity.ts new file mode 100644 index 0000000000..3b11ccf0d6 --- /dev/null +++ b/packages/agentic/desktop-test/src/wdio/identity.ts @@ -0,0 +1,8 @@ +import { createHash } from 'node:crypto'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +export function wdioGroupId(specs: readonly string[]): string { + const files = specs.map((spec) => path.resolve(spec.startsWith('file:') ? fileURLToPath(spec) : spec)); + return `wdio-${createHash('sha256').update(JSON.stringify(files)).digest('hex').slice(0, 20)}`; +} diff --git a/packages/agentic/desktop-test/src/wdio/index.ts b/packages/agentic/desktop-test/src/wdio/index.ts new file mode 100644 index 0000000000..f5f01dd4cc --- /dev/null +++ b/packages/agentic/desktop-test/src/wdio/index.ts @@ -0,0 +1,3 @@ +export { runDesktopWdioSuite } from './run.js'; +export { createDesktopWdioConfiguration } from './configuration.js'; +export type { DesktopWdioSettings } from './configuration.js'; diff --git a/packages/agentic/desktop-test/src/wdio/reporter.ts b/packages/agentic/desktop-test/src/wdio/reporter.ts new file mode 100644 index 0000000000..8a7e84a9c9 --- /dev/null +++ b/packages/agentic/desktop-test/src/wdio/reporter.ts @@ -0,0 +1,161 @@ +import { createHash, randomUUID } from 'node:crypto'; +import WDIOReporter from '@wdio/reporter'; +import type { HookStats, RunnerStats, TestStats } from '@wdio/reporter'; +import { connectReportProducer } from '../reporting/transport.js'; +import type { ReportProducer } from '../reporting/transport.js'; +import { serializeReportError } from '../reporting/protocol.js'; +import type { EventContext, EventData, ReportFailure } from '../reporting/protocol.js'; +import { wdioGroupId } from './identity.js'; + +type ReporterOptions = ConstructorParameters[0] & { desktopContext?: EventContext; specFileRetries?: number }; + +export default class DesktopWdioReporter extends WDIOReporter { + private readonly producer: Promise; + private queue = Promise.resolve(); + private pending = 0; + private context: EventContext; + private scopeId = ''; + private groupId = ''; + private workerAttempt = 0; + private readonly specFileRetries: number; + private readonly endedTests = new Set(); + + constructor(options: ReporterOptions) { + super({ ...options, stdout: false }); + this.context = options.desktopContext ?? {}; + this.specFileRetries = options.specFileRetries ?? 0; + this.producer = connectReportProducer(`wdio-${randomUUID()}`); + void this.producer.catch((error: unknown) => { + process.stderr.write(`Desktop WDIO reporting connection failed: ${String(error)}\n`); + process.exitCode = 1; + }); + } + + override get isSynchronised(): boolean { + return this.pending === 0; + } + + private schedule(action: (producer: ReportProducer) => void | Promise): void { + this.pending++; + this.queue = this.queue + .then(async () => action(await this.producer)) + .catch((error: unknown) => { + process.stderr.write(`Desktop WDIO reporting failed: ${String(error)}\n`); + process.exitCode = 1; + }) + .finally(() => { + this.pending--; + }); + } + + private report(event: EventData, context = this.context): void { + this.schedule(async (producer) => { + producer.emit(event, this.scopeId, context, this.groupId); + await producer.flush(); + }); + } + + override onRunnerStart(runner: RunnerStats): void { + this.groupId = wdioGroupId(runner.specs); + // WDIO 9 reports the remaining spec attempts (including the current worker), not an increasing attempt index. + this.workerAttempt = this.specFileRetries + 1 - (runner.retry ?? 1); + this.scopeId = `${this.groupId}-${this.workerAttempt}`; + this.context = { ...this.context, worker: runner.cid, spec: runner.specs.join(' + '), attempt: this.workerAttempt }; + this.report({ + kind: 'scope-start', + payload: { + label: `${this.context.endpoint ?? 'native'} ${this.context.renderer ?? ''} ${runner.specs.join(' + ')} (attempt ${this.workerAttempt + 1})`, + category: 'worker', + }, + }); + } + + override onTestStart(test: TestStats): void { + this.report( + { kind: 'test-start', payload: { testId: this.testId(test), title: test.fullTitle, attempt: this.attempt(test) } }, + { ...this.context, test: test.uid }, + ); + } + + override onTestEnd(test: TestStats): void { + this.testResult(test); + } + override onTestSkip(test: TestStats): void { + this.testResult(test); + } + override onTestPending(test: TestStats): void { + this.testResult(test); + } + override onTestRetry(test: TestStats): void { + this.testResult(test); + } + + private testResult(test: TestStats): void { + const testId = this.testId(test); + const attempt = this.attempt(test); + const key = `${testId}/${attempt}`; + if (this.endedTests.has(key)) return; + this.endedTests.add(key); + const status = test.state === 'pending' ? 'skipped' : test.state; + this.report( + { + kind: 'test-result', + payload: { + testId, + title: test.fullTitle, + attempt, + status, + durationMs: test.duration, + ...(status === 'skipped' ? { reason: test.pendingReason || 'Skipped by the test framework.' } : {}), + ...(status === 'failed' + ? { failure: this.failure(`${key}/failure`, test.error ?? new AggregateError(test.errors ?? [], 'WDIO test failed.'), 'test') } + : {}), + }, + }, + { ...this.context, test: test.uid, attempt }, + ); + } + + override onHookEnd(hook: HookStats): void { + if (hook.error || hook.errors?.length || hook.state === 'failed') { + this.report({ + kind: 'diagnostic', + payload: { + level: 'error', + message: `Hook failed: ${hook.title}`, + failure: this.failure( + `${this.scopeId}/hook/${hook.uid}`, + hook.error ?? new AggregateError(hook.errors ?? [], hook.title), + 'infrastructure', + ), + }, + }); + } + } + + override onRunnerEnd(runner: RunnerStats): void { + if (runner.error) + this.report({ + kind: 'diagnostic', + payload: { + level: 'error', + message: runner.error, + failure: this.failure(`${this.scopeId}/runner`, new Error(runner.error), 'infrastructure'), + }, + }); + this.report({ kind: 'scope-end', payload: { status: runner.failures || runner.error ? 'failed' : 'passed' } }); + this.schedule((producer) => producer.finish(runner.failures || runner.error ? 'failed' : 'passed')); + } + + private testId(test: TestStats): string { + return createHash('sha256') + .update(JSON.stringify([this.groupId, test.uid, test.fullTitle])) + .digest('hex'); + } + private attempt(test: TestStats): number { + return this.workerAttempt * 1_000_000 + (test.retries ?? 0); + } + private failure(id: string, error: unknown, kind: ReportFailure['kind']): ReportFailure { + return { failureId: createHash('sha256').update(id).digest('hex'), kind, error: serializeReportError(error) }; + } +} diff --git a/packages/agentic/desktop-test/src/wdio/run.ts b/packages/agentic/desktop-test/src/wdio/run.ts new file mode 100644 index 0000000000..c86eeb5efe --- /dev/null +++ b/packages/agentic/desktop-test/src/wdio/run.ts @@ -0,0 +1,83 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { resolve } from 'import-meta-resolve'; +import type { ResolvedDesktopConfig, DesktopWdioSuite } from '../config.js'; +import { resolveDesktopSpecs } from '../config.js'; +import type { DesktopSupervisor } from '../supervisor.js'; +import type { DesktopWdioSettings } from './configuration.js'; +import { wdioGroupId } from './identity.js'; + +export async function runDesktopWdioSuite(supervisor: DesktopSupervisor, config: ResolvedDesktopConfig, suiteName: string): Promise { + const suite = config.config.suites?.[suiteName]; + if (!suite || suite.kind !== 'wdio') throw new Error(`No standalone WDIO suite "${suiteName}" is configured.`); + const specs = resolveDesktopSpecs(suite, supervisor.target.endpoint, config.projectRoot); + const workIds: string[] = []; + for (const selected of specs) { + const files = typeof selected === 'string' ? [selected] : selected; + const workId = wdioGroupId(files); + workIds.push(workId); + supervisor.reporter.emit({ kind: 'work-selected', payload: { workId, label: files.join(' + ') } }); + } + await supervisor.reporter.flush(); + const parent = pathToFileURL(path.join(config.projectRoot, 'package.json')).href; + const modulePath = (name: string) => { + try { + return fileURLToPath(resolve(name, parent)); + } catch (error) { + throw new Error(`The suite workspace must install ${name}; desktop-test does not install test frameworks into applications.`, { + cause: error, + }); + } + }; + modulePath('@wdio/local-runner'); + modulePath(`@wdio/${suite.framework}-framework`); + const settings: DesktopWdioSettings = { + url: supervisor.server.url, + targetId: supervisor.target.id, + endpoint: supervisor.target.endpoint, + renderer: supervisor.target.host.renderer, + ownership: supervisor.target.host.ownership ?? 'session', + specs, + framework: suite.framework, + reporter: fileURLToPath(new URL('./reporter.js', import.meta.url)), + rootDir: config.projectRoot, + hooks: suite.hooks ? path.resolve(path.dirname(config.configPath), suite.hooks) : undefined, + specFileRetries: suite.specFileRetries ?? 0, + bail: suite.bail ?? 1, + timeoutMs: suite.timeoutMs ?? 60_000, + launchTimeoutMs: supervisor.target.launchTimeoutMs ?? 180_000, + readiness: suite.readiness ?? supervisor.target.readiness, + }; + // WDIO initializes its TypeScript loader before evaluating TypeScript configurations and their hooks. + const configPath = supervisor.reporter.artifactPath('wdio.generated.mts'); + fs.writeFileSync( + configPath, + `import { createDesktopWdioConfiguration } from ${JSON.stringify(new URL('./configuration.js', import.meta.url).href)};\n` + + `export const config = await createDesktopWdioConfiguration(${JSON.stringify(settings)});\n`, + { mode: 0o600 }, + ); + const launcher = modulePath('@wdio/cli'); + await supervisor.worker( + { + command: process.execPath, + args: [fileURLToPath(new URL('../../config/launcher.cjs', import.meta.url)), pathToFileURL(launcher).href, configPath], + cwd: config.projectRoot, + env: { E2ETEST_PLATFORM: supervisor.target.endpoint, NATIVE_TESTING: 'false' }, + label: `WDIO ${suiteName} (${supervisor.target.endpoint})`, + }, + suiteDeadline(suite, specs.length), + async (code) => { + await supervisor.reporter.assertSuccessful(workIds, suite.optional); + if (code !== 0) + throw new Error( + `WDIO suite "${suiteName}" exited with code ${code} without a corresponding engine failure; see the retained worker log.`, + ); + }, + ); +} + +function suiteDeadline(suite: DesktopWdioSuite, groups: number): number { + // A suite deadline bounds a wedged launcher; Jasmine still enforces its independent per-test timeout. + return Math.min(2_147_483_647, Math.max(300_000, (suite.timeoutMs ?? 60_000) * 100 * groups * ((suite.specFileRetries ?? 0) + 1))); +} diff --git a/packages/agentic/desktop-test/tsconfig.json b/packages/agentic/desktop-test/tsconfig.json new file mode 100644 index 0000000000..3f6765b94a --- /dev/null +++ b/packages/agentic/desktop-test/tsconfig.json @@ -0,0 +1,12 @@ +{ + "extends": "@fluentui-react-native/scripts/tsconfig", + "compilerOptions": { + "outDir": "lib", + "rootDir": "src", + "composite": true, + "rewriteRelativeImportExtensions": true, + "tsBuildInfoFile": ".cache/tsconfig.tsbuildinfo" + }, + "include": ["src"], + "references": [{ "path": "../desktop-driver/tsconfig.json" }, { "path": "../../../scripts/tsconfig.json" }] +} diff --git a/packages/agentic/storybook-desktop/README.md b/packages/agentic/storybook-desktop/README.md index 0d337b2f91..59372965ea 100644 --- a/packages/agentic/storybook-desktop/README.md +++ b/packages/agentic/storybook-desktop/README.md @@ -18,50 +18,65 @@ workspace locator, so `storybook-desktop` and `storybook-server` binaries work with Yarn's pnpm linker instead of resolving through an unmaterialized virtual workspace path. -Consuming apps own their native identity in `app.json`, story globs, generated `storybook.requires` file, component -dependencies, and exceptional platform automation. A root `storybook.config.ts` declares which packages supply stories -and overrides only platform behavior that cannot use the shared defaults: +Application-neutral targets, service ownership, and reporting belong to +[`desktop-test`](../desktop-test/README.md). This package adds explicit Storybook +discovery, authenticated readiness, traversal, and isolated CSF callbacks. +Consuming apps own their native identity in `app.json`, story globs, generated +`storybook.requires` file, and component dependencies. A root `desktop.config.mts` +imports the optional integration and overrides only exceptional platform behavior: ```ts import { createWindowsSmokeOptions, createWin32RunCommand, createWin32SmokeCommand, - makeDesktopStorybookConfig, + storybook, } from '@fluentui-react-native/storybook-desktop/config'; +import { defineDesktopConfig } from '@fluentui-react-native/desktop-test/config'; const win32Host = { component: 'MyStorybook', windowTitle: 'My Storybook (Win32)', } as const; -export default makeDesktopStorybookConfig({ +export default defineDesktopConfig({ + schemaVersion: 1, projectRoot: new URL('.', import.meta.url), - storyPackages: [ - '@scope/components', - [ - '@scope/native-component', - { - platforms: ['macos', 'windows'], - }, - ], - ], - platformOptions: { - windows: { - smoke: createWindowsSmokeOptions({ - windowTitle: 'My Storybook', - }), - }, + targets: { + macos: { host: { kind: 'react-native-test-app', renderer: 'fabric' } }, + windows: { host: { kind: 'react-native-test-app', renderer: 'fabric' } }, win32: { - run: createWin32RunCommand(win32Host), - smoke: { - command: createWin32SmokeCommand({ - ...win32Host, - testIDPrefix: 'my-storybook', + host: { kind: 'rex', renderer: 'paper', ...win32Host }, + commands: { run: createWin32RunCommand(win32Host) }, + }, + }, + suites: { stories: { kind: 'csf-wdio' } }, + integration: storybook({ + storyPackages: [ + '@scope/components', + [ + '@scope/native-component', + { + platforms: ['macos', 'windows'], + }, + ], + ], + platformOptions: { + windows: { + smoke: createWindowsSmokeOptions({ + windowTitle: 'My Storybook', }), }, + win32: { + smoke: { + command: createWin32SmokeCommand({ + ...win32Host, + testIDPrefix: 'my-storybook', + }), + }, + }, }, - }, + }), }); ``` @@ -83,12 +98,12 @@ The corresponding React Native Test App manifest supplies native identity: The app's `src/main.ts` becomes a small adapter: ```ts -import config from '../storybook.config.ts'; +import config from '../desktop.config.mts'; -export default config.getStorybookConfig(); +export default config.integration.configure(config.projectRoot).getStorybookConfig(); ``` -The returned `DesktopStorybookConfig` resolves package roots lazily and exposes app identity, display name, the custom +The integration's `configure()` returns a `DesktopStorybookConfig` that resolves package roots lazily and exposes app identity, display name, the custom `storybook.testIDPrefix` field, package metadata, resolved story packages, and generated story globs for CLI and test orchestration. A config-level `testIDPrefix` remains available as an explicit override for consumers that do not store Storybook identity in `app.json`. @@ -109,7 +124,10 @@ smoke runs configured WDIO cases across all matching stories and legacy tests fails instead of reporting a false pass. This retains normal native lease/readiness and process cleanup; it does not change input capability policy. -The `storybook-desktop` binary loads `storybook.config.ts`, `.mts`, `.js`, `.mjs`, or `.cjs` from the current package. +The `storybook-desktop` binary first loads `desktop.config.mts`, `.ts`, `.mjs`, +`.js`, or `.cjs` from the current package and requires its explicit Storybook +integration. Legacy `storybook.config.*` files and `makeDesktopStorybookConfig()` +remain supported for compatibility. Use `.mts` when the consuming package otherwise defaults JavaScript files to CommonJS. Select a target with a short platform option, or omit it to use `FURN_STORYBOOK_PLATFORM` and then the host default: @@ -262,7 +280,7 @@ yarn storybook test --macos --story 'components-button--*' yarn storybook test --macos --story 'components-button--*' --test '*activation*' ``` -Defaults come from `storybook.config.mts`; no Mocha, Jest, or WDIO config file +Defaults come from the Storybook integration in `desktop.config.mts`; no Mocha, Jest, or WDIO config file is needed: ```ts @@ -299,8 +317,10 @@ per case, not to the entire suite. Assertion failures and timeouts do not prevent other independent cases in that story from running; connection or session-cleanup failures stop execution. Do not share element handles or assume state survives between cases. -Reports and best-effort failure source/tree evidence live in -`artifacts//wdio`. Generated executable files are removed after the +CLI reports and best-effort failure source/tree evidence live in +`artifacts/desktop//evidence//wdio`. Direct API calls without +the shared reporter retain the `artifacts//wdio` fallback. +Generated executable files are removed after the run. Named results include `testName`, and failure evidence uses distinct paths for each case. Callback digests participate in manifest freshness checks: restart the driver and reload the app after editing tests. @@ -315,8 +335,11 @@ story ID, runs that story's remaining legacy `desktop-e2e` plans and selected `w then advances to the next story. Non-default stories are included. Every test gets a fresh preview; the currently selected sidebar page is not a prerequisite. The aggregate smoke report contains static results in `tests` and executable -results in `wdio`. Per-story executable reports are under -`artifacts//desktop-driver/wdio`. +results in `wdio`. With the shared CLI reporter, per-story executable reports +are under +`artifacts/desktop//evidence//static/wdio/story-`. +Direct API calls without it retain the +`artifacts//desktop-driver/wdio` fallback. For legacy JSON plans only, use the consuming app's Desktop Driver CLI: @@ -352,7 +375,9 @@ before creating the attached WebDriver session. Consumers provide only native identity, title, test-ID prefix, and optional required story IDs. The Windows helper also records React Native Test App's Debug Metro port (`8081` by default), while Storybook and Desktop Driver ports remain enlistment-specific. -Artifacts are written beneath the consuming app's `artifacts/windows` or `artifacts/win32` directory. +The CLI writes canonical reports beneath the consuming app's +`artifacts/desktop/` directory. Native lifecycle diagnostics also remain +under `artifacts/windows` or `artifacts/win32`. The Windows Fabric lifecycle restarts only its exact owned app after the full catalog traversal, then rewrites the nonce-bound application lease and runs @@ -382,13 +407,22 @@ The same operations are available without Commander: ```ts import { DesktopStorybookCli } from '@fluentui-react-native/storybook-desktop/cli'; -import config from './storybook.config.ts'; - -const storybook = new DesktopStorybookCli(config); -await storybook.bundle('macos'); -await storybook.smoke('macos', { mode: 'stories-and-tests' }); +import { getStorybookIntegration } from '@fluentui-react-native/storybook-desktop/config'; +import { loadDesktopConfig } from '@fluentui-react-native/desktop-test/config'; + +const desktop = await loadDesktopConfig(); +const config = getStorybookIntegration(desktop.config).configure(desktop.projectRoot, desktop); +const api = new DesktopStorybookCli(config); +await api.bundle('macos'); +await api.smoke('macos', { mode: 'stories-and-tests' }); ``` +Higher-level coordinators should inject their shared `reporter`, `runner`, and +`services` so API calls join the same owned lifecycle. The generic +`desktop-test --target test --suite stories` command runs the complete +owned Storybook smoke/test lifecycle; `storybook test` remains the callback-only +command against an already running app. + Command runners are injectable through the constructor for higher-level automation and tests. `DesktopCommand`, `DesktopPlatformOptions`, `createDesktopStorybookInstance()`, and the related configuration types are exported from the `/config` subpath. `server()` runs the foreground server until it is stopped, so supervisors should invoke it as a @@ -396,26 +430,31 @@ dedicated task rather than await it before another operation. ### Command logs and pipeline failures -The shared command runner writes stdout and stderr continuously to a single -log file per command under `artifacts/storybook-commands` in that command's -working directory. Both streams share one descriptor, preserving their write -order without holding an entire build log in memory. - -Normal output contains a start record with the log path and a short completion -summary. A failed command replays its complete combined log between labeled +The neutral harness owns one collector for the complete run, including nested +CLIs, the channel server, PowerShell-launched services, and isolated callbacks. +It writes `events.jsonl`, `run.json`, `junit.xml`, and attributed command logs +under `artifacts/desktop/`. Both output streams are drained continuously; +their merged log preserves collector arrival order, not an ordering guarantee +between independent producer streams. + +Normal output contains live labeled stderr and test progress, a start record +with the log path, and a short completion summary. Required-service failures +abort dependent work even if an outer wrapper remains alive. +A failed command replays its complete combined log between labeled `BEGIN`/`END` records, then reports failure on stderr. Replay groups are serialized so concurrently completing commands do not mix their output. Owned background services also replay their logs when smoke fails. -Use the global `--verbose` option to replay successful-command logs as well: +Use the global `--verbose` option to stream informational output and replay +successful-command logs as well: ```sh yarn storybook --verbose test --macos --story 'components-button--*' yarn storybook --verbose smoke --macos --mode stories-and-tests ``` -Console replay is grouped at command completion; the files update while -commands are running, so ongoing logs are available without shell redirection. +Console replay is grouped at command completion; stderr is already visible +while commands run, without shell redirection. The verbose setting is inherited by package-owned Windows and Win32 lifecycle subprocesses. @@ -427,7 +466,12 @@ including with `dot`; CLI boundaries preserve nested causes and aggregate errors. A failed test cannot be hidden by a successful worker exit or by a secondary evidence/cleanup failure. -Archive the command logs alongside the per-platform test artifacts in CI. +GitHub Actions groups, annotations, summaries, JSON, and JUnit derive from the +same collector. Retry history and original failures are retained; missing +producer handshakes and unfinished work cannot pass. Required all-skipped +cohorts fail as incomplete, while explicitly optional cohorts retain skipped +status. Archive the whole canonical run directory alongside the per-platform +test artifacts with `if: always()` in CI. Logs can contain application output and test data; apply the same access and retention policy as other test evidence. Programmatic callers can inject `output` and `errorOutput` streams and set `verbose` on `DesktopStorybookCli` diff --git a/packages/agentic/storybook-desktop/config/owned-services.ps1 b/packages/agentic/storybook-desktop/config/owned-services.ps1 new file mode 100644 index 0000000000..8053779a79 --- /dev/null +++ b/packages/agentic/storybook-desktop/config/owned-services.ps1 @@ -0,0 +1,78 @@ +$desktopServicePath = & node -p "require.resolve('@fluentui-react-native/desktop-test/service')" +if ($LASTEXITCODE -ne 0) { throw 'Could not resolve the shared desktop service supervisor.' } +if (-not $env:FURN_DESKTOP_REPORT_CONTEXT) { throw 'Desktop smoke services require an inherited reporting context.' } +$desktopReportRoot = ($env:FURN_DESKTOP_REPORT_CONTEXT | ConvertFrom-Json).artifactsRoot +$desktopControlRoot = Join-Path $desktopReportRoot 'control' +New-Item -ItemType Directory -Path $desktopControlRoot -Force | Out-Null +$desktopServices = [System.Collections.Generic.List[object]]::new() + +function Start-OwnedCommand { + param( + [Parameter(Mandatory)][string]$FilePath, + [Parameter(Mandatory)][string[]]$ArgumentList, + [Parameter(Mandatory)][string]$LogName + ) + $id = [Guid]::NewGuid().ToString() + $descriptorPath = Join-Path $desktopControlRoot "$id.json" + $stopPath = Join-Path $desktopControlRoot "$id.stop" + $stderrPath = Join-Path $desktopControlRoot "$id.stderr.log" + @{ + command = @{ command = $FilePath; args = $ArgumentList; cwd = $projectRoot } + label = $LogName + stopPath = $stopPath + } | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $descriptorPath -Encoding utf8NoBOM + $service = Start-Process -FilePath 'node' -ArgumentList @("`"$desktopServicePath`"", "`"$descriptorPath`"") ` + -WorkingDirectory $projectRoot -PassThru -WindowStyle Hidden ` + -RedirectStandardOutput (Join-Path $desktopControlRoot "$id.stdout.log") -RedirectStandardError $stderrPath + $desktopServices.Add(@{ Process = $service; StopPath = $stopPath; Label = $LogName; StderrPath = $stderrPath }) + return $service +} + +function Assert-OwnedServices { + foreach ($service in $desktopServices) { + if ($service.Process.HasExited) { + $diagnostic = if (Test-Path -LiteralPath $service.StderrPath) { Get-Content -LiteralPath $service.StderrPath -Raw } else { '' } + throw "Required service '$($service.Label)' exited with code $($service.Process.ExitCode). $diagnostic" + } + } +} + +function Wait-ForTcpPort { + param([Parameter(Mandatory)][int]$Port, [int]$TimeoutSeconds = 120) + + $deadline = (Get-Date).AddSeconds($TimeoutSeconds) + $lastFailure = 'Connection attempt timed out.' + while ((Get-Date) -lt $deadline) { + Assert-OwnedServices + $client = [System.Net.Sockets.TcpClient]::new() + try { + $task = $client.ConnectAsync('127.0.0.1', $Port) + if ($task.Wait(500) -and $client.Connected) { return } + } catch { + $cause = $_.Exception.GetBaseException() + if ($cause -isnot [System.Net.Sockets.SocketException] -or + $cause.SocketErrorCode -notin @([System.Net.Sockets.SocketError]::ConnectionRefused, [System.Net.Sockets.SocketError]::TimedOut)) { throw } + $lastFailure = $cause.Message + } finally { $client.Dispose() } + Start-Sleep -Milliseconds 250 + } + throw "Timed out waiting for port $Port. Last connection failure: $lastFailure" +} + +function Stop-OwnedServices { + $failures = [System.Collections.Generic.List[System.Exception]]::new() + for ($index = $desktopServices.Count - 1; $index -ge 0; $index -= 1) { + $service = $desktopServices[$index] + try { + if (-not $service.Process.HasExited) { + Set-Content -LiteralPath $service.StopPath -Value 'stop' -Encoding utf8NoBOM + if (-not $service.Process.WaitForExit(15000)) { + Stop-Process -Id $service.Process.Id -ErrorAction Stop + throw "Owned service '$($service.Label)' did not complete its graceful reporting flush." + } + } + if ($service.Process.ExitCode -ne 0) { throw "Owned service '$($service.Label)' failed with exit code $($service.Process.ExitCode)." } + } catch { $failures.Add($_.Exception) } + } + if ($failures.Count) { throw [System.AggregateException]::new('Desktop service cleanup failed.', $failures) } +} diff --git a/packages/agentic/storybook-desktop/config/server.cjs b/packages/agentic/storybook-desktop/config/server.cjs index 8563be5e40..48ece3be13 100644 --- a/packages/agentic/storybook-desktop/config/server.cjs +++ b/packages/agentic/storybook-desktop/config/server.cjs @@ -15,94 +15,137 @@ async function startDesktopStorybookServer({ const requireFromProject = createRequire(path.join(projectRoot, 'package.json')); const { createChannelServer } = requireFromProject('@storybook/react-native/node'); - const server = createChannelServer({ - host, - port, - configPath, - websockets: true, - experimental_mcp: true, - keepNodeProcessAlive: true, - }); - - // eslint-disable-next-line no-console - console.log(`Storybook channel server listening: - WebSocket : ws://${host}:${port}/ - MCP : http://${host}:${port}/mcp`); - + const [{ startNativeDesktopService }, { createDesktopReportSession }] = await Promise.all([ + import('@fluentui-react-native/desktop-test'), + import('@fluentui-react-native/desktop-test/reporting'), + ]); + const report = await createDesktopReportSession({ artifactsRoot: path.join(projectRoot, 'artifacts', 'desktop'), kind: 'lifecycle' }); + let server; let driver; - if (driverManifestPath) { - if (!fs.existsSync(driverManifestPath)) { - throw new Error(`Desktop Driver manifest does not exist at ${driverManifestPath}.`); - } - const driverManifest = JSON.parse(fs.readFileSync(driverManifestPath, 'utf8')); + let keepAlive; + let closing; + const close = (error) => + (closing ??= (async () => { + process.off('SIGINT', stop); + process.off('SIGTERM', stop); + const failures = error === undefined ? [] : [error]; + try { + await driver?.close(); + } catch (failure) { + report.reporter.fail(failure, 'cleanup'); + failures.push(failure); + } + try { + if (server) { + for (const client of server.clients) client.terminate(); + await new Promise((resolve, reject) => server.close((failure) => (failure ? reject(failure) : resolve()))); + } + } catch (failure) { + report.reporter.fail(failure, 'cleanup'); + failures.push(failure); + } + clearInterval(keepAlive); + process.exitCode = await report.close( + failures.length ? new AggregateError(failures, 'Storybook channel service failed.') : undefined, + ); + })()); + const stop = () => { + void close().catch((error) => { + process.stderr.write(`${error.stack ?? error}\n`); + process.exitCode = 1; + }); + }; + const fatal = (error) => { + if (closing) return; + report.reporter.emit({ kind: 'service-failed', payload: { failure: report.reporter.failure(error) } }, 'storybook-channel'); + void close(error).catch((failure) => { + process.stderr.write(`${failure.stack ?? failure}\n`); + process.exitCode = 1; + }); + }; + try { + const driverManifest = driverManifestPath ? JSON.parse(fs.readFileSync(driverManifestPath, 'utf8')) : undefined; if ( - driverManifest.schemaVersion !== 2 || - !driverManifest.nativeDriver || - !driverManifest.application || - typeof driverManifest.application.leasePath !== 'string' || - typeof driverManifest.application.leaseNonce !== 'string' - ) { + driverManifest && + (driverManifest.schemaVersion !== 2 || + !driverManifest.nativeDriver || + !driverManifest.application || + typeof driverManifest.application.leasePath !== 'string' || + typeof driverManifest.application.leaseNonce !== 'string') + ) throw new Error(`Invalid native Desktop Driver manifest at ${driverManifestPath}.`); + if (driverManifest) { + report.reporter.addSecret(driverManifest.application.leaseNonce); + report.reporter.addSecret(driverManifest.bridgeNonce); } - const [{ NativeDesktopHost, createDesktopDriverServer }, { StorybookChannelOrchestrator }] = await Promise.all([ - import('@fluentui-react-native/desktop-driver'), - import('../lib/driver/index.js'), - ]); + // The upstream HTTP server is unref'd and closes on beforeExit; own only our clearable keepalive. + server = createChannelServer({ host, port, configPath, websockets: true, experimental_mcp: true, keepNodeProcessAlive: false }); if (!server) { throw new Error('Desktop Driver Storybook orchestration requires WebSockets.'); } - let targetHost; - let orchestrator; - if (process.env.STORYBOOK_NATIVE_DRIVER_FAKE === '1') { - const { createFakeStoryWindows, FakeDesktopHost, FakeStoryOrchestrator } = - await import('@fluentui-react-native/desktop-driver/testing'); - targetHost = new FakeDesktopHost({ + server.on('error', fatal); + keepAlive = setInterval(() => undefined, 60_000); + if (driverManifest) { + const { StorybookChannelOrchestrator } = await import('../lib/driver/index.js'); + const target = { endpoint: driverManifest.endpoint, + id: driverManifest.targetId, platformName: driverManifest.endpoint === 'macos' ? 'macos' : 'windows', + renderer: driverManifest.renderer, storyRootTestId: `${driverManifest.testIDPrefix}-story-root`, - windows: createFakeStoryWindows(driverManifest.storyManifest, `${driverManifest.testIDPrefix}-story-root`), - }); - orchestrator = new FakeStoryOrchestrator(driverManifest.storyManifest, targetHost); - } else { - targetHost = new NativeDesktopHost({ - application: driverManifest.application, - artifact: driverManifest.nativeDriver, - endpoint: driverManifest.endpoint, - onStderr: (message) => process.stderr.write(`[desktop-driver-native] ${message}`), - }); - orchestrator = createChannelOrchestrator({ - channelServer: server, - driverManifest, - serverUrl: loopbackUrl(host, port), - StorybookChannelOrchestrator, - }); - } - driver = await createDesktopDriverServer({ - host, - port: driverManifest.driverPort, - targets: [ - { + launchModes: ['attach'], + }; + if (process.env.STORYBOOK_NATIVE_DRIVER_FAKE === '1') { + const { createDesktopDriverServer } = await import('@fluentui-react-native/desktop-driver'); + const { createFakeStoryWindows, FakeDesktopHost, FakeStoryOrchestrator } = + await import('@fluentui-react-native/desktop-driver/testing'); + const targetHost = new FakeDesktopHost({ endpoint: driverManifest.endpoint, - host: targetHost, - id: driverManifest.targetId, - platformName: driverManifest.endpoint === 'macos' ? 'macos' : 'windows', - renderer: driverManifest.renderer, - storyRootTestId: `${driverManifest.testIDPrefix}-story-root`, - storyOrchestrator: orchestrator, - }, - ], - }); - console.log(` WebDriver: ${driver.url}/`); - const closeDriver = () => { - driver.close().finally(() => { - process.exit(); - }); - }; - process.once('SIGINT', closeDriver); - process.once('SIGTERM', closeDriver); + platformName: target.platformName, + storyRootTestId: target.storyRootTestId, + windows: createFakeStoryWindows(driverManifest.storyManifest, target.storyRootTestId), + }); + driver = await createDesktopDriverServer({ + host, + port: driverManifest.driverPort, + targets: [ + { ...target, host: targetHost, storyOrchestrator: new FakeStoryOrchestrator(driverManifest.storyManifest, targetHost) }, + ], + }); + } else { + driver = await startNativeDesktopService({ + application: driverManifest.application, + artifact: driverManifest.nativeDriver, + endpoint: driverManifest.endpoint, + host, + port: driverManifest.driverPort, + onStderr: (text) => + report.reporter.emit({ kind: 'output', payload: { stream: 'stderr', text } }, 'native-helper', { service: 'native-helper' }), + onFailure: fatal, + target: { + ...target, + storyOrchestrator: createChannelOrchestrator({ + channelServer: server, + driverManifest, + serverUrl: loopbackUrl(host, port), + StorybookChannelOrchestrator, + }), + }, + }); + } + } + report.reporter.emit( + { kind: 'service-ready', payload: { message: `Storybook ${loopbackUrl(host, port)}${driver ? `; WebDriver ${driver.url}` : ''}` } }, + 'storybook-channel', + ); + report.onStop?.(stop); + process.once('SIGINT', stop); + process.once('SIGTERM', stop); + return { channelServer: server, driver, close }; + } catch (error) { + await close(error); + throw error; } - - return { channelServer: server, driver }; } function createChannelOrchestrator({ channelServer, driverManifest, serverUrl, StorybookChannelOrchestrator }) { diff --git a/packages/agentic/storybook-desktop/config/smoke-win32.ps1 b/packages/agentic/storybook-desktop/config/smoke-win32.ps1 index 1d680380fd..f147424403 100644 --- a/packages/agentic/storybook-desktop/config/smoke-win32.ps1 +++ b/packages/agentic/storybook-desktop/config/smoke-win32.ps1 @@ -28,6 +28,7 @@ if ($SmokeMode -eq 'stories-and-tests' -and (-not $driverPort -or -not $env:STOR } New-Item -ItemType Directory -Path $logRoot -Force | Out-Null +. (Join-Path $PSScriptRoot 'owned-services.ps1') function Invoke-DesktopCli { param([Parameter(Mandatory)][string[]]$Arguments) @@ -38,21 +39,6 @@ function Invoke-DesktopCli { } } -function Start-OwnedCommand { - param( - [Parameter(Mandatory)] - [string]$FilePath, - [Parameter(Mandatory)] - [string[]]$ArgumentList, - [Parameter(Mandatory)] - [string]$LogName - ) - - return Start-Process -FilePath $FilePath -ArgumentList $ArgumentList -WorkingDirectory $projectRoot ` - -RedirectStandardOutput (Join-Path $logRoot "$LogName.out.log") ` - -RedirectStandardError (Join-Path $logRoot "$LogName.err.log") -PassThru -WindowStyle Hidden -} - function Write-ApplicationLease { param( [Parameter(Mandatory)] @@ -86,35 +72,12 @@ function Write-ApplicationLease { return $leasePath } -function Wait-ForTcpPort { - param( - [Parameter(Mandatory)] - [int]$Port, - [int]$TimeoutSeconds = 120 - ) - - $deadline = (Get-Date).AddSeconds($TimeoutSeconds) - while ((Get-Date) -lt $deadline) { - $client = [System.Net.Sockets.TcpClient]::new() - try { - $task = $client.ConnectAsync('127.0.0.1', $Port) - if ($task.Wait(500) -and $client.Connected) { - return - } - } finally { - $client.Dispose() - } - Start-Sleep -Milliseconds 250 - } - - throw "Timed out waiting for port $Port." -} - function Wait-ForApp { param([int]$TimeoutSeconds = 120) $deadline = (Get-Date).AddSeconds($TimeoutSeconds) while ((Get-Date) -lt $deadline) { + Assert-OwnedServices $process = Get-Process -Name ReactTest -ErrorAction SilentlyContinue | Where-Object { $_.MainWindowHandle -ne 0 -and $_.MainWindowTitle -eq $WindowTitle } | Select-Object -First 1 @@ -156,6 +119,7 @@ function Wait-ForAutomationId { $deadline = (Get-Date).AddSeconds($TimeoutSeconds) while ((Get-Date) -lt $deadline) { + Assert-OwnedServices if (Find-AutomationElement -Process $Process -AutomationId $AutomationId) { return } @@ -181,6 +145,7 @@ function Add-OwnedProcess { } } +$smokeFailure = $null try { if (Get-NetTCPConnection -State Listen -LocalPort $storybookPort -ErrorAction SilentlyContinue) { throw "Storybook port $storybookPort is already in use." @@ -247,7 +212,12 @@ try { if (-not (Get-Process -Id $appProcess.Id -ErrorAction SilentlyContinue)) { throw 'The Win32 Storybook host exited during the smoke test.' } +} catch { + $smokeFailure = $_.Exception + throw } finally { + $cleanupFailure = $null + try { Stop-OwnedServices } catch { $cleanupFailure = $_.Exception } if ($applicationLeasePath) { Remove-Item -LiteralPath $applicationLeasePath -Force -ErrorAction SilentlyContinue } @@ -259,5 +229,9 @@ try { Stop-Process -Id $ownedProcessId -ErrorAction Continue } } + if ($cleanupFailure) { + if ($smokeFailure) { throw [System.AggregateException]::new('Smoke execution and service cleanup failed.', [System.Exception[]]@($smokeFailure, $cleanupFailure)) } + throw $cleanupFailure + } } } diff --git a/packages/agentic/storybook-desktop/config/smoke-windows.ps1 b/packages/agentic/storybook-desktop/config/smoke-windows.ps1 index 1b09523935..a936a51469 100644 --- a/packages/agentic/storybook-desktop/config/smoke-windows.ps1 +++ b/packages/agentic/storybook-desktop/config/smoke-windows.ps1 @@ -29,44 +29,7 @@ if ($SmokeMode -eq 'stories-and-tests' -and (-not $driverPort -or -not $env:STOR New-Item -ItemType Directory -Path $logRoot -Force | Out-Null -function Start-OwnedCommand { - param( - [Parameter(Mandatory)] - [string]$FilePath, - [Parameter(Mandatory)] - [string[]]$ArgumentList, - [Parameter(Mandatory)] - [string]$LogName - ) - - return Start-Process -FilePath $FilePath -ArgumentList $ArgumentList -WorkingDirectory $projectRoot ` - -RedirectStandardOutput (Join-Path $logRoot "$LogName.out.log") ` - -RedirectStandardError (Join-Path $logRoot "$LogName.err.log") -PassThru -WindowStyle Hidden -} - -function Wait-ForTcpPort { - param( - [Parameter(Mandatory)] - [int]$Port, - [int]$TimeoutSeconds = 120 - ) - - $deadline = (Get-Date).AddSeconds($TimeoutSeconds) - while ((Get-Date) -lt $deadline) { - $client = [System.Net.Sockets.TcpClient]::new() - try { - $task = $client.ConnectAsync('127.0.0.1', $Port) - if ($task.Wait(500) -and $client.Connected) { - return - } - } finally { - $client.Dispose() - } - Start-Sleep -Milliseconds 250 - } - - throw "Timed out waiting for port $Port." -} +. (Join-Path $PSScriptRoot 'owned-services.ps1') function Get-PortOwner { param([Parameter(Mandatory)][int]$Port) @@ -83,6 +46,7 @@ function Wait-ForApp { $deadline = (Get-Date).AddSeconds($TimeoutSeconds) while ((Get-Date) -lt $deadline) { + Assert-OwnedServices $process = Get-Process -Name ReactApp -ErrorAction SilentlyContinue | Where-Object { $_.Id -notin $ExcludedProcessIds -and @@ -262,6 +226,7 @@ function Invoke-StorybookControl { } } +$smokeFailure = $null try { if (Get-NetTCPConnection -State Listen -LocalPort $storybookPort -ErrorAction SilentlyContinue) { throw "Storybook port $storybookPort is already in use." @@ -307,7 +272,12 @@ try { $applicationLeasePath = Write-ApplicationLease -Process $appProcess -LaunchTarget $launchTarget Invoke-StorybookControl -Phase 'tests' -Process $appProcess } +} catch { + $smokeFailure = $_.Exception + throw } finally { + $cleanupFailure = $null + try { Stop-OwnedServices } catch { $cleanupFailure = $_.Exception } if ($applicationLeasePath) { Remove-Item -LiteralPath $applicationLeasePath -Force -ErrorAction SilentlyContinue } @@ -319,5 +289,9 @@ try { Stop-Process -Id $ownedProcessId -ErrorAction Continue } } + if ($cleanupFailure) { + if ($smokeFailure) { throw [System.AggregateException]::new('Smoke execution and service cleanup failed.', [System.Exception[]]@($smokeFailure, $cleanupFailure)) } + throw $cleanupFailure + } } } diff --git a/packages/agentic/storybook-desktop/config/storybook-control.cjs b/packages/agentic/storybook-desktop/config/storybook-control.cjs index 927f49c515..40df0e989b 100644 --- a/packages/agentic/storybook-desktop/config/storybook-control.cjs +++ b/packages/agentic/storybook-desktop/config/storybook-control.cjs @@ -1,12 +1,15 @@ const fs = require('node:fs'); const path = require('node:path'); const { pathToFileURL } = require('node:url'); -const { formatDesktopStorybookError, writeDesktopStorybookFailure } = require('./diagnostics.cjs'); +const { writeDesktopStorybookFailure } = require('./diagnostics.cjs'); const host = process.env.STORYBOOK_WS_HOST || '127.0.0.1'; const port = Number(process.env.STORYBOOK_WS_PORT) || 7007; const smokeMode = process.env.STORYBOOK_SMOKE_MODE || 'stories'; const smokePhase = parseSmokePhase(process.argv.slice(2)); +const controller = new AbortController(); +let report; +let runner; function parseSmokePhase(args) { if (args.length === 0) { @@ -24,8 +27,11 @@ function baseUrl() { } async function request(pathname, options) { - const response = await fetch(`${baseUrl()}${pathname}`, options); - const body = await response.json().catch(() => ({})); + const response = await fetch(`${baseUrl()}${pathname}`, { + ...options, + signal: AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]), + }); + const body = await response.json(); if (!response.ok) { throw new Error(body.error || `Storybook server returned ${response.status}`); } @@ -35,6 +41,7 @@ async function request(pathname, options) { async function selectStory(storyId, attempts = 45) { let lastError; for (let attempt = 1; attempt <= attempts; attempt += 1) { + controller.signal.throwIfAborted(); try { return await request(`/select-story-sync/${encodeURIComponent(storyId)}`, { method: 'POST' }); } catch (error) { @@ -61,17 +68,45 @@ async function smoke() { const settleMilliseconds = Number(process.env.STORYBOOK_SMOKE_SETTLE_MS) || 0; const failFast = process.env.STORYBOOK_SMOKE_FAIL_FAST === '1'; const failures = []; + for (const { id } of entries) + report.reporter.emit({ kind: 'work-selected', payload: { workId: `navigation:${id}`, label: `Navigate ${id}` } }); for (const { id } of entries) { + const started = Date.now(); + const testId = `navigation:${id}`; + const title = `Navigate ${id}`; + const context = { story: id }; + report.reporter.emit({ kind: 'scope-start', payload: { label: title, category: 'phase' } }, testId, context, testId); + report.reporter.emit({ kind: 'test-start', payload: { testId, title, attempt: 0 } }, testId, context); try { await selectStory(id); if (settleMilliseconds > 0) { await new Promise((resolve) => setTimeout(resolve, settleMilliseconds)); } - process.stdout.write(`rendered ${id}\n`); + report.reporter.emit( + { kind: 'test-result', payload: { testId, title, attempt: 0, status: 'passed', durationMs: Date.now() - started } }, + testId, + context, + ); + report.reporter.emit({ kind: 'scope-end', payload: { status: 'passed' } }, testId); } catch (error) { - failures.push({ id, error: formatDesktopStorybookError(error) }); - writeDesktopStorybookFailure(`navigation "${id}"`, error); + failures.push(error); + report.reporter.emit( + { + kind: 'test-result', + payload: { + testId, + title, + attempt: 0, + status: 'failed', + durationMs: Date.now() - started, + failure: report.reporter.failure(error, 'test'), + }, + }, + testId, + context, + ); + report.reporter.emit({ kind: 'scope-end', payload: { status: 'failed' } }, testId); if (failFast) { break; } @@ -79,9 +114,9 @@ async function smoke() { } if (failures.length > 0) { - throw new Error(`${failures.length} of ${entries.length} stories failed to render: ${failures.map(({ id }) => id).join(', ')}`); + throw new AggregateError(failures, `${failures.length} of ${entries.length} stories failed to render.`); } - process.stdout.write(`Rendered ${entries.length} stories.\n`); + report.reporter.emit({ kind: 'diagnostic', payload: { level: 'info', message: `Rendered ${entries.length} stories.` } }); } if (smokeMode === 'stories-and-tests' && smokePhase !== 'stories') { @@ -117,11 +152,44 @@ async function runAuthoredTests() { manifest: manifest.storyManifest, platform: manifest.endpoint, targetId: manifest.targetId, + report: report.reporter, + commandRunner: runner, + signal: controller.signal, + errorOutput: { + write: (text) => { + report.reporter.emit({ kind: 'diagnostic', payload: { level: 'warning', message: String(text) } }); + return true; + }, + }, }); - process.stdout.write(`${formatDesktopStorybookSmokeTestSummary(result)}\n`); + report.reporter.emit({ kind: 'diagnostic', payload: { level: 'info', message: formatDesktopStorybookSmokeTestSummary(result) } }); +} + +async function main() { + const [{ createDesktopReportSession }, { NodeDesktopCommandRunner }] = await Promise.all([ + import('@fluentui-react-native/desktop-test/reporting'), + import('@fluentui-react-native/desktop-test/commands'), + ]); + report = await createDesktopReportSession({ artifactsRoot: path.join(process.cwd(), 'artifacts', 'desktop'), kind: 'tests' }); + runner = new NodeDesktopCommandRunner({ reporter: report.reporter, environment: report.environment }); + report.onStop?.(() => controller.abort(new Error('Storybook control interrupted.'))); + let failure; + try { + await smoke(); + } catch (error) { + failure = error; + } finally { + try { + await runner.close(); + } catch (error) { + report.reporter.fail(error, 'cleanup'); + failure = failure === undefined ? error : new AggregateError([failure, error], 'Storybook control and cleanup failed.'); + } + process.exitCode = await report.close(failure); + } } -smoke().catch((error) => { +main().catch((error) => { writeDesktopStorybookFailure('smoke', error); process.exitCode = 1; }); diff --git a/packages/agentic/storybook-desktop/jest.config.cjs b/packages/agentic/storybook-desktop/jest.config.cjs index d6535664e2..8f56587ad6 100644 --- a/packages/agentic/storybook-desktop/jest.config.cjs +++ b/packages/agentic/storybook-desktop/jest.config.cjs @@ -2,6 +2,7 @@ const config = require('@fluentui-react-native/scripts/jest-config'); module.exports = { ...config, + transformIgnorePatterns: ['node_modules[/\\\\](?!.*(?:\\.mjs$|@rnx-kit(?:[/\\\\]reporter|-reporter-npm-)))'], moduleNameMapper: { ...config.moduleNameMapper, '^(\\.{1,2}/.*)\\.js$': '$1', diff --git a/packages/agentic/storybook-desktop/package.json b/packages/agentic/storybook-desktop/package.json index d9ec6b33df..b3743a10c6 100644 --- a/packages/agentic/storybook-desktop/package.json +++ b/packages/agentic/storybook-desktop/package.json @@ -27,7 +27,6 @@ "./babel": "./config/babel.cjs", "./config": { "types": "./lib/config/index.d.ts", - "import": "./lib/config/index.js", "default": "./src/config/index.ts" }, "./driver": { @@ -59,6 +58,7 @@ "@babel/core": "catalog:", "@babel/plugin-transform-typescript": "catalog:", "@fluentui-react-native/desktop-driver": "workspace:*", + "@fluentui-react-native/desktop-test": "workspace:*", "@rnx-kit/tools-react-native": "catalog:", "@wdio/logger": "catalog:", "commander": "^14.0.2", diff --git a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts index 5dd87cd1c8..7bf3787f5f 100644 --- a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts +++ b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts @@ -105,6 +105,7 @@ class RecordingRunner implements DesktopCommandRunner { readonly background: PreparedDesktopCommand[] = []; stopped = 0; failCommand?: string; + exitService?: string; async run(command: PreparedDesktopCommand): Promise { this.foreground.push(command); @@ -116,7 +117,7 @@ class RecordingRunner implements DesktopCommandRunner { start(command: PreparedDesktopCommand): RunningDesktopCommand { this.background.push(command); return { - completed: new Promise(() => {}), + completed: command.context?.service === this.exitService && this.exitService ? Promise.resolve(7) : new Promise(() => {}), stop: async () => { this.stopped += 1; }, @@ -321,14 +322,19 @@ describe('DesktopStorybookCli', () => { output: { write: () => true }, runner: new RecordingRunner(), }; + options.runner.exitService = 'storybook-channel'; const manifestPath = path.join(storybookRoot, 'storybook-desktop.generated', 'driver-manifest.macos.json'); try { - await new DesktopStorybookCli(makeConfig(), options).driver('macos'); + await expect(new DesktopStorybookCli(makeConfig(), options).driver('macos')).rejects.toThrow('exited unexpectedly'); const fabricManifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); - await new DesktopStorybookCli(makeConfig(), { ...options, paper: true }).driver('macos'); + await expect(new DesktopStorybookCli(makeConfig(), { ...options, paper: true }).driver('macos')).rejects.toThrow( + 'exited unexpectedly', + ); const paperManifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); - await new DesktopStorybookCli(makeConfig(), { ...options, paper: true }).driver('macos'); + await expect(new DesktopStorybookCli(makeConfig(), { ...options, paper: true }).driver('macos')).rejects.toThrow( + 'exited unexpectedly', + ); const repeatedPaperManifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); expect(fabricManifest.renderer).toBe('fabric'); @@ -685,6 +691,8 @@ describe('DesktopStorybookCli', () => { driverUrl: `http://127.0.0.1:${cli.instance.driverPort}`, manifest: await createEmptyStoryManifest(cli.config, 'macos'), platform: 'macos', + report: undefined, + signal: expect.any(AbortSignal), targetId: 'agenticstorybook-macos', }); expect(fetch.mock.calls.map(([input]) => input.toString())).toContain( @@ -857,6 +865,7 @@ describe('createDesktopStorybookCommand', () => { test('starts the channel and embedded driver from one server command', async () => { const runner = new RecordingRunner(); + runner.exitService = 'storybook-channel'; const program = createDesktopStorybookCommand({ ...nativeDriverTestOptions, config: makeConfig(), @@ -867,9 +876,11 @@ describe('createDesktopStorybookCommand', () => { const manifestPath = path.join(storybookRoot, 'storybook-desktop.generated', 'driver-manifest.win32.json'); try { - await program.parseAsync(['node', 'test', 'driver', '--win32', '--host', 'localhost', '--port', '7102']); + await expect(program.parseAsync(['node', 'test', 'driver', '--win32', '--host', 'localhost', '--port', '7102'])).rejects.toThrow( + 'exited unexpectedly', + ); - expect(runner.foreground[0]).toMatchObject({ + expect(runner.background[1]).toMatchObject({ args: [path.resolve(storybookRoot, '../../packages/agentic/storybook-desktop/config/server-runner.cjs')], env: { [FURN_STORYBOOK_PLATFORM]: 'win32', @@ -892,6 +903,7 @@ describe('createDesktopStorybookCommand', () => { }); const secondRunner = new RecordingRunner(); + secondRunner.exitService = 'storybook-channel'; const secondProgram = createDesktopStorybookCommand({ ...nativeDriverTestOptions, config: makeConfig(), @@ -899,7 +911,9 @@ describe('createDesktopStorybookCommand', () => { isPortAvailable: async () => true, runner: secondRunner, }); - await secondProgram.parseAsync(['node', 'test', 'driver', '--win32', '--host', 'localhost', '--port', '7102']); + await expect( + secondProgram.parseAsync(['node', 'test', 'driver', '--win32', '--host', 'localhost', '--port', '7102']), + ).rejects.toThrow('exited unexpectedly'); const secondManifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')) as Record; expect(secondManifest.bridgeNonce).toBe(firstManifest.bridgeNonce); } finally { diff --git a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts index 4dd4d310be..dd102b65cf 100644 --- a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts +++ b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts @@ -2,6 +2,8 @@ import { spawn } from 'node:child_process'; import fs from 'node:fs'; import { createServer } from 'node:net'; import path from 'node:path'; +import { DesktopServiceScope } from '@fluentui-react-native/desktop-test'; +import type { DesktopReportWriter } from '@fluentui-react-native/desktop-test/reporting'; import { writeDesktopStorybookFailure, type DesktopStorybookErrorOutput } from '../../config/diagnostics.cjs'; import { @@ -65,6 +67,8 @@ export type DesktopStorybookCliOptions = { runSmokeTests?: typeof runDesktopStorybookSmokeTests; resolveNativeDriver?: typeof resolveNativeDesktopDriver; writeMacOSApplicationLease?: typeof writeMacOSApplicationLease; + reporter?: DesktopReportWriter; + services?: DesktopServiceScope; }; export type DesktopStorybookServerOptions = { @@ -94,10 +98,13 @@ export class DesktopStorybookCli { private readonly runSmokeTests: typeof runDesktopStorybookSmokeTests; private readonly resolveNativeDriver: typeof resolveNativeDesktopDriver; private readonly writeMacOSApplicationLease: typeof writeMacOSApplicationLease; + private readonly reporter?: DesktopReportWriter; + private readonly services: DesktopServiceScope; constructor(config: DesktopStorybookConfig, options: DesktopStorybookCliOptions = {}) { this.config = config; - this.paper = options.paper ?? false; + this.reporter = options.reporter; + this.paper = options.paper ?? config.getDesktopTarget?.('macos')?.host.renderer === 'paper'; this.instance = createDesktopStorybookInstance({ projectRoot: config.projectRoot, bundleIdentifierPrefix: config.macosBundleIdentifier, @@ -105,15 +112,17 @@ export class DesktopStorybookCli { this.runner = options.runner ?? new NodeDesktopCommandRunner({ + reporter: options.reporter, output: options.output, errorOutput: options.errorOutput, verbose: options.verbose, }); + this.services = options.services ?? new DesktopServiceScope(this.runner, this.reporter); this.buildNativeDriver = options.buildNativeDriver ?? buildNativeDesktopDriver; this.createStoryManifest = options.createStoryManifest ?? createDesktopStoryManifest; this.fetch = options.fetch ?? globalThis.fetch; - this.output = options.output ?? process.stdout; - this.errorOutput = options.errorOutput ?? process.stderr; + this.output = options.output ?? reportOutput(this.reporter, 'info', process.stdout); + this.errorOutput = options.errorOutput ?? reportOutput(this.reporter, 'warning', process.stderr); this.verbose = options.verbose ?? process.env[STORYBOOK_VERBOSE] === '1'; this.isPortAvailable = options.isPortAvailable ?? isLoopbackPortAvailable; this.runSmokeTests = options.runSmokeTests ?? runDesktopStorybookSmokeTests; @@ -159,25 +168,29 @@ export class DesktopStorybookCli { }; resolvedInstance.driverManifestPath = await this.writeDriverManifest(platform, resolvedInstance, nativeDriver); this.output.write(`Storybook instance ${resolvedInstance.id}: channel=${storybookPort}, metro=${metroPort}, driver=${driverPort}\n`); - const metro = this.runner.start(this.prepareCommand(defaultMetroCommand(resolvedInstance), platform, resolvedInstance)); + this.services.start('metro', this.prepareCommand(defaultMetroCommand(resolvedInstance), platform, resolvedInstance)); const failures: unknown[] = []; try { - await this.executePlan( - { - ...command, - env: { - ...command.env, - ...(options.host ? { STORYBOOK_WS_HOST: options.host } : {}), + this.services.start( + 'storybook-channel', + this.prepareCommand( + { + ...command, + env: { + ...command.env, + ...(options.host ? { STORYBOOK_WS_HOST: options.host } : {}), + }, }, - }, - platform, - resolvedInstance, + platform, + resolvedInstance, + ), ); + await this.services.guard(new Promise(() => undefined)); } catch (error) { failures.push(error); } try { - await metro.stop({ failed: failures.length > 0 }); + await this.services.close(); } catch (error) { failures.push(error); } @@ -232,7 +245,16 @@ export class DesktopStorybookCli { ); } const results = await runWdioStoryTests( - { ...settings, ...connection, config: this.config, manifest, platform, errorOutput: this.errorOutput }, + { + ...settings, + ...connection, + config: this.config, + manifest, + platform, + errorOutput: this.errorOutput, + report: this.reporter, + signal: this.services.controller.signal, + }, this.runner, ); this.output.write( @@ -366,7 +388,6 @@ export class DesktopStorybookCli { instance: ResolvedDesktopStorybookInstance, mode: DesktopSmokeMode, ): Promise { - const backgroundCommands: RunningDesktopCommand[] = []; const failures: unknown[] = []; let primaryFailure: unknown; const serverUrl = smoke.serverUrl ?? loopbackUrl(instance.storybookPort); @@ -379,7 +400,8 @@ export class DesktopStorybookCli { const readiness: Promise[] = []; if (server) { - const runningServer = this.runner.start( + const runningServer = this.services.start( + 'storybook-channel', this.prepareCommand( { ...server, @@ -392,7 +414,6 @@ export class DesktopStorybookCli { instance, ), ); - backgroundCommands.push(runningServer); readiness.push(this.waitForUrl(new URL('/index.json', serverUrl).href, runningServer, smoke.startupTimeoutMs)); if (mode === 'stories-and-tests') { readiness.push(this.waitForUrl(loopbackUrl(instance.driverPort, '/status'), runningServer, smoke.startupTimeoutMs)); @@ -401,12 +422,11 @@ export class DesktopStorybookCli { throw new Error('The stories-and-tests smoke mode requires the Storybook server and embedded Desktop Driver.'); } if (metro) { - const runningMetro = this.runner.start(this.prepareCommand(metro, platform, instance)); - backgroundCommands.push(runningMetro); + const runningMetro = this.services.start('metro', this.prepareCommand(metro, platform, instance)); readiness.push(this.waitForUrl(metroUrl, runningMetro, smoke.startupTimeoutMs)); } - await Promise.all(readiness); + await this.services.guard(Promise.all(readiness)); await this.executeAction('run', platform, instance); if (mode === 'stories-and-tests' && platform === 'macos') { if (!instance.driverManifestPath) { @@ -414,33 +434,37 @@ export class DesktopStorybookCli { } await this.writeMacOSApplicationLease(instance.driverManifestPath, smoke.startupTimeoutMs ?? 120_000); } - await this.renderEveryStory(serverUrl, smoke.settleMs ?? 0, smoke.initialRenderTimeoutMs ?? smoke.startupTimeoutMs); + await this.services.guard( + this.renderEveryStory(serverUrl, smoke.settleMs ?? 0, smoke.initialRenderTimeoutMs ?? smoke.startupTimeoutMs), + ); if (mode === 'stories-and-tests') { - const result = await this.runSmokeTests({ - config: this.config, - commandRunner: this.runner, - errorOutput: this.errorOutput, - driverUrl: loopbackUrl(instance.driverPort), - manifest: await this.createStoryManifest(this.config, platform), - platform, - targetId: `${this.config.appName}-${platform}`.toLowerCase(), - }); + const result = await this.services.guard( + this.runSmokeTests({ + config: this.config, + commandRunner: this.runner, + errorOutput: this.errorOutput, + driverUrl: loopbackUrl(instance.driverPort), + manifest: await this.createStoryManifest(this.config, platform), + platform, + targetId: `${this.config.appName}-${platform}`.toLowerCase(), + report: this.reporter, + signal: this.services.controller.signal, + }), + ); this.output.write(`${formatDesktopStorybookSmokeTestSummary(result)}\n`); } } catch (error) { primaryFailure = error; } finally { try { - await this.executePlan(smoke.stop!, platform, instance); + await this.executePlan(smoke.stop!, platform, instance, true); } catch (error) { failures.push(error); } - for (const backgroundCommand of backgroundCommands.reverse()) { - try { - await backgroundCommand.stop({ failed: primaryFailure !== undefined }); - } catch (error) { - failures.push(error); - } + try { + await this.services.close(); + } catch (error) { + failures.push(error); } } @@ -555,8 +579,15 @@ export class DesktopStorybookCli { } const failures: Error[] = []; + for (const { id } of entries) + this.reporter?.emit({ kind: 'work-selected', payload: { workId: `navigation:${id}`, label: `Render ${id}` } }); let consecutiveFailures = 0; for (const [entryIndex, { id }] of entries.entries()) { + this.services.controller.signal.throwIfAborted(); + const started = Date.now(); + const testId = `navigation:${id}`; + this.reporter?.emit({ kind: 'scope-start', payload: { label: `Render ${id}`, category: 'worker' } }, testId, { story: id }, testId); + this.reporter?.emit({ kind: 'test-start', payload: { testId, title: `Render ${id}`, attempt: 0 } }, testId, { story: id }); try { await this.selectStory(serverUrl, id!, entryIndex === 0 ? initialRenderTimeoutMs : 15_000); consecutiveFailures = 0; @@ -564,7 +595,32 @@ export class DesktopStorybookCli { await delay(settleMs); } this.output.write(`rendered ${id}\n`); + this.reporter?.emit( + { + kind: 'test-result', + payload: { testId, title: `Render ${id}`, attempt: 0, status: 'passed', durationMs: Date.now() - started }, + }, + testId, + { story: id }, + ); + this.reporter?.emit({ kind: 'scope-end', payload: { status: 'passed' } }, testId); } catch (error) { + this.reporter?.emit( + { + kind: 'test-result', + payload: { + testId, + title: `Render ${id}`, + attempt: 0, + status: 'failed', + durationMs: Date.now() - started, + failure: this.reporter.failure(error, 'test'), + }, + }, + testId, + { story: id }, + ); + this.reporter?.emit({ kind: 'scope-end', payload: { status: 'failed' } }, testId); writeDesktopStorybookFailure(`navigation "${id}"`, error, this.errorOutput); failures.push(new Error(`Story "${id}" failed to render: ${errorMessage(error)}`, { cause: error })); consecutiveFailures += 1; @@ -584,6 +640,7 @@ export class DesktopStorybookCli { const deadline = Date.now() + timeoutMs; let lastError: unknown; do { + this.services.controller.signal.throwIfAborted(); try { await this.getJson( new URL(`/select-story-sync/${encodeURIComponent(storyId)}`, serverUrl), @@ -644,15 +701,22 @@ export class DesktopStorybookCli { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), timeoutMs); try { - return await this.fetch(input, { ...init, signal: controller.signal }); + return await this.fetch(input, { ...init, signal: AbortSignal.any([controller.signal, this.services.controller.signal]) }); } finally { clearTimeout(timeout); } } - private async executePlan(plan: DesktopCommandPlan, platform: Platforms, instance?: ResolvedDesktopStorybookInstance): Promise { + private async executePlan( + plan: DesktopCommandPlan, + platform: Platforms, + instance?: ResolvedDesktopStorybookInstance, + cleanup = false, + ): Promise { for (const command of Array.isArray(plan) ? plan : [plan]) { - await this.runner.run(this.prepareCommand(command, platform, instance)); + const work = this.runner.run(this.prepareCommand(command, platform, instance)); + if (cleanup) await work; + else await this.services.guard(work); } } @@ -686,6 +750,21 @@ export class DesktopStorybookCli { } } +function reportOutput( + reporter: DesktopReportWriter | undefined, + level: 'info' | 'warning', + fallback: Pick, +): Pick { + if (!reporter) return fallback; + return { + write(chunk: string | Uint8Array) { + const message = String(chunk).trim(); + if (message) reporter.emit({ kind: 'diagnostic', payload: { level, message } }, 'storybook'); + return true; + }, + }; +} + function readReusableBridgeNonce( manifestPath: string, instance: DesktopStorybookInstance, @@ -940,7 +1019,7 @@ function loopbackUrl(port: number, pathname = ''): string { function unsupportedAction(action: DesktopStorybookAction, platform: Platforms): Error { return new Error( - `Desktop Storybook ${action} is not configured for ${platform}. Set platformOptions.${platform}.${action} in storybook.config.ts.`, + `Desktop Storybook ${action} is not configured for ${platform}. Set platformOptions.${platform}.${action} in your Storybook integration options.`, ); } diff --git a/packages/agentic/storybook-desktop/src/cli/README.md b/packages/agentic/storybook-desktop/src/cli/README.md index 6fe4534865..9b5fa8408b 100644 --- a/packages/agentic/storybook-desktop/src/cli/README.md +++ b/packages/agentic/storybook-desktop/src/cli/README.md @@ -65,10 +65,12 @@ the syntax appropriate for that environment. A `--config ` option may be placed before the subcommand when the configuration does not use a standard root filename. -Use `--verbose` before the subcommand to replay complete output from -successful commands. By default, command output is captured continuously in -`artifacts/storybook-commands`; failures replay the ordered stdout/stderr log, -and successful commands emit short summaries. Navigation and test failures +Use `--verbose` before the subcommand to also stream informational output and +replay successful commands. Stderr and test progress are live by default. +Command output is captured continuously in `artifacts/desktop//commands`; +failures replay source-attributed logs in collector arrival order. +The same run directory holds the event journal, JSON result, and JUnit report. +Navigation and test failures also emit labeled stderr diagnostics with story/test IDs and underlying causes. See [pipeline logging](../../README.md#command-logs-and-pipeline-failures). @@ -130,7 +132,7 @@ Metro process is required. ## End-to-end tests Use `yarn storybook test --macos --story 'components-button--*'` for executable -story callbacks. It uses the app's `wdio` settings in `storybook.config.mts` +story callbacks. It uses the Storybook integration's `wdio` settings in `desktop.config.mts` and Node's built-in test runner, not a separate WDIO configuration. Start `storybook driver` and launch the app first; `--list` needs neither. For multiple cases, use a named function collection under the story's `wdio` diff --git a/packages/agentic/storybook-desktop/src/cli/commandRunner.test.ts b/packages/agentic/storybook-desktop/src/cli/commandRunner.test.ts index bf0879a0bc..db79b2fe0e 100644 --- a/packages/agentic/storybook-desktop/src/cli/commandRunner.test.ts +++ b/packages/agentic/storybook-desktop/src/cli/commandRunner.test.ts @@ -1,122 +1,67 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; -import { Writable } from 'node:stream'; - +import { NodeDesktopCommandRunner as SharedRunner } from '@fluentui-react-native/desktop-test/commands'; import { NodeDesktopCommandRunner, type PreparedDesktopCommand } from './commandRunner.js'; -function capture() { - const chunks: Buffer[] = []; - return { - write: jest.fn((chunk: string | Uint8Array) => { - chunks.push(Buffer.from(chunk)); - return true; - }), - text: () => Buffer.concat(chunks).toString('utf8'), - }; -} - -describe('ordered command output', () => { +describe('shared command output', () => { let root: string; beforeEach(() => { root = fs.mkdtempSync(path.join(os.tmpdir(), 'storybook-command-output-')); }); afterEach(() => fs.rmSync(root, { recursive: true, force: true })); - function command(source: string, label = 'fixture command'): PreparedDesktopCommand { - return { command: process.execPath, args: ['-e', source], cwd: root, env: {}, label }; + function command(source: string): PreparedDesktopCommand { + return { command: process.execPath, args: ['-e', source], cwd: root, env: {}, label: 'fixture command' }; } - const mixedOutput = "process.stdout.write('out-1\\n'); process.stderr.write('err-1\\n'); process.stdout.write('out-2\\n');"; - - test('writes both streams continuously to one log and suppresses successful command output', async () => { - const output = capture(); - const errorOutput = capture(); - const runner = new NodeDesktopCommandRunner({ output, errorOutput, verbose: false }); - await runner.run(command(mixedOutput)); - const directory = path.join(root, 'artifacts/storybook-commands'); - const files = fs.readdirSync(directory); - expect(files).toHaveLength(1); - expect(fs.readFileSync(path.join(directory, files[0]), 'utf8')).toContain('out-1\nerr-1\nout-2\n'); - expect(output.text()).toContain('[storybook] OK fixture command'); - expect(output.text()).not.toContain('out-1\n'); - expect(errorOutput.text()).toBe(''); - }); - test('replays the complete ordered log to stderr on failure exactly once', async () => { - const output = capture(); - const errorOutput = capture(); - const runner = new NodeDesktopCommandRunner({ output, errorOutput, verbose: false }); - await expect(runner.run(command(`${mixedOutput} process.exitCode = 7;`))).rejects.toThrow('exited with code 7'); - expect(errorOutput.text()).toContain('out-1\nerr-1\nout-2\n'); - expect(errorOutput.text()).toContain('[storybook] FAIL fixture command (exit 7; log:'); - expect(errorOutput.text().match(/\[storybook\] BEGIN/g)).toHaveLength(1); - expect(output.text()).not.toContain('[storybook] OK'); + test('uses the neutral command runner rather than a Storybook lifecycle copy', () => { + expect(NodeDesktopCommandRunner).toBe(SharedRunner); }); - test('verbose mode replays successful command output', async () => { - const output = capture(); - const runner = new NodeDesktopCommandRunner({ output, errorOutput: capture(), verbose: true }); - await runner.run(command(mixedOutput)); - expect(output.text()).toContain('out-1\nerr-1\nout-2\n'); - expect(output.text()).toContain('[storybook] BEGIN fixture command'); - expect(output.text()).toContain('[storybook] END fixture command'); - expect(output.text()).toContain('[storybook] OK fixture command'); - }); - - test('retains live logs and replays them when an owner stops a command after failure', async () => { - const output = capture(); - const errorOutput = capture(); - const runner = new NodeDesktopCommandRunner({ output, errorOutput, verbose: false }); - const running = runner.start(command("process.stdout.write('live-log\\n'); setInterval(() => {}, 1000);")); + test('retains attributed logs and forwards stderr without verbose', async () => { + const stdout: string[] = []; + const stderr: string[] = []; + const runner = new NodeDesktopCommandRunner({ + output: { + write: (text) => { + stdout.push(String(text)); + return true; + }, + }, + errorOutput: { + write: (text) => { + stderr.push(String(text)); + return true; + }, + }, + }); try { - const deadline = Date.now() + 5000; - while (!fs.readFileSync(running.logPath!, 'utf8').includes('\nlive-log\n')) { - if (Date.now() > deadline) throw new Error('The running command did not write its live output.'); - await new Promise((resolve) => setTimeout(resolve, 10)); - } - expect(output.text()).not.toContain('live-log\n'); - await running.stop({ failed: true }); - expect(errorOutput.text()).toContain('\nlive-log\n'); - expect(errorOutput.text()).toContain('[storybook] FAIL fixture command (stopped; log:'); + await runner.run(command("process.stdout.write('information\\n');process.stderr.write('diagnostic\\n');")); } finally { - await running.stop(); + await runner.close(); } + expect(stderr.join('')).toContain('stderr: diagnostic'); + expect(stdout.join('')).not.toContain('stdout: information'); + const runs = path.join(root, 'artifacts', 'desktop-commands'); + const run = path.join(runs, fs.readdirSync(runs)[0]); + const log = path.join(run, 'commands', fs.readdirSync(path.join(run, 'commands'))[0]); + expect(fs.readFileSync(log, 'utf8')).toContain('[stdout] information'); + expect(fs.readFileSync(log, 'utf8')).toContain('[stderr] diagnostic'); + expect(JSON.parse(fs.readFileSync(path.join(run, 'run.json'), 'utf8')).status).toBe('passed'); }); - test('does not interleave replay groups from concurrent commands', async () => { - const chunks: Buffer[] = []; - const errorOutput = new Writable({ - highWaterMark: 1, - write(chunk, _encoding, callback) { - setTimeout(() => { - chunks.push(Buffer.from(chunk)); - callback(); - }, 1); - }, - }); - const runner = new NodeDesktopCommandRunner({ output: capture(), errorOutput, verbose: false }); - await Promise.allSettled([ - runner.run(command("process.stdout.write('first-out\\n'); process.stderr.write('first-err\\n'); process.exitCode=1;", 'first')), - runner.run(command("process.stdout.write('second-out\\n'); process.stderr.write('second-err\\n'); process.exitCode=1;", 'second')), - ]); - const log = Buffer.concat(chunks).toString('utf8'); - for (const [label, other] of [ - ['first', 'second'], - ['second', 'first'], - ]) { - const group = log.slice(log.indexOf(`[storybook] BEGIN ${label}\n`), log.indexOf(`[storybook] END ${label}\n`)); - expect(group).toContain(`${label}-out\n${label}-err\n`); - expect(group).not.toContain(`${other}-out\n`); + test('keeps the original command failure in the shared result', async () => { + const runner = new NodeDesktopCommandRunner({ output: { write: () => true }, errorOutput: { write: () => true } }); + try { + await expect(runner.run(command('process.exitCode=7;'))).rejects.toThrow('exited with code 7'); + } finally { + await runner.close(); } - errorOutput.destroy(); - }); - - test('reports process startup errors and keeps the command log', async () => { - const errorOutput = capture(); - const runner = new NodeDesktopCommandRunner({ output: capture(), errorOutput, verbose: false }); - await expect(runner.run({ ...command(''), command: path.join(root, 'missing-executable') })).rejects.toThrow(); - expect(errorOutput.text()).toContain('[storybook] FAIL fixture command'); - expect(errorOutput.text()).toMatch(/ENOENT|not recognized|cannot find/i); - expect(fs.readdirSync(path.join(root, 'artifacts/storybook-commands'))).toHaveLength(1); + const runs = path.join(root, 'artifacts', 'desktop-commands'); + const result = JSON.parse(fs.readFileSync(path.join(runs, fs.readdirSync(runs)[0], 'run.json'), 'utf8')); + expect(result.status).toBe('failed'); + expect(result.failures).toHaveLength(1); + expect(result.failures[0].error.message).toContain('exited with code 7'); }); }); diff --git a/packages/agentic/storybook-desktop/src/cli/commandRunner.ts b/packages/agentic/storybook-desktop/src/cli/commandRunner.ts index 78c27ad004..613f7009cd 100644 --- a/packages/agentic/storybook-desktop/src/cli/commandRunner.ts +++ b/packages/agentic/storybook-desktop/src/cli/commandRunner.ts @@ -1,266 +1,9 @@ -import { spawn } from 'node:child_process'; -import { randomUUID } from 'node:crypto'; -import { once } from 'node:events'; -import fs from 'node:fs'; -import path from 'node:path'; -import { Writable } from 'node:stream'; - -import type { DesktopCommand } from '../config/commands.js'; -import { formatDesktopStorybookError, type DesktopStorybookErrorOutput } from '../../config/diagnostics.cjs'; +export { NodeDesktopCommandRunner } from '@fluentui-react-native/desktop-test/commands'; +export type { + DesktopCommandRunner, + NodeDesktopCommandRunnerOptions, + PreparedDesktopCommand, + RunningDesktopCommand, +} from '@fluentui-react-native/desktop-test/commands'; export const STORYBOOK_VERBOSE = 'STORYBOOK_VERBOSE'; - -export type PreparedDesktopCommand = DesktopCommand & { - args: readonly string[]; - cwd: string; - env: Readonly>; - label?: string; -}; - -export type RunningDesktopCommand = { - completed: Promise; - logPath?: string; - stop(options?: { failed?: boolean }): Promise; -}; - -export type NodeDesktopCommandRunnerOptions = { - errorOutput?: DesktopStorybookErrorOutput; - logDirectory?: string; - output?: Pick; - verbose?: boolean; -}; - -export interface DesktopCommandRunner { - run(command: PreparedDesktopCommand): Promise; - start(command: PreparedDesktopCommand): RunningDesktopCommand; -} - -export class NodeDesktopCommandRunner implements DesktopCommandRunner { - private readonly output: Pick; - private readonly errorOutput: DesktopStorybookErrorOutput; - private readonly verbose: boolean; - private readonly logDirectory?: string; - private reportQueue = Promise.resolve(); - - constructor(options: NodeDesktopCommandRunnerOptions = {}) { - this.output = options.output ?? process.stdout; - this.errorOutput = options.errorOutput ?? process.stderr; - this.verbose = options.verbose ?? process.env[STORYBOOK_VERBOSE] === '1'; - this.logDirectory = options.logDirectory; - } - - async run(command: PreparedDesktopCommand): Promise { - const running = this.launch(command, false); - let failure: unknown; - try { - const exitCode = await running.completed; - if (exitCode !== 0) { - throw new Error(`${formatCommandFailure(command, exitCode)} Log: ${running.logPath}`); - } - } catch (error) { - failure = error; - } - try { - await running.stop({ failed: failure !== undefined }); - } catch (error) { - throw new AggregateError(failure === undefined ? [error] : [failure, error], 'Command execution or log reporting failed.'); - } - if (failure !== undefined) { - throw failure; - } - } - - start(command: PreparedDesktopCommand): RunningDesktopCommand { - return this.launch(command, true); - } - - private launch(command: PreparedDesktopCommand, background: boolean): RunningDesktopCommand { - const directory = this.logDirectory ?? path.join(command.cwd, 'artifacts', 'storybook-commands'); - fs.mkdirSync(directory, { recursive: true }); - const logPath = path.join(directory, `${new Date().toISOString().replaceAll(':', '-')}-${randomUUID()}.log`); - const label = command.label ?? [command.command, ...command.args].join(' '); - const descriptor = fs.openSync(logPath, 'ax', 0o600); - let child: ReturnType; - try { - fs.writeSync(descriptor, `$ ${[command.command, ...command.args].map((argument) => JSON.stringify(argument)).join(' ')}\n`); - // A shared descriptor preserves stdout/stderr write order without buffering a whole command in memory. - child = spawnCommand(command, background, descriptor); - } finally { - fs.closeSync(descriptor); - } - const completion = completed(child); - let stopRequested = false; - let stopPromise: Promise | undefined; - let exitError: unknown; - let reportedFailure = false; - const reports: Promise[] = []; - const enqueue = (action: () => Promise | void) => { - const report = this.reportQueue.then(action); - this.reportQueue = report.catch(() => undefined); - reports.push(report); - }; - enqueue(() => writeOutput(this.output, `[storybook] START ${label} (log: ${logPath})\n`)); - - const report = (failed: boolean, status: string) => { - if (reportedFailure) { - return; - } - reportedFailure = failed; - enqueue(async () => { - const output = failed ? this.errorOutput : this.output; - if (failed || this.verbose) { - await writeOutput(output, `[storybook] BEGIN ${label}\n`); - await replayCommandLog(logPath, output); - await writeOutput(output, `\n[storybook] END ${label}\n`); - } - await writeOutput(output, `[storybook] ${failed ? 'FAIL' : 'OK'} ${label} (${status}; log: ${logPath})\n`); - if (exitError !== undefined) { - await writeOutput(output, `${formatDesktopStorybookError(exitError)}\n`); - } - }); - }; - void completion.then( - (code) => { - if (!stopRequested && code !== 0) { - report(true, child.signalCode ? `signal ${child.signalCode}` : `exit ${code}`); - } - }, - (error) => { - exitError = error; - report(true, 'could not start'); - }, - ); - - return { - completed: completion, - logPath, - stop(options = {}) { - stopPromise ??= (async () => { - stopRequested = true; - const running = exitError === undefined && child.exitCode === null && child.signalCode === null; - try { - if (running) { - await stopChildProcess(child.pid, completion, logPath); - } - } catch (error) { - exitError = error; - report(true, 'could not stop'); - await Promise.all(reports); - throw error; - } - report( - Boolean(options.failed || exitError !== undefined || (!running && child.exitCode !== 0)), - running ? 'stopped' : child.signalCode ? `signal ${child.signalCode}` : `exit ${child.exitCode}`, - ); - await Promise.all(reports); - })(); - return stopPromise; - }, - }; - } -} - -async function replayCommandLog(logPath: string, output: Pick): Promise { - for await (const chunk of fs.createReadStream(logPath)) { - await writeOutput(output, chunk); - } -} - -async function writeOutput(output: Pick, chunk: string | Uint8Array): Promise { - if (!output.write(chunk) && output instanceof Writable) { - await once(output, 'drain'); - } -} - -function spawnCommand(command: PreparedDesktopCommand, background: boolean, descriptor?: number) { - return spawn(command.command, [...command.args], { - cwd: command.cwd, - detached: background && process.platform !== 'win32', - env: { ...process.env, ...command.env }, - shell: requiresWindowsShell(command.command), - stdio: descriptor === undefined ? 'inherit' : ['inherit', descriptor, descriptor], - }); -} - -function requiresWindowsShell(command: string): boolean { - if (process.platform !== 'win32') { - return false; - } - - const executable = path.basename(command).toLowerCase(); - return path.extname(executable) !== '.exe' && !['node', 'powershell', 'pwsh'].includes(executable); -} - -function completed(child: ReturnType): Promise { - return new Promise((resolve, reject) => { - child.once('error', reject); - child.once('exit', (code, signal) => resolve(code ?? (signal ? 1 : 0))); - }); -} - -async function stopChildProcess(pid: number | undefined, completion: Promise, logPath: string): Promise { - if (!pid) { - return; - } - - if (process.platform === 'win32') { - await stopWindowsProcessTree(pid, logPath); - await completion; - return; - } - - try { - process.kill(-pid, 'SIGTERM'); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== 'ESRCH') { - throw error; - } - } - - let timer: ReturnType | undefined; - let stopped: boolean; - try { - stopped = await Promise.race([ - completion.then(() => true), - new Promise((resolve) => { - timer = setTimeout(() => resolve(false), 5000); - }), - ]); - } finally { - clearTimeout(timer); - } - if (!stopped) { - try { - process.kill(-pid, 'SIGKILL'); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== 'ESRCH') { - throw error; - } - } - await completion; - } -} - -async function stopWindowsProcessTree(pid: number, logPath: string): Promise { - const command: PreparedDesktopCommand = { - command: process.env.ComSpec ?? 'cmd.exe', - args: ['/d', '/s', '/c', `taskkill /PID ${pid} /T /F`], - cwd: path.parse(process.cwd()).root, - env: {}, - }; - const descriptor = fs.openSync(logPath, 'a'); - let child: ReturnType; - try { - child = spawnCommand(command, false, descriptor); - } finally { - fs.closeSync(descriptor); - } - const exitCode = await completed(child); - if (exitCode !== 0) { - throw new Error(`Could not stop owned process tree ${pid}.`); - } -} - -function formatCommandFailure(command: PreparedDesktopCommand, exitCode: number): string { - return `Command "${[command.command, ...command.args].join(' ')}" exited with code ${exitCode}.`; -} diff --git a/packages/agentic/storybook-desktop/src/cli/createDesktopStorybookCommand.ts b/packages/agentic/storybook-desktop/src/cli/createDesktopStorybookCommand.ts index 479df1e533..eddae15113 100644 --- a/packages/agentic/storybook-desktop/src/cli/createDesktopStorybookCommand.ts +++ b/packages/agentic/storybook-desktop/src/cli/createDesktopStorybookCommand.ts @@ -1,4 +1,9 @@ import { Command, Option } from 'commander'; +import path from 'node:path'; +import { claimDesktopWorkload, DesktopServiceScope, DesktopStopRequested } from '@fluentui-react-native/desktop-test'; +import { NodeDesktopCommandRunner } from '@fluentui-react-native/desktop-test/commands'; +import { createDesktopReportSession } from '@fluentui-react-native/desktop-test/reporting'; +import type { DesktopReportSession } from '@fluentui-react-native/desktop-test/reporting'; import { desktopSmokeModes, type DesktopSmokeMode } from '../config/commands.js'; import type { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; @@ -36,8 +41,8 @@ export function createDesktopStorybookCommand(options: CreateDesktopStorybookCom const program = new Command() .name('storybook-desktop') .description('Serve, prepare, bundle, build, run, and smoke test a React Native desktop Storybook app.') - .option('-c, --config ', 'path to storybook.config.ts') - .option('-v, --verbose', 'replay full captured output for successful commands as well as failures'); + .option('-c, --config ', 'path to desktop.config.mts (legacy storybook.config files are also accepted)') + .option('-v, --verbose', 'stream informational output as well as live errors and test progress'); let apiPromise: Promise | undefined; const getApi = (flags: PlatformFlags) => @@ -58,6 +63,8 @@ export function createDesktopStorybookCommand(options: CreateDesktopStorybookCom runSmokeTests: options.runSmokeTests, resolveNativeDriver: options.resolveNativeDriver, writeMacOSApplicationLease: options.writeMacOSApplicationLease, + reporter: options.reporter, + services: options.services, }), )); @@ -160,7 +167,55 @@ function addInstanceCommand(program: Command, getApi: GetApi): void { } export async function runDesktopStorybookCli(argv: readonly string[] = process.argv): Promise { - await createDesktopStorybookCommand().parseAsync([...argv]); + const options: CreateDesktopStorybookCommandOptions = {}; + const program = createDesktopStorybookCommand(options); + let session: DesktopReportSession | undefined; + let runner: NodeDesktopCommandRunner | undefined; + let releaseClaim: (() => void) | undefined; + let failure: unknown; + const interrupt = () => options.services?.fail(new Error('Storybook run interrupted.')); + program.hook('preAction', async (_program, action) => { + if (['manifest', 'instance'].includes(action.name()) || action.opts<{ list?: boolean }>().list) return; + options.config = await loadDesktopStorybookConfig(program.opts<{ config?: string }>().config); + session = await createDesktopReportSession({ + ...options.config.desktop?.reporting, + artifactsRoot: options.config.desktop?.reporting.artifactsRoot ?? path.join(options.config.projectRoot, 'artifacts', 'desktop'), + kind: ['test', 'smoke'].includes(action.name()) ? 'tests' : 'lifecycle', + verbose: program.opts<{ verbose?: boolean }>().verbose, + onFatal: (error) => options.services?.fail(error), + }); + if (!session.delegated && ['driver', 'smoke'].includes(action.name())) releaseClaim = claimDesktopWorkload(session.reporter.runId); + options.reporter = session.reporter; + runner = new NodeDesktopCommandRunner({ reporter: session.reporter, environment: session.environment }); + options.runner = runner; + options.services = new DesktopServiceScope(runner, session.reporter); + session.onStop?.(() => options.services!.requestStop()); + process.once('SIGINT', interrupt); + process.once('SIGTERM', interrupt); + }); + try { + await program.parseAsync([...argv]); + } catch (error) { + if (!(error instanceof DesktopStopRequested)) failure = error; + } finally { + process.off('SIGINT', interrupt); + process.off('SIGTERM', interrupt); + try { + await options.services?.close(); + await runner?.close(); + } catch (error) { + options.reporter?.fail(error, 'cleanup'); + failure = failure === undefined ? error : new AggregateError([failure, error], 'Storybook execution and cleanup failed.'); + } + try { + releaseClaim?.(); + } catch (error) { + options.reporter?.fail(error, 'cleanup'); + failure ??= error; + } + if (session) process.exitCode = await session.close(failure); + } + if (failure !== undefined) throw failure; } function addActionCommand(program: Command, action: 'bundle' | 'run' | 'build', description: string, getApi: GetApi): void { diff --git a/packages/agentic/storybook-desktop/src/cli/loadConfig.ts b/packages/agentic/storybook-desktop/src/cli/loadConfig.ts index 0e3953fbf2..f14ab95703 100644 --- a/packages/agentic/storybook-desktop/src/cli/loadConfig.ts +++ b/packages/agentic/storybook-desktop/src/cli/loadConfig.ts @@ -2,9 +2,16 @@ import fs from 'node:fs'; import path from 'node:path'; import { pathToFileURL } from 'node:url'; -import { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; +import type { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; +import { loadDesktopConfig, validateDesktopConfig } from '@fluentui-react-native/desktop-test/config'; +import { getStorybookIntegration } from '../config/integration.js'; const defaultConfigNames = [ + 'desktop.config.mts', + 'desktop.config.ts', + 'desktop.config.mjs', + 'desktop.config.js', + 'desktop.config.cjs', 'storybook.config.ts', 'storybook.config.mts', 'storybook.config.js', @@ -16,12 +23,37 @@ export async function loadDesktopStorybookConfig(configPath?: string, cwd = proc const resolvedPath = configPath ? resolveConfigPath(configPath, cwd) : findConfigPath(cwd); const loadedModule = (await import(pathToFileURL(resolvedPath).href)) as Record; const config = loadedModule.default ?? loadedModule.config; - if (!(config instanceof DesktopStorybookConfig)) { - throw new TypeError(`Desktop Storybook config at ${resolvedPath} must default-export the result of makeDesktopStorybookConfig().`); + if (config && typeof config === 'object' && 'schemaVersion' in config) { + validateDesktopConfig(config); + const resolved = await loadDesktopConfig(resolvedPath); + return getStorybookIntegration(config).configure(resolved.projectRoot, resolved); + } + if (!isLegacyConfig(config)) { + throw new TypeError( + `Desktop Storybook config at ${resolvedPath} must export a desktop config with the Storybook integration or makeDesktopStorybookConfig().`, + ); } return config; } +function isLegacyConfig(config: unknown): config is DesktopStorybookConfig { + return ( + !!config && + typeof config === 'object' && + 'projectRoot' in config && + typeof config.projectRoot === 'string' && + [ + 'getStorybookConfig', + 'getPlatformOptions', + 'getStoryPackages', + 'getCommandPlan', + 'getSmokeOptions', + 'getNativeDriverOptions', + 'resolvePackage', + ].every((key) => key in config && typeof Reflect.get(config, key) === 'function') + ); +} + function resolveConfigPath(configPath: string, cwd: string): string { const resolvedPath = path.isAbsolute(configPath) ? configPath : path.resolve(cwd, configPath); if (!fs.existsSync(resolvedPath)) { diff --git a/packages/agentic/storybook-desktop/src/cli/smokeTests.test.ts b/packages/agentic/storybook-desktop/src/cli/smokeTests.test.ts index 3d6f5c963b..da8b2d0bac 100644 --- a/packages/agentic/storybook-desktop/src/cli/smokeTests.test.ts +++ b/packages/agentic/storybook-desktop/src/cli/smokeTests.test.ts @@ -201,11 +201,13 @@ describe('runDesktopStorybookSmokeTests', () => { const runWdio: jest.MockedFunction = jest.fn(async ({ story }) => [ { storyId: story!, status: 'skipped' as const, durationMs: 0, skipReason: 'Capability unavailable' }, ]); - const result = await runDesktopStorybookSmokeTests(options, connect, runWdio); + await expect(runDesktopStorybookSmokeTests(options, connect, runWdio)).rejects.toThrow('only skipped'); expect(connect).not.toHaveBeenCalled(); expect(runWdio).toHaveBeenCalledTimes(1); - expect(result.wdio).toMatchObject([{ storyId: inputId, status: 'skipped' }]); - expect(formatDesktopStorybookSmokeTestSummary(result)).toBe('Ran 1 desktop story tests (0 passed, 1 skipped).'); + expect(JSON.parse(fs.readFileSync(path.join(root, 'artifacts/windows/desktop-driver/run.json'), 'utf8'))).toMatchObject({ + status: 'failed', + wdio: [{ storyId: inputId, status: 'skipped' }], + }); }); test('fails after persisting failed plan results and closing the session', async () => { diff --git a/packages/agentic/storybook-desktop/src/cli/smokeTests.ts b/packages/agentic/storybook-desktop/src/cli/smokeTests.ts index 3cb23fb0fd..2dcfb9a1f2 100644 --- a/packages/agentic/storybook-desktop/src/cli/smokeTests.ts +++ b/packages/agentic/storybook-desktop/src/cli/smokeTests.ts @@ -6,6 +6,7 @@ import type { DesktopStoryRunResult, DesktopStoryTestResult } from '@fluentui-re import { ArtifactManager } from '@fluentui-react-native/desktop-driver/artifacts'; import { selectDesktopStoryTests } from '@fluentui-react-native/desktop-driver/runner'; import { connectDesktopWebdriver } from '@fluentui-react-native/desktop-driver/wdio'; +import type { DesktopReportWriter } from '@fluentui-react-native/desktop-test/reporting'; import type { DesktopWebdriverOptions, DesktopWebdriverSession } from '@fluentui-react-native/desktop-driver/wdio'; import type { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; @@ -30,6 +31,8 @@ export type DesktopStorybookSmokeTestOptions = { testTag?: string; errorOutput?: DesktopStorybookErrorOutput; commandRunner?: DesktopCommandRunner; + report?: DesktopReportWriter; + signal?: AbortSignal; }; export type DesktopStorybookSmokeTestResult = DesktopStoryRunResult & { @@ -60,18 +63,33 @@ export async function runDesktopStorybookSmokeTests( if (storyIds.length === 0) { throw new Error('No desktop story tests matched the smoke selection.'); } - const artifacts = new ArtifactManager(options.artifactsRoot ?? path.join(config.projectRoot, 'artifacts', platform, 'desktop-driver')); + const artifacts = new ArtifactManager( + options.artifactsRoot ?? + (options.report + ? options.report.artifactPath(path.join('evidence', platform, 'static')) + : path.join(config.projectRoot, 'artifacts', platform, 'desktop-driver')), + ); const startedAt = new Date().toISOString(); const tests: DesktopStoryTestResult[] = []; const wdio: WdioStoryTestResult[] = []; const failures: unknown[] = []; + const staticFailures = new Map(); + for (const storyId of storyIds) options.report?.emit({ kind: 'work-selected', payload: { workId: `smoke:${storyId}`, label: storyId } }); for (const storyId of storyIds) { + const scope = `smoke:${storyId}`; + options.report?.emit( + { kind: 'scope-start', payload: { label: storyId, category: 'phase' } }, + scope, + { story: storyId, endpoint: platform }, + scope, + ); try { + options.signal?.throwIfAborted(); if (planStoryIds.has(storyId)) { - const plans = await runStoryPlans(options, storyId, artifacts.root, connect, testTag ?? 'desktop-e2e'); + const plans = await runStoryPlans(options, storyId, artifacts.root, connect, testTag ?? 'desktop-e2e', staticFailures); tests.push(...plans.tests); - if (plans.status !== 'passed' && plans.tests.every(({ status }) => status === 'passed' || status === 'skipped')) { + if (plans.status === 'failed' && plans.tests.every(({ status }) => status === 'passed' || status === 'skipped')) { throw new Error(`The static test runner failed for "${storyId}" without failed test details.`); } } @@ -90,6 +108,9 @@ export async function runDesktopStorybookSmokeTests( targetId: options.targetId, url: options.driverUrl, errorOutput: options.errorOutput, + report: options.report, + signal: options.signal, + optional: true, }, options.commandRunner, )), @@ -106,14 +127,19 @@ export async function runDesktopStorybookSmokeTests( } } catch (error) { failures.push(new Error(`Smoke tests for "${storyId}" failed: ${errorMessage(error)}`, { cause: error })); + options.report?.emit({ kind: 'scope-end', payload: { status: 'failed' } }, scope); break; } + options.report?.emit({ kind: 'scope-end', payload: { status: 'passed' } }, scope); } const failedTests = tests.filter(({ status }) => status !== 'passed' && status !== 'skipped'); if (failedTests.length) { failures.push( - new Error( + new AggregateError( + failedTests.map( + (test) => staticFailures.get(`${test.storyId}/${test.testId}`) ?? new Error(test.error ?? 'Static story test failed.'), + ), `${failedTests.length} of ${tests.length} desktop story tests failed: ${failedTests .map(({ status, storyId, testId }) => `${storyId}/${testId} (${status})`) .join(', ')}`, @@ -126,6 +152,9 @@ export async function runDesktopStorybookSmokeTests( } failures.push(new Error('Executable smoke tests reported failures.')); } + const allSkipped = tests.length + wdio.length > 0 && [...tests, ...wdio].every(({ status }) => status === 'skipped'); + if (allSkipped && failures.length === 0) + failures.push(new Error('Required desktop smoke cohort only skipped tests; it is not a passing run.')); const result: DesktopStorybookSmokeTestResult = { endpoint: platform, finishedAt: new Date().toISOString(), @@ -140,10 +169,14 @@ export async function runDesktopStorybookSmokeTests( wdio, }; try { - artifacts.writeMetadata('run', { + const reportPath = artifacts.writeMetadata('run', { ...result, ...(failures.length ? { errors: failures.map(errorMessage) } : {}), }); + options.report?.emit({ + kind: 'artifact', + payload: { path: path.relative(options.report.root, reportPath), description: 'Story smoke results' }, + }); } catch (error) { failures.push(error); writeDesktopStorybookFailure('writing the smoke report', error, options.errorOutput); @@ -163,6 +196,7 @@ async function runStoryPlans( artifactsRoot: string, connect: DesktopStorybookSmokeConnector, tag: string, + failures: Map, ): Promise { const desktop = await connect({ launchMode: 'attach', @@ -179,10 +213,42 @@ async function runStoryPlans( } const reported = new Set(); const onTestResult = (test: Readonly) => { + const key = `${test.storyId}/${test.testId}`; + if (reported.has(key)) return; + reported.add(key); + const failure = + test.status === 'passed' || test.status === 'skipped' ? undefined : new Error(test.error ?? 'Static story test failed.'); + if (failure) failures.set(key, failure); + options.report?.emit( + { + kind: 'test-result', + payload: { + testId: `static:${key}`, + title: key, + attempt: 0, + status: test.status === 'passed' || test.status === 'skipped' ? test.status : 'failed', + durationMs: test.durationMs, + ...(test.status === 'skipped' ? { reason: test.skipReason ?? 'Unsupported by the static test runner.' } : {}), + ...(failure ? { failure: options.report.failure(failure, 'test') } : {}), + }, + }, + `static:${key}`, + { story: test.storyId, test: test.testId, endpoint: options.platform }, + ); + for (const artifact of test.artifacts) + options.report?.emit( + { + kind: 'artifact', + payload: { + path: path.relative(options.report.root, path.join(artifactsRoot, artifact.path)), + description: `${key}: ${artifact.kind}`, + }, + }, + `static:${key}`, + ); if (test.status === 'passed' || test.status === 'skipped') { return; } - reported.add(`${test.storyId}/${test.testId}`); const step = test.steps.find(({ status }) => status === 'failed'); writeDesktopStorybookFailure( `test "${test.storyId}/${test.testId}" (${test.status}${step ? `, step ${step.index + 1}` : ''})`, diff --git a/packages/agentic/storybook-desktop/src/config/index.ts b/packages/agentic/storybook-desktop/src/config/index.ts index 86fb2804f9..ea2cd475f9 100644 --- a/packages/agentic/storybook-desktop/src/config/index.ts +++ b/packages/agentic/storybook-desktop/src/config/index.ts @@ -41,3 +41,5 @@ export type { export { FURN_STORYBOOK_PLATFORM, getAllPlatforms, getPlatform, isPlatform, setPlatform } from './platforms.ts'; export type { Platforms } from './platforms.ts'; export type { DesktopStorybookWdioOptions } from './wdio.ts'; +export { storybook, getStorybookIntegration } from './integration.ts'; +export type { StorybookDesktopIntegration } from './integration.ts'; diff --git a/packages/agentic/storybook-desktop/src/config/integration.ts b/packages/agentic/storybook-desktop/src/config/integration.ts new file mode 100644 index 0000000000..c82f309fe6 --- /dev/null +++ b/packages/agentic/storybook-desktop/src/config/integration.ts @@ -0,0 +1,55 @@ +import type { DesktopIntegration, ResolvedDesktopConfig } from '@fluentui-react-native/desktop-test'; +import type { DesktopConfig } from '@fluentui-react-native/desktop-test/config'; +import { DesktopStorybookConfig } from './makeDesktopStorybookConfig.ts'; +import type { DesktopStorybookConfigOptions } from './makeDesktopStorybookConfig.ts'; + +export type StorybookDesktopIntegration = DesktopIntegration & { + configure(projectRoot: string | URL, desktop?: ResolvedDesktopConfig): DesktopStorybookConfig; +}; + +export function storybook(options: DesktopStorybookConfigOptions = {}): StorybookDesktopIntegration { + return { + name: 'storybook', + validate() { + const allowed = [ + 'projectRoot', + 'storybookConfigDir', + 'appConfigPath', + 'storyPackages', + 'deviceAddons', + 'testIDPrefix', + 'platformOptions', + 'wdio', + 'platforms', + 'storyPatterns', + 'platformSettings', + ]; + for (const key of Object.keys(options)) + if (!allowed.includes(key)) throw new TypeError(`Unknown Storybook integration field "${key}".`); + new DesktopStorybookConfig(options); + }, + configure(projectRoot, desktop) { + return new DesktopStorybookConfig({ ...options, projectRoot }, desktop); + }, + async run({ config, target, supervisor, action }) { + const { DesktopStorybookCli } = await import('@fluentui-react-native/storybook-desktop/cli'); + const api = new DesktopStorybookCli(this.configure(config.projectRoot, config), { + paper: target.endpoint === 'macos' && target.host.renderer === 'paper', + reporter: supervisor.reporter, + runner: supervisor.runner, + services: supervisor.services, + }); + if (action === 'serve') await api.driver(target.endpoint); + else if (action === 'test') await api.smoke(target.endpoint, { mode: 'stories-and-tests' }); + else await api[action](target.endpoint); + }, + }; +} + +export function getStorybookIntegration(config: DesktopConfig): StorybookDesktopIntegration { + const integration = config.integration; + if (!integration || integration.name !== 'storybook' || !('configure' in integration) || typeof integration.configure !== 'function') { + throw new TypeError('Desktop config must explicitly import the Storybook integration.'); + } + return integration as StorybookDesktopIntegration; +} diff --git a/packages/agentic/storybook-desktop/src/config/makeDesktopStorybookConfig.ts b/packages/agentic/storybook-desktop/src/config/makeDesktopStorybookConfig.ts index 72488d4476..f65a554091 100644 --- a/packages/agentic/storybook-desktop/src/config/makeDesktopStorybookConfig.ts +++ b/packages/agentic/storybook-desktop/src/config/makeDesktopStorybookConfig.ts @@ -3,6 +3,7 @@ import { createRequire } from 'node:module'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import type { NativeDesktopApplicationDescriptor } from '@fluentui-react-native/desktop-driver'; +import type { ResolvedDesktopConfig, ResolvedDesktopTarget } from '@fluentui-react-native/desktop-test'; import type { DesktopCommand, DesktopCommandPlan, @@ -73,7 +74,7 @@ export type StoryPackageSpec = string | readonly [packageName: string, settings: export type DesktopStorybookConfigOptions = StorySettings & { /** - * Path to the consuming app. A file URL is convenient from storybook.config.ts. + * Path to the consuming app. A file URL is convenient from desktop.config.mts. * @default process.cwd() */ projectRoot?: string | URL; @@ -142,6 +143,7 @@ export class DesktopStorybookConfig { readonly projectRoot: string; readonly storybookConfigDir: string; readonly appConfigPath: string; + readonly desktop?: ResolvedDesktopConfig; private readonly config: DesktopStorybookConfigOptions; private readonly requireFromProject: NodeJS.Require; @@ -150,7 +152,8 @@ export class DesktopStorybookConfig { private packageManifestCache?: Readonly; private appManifestCache?: Readonly; - constructor(userConfig: DesktopStorybookConfigOptions = {}) { + constructor(userConfig: DesktopStorybookConfigOptions = {}, desktop?: ResolvedDesktopConfig) { + this.desktop = desktop; this.projectRoot = resolveProjectRoot(userConfig.projectRoot); this.storybookConfigDir = resolveFromProject(this.projectRoot, userConfig.storybookConfigDir ?? 'src'); this.appConfigPath = resolveFromProject(this.projectRoot, userConfig.appConfigPath ?? 'app.json'); @@ -259,12 +262,19 @@ export class DesktopStorybookConfig { getPlatformOptions(platformSetting: Platforms | string | undefined = this.platform): Readonly { const platform = requirePlatform(platformSetting); - const configured = this.config.platformOptions?.[platform] ?? {}; + const target = this.getDesktopTarget(platform); + if (this.desktop && !target) throw new Error(`No canonical desktop target is configured for ${platform}.`); + const legacy = this.config.platformOptions?.[platform] ?? {}; + for (const stage of ['prep', 'bundle', 'build', 'run'] as const) { + if (target?.commands?.[stage] !== undefined && legacy[stage] !== undefined) + throw new Error(`Ambiguous ${platform}.${stage}: configure it on the desktop target, not also in the Storybook integration.`); + } + const configured = { ...legacy, ...target?.commands }; const nativeProject = { ...defaultNativeProjectOptions(this, platform), ...configured.nativeProject, }; - const configuredNativeDriver = configured.nativeDriver; + const configuredNativeDriver = configured.nativeDriver === false ? false : { ...target?.nativeDriver, ...configured.nativeDriver }; return Object.freeze({ nativeDriver: @@ -278,8 +288,13 @@ export class DesktopStorybookConfig { installRoot: resolveOptionalProjectPath(this.projectRoot, configuredNativeDriver?.installRoot), application: Object.freeze({ ...defaultNativeApplication(this, platform), + ...(target?.host.windowTitle ? { windowTitle: target.host.windowTitle } : {}), + ...target?.host.application, ...configuredNativeDriver?.application, - executablePath: resolveOptionalProjectPath(this.projectRoot, configuredNativeDriver?.application?.executablePath), + executablePath: resolveOptionalProjectPath( + this.projectRoot, + configuredNativeDriver?.application?.executablePath ?? target?.host.application?.executablePath, + ), }), }), nativeProject: Object.freeze(nativeProject), @@ -292,6 +307,12 @@ export class DesktopStorybookConfig { }); } + getDesktopTarget(platform: Platforms): ResolvedDesktopTarget | undefined { + const targets = Object.values(this.desktop?.targets ?? {}).filter((target) => target.endpoint === platform); + if (targets.length > 1) throw new Error(`Storybook requires one unambiguous target per endpoint; ${platform} has ${targets.length}.`); + return targets[0]; + } + getCommandPlan( action: Exclude, platformSetting: Platforms | string | undefined = this.platform, diff --git a/packages/agentic/storybook-desktop/src/config/sourceConfig.test.ts b/packages/agentic/storybook-desktop/src/config/sourceConfig.test.ts index 4e1859c455..e46fdaf246 100644 --- a/packages/agentic/storybook-desktop/src/config/sourceConfig.test.ts +++ b/packages/agentic/storybook-desktop/src/config/sourceConfig.test.ts @@ -26,7 +26,8 @@ test('loads the source config through the CommonJS prebuild fallback without emi storyPackages: ['@fluentui-react-native/components'], wdio: { test: '*activation*' }, }); - process.stdout.write(JSON.stringify({ entry, stories: config.getStorybookConfig('macos').stories, wdio: config.wdio })); + const application = requireFromProject('./src/main.ts').default; + process.stdout.write(JSON.stringify({ entry, stories: config.getStorybookConfig('macos').stories, wdio: config.wdio, application })); `, projectRoot, ], @@ -40,5 +41,42 @@ test('loads the source config through the CommonJS prebuild fallback without emi entry: path.resolve(__dirname, 'index.ts'), stories: ['../../../packages/agentic/components/src/**/*.stories.?(ts|tsx)'], wdio: { test: '*activation*', timeoutMs: 30_000, reporter: 'spec', clickMode: 'auto' }, + application: { stories: expect.arrayContaining([expect.stringContaining('packages/agentic/components')]) }, }); }); + +test('the canonical Storybook config resolves target commands and identity without a second schema', () => { + const projectRoot = path.resolve(__dirname, '../../../../../apps/storybook'); + const result = spawnSync( + process.execPath, + [ + '--input-type=module', + '-e', + ` + import { loadDesktopConfig, selectDesktopRenderer } from '@fluentui-react-native/desktop-test/config'; + import { getStorybookIntegration } from '@fluentui-react-native/storybook-desktop/config'; + const desktop = await loadDesktopConfig(); + const config = getStorybookIntegration(desktop.config).configure(desktop.projectRoot, desktop); + const win32 = config.getPlatformOptions('win32'); + process.stdout.write(JSON.stringify({ + target: config.getDesktopTarget('win32'), + run: win32.run, + windowTitle: win32.nativeDriver.application.windowTitle, + fabricId: desktop.targets.macos.id, + paper: selectDesktopRenderer(desktop.targets.macos, 'paper'), + })); + `, + ], + { cwd: projectRoot, encoding: 'utf8', timeout: 10_000 }, + ); + if (result.status !== 0) throw new Error(result.stderr || result.error?.message || `Canonical config exited with ${result.status}.`); + const value = JSON.parse(result.stdout); + expect(value).toMatchObject({ + target: { endpoint: 'win32', host: { kind: 'rex', renderer: 'paper' } }, + run: { command: process.execPath }, + windowTitle: 'Agentic Components Storybook (Win32)', + paper: { endpoint: 'macos', host: { renderer: 'paper' } }, + }); + expect(value.target.id).toContain('win32'); + expect(value.paper.id).not.toBe(value.fabricId); +}); diff --git a/packages/agentic/storybook-desktop/src/driver/driverManifest.ts b/packages/agentic/storybook-desktop/src/driver/driverManifest.ts index efc6034582..d73a730baa 100644 --- a/packages/agentic/storybook-desktop/src/driver/driverManifest.ts +++ b/packages/agentic/storybook-desktop/src/driver/driverManifest.ts @@ -3,6 +3,7 @@ import fs from 'node:fs'; import path from 'node:path'; import type { DesktopStoryManifest, NativeDesktopApplicationDescriptor, NativeDriverArtifact } from '@fluentui-react-native/desktop-driver'; +import { selectDesktopRenderer } from '@fluentui-react-native/desktop-test/config'; import type { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; import type { DesktopStorybookInstance } from '../config/instance.js'; @@ -57,6 +58,8 @@ export function createDesktopStorybookDriverManifest({ throw new Error(`Native Desktop Driver is disabled for ${platform}.`); } const leasePath = path.join(config.projectRoot, 'storybook-desktop.generated', `application-lease.${platform}.json`); + const configuredTarget = config.getDesktopTarget?.(platform); + const target = configuredTarget && platform === 'macos' ? selectDesktopRenderer(configuredTarget, renderer) : configuredTarget; return Object.freeze({ application: Object.freeze({ ...nativeOptions.application, @@ -78,7 +81,7 @@ export function createDesktopStorybookDriverManifest({ schemaVersion: 2, storyManifest, storybookPort: instance.storybookPort, - targetId: `${config.appName}-${platform}`.toLowerCase(), + targetId: target?.id ?? `${config.appName}-${platform}`.toLowerCase(), testIDPrefix: config.testIDPrefix, wdio: config.wdio, }); diff --git a/packages/agentic/storybook-desktop/src/driver/serverIntegration.test.ts b/packages/agentic/storybook-desktop/src/driver/serverIntegration.test.ts index ffc2d9c832..9e6fd4c7ed 100644 --- a/packages/agentic/storybook-desktop/src/driver/serverIntegration.test.ts +++ b/packages/agentic/storybook-desktop/src/driver/serverIntegration.test.ts @@ -1,10 +1,12 @@ -import { spawn } from 'node:child_process'; import fs from 'node:fs'; import { createServer } from 'node:net'; import os from 'node:os'; import path from 'node:path'; import { createDesktopDriverClient } from '@fluentui-react-native/desktop-driver/client'; +import { NodeDesktopCommandRunner } from '@fluentui-react-native/desktop-test/commands'; +import { deadline } from '@fluentui-react-native/desktop-test/commands'; +import { DesktopRunReporter, startReportTransport } from '@fluentui-react-native/desktop-test/reporting'; import type { DesktopStorybookDriverManifest } from './driverManifest.js'; @@ -73,9 +75,27 @@ describe('desktop Storybook server integration', () => { }; fs.writeFileSync(manifestPath, JSON.stringify(driverManifest)); - const child = spawn(process.execPath, [path.resolve(__dirname, '../../config/server-runner.cjs')], { + const output: string[] = []; + const sink = { + write: (text: string) => { + output.push(text); + return true; + }, + }; + const reporter = new DesktopRunReporter({ + artifactsRoot: temporaryDirectory, + kind: 'lifecycle', + output: sink, + errorOutput: sink, + summary: false, + }); + const transport = await startReportTransport(reporter); + const runner = new NodeDesktopCommandRunner({ reporter, environment: transport.environment }); + const child = runner.start({ + command: process.execPath, + args: [path.resolve(__dirname, '../../config/server-runner.cjs')], + cwd: projectRoot, env: { - ...process.env, STORYBOOK_CONFIG_PATH: path.join(projectRoot, 'src'), STORYBOOK_DRIVER_MANIFEST: manifestPath, STORYBOOK_NATIVE_DRIVER_FAKE: '1', @@ -83,11 +103,7 @@ describe('desktop Storybook server integration', () => { STORYBOOK_SMOKE_MODE: 'stories-and-tests', STORYBOOK_WS_PORT: String(storybookPort), }, - stdio: ['ignore', 'pipe', 'pipe'], }); - const output: Buffer[] = []; - child.stdout.on('data', (chunk: Buffer) => output.push(chunk)); - child.stderr.on('data', (chunk: Buffer) => output.push(chunk)); try { const [indexResponse, driverResponse] = await Promise.all([ @@ -108,6 +124,7 @@ describe('desktop Storybook server integration', () => { alwaysMatch: { platformName: 'windows', 'furn:target': 'integration-windows', + 'furn:launchMode': 'attach', }, }); await expect(session.selectStory('components-button--default', 'integration-run')).resolves.toEqual({ @@ -117,15 +134,111 @@ describe('desktop Storybook server integration', () => { }); await session.delete(); } finally { - child.kill(); - await waitForExit(child); - fs.rmSync(temporaryDirectory, { force: true, recursive: true }); + try { + await child.stop(); + expect(await child.completed).toBe(0); + await runner.close(); + await transport.close(); + expect((await reporter.finish()).status).toBe('passed'); + } finally { + fs.rmSync(temporaryDirectory, { force: true, recursive: true }); + } } - expect(Buffer.concat(output).toString('utf8')).toContain('WebDriver:'); + expect(output.join('')).toContain('WebDriver '); }); }); +describe('PowerShell owned service integration', () => { + const windowsTest = process.platform === 'win32' ? test : test.skip; + windowsTest.each([false, true])( + 'forwards live grandchild diagnostics and drains services (failure=%s)', + async (failService) => { + const port = await getAvailablePort(); + const temporaryDirectory = fs.mkdtempSync(path.join(os.tmpdir(), 'furn-powershell-services-')); + const output: string[] = []; + const sink = { + write: (text: string) => { + output.push(text); + return true; + }, + }; + const reporter = new DesktopRunReporter({ + artifactsRoot: temporaryDirectory, + kind: 'lifecycle', + output: sink, + errorOutput: sink, + summary: false, + }); + const transport = await startReportTransport(reporter); + const runner = new NodeDesktopCommandRunner({ reporter, environment: transport.environment }); + const quote = (text: string) => `'${text.replaceAll("'", "''")}'`; + const projectRoot = path.resolve(__dirname, '../../../../../apps/storybook'); + const script = path.join(temporaryDirectory, 'fixture with spaces.ps1'); + const marker = path.join(temporaryDirectory, 'service-pid.json'); + const source = ` + require('node:fs').writeFileSync(${JSON.stringify(marker)}, JSON.stringify(process.pid)); + const server = require('node:net').createServer(socket => socket.end()); + setTimeout(() => server.listen(${port}, '127.0.0.1', () => { + process.stderr.write('live PowerShell grandchild diagnostic\\n'); + ${failService ? "setTimeout(() => { process.stderr.write('original nested service failure\\n'); process.exit(7); }, 700);" : ''} + }), 500);`; + fs.writeFileSync( + script, + ` + $ErrorActionPreference = 'Stop' + $projectRoot = ${quote(projectRoot)} + . ${quote(path.resolve(__dirname, '../../config/owned-services.ps1'))} + try { + $service = Start-OwnedCommand -FilePath ${quote(process.execPath)} -ArgumentList @('-e', ${quote(source)}) -LogName 'contract-service' + Wait-ForTcpPort -Port ${port} -TimeoutSeconds 10 + for ($index = 0; $index -lt 20; $index += 1) { Assert-OwnedServices; Start-Sleep -Milliseconds 100 } + } finally { Stop-OwnedServices } + `, + ); + let completed = false; + const running = runner.start({ + command: 'pwsh', + args: ['-NoProfile', '-NonInteractive', '-File', script], + cwd: projectRoot, + env: {}, + label: 'PowerShell fixture', + }); + void running.completed.then(() => { + completed = true; + }); + try { + await deadline( + (async () => { + while (!output.join('').includes('live PowerShell grandchild diagnostic')) { + if (completed) throw new Error(output.join('')); + await new Promise((resolve) => setTimeout(resolve, 50)); + } + })(), + 15_000, + 'PowerShell diagnostic', + ); + expect(completed).toBe(false); + expect(await running.completed).toBe(failService ? 1 : 0); + } finally { + try { + await running.stop(); + await runner.close(); + await transport.close(); + const result = await reporter.finish(); + expect(result.status).toBe(failService ? 'failed' : 'passed'); + if (failService) expect(output.join('')).toContain('original nested service failure'); + const pid: number = JSON.parse(fs.readFileSync(marker, 'utf8')); + expect(() => process.kill(pid, 0)).toThrow(); + } finally { + fs.rmSync(temporaryDirectory, { recursive: true, force: true }); + } + } + }, + 45_000, + ); +}); + function getAvailablePort(): Promise { return new Promise((resolve, reject) => { const server = createServer(); @@ -159,13 +272,6 @@ async function waitForResponse(url: string): Promise { throw new Error(`Timed out waiting for ${url}: ${(lastError as Error)?.message ?? 'not ready'}`); } -function waitForExit(child: ReturnType): Promise { - if (child.exitCode !== null || child.signalCode !== null) { - return Promise.resolve(); - } - return new Promise((resolve) => child.once('exit', () => resolve())); -} - function loopbackUrl(port: number, pathname: string): string { // eslint-disable-next-line @microsoft/sdl/no-insecure-url -- test-only loopback service return `http://127.0.0.1:${port}${pathname}`; diff --git a/packages/agentic/storybook-desktop/src/testing/runWdioTests.ts b/packages/agentic/storybook-desktop/src/testing/runWdioTests.ts index 73259b9a66..52cacc56c1 100644 --- a/packages/agentic/storybook-desktop/src/testing/runWdioTests.ts +++ b/packages/agentic/storybook-desktop/src/testing/runWdioTests.ts @@ -7,6 +7,8 @@ import type { DesktopStoryManifest, DesktopStoryManifestEntry } from '@fluentui- import type { DesktopArtifact } from '@fluentui-react-native/desktop-driver/authoring'; import { ArtifactManager, captureDesktopFailureEvidence } from '@fluentui-react-native/desktop-driver/artifacts'; import { connectDesktopWebdriver } from '@fluentui-react-native/desktop-driver/wdio'; +import type { DesktopReportWriter } from '@fluentui-react-native/desktop-test/reporting'; +import { DesktopReportedError } from '@fluentui-react-native/desktop-test/reporting'; import type { DesktopStorybookConfig } from '../config/makeDesktopStorybookConfig.js'; import type { Platforms } from '../config/platforms.js'; @@ -35,6 +37,9 @@ export type RunWdioStoryTestsOptions = DesktopStorybookWdioOptions & { url: string; artifactsRoot?: string; errorOutput?: DesktopStorybookErrorOutput; + report?: DesktopReportWriter; + signal?: AbortSignal; + optional?: boolean; }; export class WdioStoryTestRunError extends Error { @@ -42,7 +47,7 @@ export class WdioStoryTestRunError extends Error { constructor(results: readonly WdioStoryTestResult[], failures: readonly unknown[]) { const cause = failures.length === 1 ? failures[0] : new AggregateError(failures, 'Executable wdio tests and evidence/cleanup failed.'); - super(errorMessage(cause), { cause }); + super(`Executable wdio stories failed: ${errorMessage(cause)}`, { cause }); this.name = 'WdioStoryTestRunError'; this.results = [...results]; } @@ -61,7 +66,7 @@ export function selectWdioStories( export async function runWdioStoryTests( options: RunWdioStoryTestsOptions, - runner: DesktopCommandRunner = new NodeDesktopCommandRunner(), + runner?: DesktopCommandRunner, ): Promise { const settings = resolveWdioOptions(options); const stories = selectWdioStories(options.manifest, settings); @@ -80,22 +85,35 @@ export async function runWdioStoryTests( }); if (cases.length === 0) throw new Error('No executable wdio tests matched the requested selection.'); const artifacts = new ArtifactManager( - options.artifactsRoot ?? path.join(options.config.projectRoot, 'artifacts', options.platform, 'wdio'), + options.artifactsRoot ?? + (options.report + ? options.report.artifactPath(path.join('evidence', options.platform, 'wdio')) + : path.join(options.config.projectRoot, 'artifacts', options.platform, 'wdio')), ); const generatedRoot = path.join(options.config.projectRoot, 'storybook-desktop.generated', 'wdio'); - fs.mkdirSync(generatedRoot, { recursive: true }); - const directory = fs.mkdtempSync(path.join(generatedRoot, 'run-')); const workerUrl = pathToFileURL( path.join(options.config.resolvePackage('@fluentui-react-native/storybook-desktop').root, 'lib', 'testing', 'worker.js'), ).href; const results: WdioStoryTestResult[] = []; + const executionFailures: unknown[] = []; + for (const { entry, testName } of cases) + options.report?.emit({ + kind: 'work-selected', + payload: { workId: csfTestId(entry.id, testName), label: `${entry.id} / ${testName ?? 'default'}` }, + }); + const ownedRunner = runner ? undefined : new NodeDesktopCommandRunner({ reporter: options.report }); + const commands = runner ?? ownedRunner!; + let directory: string | undefined; let infrastructureFailure: unknown; let activeStory: string | undefined; let phase = 'setup'; let activeResult: WdioStoryTestResult | undefined; let activeStarted = 0; try { + fs.mkdirSync(generatedRoot, { recursive: true }); + directory = fs.mkdtempSync(path.join(generatedRoot, 'run-')); for (const [index, { entry, code, testName }] of cases.entries()) { + options.signal?.throwIfAborted(); const displayName = testName === undefined ? entry.id : `${entry.id} / ${testName}`; const artifactId = testName === undefined ? entry.id : `${entry.id}-${createHash('sha256').update(testName).digest('hex').slice(0, 16)}`; @@ -110,6 +128,10 @@ export async function runWdioStoryTests( durationMs: 0, }; activeResult = result; + const testId = csfTestId(entry.id, testName); + const context = { story: entry.id, test: testName ?? 'default', endpoint: options.platform }; + options.report?.emit({ kind: 'scope-start', payload: { label: displayName, category: 'worker' } }, testId, context, testId); + options.report?.emit({ kind: 'test-start', payload: { testId, title: displayName, attempt: 0 } }, testId, context); // The supervisor owns the session, so even a crashed or timed-out Node worker cannot strand it. const desktop = await connectDesktopWebdriver({ clickMode: settings.clickMode, @@ -149,7 +171,7 @@ export async function runWdioStoryTests( ); phase = 'execution'; const outcome = await runWorker( - runner, + commands, { command: process.execPath, label: `wdio "${displayName}"`, @@ -164,10 +186,13 @@ export async function runWdioStoryTests( ], cwd: options.config.projectRoot, env: {}, + category: 'worker', + context, }, Math.min(settings.timeoutMs + 5000, 2_147_483_647), resultPath, displayName, + options.signal, ); result.status = outcome.status; if (outcome.status === 'skipped') { @@ -175,12 +200,25 @@ export async function runWdioStoryTests( } } catch (error) { failure = error; + executionFailures.push(error); result.error = formatDesktopStorybookError(error); writeDesktopStorybookFailure(`wdio "${displayName}" during ${phase}`, error, options.errorOutput); const evidence = await captureDesktopFailureEvidence(desktop.session, artifacts, artifactId); result.artifacts = evidence.artifacts; result.evidenceErrors = evidence.errors; + for (const artifact of result.artifacts) + options.report?.emit( + { + kind: 'artifact', + payload: { + path: path.relative(options.report.root, path.join(artifacts.root, artifact.path)), + description: `${displayName}: ${artifact.kind}`, + }, + }, + testId, + ); for (const evidenceError of result.evidenceErrors) { + options.report?.fail(new Error(evidenceError), 'evidence', testId); writeDesktopStorybookFailure(`evidence for "${displayName}"`, evidenceError, options.errorOutput); } } @@ -188,6 +226,7 @@ export async function runWdioStoryTests( try { await desktop.delete(); } catch (cleanupError) { + options.report?.fail(cleanupError, 'cleanup', testId); throw new AggregateError( failure === undefined ? [cleanupError] : [failure, cleanupError], `Could not release the wdio session for ${displayName}; stopping before another test starts.`, @@ -195,46 +234,94 @@ export async function runWdioStoryTests( } result.durationMs = Date.now() - started; results.push(result); + options.report?.emit( + { + kind: 'test-result', + payload: { + testId, + title: displayName, + attempt: 0, + status: result.status, + durationMs: result.durationMs, + ...(result.status === 'skipped' ? { reason: result.skipReason } : {}), + ...(failure !== undefined ? { failure: options.report.failure(failure, 'test') } : {}), + }, + }, + testId, + context, + ); + options.report?.emit({ kind: 'scope-end', payload: { status: result.status } }, testId); } } catch (error) { infrastructureFailure = error; + options.report?.fail(error); if (activeResult && !results.includes(activeResult)) { activeResult.status = 'failed'; activeResult.error = formatDesktopStorybookError(error); activeResult.durationMs = Date.now() - activeStarted; results.push(activeResult); + const testId = csfTestId(activeResult.storyId, activeResult.testName); + options.report?.emit( + { + kind: 'test-result', + payload: { + testId, + title: activeStory!, + attempt: 0, + status: 'failed', + durationMs: activeResult.durationMs, + failure: options.report.failure(error), + }, + }, + testId, + { endpoint: options.platform, story: activeResult.storyId, test: activeResult.testName ?? 'default' }, + ); + options.report?.emit({ kind: 'scope-end', payload: { status: 'failed' } }, testId); } writeDesktopStorybookFailure(`wdio "${activeStory ?? 'run'}" during ${phase}`, error, options.errorOutput); } const failures: unknown[] = infrastructureFailure === undefined ? [] : [infrastructureFailure]; - if (results.some(({ status }) => status === 'failed')) { - failures.push( - new Error( - `Executable wdio stories failed: ${results - .filter(({ status }) => status === 'failed') - .map(({ storyId, testName }) => (testName === undefined ? storyId : `${storyId} / ${testName}`)) - .join(', ')}. ` + `See ${path.join(artifacts.root, 'run.json')}.`, - ), - ); - } + failures.push(...executionFailures); + const allSkipped = results.every(({ status }) => status === 'skipped'); + const incomplete = results.length !== cases.length || (allSkipped && !options.optional); + if (incomplete && failures.length === 0) + failures.push(new Error('Required executable story cohort did not execute any tests or left selected work unfinished.')); try { - artifacts.writeMetadata('run', { + const reportPath = artifacts.writeMetadata('run', { endpoint: options.platform, platformManifestDigest: options.manifest.platformManifestDigest, - status: failures.length ? 'failed' : 'passed', + status: failures.length + ? incomplete && executionFailures.length === 0 && infrastructureFailure === undefined + ? 'incomplete' + : 'failed' + : allSkipped + ? 'skipped' + : 'passed', ...(infrastructureFailure === undefined ? {} : { error: errorMessage(infrastructureFailure) }), tests: results, }); + options.report?.emit({ + kind: 'artifact', + payload: { path: path.relative(options.report.root, reportPath), description: 'Executable story results' }, + }); } catch (error) { failures.push(error); writeDesktopStorybookFailure('writing the wdio report', error, options.errorOutput); } try { - fs.rmSync(directory, { recursive: true, force: true }); + if (directory) fs.rmSync(directory, { recursive: true, force: true }); } catch (error) { failures.push(error); writeDesktopStorybookFailure('cleaning generated wdio files', error, options.errorOutput); } + if (ownedRunner) { + if (failures.length) ownedRunner.reporter?.fail(new AggregateError(failures, 'Executable story failures.')); + try { + await ownedRunner.close(); + } catch (error) { + failures.push(error); + } + } if (failures.length) throw new WdioStoryTestRunError(results, failures); return results; } @@ -254,6 +341,12 @@ export function matchesWdioTest(name: string, pattern?: string): boolean { return pattern === undefined || path.matchesGlob(name, pattern); } +function csfTestId(storyId: string, testName?: string): string { + return `csf-${createHash('sha256') + .update(JSON.stringify([storyId, testName ?? 'default'])) + .digest('hex')}`; +} + function errorMessage(error: unknown): string { return error instanceof Error ? error.message : String(error); } @@ -264,14 +357,25 @@ async function runWorker( timeoutMs: number, resultPath: string, storyId: string, + signal?: AbortSignal, ): Promise { const worker = runner.start(command); let timer: ReturnType | undefined; let failure: unknown; let result: WorkerResult | undefined; + let abort: (() => void) | undefined; try { const exitCode = await Promise.race([ worker.completed, + ...(signal + ? [ + new Promise((_resolve, reject) => { + abort = () => reject(signal.reason); + if (signal.aborted) abort(); + else signal.addEventListener('abort', abort, { once: true }); + }), + ] + : []), new Promise((_resolve, reject) => { timer = setTimeout(() => reject(new Error(`Executable wdio worker exceeded its ${timeoutMs}ms process deadline.`)), timeoutMs); }), @@ -285,8 +389,10 @@ async function runWorker( 'error' in outcome && typeof outcome.error === 'string' ) { + if ('failure' in outcome && outcome.failure !== undefined) throw new DesktopReportedError(outcome.failure); throw new Error(`Executable wdio callback "${storyId}" failed:\n${outcome.error}`); } + if (exitCode !== 0) { throw new Error(`Executable wdio worker for "${storyId}" exited with code ${exitCode}.`); } @@ -298,6 +404,7 @@ async function runWorker( failure = error; } finally { clearTimeout(timer); + if (abort) signal?.removeEventListener('abort', abort); } try { await worker.stop({ failed: failure !== undefined }); diff --git a/packages/agentic/storybook-desktop/src/testing/wdioRunner.contract.cjs b/packages/agentic/storybook-desktop/src/testing/wdioRunner.contract.cjs index aa90e3e4fa..b15486c8a4 100644 --- a/packages/agentic/storybook-desktop/src/testing/wdioRunner.contract.cjs +++ b/packages/agentic/storybook-desktop/src/testing/wdioRunner.contract.cjs @@ -13,11 +13,13 @@ async function main() { { runWdioStoryTests, WdioStoryTestRunError }, { createDesktopDriverStoryHarness, createFakeStoryWindows }, { makeDesktopStorybookConfig }, + { DesktopRunReporter }, ] = await Promise.all([ import(pathToFileURL(path.join(packageRoot, 'lib/driver/storyManifest.js')).href), import(pathToFileURL(path.join(packageRoot, 'lib/testing/runWdioTests.js')).href), import('@fluentui-react-native/desktop-driver/testing'), import(pathToFileURL(path.join(packageRoot, 'lib/config/index.js')).href), + import('@fluentui-react-native/desktop-test/reporting'), ]); fs.writeFileSync( path.join(root, 'fixture.stories.ts'), @@ -180,7 +182,27 @@ async function main() { namedTimeout = error.results.map(({ testName, status }) => ({ testName, status })); } const passed = await run('pass'); - const skipped = await run('skip'); + let skipped; + try { + await run('skip'); + throw new Error('An all-skipped required cohort must not pass.'); + } catch (error) { + if ( + !(error instanceof Error) || + !(error instanceof WdioStoryTestRunError) || + !('results' in error) || + !Array.isArray(error.results) || + error.results[0]?.status !== 'skipped' + ) + throw error; + skipped = error.results; + } + const skipReport = new DesktopRunReporter({ artifactsRoot: path.join(root, 'reports'), optional: true, summary: false }); + await run('skip', { optional: true, report: skipReport }); + const skipResult = await skipReport.finish(); + if (skipResult.status !== 'skipped' || skipResult.work.some(({ status }) => status !== 'skipped')) { + throw new Error('A skipped callback was reported as passed selected work.'); + } const failures = []; for (const story of ['fail', 'timeout', 'spin', 'crash', 'exit-zero', 'stale', 'navigation-failure']) { let rejected = false; @@ -232,7 +254,9 @@ async function main() { selections.length = 0; const control = await runSmokeControl(packageRoot, root, smokeManifestPath); if (control.code !== 0) throw new Error(control.stderr || control.stdout || `Smoke control exited with ${control.code}.`); - const smokeReport = JSON.parse(fs.readFileSync(path.join(root, 'artifacts/windows/desktop-driver/run.json'), 'utf8')); + const reportRoot = path.join(root, 'artifacts', 'desktop'); + const runDirectory = fs.readdirSync(reportRoot)[0]; + const smokeReport = JSON.parse(fs.readFileSync(path.join(reportRoot, runDirectory, 'evidence/windows/static/run.json'), 'utf8')); const smoke = { plans: smokeReport.tests.map(({ storyId }) => storyId), callbacks: smokeReport.wdio.map(({ storyId }) => storyId), diff --git a/packages/agentic/storybook-desktop/src/testing/worker.ts b/packages/agentic/storybook-desktop/src/testing/worker.ts index 6b70623684..4d9607a12d 100644 --- a/packages/agentic/storybook-desktop/src/testing/worker.ts +++ b/packages/agentic/storybook-desktop/src/testing/worker.ts @@ -1,9 +1,11 @@ import fs from 'node:fs'; +import path from 'node:path'; import { test } from 'node:test'; import { attachDesktopWebdriver, type DesktopWebdriverAttachment } from '@fluentui-react-native/desktop-driver/wdio'; import { expect } from 'expect-webdriverio'; import type { DesktopEndpoint } from '@fluentui-react-native/desktop-driver'; +import { createDesktopReportSession, DESKTOP_REPORT_CONTEXT } from '@fluentui-react-native/desktop-test/reporting'; import type { WdioStoryTest } from './types.js'; import { formatDesktopStorybookError, writeDesktopStorybookFailure } from '../../config/diagnostics.cjs'; @@ -20,20 +22,34 @@ export type WdioWorkerOptions = { export function registerWdioStoryTest(options: WdioWorkerOptions, callback: WdioStoryTest): void { const name = options.testName === undefined ? options.storyId : `${options.storyId} / ${options.testName}`; test(name, { timeout: options.timeoutMs }, async (context) => { + const report = process.env[DESKTOP_REPORT_CONTEXT] + ? await createDesktopReportSession({ artifactsRoot: path.dirname(options.resultPath), kind: 'lifecycle' }) + : undefined; + const controller = new AbortController(); + const signal = AbortSignal.any([context.signal, controller.signal]); + report?.onStop?.(() => controller.abort(new Error('Owned story worker stop requested.'))); let recorded = false; + let failure: unknown; const recordFailure = (error: unknown) => { - const diagnostic = formatDesktopStorybookError(error); + failure = error; + const diagnostic = report?.reporter.redact(formatDesktopStorybookError(error)) ?? formatDesktopStorybookError(error); try { - fs.writeFileSync(options.resultPath, JSON.stringify({ status: 'failed', error: diagnostic })); + fs.writeFileSync( + options.resultPath, + JSON.stringify({ status: 'failed', error: diagnostic, failure: report?.reporter.failure(error, 'test') }), + { mode: 0o600 }, + ); recorded = true; } catch (writeError) { writeDesktopStorybookFailure(`wdio "${name}"`, error); throw new AggregateError([error, writeError], 'Could not persist the worker failure diagnostic.'); } }; - context.after(() => { - if (!recorded && context.signal.aborted) { - recordFailure(context.signal.reason); + context.after(async () => { + try { + if (!recorded && signal.aborted) recordFailure(signal.reason); + } finally { + await report?.close(failure); } }); try { @@ -41,14 +57,14 @@ export function registerWdioStoryTest(options: WdioWorkerOptions, callback: Wdio if (!['macos', 'windows', 'win32'].includes(options.platform) || desktop.session.capabilities['furn:endpoint'] !== options.platform) { throw new Error('The wdio worker platform does not match the attached desktop endpoint.'); } - context.signal.throwIfAborted(); + signal.throwIfAborted(); let skipReason: string | undefined; await callback({ browser: desktop.browser, desktop, expect, platform: options.platform, - signal: context.signal, + signal, skip: (reason) => { if (!reason?.trim()) { throw new TypeError('Skipping a wdio story requires a reason.'); @@ -57,7 +73,7 @@ export function registerWdioStoryTest(options: WdioWorkerOptions, callback: Wdio context.skip(reason); }, }); - context.signal.throwIfAborted(); + signal.throwIfAborted(); fs.writeFileSync(options.resultPath, JSON.stringify(skipReason ? { status: 'skipped', skipReason } : { status: 'passed' })); recorded = true; } catch (error) { diff --git a/packages/agentic/storybook-desktop/tsconfig.json b/packages/agentic/storybook-desktop/tsconfig.json index cc8223069e..3619259f39 100644 --- a/packages/agentic/storybook-desktop/tsconfig.json +++ b/packages/agentic/storybook-desktop/tsconfig.json @@ -9,6 +9,9 @@ }, "include": ["src"], "references": [ + { + "path": "../desktop-test/tsconfig.json" + }, { "path": "../desktop-driver/tsconfig.json" }, diff --git a/tsconfig.json b/tsconfig.json index 1bb404723f..e756bb74e9 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -28,6 +28,9 @@ { "path": "packages/agentic/desktop-driver/tsconfig.json" }, + { + "path": "packages/agentic/desktop-test/tsconfig.json" + }, { "path": "packages/agentic/native-core/tsconfig.json" }, diff --git a/yarn.lock b/yarn.lock index 10f6c9840d..db27377bb1 100644 --- a/yarn.lock +++ b/yarn.lock @@ -2567,6 +2567,7 @@ __metadata: "@babel/core": "catalog:" "@fluentui-react-native/components": "workspace:*" "@fluentui-react-native/desktop-driver": "workspace:*" + "@fluentui-react-native/desktop-test": "workspace:*" "@fluentui-react-native/native-core": "workspace:*" "@fluentui-react-native/scripts": "workspace:*" "@fluentui-react-native/storybook-desktop": "workspace:*" @@ -3500,6 +3501,27 @@ __metadata: languageName: unknown linkType: soft +"@fluentui-react-native/desktop-test@workspace:*, @fluentui-react-native/desktop-test@workspace:packages/agentic/desktop-test": + version: 0.0.0-use.local + resolution: "@fluentui-react-native/desktop-test@workspace:packages/agentic/desktop-test" + dependencies: + "@fluentui-react-native/desktop-driver": "workspace:*" + "@fluentui-react-native/scripts": "workspace:*" + "@rnx-kit/reporter": "catalog:" + "@wdio/cli": "catalog:" + "@wdio/jasmine-framework": "catalog:" + "@wdio/local-runner": "catalog:" + "@wdio/reporter": "npm:9.24.0" + "@wdio/runner": "catalog:" + commander: "npm:^14.0.2" + expect-webdriverio: "catalog:" + import-meta-resolve: "npm:4.2.0" + webdriverio: "catalog:" + bin: + desktop-test: ./config/cli.cjs + languageName: unknown + linkType: soft + "@fluentui-react-native/divider@workspace:*, @fluentui-react-native/divider@workspace:packages/components/Divider": version: 0.0.0-use.local resolution: "@fluentui-react-native/divider@workspace:packages/components/Divider" @@ -3643,6 +3665,8 @@ __metadata: dependencies: "@babel/core": "catalog:" "@babel/runtime": "catalog:" + "@fluentui-react-native/desktop-driver": "workspace:*" + "@fluentui-react-native/desktop-test": "workspace:*" "@fluentui-react-native/focus-zone": "workspace:*" "@fluentui-react-native/scripts": "workspace:*" "@office-iss/react-native-win32": "npm:^0.81.0" @@ -5144,6 +5168,7 @@ __metadata: "@babel/core": "catalog:" "@babel/plugin-transform-typescript": "catalog:" "@fluentui-react-native/desktop-driver": "workspace:*" + "@fluentui-react-native/desktop-test": "workspace:*" "@fluentui-react-native/scripts": "workspace:*" "@react-native/babel-preset": "npm:^0.81.0" "@rnx-kit/tools-react-native": "catalog:" @@ -5475,6 +5500,7 @@ __metadata: dependencies: "@babel/core": "catalog:" "@babel/runtime": "catalog:" + "@fluentui-react-native/desktop-test": "workspace:*" "@fluentui-react-native/scripts": "workspace:*" "@fluentui-react-native/tester-core": "workspace:*" "@office-iss/react-native-win32": "npm:^0.81.0" @@ -5511,6 +5537,7 @@ __metadata: "@babel/core": "catalog:" "@babel/runtime": "catalog:" "@fluentui-react-native/callout": "workspace:*" + "@fluentui-react-native/desktop-test": "workspace:*" "@fluentui-react-native/experimental-appearance-additions": "workspace:*" "@fluentui-react-native/experimental-avatar": "workspace:*" "@fluentui-react-native/experimental-checkbox": "workspace:*" @@ -16953,7 +16980,7 @@ __metadata: languageName: node linkType: hard -"import-meta-resolve@npm:^4.0.0, import-meta-resolve@npm:^4.1.0, import-meta-resolve@npm:^4.2.0": +"import-meta-resolve@npm:4.2.0, import-meta-resolve@npm:^4.0.0, import-meta-resolve@npm:^4.1.0, import-meta-resolve@npm:^4.2.0": version: 4.2.0 resolution: "import-meta-resolve@npm:4.2.0" checksum: 10c0/3ee8aeecb61d19b49d2703987f977e9d1c7d4ba47db615a570eaa02fe414f40dfa63f7b953e842cbe8470d26df6371332bfcf21b2fd92b0112f9fea80dde2c4c From 31098f49646acaea6fde229dc29c9a32619a3bc8 Mon Sep 17 00:00:00 2001 From: Jason Morse Date: Thu, 22 Oct 2026 09:20:42 -0700 Subject: [PATCH 3/8] Qualify the desktop driver refactor on macOS Fix macOS-only harness defects found while running the neutral desktop-test harness and Storybook integration on macOS: - Canonicalize the reporting root (/var -> /private/var) and its contract fixture root. - Skip stop frames to producers whose Unix sockets already half-closed. - Request the renderer-scoped Storybook target the embedded driver registers. - Launch the exact built RN Test App product on macOS, failing closed on a bundle identifier, renderer, or embedded bundle mismatch. - Label retried attempts by ordinal. Scroll FluentTester buttons until their click point is unobscured, and let specs that share a page find it already selected. Extend the RNmacOS patch so accessible Fabric paragraphs are exposed to accessibility on macOS. macOS Fabric and Paper both pass Storybook 280/280 and the FluentTester pilot and legacy (62 tests) suites with no skips or retries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .changeset/desktop-test-macos-fixes.md | 14 ++++ ...t-native-macos-npm-0.81.9-aa583a0648.patch | 20 ++++++ apps/E2E/README.md | 18 +++++ apps/E2E/src/common/BasePage.ts | 3 +- apps/E2E/src/common/nativeDesktop.ts | 18 ++++- driver-next-plan.md | 33 ++++++++- .../desktop-driver/native/macos/README.md | 5 +- packages/agentic/desktop-test/README.md | 18 +++-- packages/agentic/desktop-test/src/adapters.ts | 57 +++++++++++++++- .../desktop-test/src/contracts.contract.cjs | 67 ++++++++++++++++++- .../desktop-test/src/contracts.test.ts | 1 + .../desktop-test/src/reporting/collector.ts | 13 ++-- .../desktop-test/src/reporting/transport.ts | 3 +- packages/agentic/native-core/README.md | 4 ++ .../src/cli/DesktopStorybookCli.test.ts | 54 ++++++++++++++- .../src/cli/DesktopStorybookCli.ts | 11 ++- .../src/driver/driverManifest.ts | 15 ++++- yarn.lock | 4 +- 18 files changed, 326 insertions(+), 32 deletions(-) create mode 100644 .changeset/desktop-test-macos-fixes.md diff --git a/.changeset/desktop-test-macos-fixes.md b/.changeset/desktop-test-macos-fixes.md new file mode 100644 index 0000000000..0bd09f309d --- /dev/null +++ b/.changeset/desktop-test-macos-fixes.md @@ -0,0 +1,14 @@ +--- +'@fluentui-react-native/desktop-test': patch +'@fluentui-react-native/storybook-desktop': patch +--- + +Fix macOS desktop runs: canonicalize the reporting root so artifacts under +symlinked temporary directories stay confined, skip stop requests to producers +whose Unix-domain sockets have already half-closed, and request the canonical +desktop target that Storybook's embedded driver registers for Fabric and Paper +smoke tests. macOS RN Test App sessions without explicit identity now launch +the project's exact built product instead of any Launch Services copy sharing +its bundle identifier, and fail closed when its bundle identifier, renderer, or +embedded JavaScript bundle does not match. Console test labels number retried +attempts by ordinal rather than by WDIO's sparse attempt keys. diff --git a/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch b/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch index 19175c6890..09efd72969 100644 --- a/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch +++ b/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch @@ -53,6 +53,26 @@ index 09822246f92767208a7d2ad561d4b2954c0fef33..6e5fce575e507ed0c3e3f6794115dd76 @"timer": NSAccessibilityStaticTextRole, // no exact match on macOS @"toolbar": NSAccessibilityToolbarRole, // Roles/traits that are macOS specific and are used by some of the core components (Lists): +diff --git a/React/Fabric/Mounting/ComponentViews/Text/RCTParagraphComponentView.mm b/React/Fabric/Mounting/ComponentViews/Text/RCTParagraphComponentView.mm +index 804eec8079a01f7580d0d6e7b4c2ede4a4a61c7f..a6d967119c4847be709c5a73b518862f781f7634 100644 +--- a/React/Fabric/Mounting/ComponentViews/Text/RCTParagraphComponentView.mm ++++ b/React/Fabric/Mounting/ComponentViews/Text/RCTParagraphComponentView.mm +@@ -344,10 +344,15 @@ static void RCTCancelTouchesForView(RCTPlatformView *view) + + - (BOOL)isAccessibilityElement + { ++#if TARGET_OS_OSX // [macOS ++ // macOS has no `accessibilityElements` provider, so an accessible paragraph exposes itself as static text. ++ return static_cast(*_props).accessible; ++#else // macOS] + // All accessibility functionality of the component is implemented in `accessibilityElements` method below. + // Hence to avoid calling all other methods from `UIAccessibilityContainer` protocol (most of them have default + // implementations), we return here `NO`. + return NO; ++#endif // [macOS] + } + + #if !TARGET_OS_OSX // [macOS] diff --git a/React/Fabric/Mounting/ComponentViews/TextInput/RCTTextInputComponentView.mm b/React/Fabric/Mounting/ComponentViews/TextInput/RCTTextInputComponentView.mm index e7254907271b70783e886d378c7824df46e04e93..fbe1d5d7e834afb4553adb03253e34f6b9ce69d7 100644 --- a/React/Fabric/Mounting/ComponentViews/TextInput/RCTTextInputComponentView.mm diff --git a/apps/E2E/README.md b/apps/E2E/README.md index c901113e25..1703182121 100644 --- a/apps/E2E/README.md +++ b/apps/E2E/README.md @@ -34,6 +34,24 @@ and popup opening. This probe is not the complete legacy baseline or an old/new comparative run. It does not enable restricted input or waive unsupported assertions. +On macOS, bundle before `prep`, because CocoaPods embeds only resources that exist +during `pod install`. Add `--paper` to `prep` and `test` for Paper: + +```sh +yarn workspace @fluentui-react-native/tester desktop-test --target macos build-driver +yarn workspace @fluentui-react-native/tester desktop-test --target macos bundle +yarn workspace @fluentui-react-native/tester desktop-test --target macos prep +yarn workspace @fluentui-react-native/tester desktop-test --target macos build +yarn workspace @fluentui-react-native/e2e-testing desktop-test --target macos test --suite legacy +``` + +On October 22, 2026 (UTC), the macOS pilot and all 62 `legacy` tests pass on +both Fabric and Paper, with no skips or retries. Fabric depends on the +repository's RNmacOS Text accessibility patch, described in the +[native-core README](../../packages/agentic/native-core/README.md#repository-local-rnmacos-prerequisite-patch). +Selected tester buttons are scrolled until their center is inside the list +before clicking, because the native click targets that point. + Jasmine globals, hooks, matchers, retries, and grouped specs remain WDIO-owned. The shared page adapter maps only established native attributes, uses checked state for toggles, explicit key actions for chords, and native wheel input for diff --git a/apps/E2E/src/common/BasePage.ts b/apps/E2E/src/common/BasePage.ts index 15f3b5596f..4de0e8a35c 100644 --- a/apps/E2E/src/common/BasePage.ts +++ b/apps/E2E/src/common/BasePage.ts @@ -89,7 +89,8 @@ export abstract class BasePage { const pageButton = await this._pageButton; // sendKeys will move focus to the button, so we dont have to worry about the button being offscreen in the scrollable area. await this.sendKeys(pageButton, [Keys.ENTER]); - } else { + } else if (!isNativeDesktop() || (await this._pageButton.isEnabled())) { + // The tester disables the selected page's button; specs that share a page (e.g. IconLegacy and IconV1) find it already open. await this._pageButton.click(); } diff --git a/apps/E2E/src/common/nativeDesktop.ts b/apps/E2E/src/common/nativeDesktop.ts index 675b0bf1d6..8d7f8146cd 100644 --- a/apps/E2E/src/common/nativeDesktop.ts +++ b/apps/E2E/src/common/nativeDesktop.ts @@ -33,19 +33,31 @@ export async function sendNativeDesktopKeys(element: ChainablePromiseElement, ke if (failure !== undefined) throw failure; } +async function isCenterWithinScroller(element: ChainablePromiseElement, scroller: ChainablePromiseElement): Promise { + if (!(await element.isDisplayed())) return false; + const target = await browser.getElementRect(await element.elementId); + const viewport = await browser.getElementRect(await scroller.elementId); + const x = target.x + target.width / 2; + const y = target.y + target.height / 2; + return x > viewport.x && y > viewport.y && x < viewport.x + viewport.width && y < viewport.y + viewport.height; +} + export async function scrollNativeDesktopElement(element: ChainablePromiseElement, scroller: ChainablePromiseElement): Promise { - if (await element.isDisplayed()) return; + // A partly clipped control still reports displayed, but the driver clicks its center point, which may be covered. + if (await isCenterWithinScroller(element, scroller)) return; if (!browser.capabilities['furn:features']?.wheel) throw new Error('This desktop does not support native wheel input; navigation cannot be qualified.'); + // Native wheel input goes to the topmost window under the pointer, so the tester window must be in front first. + await browser.switchToWindow(await browser.getWindowHandle()); const origin = { 'element-6066-11e4-a52e-4f735466cecf': await scroller.elementId }; await browser.waitUntil( async () => { await browser.performActions([ { type: 'wheel', id: 'tester-scroll', actions: [{ type: 'scroll', origin, x: 0, y: 0, deltaX: 0, deltaY: 300, duration: 100 }] }, ]); - return element.isDisplayed(); + return isCenterWithinScroller(element, scroller); }, - { timeout: 30_000, timeoutMsg: 'The native scroll container did not expose the selected tester control.' }, + { timeout: 30_000, timeoutMsg: 'The native scroll container did not bring the selected tester control into view.' }, ); } diff --git a/driver-next-plan.md b/driver-next-plan.md index 2bb5095921..f9aa1c6168 100644 --- a/driver-next-plan.md +++ b/driver-next-plan.md @@ -54,6 +54,32 @@ pass does not attribute the availability change to the adapter fixes; no input policy was relaxed. These are implementation and limited app evidence, not complete behavioral parity. +**macOS:** after merging `origin/main` (`60121de61`), the native helper builds, +handshakes, self-tests, and reports ready permissions. The opt-in driver tests +pass 68 with two skipped. Five macOS-only defects were fixed: + +- the reporting root is canonicalized (`/var` resolves to `/private/var`); +- stop frames are no longer written to producers whose Unix sockets have half-closed; +- the Storybook CLI requests the renderer-scoped target ID its embedded driver registers; +- the RN Test App adapter launches the exact built product, not any Launch Services copy; +- that product fails closed on a bundle identifier, renderer, or embedded-bundle mismatch. + +Two app-side fixes followed: + +- The E2E page adapter scrolls tester buttons until their click point is unobscured. +- The repository RNmacOS patch exposes accessible Fabric `` paragraphs, whose + `testID`s were otherwise invisible. + +Results: + +| Lane | Fabric | Paper | +| -------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------- | +| Storybook lifecycle | 280/280 (`812ed972-9b9f-4862-ac54-4ae7f2a4d837`, with the patch) | 280/280 (`a8a511be-e082-4700-8546-e52694a5e941`) | +| FluentTester pilot | 2/2 (`1780e120-5560-44c2-80b9-c01aae342589`) | 2/2 | +| FluentTester `legacy` (62 tests) | all pass (`2673c35b-a00d-44d9-8fb6-2015655439fa`) | all pass (`202e7daf-a7c9-45cc-8bfe-07f4a812a9fe`) | + +None of these runs has skips or retries. Storybook reports are under `apps/storybook/artifacts/desktop/`; FluentTester reports are under `apps/E2E/artifacts/desktop/`. + **Repository validation:** package checks, publishing, changesets, and all changed Markdown links pass. After `yarn clean-lage-cache`, the final aggregate passes with `yarn lage buildci --concurrency 4`. Initial runs exposed missing @@ -65,9 +91,10 @@ remains undiagnosed. The separate opt-in Windows driver run passes 68 tests with two macOS-only tests skipped. The aggregate Markdown-link task remains blocked by Windows zx shell quoting; direct checks of all changed files pass. -**Still open:** per-test assertion mapping and equivalent-build old/new comparisons; actual -FluentTester Windows and macOS renderer/runtime qualification; macOS native -compilation; all Storybook platform/renderer regressions; and real Actions +**Still open:** per-test assertion mapping and equivalent-build old/new comparisons; +actual FluentTester Windows renderer/runtime qualification; the remaining +Storybook platform/renderer regressions; an upstream or consumer-integrated +replacement for the RNmacOS Text accessibility patch; and real Actions grouping/summary/upload verification. The 100-run/14-day reliability gate and desktop Appium retirement cannot be completed by this implementation session. No lane is retired and no capability skip is counted as a native pass. diff --git a/packages/agentic/desktop-driver/native/macos/README.md b/packages/agentic/desktop-driver/native/macos/README.md index 779ac1ddac..9fed8f98ee 100644 --- a/packages/agentic/desktop-driver/native/macos/README.md +++ b/packages/agentic/desktop-driver/native/macos/README.md @@ -173,8 +173,9 @@ Launch honors an explicitly configured `launchTimeoutMs`, or uses the native 90-second default within the server's 120-second default deadline. If LaunchServices completes after timeout or cancellation, the late completion closes only a new application causally owned by that request, never a preexisting app. -This changed timeout path still needs matching-macOS build and runtime -qualification; Windows tests do not validate it. +On October 22, 2026 (UTC), the helper built, handshook, and self-tested on +macOS, and real FluentTester launches with a configured 180-second timeout +succeeded. The late-completion branch remains covered only by code review. Leases record whether the app was launched or attached. Session deletion preserves attached apps and closes only an exact process owned by the helper. diff --git a/packages/agentic/desktop-test/README.md b/packages/agentic/desktop-test/README.md index 042326817e..d34fbfb47b 100644 --- a/packages/agentic/desktop-test/README.md +++ b/packages/agentic/desktop-test/README.md @@ -43,11 +43,11 @@ names must supply `endpoint`. Renderer is optional for ordinary native apps. the default permits owned launch only. Windows identity requires an exact title and an executable, AUMID, or verified owner lease, never a title alone. -| Host adapter | Behavior | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `installed` | Use the trusted application descriptor; build/prep/bundle/run stages require explicit commands. | -| `react-native-test-app` | Derive native commands and identity from `app.json`. Windows verifies the generated renderer and registered package identity. | -| `rex` | Win32 prebuilt host with declared `component` and `windowTitle`; use the existing bundle, executable, working directory, and fixed arguments. Native build is unsupported. | +| Host adapter | Behavior | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `installed` | Use the trusted application descriptor; build/prep/bundle/run stages require explicit commands. | +| `react-native-test-app` | Derive native commands and identity from `app.json`. Windows verifies the generated renderer and registered package identity; macOS launches the exact built product after verifying its bundle identifier, renderer, and embedded bundle. | +| `rex` | Win32 prebuilt host with declared `component` and `windowTitle`; use the existing bundle, executable, working directory, and fixed arguments. Native build is unsupported. | `commands` overrides individual `prep`, `bundle`, `build`, or `run` stages. Each value is `{ command, args?, cwd?, env? }`, an ordered array of commands, or @@ -117,7 +117,13 @@ URL/target, and waits for an owned stop; clients create sessions explicitly. `--json` keeps stdout machine-readable and diagnostics on stderr. `prep`, `bundle`, `build`, and `run` expose supported adapter stages. macOS RN Test App targets accept `--paper`; regenerate with `prep` after changing -renderer. Native tests use verified prebuilt helpers only: run `build-driver` +renderer. Run `bundle` before `prep` there: CocoaPods embeds only `app.json` +resources that exist during `pod install`. Without explicit `application` +identity, macOS sessions launch the single app in +`macos/DerivedData/Build/Products/Debug` rather than any Launch Services copy +sharing its bundle identifier, and fail closed when its `CFBundleIdentifier`, +`RCTNewArchEnabled` renderer, or declared `.jsbundle` resource does not match. +Native tests use verified prebuilt helpers only: run `build-driver` explicitly first, or provision a compatible helper/cache through [`nativeDriver` settings](../desktop-driver/references/native-helpers.md). diff --git a/packages/agentic/desktop-test/src/adapters.ts b/packages/agentic/desktop-test/src/adapters.ts index 0062735562..ee252d2ce5 100644 --- a/packages/agentic/desktop-test/src/adapters.ts +++ b/packages/agentic/desktop-test/src/adapters.ts @@ -48,7 +48,11 @@ export async function resolveDesktopApplication( const windowTitle = component?.displayName ?? app.displayName ?? app.name; if (!windowTitle) throw new Error('RN Test App manifest does not define its display identity.'); if (target.endpoint === 'macos') { - application = { bundleIdentifier: app.macos?.bundleIdentifier ?? 'com.microsoft.ReactTestApp', windowTitle, ...configured }; + const bundleIdentifier = app.macos?.bundleIdentifier ?? 'com.microsoft.ReactTestApp'; + application = + configured.bundleIdentifier || configured.executablePath || configured.leasePath + ? { bundleIdentifier, windowTitle, ...configured } + : { executablePath: await resolveBuiltMacosApplication(target, app, bundleIdentifier, signal), windowTitle, ...configured }; } else { const propertiesPath = path.join(target.projectRoot, 'windows', 'ExperimentalFeatures.props'); const properties = fs.readFileSync(propertiesPath, 'utf8'); @@ -143,7 +147,58 @@ type AppManifest = { displayName?: string; components?: { appKey?: string; displayName?: string }[]; macos?: { bundleIdentifier?: string }; + resources?: string[] | { macos?: string[] }; }; + +// Every RN Test App copy shares one bundle identifier, so Launch Services may pick a stale build from +// another checkout. Resolve this project's built product and launch that exact executable instead. +async function resolveBuiltMacosApplication( + target: ResolvedDesktopTarget, + app: AppManifest, + bundleIdentifier: string, + signal?: AbortSignal, +): Promise { + const products = path.join(target.projectRoot, 'macos', 'DerivedData', 'Build', 'Products', 'Debug'); + const bundles = fs.existsSync(products) ? fs.readdirSync(products).filter((name) => name.endsWith('.app')) : []; + if (bundles.length !== 1) { + throw new Error( + bundles.length === 0 + ? `Built macOS application missing in ${products}; run the build stage or configure a verified application identity.` + : `Expected one built macOS application in ${products} but found ${bundles.join(', ')}; remove stale products and rebuild.`, + ); + } + const bundlePath = path.join(products, bundles[0]); + const infoPath = path.join(bundlePath, 'Contents', 'Info.plist'); + const { stdout } = await promisify(execFile)('plutil', ['-convert', 'json', '-o', '-', infoPath], { + timeout: 10_000, + signal, + maxBuffer: 1024 * 1024, + }); + const info = JSON.parse(stdout) as Record; + if (info.CFBundleIdentifier !== bundleIdentifier) + throw new Error(`Built macOS application ${bundlePath} does not have the expected bundle identifier ${bundleIdentifier}.`); + if (typeof info.RCTNewArchEnabled !== 'boolean') + throw new Error(`Built macOS application ${bundlePath} does not declare RCTNewArchEnabled; run prep and build again.`); + const actualRenderer = info.RCTNewArchEnabled ? 'fabric' : 'paper'; + if (!target.host.renderer || target.host.renderer !== actualRenderer) { + throw new Error( + `Configured macOS renderer ${target.host.renderer ?? '(not specified)'} does not match built ${actualRenderer}; run prep and build for the configured renderer.`, + ); + } + const resources = Array.isArray(app.resources) ? app.resources : (app.resources?.macos ?? []); + for (const resource of resources.filter((name) => name.endsWith('.jsbundle'))) { + if (!fs.existsSync(path.join(bundlePath, 'Contents', 'Resources', path.basename(resource)))) { + throw new Error( + `Built macOS application does not embed ${path.basename(resource)}. CocoaPods only includes resources that exist during pod install: run bundle, then prep, then build.`, + ); + } + } + const executable = info.CFBundleExecutable; + if (typeof executable !== 'string' || !/^[^/\\]+$/.test(executable)) + throw new Error(`Built macOS application ${bundlePath} has an invalid CFBundleExecutable.`); + return path.join(bundlePath, 'Contents', 'MacOS', executable); +} + function readAppManifest(target: ResolvedDesktopTarget): AppManifest { const filename = target.host.appConfigPath ?? path.join(target.projectRoot, 'app.json'); const value: unknown = JSON.parse(fs.readFileSync(filename, 'utf8')); diff --git a/packages/agentic/desktop-test/src/contracts.contract.cjs b/packages/agentic/desktop-test/src/contracts.contract.cjs index ccff35d9ac..6cbcf1248d 100644 --- a/packages/agentic/desktop-test/src/contracts.contract.cjs +++ b/packages/agentic/desktop-test/src/contracts.contract.cjs @@ -31,7 +31,8 @@ async function main() { const { NodeDesktopCommandRunner } = await import(lib('commands.js')); const { DesktopServiceScope } = await import(lib('services.js')); const { loadDesktopConfig, defineDesktopConfig, resolveDesktopSpecs } = await import(lib('config.js')); - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'desktop-harness-contract-')); + // Config resolution canonicalizes paths; macOS temporary directories are symlinked (/var -> /private/var). + const root = fs.realpathSync.native(fs.mkdtempSync(path.join(os.tmpdir(), 'desktop-harness-contract-'))); const out = capture(); const err = capture(); const reporter = (options = {}) => @@ -78,21 +79,83 @@ async function main() { () => resolveDesktopSpecs({ kind: 'wdio', framework: 'jasmine', specs: ['missing.*'] }, 'windows', root), /refusing reduced coverage/, ); + } else if (mode === 'adapters') { + const { resolveDesktopApplication } = await import(lib('adapters.js')); + const project = path.join(root, 'app'); + const products = path.join(project, 'macos', 'DerivedData', 'Build', 'Products', 'Debug'); + const contents = path.join(products, 'ReactTestApp.app', 'Contents'); + const embedded = path.join(contents, 'Resources', 'index.macos.jsbundle'); + fs.mkdirSync(project); + fs.writeFileSync( + path.join(project, 'app.json'), + JSON.stringify({ + name: 'Fixture', + components: [{ appKey: 'fixture', displayName: 'Fixture Window' }], + resources: { macos: ['dist/assets', 'dist/index.macos.jsbundle'] }, + }), + ); + const target = (renderer, application) => ({ + endpoint: 'macos', + projectRoot: project, + host: { kind: 'react-native-test-app', renderer, application }, + }); + const leasePath = path.join(root, 'lease.json'); + assert.deepEqual( + { ...(await resolveDesktopApplication(target('paper', { leasePath }))) }, + { bundleIdentifier: 'com.microsoft.ReactTestApp', windowTitle: 'Fixture Window', leasePath }, + 'configured identity must not require local build products', + ); + if (process.platform === 'darwin') { + const writeInfo = ({ identifier = 'com.microsoft.ReactTestApp', newArch = 'false' } = {}) => + fs.writeFileSync( + path.join(contents, 'Info.plist'), + `CFBundleIdentifier${identifier}` + + `CFBundleExecutableReactTestApp${newArch ? `RCTNewArchEnabled<${newArch}/>` : ''}`, + ); + await assert.rejects(resolveDesktopApplication(target('paper')), /Built macOS application missing/); + fs.mkdirSync(path.join(contents, 'MacOS'), { recursive: true }); + fs.mkdirSync(path.join(contents, 'Resources')); + fs.writeFileSync(path.join(contents, 'MacOS', 'ReactTestApp'), ''); + fs.writeFileSync(embedded, ''); + writeInfo(); + assert.deepEqual( + { ...(await resolveDesktopApplication(target('paper'))) }, + { executablePath: path.join(contents, 'MacOS', 'ReactTestApp'), windowTitle: 'Fixture Window' }, + 'unconfigured identity must launch the exact built product, not a Launch Services bundle-identifier match', + ); + await assert.rejects(resolveDesktopApplication(target('fabric')), /renderer fabric does not match built paper/); + writeInfo({ newArch: 'true' }); + assert.equal((await resolveDesktopApplication(target('fabric'))).executablePath, path.join(contents, 'MacOS', 'ReactTestApp')); + writeInfo({ newArch: '' }); + await assert.rejects(resolveDesktopApplication(target('paper')), /does not declare RCTNewArchEnabled/); + writeInfo({ identifier: 'com.example.Other' }); + await assert.rejects(resolveDesktopApplication(target('paper')), /expected bundle identifier com\.microsoft\.ReactTestApp/); + writeInfo(); + fs.rmSync(embedded); + await assert.rejects(resolveDesktopApplication(target('paper')), /does not embed index\.macos\.jsbundle.*bundle, then prep/); + fs.writeFileSync(embedded, ''); + fs.mkdirSync(path.join(products, 'Stale.app')); + await assert.rejects(resolveDesktopApplication(target('paper')), /Expected one built macOS application/); + } } else if (mode === 'reporting') { const run = reporter({ secrets: ['secret-value'], console: 'github-actions' }); run.emit({ kind: 'output', payload: { stream: 'stderr', text: '::error::untrusted secret-value\n' } }); const error = run.failure(new Error('original secret-value', { cause: new Error('root cause') }), 'test'); const title = 'retries \u{1f680} <&\u0001'; + run.emit({ kind: 'test-start', payload: { testId: 'retry', title, attempt: 0 } }); run.emit({ kind: 'test-result', payload: { testId: 'retry', title, attempt: 0, status: 'failed', durationMs: 2, failure: error }, }); - run.emit({ kind: 'test-result', payload: { testId: 'retry', title, attempt: 1, status: 'passed', durationMs: 3 } }); + run.emit({ kind: 'test-start', payload: { testId: 'retry', title, attempt: 1_000_000 } }); + run.emit({ kind: 'test-result', payload: { testId: 'retry', title, attempt: 1_000_000, status: 'passed', durationMs: 3 } }); const result = await run.finish(); assert.equal(result.status, 'passed'); assert.equal(result.counts.flaky, 1); assert.equal(result.counts.selected, 1); assert.equal(result.counts.retries, 1); + assert(out.text().includes('(attempt 2)'), 'sparse attempt keys are labeled by ordinal'); + assert(!out.text().includes('(attempt 1000001)')); const journal = fs.readFileSync(path.join(run.root, 'events.jsonl'), 'utf8'); assert(!journal.includes('secret-value')); assert(journal.includes('root cause')); diff --git a/packages/agentic/desktop-test/src/contracts.test.ts b/packages/agentic/desktop-test/src/contracts.test.ts index 8b333f8a26..6702f2fcae 100644 --- a/packages/agentic/desktop-test/src/contracts.test.ts +++ b/packages/agentic/desktop-test/src/contracts.test.ts @@ -4,6 +4,7 @@ import path from 'node:path'; describe('neutral desktop harness contracts', () => { test.each([ 'config', + 'adapters', 'reporting', 'commands', 'transport', diff --git a/packages/agentic/desktop-test/src/reporting/collector.ts b/packages/agentic/desktop-test/src/reporting/collector.ts index dfaa761417..86390fb9ac 100644 --- a/packages/agentic/desktop-test/src/reporting/collector.ts +++ b/packages/agentic/desktop-test/src/reporting/collector.ts @@ -75,8 +75,10 @@ export class DesktopRunReporter { constructor(options: DesktopRunReporterOptions) { this.options = options; - this.root = path.join(path.resolve(options.artifactsRoot), this.runId); - fs.mkdirSync(this.root, { recursive: true, mode: 0o700 }); + const root = path.join(path.resolve(options.artifactsRoot), this.runId); + fs.mkdirSync(root, { recursive: true, mode: 0o700 }); + // Producers report paths relative to canonical artifact roots (macOS temp directories are /var -> /private/var). + this.root = fs.realpathSync.native(root); this.actions = !options.structured && (options.console === 'github-actions' || ((!options.console || options.console === 'auto') && process.env.GITHUB_ACTIONS === 'true')); @@ -297,9 +299,12 @@ export class DesktopRunReporter { case 'service-failed': if (firstFailure) this.logger.error(`${prefix} FAILED ${event.payload.failure.error.message}`); break; - case 'test-start': - this.logger.log(`${prefix} TEST ${event.payload.title} (attempt ${event.payload.attempt + 1})`); + case 'test-start': { + // Producers may encode attempts sparsely (WDIO combines spec-file and test retries), so label the observed ordinal. + const ordinal = this.tests.get(event.payload.testId)!.attempts.findIndex(({ attempt }) => attempt === event.payload.attempt) + 1; + this.logger.log(`${prefix} TEST ${event.payload.title} (attempt ${ordinal})`); break; + } case 'test-result': this.logger.log( `${prefix} ${event.payload.status.toUpperCase()} ${event.payload.title}${event.payload.reason ? `: ${event.payload.reason}` : ''}`, diff --git a/packages/agentic/desktop-test/src/reporting/transport.ts b/packages/agentic/desktop-test/src/reporting/transport.ts index 0af657a0d0..771bbfa95a 100644 --- a/packages/agentic/desktop-test/src/reporting/transport.ts +++ b/packages/agentic/desktop-test/src/reporting/transport.ts @@ -97,7 +97,8 @@ export async function startReportTransport(reporter: DesktopRunReporter, onFatal clearTimeout(timer); if (!socket.destroyed && !socket.writableEnded) socket.end(JSON.stringify({ controlled: children.length }) + '\n'); }); - for (const [child] of children) child.write('{"stop":true}\n'); + // A producer that already half-closed after its terminal handshake only needs to be awaited. + for (const [child] of children) if (!child.readableEnded && !child.writableEnded) child.write('{"stop":true}\n'); return; } if (!isHello(frame) || frame.runId !== context.runId || !sameSecret(frame.nonce, nonce)) diff --git a/packages/agentic/native-core/README.md b/packages/agentic/native-core/README.md index f271b02877..e038df613d 100644 --- a/packages/agentic/native-core/README.md +++ b/packages/agentic/native-core/README.md @@ -374,6 +374,10 @@ onto the actual accessible element, removes the structural Fabric TextInput's extra Tab stop, and separates read-only single-line editing from disabled state on both renderers. Read-only editors remain selectable/focusable; disabled editors remain disabled. It also maps their `textbox` role through the existing converter. +Fabric paragraphs (``) set `accessible` now expose themselves as static +text on macOS. RNmacOS otherwise rejects them unconditionally because its iOS +element provider is compiled out, which hides every Text label and `testID` +from accessibility clients. These fixes belong in RNmacOS, not in a MenuItem-specific native host or a JavaScript accessibility-value workaround. The patch is used by this repository's diff --git a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts index 7bf3787f5f..235a0c1be9 100644 --- a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts +++ b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.test.ts @@ -3,9 +3,11 @@ import os from 'node:os'; import path from 'node:path'; import type { NativeDriverArtifact } from '@fluentui-react-native/desktop-driver'; +import type { ResolvedDesktopConfig, ResolvedDesktopTarget } from '@fluentui-react-native/desktop-test'; +import { selectDesktopRenderer } from '@fluentui-react-native/desktop-test/config'; import { STORYBOOK_SMOKE_MODE } from '../config/commands'; -import { makeDesktopStorybookConfig } from '../config/makeDesktopStorybookConfig'; +import { DesktopStorybookConfig, makeDesktopStorybookConfig } from '../config/makeDesktopStorybookConfig'; import { FURN_STORYBOOK_BUNDLE_IDENTIFIER, FURN_STORYBOOK_INSTANCE_ID } from '../config/instance'; import { FURN_STORYBOOK_PLATFORM } from '../config/platforms'; import type { DesktopCommandRunner, PreparedDesktopCommand, RunningDesktopCommand } from './commandRunner'; @@ -764,6 +766,56 @@ describe('createDesktopStorybookCommand', () => { expect(runner.stopped).toBe(2); }); + test.each([[[]], [['--paper']]])('requests the canonical target registered by the driver manifest (%j)', async (flags) => { + const runner = new RecordingRunner(); + const runSmokeTests = jest.fn(async ({ targetId }: { targetId: string }) => ({ + endpoint: 'macos' as const, + finishedAt: '2026-10-02T08:00:01.000Z', + manifest: { platform: 'macos-digest', portable: 'portable-digest' }, + platformName: 'macos' as const, + runId: 'canonical-smoke', + schemaVersion: 1 as const, + startedAt: '2026-10-02T08:00:00.000Z', + status: 'passed' as const, + targetId, + tests: [], + })); + const target = { + configPath: path.join(storybookRoot, 'desktop.config.mts'), + endpoint: 'macos', + host: { kind: 'react-native-test-app', renderer: 'fabric' }, + id: 'macos-configured', + instanceId: 'configured', + projectRoot: storybookRoot, + } as unknown as ResolvedDesktopTarget; + const config = new DesktopStorybookConfig( + { + projectRoot: storybookRoot, + storyPackages: [], + platformOptions: { macos: { run: { command: 'launch-storybook' }, smoke: { stop: { command: 'stop-storybook' } } } }, + }, + { targets: { macos: target } } as unknown as ResolvedDesktopConfig, + ); + const program = createDesktopStorybookCommand({ + ...nativeDriverTestOptions, + config, + createStoryManifest: createEmptyStoryManifest, + fetch: async () => new Response(JSON.stringify({ entries: { story: { id: 'first--story', type: 'story' } } })), + isPortAvailable: async () => true, + output: { write: () => true }, + runner, + runSmokeTests, + }); + + await program.parseAsync(['node', 'storybook', 'smoke', '--macos', ...flags, '--mode', 'stories-and-tests']); + + const manifest = JSON.parse( + fs.readFileSync(path.join(storybookRoot, 'storybook-desktop.generated', 'driver-manifest.macos.json'), 'utf8'), + ); + expect(manifest.targetId).toBe(selectDesktopRenderer(target, flags.length ? 'paper' : 'fabric').id); + expect(runSmokeTests).toHaveBeenCalledWith(expect.objectContaining({ platform: 'macos', targetId: manifest.targetId })); + }); + test.each(['windows', 'win32'])('rejects --paper with explicit --%s', async (platform) => { const runner = new RecordingRunner(); const program = createDesktopStorybookCommand({ config: makeConfig(), runner }); diff --git a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts index dd102b65cf..33755cd51a 100644 --- a/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts +++ b/packages/agentic/storybook-desktop/src/cli/DesktopStorybookCli.ts @@ -34,6 +34,7 @@ import { writeDesktopStorybookDriverManifest, writeDesktopStoryManifest, } from '../driver/index.js'; +import { desktopStorybookTargetId } from '../driver/driverManifest.js'; import { NodeDesktopCommandRunner, STORYBOOK_VERBOSE } from './commandRunner.js'; import type { DesktopCommandRunner, PreparedDesktopCommand, RunningDesktopCommand } from './commandRunner.js'; import { formatDesktopStorybookSmokeTestSummary, runDesktopStorybookSmokeTests } from './smokeTests.js'; @@ -290,7 +291,7 @@ export class DesktopStorybookCli { manifest.driverPort < 1 || manifest.driverPort > 65535 || !('targetId' in manifest) || - manifest.targetId !== `${this.config.appName}-${platform}`.toLowerCase() + manifest.targetId !== this.targetId(platform) ) { throw new Error(`Invalid Storybook driver instance at ${manifestPath}; restart storybook driver --${platform}.`); } @@ -304,7 +305,7 @@ export class DesktopStorybookCli { ...this.instance, endpoint: platform, renderer: this.renderer(platform), - targetId: `${this.config.appName}-${platform}`.toLowerCase(), + targetId: this.targetId(platform), testIDPrefix: this.config.testIDPrefix, }, null, @@ -446,7 +447,7 @@ export class DesktopStorybookCli { driverUrl: loopbackUrl(instance.driverPort), manifest: await this.createStoryManifest(this.config, platform), platform, - targetId: `${this.config.appName}-${platform}`.toLowerCase(), + targetId: this.targetId(platform), report: this.reporter, signal: this.services.controller.signal, }), @@ -534,6 +535,10 @@ export class DesktopStorybookCli { return outputPath; } + private targetId(platform: Platforms): string { + return desktopStorybookTargetId(this.config, platform, this.renderer(platform)); + } + private renderer(platform: Platforms): 'fabric' | 'paper' { return platform === 'win32' || (platform === 'macos' && this.paper) ? 'paper' : 'fabric'; } diff --git a/packages/agentic/storybook-desktop/src/driver/driverManifest.ts b/packages/agentic/storybook-desktop/src/driver/driverManifest.ts index d73a730baa..52f280894e 100644 --- a/packages/agentic/storybook-desktop/src/driver/driverManifest.ts +++ b/packages/agentic/storybook-desktop/src/driver/driverManifest.ts @@ -44,6 +44,17 @@ export type CreateDesktopStorybookDriverManifestOptions = { storyManifest: DesktopStoryManifest; }; +/** Returns the WebDriver target ID the embedded Storybook driver registers for an endpoint and renderer. */ +export function desktopStorybookTargetId( + config: Pick, + platform: Platforms, + renderer: DesktopStorybookDriverManifest['renderer'], +): string { + const configuredTarget = config.getDesktopTarget?.(platform); + const target = configuredTarget && platform === 'macos' ? selectDesktopRenderer(configuredTarget, renderer) : configuredTarget; + return target?.id ?? `${config.appName}-${platform}`.toLowerCase(); +} + export function createDesktopStorybookDriverManifest({ bridgeNonce = randomBytes(24).toString('base64url'), config, @@ -58,8 +69,6 @@ export function createDesktopStorybookDriverManifest({ throw new Error(`Native Desktop Driver is disabled for ${platform}.`); } const leasePath = path.join(config.projectRoot, 'storybook-desktop.generated', `application-lease.${platform}.json`); - const configuredTarget = config.getDesktopTarget?.(platform); - const target = configuredTarget && platform === 'macos' ? selectDesktopRenderer(configuredTarget, renderer) : configuredTarget; return Object.freeze({ application: Object.freeze({ ...nativeOptions.application, @@ -81,7 +90,7 @@ export function createDesktopStorybookDriverManifest({ schemaVersion: 2, storyManifest, storybookPort: instance.storybookPort, - targetId: target?.id ?? `${config.appName}-${platform}`.toLowerCase(), + targetId: desktopStorybookTargetId(config, platform, renderer), testIDPrefix: config.testIDPrefix, wdio: config.wdio, }); diff --git a/yarn.lock b/yarn.lock index 320b66c417..d3e2e2f1e3 100644 --- a/yarn.lock +++ b/yarn.lock @@ -21869,7 +21869,7 @@ __metadata: "react-native-macos@patch:react-native-macos@npm%3A0.81.9#~/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch": version: 0.81.9 - resolution: "react-native-macos@patch:react-native-macos@npm%3A0.81.9#~/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch::version=0.81.9&hash=e2b496" + resolution: "react-native-macos@patch:react-native-macos@npm%3A0.81.9#~/.yarn/patches/react-native-macos-npm-0.81.9-aa583a0648.patch::version=0.81.9&hash=288fc3" dependencies: "@jest/create-cache-key-function": "npm:^29.7.0" "@react-native-macos/virtualized-lists": "npm:0.81.9" @@ -21915,7 +21915,7 @@ __metadata: bin: react-native: cli.js react-native-macos: cli.js - checksum: 10c0/06109070d3f83a2200b279929156004171ef55f3ab2eedb2500fbe45573fac289819b011b9dd1a0e0e012cab66a9c2ab4951b29d131a822e4eb4df65e67e358c + checksum: 10c0/cf1919114a1b4236d68bcd6fd2833b448859bde13ac395db8ebc0e65cce764c30179385c7c1de02a13d2f302ec6a86fdc7cbb3cd6d26fff1873d1732390a87aa languageName: node linkType: hard From a27723f13ea4147f66e48efa19adaa169032dd3e Mon Sep 17 00:00:00 2001 From: Jason Morse Date: Thu, 22 Oct 2026 14:20:29 -0700 Subject: [PATCH 4/8] embed instructions in packages --- .changeset/desktop-package-guidance.md | 9 + .../agentic-storybook-development/SKILL.md | 46 +- .github/skills/desktop-testing/SKILL.md | 37 + .../skills/maintain-desktop-testing/SKILL.md | 43 + AGENTS.md | 38 +- CONTRIBUTING.md | 5 + apps/fluent-tester/README.md | 4 + apps/storybook/AGENTS.md | 245 +--- apps/storybook/README.md | 396 +----- apps/storybook/agent-map.yaml | 4 +- apps/win32/README.md | 4 + driver-next-plan.md | 1066 ----------------- lage.config.mjs | 14 +- package.json | 1 + packages/agentic/desktop-driver/AGENTS.md | 38 +- packages/agentic/desktop-driver/README.md | 7 +- packages/agentic/desktop-driver/package.json | 14 + .../desktop-driver/references/README.md | 20 + .../desktop-driver/references/architecture.md | 7 + .../references/ci-integration.md | 7 + .../desktop-driver/references/contributing.md | 13 +- .../references/getting-started.md | 7 + .../references/native-helpers.md | 9 + .../desktop-driver/references/protocol.md | 9 +- .../desktop-driver/references/security.md | 7 + .../desktop-driver/references/service.md | 15 +- .../references/test-integration.md | 11 +- packages/agentic/desktop-test/AGENTS.md | 34 + packages/agentic/desktop-test/README.md | 181 +-- packages/agentic/desktop-test/package.json | 11 + .../agentic/desktop-test/references/README.md | 18 + .../desktop-test/references/commands.md | 32 + .../desktop-test/references/configuration.md | 52 + .../desktop-test/references/contributing.md | 21 + .../desktop-test/references/lifecycle.md | 25 + .../desktop-test/references/platforms.md | 30 + .../desktop-test/references/reporting.md | 30 + .../desktop-test/references/wdio-suites.md | 36 + .../storybook-desktop-runtime/AGENTS.md | 31 + .../storybook-desktop-runtime/README.md | 49 +- .../storybook-desktop-runtime/package.json | 12 + .../references/README.md | 13 + .../references/integration.md | 25 + .../references/runtime-contract.md | 40 + packages/agentic/storybook-desktop/AGENTS.md | 29 + packages/agentic/storybook-desktop/README.md | 525 +------- .../agentic/storybook-desktop/package.json | 12 + .../storybook-desktop/references/README.md | 20 + .../storybook-desktop/references/commands.md | 67 ++ .../references/contributing.md | 21 + .../references/integration.md | 62 + .../references/platforms/macos.md | 23 + .../references/platforms/win32.md | 21 + .../references/platforms/windows.md | 20 + .../storybook-desktop/references/readiness.md | 31 + .../references/story-tests.md | 75 ++ .../references/troubleshooting.md | 26 + .../storybook-desktop/src/cli/README.md | 225 +--- references/agents/README.md | 3 + .../components/agentic/tests-and-stories.md | 11 +- .../agents/components/classification.md | 4 +- .../agents/components/legacy-test-apps.md | 4 +- ...p-guidance-helper-permissions-holdout.json | 45 + ...-guidance-storybook-readiness-holdout.json | 56 + ...op-guidance-wdio-page-adapter-holdout.json | 48 + .../desktop-package-review-readonly.json | 31 + .../cases/desktop-test-new-app-target.json | 40 + references/agents/repository/development.md | 1 + .../repository/package-guidance-packages.json | 9 + .../agents/repository/package-guidance.md | 66 + references/agents/repository/permissions.md | 2 +- scripts/src/cli.ts | 2 + scripts/src/tasks/agentInstructions.test.ts | 137 +++ scripts/src/tasks/agentInstructions.ts | 218 +++- .../src/tasks/checkAgentInstructionsTask.ts | 6 +- scripts/src/tasks/checkPackageContentsTask.ts | 127 ++ 76 files changed, 2071 insertions(+), 2612 deletions(-) create mode 100644 .changeset/desktop-package-guidance.md create mode 100644 .github/skills/desktop-testing/SKILL.md create mode 100644 .github/skills/maintain-desktop-testing/SKILL.md delete mode 100644 driver-next-plan.md create mode 100644 packages/agentic/desktop-driver/references/README.md create mode 100644 packages/agentic/desktop-test/AGENTS.md create mode 100644 packages/agentic/desktop-test/references/README.md create mode 100644 packages/agentic/desktop-test/references/commands.md create mode 100644 packages/agentic/desktop-test/references/configuration.md create mode 100644 packages/agentic/desktop-test/references/contributing.md create mode 100644 packages/agentic/desktop-test/references/lifecycle.md create mode 100644 packages/agentic/desktop-test/references/platforms.md create mode 100644 packages/agentic/desktop-test/references/reporting.md create mode 100644 packages/agentic/desktop-test/references/wdio-suites.md create mode 100644 packages/agentic/storybook-desktop-runtime/AGENTS.md create mode 100644 packages/agentic/storybook-desktop-runtime/references/README.md create mode 100644 packages/agentic/storybook-desktop-runtime/references/integration.md create mode 100644 packages/agentic/storybook-desktop-runtime/references/runtime-contract.md create mode 100644 packages/agentic/storybook-desktop/AGENTS.md create mode 100644 packages/agentic/storybook-desktop/references/README.md create mode 100644 packages/agentic/storybook-desktop/references/commands.md create mode 100644 packages/agentic/storybook-desktop/references/contributing.md create mode 100644 packages/agentic/storybook-desktop/references/integration.md create mode 100644 packages/agentic/storybook-desktop/references/platforms/macos.md create mode 100644 packages/agentic/storybook-desktop/references/platforms/win32.md create mode 100644 packages/agentic/storybook-desktop/references/platforms/windows.md create mode 100644 packages/agentic/storybook-desktop/references/readiness.md create mode 100644 packages/agentic/storybook-desktop/references/story-tests.md create mode 100644 packages/agentic/storybook-desktop/references/troubleshooting.md create mode 100644 references/agents/evals/cases/desktop-guidance-helper-permissions-holdout.json create mode 100644 references/agents/evals/cases/desktop-guidance-storybook-readiness-holdout.json create mode 100644 references/agents/evals/cases/desktop-guidance-wdio-page-adapter-holdout.json create mode 100644 references/agents/evals/cases/desktop-package-review-readonly.json create mode 100644 references/agents/evals/cases/desktop-test-new-app-target.json create mode 100644 references/agents/repository/package-guidance-packages.json create mode 100644 references/agents/repository/package-guidance.md create mode 100644 scripts/src/tasks/checkPackageContentsTask.ts diff --git a/.changeset/desktop-package-guidance.md b/.changeset/desktop-package-guidance.md new file mode 100644 index 0000000000..eab0559a6b --- /dev/null +++ b/.changeset/desktop-package-guidance.md @@ -0,0 +1,9 @@ +--- +"@fluentui-react-native/desktop-driver": patch +"@fluentui-react-native/desktop-test": patch +"@fluentui-react-native/storybook-desktop": patch +"@fluentui-react-native/storybook-desktop-runtime": patch +--- + +Add package-owned agent guidance and user entry points, and restrict published +contents to the package files required at runtime and for documentation. diff --git a/.github/skills/agentic-storybook-development/SKILL.md b/.github/skills/agentic-storybook-development/SKILL.md index de68596067..de3e4a060a 100644 --- a/.github/skills/agentic-storybook-development/SKILL.md +++ b/.github/skills/agentic-storybook-development/SKILL.md @@ -1,31 +1,31 @@ --- name: agentic-storybook-development -description: Develop, run, bundle, or troubleshoot the agentic-components Storybook app on macOS or Windows, including CocoaPods, Xcode, Metro, Fabric, and offline builds. +description: Develop, bundle, build, or troubleshoot the agentic-components Storybook app in apps/storybook on macOS, Windows, or Win32, including app catalog configuration, Metro, CocoaPods, native projects, and renderer builds. Not for configuring or running desktop tests or changing desktop package source. license: MIT --- -# Agentic Storybook development +# Agentic Storybook app development -Work on the native Storybook application in `apps/storybook`. +This workflow covers the `apps/storybook` consumer and its native toolchain. +For the shared Storybook integration API and CLI use +[`storybook-desktop` references](../../../packages/agentic/storybook-desktop/references/README.md); +for app identity, discovery, and diagnostics use the app's +[`agent-map.yaml`](../../../apps/storybook/agent-map.yaml) and +[`AGENTS.md`](../../../apps/storybook/AGENTS.md). -## Workflow +1. Inspect `desktop.config.mts` and the app's declared scripts; do not invent + aliases or direct native runner commands. +2. Use the package's platform recipes for prep, bundle, build, smoke, and + diagnostics. Treat Pods, workspaces, generated solutions, build directories, + and DerivedData as disposable. Fix the owning hand-authored source. +3. Reinstall/regenerate affected native dependency sources and use a clean + build after dependency or workaround changes. +4. Validate the JavaScript bundle and affected native build. Run the app's + format/lint and root validation when shared manifests or references change. +5. Finish code/configuration changes with `yarn clean-lage-cache` then + `yarn lage buildci` (R-VALID-003). -1. Read the repository `AGENTS.md`, then the compact - [agent map](../../../apps/storybook/agent-map.yaml). Read the - [Storybook instructions](../../../apps/storybook/AGENTS.md), including its native diagnostics commands, and - [README](../../../apps/storybook/README.md) only as needed for the task. -2. Inspect the Storybook `package.json` and run its declared workspace scripts; do not invent direct runner commands. -3. Use `storybook-agent` to query runtime story IDs, select stories, and sweep renderability. Reproduce the first - failure from the Storybook workspace so pnpm-linked React Native tooling resolves from the app. -4. Treat native workspaces, Pods, lockfiles, generated solutions, build directories, and DerivedData as disposable - outputs. Fix the owning manifest, Podfile, configuration, or script instead. -5. After dependency or workaround changes, regenerate the affected native dependency source and use a clean build so - stale output cannot produce a false success. - When adding a codegen-enabled workspace dependency to existing macOS Pods, the first pod install can generate its - spec without refreshing ReactCodegen's source list. If the clean build then reports a missing generated spec header, - run `pods:macos` once more before changing native configuration. -6. Validate the JavaScript bundle and the affected native build. Run Storybook format/lint and the root build when - manifests or shared package references changed. Finish with `yarn clean-lage-cache` and then `yarn lage buildci` (R-VALID-003). - -Keep component API, slot, token, and rendering work in the `agentic-component-authoring` skill; use this skill for the -Storybook application and its native toolchains. +For test configuration, helper provisioning, test runs, smoke reports, or CI +lanes, use `desktop-testing`. For package source changes, use +`maintain-desktop-testing`. Component API and assertion work stays with +`agentic-component-authoring`. diff --git a/.github/skills/desktop-testing/SKILL.md b/.github/skills/desktop-testing/SKILL.md new file mode 100644 index 0000000000..7592ad6b5e --- /dev/null +++ b/.github/skills/desktop-testing/SKILL.md @@ -0,0 +1,37 @@ +--- +name: desktop-testing +description: Configure, run, or troubleshoot native desktop tests using desktop-test, desktop-driver, and storybook-desktop, including app targets and suites, helpers, permissions, WDIO suites, Storybook story tests and smoke runs, reports, and CI. Not for package source changes or the Storybook app build toolchain. +license: MIT +--- + +# Desktop testing + +Package references own commands, schemas, and troubleshooting. Read only the +matching topic; do not copy its content here. + +| Task | Read | +| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an app target or suite | [`desktop-test` configuration](../../../packages/agentic/desktop-test/references/configuration.md) | +| Run stages, suites, or inspect results | [`desktop-test` commands](../../../packages/agentic/desktop-test/references/commands.md) and [reporting](../../../packages/agentic/desktop-test/references/reporting.md) | +| Prepare a platform or provision a helper | [`desktop-test` platforms](../../../packages/agentic/desktop-test/references/platforms.md) and [`desktop-driver` helpers](../../../packages/agentic/desktop-driver/references/native-helpers.md) | +| Integrate Storybook or run callbacks | [`storybook-desktop` integration](../../../packages/agentic/storybook-desktop/references/integration.md), [commands](../../../packages/agentic/storybook-desktop/references/commands.md), and [tests](../../../packages/agentic/storybook-desktop/references/story-tests.md) | +| Fix E2E adapters or diagnose failures | [`desktop-test` WDIO suites](../../../packages/agentic/desktop-test/references/wdio-suites.md) and [lifecycle](../../../packages/agentic/desktop-test/references/lifecycle.md) | + +## Workflow + +1. Identify the consumer workspace; read its config and declared scripts. For + `apps/storybook`, also read `AGENTS.md` and `agent-map.yaml`. +2. Reproduce with the narrowest declared stage, suite, story, or filter. + Provision the verified helper explicitly; never guess flags or call runner + internals. +3. Do not change privacy settings without authorization or overlap interactive + desktop workloads. +4. Report failures, skips, unsupported capabilities, and incomplete work + distinctly. Skips are not passes. +5. If a published package changed, add its changeset. Finish code/config + changes with `yarn clean-lage-cache` then `yarn lage buildci`. + +For package source changes or reviews, use `maintain-desktop-testing`. For the +`apps/storybook` build/toolchain, use `agentic-storybook-development`. +Component story assertions stay with `agentic-component-authoring`; v0/v1 +FluentTester assertions stay with `maintain-v0-v1-component`. diff --git a/.github/skills/maintain-desktop-testing/SKILL.md b/.github/skills/maintain-desktop-testing/SKILL.md new file mode 100644 index 0000000000..10f29194ff --- /dev/null +++ b/.github/skills/maintain-desktop-testing/SKILL.md @@ -0,0 +1,43 @@ +--- +name: maintain-desktop-testing +description: Change, review, or plan source changes to desktop-driver, desktop-test, storybook-desktop, or storybook-desktop-runtime, including native helpers. Applies package invariants, documentation, native and consumer validation, and changesets. +license: MIT +--- + +# Maintain desktop testing packages + +| Owner | Entry | +| --------------------------------------------- | ---------------------------------------------------------------------------- | +| WebDriver service, clients, helpers | [`desktop-driver`](../../../packages/agentic/desktop-driver/AGENTS.md) | +| App/suite config, lifecycle, WDIO, reporting | [`desktop-test`](../../../packages/agentic/desktop-test/AGENTS.md) | +| Storybook integration, CLI, extraction, tests | [`storybook-desktop`](../../../packages/agentic/storybook-desktop/AGENTS.md) | +| On-device bridge, Metro, theme, chrome | [`runtime`](../../../packages/agentic/storybook-desktop-runtime/AGENTS.md) | + +Read the matching contributing reference from the package index. + +## Mode first + +- **Review/plan:** read-only. Do not implement, run write-mode formatting, + build, create changesets, launch apps, or run native contracts. Report + validation not run. +- **Implementation:** continue below. + +## Implementation + +1. Change the lowest package that owns the behavior; update dependent + references when needed. Preserve exact identity, fail-closed behavior, + cancellation, ownership-safe cleanup, and explicit unsupported/skip status. +2. For standalone Swift/C++ helpers, read the provider README and + `native/PROTOCOL.md`; keep protocol schema and docs in sync. Apply + `implement-native-component` as a native overlay. React-module and Office + bridging-header steps do not apply to these standalone helpers. +3. Update owning references and affected README/AGENTS files in the same + change. Keep `PLAN.md` to unfinished or deferred work. +4. Run package format, lint, build, and test. Native changes also need the + matching-OS helper contract and affected consumer lane; JS tests do not + qualify native behavior. +5. Add a patch changeset for each changed published package. Finish with + `yarn clean-lage-cache` then `yarn lage buildci`. + +When delegating, pass the package AGENTS.md and selected reference paths; +subagents do not inherit them automatically. diff --git a/AGENTS.md b/AGENTS.md index 2a4d226871..d2811c4b3d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,8 +6,9 @@ routes each task to the workflow and references it needs; load only what the tas ## How guidance is organized -- **`AGENTS.md` files** hold safeguards and local invariants. Copilot CLI loads this file plus the `AGENTS.md` in the - working directory and each ancestor. From the repository root, read the scoped files below when a task touches them. +- **`AGENTS.md` files** hold safeguards and local invariants. Copilot CLI loads applicable files from the repository, + working directory, ancestors, and directories in the path of files being worked on. Read the scoped entries below + when a task touches them. - **Skills** in [`.github/skills`](.github/skills) define task workflows. - **[Shared references](references/agents/README.md)** in `references/agents/` hold reusable recipes and repository knowledge, indexed by task. @@ -51,26 +52,33 @@ Classify the change with the [classification guide](references/agents/components | Review a component change of any generation | [review-component](.github/skills/review-component/SKILL.md) | | Native component, TurboModule, service, codegen, or a native-core `/legacy` wrapper | [implement-native-component](.github/skills/implement-native-component/SKILL.md) | | Cross-component agentic extraction or dependency direction | [agentic-component-optimization](.github/skills/agentic-component-optimization/SKILL.md) | -| Storybook app, Metro, CocoaPods, or desktop native toolchain | [agentic-storybook-development](.github/skills/agentic-storybook-development/SKILL.md) | +| Desktop test configuration, execution, reporting, helpers, or CI | [desktop-testing](.github/skills/desktop-testing/SKILL.md) | +| Change or review the desktop testing package source | [maintain-desktop-testing](.github/skills/maintain-desktop-testing/SKILL.md) | +| Storybook app catalog, Metro, CocoaPods, or native builds | [agentic-storybook-development](.github/skills/agentic-storybook-development/SKILL.md) | | Dependency vulnerability remediation | [fix-security-issues](.github/skills/fix-security-issues/SKILL.md) | Other repository work starts from the [reference index](references/agents/README.md): architecture, commands and validation, TypeScript, release, design tokens, focus, and native guidance. +For package-owned knowledge, follow the +[package guidance standard](references/agents/repository/package-guidance.md). ## Scoped instructions -| Path | Scope | -| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------ | -| [packages/agentic/components](packages/agentic/components/AGENTS.md) | Agentic package invariants, contracts, and story tests | -| [packages/agentic/components/src](packages/agentic/components/src/AGENTS.md) | Source boundaries and extraction | -| [.../src/components](packages/agentic/components/src/components/AGENTS.md) | Higher-order components and the focus visual policy | -| [.../src/primitives](packages/agentic/components/src/primitives/AGENTS.md) | Public primitives | -| [packages/agentic/desktop-driver](packages/agentic/desktop-driver/AGENTS.md) | Desktop WebDriver driver | -| [packages/framework-base](packages/framework-base/AGENTS.md) | Shared component patterns | -| [packages/framework-base/src/accessibility](packages/framework-base/src/accessibility/AGENTS.md) | Accessibility helpers and native action qualification | -| [packages/framework/framework](packages/framework/framework/AGENTS.md) | v1 framework package | -| [packages/deprecated/foundation-compose](packages/deprecated/foundation-compose/AGENTS.md) | v0 framework package | -| [apps/storybook](apps/storybook/AGENTS.md) | Storybook app, native workflows, and diagnostics | +| Path | Scope | +| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | +| [packages/agentic/components](packages/agentic/components/AGENTS.md) | Agentic package invariants, contracts, and story tests | +| [packages/agentic/components/src](packages/agentic/components/src/AGENTS.md) | Source boundaries and extraction | +| [.../src/components](packages/agentic/components/src/components/AGENTS.md) | Higher-order components and the focus visual policy | +| [.../src/primitives](packages/agentic/components/src/primitives/AGENTS.md) | Public primitives | +| [packages/agentic/desktop-driver](packages/agentic/desktop-driver/AGENTS.md) | Desktop WebDriver driver | +| [packages/agentic/desktop-test](packages/agentic/desktop-test/AGENTS.md) | Application-neutral desktop test harness | +| [packages/agentic/storybook-desktop](packages/agentic/storybook-desktop/AGENTS.md) | Storybook desktop integration | +| [packages/agentic/storybook-desktop-runtime](packages/agentic/storybook-desktop-runtime/AGENTS.md) | On-device Storybook runtime | +| [packages/framework-base](packages/framework-base/AGENTS.md) | Shared component patterns | +| [packages/framework-base/src/accessibility](packages/framework-base/src/accessibility/AGENTS.md) | Accessibility helpers and native action qualification | +| [packages/framework/framework](packages/framework/framework/AGENTS.md) | v1 framework package | +| [packages/deprecated/foundation-compose](packages/deprecated/foundation-compose/AGENTS.md) | v0 framework package | +| [apps/storybook](apps/storybook/AGENTS.md) | Storybook app, native workflows, and diagnostics | ## Quick facts diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ac4c660f1d..cdf43fba77 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,6 +13,11 @@ This guide assumes you: There are some specific quirks to this repository that one should be familiar with. +When configuring or maintaining a package, start with its `AGENTS.md` and +`README.md`; package-owned recipes and shared agent guidance live in its +indexed `references/` directory. Follow the +[package-owned guidance standard](./references/agents/repository/package-guidance.md). + ### Component generations The repository contains three generations of components. Identify the generation from a component's imports and diff --git a/apps/fluent-tester/README.md b/apps/fluent-tester/README.md index e0ea234389..51a4d3d80a 100644 --- a/apps/fluent-tester/README.md +++ b/apps/fluent-tester/README.md @@ -7,3 +7,7 @@ - [macOS](./docs/macos.md) - [Windows](./docs/windows.md) - [Win32](./../win32/README.md) + +For native desktop E2E configuration and suite execution, see the +`@fluentui-react-native/desktop-test` package guide at +`packages/agentic/desktop-test/README.md`. diff --git a/apps/storybook/AGENTS.md b/apps/storybook/AGENTS.md index 5821ffb000..290ea99699 100644 --- a/apps/storybook/AGENTS.md +++ b/apps/storybook/AGENTS.md @@ -1,191 +1,54 @@ -# Agentic Components Storybook development - -These instructions apply to `apps/storybook` and its descendants. - -Read [`agent-map.yaml`](agent-map.yaml) first for the compact architecture, lookup, and interaction map. Read this file, -`README.md`, `package.json`, and `desktop.config.mts` before changing the Storybook application or its native projects. - -## Agent-efficient discovery and interaction - -- Generate the platform manifest with `yarn storybook manifest --` instead of guessing IDs or searching the - sidebar. -- Query and run stories through `yarn desktop-driver` against the listener started by - `yarn storybook driver --`. -- Use screenshots only for visual evidence. Use WebSocket events and accessibility bounds for lookup and interaction. -- Treat `src/storybook.requires.ts` as generated output; use `desktop.config.mts` for story package discovery and - `src/main.ts` only as the shared config adapter. -- Keep `agent-map.yaml` synchronized when stable paths, scripts, services, or interaction contracts change. Do not list - individual stories there; the runtime index is authoritative. - -## Command and dependency discipline - -- From the repository root, invoke scripts with - `yarn workspace @fluentui-react-native/agentic-components-storybook