test: migrate Maestro e2e to the e2e runner (stable) - #1153
Open
okwasniewski wants to merge 9 commits into
Open
okwasniewski wants to merge 9 commits into
okwasniewski wants to merge 9 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The platform jobs miss shared source changes, and the new e2e TypeScript project is not statically checked in CI.
Review effort: Balanced
Findings: 3
Open (4)
What changed in this PR
Migrates mobile E2E coverage from Maestro YAML to the stable TypeScript e2e runner for iOS and Android.
Changes:
- Replaces 11 Maestro scenarios with deterministic TypeScript tests.
- Adds shared device setup, CI reporting, and multi-device configuration.
- Removes legacy Maestro scripts, flows, and skill documentation.
| File | Description |
|---|---|
tsconfig.json |
Excludes the standalone e2e project. |
skills-lock.json |
Replaces the Maestro skill lock. |
scripts/run-maestro-tests.sh |
Removes the Maestro runner. |
package.json |
Adds e2e build and run scripts. |
example/ios/Podfile.lock |
Updates generated pod versions/checksums. |
e2e/tsconfig.json |
Configures strict e2e TypeScript checking. |
e2e/tests/tab-view-inside-scroll-view.e2e.ts |
Migrates TabView/ScrollView coverage. |
e2e/tests/support/test.ts |
Adds app installation and relaunch hooks. |
e2e/tests/support/basic-pager.ts |
Shares basic pager assertions. |
e2e/tests/support/app.ts |
Adds navigation and layout helpers. |
e2e/tests/smoke.e2e.ts |
Adds the smoke scenario. |
e2e/tests/scrollable-pager-view.e2e.ts |
Migrates scrollable pager coverage. |
e2e/tests/on-page-selected.e2e.ts |
Migrates page-selection alert coverage. |
e2e/tests/nested-pager-view.e2e.ts |
Migrates nested pager coverage. |
e2e/tests/material-top-bar.e2e.ts |
Migrates material tabs coverage. |
e2e/tests/issues/issue-1098-nested-pager.e2e.ts |
Adds #1098 regression coverage. |
e2e/tests/issues/issue-1083-modal-set-page.e2e.ts |
Adds #1083 regression coverage. |
e2e/tests/basic-pager.e2e.ts |
Covers horizontal, vertical, and RTL paging. |
e2e/targets.ts |
Defines mobile targets and device pools. |
e2e/README.md |
Documents setup, execution, and CI. |
e2e/package.json |
Declares runner dependencies and commands. |
e2e/e2e.config.ts |
Configures timeouts, workers, and reporting. |
e2e/bun.lock |
Locks e2e dependencies. |
.maestro/tests/tab_view_inside_scroll_view_example.yaml |
Removes legacy TabView flow. |
.maestro/tests/scrollable_pagerView_example.yaml |
Removes legacy scroll flow. |
.maestro/tests/pager_vertical_basic_example.yaml |
Removes vertical pager flow. |
.maestro/tests/pager_rtl_example.yaml |
Removes RTL pager flow. |
.maestro/tests/pager_basic_example.yaml |
Removes horizontal pager flow. |
.maestro/tests/on_page_selected_example.yaml |
Removes selection flow. |
.maestro/tests/nested_pagerView_example.yaml |
Removes nested pager flow. |
.maestro/tests/material_top_bar_example.yaml |
Removes material tabs flow. |
.maestro/smoke-test.yaml |
Removes legacy smoke flow. |
.maestro/setup/tab_view_inside_scroll_view_example_setup.yaml |
Removes TabView setup. |
.maestro/setup/scrollable_pagerView_example_setup.yaml |
Removes scroll setup. |
.maestro/setup/on_page_selected_example_setup.yaml |
Removes selection setup. |
.maestro/setup/nested_pagerView_example_setup.yaml |
Removes nested pager setup. |
.maestro/setup/material_top_bar_example_setup.yaml |
Removes material tabs setup. |
.maestro/setup/issue_1098_nested_pager_repro_setup.yaml |
Removes #1098 setup. |
.maestro/setup/issue_1083_modal_set_page_repro_setup.yaml |
Removes #1083 setup. |
.maestro/README.md |
Removes Maestro documentation. |
.maestro/issues/issue_1098_nested_pager_repro.yaml |
Removes #1098 Maestro test. |
.maestro/issues/issue_1083_modal_set_page_repro.yaml |
Removes #1083 Maestro test. |
.maestro/flows/basic-pager/verify-vertical-swipe.yaml |
Removes vertical helper flow. |
.maestro/flows/basic-pager/verify-horizontal-rtl-swipe.yaml |
Removes RTL helper flow. |
.maestro/flows/basic-pager/verify-horizontal-ltr-swipe.yaml |
Removes LTR helper flow. |
.maestro/flows/basic-pager/verify-controls.yaml |
Removes controls helper flow. |
.maestro/flows/basic-pager/open.yaml |
Removes pager-opening helper. |
.maestro/flows/basic-pager/ensure-rtl.yaml |
Removes RTL setup helper. |
.maestro/flows/basic-pager/ensure-ltr.yaml |
Removes LTR setup helper. |
.gitignore |
Ignores e2e output artifacts. |
.github/workflows/ios.yml |
Runs and uploads iOS e2e results. |
.github/workflows/android.yml |
Runs and uploads Android e2e results. |
.agents/skills/maestro-mobile-testing/SKILL.md |
Removes the Maestro skill. |
.agents/skills/e2e/SKILL.md |
Adds e2e runner guidance. |
.agents/skills/e2e/references/writing-tests.md |
Documents test APIs. |
.agents/skills/e2e/references/setup.md |
Documents runner setup. |
.agents/skills/e2e/references/running.md |
Documents execution and CI. |
.agents/skills/e2e/references/mcp.md |
Documents MCP operation. |
.agents/skills/e2e/references/explore.md |
Documents exploratory runs. |
.agents/skills/e2e/references/debugging.md |
Documents failure diagnosis. |
.agents/skills/e2e/references/bug-bash.md |
Documents bug-bash workflows. |
.agents/skills/e2e/references/agent.md |
Documents agentic test steps. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - '.github/workflows/android.yml' | ||
| - 'android/**' | ||
| - 'example/android/**' | ||
| - 'e2e/**' |
| - '.github/workflows/ios.yml' | ||
| - 'ios/**' | ||
| - 'example/ios/**' | ||
| - 'e2e/**' |
| "test:ios": "e2e run --target ios", | ||
| "test:android": "e2e run --target android", | ||
| "list": "e2e list", | ||
| "typecheck": "tsc --noEmit" |
|
|
||
| `targets.ts` turns the list into a device pool. The engine boots every device | ||
| up front, declares one worker per device, and the runner spreads the test | ||
| files across them. 13 tests: about 375 s on one simulator, 190 s on two, 130 s |
- replace .maestro YAML flows 1:1 with TypeScript tests in e2e/ on the public stable packages (e2e 0.15.1, @e2e-dev/mobile 0.8.1); no agentic flows, no NPM_TOKEN - engine installs/launches nothing on stable: freshAppBeforeEach() in every file installs E2E_*_APP_PATH and relaunches via device.openApp relaunch - add @e2e-dev/github PR reporter; run suites in ios.yml/android.yml after the release build, upload .e2e report artifacts - sync .agents/skills/e2e to the skill shipped with stable, drop the maestro skill
okwasniewski
force-pushed
the
oskar/e2e-migration
branch
from
October 1, 2026 12:49
f345574 to
d0df17c
Compare
installApp replaces the binary and keeps data, so repeating it when E2E_*_APP_PATH is set (~2s) beats tracking per-device state across realms and pools
- issue_1096 keyboard shrink and issue_1142 search bar inset flows came in via rebase with no runner left; ported 1:1 to e2e/tests/issues (1142 pinned to ios: the UISearchController inset never applies on Android) - clamp pod deployment targets to 15.1: Xcode 27 rejects RNSVG's 12.4 resource-bundle targets, which RN's post_install skips - verified: ios 13/13, android 12/12 (1083/1096/1098 rechecked on emulator)
- trigger platform e2e on src/** and example/src/** changes - typecheck e2e project in the lint workflow - fix stale test count/timings in e2e README
tsconfig.build.json exclude replaces the base exclude, so bob's tsc swept e2e/ (deps only in e2e/node_modules) and prepare failed every CI install
- gradle inside the emulator step starved the 4-vcpu runner: System UI ANRed and the dialog blocked all 12 tests (seen in run artifacts) - assembleRelease x86_64 runs before the AVD boots; emulator step only installs the apk and runs the suite, with hide_error_dialogs 1 - android bundle was --dev true (ios already false): dev-mode JS is slow enough on swiftshader to trigger ANRs - add job timeouts (60m ios, 75m android)
- RTL restore raced I18nManager's async pref write: the toggle now reflects the pending direction via state, and the helper waits for the flip before the relaunch's force-stop - nested pager: swipe the inner pagers by their own testIDs with a slow drag; Android hands a fast fling on the shared horizontal axis to the outer pager (seen as landing on 3-rd-pager-view) - tab-view: scroll the ScrollView node, not the viewport; on 1920px screens the viewport centre lands inside the TabView pager - issue-1096 pinned ios-only like the original Maestro flow (Android adjustResize shrinks the window by design); refocus input before Enter - ios.yml detaches the simulator hardware keyboard: Simulator.app's default hides the software keyboard, leaving keyboardReturn nothing to press verified: ios 13/13 (2m26s), android 11/11 (4m01s) at CI's 1080x1920
a fling on the software-rendered CI emulator empties the a11y tree long enough for scrollUntilVisible to give up (CI artifact shows 27 bare nodes mid-scroll); verified on both platforms locally
tab-view fails only on CI: the AOSP image's accessibility tree drops the TabView contents mid-scroll (27 bare nodes in two runs' artifacts, passes locally at identical resolution); google_apis images are the standard for a11y-driven UI testing. Pin actions/cache like the other actions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the Maestro YAML suite (
.maestro/) 1:1 with a TypeScript e2e suite ine2e/, running on the public stable e2e runner (e2e@0.15.1+@e2e-dev/mobile@0.8.1). No private packages orNPM_TOKENneeded; no model/API key either - the suite is deterministic only (no agentic flows).freshAppBeforeEach()(installsE2E_IOS_APP_PATH/E2E_ANDROID_APK_PATHonce per device, relaunches the app fresh before each test viadevice.openApp(APP_ID, { relaunch: true })).ios.ymlandandroid.ymlnow run the suite after building the release example app and upload.e2e/report.json+ failure artifacts. The@e2e-dev/githubreporter posts a job summary and one PR comment per run (needspull-requests: write, passedGITHUB_TOKEN); it no-ops locally..agents/skills/e2esynced to the skill shipped with the stable package; Maestro skill andscripts/run-maestro-tests.shremoved.Test Plan
What's required for testing (prerequisites)?
Xcode with a simulator runtime or the Android SDK with an emulator (
npx agent-device doctor), plus the release example app build (bun run e2e:ios/bun run e2e:androiddoes bundle + build + install + run). Android gradle needs JDK <= 21 locally; CI pins zulu 17.What are the steps to reproduce (after prerequisites)?
Both suites ran green locally on both platforms; failures during bring-up surfaced real stable-runner semantics (resume-vs-relaunch, per-file hook registration, Android cleanup budget) and are handled in
e2e/tests/support/with the reasoning documented ine2e/README.md.Compatibility
Checklist
README.md