diff --git a/README.md b/README.md index 7ed2e1c..3db51b6 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ curl -s "$HOST/api/v1/products/$PRODUCT_ID/mobile_binary_files" -H "Authorizatio | jq '.mobile_binary_files[] | {id, filename, platform}' ``` -Listing binaries needs an owner token, so do it once yourself and store the ID — the CI token can use a binary ID without being able to list them. +Listing binaries needs an owner or engineer token, so do it once yourself and store the ID — the CI token can use a binary ID without being able to list them. You need no ID at all if CI uploads the build it just made; see [Fresh builds](#fresh-builds). To send browser traffic through a corporate proxy, list the configured ones and take the ID: @@ -144,7 +144,8 @@ Both channels take these: | `manufacturer` | | | Narrows auto-selection to this manufacturer | | `device-location` | | | Narrows auto-selection to this datacenter, e.g. `EU` or `US` | | `device-backend` | | `mobitru` | Device provider | -| `app-binary-id` | | | UUID of an uploaded binary to install before the run | +| `app-binary-path` | | | Path to an `.apk`, `.aab` or `.ipa` in the workspace. Uploaded before the run, so the session tests the build this workflow made | +| `app-binary-id` | | | UUID of an already uploaded binary to install before the run | | `app-package` | | | Package name (Android) or Bundle ID (iOS) of an app already on the device | | `mobile-browser` | | `false` | Test the device browser instead of an app | | `prerequisites` | | | Free-form setup notes to run before the checks, e.g. login steps | @@ -160,7 +161,8 @@ A mobile run needs `channel: mobile`, a `product-id` for a mobile product, and t **What runs on it.** Exactly one of: -- `app-binary-id` — install an uploaded APK or IPA first. +- `app-binary-path` — upload a build from the workspace and install that. See [Fresh builds](#fresh-builds). +- `app-binary-id` — install a binary that was uploaded earlier. - `app-package` — an app already installed on the device. Needs `device-serial`, since it has to be a device you picked. - `mobile-browser: true` — the device's own browser. Nothing is installed. @@ -185,6 +187,18 @@ Two things worth knowing: Concurrency also costs worker capacity on the installation, not just devices: each running check occupies one agent slot for its whole duration. Setting `max-concurrency` above the number of slots books devices that then sit idle waiting their turn, so it is worth checking with whoever runs the installation before going wide. +### Fresh builds + +`app-binary-id` installs a binary someone uploaded at some earlier point, which is rarely the code you just pushed. `app-binary-path` takes a file out of the runner's workspace instead: the action uploads it to the product and starts the session on it, so the run tests this commit's build. + +The action builds nothing. The file comes from the run itself: + +- **A build job in the same workflow.** Gradle leaves the APK under `app/build/outputs/apk/`, `xcodebuild -exportArchive` leaves the IPA in the export directory. Jobs get separate runners, so hand the file over with `actions/upload-artifact` and `actions/download-artifact`. +- **The same job.** Build and check in one job and the path is just the build output, with no artifact hop. +- **Somewhere else.** A release asset, a nightly build, an internal build service — download it into the workspace first with `gh release download` or `curl`, then point at the file. + +Uploading needs an owner or engineer token, the same level that lists binaries; a plain CI token can still run on an `app-binary-id`. The file must be an `.apk`, `.aab` or `.ipa` within the installation's size limit, 2 GB unless it was changed. Every run adds a binary to the product and nothing is replaced, so uploading on every commit grows that list. + ## Outputs | Name | Description | @@ -323,6 +337,44 @@ Run a mobile suite on an auto-selected Android phone, installing a binary: app-binary-id: ${{ vars.AGENTIC_QA_APP_BINARY }} ``` +Check the build this run produced, instead of whatever was uploaded last: + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: ./gradlew assembleDebug + - uses: actions/upload-artifact@v4 + with: + name: apk + path: app/build/outputs/apk/debug/app-debug.apk + + mobile-checks: + needs: build + runs-on: ubuntu-latest + steps: + - uses: actions/download-artifact@v4 + with: + name: apk + path: build + - uses: test-IO/agentic-qa-github-action@v1 + with: + host: ${{ vars.AGENTIC_QA_HOST }} + token: ${{ secrets.AGENTIC_QA_TOKEN }} + project-id: ${{ vars.AGENTIC_QA_PROJECT }} + check-suite-id: ${{ vars.AGENTIC_QA_MOBILE_SUITE }} + channel: mobile + product-id: ${{ vars.AGENTIC_QA_MOBILE_PRODUCT }} + device-platform: android + app-binary-path: build/app-debug.apk +``` + +An artifact keeps the file, not the directory it was built in, so the download lands it at `build/app-debug.apk`. + +For a release build, `assembleRelease` writes `app-release.apk` only when the release build type has a signing config. Without one you get `app-release-unsigned.apk`, which Android will not install. + Five checks at a time on EU phones, for a suite that would otherwise take hours: ```yaml @@ -447,13 +499,13 @@ Fire and forget, for a nightly run you inspect in the UI: **Proxies are web only.** The mobile API takes no proxy field, so `proxy-config-id` on a mobile run warns and is dropped. -**Listing app binaries needs an owner token.** `GET /products/:id/mobile_binary_files` is owner-only, though `app-binary-id` works with any token that can reach the product. Look the ID up once and store it as a repository variable. +**Listing and uploading app binaries need an owner or engineer token.** `GET /products/:id/mobile_binary_files` and both halves of the upload refuse every other role, though `app-binary-id` works with any token that can reach the product. Either give CI an owner or engineer token so `app-binary-path` can upload, or look the ID up once yourself and store it as a repository variable. **`device-location` only narrows auto-selection.** It is dropped with a warning alongside a pinned `device-serial`, which already names one device. Older installations may not accept it — the filter was added to the mobile API after the datacenter values themselves were exposed. **Private installations need a reachable host.** GitHub-hosted runners must be able to open an HTTPS connection to `host`. If your installation sits behind a firewall or VPN, use a self-hosted runner. -**Requirements.** `bash`, `curl` and `jq`. All are present on GitHub-hosted runners. +**Requirements.** `bash`, `curl` and `jq`, plus `openssl` for `app-binary-path`. All are present on GitHub-hosted runners. ## Development @@ -461,7 +513,7 @@ Fire and forget, for a nightly run you inspect in the UI: tests/run_tests.sh ``` -The tests run `scripts/run.sh` end to end against `tests/stub_api.py`, a small stand-in for the REST API. They cover the pass and fail gates, blocked handling, timeouts, expired tokens, JUnit output and escaping, both channels' request bodies, and the mobile input validation. +The tests run `scripts/run.sh` end to end against `tests/stub_api.py`, a small stand-in for the REST API. They cover the pass and fail gates, blocked handling, timeouts, expired tokens, JUnit output and escaping, both channels' request bodies, the binary upload, and the mobile input validation. `scripts/run.sh` uses bash 4 parameter expansion, so run the tests with bash 4 or newer. That is what GitHub-hosted runners have; macOS ships bash 3.2, where you need `brew install bash` and `/opt/homebrew/bin/bash tests/run_tests.sh`. diff --git a/action.yml b/action.yml index bf713ff..35bf8d5 100644 --- a/action.yml +++ b/action.yml @@ -109,7 +109,11 @@ inputs: required: false default: '' app-binary-id: - description: 'UUID of an uploaded app binary to install before the run. One app source is required for mobile.' + description: 'UUID of an already uploaded app binary to install before the run. One app source is required for mobile.' + required: false + default: '' + app-binary-path: + description: 'Path to an .apk, .aab or .ipa in the workspace. Uploaded to the product before the run, so a CI build is tested as built.' required: false default: '' app-package: @@ -197,6 +201,7 @@ runs: AQ_DEVICE_LOCATION: ${{ inputs.device-location }} AQ_DEVICE_BACKEND: ${{ inputs.device-backend }} AQ_APP_BINARY_ID: ${{ inputs.app-binary-id }} + AQ_APP_BINARY_PATH: ${{ inputs.app-binary-path }} AQ_APP_PACKAGE: ${{ inputs.app-package }} AQ_MOBILE_BROWSER: ${{ inputs.mobile-browser }} AQ_PREREQUISITES: ${{ inputs.prerequisites }} diff --git a/scripts/run.sh b/scripts/run.sh index 420a7c7..df8d8fa 100755 --- a/scripts/run.sh +++ b/scripts/run.sh @@ -38,6 +38,7 @@ echo "::add-mask::$AQ_TOKEN" : "${AQ_MANUFACTURER:=}" : "${AQ_DEVICE_BACKEND:=}" : "${AQ_APP_BINARY_ID:=}" +: "${AQ_APP_BINARY_PATH:=}" : "${AQ_APP_PACKAGE:=}" : "${AQ_MOBILE_BROWSER:=false}" : "${AQ_PREREQUISITES:=}" @@ -133,14 +134,22 @@ else # notices the missing upload when the session starts, which would leave an # unstartable session behind, so settle it before anything is created. sources=0 - if is_true "$AQ_MOBILE_BROWSER"; then sources=$((sources + 1)); fi - if [[ -n "$AQ_APP_PACKAGE" ]]; then sources=$((sources + 1)); fi - if [[ -n "$AQ_APP_BINARY_ID" ]]; then sources=$((sources + 1)); fi + if is_true "$AQ_MOBILE_BROWSER"; then sources=$((sources + 1)); fi + if [[ -n "$AQ_APP_PACKAGE" ]]; then sources=$((sources + 1)); fi + if [[ -n "$AQ_APP_BINARY_ID" ]]; then sources=$((sources + 1)); fi + if [[ -n "$AQ_APP_BINARY_PATH" ]]; then sources=$((sources + 1)); fi if (( sources == 0 )); then - die "mobile needs an app source: set one of app-binary-id, app-package or mobile-browser." + die "mobile needs an app source: set one of app-binary-path, app-binary-id, app-package or mobile-browser." fi if (( sources > 1 )); then - die "set only one app source out of app-binary-id, app-package and mobile-browser." + die "set only one app source out of app-binary-path, app-binary-id, app-package and mobile-browser." + fi + if [[ -n "$AQ_APP_BINARY_PATH" ]]; then + [[ -f "$AQ_APP_BINARY_PATH" ]] || die "app-binary-path '$AQ_APP_BINARY_PATH' is not a file. It is read from the workspace, so check out or download the build first." + case "${AQ_APP_BINARY_PATH,,}" in + *.apk|*.aab|*.ipa) ;; + *) die "app-binary-path must be an .apk, .aab or .ipa, got '$AQ_APP_BINARY_PATH'." ;; + esac fi if [[ -n "$AQ_APP_PACKAGE" && -z "$AQ_DEVICE_SERIAL" ]]; then die "app-package runs an app already installed on one device, so it needs device-serial." @@ -186,8 +195,47 @@ if [[ -z "$session_name" ]]; then fi fi +# Two-phase direct upload: reserve a blob, stream the bytes to wherever the API +# points (local disk, GCS), then turn it into a binary record. The PUT carries +# only the headers the API handed back — never the API key, which would leak it +# to the storage host. +upload_app_binary() { + local path="$1" name size checksum init url signed + command -v openssl >/dev/null 2>&1 || die "openssl is required to upload app-binary-path." + name=$(basename "$path") + size=$(wc -c < "$path" | tr -d '[:space:]') + checksum=$(openssl dgst -md5 -binary "$path" | base64 | tr -d '\n') + + api POST "/api/v1/products/$AQ_PRODUCT_ID/mobile_binary_files/initiate_upload" \ + "$(jq -n --arg f "$name" --argjson s "$size" --arg c "$checksum" \ + '{filename: $f, byte_size: $s, checksum: $c}')" + [[ "$API_STATUS" == "200" ]] || die "could not start the binary upload: $(api_error)" + init="$API_BODY" + url=$(jq -r '.direct_upload_url' <<<"$init") + signed=$(jq -r '.blob_signed_id' <<<"$init") + [[ -n "$url" && "$url" != "null" ]] || die "upload did not return a direct_upload_url: $init" + + local -a put=(-sS -o /dev/null -w '%{http_code}' -X PUT --upload-file "$path" --max-time 600) + while IFS= read -r header; do + [[ -n "$header" ]] && put+=(-H "$header") + done < <(jq -r '(.direct_upload_headers // {}) | to_entries[] | "\(.key): \(.value)"' <<<"$init") + local code + if ! code=$(curl "${put[@]}" "$url" 2>&1); then + die "could not upload $name to the storage host." + fi + [[ "$code" =~ ^2[0-9][0-9]$ ]] || die "uploading $name failed with HTTP $code." + + api POST "/api/v1/products/$AQ_PRODUCT_ID/mobile_binary_files/commit_upload" \ + "$(jq -n --arg b "$signed" '{blob_signed_id: $b}')" + [[ "$API_STATUS" == "201" ]] || die "could not finish the binary upload: $(api_error)" + AQ_APP_BINARY_ID=$(jq -r '.id' <<<"$API_BODY") + [[ -n "$AQ_APP_BINARY_ID" && "$AQ_APP_BINARY_ID" != "null" ]] || die "upload did not return a binary id: $API_BODY" + echo "Uploaded $name ($size bytes) as $AQ_APP_BINARY_ID" +} + if [[ "$AQ_CHANNEL" == "mobile" ]]; then if is_true "$AQ_MOBILE_BROWSER"; then browser_json=true; else browser_json=false; fi + if [[ -n "$AQ_APP_BINARY_PATH" ]]; then upload_app_binary "$AQ_APP_BINARY_PATH"; fi create_path="/api/v1/projects/$AQ_PROJECT_ID/mobile/test_sessions" payload=$(jq -n \ --arg name "$session_name" \ diff --git a/tests/run_tests.sh b/tests/run_tests.sh index 565cd9a..6bbbb4d 100755 --- a/tests/run_tests.sh +++ b/tests/run_tests.sh @@ -38,7 +38,7 @@ run_action() { AQ_JUNIT_PATH="$WORK/report.xml" \ AQ_CHANNEL="web" AQ_PRODUCT_ID="" AQ_DEVICE_SERIAL="" AQ_DEVICE_PLATFORM="" \ AQ_DEVICE_TYPE="" AQ_OS_VERSION="" AQ_MANUFACTURER="" AQ_DEVICE_BACKEND="" \ - AQ_APP_BINARY_ID="" AQ_APP_PACKAGE="" AQ_MOBILE_BROWSER="false" AQ_PREREQUISITES="" \ + AQ_APP_BINARY_ID="" AQ_APP_BINARY_PATH="" AQ_APP_PACKAGE="" AQ_MOBILE_BROWSER="false" AQ_PREREQUISITES="" \ AQ_DEVICE_LOCATION="" AQ_MAX_CONCURRENCY="" \ AQ_PROXY_CONFIG_ID="" AQ_RESULTS_SETTLE_SECONDS="30" \ "$@" bash "$ROOT/scripts/run.sh" > "$WORK/log" 2>&1 @@ -152,6 +152,60 @@ assert_file_has "$WORK/payload.json" '"manufacturer": "Google"' "sends the manu assert_file_has "$WORK/payload.json" '"mobile_browser": true' "sends mobile_browser" assert_file_has "$WORK/payload.json" '"prerequisites": "log in first"' "sends prerequisites" +# --- uploading a fresh build --- + +BUILD="$WORK/app-release.apk" +printf 'apk!' > "$BUILD" + +run_action green 0 "uploads a fresh build and runs the session on it" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$BUILD" +assert_file_has "$WORK/initiate.json" '"filename": "app-release.apk"' "sends the filename" +assert_file_has "$WORK/initiate.json" '"byte_size": 4' "sends the byte size" +assert_file_has "$WORK/initiate.json" '"checksum"' "sends a checksum" +assert_file_has "$WORK/commit.json" '"blob_signed_id": "signed-blob-1"' "commits the blob it was given" +assert_file_has "$WORK/payload.json" '"selected_artifact_id": "binary-99"' "runs on the uploaded binary" +assert_file_has "$WORK/log" "Uploaded app-release.apk" "reports the upload" + +if [[ "$(cat "$WORK/uploaded.bin" 2>/dev/null)" == "apk!" ]]; then + echo "ok streams the file bytes to the storage host" + pass=$((pass + 1)) +else + echo "FAIL streams the file bytes to the storage host" + fail=$((fail + 1)) +fi + +# The storage host is not ours; the API key must not travel to it. +if grep -qi "ApiKey" "$WORK/upload_headers.json"; then + echo "FAIL keeps the API key out of the storage upload" + fail=$((fail + 1)) +else + echo "ok keeps the API key out of the storage upload" + pass=$((pass + 1)) +fi + +run_action green 1 "fails when the build file is missing" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$WORK/not-here.apk" +assert_file_has "$WORK/log" "is not a file" "explains a missing build file" + +printf 'x' > "$WORK/build.txt" +run_action green 1 "rejects a build that is not an apk, aab or ipa" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$WORK/build.txt" + +run_action green 1 "rejects both a build path and a binary id" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$BUILD" AQ_APP_BINARY_ID=bin-1 + +run_action upload_rejected 1 "fails when the upload is refused" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$BUILD" +assert_file_has "$WORK/log" "could not start the binary upload" "explains a refused upload" + +run_action storage_down 1 "fails when the storage host rejects the bytes" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$BUILD" +assert_file_has "$WORK/log" "failed with HTTP 500" "explains a failed storage upload" + +run_action commit_rejected 1 "fails when the commit is refused" \ + "${MOBILE_BASE[@]}" AQ_DEVICE_SERIAL=ABC123 AQ_APP_BINARY_PATH="$BUILD" +assert_file_has "$WORK/log" "could not finish the binary upload" "explains a refused commit" + run_action green 0 "runs a mobile session several checks at a time" \ "${MOBILE_BASE[@]}" AQ_DEVICE_PLATFORM=android AQ_MOBILE_BROWSER=true \ AQ_MAX_CONCURRENCY=5 AQ_DEVICE_LOCATION=EU diff --git a/tests/stub_api.py b/tests/stub_api.py index cbf07c2..ad5345f 100644 --- a/tests/stub_api.py +++ b/tests/stub_api.py @@ -85,6 +85,29 @@ def do_POST(self): body = json.loads(self.rfile.read(length) or "{}") if self.path.endswith("/run"): return self.reply(200, {"test_session": {"id": "sess-1", "status": "started"}}) + if self.path.endswith("/mobile_binary_files/initiate_upload"): + with open(os.path.join(WORK, "initiate.json"), "w") as handle: + json.dump(body, handle, indent=2) + if MODE == "upload_rejected": + return self.reply(422, {"error": "byte_size exceeds maximum of 10 bytes"}) + return self.reply(200, { + "direct_upload_url": f"http://127.0.0.1:{PORT}/blob/put", + "direct_upload_headers": {"Content-Type": "application/octet-stream"}, + "blob_signed_id": "signed-blob-1", + "max_bytes": 2147483648, + "platform": "android", + }) + if self.path.endswith("/mobile_binary_files/commit_upload"): + with open(os.path.join(WORK, "commit.json"), "w") as handle: + json.dump(body, handle, indent=2) + if MODE == "commit_rejected": + return self.reply(422, {"error": "Upload not completed"}) + return self.reply(201, { + "id": "binary-99", + "filename": "app-release.apk", + "platform": "android", + "content_length": 4, + }) with open(os.path.join(WORK, "payload.json"), "w") as handle: json.dump(body, handle, indent=2) with open(os.path.join(WORK, "create_path"), "w") as handle: @@ -94,4 +117,17 @@ def do_POST(self): return self.reply(201, {"test_session": {"id": "sess-1", "status": "created"}}) + def do_PUT(self): + length = int(self.headers.get("Content-Length") or 0) + payload = self.rfile.read(length) + with open(os.path.join(WORK, "uploaded.bin"), "wb") as handle: + handle.write(payload) + with open(os.path.join(WORK, "upload_headers.json"), "w") as handle: + json.dump(dict(self.headers), handle, indent=2) + if MODE == "storage_down": + return self.reply(500, {"error": "storage unavailable"}) + self.send_response(204) + self.end_headers() + + HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()