diff --git a/SUMMARY.md b/SUMMARY.md index 8929180..9dfee9c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -74,6 +74,7 @@ ## Advanced * [Async Execution](advanced/async-execution.md) +* [Cancelling Superseded Runs](advanced/cancel-previous.md) * [Retry Strategies](advanced/retry-strategies.md) * [Chrome Onboarding](advanced/chrome-onboarding.md) * [Exit Codes](advanced/exit-codes.md) diff --git a/advanced/cancel-previous.md b/advanced/cancel-previous.md new file mode 100644 index 0000000..75d2b7b --- /dev/null +++ b/advanced/cancel-previous.md @@ -0,0 +1,151 @@ +# Cancelling superseded runs + +Push twice in quick succession and the first run's queued tests are dead weight: +nobody is waiting on their verdict, but they still sit in the queue and still +cost you. `--cancel-previous` cancels them when the newer run is submitted. + +It is opt-in and off by default. Without it, nothing about your runs changes. +Requires DeviceCloud CLI 5.6.0 or later. + +## What it does + +When a run carrying the flag is submitted, DeviceCloud looks for the previous +run from the same CI context and cancels the tests of that run which have not +started yet. + +* **Both runs need the flag.** A run is only superseded if it was itself + submitted with `--cancel-previous`. +* **Queued tests only.** Anything already running on a device finishes and + reports normally — cancelling it would free no device and refund nothing. +* **Cancelled tests are refunded at 75%**, the same as cancelling a + not-yet-started test by hand in the console. +* **The superseded run goes quiet.** It sends no completion email, Slack + message or webhook, and its GitHub check, if it has one, is closed as + `skipped` so it cannot block a pull request. `dcd cloud` exits `0` for it + rather than failing your build; see + [How a superseded run reports](#how-a-superseded-run-reports) for how each + integration treats it. + +## What counts as "the same CI context" + +The context is built from three pieces of the run's metadata. Each tab under +[Usage](#usage) shows how to supply them. + +| Part | Where it comes from | +|---|---| +| Repository | `gh_repo` | +| Branch or PR | `gh_pr_number` if present, otherwise `gh_branch` | +| Job | `gh_check_name` — the job's check name | + +A pull request and a push to the same branch are **different** contexts, which +matches how GitHub Actions treats them. + +If a run has no repository, or no branch and no PR, there is no context to group +by and nothing is cancelled. The run's output says so rather than guessing. + +{% hint style="warning" %} +**Set a check name per job.** The check name is what keeps your iOS job from +cancelling your Android job. If one commit runs tests more than once and those +runs share a check name, they share a group — and the second will cancel the +first's queued tests. Without a check name, every job on the branch or PR +shares one group, and the submit output warns you. +{% endhint %} + +Runs that report the **same** CI run ID never cancel each other, so jobs within +one CI run are safe even before you set check names. Where the run ID comes +from: + +* **CLI:** `--metadata gh_run_id=`. Without it, runs have no run ID. +* **EAS Workflows:** the EAS build ID, which differs between your iOS and + Android build jobs, so those two jobs don't count as one run. Give each its + own `DCD_CHECK_NAME`, or set the same `DCD_GH_RUN_ID` on both. + +## Usage + +`--cancel-previous` works with the CLI and with EAS Workflows. The GitHub +Action, the Bitrise step and the Bitbucket pipe don't have an option for it yet; +in those pipelines you can call the CLI directly, as below. + +{% tabs %} +{% tab title="CLI" %} +```bash +dcd cloud --app-file build/app.apk --flows ./.maestro \ + --cancel-previous \ + --repo-name acme/my-app \ + --branch "$GIT_BRANCH" \ + --metadata gh_check_name=Android \ + --metadata gh_run_id="$CI_RUN_ID" +``` + +The CLI doesn't read your CI provider's environment, so pass the context +yourself. `--repo-name` plus `--branch` or `--pr-number` are required; without +them nothing is cancelled. `gh_check_name` scopes the group to this job, and +`gh_run_id` stops jobs of the same CI run from cancelling each other. +{% endtab %} + +{% tab title="EAS Workflows" %} +```yaml +e2e_android: + needs: [build_android] + runs_on: linux-medium + env: + DCD_GH_REPO: acme/my-app # your repository, as owner/repo + DCD_GH_BRANCH: ${{ github.ref_name }} + DCD_CHECK_NAME: Android # a different value in each job + steps: + - uses: eas/checkout + - id: download + uses: eas/download_build + with: + build_id: ${{ needs.build_android.outputs.build_id }} + - run: | + npx --yes @devicecloud.dev/eas-workflow@v1 \ + --app-file ${{ steps.download.outputs.artifact_path }} \ + --flows ./.maestro \ + --cancel-previous +``` + +The wrapper passes `--cancel-previous` straight to the CLI and builds the +context from the job's `env:` block: `DCD_GH_REPO` plus `DCD_GH_BRANCH` or +`DCD_GH_PR_NUMBER` are required. Giving `DCD_GH_REPO` as a fixed string, as +above, keeps it valid on manual runs too. Instead of a check name per job, you +can set `DCD_GH_RUN_ID: ${{ workflow.id }}` on every job so the jobs of one +workflow run count as one run. See +[EAS Workflows](../ci-cd/eas-workflows.md#git-context-optional) for the +variables. +{% endtab %} +{% endtabs %} + +## How a superseded run reports + +The superseded run's CLI keeps waiting until any of its tests that were already +running have finished, then prints: + +``` +⚠ Run superseded by a newer run from the same CI context — exiting 0 + ⎿ superseded by https://console.devicecloud.dev/results?upload= +``` + +`dcd cloud` then exits `0`, even if one of the run's tests had already failed. +Under `--json`, `status` is `SUPERSEDED` — a third value alongside `PASSED` and +`FAILED`, so update any script that switches on it. + +That value only comes from `dcd cloud` itself: `dcd status` and +[`GET /uploads/status`](../api/uploads.md) report a superseded run as `FAILED`, +because its cancelled tests count against it. So whether the older CI job goes +green depends on how you run the tests: + +* **CLI:** the job passes, because `dcd cloud` exits `0`. +* **EAS Workflows:** the job still fails. The wrapper checks the run's status + after the CLI exits, and that reports `FAILED`. + +Each result cancelled this way carries a `cancellation_reason` of +`superseded_by:` in [`GET /results/{uploadId}`](../api/results.md). + +## Limitations + +* It cannot stop a test that is already running on a device. A run whose tests + had all started is unaffected. +* A re-run that reports the same CI run ID, such as GitHub's "Re-run failed + jobs", does not supersede the original attempt. +* A previous run older than 24 hours is not superseded. diff --git a/advanced/exit-codes.md b/advanced/exit-codes.md index 7fcc9b7..9e542b9 100644 --- a/advanced/exit-codes.md +++ b/advanced/exit-codes.md @@ -27,3 +27,7 @@ The two JSON flags behave differently around test failures: {% hint style="info" %} Use `--json-file` when you want to inspect the result yourself rather than have a non-zero exit code fail the build. Use `--json` when you still want the exit code to gate your pipeline. {% endhint %} + +## Superseded Runs + +A run superseded by a newer run through [`--cancel-previous`](cancel-previous.md) exits `0`, even if one of its tests had already failed. With `--json` or `--json-file`, its `status` is `SUPERSEDED`. diff --git a/api/results.md b/api/results.md index 3574ad6..b9a64fe 100644 --- a/api/results.md +++ b/api/results.md @@ -39,6 +39,17 @@ curl https://api.devicecloud.dev/results/7e12345f-eb12-12ec-a30b-bb1234f1d12a \ } ``` +Each result also has a `cancellation_reason`, which is `null` unless the test +was cancelled for a recorded reason: + +* `superseded_by:` — a newer run from the same CI context replaced + this one. See [Cancelling superseded runs](../advanced/cancel-previous.md). +* `Cancelled: payment for this run failed.` — the run's payment failed after + its tests were created. + +A test you cancel yourself has a `null` reason. Match on the `superseded_by:` +prefix rather than comparing whole values, as more reasons may be added. + --- ## Download JUnit report diff --git a/ci-cd/eas-workflows.md b/ci-cd/eas-workflows.md index 1a3b85d..0d46c38 100644 --- a/ci-cd/eas-workflows.md +++ b/ci-cd/eas-workflows.md @@ -155,7 +155,7 @@ The build artifact itself is downloaded by `eas/download_build` and passed via ` |----------|-------------|-----------| | `DCD_GH_SHA` | `${{ github.sha }}` | `gh_sha` metadata | | `DCD_GH_BRANCH` | `${{ github.ref_name }}` | `gh_branch` metadata | -| `DCD_GH_RUN_ID` | `${{ github.run_id }}` | `gh_run_id` metadata | +| `DCD_GH_RUN_ID` | `${{ workflow.id }}` | `gh_run_id` metadata — the same value on every job of one workflow run | | `DCD_GH_PR_NUMBER` | `${{ github.event.pull_request.number }}` | `gh_pr_number` metadata | | `DCD_GH_PR_URL` | `${{ github.event.pull_request.html_url }}` | `gh_pr_url` metadata | | `DCD_GH_REPO` | `${{ github.repository }}` | `gh_repo` metadata | @@ -164,7 +164,7 @@ The build artifact itself is downloaded by `eas/download_build` and passed via ` Set `DCD_CHECK_NAME` when a commit is tested by more than one job, so each gets a check of its own that can be required separately in branch protection. Keep it fixed per job — GitHub matches required checks by name. {% hint style="warning" %} -`${{ github.event.pull_request.number }}`, `${{ github.event.pull_request.html_url }}` and `${{ github.repository }}` resolve to `null` on manual triggers and EAS rejects them as invalid env values. Only set the PR and repo variables inside an `if:` guard that limits the job to PR events, or omit them. `github.sha`, `github.ref_name` and `github.run_id` coerce to empty strings safely. +`${{ github.event.pull_request.number }}`, `${{ github.event.pull_request.html_url }}` and `${{ github.repository }}` resolve to `null` on manual triggers and EAS rejects them as invalid env values. Only set the PR and repo variables inside an `if:` guard that limits the job to PR events, or omit them. `github.sha` and `github.ref_name` coerce to empty strings safely. {% endhint %} ### Advanced @@ -229,6 +229,7 @@ See the [Devices & OS Versions](../getting-started/devices-configuration.md) pag | Flag | Description | |------|-------------| | `--async` | Exit immediately without waiting for results (exit code `0` regardless). See [Async Execution](../advanced/async-execution.md). | +| `--cancel-previous` | Cancel the still-queued tests of the previous run of this job on the same branch or PR. Needs `DCD_GH_REPO` plus `DCD_GH_BRANCH` or `DCD_GH_PR_NUMBER`; set `DCD_CHECK_NAME` per job. See [Cancelling superseded runs](../advanced/cancel-previous.md). | | `--download-artifacts ` | Download logs/screenshots/videos. Options: `ALL`, `FAILED`. | | `--disable-animations` | Disable device animations. See [Animations](../configuration/disable-animations.md). | | `--maestro-chrome-onboarding` | Android only. See [Chrome Onboarding](../advanced/chrome-onboarding.md). | diff --git a/ci-cd/overview.md b/ci-cd/overview.md index e4aadb7..6ceefb4 100644 --- a/ci-cd/overview.md +++ b/ci-cd/overview.md @@ -13,6 +13,7 @@ DeviceCloud supports a wide range of CI/CD options. If you don't see your provid ### Useful Features For CI/CD * [Async Execution](../advanced/async-execution.md) - Fire-and-forget tests without blocking your pipeline +* [Cancelling Superseded Runs](../advanced/cancel-previous.md) - Drop the previous run's queued tests when a newer commit arrives * [dcd status](../cli/dcd-status.md) - Poll for results after an async run * [Report Formats](../artifacts/report-formats.md) - Generate JUnit/HTML reports for your CI system * [Artifacts & Downloads](../artifacts/artifacts.md) - Access logs, screenshots, and videos diff --git a/cli/dcd-cloud.md b/cli/dcd-cloud.md index 3a7a1c3..4b4ec3d 100644 --- a/cli/dcd-cloud.md +++ b/cli/dcd-cloud.md @@ -98,8 +98,9 @@ Attach Git and pull request metadata to a run. These values are displayed in the | Flag | Description | |------|-------------| | `--async` | Submit tests and return immediately (exit `0`) without waiting for results (see [Async Execution](../advanced/async-execution.md)) | +| `--cancel-previous` | Cancel the still-queued tests of the previous run from the same CI context. Needs `--repo-name` plus `--branch` or `--pr-number` (see [Cancelling superseded runs](../advanced/cancel-previous.md)) | | `--quiet`, `-q` | Suppress per-test progress; print only the final summary | -| `--json` | Output results as JSON. Exits `0` on success, `2` on test failure, `1` on CLI/infrastructure errors | +| `--json` | Output results as JSON. Exits `0` on success, `2` on test failure, `1` on CLI/infrastructure errors. A run superseded through `--cancel-previous` has `status` `SUPERSEDED` and exits `0` | | `--json-file` | Write JSON results to a file (`_dcd.json` by default). Exits `0` even if the test run fails; infrastructure errors still exit `1` | | `--json-file-name ` | Custom name (or relative path) for the JSON file. Requires `--json-file` | | `--dry-run` | Simulate the run without uploading or triggering a test — useful for debugging workflow issues | diff --git a/notifications/email-notifications.md b/notifications/email-notifications.md index e22b723..8e195f1 100644 --- a/notifications/email-notifications.md +++ b/notifications/email-notifications.md @@ -18,7 +18,7 @@ notifications: - devs@example.com ``` -By default you'll only be emailed when a suite has at least one failed or cancelled flow. +By default you'll only be emailed when a suite has at least one failed or cancelled flow. A run superseded by a newer run through [`--cancel-previous`](../advanced/cancel-previous.md) is never emailed. {% hint style="info" %} The `config.yaml` is picked up automatically from the directory you pass to `dcd cloud`. See [Workspace Configuration](../configuration/workspace-config.md) for where the file lives and how it's loaded. diff --git a/notifications/slack-notifications.md b/notifications/slack-notifications.md index 03c1d32..6f4e2ca 100644 --- a/notifications/slack-notifications.md +++ b/notifications/slack-notifications.md @@ -23,6 +23,8 @@ Once a channel is selected, use **Send test** to post a sample message to that c By default DeviceCloud posts after **every** completed run. To cut down on noise, turn on **Only notify on failures** in the Slack section — passing runs will then be skipped and you'll only get a message when a run has at least one failed flow. +A run superseded by a newer run through [`--cancel-previous`](../advanced/cancel-previous.md) never posts, whatever this setting. + ### What's in the message Each notification includes: diff --git a/notifications/webhook-notifications.md b/notifications/webhook-notifications.md index 8d5276a..cfbf2dd 100644 --- a/notifications/webhook-notifications.md +++ b/notifications/webhook-notifications.md @@ -1,6 +1,6 @@ # Webhook Notifications -DeviceCloud can send a POST request to a URL of your choice when a set of tests completes. +DeviceCloud can send a POST request to a URL of your choice when a set of tests completes. No request is sent for a run superseded by a newer run through [`--cancel-previous`](../advanced/cancel-previous.md). ### Enabling Webhooks