test: [SDK-5481] add a fault-injection harness for request retries

.gitignore

@@ -21,4 +21,6 @@ iOS_SDK/Carthage/Build
 .build/
 
 examples/demo/App/Secrets.plist
-examples/demo/Local.xcconfig
+examples/demo/Local.xcconfig
+examples/demo/scripts/retry/.state/
+examples/demo/scripts/retry/__pycache__/

examples/demo/scripts/retry/README.md

@@ -0,0 +1,128 @@
+# Retry test harness
+
+Injects failures into the SDK's requests from the demo app, so executor retries and backoff can be
+watched and checked. A [mitmproxy](https://mitmproxy.org) addon fails only requests that carry the
+SDK's `SDK-Version` header. The demo's own REST calls and all other traffic pass through. Only
+`onesignal.com` traffic is decrypted.
+
+```bash
+brew install mitmproxy mobile-dev-inc/tap/maestro
+cd examples/demo/scripts/retry
+./retry-test.sh help
+```
+
+[Maestro](https://maestro.mobile.dev) drives the demo's UI (`tap`) through the flows in `flows/`.
+
+## Setup
+
+1. `xcrun simctl create "OneSignal Retry Test" "iPhone 17"`. The harness uses its own simulator
+   (`RETRY_TEST_DEVICE`, by name or UDID) because the other OneSignal demos share the bundle ID.
+2. `./retry-test.sh install` builds the demo and installs it there.
+3. `./retry-test.sh start` starts mitmproxy on port 8899, points the Mac's Wi-Fi/Ethernet HTTP(S)
+   proxy at it (the simulator uses the Mac's proxy), and trusts its CA on the simulator.
+4. `./retry-test.sh fresh`, then `./retry-test.sh tap allow-notifications`. The permission prompt
+   holds the app inactive, so foreground retries never fire while it is showing. `fresh` wipes the
+   demo's data without reinstalling, so the permission is kept from then on.
+5. In another terminal, `./retry-test.sh watch` follows each SDK request.
+6. When done, **`./retry-test.sh stop`** restores the Mac's proxy settings.
+
+Leave the Mac's proxy alone with `RETRY_TEST_PROXY_MODE=local` (mitmproxy captures the `App`
+process, which needs its network extension approved once). For a physical device, use
+`RETRY_TEST_PROXY_MODE=manual` and set the device's Wi-Fi proxy to the Mac, as `start` explains.
+
+## Commands
+
+| Command | Does |
+| --- | --- |
+| `fail <route> <status> [--times N] [--retry-after S] [--add]` | Fail matching requests. Status `0` drops the connection, like being offline. Without `--times` it fails until `clear`. `--add` keeps earlier rules. Applies right away. |
+| `clear` / `rules` | Remove or show the rules |
+| `report [regex]` | Per route: each attempt, the gap since the previous one, and the gap expected |
+| `reset-log` | Forget recorded attempts, for example between scenarios |
+| `tap <action> [args]` | Use the demo: `add-tag <key> <value>`, `add-alias <label> <id>`, `add-email <email>`, `login <external id> [jwt]`, `update-jwt <external id> [jwt]`, `logout`, `toggle-push`, `pause-iams`, `allow-notifications`. With `RETRY_TEST_IV_KEY` set, `login` and `update-jwt` sign a JWT when none is given |
+| `jwt <external id> [label=id] [email:addr] [sms:number]` | Print an Identity Verification JWT, valid for an hour, that claims those aliases and subscriptions |
+| `install` | Build the demo and install it on the simulator |
+| `fresh` / `relaunch` | New install without reinstalling / kill and relaunch |
+| `background` / `foreground` | Background the demo by opening Settings / bring it back |
+| `network-flap [s]` | Turn the Mac's Wi-Fi off for `s` seconds (default 10). Drops your Mac's connection too. |
+| `logs` | Stream the demo's SDK request and retry logs |
+
+Routes: `create-user`, `fetch-user`, `update-user` (tags, language, session data), `identity` (add
+alias and login's Identify User share it), `remove-alias`, `create-sub`, `update-sub`,
+`delete-sub`, `identity-by-sub`, `custom-events`, `all`.
+
+## Reading `report`
+
+```
+POST apps/<app>/users  (User executor)
+    #     time status     body      gap         expected  note
+    1     0.0s   408* edc1f203
+    2     2.8s   408* 8f1a8f24     2.8s         2.5-5.0s  executor backoff #1
+    3    11.4s   408* d85e5a15     8.6s        5.0-10.0s  executor backoff #2
+    4    31.0s   201  3a910c5a    19.6s       10.0-20.0s  executor backoff #3
+```
+
+What the SDK should do:
+
+- **Executor backoff**: 5s, doubling to 300s, times a jitter of 0.5 to 1. `Retry-After` is the
+  minimum wait (capped at 1h), and a 429 without one waits 60s.
+- **Client reattempts first**: `OneSignalClient` itself re-sends a 5xx or status 0 four times
+  (5, 15, 45, 135s) before the executor sees the failure. That count is kept per request for the
+  run, so later failures of the same request go straight to the executor backoff. Use a
+  retryable status the client does not reattempt, like **408**, to see the executor backoff right
+  away. The client resends ignore foreground and network return, so after going offline, requests
+  can wait for the next client resend (up to 135s).
+- **Every executor resends on the 5s poll**, so the window allows up to 5s more.
+- **Foreground, and network return while in the foreground, end the executor backoff early**, but
+  not `Retry-After`. `EARLY` right after those is expected. One case happens without you doing
+  anything: at cold start, the first failure often lands just before the app goes active, so the
+  second attempt follows within a second.
+- `body` is a hash of the request body. A new hash on each attempt is normal, since bodies hold
+  timestamps.
+- Attempts are grouped by method and path with IDs left out, so two requests on one route in the
+  same window (two logins, or a session update and a tag) share one series. `reset-log` between
+  scenarios, and run one action at a time.
+
+## Scenarios
+
+Run `reset-log` before each one. Unless noted, the request should succeed after the last failure,
+and nothing should need a relaunch.
+
+1. **Exponential backoff, User executor**: `fail create-user 408 --times 4`, then `fresh`. Gaps
+   should fall in the windows above, then 201.
+2. **Repo executor**: `fail update-user 408 --times 3`, then `tap add-tag k v`. The PATCH
+   retries on the next poll after each backoff.
+3. **Retry-After**: `fail update-user 429 --times 2 --retry-after 20`, then `tap add-tag k v`. Gaps of at
+   least 20s. `background` then `foreground` during the wait must not cut it short.
+4. **429 without Retry-After**: `fail update-user 429 --times 1`, then `tap add-tag k v`. About 60s.
+5. **5xx**: `fail update-user 503 --times 6`, then `tap add-tag k v`. Four client reattempts (5, 15, 45,
+   135s), then executor backoff. Slow, about 3.5 minutes.
+6. **Offline**: `fail all 0`, then `tap add-tag k v`, `tap add-alias l i` and `tap add-email e`.
+   Wait for a few failures, then `clear`. Everything goes out by its next client resend or backoff.
+7. **Foreground ends the backoff**: `fail create-user 408 --times 6`, then `fresh`. After about
+   25s, `background`, wait, `foreground`. An attempt goes out right as the app becomes active.
+8. **Network return**: as in 7, but `network-flap 5` with the app in the foreground. An attempt goes
+   out when Wi-Fi comes back. Repeat with the app in the background: no attempt then.
+9. **Order is kept**: `fail create-user 408 --times 3`, then `fresh` and `tap login <id>` during the
+   failures. Identify User (PATCH `.../identity`) is not sent until Create User succeeds.
+10. **Fetch User**: `fail fetch-user 408 --times 3`, then `tap login <id>`.
+    Fetch User retries in the same run, and later user requests wait for it.
+11. **Kill during backoff**: `fail update-user 408`, `tap add-tag k v`, wait for two failures, `clear`,
+    then `relaunch`. The cached request is sent right after launch.
+12. **Non-retryable is not retried**: `fail update-user 400 --times 1`, then `tap add-tag k v`. One attempt,
+    no retries.
+13. **Update Subscription no longer retries every 5s**: `fail update-sub 408 --times 4`, then
+    `tap toggle-push`. Gaps grow instead of repeating every 5s.
+14. **Login/logout while failing**: `fail all 408`, `tap login b`, `tap logout`, `tap login c`, then
+    `clear`. Requests for each user go out in order, with no errors about the wrong user.
+15. **Identity Verification**: needs an app with Identity Verification on and its ES256 private
+    key. Ask the SDK team for both and keep the key outside the repo. Set the app's ID as
+    `ONESIGNAL_APP_ID` in `examples/demo/App/Secrets.plist` and `install` again, then set
+    `RETRY_TEST_IV_APP_ID=<app id>` and `RETRY_TEST_IV_KEY=<path to the key>`. `tap login <id>` now
+    signs a JWT. No anonymous Create User is sent, so in 1 and 9 the Create User comes from the
+    login. If the app has in-app messages, `tap pause-iams` first, or a message can take a `tap`.
+    Repeat 1, 2, 6 and 9. In 6, log in first with a JWT that claims the alias and email:
+    `tap login <id> "$(./retry-test.sh jwt <id> l=i email:e)"`. To see a token rejected in the middle of
+    retries, `fail create-user 408 --times 2` and log in with a token signed by another key. After
+    the 401 nothing is sent until `tap update-jwt <id>`. `report` treats a 401 or 403 as holding the
+    request, with its retry counts, only when the request carried a Bearer token. Create User keeps
+    its counts after any 4xx, since the SDK resends it on the next session.

examples/demo/scripts/retry/fault_proxy.py

@@ -0,0 +1,126 @@
+"""
+mitmproxy addon that injects failures into OneSignal SDK requests so executor retries can be
+observed in the demo app. Driven by `retry-test.sh`; see README.md.
+
+Only requests carrying the SDK's `SDK-Version` header are touched, so the demo's own REST calls
+pass through. Every SDK request is appended to attempts.jsonl for `retry-test.sh report`.
+
+Rules are re-read whenever rules.json changes. Each rule:
+  {"id": "1f3a9c2e", "route": "update-user", "method": "PATCH", "path": "<regex>", "status": 503,
+   "retry_after": "30", "times": 3}
+`status` 0 drops the connection (the SDK sees status 0, like being offline). `times` is how many
+matching requests fail before the rule passes them through; null fails forever.
+"""
+
+import hashlib
+import json
+import os
+import re
+import time
+
+from mitmproxy import http
+
+STATE_DIR = os.environ["RETRY_TEST_STATE_DIR"]
+RULES_PATH = os.path.join(STATE_DIR, "rules.json")
+ATTEMPTS_PATH = os.path.join(STATE_DIR, "attempts.jsonl")
+
+API_PATH_PREFIX = re.compile(r"^/(api/v1/)?")
+
+
+def body_hash(raw):
+    # Key order can change between attempts of the same request, so hash the parsed JSON.
+    try:
+        raw = json.dumps(json.loads(raw), sort_keys=True).encode()
+    except ValueError:
+        pass
+    return hashlib.sha1(raw).hexdigest()[:8]
+
+
+class FaultProxy:
+    def __init__(self):
+        self.rules_mtime = None
+        self.rules = []
+        self.remaining = []
+
+    def load_rules(self):
+        try:
+            mtime = os.stat(RULES_PATH).st_mtime_ns
+        except FileNotFoundError:
+            mtime = None
+        if mtime == self.rules_mtime:
+            return
+        self.rules_mtime = mtime
+        old_remaining = {rule["id"]: left for rule, left in zip(self.rules, self.remaining) if rule.get("id")}
+        self.rules = []
+        if mtime is not None:
+            with open(RULES_PATH) as f:
+                self.rules = json.load(f)
+        # `fail --add` rewrites the earlier rules with their ids; keep the counts they have used up.
+        # Every `fail` issues a new id, so re-issuing an identical rule starts a fresh count.
+        self.remaining = [
+            old_remaining[rule["id"]] if rule.get("id") in old_remaining else rule.get("times")
+            for rule in self.rules
+        ]
+        print(f"[fault] rules: {json.dumps(self.rules) if self.rules else 'none, passing everything'}")
+
+    def match(self, flow):
+        path = API_PATH_PREFIX.sub("", flow.request.path.split("?")[0])
+        for i, rule in enumerate(self.rules):
+            if rule.get("method") and rule["method"] != flow.request.method:
+                continue
+            if not re.search(rule["path"], path):
+                continue
+            if self.remaining[i] is not None:
+                if self.remaining[i] <= 0:
+                    continue
+                self.remaining[i] -= 1
+            return rule
+        return None
+
+    def request(self, flow: http.HTTPFlow):
+        if "onesignal.com" not in flow.request.pretty_host or "SDK-Version" not in flow.request.headers:
+            return
+        self.load_rules()
+        rule = self.match(flow)
+        flow.metadata["fault_rule"] = rule
+        if rule is None:
+            return
+        if rule["status"] == 0:
+            self.record(flow, 0, rule)
+            flow.kill()
+            return
+        headers = {"Content-Type": "application/json"}
+        if rule.get("retry_after") is not None:
+            headers["Retry-After"] = str(rule["retry_after"])
+        body = json.dumps({"errors": [{"title": f"Injected by fault_proxy ({rule['route']})"}]})
+        flow.response = http.Response.make(rule["status"], body, headers)
+
+    def response(self, flow: http.HTTPFlow):
+        if "fault_rule" not in flow.metadata:
+            return
+        self.record(flow, flow.response.status_code, flow.metadata["fault_rule"])
+
+    def error(self, flow: http.HTTPFlow):
+        if "fault_rule" in flow.metadata and flow.metadata["fault_rule"] is None:
+            self.record(flow, 0, None)
+
+    def record(self, flow, status, rule):
+        path = API_PATH_PREFIX.sub("", flow.request.path.split("?")[0])
+        entry = {
+            "t": time.time(),
+            "method": flow.request.method,
+            "path": path,
+            "status": status,
+            "injected": rule is not None,
+            "retry_after": rule.get("retry_after") if rule else None,
+            "body": body_hash(flow.request.raw_content or b""),
+            "bearer": flow.request.headers.get("Authorization", "").startswith("Bearer "),
+        }
+        with open(ATTEMPTS_PATH, "a") as f:
+            f.write(json.dumps(entry) + "\n")
+        tag = "INJECTED" if rule else "passed"
+        stamp = time.strftime("%H:%M:%S", time.localtime(entry["t"]))
+        print(f"[fault] {stamp} {status:>3} {tag:<8} {flow.request.method:<6} {path} body={entry['body']}", flush=True)
+
+
+addons = [FaultProxy()]

examples/demo/scripts/retry/flows/_scroll-to.yaml

@@ -0,0 +1,11 @@
+appId: ${APP_ID}
+---
+- scrollUntilVisible:
+    element:
+      id: app_section
+    direction: UP
+    speed: 100
+- scrollUntilVisible:
+    element:
+      id: ${TARGET}
+    centerElement: true

examples/demo/scripts/retry/flows/add-alias.yaml

@@ -0,0 +1,16 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: add_alias_button
+- tapOn:
+    id: add_alias_button
+- tapOn:
+    id: alias_label_input
+- inputText: ${KEY}
+- tapOn:
+    id: alias_id_input
+- inputText: ${VALUE}
+- tapOn:
+    id: singlepair_confirm_button

examples/demo/scripts/retry/flows/add-email.yaml

@@ -0,0 +1,13 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: add_email_button
+- tapOn:
+    id: add_email_button
+- tapOn:
+    id: email_input
+- inputText: ${VALUE}
+- tapOn:
+    id: singleinput_confirm_button

examples/demo/scripts/retry/flows/add-tag.yaml

@@ -0,0 +1,16 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: add_tag_button
+- tapOn:
+    id: add_tag_button
+- tapOn:
+    id: tag_key_input
+- inputText: ${KEY}
+- tapOn:
+    id: tag_value_input
+- inputText: ${VALUE}
+- tapOn:
+    id: singlepair_confirm_button

examples/demo/scripts/retry/flows/allow-notifications.yaml

@@ -0,0 +1,6 @@
+appId: ${APP_ID}
+---
+- launchApp
+- tapOn:
+    text: Allow
+    optional: true

examples/demo/scripts/retry/flows/login.yaml

@@ -0,0 +1,20 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: login_user_button
+- tapOn:
+    id: login_user_button
+- tapOn:
+    id: login_user_id_input
+- inputText: ${VALUE}
+- runFlow:
+    when:
+      true: ${JWT != ''}
+    commands:
+      - tapOn:
+          id: login_user_jwt_input
+      - inputText: ${JWT}
+- tapOn:
+    id: singleinput_confirm_button

examples/demo/scripts/retry/flows/logout.yaml

@@ -0,0 +1,8 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: logout_user_button
+- tapOn:
+    id: logout_user_button

examples/demo/scripts/retry/flows/pause-iams.yaml

@@ -0,0 +1,8 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: pause_iam_toggle
+- tapOn:
+    id: pause_iam_toggle

examples/demo/scripts/retry/flows/toggle-push.yaml

@@ -0,0 +1,8 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: push_enabled_toggle
+- tapOn:
+    id: push_enabled_toggle

examples/demo/scripts/retry/flows/update-jwt.yaml

@@ -0,0 +1,18 @@
+appId: ${APP_ID}
+---
+- runFlow:
+    file: _scroll-to.yaml
+    env:
+      TARGET: update_user_jwt_button
+- tapOn:
+    id: update_user_jwt_button
+# Prefilled with the external id the SDK asked about.
+- tapOn:
+    id: update_jwt_external_id_input
+- eraseText
+- inputText: ${KEY}
+- tapOn:
+    id: update_jwt_token_input
+- inputText: ${VALUE}
+- tapOn:
+    id: singlepair_confirm_button