Skip to content

test: [SDK-5535] add a fault-injection harness for operation retries - #2791

Merged
fadi-george merged 8 commits into
mainfrom
fadi/sdk-5535-retry-harness
Oct 14, 2026
Merged

fadi-george merged 8 commits into
mainfrom
fadi/sdk-5535-retry-harness

Conversation

@fadi-george

@fadi-george fadi-george commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

One Line Summary

Adds examples/demo/scripts/retry, a harness that injects failures into the demo's SDK requests and checks retry timing. It is a port of the iOS harness from OneSignal/OneSignal-iOS-SDK#1775.

Details

Motivation

Unit tests for #2790 mock the executors, so they don't show real retry gaps, or whether foreground and network return wake backed-off operations on a device. This lets anyone reproduce those cases against the demo app.

Scope

  • fault_proxy.py: mitmproxy addon, based on iOS. It fails only requests with the SDK-Version header and logs every attempt, timed at the request. It listens on loopback only.
  • report.py: per record (or per body for id-less routes), the gap between attempts vs the expected window: 5s doubling to 300s times 0.5-1 jitter, Retry-After as the minimum, and no client-side re-sends. It also lists any SDK request sent while HttpClient's Retry-After gate should have held it.
  • retry-test.sh: rewritten for adb. It sets the device's global HTTP proxy (10.0.2.2 on emulators, adb reverse on devices), clears data with pm clear, presses Home to background, and turns Wi-Fi and data off for a network flap, restoring only what was on even after Ctrl-C. A failed Maestro flow fails tap. install builds the demo against local SDK source.
  • Maestro flows: copied from iOS. The Android demo uses the same test tags, and only the scroll step changed.
  • Demo app: adds a network_security_config with debug-overrides only. When built with -PRETRY_TEST_CA=<pem>, the debug build also trusts that CA, so no certificate has to be installed on the device. Release builds are unaffected.

Testing

Manual testing

Android 16 emulator (API 36), demo built from main after #2790:

  • Backoff: Create User failing with 408 three times was retried after 3.8s, 6.5s and 11.8s, then 201. All inside the expected windows.
  • Other records keep going: with a tag update failing, a push toggle PATCH went out while the tag kept retrying (4.7s, 8.7s, 17.0s, 35.4s).
  • Foreground: background then foreground during a 40-80s backoff sent the retry 0.5s after foreground, with no batching wait.
  • Network return: a 5s flap in the foreground during an 80-160s backoff sent the retry right as the network came back, and Wi-Fi and data were restored.
  • Not rerun after fix: [SDK-5535] retry failed operations without blocking the queue #2790's final changes: Retry-After, order, kill during backoff, offline, login/logout, and Identity Verification.

Affected code checklist

  • Notifications
    • Display
    • Open
    • Push Processing
    • Confirm Deliveries
  • Outcomes
  • Sessions
  • In-App Messaging
  • REST API requests
  • Public API changes

Checklist

Overview

  • I have filled out all REQUIRED sections above
  • PR does one thing
  • Any Public API changes are explained in the PR details and conform to existing APIs

Testing

  • I have included test coverage for these changes, or explained why they are not needed
  • All automated tests pass, or I explained why that is not possible
  • I have personally tested this on my device, or explained why that is not possible

Final pass

  • Code is as readable as possible.
  • I have reviewed this PR myself, ensuring it meets each checklist item

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

📊 Diff Coverage Report

✓ Coverage check passed (no source files changed)

📥 View workflow run

@fadi-george
fadi-george added this pull request to stack #2792 October 14, 2026 02:19
@fadi-george
fadi-george marked this pull request as ready for review October 14, 2026 02:19
@fadi-george
fadi-george requested a review from a team as a code owner October 14, 2026 02:19
@fadi-george
fadi-george force-pushed the fadi/sdk-5535-retry-harness branch from 21a8eaf to 5a731af Compare October 14, 2026 02:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-model review (Opus 5.5, GPT 5.6 Sol, Grok 4.7)

Backoff math, route regexes, and debug-only CA trust look right for the single-operation happy path. A few harness bugs will make report lie or leave the device in a bad state.

Act on (3/3)

  • report.py groups by method + stripped path, so different records share one backoff counter. Scenario 11 (and any parallel ops on the same route) will print false EARLY/LATE. Key on the full path or body hash.
  • tap pipes Maestro through grep … || true, so a failed flow still exits 0. Scenarios that assert “not sent” become untrustworthy.
  • network-flap has no trap and always enables both radios. Ctrl-C, a failed svc, or wireless adb leaves the device offline.

Consider (2/3)

  • --add reloads remaining from original times, so an in-flight --times N rule starts over.
  • 404 is treated as terminal, but user executors retry 404 inside the post-create window.

Noted

  • Bind mitmdump to 127.0.0.1 (it currently listens on all interfaces).
  • fail … 0 --retry-after records a floor the SDK never sees.
  • float() Retry-After vs SDK toIntOrNull() (e.g. 1.5 → report 1.5s, SDK 60s).

Dismissed

  • Release CA leak — debug-overrides plus the debug source set only.
Open in Web View Automation 

Sent by Cursor Automation: PR Reviews

Comment thread examples/demo/scripts/retry/report.py Outdated
Comment thread examples/demo/scripts/retry/retry-test.sh Outdated
Comment thread examples/demo/scripts/retry/retry-test.sh Outdated
Comment thread examples/demo/scripts/retry/fault_proxy.py Outdated

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a trap in retry-test.sh that turns the radios back on and clears the proxy on exit. Drop || true on Maestro. Update the body, since #2790 now removes the global delay.

@fadi-george
fadi-george force-pushed the fadi/sdk-5535-retry-harness branch from 5a731af to e54c53b Compare October 14, 2026 18:11
@fadi-george
fadi-george force-pushed the fadi/sdk-5535-retry-harness branch from b70b016 to 17692ea Compare October 14, 2026 18:49

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

034b9f5 (trap, Maestro exit code, per-record report) isn't on the branch anymore. The rebase dropped it. When restoring it, leave out the .pyc. The README and report.py still describe network return ending backoff and "other routes are not held", neither of which matches #2790 now.

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still needs a rebase onto 9f61748 and 034b9f5 restored. report.py and the README model linear 15s backoff with no jitter. The new schedule is 5s doubling to 300s with 0.5 to 1x jitter, the gate holds all traffic during Retry-After, and a foreground skips the 5s batching wait.

Base automatically changed from fadi/sdk-5535 to main October 14, 2026 20:09
fadi-george and others added 4 commits October 14, 2026 13:12
Port of the iOS retry harness to the Android demo: mitmproxy fails SDK
requests, Maestro drives the demo, and report.py checks retry gaps
against OperationRepo's per-op backoff. Debug demo builds trust the
mitmproxy CA when built with -PRETRY_TEST_CA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… operation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… review issues

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fadi-george
fadi-george force-pushed the fadi/sdk-5535-retry-harness branch from 17692ea to 8dcf953 Compare October 14, 2026 20:19
@fadi-george

Copy link
Copy Markdown
Contributor Author

@abdulraqeeb33 rebased onto main, so this is just the harness now. Redid the fixes in 8dcf953:

  • network-flap has a trap that turns back on only the radios that were on, even after Ctrl-C, and it refuses wireless adb.
  • A failed Maestro flow fails tap now.
  • report groups attempts per record (by body for Create User and custom events), so other records don't share a counter.
  • README and report match the final fix: [SDK-5535] retry failed operations without blocking the queue #2790: 5s doubling to 300s with jitter, the gate holds all SDK traffic during Retry-After (report lists anything sent inside it), and a foreground skips the batching wait.
  • No .pyc, and __pycache__ is ignored.

From the Cursor review: --add keeps remaining counts, mitmdump listens on loopback only, status 0 with --retry-after is rejected, and Retry-After is parsed like HttpClient.

I left the device proxy alone on exit since it's meant to stay set between commands until stop. 404 also stays non-retried, since a retried one shows up as a new attempt rather than a false EARLY.

Reran backoff, other records, foreground, and network return on an emulator. Details are in the body.

…er check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@abdulraqeeb33 abdulraqeeb33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all the earlier points are fixed and the docs match what merged. Two small ones in retry-test.sh:

  1. A typo'd route installs a match-all rule. die inside $(route_rule ...) only exits the subshell, so read gets empty strings and the empty path matches every request. spec="$(route_rule "$route")" || exit $? before the read fixes it.
  2. network-flap only refuses serials with :. Android 11+ wireless debugging serials look like adb-XXXX._adb-tls-connect._tcp, so Wi-Fi goes off, adb drops, and the radios stay off. Also refuse *._adb-tls-connect._tcp, and don't print "Network restored." when svc enable fails.

Nits: restore stylus_handwriting_enabled on stop, and skip a truncated last line in report.py instead of crashing.

… in the retry harness

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fadi-george

Copy link
Copy Markdown
Contributor Author

@abdulraqeeb33 thanks, both were real. Fixed in 2c8c68d:

  1. A typo'd route now exits before any rule is written.
  2. network-flap also refuses ._adb-tls-connect._tcp serials, and it only says "Network restored." if svc enable worked. While I was in there, it now treats wifi_on=2 (Wi-Fi on in airplane mode) as on, and a SIGTERM runs the restore right away.

report.py skips a half-written last line now.

I'd leave stylus_handwriting_enabled off on stop. install sets it, not start, so after a stop and start, tap would start typing nothing again. It only turns off the handwriting prompt on a test device.

fadi-george and others added 2 commits October 14, 2026 14:22
…erification apps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fadi-george
fadi-george merged commit e79040a into main Oct 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants