Repository navigation
fix(odin): Make check-ins resilient to startup failures and transient network errors (EDG-1032) - #588
vikramlc-cognite wants to merge 5 commits into
Conversation
… network errors - Log and treat as fatal any failure building/reporting the startup request instead of letting the ExtractorCheckin daemon thread die silently. - Make the extractor process exit with a dedicated non-zero code on a fatal check-in error and have the runtime exit 1 instead of 0. - Treat raw requests/httpx transport errors and timeouts (including from the OAuth token refresh) as transient: warn, requeue and retry, not fatal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request improves the resilience of the extractor's check-in mechanism by ensuring that transient network errors are logged as warnings and retried, while unexpected failures during check-in or startup are treated as fatal, causing the extractor process to exit with a specific exit code. A critical review comment identifies a potential infinite restart loop: if a fatal error triggers cancellation, any exception raised during the shutdown of extractor.run() will bypass the fatal error check and trigger a restart. A code suggestion is provided to check the fatal error event before attempting a restart.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #588 +/- ##
==========================================
+ Coverage 84.20% 84.69% +0.49%
==========================================
Files 46 46
Lines 4711 4751 +40
==========================================
+ Hits 3967 4024 +57
+ Misses 744 727 -17
🚀 New features to boost your workflow:
|
… network errors - Log and treat as fatal any failure building/reporting the startup request instead of letting the ExtractorCheckin daemon thread die silently. - Make the extractor process exit with a dedicated non-zero code on a fatal check-in error and have the runtime exit 1 instead of 0. - Treat raw requests/httpx transport errors and timeouts (including from the OAuth token refresh) as transient: warn, requeue and retry, not fatal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
df3ae15 to
90affef
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request improves check-in resilience in the unstable package by ensuring that transient network errors (such as timeouts and connection issues from requests/httpx) are retried instead of shutting down the extractor, while actual fatal errors (like bad credentials or startup failures) correctly trigger a fatal shutdown with a dedicated exit code. The review feedback identifies two critical issues: first, a potential infinite restart loop if the extractor crashes during shutdown after a fatal error has already been set; and second, transient 5xx server errors from the identity provider during token refresh being incorrectly classified as fatal configuration errors rather than transient issues.
…itedata/python-extractor-utils into EDG-1032-checkin-resilience
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request improves the resilience of the extractor's check-in mechanism by distinguishing between transient network errors and fatal configuration or authentication issues. Transient errors (such as timeouts and connection failures) are now logged as warnings and retried, while unexpected failures during check-ins or startup are treated as fatal, causing the extractor to shut down and exit with a specific exit code to prevent automatic restarts. A comprehensive test suite has been added to verify these resilience behaviors. There are no review comments to address, and I have no additional feedback to provide.
Summary
The check-in no longer dies silently on startup errors, and transient network errors (including a timeout during OAuth token refresh) are retried instead of shutting the extractor down.
Type of change
What changed
base.py_run_checkin: any failure building or reporting the startup request (for example more than 100 available actions, or "Could not report startup") is logged at CRITICAL on the normal logger, treated as fatal, and cancels the extractor.checkin_worker.py_wrap_checkin_like_request: connection errors and timeouts (requestsTimeout/ConnectionError,httpxtimeouts/network errors,TimeoutError,ConnectionError, and an auth error whose cause is one of those) are logged as WARNING, requeued and retried at the next check-in. 401/403, bad credentials, config errors (invalid URL, missing schema, unsupported protocol) and unknown errors stay fatal.runtime.py: when the extractor stops because of a fatal check-in error, the child exits with code 3 and the runtime exits with a non-zero code (previously 0).notify_fatal_erroron the worker; tests intests/test_unstable/test_checkin_resilience.py; CHANGELOG.Why it changed
What to focus on during review
runtime.py(child code 3 -> runtimesys.exit(1)).Test evidence
tests/test_unstable/test_checkin_resilience.py: 26 tests (101 actions fatal and logged, 100 actions fine, transient errors requeued and not fatal, 401/403/bad credentials/config errors fatal, startup retry, child exit code)uv run pytest tests/test_unstable: 1 failed, 48 errors, all from missing live-CDF env vars, identical to a cleanmaster; passed count 248 -> 274ruff,ruff format,mypy,pre-commit run --all-filespassRisks and unknowns
sys.exit(1)) is covered on the child side only; it needs a manual check.Rollout and rollback
Checklist