Skip to content

fix(odin): Make check-ins resilient to startup failures and transient network errors (EDG-1032) - #588

Open
vikramlc-cognite wants to merge 5 commits into
masterfrom
EDG-1032-checkin-resilience
Open

vikramlc-cognite wants to merge 5 commits into
masterfrom
EDG-1032-checkin-resilience

Conversation

@vikramlc-cognite

Copy link
Copy Markdown
Contributor

Summary

The check-in no longer dies silently on startup errors, and transient network errors (including a timeout during OAuth token refresh) are retried instead of shutting the extractor down.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Refactor (no functional change)
  • Documentation update
  • Chore / tooling / CI

What changed

  • base.py _run_checkin: any failure building or reporting the startup request (for example more than 100 available actions, or "Could not report startup") is logged at CRITICAL on the normal logger, treated as fatal, and cancels the extractor.
  • checkin_worker.py _wrap_checkin_like_request: connection errors and timeouts (requests Timeout/ConnectionError, httpx timeouts/network errors, TimeoutError, ConnectionError, and an auth error whose cause is one of those) are logged as WARNING, requeued and retried at the next check-in. 401/403, bad credentials, config errors (invalid URL, missing schema, unsupported protocol) and unknown errors stay fatal.
  • runtime.py: when the extractor stops because of a fatal check-in error, the child exits with code 3 and the runtime exits with a non-zero code (previously 0).
  • New notify_fatal_error on the worker; tests in tests/test_unstable/test_checkin_resilience.py; CHANGELOG.

Why it changed

  • With 50 scheduled tasks the startup request exceeds the 100-action limit, the check-in thread died with only a stderr traceback, and the integration looked healthy but never checked in. A raw timeout from the token endpoint was treated as fatal and stopped the extractor with exit code 0.
  • Related issue: EDG-1032 (parent: EDG-1031)
  • Related docs / discussion: actions bug bash findings Don't specify minor sdk version #3 and Fix bug in config verification #4

What to focus on during review

  • The transient-error list is deliberately narrow; check it against errors you have seen in production.
  • The new exit-code path in runtime.py (child code 3 -> runtime sys.exit(1)).
  • Any exception escaping the periodic loop is now fatal instead of silently killing the thread.

Test evidence

  • tests/test_unstable/test_checkin_resilience.py: 26 tests (101 actions fatal and logged, 100 actions fine, transient errors requeued and not fatal, 401/403/bad credentials/config errors fatal, startup retry, child exit code)
  • uv run pytest tests/test_unstable: 1 failed, 48 errors, all from missing live-CDF env vars, identical to a clean master; passed count 248 -> 274
  • ruff, ruff format, mypy, pre-commit run --all-files pass
  • 14 of the 23 original tests fail with the production change reverted (the rest are regression guards)

Risks and unknowns

  • The runtime path (child code 3 -> sys.exit(1)) is covered on the child side only; it needs a manual check.
  • Without a runtime (extractor run directly) there is no fatal handler: it only logs and cancels, no exit code.
  • Live-CDF tests could not run locally; CI should be the real check.

Rollout and rollback

  • No config changes. Revert the commit to roll back.

Checklist

  • Self-reviewed the diff
  • Tests added or updated (or N/A with reason)
  • Docs updated (or N/A): CHANGELOG
  • No secrets, credentials, or PII committed
  • Breaking changes called out above and communicated to affected teams: exit code of a fatal check-in error is now non-zero

… network errors

- Log and treat as fatal any failure building/reporting the startup request
  instead of letting the ExtractorCheckin daemon thread die silently.
- Make the extractor process exit with a dedicated non-zero code on a fatal
  check-in error and have the runtime exit 1 instead of 0.
- Treat raw requests/httpx transport errors and timeouts (including from the
  OAuth token refresh) as transient: warn, requeue and retry, not fatal.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vikramlc-cognite
vikramlc-cognite requested a review from a team as a code owner October 26, 2026 09:16

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the resilience of the extractor's check-in mechanism by ensuring that transient network errors are logged as warnings and retried, while unexpected failures during check-in or startup are treated as fatal, causing the extractor process to exit with a specific exit code. A critical review comment identifies a potential infinite restart loop: if a fatal error triggers cancellation, any exception raised during the shutdown of extractor.run() will bypass the fatal error check and trigger a restart. A code suggestion is provided to check the fatal error event before attempting a restart.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.36364% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.69%. Comparing base (5b7c577) to head (9757298).

Files with missing lines Patch % Lines
cognite/extractorutils/unstable/core/runtime.py 77.77% 4 Missing ⚠️
cognite/extractorutils/unstable/core/base.py 77.77% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #588      +/-   ##
==========================================
+ Coverage   84.20%   84.69%   +0.49%     
==========================================
  Files          46       46              
  Lines        4711     4751      +40     
==========================================
+ Hits         3967     4024      +57     
+ Misses        744      727      -17     
Files with missing lines Coverage Δ
...ite/extractorutils/unstable/core/checkin_worker.py 98.41% <100.00%> (+5.58%) ⬆️
cognite/extractorutils/unstable/core/base.py 86.68% <77.77%> (-0.26%) ⬇️
cognite/extractorutils/unstable/core/runtime.py 77.03% <77.77%> (+4.47%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… network errors

- Log and treat as fatal any failure building/reporting the startup request
  instead of letting the ExtractorCheckin daemon thread die silently.
- Make the extractor process exit with a dedicated non-zero code on a fatal
  check-in error and have the runtime exit 1 instead of 0.
- Treat raw requests/httpx transport errors and timeouts (including from the
  OAuth token refresh) as transient: warn, requeue and retry, not fatal.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vikramlc-cognite
vikramlc-cognite force-pushed the EDG-1032-checkin-resilience branch from df3ae15 to 90affef Compare November 2, 2026 12:00
@vikramlc-cognite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves check-in resilience in the unstable package by ensuring that transient network errors (such as timeouts and connection issues from requests/httpx) are retried instead of shutting down the extractor, while actual fatal errors (like bad credentials or startup failures) correctly trigger a fatal shutdown with a dedicated exit code. The review feedback identifies two critical issues: first, a potential infinite restart loop if the extractor crashes during shutdown after a fatal error has already been set; and second, transient 5xx server errors from the identity provider during token refresh being incorrectly classified as fatal configuration errors rather than transient issues.

@vikramlc-cognite
vikramlc-cognite marked this pull request as ready for review November 3, 2026 08:33
@vikramlc-cognite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the resilience of the extractor's check-in mechanism by distinguishing between transient network errors and fatal configuration or authentication issues. Transient errors (such as timeouts and connection failures) are now logged as warnings and retried, while unexpected failures during check-ins or startup are treated as fatal, causing the extractor to shut down and exit with a specific exit code to prevent automatic restarts. A comprehensive test suite has been added to verify these resilience behaviors. There are no review comments to address, and I have no additional feedback to provide.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants