Skip to content

fix(odin): Report the real outcome of Start and Stop task actions (EDG-1035) - #591

Draft
vikramlc-cognite wants to merge 1 commit into
EDG-1033-clean-shutdownfrom
EDG-1035-action-outcome-status
Draft

vikramlc-cognite wants to merge 1 commit into
EDG-1033-clean-shutdownfrom
EDG-1035-action-outcome-status

Conversation

@vikramlc-cognite

Copy link
Copy Markdown
Contributor

Summary

Start <task> now reports how the run actually went (failed, completed_with_errors, succeeded), and Stop <task> reports succeeded only once the task has ended.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Refactor (no functional change)
  • Documentation update
  • Chore / tooling / CI

What changed

  • tasks.py: thread-safe ErrorSummary on TaskContext, recording the count and first description/details per ErrorLevel.
  • base.py _start_task_outcome (rule, in this order): cancelled -> canceled; any fatal error or a raising task -> failed with the first fatal error's description and details as resultMessage (truncated); else any error-level error -> completed_with_errors ("Completed with N error(s): "); else succeeded (warnings only or nothing).
  • Stop <task>: reports running, waits until the task has ended, then succeeded. failed if the task is idle ("not currently running") or has not ended within STOP_TASK_ACTION_TIMEOUT_SECONDS (300 s); canceled if the Stop itself is cancelled (shutdown, user).
  • _dto.py: ActionStatus.completed_with_errors. Module docstring documents the rule; CHANGELOG.
  • Scheduled (non-action) runs are unaffected.

Why it changed

  • Tasks report problems on their context instead of raising, so a failed run ended succeeded (DB broken-query, File with a missing source). Stop reported succeeded as soon as the signal was sent.
  • Related issue: EDG-1035 (parent: EDG-1031)
  • Related docs / discussion: PM decision: failed run -> Failed with the error text, non-fatal errors -> "Completed with Errors", Stop only once stopped, one rule for all extractors. Stacked on EDG-1033. Needs Odin completed_with_errors (cognitedata/odin#324).

What to focus on during review

  • The mapping order (cancel wins over errors) and which levels count as run-ending. The DB extractor reports a whole failed query at level error, so until it reports it as fatal (follow-up in db-extractor) it ends completed_with_errors, not failed.
  • The Stop wait: it runs on its own action thread, is registered as a running action so shutdown and cancel can wake it, and has a 300 s bound.
  • One existing test was changed (it would block for 300 s under the new Stop behaviour).

Test evidence

  • tests/test_unstable/test_action_outcome.py: 14 tests (fatal, truncation, error-only, ctx.exception, warning-only, none, raising task, cancel, scheduled run, Stop waits, Stop timeout, Stop idle, Stop on shutdown)
  • uv run pytest tests/test_unstable: 300 passed; the same 1 failed + 48 environment errors as master
  • ruff, ruff format, mypy, pre-commit run --all-files pass
  • 8 of the 14 new tests fail with base.py/tasks.py reverted; the other 5 guard unchanged behaviour

Risks and unknowns

  • Behaviour change for every extractor: runs that report errors no longer end succeeded.
  • Until Odin deploys completed_with_errors (Add status codes to time series queue #324), a check-in carrying it may be rejected. Deploy Odin first.
  • A Stop on a task that is stuck in a blocking call keeps the action running in Odin for up to 300 s.

Rollout and rollback

Checklist

  • Self-reviewed the diff
  • Tests added or updated (or N/A with reason)
  • Docs updated (or N/A): module docstring, CHANGELOG
  • No secrets, credentials, or PII committed
  • Breaking changes called out above and communicated to affected teams: action outcome semantics changed (PM-approved); pending product communication

@vikramlc-cognite
vikramlc-cognite requested a review from a team as a code owner October 26, 2026 09:24

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new 'completed_with_errors' action status for tasks and improves the reporting of task outcomes. It adds an 'ErrorSummary' class to track errors during task execution, allowing 'Start ' actions to report more accurate statuses (failed, completed_with_errors, or succeeded) based on reported errors. Additionally, it updates 'Stop ' actions to wait for the task to finish before reporting success, with a configurable timeout. There are no review comments to address.

@vikramlc-cognite
vikramlc-cognite marked this pull request as draft October 26, 2026 10:22
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.10145% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.25%. Comparing base (e046cd1) to head (370f43a).

Files with missing lines Patch % Lines
cognite/extractorutils/unstable/core/base.py 96.00% 2 Missing ⚠️
Additional details and impacted files
@@                     Coverage Diff                     @@
##           EDG-1033-clean-shutdown     #591      +/-   ##
===========================================================
+ Coverage                    84.93%   85.25%   +0.31%     
===========================================================
  Files                           46       46              
  Lines                         4819     4882      +63     
===========================================================
+ Hits                          4093     4162      +69     
+ Misses                         726      720       -6     
Files with missing lines Coverage Δ
cognite/extractorutils/unstable/core/_dto.py 98.44% <100.00%> (+0.01%) ⬆️
cognite/extractorutils/unstable/core/tasks.py 100.00% <100.00%> (ø)
cognite/extractorutils/unstable/core/base.py 89.31% <96.00%> (+0.87%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Start <task> now maps the errors reported on the TaskContext to the action
status (failed / completed_with_errors / succeeded), and Stop <task> stays
running until the task has ended.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@vikramlc-cognite
vikramlc-cognite force-pushed the EDG-1035-action-outcome-status branch from 5b694dc to 370f43a Compare November 2, 2026 12:00

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants