Skip to content

Fix KeyError in SocketInterface.shutdown() when worker exits during wait - #1063

Merged
jan-janssen merged 2 commits into
mainfrom
fix/socket-interface-shutdown-race
Sep 11, 2026
Merged

Fix KeyError in SocketInterface.shutdown() when worker exits during wait#1063
jan-janssen merged 2 commits into
mainfrom
fix/socket-interface-shutdown-race

Conversation

@jan-janssen

@jan-janssen jan-janssen commented Sep 11, 2026

Copy link
Copy Markdown
Member

Summary

  • SocketInterface.shutdown() unconditionally indexed the reply dict returned by send_and_receive_dict() with ["result"].
  • receive_dict() can instead return {"error": ExecutorlibSocketError(...)} if the worker process exits while shutdown() is waiting for its reply (a benign race - the worker simply finished/exited on its own).
  • That race raised a KeyError: 'result' inside SocketInterface.__del__, which Python cannot propagate from a destructor, so it prints a confusing Exception ignored in: <function SocketInterface.__del__> traceback during garbage collection (e.g. when Jupyter releases old cached objects) even though nothing actually crashed.
  • Fix: only read the "result" key when it's present, otherwise shutdown() returns None for this benign race instead of raising.

Test plan

  • Added test_interface_shutdown_with_process_exiting_during_wait, which uses a spawner whose poll() reports the process alive only once (matching the real race window) and asserts shutdown() no longer raises.
  • Full test suite run pending (this environment's MPI/network setup makes the broader suite slow to run interactively).

Summary by CodeRabbit

  • Bug Fixes
    • Improved shutdown handling when a worker exits while the interface is waiting for a response.
    • Shutdown now completes reliably when the worker’s response does not include a result.
    • Ensured shutdown returns the expected empty result and properly resets the connection state.
    • Prevented shutdown waits from leaving the communication interface in an inconsistent state.

shutdown() unconditionally indexed the reply dict with ["result"], but
receive_dict() returns {"error": ExecutorlibSocketError(...)} instead
when the worker process exits while shutdown() is waiting for its
reply. That race raises a KeyError inside SocketInterface.__del__,
which surfaces as a confusing "Exception ignored in ... __del__"
traceback during garbage collection even though nothing actually
crashed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 6c26f541-ee3b-4327-a2cb-c47337d1cc6a

📥 Commits

Reviewing files that changed from the base of the PR and between a9037e9 and f839b1b.

📒 Files selected for processing (1)
  • tests/unit/standalone/interactive/test_communication.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

SocketInterface.shutdown now tolerates shutdown responses without a "result" key. A regression test simulates a worker exiting while shutdown(wait=True) waits for a response and verifies that the method returns None.

Changes

Interactive shutdown

Layer / File(s) Summary
Shutdown response handling and regression coverage
src/executorlib/standalone/interactive/communication.py, tests/unit/standalone/interactive/test_communication.py
SocketInterface.shutdown retains None when the response has no "result" key. The test double simulates process exit during the wait, and the test verifies that shutdown(wait=True) returns None.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to f839b

Interactive shutdown now handles a worker exiting during response waiting without raising a destructor traceback.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: fixing the KeyError in SocketInterface.shutdown() when the worker exits during the wait.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/socket-interface-shutdown-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jan-janssen
jan-janssen marked this pull request as draft September 11, 2026 06:42
Updated test to bind to a random port and connect a peer to prevent blocking during shutdown.
@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.26%. Comparing base (851b1a0) to head (f839b1b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1063   +/-   ##
=======================================
  Coverage   94.25%   94.26%           
=======================================
  Files          39       39           
  Lines        2176     2178    +2     
=======================================
+ Hits         2051     2053    +2     
  Misses        125      125           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jan-janssen
jan-janssen marked this pull request as ready for review September 11, 2026 07:37
@jan-janssen
jan-janssen merged commit d2f7916 into main Sep 11, 2026
66 of 69 checks passed
@jan-janssen
jan-janssen deleted the fix/socket-interface-shutdown-race branch September 11, 2026 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant