Skip to content

fix: Do not resend commands after a timeout or server disconnect - #2286

Closed
iMicknl wants to merge 4 commits into
mainfrom
fix/no-timeout-retry-non-idempotent
Closed

iMicknl wants to merge 4 commits into
mainfrom
fix/no-timeout-retry-non-idempotent

Conversation

@iMicknl

@iMicknl iMicknl commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Problem

retry_on_connection_failure (TimeoutError, ClientConnectorError, ServerDisconnectedError) wraps every request helper, including _post, _put, and _delete. A timeout or dropped connection can happen after the server has already accepted a request. When that happens, writes such as exec/apply, exec/schedule, cancelling an execution, local token activation, openPairing, and developer mode get resent. A command can then run twice, or the retry hits DuplicateActionOnDeviceError while the first one is still running. The 15s default session timeout makes this more likely.

This is a v1 to v2 regression. In v1 only fetch_events retried on timeouts. execute_commands did not retry.

Fix

  • retry_on_unsent_request (ClientConnectorError, ConnectionTimeoutError) covers failures where no connection was made, so the request was never sent. It applies to all requests, including PUT and DELETE.
  • retry_on_connection_failure keeps its existing budget, but it now only applies to idempotent requests: _get, _authenticate, and _post(..., idempotent=True). Only events/register, events/{id}/fetch, and reference/devices/search pass that flag.
  • The intended behaviour from feat: harden HTTP resiliency with connection retries, request timeout, and docs #2148 and fix: Bound authentication and listener recovery retries #2262 is unchanged for idempotent calls: ServerDisconnectedError is retried without a relogin, and auth and listener registration keep their transport budgets.
  • docs/resiliency.md is updated.

Callers of execute_action_group and other writes will now see TimeoutError or ServerDisconnectedError raised directly instead of a silent resend.

Tests

  • execute_action_group is not retried on TimeoutError or ServerDisconnectedError (1 POST).
  • execute_action_group is still retried on ClientConnectorError.
  • fetch_events (an idempotent POST) is still retried on TimeoutError.
  • The existing GET test still covers retries on TimeoutError.

648 tests pass. ruff, ruff format, mypy, and ty are clean.

Timeouts and server disconnects may occur after the server accepted the
request, so resending exec/apply and other writes could run a command
twice. Only retry these failures for idempotent requests (GETs, auth,
event listener register/fetch, reference search); retry connection
setup failures for every request.
@iMicknl
iMicknl requested a review from tetienne as a code owner October 11, 2026 17:51
@iMicknl iMicknl added the bug Something isn't working label Oct 11, 2026
Copilot AI balanced review requested due to automatic review settings October 11, 2026 17:51
@iMicknl iMicknl added the bug Something isn't working label Oct 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Timeout documentation remains contradictory, and the new connection-timeout behavior lacks direct regression coverage.

3 open findings
What changed in this PR

Prevents duplicate write operations after ambiguous network failures while preserving retries for unsent and idempotent requests.

Changes:

  • Splits retry policies by request safety.
  • Marks safe POST endpoints as idempotent.
  • Adds regression tests and resiliency documentation.
File Description
pyoverkiz/​client.py Applies safety-aware retry policies.
tests/​test_client.py Tests retry and non-retry behavior.
docs/​resiliency.md Documents updated retry semantics.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_client.py Outdated
Comment on lines +97 to +100
side_effect=[
aiohttp.ClientConnectorError(MagicMock(), OSError("refused")),
resp,
],

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now exercises both ClientConnectorError and ConnectionTimeoutError on execute_action_group; plain TimeoutError non-resend test kept. (43438db)

Comment thread docs/resiliency.md Outdated
Comment on lines +16 to +17
| Connection not established — `ClientConnectorError`, `ConnectionTimeoutError` | yes | 3 tries / ~30s | reopen the connection |
| Timeouts and dropped connections — `TimeoutError`, `ServerDisconnectedError` | idempotent requests only (GETs, authentication, event listener register/fetch, reference device search) | 3 tries / ~30s | reopen the connection |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TimeoutError row now explicitly excludes ConnectionTimeoutError. (43438db)

Comment thread docs/resiliency.md Outdated
Comment on lines +60 to +64
Commands and other writes (`execute_action_group`, cancelling executions, pairing,
developer mode, and so on) are **not** resent after a `TimeoutError` or
`ServerDisconnectedError`: the server may already have accepted the request, so a
resend could run the command twice. Handle these errors yourself, for example by
checking `get_current_executions()` before trying again.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Qualified the max_time note and Request timeout section as retried for idempotent requests only. (43438db)

@iMicknl iMicknl closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants