Skip to content

fix(s3): retry refused connections and unresolvable hosts - #170

Closed
HarshMN2345 wants to merge 1 commit into
mainfrom
fix/retry-connection-failures
Closed

fix(s3): retry refused connections and unresolvable hosts#170
HarshMN2345 wants to merge 1 commit into
mainfrom
fix/retry-connection-failures

Conversation

@HarshMN2345

Copy link
Copy Markdown
Member

Problem

S3\RetryStrategy::delay() opened with:

if ($attempt > $this->retries || ! $response instanceof ResponseInterface) {
    return null;
}

The second clause declines whenever there is no response, so the strategy only retried failures the service had answered — 429/503, SlowDown, Throttling. A refused connection or an unresolvable host went straight through on the first attempt, even though the Retry decorator already catches ClientExceptionInterface and calls delay($request, $attempt, null, $error). The $error argument was simply never used.

Production impact

A DigitalOcean Spaces endpoint refused connections for ~35 minutes. Every upload through it failed on the first attempt:

Failed to connect to <bucket>.fra1.digitaloceanspaces.com:443
after 5 ms: Could not connect to server        (CURLE_COULDNT_CONNECT)

In our case the payloads had already been produced by earlier work, so each failure discarded work that had succeeded.

Change

Retry when the request never reached the service:

  • ConnectionExceptionCURLE_COULDNT_CONNECT and friends
  • DnsExceptionCURLE_COULDNT_RESOLVE_HOST

Replaying these cannot duplicate an effect, because nothing was delivered.

Deliberately unchanged: NetworkException, TimeoutException and other transport failures stay non-retryable — the request may have been applied server-side, so a retry could duplicate it. The existing testTransportErrorsAreNotRetried (which uses NetworkException) still passes unmodified.

This does widen the strategy's scope beyond rate-limiting, so the class docblock is updated to say so.

Tests

Three added: refused connection retried, unresolvable host retried, and the retry cap honoured on the new path.

Verified against the real classes — the two new paths retry, and all existing behaviour is unchanged:

PASS  refused connection is retried
PASS  unresolvable host is retried
PASS  network error still NOT retried
PASS  SlowDown still retried
PASS  non-transient XML still NOT retried
PASS  connection retries respect the cap

Note: composer.json has no require-dev, so phpunit is not vendored and I could not run the suite through it locally — the above exercises the same assertions directly. Worth a CI run to confirm.

🤖 Generated with Claude Code

The strategy returned null whenever there was no response, so it only
retried failures the service had answered. A refused connection or an
unresolvable host was passed straight through, even though the request never
reached the service and replaying it cannot duplicate an effect.

Seen in production: a Spaces endpoint refused connections for ~35 minutes and
every upload failed on the first attempt with CURLE_COULDNT_CONNECT.

Other transport failures stay non-retryable, since a reset or a timeout may
have been applied server-side.
@github-actions

Copy link
Copy Markdown

Thanks for contributing! This repository is a read-only mirror; development for this library happens in packages/storage in the utopia-php monorepo. Please open this pull request there instead.

@github-actions github-actions Bot closed this Aug 28, 2026
@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown

Greptile Summary

The PR extends the S3 retry strategy to retry DNS and connection exceptions when no HTTP response exists.

  • Adds retry handling for ConnectionException and DnsException.
  • Adds tests for refused connections, DNS failures, and retry exhaustion.
  • Broad ConnectionException handling also includes ambiguous post-delivery failures, which can replay mutating requests.

Confidence Score: 4/5

The PR should not merge until ConnectionException retries are restricted to failures that guarantee the request was not delivered.

The new broad ConnectionException branch retries send and receive failures that may occur after a mutating S3 request has already been applied, allowing operations such as multipart-upload creation to execute twice.

Files Needing Attention: src/Storage/Device/S3/RetryStrategy.php, tests/Storage/Device/S3/RetryStrategyTest.php

Important Files Changed

Filename Overview
src/Storage/Device/S3/RetryStrategy.php Adds no-response retries, but the ConnectionException check is broader than refused connections and can replay requests after ambiguous delivery.
tests/Storage/Device/S3/RetryStrategyTest.php Covers refused and DNS failures plus the retry cap, but constructs only a refused ConnectionException and does not cover post-delivery failures represented by the same class.

Fix all with Greploop Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
### Issue 1
src/Storage/Device/S3/RetryStrategy.php:68
**Retries include delivered requests**

When a mutating S3 request fails during sending or response reception, the client reports several such failures as `ConnectionException`, and this branch retries the request even though the service may already have applied it. Replaying a non-idempotent request such as multipart-upload creation can create a second orphaned upload with associated storage costs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "fix(s3): retry refused connections and u..." | Re-trigger Greptile


if (! $this->isTransient($response)) {
$retryable = $response instanceof ResponseInterface
? $this->isTransient($response)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Retries include delivered requests

When a mutating S3 request fails during sending or response reception, the client reports several such failures as ConnectionException, and this branch retries the request even though the service may already have applied it. Replaying a non-idempotent request such as multipart-upload creation can create a second orphaned upload with associated storage costs.

Knowledge Base Used: S3 transport resilience

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/Storage/Device/S3/RetryStrategy.php
Line: 68

Comment:
**Retries include delivered requests**

When a mutating S3 request fails during sending or response reception, the client reports several such failures as `ConnectionException`, and this branch retries the request even though the service may already have applied it. Replaying a non-idempotent request such as multipart-upload creation can create a second orphaned upload with associated storage costs.

**Knowledge Base Used:** [S3 transport resilience](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/utopia-php/storage/-/docs/s3-transport-resilience.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant