Repository navigation
Fix gateway recovery after high request load - #4351
Merged
Merged
Conversation
peterschmidt85
marked this pull request as draft
October 7, 2026 12:36
peterschmidt85
marked this pull request as ready for review
October 7, 2026 13:43
jvstme
reviewed
Oct 7, 2026
| except BaseException: | ||
| # HTTPcore removes the failed/cancelled request, but can leave its newly | ||
| # assigned connection behind before it starts. That slot is never reused. | ||
| # Remove this orphan cleanup once https://github.com/encode/httpcore/pull/1099 ships. |
Collaborator
There was a problem hiding this comment.
As per the discussion in this PR, it may be a long time until the fix lands in httpcore and httpx, but the bug is already fixed in httpx2.
Should we consider migrating to httpx2? It's a drop-in replacement, so migrating might be easier than pinning, patching, and redistributing httpcore with its license, and we'll also benefit from other fixes and improvements.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4338.
High request load can leave services returning HTTP 500 even after the load stops.
LimitNOFILE=65535for the gateway Python service; nginx's limit does not apply to it.httpx2for authorization requests to dstack-server. Its pool releases connection slots abandoned by cancelled or timed-out requests and avoids scanning every connection for each waiting request.The migration is limited to gateway authorization; other dstack HTTP clients continue using
httpx.Deploying the fixed server release automatically updates and restarts existing gateways under default settings, including applying the new file-descriptor limit.
Follow-up: stress-test gateway capacity and recovery under overload. Document supported request rates and concurrency, latency/error targets, and which scaling options help.