Repository navigation
Reopen gateway replica SSH tunnels whose ssh process exited - #4353
Closed
alvils-ailab-stratcom wants to merge 1 commit into
Closed
alvils-ailab-stratcom wants to merge 1 commit into
alvils-ailab-stratcom wants to merge 1 commit into
Conversation
The gateway opens one SSH tunnel per service replica when the replica is registered and reuses it for every request, but never checks it again. If the tunnel's ssh process exits (the replica's sshd drops a gateway that missed keepalives under load, a network blip, or the process is killed), nginx keeps forwarding to the abandoned Unix socket and the service returns 502 until the gateway is restarted, while probes, which use their own tunnels, stay green. Check every replica tunnel periodically (DSTACK_PROXY_TUNNEL_CHECK_INTERVAL, default 15 s) through its control socket and reopen the ones whose ssh process has exited. The local app socket path is kept, so nginx and the HTTP client need no reconfiguration. A stale control socket left by a killed ssh process is removed first, otherwise the new ssh would run without one and fail every check. A per-connection lock keeps a reopen from resurrecting a connection that is being closed. Fixes dstackai#4352
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopen dead gateway replica tunnels.
Fixes #4352