Repository navigation
sdk/python: a parked connection's liveness check works above fd 1024 - #319
Merged
Merged
Conversation
select.select raises ValueError for an fd at or above FD_SETSIZE, which the quiet check read as a dead peer, so in a process with 1024+ open fds every parked relay and control-plane connection was closed on take and each call dialed afresh. Sockets below FD_SETSIZE keep the select path, since poll costs 8 us per check on macOS against 0.5 us for select; higher fds use poll, and platforms without poll (Windows, whose select has no fd-number cap) keep select.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Conn.quiet()(conn.py) and_quiet()(client.py) checked a parked keep-alive connection withselect.select([sock], [], [], 0).selectraisesValueErrorfor any fd at or aboveFD_SETSIZE(1024). Both helpers caught that and returnedFalse, so the check reported the peer as dead. In a process with 1024 or more open fds, every parked relay connection and every parked control-plane HTTP connection was closed on take, and each call dialed again. Keep-alive stopped working with no error. The Go SDK does not have this problem because it peeks the raw fd withrecv(MSG_PEEK).Fix
The shared helper
conn.is_quiet(sock)now does both checks:selectcall as before, so nothing changes in the normal case (the extrafileno()costs +25 ns per check: 520 → 545 ns on macOS).select.poll()withPOLLIN. Any event (data, EOF,POLLHUP,POLLERR,POLLNVAL) means not quiet, which is whatselectreported as readable.select.poll(Windows) keepselect. Windowsselectlimits how many sockets you pass, not how high the handle numbers go.pollis not used for every fd because it costs 8 µs per check on macOS (Darwin builds it on kqueue), against 0.5 µs forselect. On Linux the two cost about the same (347 vs 379 ns).Both TLS and plain sockets work:
pollregisters the raw fd, asselectdid. I checked by hand against a self-signed TLS server, with the client SSLSocket at fd 1026 on macOS and at fd 1027 on Linux. An idle connection reports quiet. An unprompted byte reports not quiet. After that byte is read, it reports quiet again. A peer close reports not quiet, and so does a closed socket.MSG_PEEK | MSG_DONTWAITwas rejected.SSLSocket.recvrefuses flags. A socket with a timeout, as http.client's sockets have, waits up to that timeout before the peek.Tests
A new
fd_floorfixture inconftest.pyruns each test twice.low_fdopens no filler fds.high_fdraisesRLIMIT_NOFILEif needed and opens/dev/nulluntil it gets an fd of 1024 or higher, so every socket the SDK opens has an fd of 1024 or higher. Four existing keep-alive tests now take the fixture:test_calls_share_one_connection(relay, 3 stat + 1 exec), which also asserts that the parked socket's fd is at or above the floortest_peer_hang_up_is_noticed_before_reuse(relay, dead-peer detection through the poll branch)test_control_calls_reuse_one_connection(control-plane HTTP)test_a_parked_connection_the_node_closed_is_not_reused(control-plane HTTP, dead-peer detection through the poll branch)Reproduction: these are the same tests run against the old
cocoonsandbox/from ec9f99d, with dials counted per call.test_calls_share_one_connection(relay upgrades)AssertionError: 4)test_control_calls_reuse_one_connection[30.0-1](HTTP conns)AssertionError: 2)Timing
Setup: one process with fds 0–1024 taken, the SDK's sockets above 1024, and the in-process fakes from the test suite (
FakeAgent,KeptNode). Each run does 200 RPCs per kind after a warm-up call and reports the per-RPC median. Runs are interleaved old, new, new, old. Here "old" means the pre-fixselectcheck patched in. The fakes were set toTCP_NODELAY, as Go's listeners are by default. Without that, Linux shows 40 ms Nagle and delayed-ACK stalls on reused connections, which come from the fakes and not from the SDK.linux/arm64 (python:3.12 container), p50 per RPC:
infomacOS arm64, p50 per RPC:
infoOn loopback that is 5–10× per relay RPC and about 3.5× per control-plane call on Linux. Against a real node, each dial the fix removes also costs a network round trip, plus the TLS handshake on https.
Gates
macOS (python 3.12 venv, ruff 0.15.22, pytest 8.4.2, mypy 1.20.2), from
sdk/python:linux/arm64 (
python:3.12container):Not run: Windows. That path keeps the old
selectcall unchanged.