Skip to content

sdk/python: a parked connection's liveness check works above fd 1024 - #319

Merged
CMGS merged 1 commit into
mainfrom
perf/py-keepalive-high-fd
Oct 5, 2026
Merged

CMGS merged 1 commit into
mainfrom
perf/py-keepalive-high-fd

Conversation

@CMGS

@CMGS CMGS commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Bug

Conn.quiet() (conn.py) and _quiet() (client.py) checked a parked keep-alive connection with select.select([sock], [], [], 0). select raises ValueError for any fd at or above FD_SETSIZE (1024). Both helpers caught that and returned False, so the check reported the peer as dead. In a process with 1024 or more open fds, every parked relay connection and every parked control-plane HTTP connection was closed on take, and each call dialed again. Keep-alive stopped working with no error. The Go SDK does not have this problem because it peeks the raw fd with recv(MSG_PEEK).

Fix

The shared helper conn.is_quiet(sock) now does both checks:

  • fd < 1024: the same select call as before, so nothing changes in the normal case (the extra fileno() costs +25 ns per check: 520 → 545 ns on macOS).
  • fd >= 1024: select.poll() with POLLIN. Any event (data, EOF, POLLHUP, POLLERR, POLLNVAL) means not quiet, which is what select reported as readable.
  • Platforms without select.poll (Windows) keep select. Windows select limits how many sockets you pass, not how high the handle numbers go.

poll is not used for every fd because it costs 8 µs per check on macOS (Darwin builds it on kqueue), against 0.5 µs for select. On Linux the two cost about the same (347 vs 379 ns).

Both TLS and plain sockets work: poll registers the raw fd, as select did. I checked by hand against a self-signed TLS server, with the client SSLSocket at fd 1026 on macOS and at fd 1027 on Linux. An idle connection reports quiet. An unprompted byte reports not quiet. After that byte is read, it reports quiet again. A peer close reports not quiet, and so does a closed socket.

MSG_PEEK | MSG_DONTWAIT was rejected. SSLSocket.recv refuses flags. A socket with a timeout, as http.client's sockets have, waits up to that timeout before the peek.

Tests

A new fd_floor fixture in conftest.py runs each test twice. low_fd opens no filler fds. high_fd raises RLIMIT_NOFILE if needed and opens /dev/null until it gets an fd of 1024 or higher, so every socket the SDK opens has an fd of 1024 or higher. Four existing keep-alive tests now take the fixture:

  • test_calls_share_one_connection (relay, 3 stat + 1 exec), which also asserts that the parked socket's fd is at or above the floor
  • test_peer_hang_up_is_noticed_before_reuse (relay, dead-peer detection through the poll branch)
  • test_control_calls_reuse_one_connection (control-plane HTTP)
  • test_a_parked_connection_the_node_closed_is_not_reused (control-plane HTTP, dead-peer detection through the poll branch)

Reproduction: these are the same tests run against the old cocoonsandbox/ from ec9f99d, with dials counted per call.

test (high_fd) RPCs dials before dials after
test_calls_share_one_connection (relay upgrades) 4 4 (AssertionError: 4) 1
test_control_calls_reuse_one_connection[30.0-1] (HTTP conns) 2 2 (AssertionError: 2) 1
# old code, macOS and linux/arm64 alike
FAILED tests/test_client.py::test_control_calls_reuse_one_connection[high_fd-30.0-1]
FAILED tests/test_keepalive.py::test_calls_share_one_connection[high_fd] - As...
2 failed, 4 passed, 223 deselected

Timing

Setup: one process with fds 0–1024 taken, the SDK's sockets above 1024, and the in-process fakes from the test suite (FakeAgent, KeptNode). Each run does 200 RPCs per kind after a warm-up call and reports the per-RPC median. Runs are interleaved old, new, new, old. Here "old" means the pre-fix select check patched in. The fakes were set to TCP_NODELAY, as Go's listeners are by default. Without that, Linux shows 40 ms Nagle and delayed-ACK stalls on reused connections, which come from the fakes and not from the SDK.

linux/arm64 (python:3.12 container), p50 per RPC:

run exec stat control-plane info dials/RPC
old 147 µs 129 µs 191 µs 1.00
new 28 µs 14 µs 62 µs 0.00
new 28 µs 13 µs 61 µs 0.00
old 180 µs 146 µs 254 µs 1.00
old 168 µs 131 µs 228 µs 1.00
new 28 µs 14 µs 59 µs 0.00
new 28 µs 13 µs 57 µs 0.00
old 144 µs 129 µs 211 µs 1.00

macOS arm64, p50 per RPC:

run exec stat control-plane info dials/RPC
old 252 µs 201 µs 252 µs 1.00
new 112 µs 50 µs 112 µs 0.00
new 103 µs 81 µs 112 µs 0.00
old 261 µs 186 µs 249 µs 1.00

On loopback that is 5–10× per relay RPC and about 3.5× per control-plane call on Linux. Against a real node, each dial the fix removes also costs a network round trip, plus the TLS handshake on https.

Gates

macOS (python 3.12 venv, ruff 0.15.22, pytest 8.4.2, mypy 1.20.2), from sdk/python:

$ ruff format --check .
23 files already formatted
$ ruff check .
All checks passed!
$ python -m mypy
Success: no issues found in 9 source files
$ python -m pytest -q
229 passed in 38.37s

linux/arm64 (python:3.12 container):

23 files already formatted
All checks passed!
229 passed in 35.88s
PASSED tests/test_client.py::test_control_calls_reuse_one_connection[high_fd-30.0-1]
PASSED tests/test_client.py::test_control_calls_reuse_one_connection[high_fd-0-2]
PASSED tests/test_client.py::test_a_parked_connection_the_node_closed_is_not_reused[high_fd]
PASSED tests/test_keepalive.py::test_calls_share_one_connection[high_fd]
PASSED tests/test_keepalive.py::test_peer_hang_up_is_noticed_before_reuse[high_fd]

Not run: Windows. That path keeps the old select call unchanged.

select.select raises ValueError for an fd at or above FD_SETSIZE, which the
quiet check read as a dead peer, so in a process with 1024+ open fds every
parked relay and control-plane connection was closed on take and each call
dialed afresh. Sockets below FD_SETSIZE keep the select path, since poll
costs 8 us per check on macOS against 0.5 us for select; higher fds use poll,
and platforms without poll (Windows, whose select has no fd-number cap) keep
select.
@CMGS
CMGS merged commit 789507b into main Oct 5, 2026
2 checks passed
@CMGS
CMGS deleted the perf/py-keepalive-high-fd branch October 5, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant