Repository navigation
fix(spdk-start): pass the host's fence ports to the SPDK pod so the entrypoint stops flushing iptables (R26.3) - #1484
Merged
Conversation
…ntrypoint stops flushing iptables The ultra entrypoint ran `iptables -F` before every SPDK start to clear port fences left by a previous SPDK instance. The pod shares the host network namespace with the CNI, so the flush also wiped cilium's host chains: every SPDK restart on the ams1 cluster (2026-09/10) dropped host-to-pod traffic until the cilium agent was restarted. The entrypoint now deletes DROP/REJECT rules for an explicit port list only. It cannot derive that list itself (it sees RPC_PORT and FW_PORT, the lvol and hublvol ports are allocated here), so the control plane computes it at restart and passes it through spdk_process_start into the pod as FENCE_PORTS. The list is every port the control plane may have fenced on that host for this SPDK instance: the lvol and hublvol ports of its own lvstore and of every lvstore it backs as secondary or tertiary, read from both the per-lvstore map and the legacy node-level fields. It has to cover what a 25.10.x control plane fenced, because the online upgrade restarts the node onto an image that no longer runs the iptables firewall service; after that the control plane only talks to SPDK's port-block RPCs and the health check no longer reads iptables, so nothing else would ever remove those rules. The parameter is omitted when empty, so an older storage-node API sees the same request as before, and an older image ignores the env var. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Hamdy-khader
approved these changes
Oct 15, 2026
This was referenced Oct 16, 2026
Merged
wmousa
added a commit
that referenced
this pull request
Oct 17, 2026
…ient fallback (R26.3) (#1496) * Revert "fix(spdk-start): pass the host's fence ports to the SPDK pod so the entrypoint stops flushing iptables (#1484)" This reverts commit 88a50e2. The SPDK image no longer touches iptables at all (ultra #218): 25.10.x -> 26.3 upgrades are offline, so no 25.10 fence rules need cleaning at SPDK start, and nothing reads FENCE_PORTS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(port-block): fence only through the SPDK RPC, drop the FirewallClient fallback port_block.set_port sent fences to the legacy firewall webapp (FirewallClient, mgmt_ip:firewall_port) whenever node.spdk_version was empty, and redirected a method-not-found from nvmf_port_block there too. 25.10.x node records never carried spdk_version, and it is only set by a restart that passes --spdk-proxy-image, so a node upgraded to 26.3 kept "" and every fence on it went to a port nothing listens on: the SPDK image no longer ships the webapp or iptables (ultra #218). The fence failed with connection refused. - port_block: set_port / get_blocked_ports_set / is_port_blocked use the SPDK RPCs only; an RPC error, method-not-found included, propagates. - health_controller.check_ports_on_node: drop the per-port legacy branch. - Delete fw_api_client.py; port_block was its only user. - Tests: the ftt2 and dual-FT fixtures patch port_block instead of FirewallClient; the expansion simulator answers nvmf_port_block / nvmf_port_unblock / nvmf_get_blocked_ports instead of stubbing FirewallClient; regression tests in test_port_block_rpc_only.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lint: annotate TTLCache._instances as ClassVar (RUF012) The Linter job installs an unpinned ruff; 0.17.1 flags the class-level WeakSet registry as a mutable class attribute. It is meant to be shared across instances, so ClassVar states that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: remove the firewall-port plumbing and the jc dependency Nothing has read any of this since fencing moved to the SPDK nvmf_port_block RPC and the SPDK image dropped iptables and the firewall webapp (ultra #218): - StorageNode.firewall_port, the spdk_process_start(firewall_port=) param, the field in both node-API SPDKParams, the FW_PORT env on the SPDK container (docker and the k8s template) and constants.FW_PORT_START. - utils.get_next_fw_port: no callers. - The deprecated node-API routes /snode/firewall_set_port and /snode/get_firewall, which only returned False. - StorageNode.spdk_version: only the removed FirewallClient fallback read it. - The jc dependency (it parsed iptables output), from requirements.txt. Existing FDB records keep their stored firewall_port / spdk_version keys; BaseModel.from_dict only reads declared attributes, so they are ignored. (cherry picked from commit d3459b9, part of #1497) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: remove utils.port_block, fence through the RPC client directly With the FirewallClient fallback gone, port_block was a pass-through (review on #1496): - set_port only chose nvmf_port_block or nvmf_port_unblock, and every caller passed block= as a constant: the 13 call sites now call node.rpc_client(timeout=.., retry=..).nvmf_port_block/unblock(port) with the budget each already used. The rationale for the short fence budget moves to RPCClient.nvmf_port_block. - get_blocked_ports_set was the only caller of RPCClient.nvmf_get_blocked_ports: the projection to a set of port numbers moves into that method. - is_port_blocked was a one-line membership test; check_port_on_node does it inline. Tests: they faked fencing by patching port_block.set_port, which also told them which node a fence was for. tests._mocks.route_fences keeps that: it wraps each node's rpc_client so nvmf_port_block/unblock reach the same (node, port, block) callback the tests already used. Source text anchors and "must not fence" assertions now look for the RPC calls, and mocks of nvmf_get_blocked_ports return a set. R26.3 port of 59a238b (part of #1497): the same edits replayed on R26.3, where RPCClient calls _request3 and the two-node and failback-listener tests do not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ultra SPDK entrypoint ran
iptables -Fbefore every SPDK start to clear port fences left by a previous SPDK instance. The pod shares the host network namespace with the CNI, so the flush also wiped cilium's host chains: every SPDK restart on the ams1 cluster (2026-09/10) dropped host-to-pod traffic until the cilium agent was restarted.The entrypoint is being changed (ultra
fix/r263-scoped-fence-cleanup) to delete DROP/REJECT rules for an explicit port list only. It cannot derive that list itself: it seesRPC_PORTandFW_PORT, while the lvol and hublvol ports are allocated by the control plane.Change
port_block.host_fence_ports(node, db): every port the control plane may have fenced on that host for this SPDK instance. The lvol and hublvol ports of the node's own lvstore and of every lvstore it backs as secondary or tertiary, read from both the per-lvstore map and the legacy node-level fields. The legacy fields matter: records written by 25.10.x only carry those, and the 25.10.x control plane is the one that leaves iptables fences behind during an online upgrade. After the node is on the 26.x image the control plane only uses the SPDK port-block RPCs and the health check no longer reads iptables, so nothing else would ever remove them.restart_storage_nodecomputes the list and passes it tospdk_process_start;SNodeClientsends it asfence_portsonly when non-empty, so an older storage-node API sees the same request as before.kubernetes.py,docker.py) acceptfence_portsand render it as theFENCE_PORTSenv of the SPDK container (space-separated); the pod template gets the env entry. An older image ignores the variable.Verification
ruffandmypyclean on the touched files.tests/unit/test_fence_ports_for_spdk_start.py: legacy 25.10 record, primaries backed by the node, per-lvstore map with dedup and a malformed entry, unassigned ports, env rendering, client omits the param when empty, bothSPDKParamsaccept it, template renders it.iptables -Son that host has no leftover--dport 91xx -j REJECTand that cilium's chains are intact.Systemic causes
port_block.set_portand_node_lvs_portsintasks_runner_port_allow.py. This PR adds a second derivation. One owner (the model: "ports this host serves") would remove both; cost moderate, it touches the port-allow recovery path.port_block.is_port_blocked), so a leftover host rule on a lvol port is invisible. A cheap guard: read iptables once per restart on the storage-node API side and log any--dportDROP/REJECT it finds on a simplyblock port.spdk_process_starttakes 24 positional/keyword parameters and grows by one per feature; a request model shared by client and both APIs would make additions like this one a single edit.🤖 Generated with Claude Code