Skip to content

fix(spdk-start): pass the host's fence ports to the SPDK pod so the entrypoint stops flushing iptables (R26.3) - #1484

Merged
wmousa merged 1 commit into
R26.3from
fix/r263-scoped-fence-cleanup
Oct 15, 2026
Merged

wmousa merged 1 commit into
R26.3from
fix/r263-scoped-fence-cleanup

Conversation

@wmousa

@wmousa wmousa commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Problem

The ultra SPDK entrypoint ran iptables -F before every SPDK start to clear port fences left by a previous SPDK instance. The pod shares the host network namespace with the CNI, so the flush also wiped cilium's host chains: every SPDK restart on the ams1 cluster (2026-09/10) dropped host-to-pod traffic until the cilium agent was restarted.

The entrypoint is being changed (ultra fix/r263-scoped-fence-cleanup) to delete DROP/REJECT rules for an explicit port list only. It cannot derive that list itself: it sees RPC_PORT and FW_PORT, while the lvol and hublvol ports are allocated by the control plane.

Change

  • port_block.host_fence_ports(node, db): every port the control plane may have fenced on that host for this SPDK instance. The lvol and hublvol ports of the node's own lvstore and of every lvstore it backs as secondary or tertiary, read from both the per-lvstore map and the legacy node-level fields. The legacy fields matter: records written by 25.10.x only carry those, and the 25.10.x control plane is the one that leaves iptables fences behind during an online upgrade. After the node is on the 26.x image the control plane only uses the SPDK port-block RPCs and the health check no longer reads iptables, so nothing else would ever remove them.
  • restart_storage_node computes the list and passes it to spdk_process_start; SNodeClient sends it as fence_ports only when non-empty, so an older storage-node API sees the same request as before.
  • Both storage-node APIs (kubernetes.py, docker.py) accept fence_ports and render it as the FENCE_PORTS env of the SPDK container (space-separated); the pod template gets the env entry. An older image ignores the variable.

Verification

  • ruff and mypy clean on the touched files.
  • New tests/unit/test_fence_ports_for_spdk_start.py: legacy 25.10 record, primaries backed by the node, per-lvstore map with dedup and a malformed entry, unassigned ports, env rendering, client omits the param when empty, both SPDKParams accept it, template renders it.
  • Unit suite green locally (run with the 3.12 env; this branch's tox pins 3.9, so CI is the authoritative run).
  • Not cluster-tested yet. Needs: 25.10.x cluster, upgrade control plane to this build, restart one node onto the companion ultra image, confirm iptables -S on that host has no leftover --dport 91xx -j REJECT and that cilium's chains are intact.

Systemic causes

  • The port list a host can be fenced on existed only implicitly, scattered over the call sites of port_block.set_port and _node_lvs_ports in tasks_runner_port_allow.py. This PR adds a second derivation. One owner (the model: "ports this host serves") would remove both; cost moderate, it touches the port-allow recovery path.
  • The 26.x health check never consults iptables once the RPC exists (port_block.is_port_blocked), so a leftover host rule on a lvol port is invisible. A cheap guard: read iptables once per restart on the storage-node API side and log any --dport DROP/REJECT it finds on a simplyblock port.
  • spdk_process_start takes 24 positional/keyword parameters and grows by one per feature; a request model shared by client and both APIs would make additions like this one a single edit.

🤖 Generated with Claude Code

…ntrypoint stops flushing iptables

The ultra entrypoint ran `iptables -F` before every SPDK start to clear
port fences left by a previous SPDK instance. The pod shares the host
network namespace with the CNI, so the flush also wiped cilium's host
chains: every SPDK restart on the ams1 cluster (2026-09/10) dropped
host-to-pod traffic until the cilium agent was restarted.

The entrypoint now deletes DROP/REJECT rules for an explicit port list
only. It cannot derive that list itself (it sees RPC_PORT and FW_PORT,
the lvol and hublvol ports are allocated here), so the control plane
computes it at restart and passes it through spdk_process_start into the
pod as FENCE_PORTS.

The list is every port the control plane may have fenced on that host for
this SPDK instance: the lvol and hublvol ports of its own lvstore and of
every lvstore it backs as secondary or tertiary, read from both the
per-lvstore map and the legacy node-level fields. It has to cover what a
25.10.x control plane fenced, because the online upgrade restarts the node
onto an image that no longer runs the iptables firewall service; after
that the control plane only talks to SPDK's port-block RPCs and the health
check no longer reads iptables, so nothing else would ever remove those
rules.

The parameter is omitted when empty, so an older storage-node API sees
the same request as before, and an older image ignores the env var.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot added the target/R26.3 Targets release branch R26.3 label Oct 15, 2026
@wmousa
wmousa merged commit 88a50e2 into R26.3 Oct 15, 2026
8 checks passed
@wmousa
wmousa deleted the fix/r263-scoped-fence-cleanup branch October 15, 2026 13:06
wmousa added a commit that referenced this pull request Oct 17, 2026
…ient fallback (R26.3) (#1496)

* Revert "fix(spdk-start): pass the host's fence ports to the SPDK pod so the entrypoint stops flushing iptables (#1484)"

This reverts commit 88a50e2. The SPDK image no longer touches iptables
at all (ultra #218): 25.10.x -> 26.3 upgrades are offline, so no 25.10
fence rules need cleaning at SPDK start, and nothing reads FENCE_PORTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(port-block): fence only through the SPDK RPC, drop the FirewallClient fallback

port_block.set_port sent fences to the legacy firewall webapp
(FirewallClient, mgmt_ip:firewall_port) whenever node.spdk_version was
empty, and redirected a method-not-found from nvmf_port_block there too.
25.10.x node records never carried spdk_version, and it is only set by a
restart that passes --spdk-proxy-image, so a node upgraded to 26.3 kept
"" and every fence on it went to a port nothing listens on: the SPDK
image no longer ships the webapp or iptables (ultra #218). The fence
failed with connection refused.

- port_block: set_port / get_blocked_ports_set / is_port_blocked use the
  SPDK RPCs only; an RPC error, method-not-found included, propagates.
- health_controller.check_ports_on_node: drop the per-port legacy branch.
- Delete fw_api_client.py; port_block was its only user.
- Tests: the ftt2 and dual-FT fixtures patch port_block instead of
  FirewallClient; the expansion simulator answers nvmf_port_block /
  nvmf_port_unblock / nvmf_get_blocked_ports instead of stubbing
  FirewallClient; regression tests in test_port_block_rpc_only.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lint: annotate TTLCache._instances as ClassVar (RUF012)

The Linter job installs an unpinned ruff; 0.17.1 flags the class-level
WeakSet registry as a mutable class attribute. It is meant to be shared
across instances, so ClassVar states that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: remove the firewall-port plumbing and the jc dependency

Nothing has read any of this since fencing moved to the SPDK
nvmf_port_block RPC and the SPDK image dropped iptables and the firewall
webapp (ultra #218):

- StorageNode.firewall_port, the spdk_process_start(firewall_port=) param,
  the field in both node-API SPDKParams, the FW_PORT env on the SPDK
  container (docker and the k8s template) and constants.FW_PORT_START.
- utils.get_next_fw_port: no callers.
- The deprecated node-API routes /snode/firewall_set_port and
  /snode/get_firewall, which only returned False.
- StorageNode.spdk_version: only the removed FirewallClient fallback
  read it.
- The jc dependency (it parsed iptables output), from requirements.txt.

Existing FDB records keep their stored firewall_port / spdk_version keys;
BaseModel.from_dict only reads declared attributes, so they are ignored.

(cherry picked from commit d3459b9, part of #1497)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: remove utils.port_block, fence through the RPC client directly

With the FirewallClient fallback gone, port_block was a pass-through
(review on #1496):

- set_port only chose nvmf_port_block or nvmf_port_unblock, and every
  caller passed block= as a constant: the 13 call sites now call
  node.rpc_client(timeout=.., retry=..).nvmf_port_block/unblock(port)
  with the budget each already used. The rationale for the short fence
  budget moves to RPCClient.nvmf_port_block.
- get_blocked_ports_set was the only caller of
  RPCClient.nvmf_get_blocked_ports: the projection to a set of port
  numbers moves into that method.
- is_port_blocked was a one-line membership test; check_port_on_node
  does it inline.

Tests: they faked fencing by patching port_block.set_port, which also
told them which node a fence was for. tests._mocks.route_fences keeps
that: it wraps each node's rpc_client so nvmf_port_block/unblock reach
the same (node, port, block) callback the tests already used. Source
text anchors and "must not fence" assertions now look for the RPC
calls, and mocks of nvmf_get_blocked_ports return a set.

R26.3 port of 59a238b (part of #1497): the same edits replayed on
R26.3, where RPCClient calls _request3 and the two-node and
failback-listener tests do not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

target/R26.3 Targets release branch R26.3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants