Skip to content

fix(port-block): fence only through the SPDK RPC, drop the FirewallClient fallback (R26.3) - #1496

Merged
wmousa merged 5 commits into
R26.3from
fix/r263-remove-firewall-fallback
Oct 17, 2026
Merged

wmousa merged 5 commits into
R26.3from
fix/r263-remove-firewall-fallback

Conversation

@wmousa

@wmousa wmousa commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The control plane now fences ports only through the SPDK RPC, and the iptables firewall-API fallback is gone. This is the control-plane half of ultra #218, which removed iptables and the firewall webapp from the SPDK image (already cherry-picked to every ultra branch from R26.3 up and main).

Commit 1: revert #1484. FENCE_PORTS existed only so the ultra entrypoint could clean up 25.10 fence rules. That entrypoint code is gone, and 25.10.x → 26.3 upgrades are offline, so nothing reads the variable any more.

Commit 2: drop the fallback. port_block.set_port used to send fences to FirewallClient (mgmt_ip:firewall_port) in two cases:

  • whenever node.spdk_version == "";
  • whenever nvmf_port_block returned method-not-found.

25.10.x node records have no spdk_version, and it is only set by a restart that passes --spdk-proxy-image. Every upgraded node therefore kept "", and every fence on it went to a port that nothing listens on in 26.3. The fence failed with connection refused.

Changes:

  • port_block: set_port, get_blocked_ports_set and is_port_blocked use the SPDK RPCs only. Any RPC error, method-not-found included, propagates.
  • health_controller.check_ports_on_node: the per-port legacy branch is removed.
  • fw_api_client.py: deleted; port_block was its only user.
  • Tests:
    • The ftt2 and dual-FT fixtures patch port_block instead of FirewallClient.
    • The expansion simulator answers the three port RPCs instead of stubbing FirewallClient.
    • New regression tests in tests/unit/test_port_block_rpc_only.py, and the old fallback test in test_port_block_leak_k8s_0911.py now asserts that the error propagates.

Test plan

  • ruff check, mypy simplyblock_web simplyblock_cli simplyblock_core: clean
  • Unit suite: 2773 passed, 7 skipped
  • Integration tier (needs FDB; it collects, 278 tests in the touched suites, but was not run locally): CI
  • On a cluster upgraded from 25.10 without --spdk-proxy-image: a fence and unfence round-trips through nvmf_port_block / nvmf_port_unblock

Systemic causes

  • Which fencing mechanism to use was decided by an image-tag string. It came from spdk_version, derived from the proxy image tag at storage_node_ops.py, rather than from what the SPDK on the node supports. An upgraded record silently picked the dead path.
  • Nothing tested the production RPC path. Before this PR no test referenced nvmf_port_block: every fixture node had spdk_version == "", so the integration suites ran fencing through the patched FirewallClient only.
  • Left over, not removed here: spdk_version is now unused, and so are the firewall_port field and FW_PORT env, plus the jc dependency. They can go in a follow-up.

Backported from #1497 (main)

  • e46e0875e, remove the firewall-port plumbing: firewall_port / FW_PORT, get_next_fw_port, the deprecated /snode/firewall_set_port and /snode/get_firewall routes, spdk_version, and jc (removed from requirements.txt; R26.3 has no uv.lock).
  • 79af1723d, remove utils.port_block: callers use node.rpc_client(timeout=…, retry=…).nvmf_port_block/unblock(port) directly, RPCClient.nvmf_get_blocked_ports returns the set of ports, and tests use tests._mocks.route_fences. This is the same change as on main, applied by hand because the branches have diverged: on R26.3 the client calls _request3, and four of main's test files don't exist.
  • 4cd415bae: ClassVar fix for RUF012. The Linter installs an unpinned ruff, and 0.17.1 flags ttl_cache.py.

Local check against the branch before the backport (e46e0875e~1):

  • Unit suite: 2764 passed in both.
  • The integration files this touches, run on mocks with the unit-tier FDB stub: same results before and after. ftt2/test_hublvol_mock_rpc.py is flaky only when two runs share its fixed ports; run alone, it passes 52/52 on both.

🤖 Generated with Claude Code

wmousa and others added 2 commits October 16, 2026 17:40
…so the entrypoint stops flushing iptables (#1484)"

This reverts commit 88a50e2. The SPDK image no longer touches iptables
at all (ultra #218): 25.10.x -> 26.3 upgrades are offline, so no 25.10
fence rules need cleaning at SPDK start, and nothing reads FENCE_PORTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ient fallback

port_block.set_port sent fences to the legacy firewall webapp
(FirewallClient, mgmt_ip:firewall_port) whenever node.spdk_version was
empty, and redirected a method-not-found from nvmf_port_block there too.
25.10.x node records never carried spdk_version, and it is only set by a
restart that passes --spdk-proxy-image, so a node upgraded to 26.3 kept
"" and every fence on it went to a port nothing listens on: the SPDK
image no longer ships the webapp or iptables (ultra #218). The fence
failed with connection refused.

- port_block: set_port / get_blocked_ports_set / is_port_blocked use the
  SPDK RPCs only; an RPC error, method-not-found included, propagates.
- health_controller.check_ports_on_node: drop the per-port legacy branch.
- Delete fw_api_client.py; port_block was its only user.
- Tests: the ftt2 and dual-FT fixtures patch port_block instead of
  FirewallClient; the expansion simulator answers nvmf_port_block /
  nvmf_port_unblock / nvmf_get_blocked_ports instead of stubbing
  FirewallClient; regression tests in test_port_block_rpc_only.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the target/R26.3 Targets release branch R26.3 label Oct 16, 2026
@Hamdy-khader
Hamdy-khader requested a review from mxsrc October 16, 2026 15:55

@Hamdy-khader Hamdy-khader left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@mxsrc mxsrc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this alot, always in for removing code :D
At this point we can simply remove the port_block module altogether I feel:

  • set_port just calls either block or unblock, there's no call sites that don't statically define the block argument so it's a direct call at each site.
  • get_blocked_ports_set is the only call-site of RPCClient.nvmf_get_blocked_ports so the projection can just be done in that helper
  • is_port_blocked is a simple predicate of that and merits no additional helper

@wmousa

wmousa commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Agreed, with the fallback gone port_block is a thin layer and can go. I'd rather keep #1496 minimal since it's on R26.3 and gets cherry-picked, and do the removal on main as a follow-up. That means inlining the 13 set_port sites as direct nvmf_port_block / nvmf_port_unblock calls, moving the blocked-ports set into RPCClient.nvmf_get_blocked_ports, and dropping is_port_blocked. Most of the work is the ~40 test files that mock fencing through port_block.set_port, so it's better as its own PR.

The Linter job installs an unpinned ruff; 0.17.1 flags the class-level
WeakSet registry as a mutable class attribute. It is meant to be shared
across instances, so ClassVar states that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wmousa added a commit that referenced this pull request Oct 16, 2026
With the FirewallClient fallback gone, port_block was a pass-through
(review on #1496):

- set_port only chose nvmf_port_block or nvmf_port_unblock, and every
  caller passed block= as a constant: the 13 call sites now call
  node.rpc_client(timeout=.., retry=..).nvmf_port_block/unblock(port)
  with the budget each already used. The rationale for the short fence
  budget moves to RPCClient.nvmf_port_block.
- get_blocked_ports_set was the only caller of
  RPCClient.nvmf_get_blocked_ports: the projection to a set of port
  numbers moves into that method.
- is_port_blocked was a one-line membership test; check_port_on_node
  does it inline.

Tests: they faked fencing by patching port_block.set_port, which also
told them which node a fence was for. tests._mocks.route_fences keeps
that: it wraps each node's rpc_client so nvmf_port_block/unblock reach
the same (node, port, block) callback the tests already used. Source
text anchors and "must not fence" assertions now look for the RPC
calls, and mocks of nvmf_get_blocked_ports return a set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wmousa added a commit that referenced this pull request Oct 17, 2026
…rt plumbing (#1497)

* fix(port-block): fence only through the SPDK RPC, drop the FirewallClient fallback

port_block.set_port sent fences to the legacy firewall webapp
(FirewallClient, mgmt_ip:firewall_port) whenever node.spdk_version was
empty, and redirected a method-not-found from nvmf_port_block there too.
25.10.x node records never carried spdk_version, and it is only set by a
restart that passes --spdk-proxy-image, so a node upgraded to 26.3 kept
"" and every fence on it went to a port nothing listens on: the SPDK
image no longer ships the webapp or iptables (ultra #218). The fence
failed with connection refused.

- port_block: set_port / get_blocked_ports_set / is_port_blocked use the
  SPDK RPCs only; an RPC error, method-not-found included, propagates.
- health_controller.check_ports_on_node: drop the per-port legacy branch.
- Delete fw_api_client.py; port_block was its only user.
- Tests: the ftt2 and dual-FT fixtures patch port_block instead of
  FirewallClient; the expansion simulator answers nvmf_port_block /
  nvmf_port_unblock / nvmf_get_blocked_ports instead of stubbing
  FirewallClient; regression tests in test_port_block_rpc_only.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: remove the firewall-port plumbing and the jc dependency

Nothing has read any of this since fencing moved to the SPDK
nvmf_port_block RPC and the SPDK image dropped iptables and the firewall
webapp (ultra #218):

- StorageNode.firewall_port, the spdk_process_start(firewall_port=) param,
  the field in both node-API SPDKParams, the FW_PORT env on the SPDK
  container (docker and the k8s template) and constants.FW_PORT_START.
- utils.get_next_fw_port: no callers.
- The deprecated node-API routes /snode/firewall_set_port and
  /snode/get_firewall, which only returned False.
- StorageNode.spdk_version: only the removed FirewallClient fallback
  read it.
- The jc dependency (it parsed iptables output); uv.lock regenerated
  with uv 0.12.22, the version docker/Dockerfile pins.

Existing FDB records keep their stored firewall_port / spdk_version keys;
BaseModel.from_dict only reads declared attributes, so they are ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: remove utils.port_block, fence through the RPC client directly

With the FirewallClient fallback gone, port_block was a pass-through
(review on #1496):

- set_port only chose nvmf_port_block or nvmf_port_unblock, and every
  caller passed block= as a constant: the 13 call sites now call
  node.rpc_client(timeout=.., retry=..).nvmf_port_block/unblock(port)
  with the budget each already used. The rationale for the short fence
  budget moves to RPCClient.nvmf_port_block.
- get_blocked_ports_set was the only caller of
  RPCClient.nvmf_get_blocked_ports: the projection to a set of port
  numbers moves into that method.
- is_port_blocked was a one-line membership test; check_port_on_node
  does it inline.

Tests: they faked fencing by patching port_block.set_port, which also
told them which node a fence was for. tests._mocks.route_fences keeps
that: it wraps each node's rpc_client so nvmf_port_block/unblock reach
the same (node, port, block) callback the tests already used. Source
text anchors and "must not fence" assertions now look for the RPC
calls, and mocks of nvmf_get_blocked_ports return a set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wmousa and others added 2 commits October 17, 2026 08:59
Nothing has read any of this since fencing moved to the SPDK
nvmf_port_block RPC and the SPDK image dropped iptables and the firewall
webapp (ultra #218):

- StorageNode.firewall_port, the spdk_process_start(firewall_port=) param,
  the field in both node-API SPDKParams, the FW_PORT env on the SPDK
  container (docker and the k8s template) and constants.FW_PORT_START.
- utils.get_next_fw_port: no callers.
- The deprecated node-API routes /snode/firewall_set_port and
  /snode/get_firewall, which only returned False.
- StorageNode.spdk_version: only the removed FirewallClient fallback
  read it.
- The jc dependency (it parsed iptables output), from requirements.txt.

Existing FDB records keep their stored firewall_port / spdk_version keys;
BaseModel.from_dict only reads declared attributes, so they are ignored.

(cherry picked from commit d3459b9, part of #1497)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the FirewallClient fallback gone, port_block was a pass-through
(review on #1496):

- set_port only chose nvmf_port_block or nvmf_port_unblock, and every
  caller passed block= as a constant: the 13 call sites now call
  node.rpc_client(timeout=.., retry=..).nvmf_port_block/unblock(port)
  with the budget each already used. The rationale for the short fence
  budget moves to RPCClient.nvmf_port_block.
- get_blocked_ports_set was the only caller of
  RPCClient.nvmf_get_blocked_ports: the projection to a set of port
  numbers moves into that method.
- is_port_blocked was a one-line membership test; check_port_on_node
  does it inline.

Tests: they faked fencing by patching port_block.set_port, which also
told them which node a fence was for. tests._mocks.route_fences keeps
that: it wraps each node's rpc_client so nvmf_port_block/unblock reach
the same (node, port, block) callback the tests already used. Source
text anchors and "must not fence" assertions now look for the RPC
calls, and mocks of nvmf_get_blocked_ports return a set.

R26.3 port of 59a238b (part of #1497): the same edits replayed on
R26.3, where RPCClient calls _request3 and the two-node and
failback-listener tests do not exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wmousa
wmousa merged commit cb5072e into R26.3 Oct 17, 2026
8 checks passed
@wmousa
wmousa deleted the fix/r263-remove-firewall-fallback branch October 17, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

target/R26.3 Targets release branch R26.3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants