fix(lvol): confirm crypto bdev absence instead of raising on -ENODEV

[Hamdy-khader]

Summary

• Fixes SFAM-2826: an encrypted volume's delete never converged because _remove_bdev_stack's crypto branch had no handling for the bdev (or its key) already being removed on a prior teardown pass.
• lvol_crypto_delete/lvol_crypto_key_delete raise RPCRemoteError on SPDK's -ENODEV answer; left uncaught, that exception escaped _remove_bdev_stack entirely — skipping the all_removed=False path and the caller's RuntimeError — so lvol_monitor retried the same failing delete indefinitely (92 attempts / 57 minutes in the reported run).
• Mirrors the existing bdev_lvol/bdev_lvol_clone handling: probe with bdev_get before attempting the delete, and treat -ENODEV from either the bdev delete or the key delete as confirmation the resource is already gone, marking the stack entry deleted instead of raising.

Systemic causes (not fixed here, left for follow-up)

• The per-entry "already deleted" skip at the top of the loop (lvol_controller.py:2156) is commented out, so every retry re-attempts RPCs against entries already stamped deleted — a narrower, probe-confirmed version of that skip (as suggested in the ticket) would help all branches, not just crypto.
• No shared helper exists for the "catch RPCRemoteError, treat -ENODEV as confirmed absence" idiom that 4 of 6 branches in this function already need by hand — a future new branch could reintroduce this bug class.
• The crypto branch had zero test coverage in tests/unit/test_lvol_teardown_confirmation.py despite every sibling branch being covered — that gap is why this shipped unnoticed.

Test plan

• tox run -e lint,types on the changed files — clean
• tox run -e unit -- tests/unit/test_lvol_teardown_confirmation.py — new TestRemoveBdevStackCrypto suite (6 tests) covers: genuine failure stays unconfirmed, normal successful delete, absent-bdev probe skip, -ENODEV from the bdev delete, -ENODEV from the key delete, and sync=True still bypassing the crypto branch
• tox run -e unit (full suite) — 4700 passed; the one pre-existing failure (test_metrics_are_served, missing process_open_fds on macOS/no-/proc) reproduces identically on main and is unrelated to this change

🤖 Generated with Claude Code

[mxsrc]

With some of the other RPC functions I moved the error check into the helper, adding an idempotent check, e.g.:

def some_rpc_deleter(self, name, *, idempotent=False):
    try:
        self._request("delete_resource", name=name)
    except RPCRemoteError as e:
        if idempotent and e.code == -errno.ENODEV:
            return
        raise

With that the code at the caller-site needs to do fewer distinctions, in this case the whole block could just be:

rpc_client.lvol_crypto_delete(name, idempotent=True)
rpc_client.lvol_crypto_key_delete(f"key_{name}", idempotent=True)
bdev["status"] = "deleted"

Just checked: bdev_crypto_delete yields -ENODEV on missing entity, lvol_crypto_key_delete gets RPCErrorCode.invalid_params with a message "No key object found".

[Hamdy-khader]

Good call — pushed up 46211b2 addressing this. Moved the idempotent-absorption into RPCClient.lvol_crypto_delete/lvol_crypto_key_delete via an idempotent= kwarg (same shape as nvmf_subsystem_add_ns's pattern), so _remove_bdev_stack's crypto branch is now just:

try:
    rpc_client.lvol_crypto_delete(name, idempotent=True)
    rpc_client.lvol_crypto_key_delete(f'key_{name}', idempotent=True)
    ret = True
except RPCRemoteError:
    ret = None

One adjustment from your snippet: since the two calls are independently idempotent now rather than one gating the other, the key delete runs even when the bdev delete hit the absorbed -ENODEV path — previously it would've been skipped. That's intentional (the two registries have independent absent-signatures, as you found), but flagging the behavior change explicitly.

Also confirmed and encoded your finding as a comment + test: accel_crypto_key_destroy rejects a missing key as invalid_params with message "No key object found", not -ENODEV, so lvol_crypto_key_delete's idempotent path matches on that message (narrowed so it doesn't swallow some other unrelated invalid_params failure).

Added direct RPCClient-level tests for both methods in tests/unit/rpc/test_client.py and updated the _remove_bdev_stack crypto tests accordingly.

[mxsrc]

Looks great, thanks!