Repository navigation
Product-limits stress, dual-outage matrix, resume, and lblk integrity… - #1357
Open
RaunakJalan wants to merge 292 commits into
Open
RaunakJalan wants to merge 292 commits into
RaunakJalan wants to merge 292 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
You asked why there was no other way, and there was. I said device hot-remove and network isolation had no kubectl equivalent. Both were wrong, and the suite already does each: _network_outage_dual in e2e_tests/security/test_lvol_security.py, itself ported from stress_test/continuous_k8s_native_failover.py, applies iptables DROP from inside the privileged hostNetwork SPDK pod and schedules the flush as a HOST process via `nsenter --target 1`, so it survives SPDK's abort timer killing the container. Without that the rules would be permanent. The same nsenter lands in the host's mount namespace, which is all /sys/block/<dev>/device/delete needs. So _host_cmd and _network_outage are now dual, and all ten lblk classes run on their platform. No skips left in this file. And on restart: you are right that the container is always recreated, so waiting for a pause that cannot happen was the wrong shape. When the control plane replaces the process the test now restarts the node and CONTINUES, rather than abandoning the run. What it can still prove is that a partitioned node which gets restarted leaves no corruption behind -- data verifies, the ring is consistent, no blobstore-CRC errors -- and that is the path a real cluster actually takes. It records that the stale-append window did not occur and says why, instead of reporting a clean pass as if the fence had been tested. The unfenced-append window itself still needs auto_restart_disabled held for the duration, which nothing exposes; storage_node_monitor queues a restart as soon as status reaches OFFLINE and polls every 3s, so it cannot be raced. That is a finding for the dev team rather than something to keep tuning.
Demoting md5 to a warning everywhere was wrong, and it would have hidden the exact corruption these tests exist to catch. The demotion has a real justification, but only on some hardware. SshUtils.run_fio_test hardcodes --verify=md5 and enables verify_backlog, which its own comment says bypasses the rand_seed check -- so on a device with no 4K atomic-write guarantee a mismatch genuinely can be an artefact of the harness. On a device that DOES guarantee 4K atomicity there is no such excuse. GCP is the case that needs it: pd-balanced disks are the non-atomic hardware this whole mode exists for. The lab's Samsung PM983s are 4K-atomic, as is everything else we run on, so md5 stays fatal there. CLUSTER_ENV drives it and had to be plumbed: the workflows computed it for their own shell steps but never exported it to the test, so the test could not have made this distinction even in principle. Added to the runner step of the five k8s workflows that have one. Docker has no such input, so it defaults to error -- which is correct for the bare-metal lab. Defaults to error on an unset or unknown value, so a new environment is strict until someone deliberately says otherwise.
Two things, and the first corrects a claim I made twice. Raw block on k8s is not a platform limitation, it was a gap in our helpers. spdkcsi supports it -- csi-driver/internal/csi/node/publish.go branches on GetVolumeCapability().GetBlock() and expand.go handles block expansion. What was missing: create_pvc had no volumeMode, and every pod builder uses volumeMounts, which is filesystem-only. A Block PVC needs volumeDevices. So create_pvc takes volume_mode, create_raw_device_pod attaches a Block PVC to a long-lived fio pod through volumeDevices, and _PodDeviceRunner gives RawDeviceVerifier the two methods it calls -- exec_command and is_block_device -- backed by kubectl instead of ssh. The verifier is unchanged, so both platforms run byte-identical FIO jobs rather than two implementations that drift. Long-lived pod rather than a Job because stamp, churn and verify are separate commands against the same device. This matters most on GCP, where md5 is now only a warning: without a raw lane that cluster would have had no strong data gate at all. Second, both platforms now run a MIX, because the two lanes fail differently. Raw is fio straight at the device with crc32c and nothing in the way, so a torn write cannot be absorbed -- that is the gate. Formatted is fio through a filesystem, which is what an application actually does; weaker as a detector, but it exercises the customer path and carries the snapshot/clone coverage. Running only one would either miss the realistic path or trust a check that a journalling filesystem can mask. Verified the rendered manifests: a Block PVC gets volumeMode: Block, a normal one is left at the cluster default, and the pod maps the device through volumeDevices with no volumeMounts. Also widened exec_in_pod with a timeout -- it defaulted to 300s, which would cut a raw verify off mid-run and read as a storage failure rather than a harness one.
Checked what the formatted lane was doing before claiming it was a weaker detector, and on k8s it was not a detector at all: _run_fio_dual's job config carries ioengine, direct, bs, iodepth, numjobs, rw, size, nrfiles -- and no verify of any kind. It moved IO and checked nothing. Only the docker path verified, through the --verify=md5 hardcoded inside run_fio_test, so the two platforms were never checking the same thing and the md5 severity work applied to one of them. _run_fio_dual now takes verify= and verify_fatal=, both off by default so every existing caller is untouched. The lblk formatted lane passes verify="md5". verify_fatal follows the hardware, which is the rule you set. Where a 4K atomic write is guaranteed a mismatch is a real defect and fails the job outright; on GCP's pd-balanced disks it can be an artefact of run_fio_test's own verify_backlog, so it is recorded and triaged rather than failing the run. Same decision as _md5_severity, derived from CLUSTER_ENV in both places. Rendered all three configs to confirm: existing callers get no verify line, atomic hardware gets verify=md5 plus verify_fatal=1, GCP gets verify=md5 without it. So the two lanes now genuinely differ in what they can prove, which was the point of running both: raw crc32c is unconditional and nothing can absorb a torn write, while md5-through-a-filesystem is trusted only where the hardware earns it.
It called _create_and_connect and handed the result to the raw verifier. That was fine while RAW_VERIFY was docker-only, but k8s now runs the raw lane too -- and there _create_and_connect makes a FILESYSTEM PVC and returns its name, not a block device. LblkFunctionalK8s would have tried to stamp a PVC name. Uses _provision_raw, which asks for volumeMode: Block on k8s and an NVMe-oF device on docker, so both platforms hold a real device. Also adds the formatted volume to the smoke test, so a pass means both shapes of volume work rather than only the one the gate uses. A bring-up check that covers the raw path but not the path customers actually take is not a bring-up check.
Audited every ssh_obj call on the lblk path. All of them are either behind a k8s branch, inside a docker-only mixin, or pure bookkeeping (notify_outage_started only resets an in-memory clock). One was not: _init_lblk built a RawDeviceVerifier over ssh_obj on both platforms. On k8s it was inert -- _provision_raw replaces it with the pod-backed runner, and _lblk_devices is empty until then, so nothing called it. But an ssh-backed verifier sitting on a k8s object is an invitation for the next change to use it and hang until the ssh timeout, which is exactly how this path produced Exception: All usernames failed for 10.0.0.10. Last error: timed out None on k8s makes the mistake impossible rather than merely unlikely.
(Line: 1556, Col: 9): 'env' is already defined Wiring CLUSTER_ENV in, I added an `env:` block to steps that already had one. GitHub rejects a duplicate key outright, so all five k8s workflows failed to parse -- including the NVMe ones, which had nothing to do with lblk. CLUSTER_ENV now goes into each step's existing env block instead. My validation missed it because PyYAML ALLOWS duplicate keys -- last one wins -- so yaml.safe_load parsed the broken files happily. Re-checked with a loader that rejects duplicates, which is what GitHub does, and all 34 parseable workflows are clean. k8s-e2e-ha.yaml still fails, as it did at HEAD before any of this. k8s-native-upgrade.yaml is reverted: my pass had added CLUSTER_ENV there too, and while it landed in a different step and was not a duplicate, upgrade is meant to stay untouched.
…detection All four k8s failures, one cause. Every k8s test in the suite calls _k8s_ensure_storage_class() right after creating the pool -- 54 of them do -- and the lblk tests did not. _add_pool_dual does not create a StorageClass, so simplyblock-csi-sc never existed and every PVC sat Pending: TimeoutError: [K8sUtils] PVC 'lblkfnraw2545' not Bound within 300s It hit the plain filesystem PVCs too (lblkjr901, lblkfence718), so it was never about volumeMode: Block. Now in _make_pool, used by all five classes, because a missing StorageClass surfaces as a five-minute timeout that never mentions one. LblkIntegrityDocker: "md journal append failed" is NOT corruption, and I had it in JOURNAL_FATAL_MARKERS. From spdk_4424's log on 192.168.10.203, the whole burst inside 700 microseconds: alg_io_split_parity2.cpp:922 DISTRIBD stopped write IO alg_journal.cpp:3420 received a change_leadership event for jm_vuid=1 blob_md_journal.c:511 md journal home write failed: -5 (retry) blob_md_journal.c:355 md journal append failed: -5 lvol.c:2942 Cannot update lvolstore on failover blobstore.c:10903 Updating failed and unfreeze IOs on failover That is the container_stop outage we injected. The distrib layer blocks IO while leadership moves (b_block_new_io=1), so the append gets EIO by design, and append_write_cpl handles it: rolls mem_head back, decrements used_slots, completes the op with the error, pumps the queue. The slot is un-allocated, so nothing is half-written -- the journal refusing to acknowledge an entry it could not persist is the opposite of corruption. Moved to JOURNAL_TRANSIENT_MARKERS, reported and never fatal. Recovery failures stay fatal. LblkUnfencedJournalDocker: "never demoted ... Ring head moved 4351 -> 0" was my detection being wrong, not the isolation. A head of 0 after 4351 IS the proof -- mem_head only advances within a process's lifetime, so a lower value means a fresh SPDK. I only checked the container state, and by the time we look the control plane has recreated it, so it reads "running" and the restart went unnoticed. Now either signal counts, and the head check works on k8s too where there is no container to inspect.
The previous commit read `after` in the restart check, but `after` is not computed until further down -- F821, and it would have raised NameError on every run of this test. My fault for committing while ruff was reporting it. The ring is now sampled at the decision point. Verified the logic against the numbers from the actual run: head 4351 -> 0 with the container back to "running" is correctly read as a restart, a head that advanced is not, a missing container is, and an unreadable ring falls back to container state rather than guessing.
Three defects, one chain, found by RCA of the 2026-09-18 docker run.
The verify context reaches a tmux session name, an fio job name and a log
path. Contexts contain spaces ("after snapshot+clone"), so the shell split
"fio_lblkafter snapshot+clone-lblkfs0132" and fio never started. Slugify at
the single chokepoint in _fs_fio, so no future context can reintroduce it.
validate_fio_test then read the log that was never written, matched no
markers in zero lines, and returned clean. An FIO log always carries at
least the job header, so empty means the job did not run: raise instead of
passing. This is the same silent-pass class as the empty-string log scan.
Finally, sweep for core dumps at each outage boundary. e2e.py's check only
detects a core and stops the run, so an lblk crash left a bare .zst with no
backtrace. Swept cluster-wide rather than narrowed to the outaged node: we
shut down .201 and .203 was the node that aborted.
_run_fio_dual returns a Thread on docker and a job-name string on k8s. hasattr(handle, "join") is true for both, because str.join exists, so every k8s run called "jobname".join() and died with TypeError: str.join() takes exactly one argument. That is what failed LblkFunctionalK8s and LblkIntegrityK8s. Test for threading.Thread instead. LblkUnfencedJournalK8s then failed reading journal stats from a node that sbcli reported online and healthy but kubectl could not reach yet: the transport is kubectl exec, which dials the kubelet on 10250, and after a partition that listener can still time out. Retry the post-recovery read for up to 240s rather than failing on the first attempt. Retrying, not tolerating: the caller needs those counters for its assertion, so a read that never succeeds still fails the test.
gl_fetch_container_logs asked for `container_name:*name*`. Graylog on the mgmt node ships `allow_leading_wildcard_searches = false`, so the server refuses any query starting with '*' and returns HTTP 500 "Missing search type result!". Proven against 192.168.10.210: `container_name:*` succeeds, `container_name:*app_BackupService*` 500s, the exact and prefix forms both return 3480 hits for app_CapacityMonitor. The probe treated that failure as "no messages", wrote an empty file and returned 0, so every container produced an empty log and the only trace was a WARN on stderr. Ask exact first, then prefix, never leading with '*', and say so explicitly when a container really has nothing in the window -- app_BackupService legitimately has 0 messages and should not look the same as a query the server rejected. The OpenSearch path is unaffected: it queries OpenSearch directly, where allow_leading_wildcard defaults to true.
The fallback only fires when TOTAL_CPUS comes back 0 or unreadable, which is exactly when the node is least likely to be small. 4 was low enough to start the cluster underprovisioned and hide the real problem behind slow runs. All 8 sites across the 7 k8s-native workflows, so the fallback is the same number wherever it fires.
Two failures from the 20260918-1451 k8s run. LblkUnfencedJournalK8s left 10.0.0.10 cut off for the rest of the run. The iptables restore was scheduled with `nsenter --target 1 --mount --net -- nohup sleep N && iptables -F`. Without --pid that process stays in the CONTAINER's pid namespace, so SPDK's abort timer killed the container mid-sleep and took the restore with it, making the DROP permanent -- the comment claimed it was a host process, and it was not. Hand the restore to the host's systemd as a transient timer instead, which pid 1 owns and the container cannot take down, with a --pid'd detached shell as fallback. Delete our two rules rather than `iptables -F`, which would also flush kube-proxy's jumps out of INPUT/OUTPUT. LblkIntegrityK8s could not bind clone PVCs: "requested volume size 20000000000 is less than the size 21474836480 for the source snapshot". 20G is 20*10^9 to Kubernetes, but the source snapshot came from a PVC sized in Gi. _create_lvol_dual already normalises G->Gi; _create_clone_dual did not, so the two disagreed. Apply the same normalisation, and record the size actually requested. A no-op for callers already passing Gi.
worker-0 (10.0.0.10) has been going NotReady on every k8s lblk run. Cause confirmed: the node went NotReady at 15:42:38Z on 2026-09-18, inside lblk_unfenced_journal_k8s-20260918-153755, with "Kubelet stopped posting node status" -- the signature of a node that is up but cannot reach anything. It is the same node every time because the unfenced test isolates _journal_lvs, which assert_journals_live sets to the FIRST journal target in API order. That is worker-0 on this cluster. Two defects in _network_outage, not one. The restore ran as `nsenter --target 1 --mount --net -- nohup sleep N && iptables -F`: without --pid it stayed in the container's pid namespace, and this test exists precisely because the outage makes SPDK abort and the container be replaced, so the restore died mid-sleep and the DROP became permanent. Separately, apply and restore were two commands, so a restore that failed to schedule still left the apply to run. Both are fixed by putting apply, hold and release in a single transient systemd unit owned by pid 1: the thing that cuts the node off is the thing that restores it, and if the unit cannot be created the node is never cut off at all. Fallback is a setsid'd --pid host shell running the same script. Deleting our own two rules instead of `iptables -F` also stops us flushing kube-proxy's jumps out of INPUT/OUTPUT. Recovering a node already stranded this way needs an out-of-band reboot; iptables is not persisted on RHCOS, so a reboot clears it.
…outage LblkIntegrityK8s got past the GiB fix and hit the next wall: "requested volume modifies the mode of the source volume but does not have permission to do so. allow-volume-mode-change annotation is not present". The clone is formatted and mounted, so on k8s it is a Filesystem PVC, and the raw lvols it was snapshotting are Block PVCs -- Kubernetes will not clone a Block snapshot into a Filesystem PVC, so that clone could never bind. Snapshot a formatted volume instead, keeping the caller's alternation so both lvstores still see metadata work. Docker has no volumeMode and is untouched. LblkUnfencedJournalK8s reported INCONCLUSIVE, correctly. Two reasons, both addressed. outage.join() was nearly a no-op on k8s, because _network_outage only schedules the host unit and returns -- so the test logged "links restored" while the node was still cut off and never actually waited out the window. And 90s was measurably too short: no node logged writer_conflict, lock conflict or non_leader anywhere in that run, and the ring head advanced 17 -> 23 on the original process, so the peers never noticed it was gone. Wait out the window for real, poll the control plane while waiting and record when it first marks the node offline, and raise the window to 180s. Nothing restarted the node at 90s, so there is room. The recorded offline time means the next adjustment can come from a measurement rather than a guess, and the guard still refuses to claim a pass it has not earned.
Last run showed the isolation never happened at all: no stats read ever failed, all six lvol creates succeeded, and the cluster never marked the node offline. The test was reporting INCONCLUSIVE about a partition it had not created. Three things were wrong, each invisible. nsenter --target 1 from the SPDK pod resolves /proc/1 of the CONTAINER, not the host, because the pod is hostNetwork but not hostPID. The host unit I added last round could never be created -- "System has not been booted with systemd as init system (PID 1). Can't operate." -- so nothing was applied. Verified directly against snode-spdk-pod-4420-84006c. exec_in_spdk_container never raises on a non-zero exit, it returns (stdout, stderr). So the failure was silent and the try/except around it could not fire. Check the result instead of assuming it. And a blanket INPUT/OUTPUT DROP is unfixable by design here: it blocks the kubelet too, so once the in-container timer dies with the container there is no route left to undo it. That is what needed an out-of-band reboot. Block only the peer storage nodes. The peers stop hearing from this node -- which is what the test actually wants -- while the kubelet, the API server and kubectl exec stay reachable, so the restore can always be driven from outside. Verify with iptables -S that the rules landed and undo them and fail loudly if they did not; restore explicitly after the window rather than trusting the in-node backstop; and pre-clean before isolating so a leftover cannot make the next run silently isolate nothing.
…sult The partition is now real -- the control plane marked the node 'unreachable' 39s in, the node appended nothing for the full 180s (ring head 2174 -> 2174), and the restore was clean. But drain_demoted is still False, and the reason is not the timeout: NO node in the cluster logged writer_conflict, non_leader or a jm lock conflict at any point in the run. A peer going quiet does not move lvstore leadership here. It moves when a node leaves or rejoins deliberately -- the 2026-09-18 docker RCA has a graceful shutdown producing a writer-lock conflict in 31s. So the stale-writer window cannot be manufactured by partitioning a peer, and no amount of extra isolation time will change that. Two changes. Run the log scan and integrity verification FIRST and unconditionally: they used to sit after the demotion check, so an inconclusive run discarded the one result it had actually earned -- that a node cut off and brought back leaves no corruption. And raise SkippedTestsException rather than LblkPreconditionError when leadership never moved, so it reports as SKIPPED with the evidence instead of as a failure that buries real ones. Reproducing the fence needs leadership to actually move; the skip message says so and names the two ways in.
The k8s run of 2026-09-19 18:12 reported "Passed 3/4, Failed 0, Skipped 2" for four cases. A case that raises SkippedTestsException incremented skipped_cases in the handler, and was then counted again by the closing subtraction, which attributes everything neither passed nor failed to skips. Let the subtraction own in-run skips; the handler counter now carries only the cases dropped before the run, which never enter test_class_run at all and so cannot be derived that way. Verified against the three real shapes: 3 pass + 1 self-skip, 1 pass + 1 fail + 2 never-ran, and a pre-run skip.
The _network_outage docstring still described the nsenter host-timer design that was removed when it turned out the SPDK pod has no hostPID; there was a leftover note about nohup and ssh sessions that applies to neither platform now; and the ISOLATION_SEC rationale argued 90s was too short, which was written before I found the isolation had not been happening at all. Length is not the binding constraint -- leadership does not move on a peer partition -- and the comments now say so.
The unfenced-journal case was a reproducer for an SPDK-internal gap and it could never set its own scenario up: leadership does not move because a peer goes quiet, on either platform, so it had never once reached its assertion. Removed, along with the two _spdk_freeze implementations that existed only to serve it. In its place, the question a POC actually has to answer: take a node out, one at a time, every way a node can go, and does IO carry on and does the data still match. Two lanes, because they fail differently. Static volumes are written once before any fault and never touched again, so md5 on them has no overlapping IO and no verify_backlog to produce a false mismatch -- that makes it a stronger durability check than anything else in the lblk suite, and it is re-checked after every outage so a divergence names the outage that caused it. Live FIO runs across the whole sequence as the availability gate: with ndcs/npcs 2/2 one node down is survivable by design, so any io_u error is a defect. Sudden loss is the normal case in the field, so a crash-type outage is held to exactly the same bar as a planned one. The volume set is mixed on purpose: plain, encrypted and DHCHAP volumes, formatted and raw, plus a clone from a snapshot. Encryption and DHCHAP on lblk have never been exercised; a failure confined to those lanes is a finding about lblk, not about this test, and the error names the flavour. interface_full_network_interrupt is now a first-class outage type in _outage_and_recover rather than something only one test could reach.
The live lane was plain-only for no reason other than how I wrote it. That left the most likely failure untested: the crypto layer sits in the IO path, so an encrypted volume carrying load while a node disappears is exactly where this is most likely to come apart, and a plain-only lane never touches it. One live volume per flavour now, matching the static set. Fixing that surfaced a worse bug in the same file. _provision_typed registered the static volumes in _fs_volumes, and _verify_all runs a write workload over everything in there -- so the loop would have written to the very volumes whose md5 is only trustworthy because nothing writes to them. Static volumes stay out of _fs_volumes, and the loop now verifies the raw device only: the static md5 check already covers the formatted volumes, and covers them more strictly than a fresh FIO would.
The loop applied one outage per node, so on a six-node cluster four nodes saw one outage each and two saw none. Now it is the full cross product: every type against every machine, 24 cycles on six nodes. Ordered by outage rather than by node, so consecutive cycles land on different machines. Taking one node down four times in a row tests recovery from a warm cache more than it tests the cluster, and leaves the others untouched for a quarter of the run. Two things had to change to make that safe. Live FIO was pinned to a fixed 70-minute runtime, which a four-hour sequence would outlive -- the job would have ended part way through and the availability lane would have stopped watching without saying so. It is sized from the cycle count now. And the md5 check created and deleted a utility pod per volume per call; across 24 cycles that is over a hundred pod lifecycles, more wall-clock than the outages. One pod per static volume now, kept for the run and removed by the standard teardown. The hash path also refuses to return a non-digest. An md5sum that prints nothing would otherwise compare equal to the next empty result and report the data as unchanged, which is precisely what this lane exists to catch. LBLK_MATRIX_NODES trims the node count for a shorter slot. It never drops an outage type -- every type still runs, against fewer machines -- and the log says plainly that node coverage was reduced.
Full coverage is 24 cycles on a six-node cluster, about four hours. That does not belong in an e2e slot that otherwise finishes in an hour, so it moves to stress_test/ with the other long-running families and out of get_lblk_tests. Not a duplicate of DualOutageMatrix, which is the opposite shape: that one takes several nodes down at once to explore lvstore chain separations. This one takes exactly one node down at a time and asks whether a mixed set of volumes -- plain, encrypted, DHCHAP, formatted, raw, cloned -- keeps serving and keeps its bytes through every outage type on every node.
The static lane wrapped provisioning failures in an error naming the flavour; the live lane called the same helper bare, so a crypto or DHCHAP volume that would not bind there surfaced as 'PVC not Bound within 300s' with nothing saying which combination was unsupported. The wrapper moves into _provision_typed so both lanes report it.
k8s-native-e2e runs our 12-phase cleanup_upgrade_test.sh first and the operator's cleanup-simplyblock.sh second, so the operator's script sees whatever ours could not resolve. The stress lane had it the other way round AND never ran the 12-phase script at all -- only cleanup_k8s.sh, which does not touch CRDs, cert-manager, node labels, hugepages or CSI hostpath data. Stale CRDs therefore survived into the next run, where the apiserver silently prunes fields it does not recognise from the CRs. That was an oversight when the e2e lane was fixed, not a decision, and it matters now that the lblk outage matrix runs from this pipeline. cleanup_k8s.sh stays as a final sweep: by then the namespace should be empty, and it is the pass tuned for a cluster carrying hundreds of thousands of objects, which is a state only a stress run reaches. The KMS apiservice deletion and vault force-finalize were already here.
Both ran the operator's cleanup-simplyblock.sh first and only cleanup_k8s.sh second, so ours inherited half-removed state and nothing ever cleared CRDs, cert-manager, node labels, hugepages or CSI hostpath data. Stale CRDs then survived into the next run, where the apiserver silently prunes fields it does not recognise from the CRs. Now both run cleanup_upgrade_test.sh first, the operator's script second and cleanup_k8s.sh as a final sweep, matching k8s-native-e2e and -stress. monitoring-suite also gains the KMS half: orphaned aggregated APIServices are deleted before waiting on the vault namespace, and the namespace is force-finalized if it is still Terminating. One dead cluster-scoped APIService blocks every namespace deletion on the cluster, which is what had vault stuck for 25h. upgrade needs no KMS change -- that lane never installs vault -- and its inline Helm and kube-system phases are left alone: they are idempotent, so the 12-phase pass simply finds nothing left to do. All seven k8s lanes now share the order. cross-cluster-restore and upgrade carry no KMS cleanup because neither deploys it.
…umes The run failed with "Pool not visible in sbcli after 300s" on the DHCHAP pool. Cause: I created a second pool with dhchap=True and NO allowedNodes, then built my own StorageClass for it. A DHCHAP pool with no allowedNodes has no hosts to authorise, so the operator never reconciled the CRD. My audit had checked StorageClass and binding mode -- what broke before -- and never checked that a second pool works on k8s at all. DHCHAP on k8s now follows the path the security suite established, and all four parts are required: allowedNodes is a strict subset of the workers, so the operator can derive each allowed node's NQN and label those nodes. The StorageClass is the operator's own, not ours: it carries dhchap_node_label, and the nodeAffinity CSI writes from it is what actually enforces the restriction at mount. The CSI node daemonset is restarted once per pool, because the node plugin only snapshots node labels as topology keys at registration and otherwise every PVC fails on allowedTopologies. And that class is WaitForFirstConsumer, so the PVC is bound by a throwaway pod pinned to an allowed node. Because a DHCHAP PV carries that nodeAffinity, every pod that touches one is pinned too: the seed pod, the md5 pod and the live FIO job. _run_fio_dual gained a node_selector passthrough for the last of those -- it was building the job without one, so an unpinned job could be scheduled somewhere that cannot mount the volume and it would have read as a storage fault. Namespaced volumes are a fourth flavour now, in both the static and live lanes: several namespaces packed into one subsystem rather than a subsystem per lvol. Docker sets it per lvol, k8s through max_namespace_per_subsys on the class.
Four places still reached for the CLI on the Kubernetes arm, which
defeats the point of having one. A volume there is a PVC and a snapshot
is a VolumeSnapshot; anything else is invisible to the operator, is not
what a user has, and is not cleaned up with the claims.
The capacity filler in MIG-N-005 was a `volume add`. It is a PVC now,
through make_volume like every other volume in the lane.
The bulk snapshot chains in MIG-H-007, MIG-B-007 and every MIG-F stage
were `snapshot add`. The excuse was that they exist only to give
snap_copy real work -- never cloned, never mounted, never named in an
assertion -- and that a hundred VolumeSnapshots with a readiness wait
each is slower. True, and beside the point: a snapshot the CSI layer did
not make is not the thing being tested, and it outlives the claim it
belongs to because teardown does not know about it. seed_snapshots()
takes them natively on k8s and through the CLI on docker, and tolerates
a refusal -- a chain of eighteen tests what a chain of twenty does, and
raising there would lose the case over something it is not about.
QoS is commented out rather than deleted. The suite does not support it,
so the result meant nothing either way, and the line is the natural home
for the check if QoS is ever taken on. It is also broken:
volume qos-set <id> --max-rw-iops 10000
TypeError: '<' not supported between instances of 'NoneType' and 'int'
from lvol_controller.py set_lvol, where the validation block compares
max_* < 0 on all four limits with no None check while the assignment
block ten lines below guards each one properly -- so passing fewer than
four limits can never be accepted. Reported separately; nothing in the
lane depends on it.
Still CLI on k8s, deliberately and noted here so it is a decision rather
than an oversight: the read-only verbs (volume list, migrate-list,
snapshot list), which answer identically in the admin pod, and
MIG-N-001..004, which assert the control plane's own refusals. The
Kubernetes-native form of those four would be the PersistentVolumeOps
admission webhook's rejections, which is a different test with different
assertions rather than a port of these.
The previous commit put the uncordon inline at the end of MIG-F-013, which fixes the failure it was written for and nothing else. Anything that raises between the reboot and that line -- the retry assertion, the data verification, a timeout waiting for the node -- leaves the worker cordoned, and a cordoned worker cannot run a storage pod. Every case after it then fails its pre-test health check on a node with nothing wrong with it, which is exactly the cascade that cost run 20261013-143016 three classes. That is the same mistake the StorageClass leak was: cleanup that only runs on the happy path is not cleanup. So the cordon is tracked in a set and undone in teardown(), which the framework calls after every test whatever the outcome. Registered BEFORE the reboot rather than after, because drain cordons first: if reboot_node itself raises, the node is already cordoned and teardown still has to know. Registering afterwards would miss precisely the case that matters. The inline uncordon stays as the fast path and discards from the set when it succeeds, so teardown has nothing to do on a clean run. Found by Raunak reading the previous fix, not by a run.
…ding
Run 20261013-221421 reached 14/21. Two of the five failures were this
lane accusing the product of something it had not done.
MIG-H-008 reported "only 0 of 20 snapshots survived the migration" and
MIG-B-008 "member migtree39226n0 has 0 of 5 snapshots after the group
migrated". All twenty-five were present. Both counted snapshots whose
NAME began with the prefix they had asked for, which works on docker and
cannot work on k8s: there a snapshot is a VolumeSnapshot and the CSI
driver names the backend object, so `snapshot list` comes back holding
"Name": "snapshot-<uuid>", "BDev": "LVS_19/SNAP_69",
"LVol ID": "8f001596-..."
and the logical name appears nowhere. The same root cause as the
snap_id() fix, in two more places that the fix did not reach.
snapshots_of(vol_id) counts by parent volume id instead. That is
backend-neutral, survives a migration -- a volume keeps its uuid and
only changes node -- and is a closer statement of what the case means:
the history moved with the volume, rather than the names did.
MIG-N-010 is the opposite mistake and the more instructive one. It was
matching the product's words for a refusal, and the product neither
refused nor silently succeeded: it narrated itself.
INFO: Removing snapshot: 4c44671f-f559-49e8-8f33-61eb5d22cf86
DEBUG: Requesting method: bdev_lvol_get_lvstores
which is the delete going ahead. Matching words had already failed in
the other direction too -- MIG-N-003 "passed" because "Connection
refused" contains "refus" -- so this now asks the only question the case
is about: is the snapshot still there. Whether the refusal is a
sentence, an error code or silence does not matter.
And on the evidence it is not there, which is a real gap worth reporting
once a run confirms it: the other two guards held on the same volume
seconds earlier -- "Cannot delete lvol ...: active migration" and
"Cannot resize lvol ...: active migration" -- so deletion and resize are
enforced while snapshot deletion is not, though the three are documented
in one sentence.
Still open from this run, unchanged here because they are not ours to
fix: two migrations failed with "the migration failed: target node
offline (status=down)", and in both the rollback failed as well ("taking
the migration back also failed: cancel migration ..."); and one spent
its budget in "step Validating outlived its deadline". The validation
image was not the cause -- no pull errors, no stuck Jobs, and other
migrations on the same cluster succeeded.
Both defaults said one namespace per subsystem, and neither was asked for. create_storage_class carried max_namespace_per_subsys=1, which cluster_test_base inherits for simplyblock-csi-sc -- the class every k8s test uses -- and add_lvol left the field out of the body entirely. Leaving it out is not neutral: the CSI driver sends an explicit 1 for a class that omits it (operator atlas-lib/controlplane/volumes.go:292), so the result is the same either way. That is not a smaller version of the normal configuration, it is a different one. migration_controller.py:1219 picks the migration runner by actual subsystem membership -- more than one member and the request must go through create_batch_migration into tasks_runner_batch_migration.py, otherwise it takes the single-volume runner. A subsystem that never holds two volumes never reaches the batch runner, so every defect that lives there is unreachable by construction. SFAM-2820 is one of them, which is how it can be open against docker and absent from 44 k8s migrations running the same control-plane code. Both now default to 30; the product hard limit is 50. Raising the ceiling does not by itself group anything -- the check above counts real members, and the comment above it says so explicitly -- so a lone volume still migrates on the single path. Grouping still needs namespaced=True on docker or the CSI packing onto one node on k8s. Nothing in the tree asserts an isolated subsystem, and the cases that care about a specific ceiling pass their own.
…twice
A migration that reports success is not the same as a migration that
finished. SFAM-2820 is the gap between the two: the snapshot bdev keeps
the m suffix the cutover is meant to rename away, its record still
names the lvstore on the node the volume has left, and nothing says so
until the next migration resolves the name and fails with "No such
device". Every assertion the lane had would pass on that volume --
including MIG-H-008, which counts the snapshots and finds them all
present, because they are.
So two invariants, asked of SPDK on the node rather than of the
control plane's own status, which makes them the same question and the
same answer on both backends:
assert_no_migration_artefacts no LVS_n/SNAP_n m bdev survives
on either the source or target
assert_snapshots_on_current_lvstore every snapshot record names an
lvstore the volume's node holds
Wired into the cases where migrations are serialised and a leftover is
therefore attributable -- MIG-H-005, MIG-H-006, MIG-H-008 -- and
deliberately not into full_migration, because MIG-S migrates many
volumes at once and another case's in-flight copy would read as this
one's leak.
MIG-B-009 is new and is the part that makes the defect reachable at
all. MIG-B-007/008 already builds what the ticket needs -- a shared
subsystem whose members carry snapshot chains -- and then migrates it
once, which cannot see a leftover: the leftover only collides with the
next migration of the same object. The second hop supplies that, with
the invariants checked after each one so a failure says which hop left
the state rather than only that the second one could not run.
The merge brought in 58553c3, which removed the fire-and-log --backup flag from snapshot add, and 1015c40, which made the backup its own subcommand taking a snapshot id. TC-SEC-150 still used the old single-command spelling. It would not have failed where the mistake is. argparse rejects the unknown flag and the snapshot is never created, so the first thing to notice would have been TC-SEC-151 timing out after five minutes looking through `backup list` for a backup of a snapshot that does not exist, and reporting "Could not find backup ID" -- which reads as a backup that did not complete rather than a snapshot that was never taken. Both asserts now go through cli_failed instead of the "error" not in err.lower() substring test, which is the check that let this stay quiet: argparse writes its complaint to stderr without the word error in the place being looked for. The commented-out TC-SEC-116 block carried the same spelling and is corrected in place rather than left to be revived wrong.
MIG-F-017. Four cancel points existed and none of them asked whether the volume could still be moved afterwards, which is the thing a rollback exists to preserve. Everything the class asserted was about the state on that node at that moment -- terminal, on one node, data intact, no leftovers the bdev scan can see -- and a volume can satisfy all four and still be stuck. Run 20261013-221421 is why it matters. Two migrations failed and both reported that the rollback failed as well: the migration failed: target node offline (status=down); migration failed (taking the migration back also failed: cancel migration 49999924-8200-42f7-8aeb-e0166d16e373: ... the migration failed: delete LVS_31/SNAP_187m: delete bdev LVS_31/SNAP_187m: initiation failed (taking the migration back also failed: cancel migration 58205fd2-2a99-4579-a0ec-af753617919d: ... The second of those happened inside MigrationSourceAndTargetFaults during deliberate fault injection, and that case PASSED. Nothing asked the follow-up question. MIG-F-013 does retry after a cancel, but it reboots the target first, so a rollback that is only recoverable by a reboot still passes there. MIG-F-017 retries immediately, from wherever the volume actually ended up, so it holds whether or not the cancel took. Both quotes above are also as much as we have, because the poller truncated the error at 200 characters and the clause that was cut is exactly the one that says why the rollback failed. Terminal errors are now logged whole; the polling lines stay truncated, since that error repeats on every poll.
MIG-N-010 called `snapshot delete <id> --force` and then failed the
product for deleting the snapshot. snapshot_controller.py:1150 guards
with
if active_mig and not force_delete:
so --force is a documented bypass and the delete was correct. The run
log shows it going through exactly as designed: "Removing snapshot:
4c44671f-..." and then "Snapshot deleted: 4c44671f-..., LVS_1/SNAP_145".
The case now passes no --force, which asks what it meant to ask.
What made this convincing rather than obviously wrong is that
MIG-N-008 passes --force to `volume delete` on the same volume moments
earlier and IS refused. The two are not inconsistent by accident:
lvol_controller.py uses a bare `if active_mig:` for both delete (2725)
and resize (3827), while the guards immediately around them -- node id
not found, STATUS_RESTORING -- all honour force_delete. So two of the
three guards are deliberately absolute and the third is deliberately
waivable, which reads oddly against a doc sentence that lists all
three together. That is a question for dev, not a defect, and it is
recorded at the call site rather than acted on.
The assertion text now also states that no --force was passed, so the
next person to read this failure does not have to rule the bypass out
the slow way.
MIG-N-008/009/010 ran sbctl on both backends. On k8s that reaches the control plane directly, past the operator and past the CSI driver, so whatever it proved said nothing about what a k8s user meets -- the same mistake as creating volumes with the CLI, in the lane whose whole subject is what the product refuses. A k8s user has no sbctl. They delete a PVC, they patch its requested size, they delete a VolumeSnapshot, and the driver turns those into the DeleteVolume, ControllerExpandVolume and DeleteSnapshot the guard actually sees. That is what the three cases do now when k8s_test. This also had to change what the cases accept as a refusal, because on k8s there is not one to read. A refused PVC delete prints nothing -- the claim sits in Terminating behind its finalizer. A refused expand prints nothing either -- the request is accepted into spec and status.capacity never follows. So all three now judge the same thing on both backends: whether what the guard protects is still there. The volume still has a node, the size did not move, the snapshot still resolves. MIG-N-010 already worked this way after the last commit; 008 and 009 now match it, and 009 needed lvol_size to do so. Worth noting for the k8s side specifically: snapshot_controller's guard is `if active_mig and not force_delete`, and the CSI path has no way to pass force_delete. So k8s exercises that guard as written, where the docker case had to stop passing --force to do the same.
Two gaps, both in the shape of an assertion that looks like it covers something and does not. MIG-T-008 is a SECOND migration of a volume carrying a snapshot and clone tree, which is exactly the shape SFAM-2820 describes, and it already called assert_no_target_leftovers. That helper greps the node's bdevs for the volume's uuid prefix, and the artefact is named LVS_31/SNAP_187m -- lvstore and snapshot, no uuid anywhere in it. So the one check standing there could not see the one thing this family is most likely to produce. It now also runs the invariants. MIG-F-018 is the other half of MIG-F-017. That one re-migrates after a CANCEL, which is a deliberate and orderly abort; this one re-migrates after a migration that failed on its own, which has the worse cleanup path and is the one that actually bit. Run 20261013-221421 failed inside MigrationSourceAndTargetFaults with the migration failed: delete LVS_31/SNAP_187m: delete bdev LVS_31/SNAP_187m: initiation failed (taking the migration back also failed: cancel migration 58205fd2-... and the case PASSED, because the class asked only that the migration end terminal with the volume on one of two nodes -- which a volume that can never move again satisfies. The re-migration runs after the fault is undone and after the existing 90s settle, so a refusal is about residue from the failed attempt rather than the fault. assert_target_paths_released is the evidence half, for SFAM-2824. discardMigration clears status.migration.connections and validationJobs only after deleting the validation Jobs and releasing the paths on every node it connected, so those fields surviving a finished operation is the signature of that release being skipped. It is read before the undo, while the record is still the one the failure left. Advisory rather than fatal: that reading has not been confirmed against a real cluster, and failing a case on an unproven inference is how this lane produced "0 of 20 snapshots survived". It logs the CR name and says to check nvme list-subsys on the consumers before the cluster goes away.
Two things found while checking the new cases would survive a run rather than cost one. Order. On docker a refused `volume delete` changes nothing, so MIG-N-008 could sit anywhere among the three guards. A PVC delete is not the same operation: the API server ACCEPTS it and the claim sits in Terminating behind its finalizer, with only the CSI layer declining to finish. Running it first -- which is where it was -- handed MIG-N-009 a terminating PVC to patch and MIG-N-010 a volume already on its way out, so both would have been measuring the delete instead of their own guard. It now runs after the other two and before the check that the guards lift. Severity. assert_snapshots_on_current_lvstore compares the lvstore named in a snapshot's BDev field against what bdev_lvol_get_lvstores reports on the node, and no run has yet confirmed the two spell it the same way. If they do not, every healthy volume reads as stale and the lane fails good cases -- the same inference that produced "0 of 20 snapshots survived" and accused the product of dropping every recovery point. It warns and returns the suspects instead, with what to compare before believing it, and a note to promote it once a run shows the names line up on a volume known to be good. The *m artefact check beside it stays fatal. That one is not an inference: an intermediate bdev outliving the cutover is what SFAM-2820 describes.
Four of the five failures were ours, and the fifth cannot be read yet because of a hole in our logging. MIG-F-018 was a false positive and the most important thing to get right, because it accuses the product. After nic_down on the target, pick_target chose that very node -- the control plane had re-listed it as online while its SPDK was not serving -- and the retry failed with "target node offline (status=down)". That is the fault still clearing, not residue from the failed attempt. It now waits for real node health before retrying, skips with a clear message if health does not return in 600s, and avoids the node just faulted. The case is not wrong in principle: it PASSED after MIG-F-004 in the same run, forty minutes earlier. MIG-N-008 gets its own volume on k8s. Moving the PVC delete last was not enough -- the claim still goes to Terminating behind its finalizer, and the post-migration verify then tried to start a cksum pod on it and waited out its full 300s. The guard only needs an ACTIVE migration, not a finished one, so this migrates a throwaway volume without continuing, deletes its claim, asks, and cancels. MIG-N-009 and MIG-N-010 both passed in this run, and MIG-N-010 passing silently is the confirmation that the snapshot guard holds when no --force is in play -- the gap reported earlier really was our flag. await_batch never logged an error at all, which is worse than the truncation fixed yesterday. MIG-T-008 failed the second migration of a tree -- the shape SFAM-2820 describes -- and the whole record is "batch status=failed phase=migrating members=3". It now logs like the single-migration poller, full once terminal, and reads error_message, which is the key the flattening actually writes. Three bare json.loads on sbctl output now tolerate not getting JSON. Node 0f28a2bb was stuck in_restart, `sbctl sn get` returned nothing, and JSONDecodeError came out of a status poll -- which reads as a harness crash rather than as the node still being down. It cost two classes: MigrationRetryAfterTargetReboot raised on the parse, and MigrationHaPartnerRestart then failed its health check on the same node. _run_json already behaved this way; these three did not. Not addressed here, because it is not ours: that node never came back from the reboot MIG-F-013 performs.
Truncation has now cost this lane two diagnoses in two days. The migration poller cut its error at 200 characters, which landed exactly on "(taking the migration back also failed: ...)", so the more serious half of both rollback failures is the half that went. await_batch logged no error at all, so MIG-T-008 failing the SECOND migration of a tree -- the shape SFAM-2820 describes -- exists only as "batch status=failed phase=migrating members=3". The cluster is destroyed when a run ends, so a truncated product error is not a smaller clue than a whole one. It is no clue, and recovering it costs another seven-hour run. detail() replaces the bare slice. It logs the whole text and returns a short form that says how long the real thing was and that it is in the log above, so messages stay readable and nothing is lost. 84 sites across the eight migration files now go through it; the five slices left are on names, ids and a sample list, where a slice is the point. A preflight check keeps it that way. It matches [:n] applied to captured output -- exc, err, out, combined, last, stderr, stdout -- and was confirmed to fail on a planted regression before being left green.
RandomRapidFailoverNoGapV2WithMigration injected three outage types and LblkRapidOutageStressDocker injected four, so a failure in one lane was not comparable with a failure in the other -- and the lane that runs most often was the one missing a type. The missing type is storage_node_reboot. The base has always had a branch for it at _perform_outage, deliberately kept out of the default list with a comment explaining that appending it without the branch would "count an outage that never happened". lblk_stress.py opted in; nothing else did. It is the only outage of the four that reloads a node's lvstore from disk rather than restarting a process. Without it the lane never exercises what a node actually does after a host reboot, and on lblk that path goes back through the kernel block layer, where the 512-byte sector handling lives. It is also the outage that surfaced the 24-minute restart stall on k8s (SFAM-2825), which nothing on docker would have reached. Both halves of the handler were already complete: the injection waits for offline/unreachable with a 900s budget, and the recovery waits for online with 1200s, longer than its siblings because a reboot is kernel, then node agent, then SPDK. LblkRapidOutageStressDocker inherits from this class directly, so the two lists are now identical by construction and lblk's own append is a guarded no-op. It is kept rather than deleted so the lblk soak still gets the reboot if the base list is ever narrowed again, and its docstring now says that instead of claiming to add a fourth type. The two k8s filters later in run() strip network_interrupt types only, so this survives them.
SshUtils.reboot_node remounts /mnt/nfs_share when ssh comes back, which is the fix written for run 20261004-082535 -- the one where 192.168.10.201 wrote 20G of artefacts onto an unmounted path, reached 95% and then could not come back from the next reboot, taking the run down at 8h17m and the node out of the lab. That remount is inside reboot_node's own wait, which is 300s. The outage recovery here allows 1200s for the node to come online. A reboot whose ssh returns between those two numbers gets no remount and the run carries on regardless, because reboot_node's False is not fatal here -- the recovery waits on node status separately. Nothing would notice: writes to the unmounted path still succeed, they just land on the root filesystem. So the recovery branch asks again once the node is actually online. It goes through _remount_nfs_after_reboot rather than ensure_nfs_mounted directly, because that helper owns the SKIP_NFS escape hatch and the warning that says where the writes went; calling the mount underneath it would ignore both. The helper greps the mount table first, so on the normal path this is a no-op. Relevant now because the nvme rapid soak just gained storage_node_reboot to match the lblk one, so this path is about to run far more often than it has.
Added this morning to match the lblk lane; reverted the same day. The first run with it reached 52 FIO I/O errors in 1h03m, on its second reboot, against a lane that otherwise soaks for 25-30 hours. It is not a harness fault and the reboot path works -- both reboots completed, nodes came back in 2m13s and 3m40s, and the NFS remount added alongside it stayed silent, which is its no-op path. What the reboots exposed is SFAM-2827: rebooting one node marks every OTHER node unhealthy for about 50 seconds, because _peer_connections_relevant counts UNREACHABLE among the states where a peer's connections must still work, and UNREACHABLE is exactly what a rebooting node passes through on its way to offline. Two reboots a few minutes apart overlap those windows, leave no healthy node, and the client logs "no usable path - requeuing I/O". So the lane cannot carry this type until that is fixed: every run will end inside the first hour and the endurance signal is lost. Reverting costs no coverage that was working -- LblkRapidOutageStressDocker has injected reboots since 2026-09-25 and is where the defect was first reached (lblk_rapid_outage_docker-20261002-195205, 42 I/O errors after storage_node_reboot), and it keeps them. Kept: the base's reboot branches, which lblk needs, and the NFS remount re-check in the recovery branch, which closes a real gap for lblk's reboots -- reboot_node only remounts if ssh returns inside its 300s budget while the recovery here allows 1200s. The comment left in place of the append says when to restore it.
Run 20261015-174302 failed in bootstrap with "Clone did not connect"
and the log could not say why. Two clones resolved to the same device:
17:56:20 clnD3SLDSFR9Z4W_ON1TPG6A shares a subsystem; resolved nsid=3
to /dev/nvme4n3
17:57:21 resolved /dev/nvme4n3 for clnD3SLDSFR9Z4W_EFGWFS74 but it is
already in use; refusing it
They are clones of DIFFERENT snapshots (e621e7a5 and 2ce4534f), so they
sit in different subsystems with different NQNs, and ns_id=3 in each is
legal -- a namespace id is per-subsystem. get_nvme_device_for_nqn
filters by NQN at both its sources, so one of those two resolutions is
wrong. Which one cannot be determined from what was logged: the
refusal records neither the NQN it searched, nor the nsid, nor who
holds the device, nor what the client's nvme state actually was.
It now records all four. The nvme dump is taken at the moment of
refusal, because by teardown the subsystems have changed.
Not caused by the max_namespace_per_subsys default going from 1 to 30
on 10-14, which was the obvious suspect: this resolver ran 17 times in
the 10-13 run, before that change. Clones have always landed in their
parent's subsystem -- snapshot clone --namespaced defaults to true and
cannot be turned off from the CLI (SFAM-2819) -- so the path was always
live. It refused once in four runs, so it is intermittent and
pre-existing.
main's cac3c07 turns on ruff's PERF rules, and three of our files fail them. All three are ours -- none exists on main -- so they would have failed CI on the merge, not been inherited. migration_base.pvc_of iterated .items() and used only the value test_lblk churn loop same, at :536; the sibling at :500 keeps .items() because it really does use the key _load_base read loop append() per row from csv.DictReader, now one extend() Behaviour is unchanged in all three.
add_lvol's max_namespace_per_subsys was defaulted to 30 on 2026-10-14 so that every test would exercise a shared subsystem. That is right for the k8s StorageClass, where it is a class parameter, and wrong as a blanket default on volume creation. A child created with namespace=True joins a subsystem the PARENT already made, with the ceiling the parent chose -- 10 in this lane. The default made the child send 30 alongside "namespaced", declaring a different ceiling for a subsystem it was only joining, and the join then stopped producing a block device on the client. It failed silently, which is why it took four runs to see. The control plane still reported the child as joined; only the client-side device was missing, and the lane drops a child it cannot attach. The rate climbed every run: 10-13 21:28 4 joined, 0 without a device <- before the change 10-14 22:29 6 joined, 2 without a device 10-15 11:14 5 joined, 2 without a device 10-15 17:43 6 joined, 5 without a device 10-15 22:00 6 joined, 6 without a device <- bootstrap failed At 6 of 6 no subsystem had more than one member, so the standing namespace assertion fired: "No subsystem is shared by more than one volume: 15 distinct NQNs across 15 lvols." I had looked at this lane after the change and concluded it was unaffected, because the assertion was still passing. It was passing on a shrinking margin. The k8s StorageClass default stays at 30 -- different path, and the k8s runs show no sign of this.
The diagnostic added yesterday captured `nvme list-subsys`, which shows subsystems and their PATHS and no namespaces at all. Run 20261015-225408 produced that dump and it still could not answer the question, which is which namespace belongs to which subsystem. It now captures `nvme list -o json` (DevicePath + SubsystemNQN per device) and walks /sys/class/nvme-subsystem printing each subsystem's NQN and every nvmeXnY under it with its nsid. Those are two independent views of the same mapping -- nvme-cli's and the kernel's -- and get_nvme_device_for_nqn reads both, so if they disagree that is itself the finding. What is known so far, from 20261015-225408: two clones, DIFFERENT NQNs (...lvol:0704cb86 and ...lvol:6147603f), both reporting nsid=7, both resolved to /dev/nvme12n7. Both have their own subsystem in the dump -- 12 subsystems, each NQN appearing once -- so they are not siblings sharing one. Yet _parse_nvme_list_json matches SubsystemNQN with == and get_ns_heads_for_nqn matches sysfs subsysnqn with =, both exact. An exact match on two different NQNs cannot legitimately return the same device, so one of those two sources is answering for the wrong subsystem, and this dump will say which.
The fallback that resolves a clone sharing a subsystem checked the
device it found against a set of already-claimed devices, and that set
held bare path strings pooled from both clients:
claimed = {d.get("Device") for d in self.lvol_mount_details.values()}
claimed |= {d.get("Device") for d in self.clone_mount_details.values()}
if _dev and _dev not in claimed:
The kernel numbers devices per host. /dev/nvme12n7 on 192.168.10.165
and /dev/nvme12n7 on 192.168.10.166 are different disks on different
machines that happen to share a path string, so one client's volume
reserved the other client's device.
Run 20261015-225408: clnOCXUWYB9UC70_G5GMOBVA took /dev/nvme12n7 on
.165; clnOCXUWYB9UC70_WVKRKYFU then resolved /dev/nvme12n7 on .166 and
was refused as "already in use". The device was free. With no device
the clone is unusable and bootstrap raised "Clone did not connect".
The two clones are in different subsystems (different NQNs) and both
legitimately hold nsid 7, since nsid is numbered per subsystem -- so
the identical path was pure coincidence between two hosts.
The set is now keyed by (client, device), and the refusal message
resolves the holder the same way, so it can no longer name a volume on
the wrong machine.
This code is unchanged since 73a452e (09-15) -- the bug is a month
old and needed two clones to coincide on both controller index and
nsid across two clients, which is why it only started failing now.
Nothing we changed causes it; the comparison was always wrong.
Not the same issue as "no device surfaced for child": that path has no
claimed set at all, get_nvme_device_for_nqn simply returns nothing for
a child the control plane reports as joined. Still open.
The backup setup step fails at "kubectl wait --for=condition=ready pod -l app=minio": the pod never starts, because docker.io/minio/minio is no longer publicly pullable. Failed to pull image "minio/minio:latest": reading manifest latest in docker.io/minio/minio: requested access to the resource is denied Not a rate limit, which reports toomanyrequests, and not a general registry problem. Tested on the lab cluster: minio/minio:latest access denied minio/minio:RELEASE.2025-04-22T22-12-26Z access denied minio/minio:RELEASE.2024-12-18T13-15-44Z access denied quay.io/minio/minio:latest unauthorized busybox:latest Running simplyblock/simplyblock:main Running So docker.io works, including for our own images; it is the minio repository specifically, on every tag tried. Nothing changed on our side, which is why this lane set up correctly until recently, and no node has the image cached to fall back on. quay.io/simplyblock-io/minio:RELEASE.2024-01-16T16-07-38Z is pinned instead. Verified on the cluster: Running in 20s, banner reports RELEASE.2024-01-16T16-07-38Z, and /minio/health/live -- the endpoint these workflows probe -- answers OK from another pod. Pinned rather than floating so this cannot be taken away again without us choosing it. Note the fallback is the only path available: the external MinIO at 192.168.10.164:9000 is healthy from the lab (HTTP 200) but no pod can reach it. 192.168.10.164:9000, 192.168.10.210:80 and 192.168.10.72:22 are all BLOCKED from inside a pod, so the cluster has no route to 192.168.10.0/24 at all. The reachability probe was right to fall through; fixing that routing would remove the need for an in-cluster MinIO entirely and is worth raising separately.
…uster's pool
All 26 backup classes failed on run 20261016-100347 and not one backup
was taken. Two causes, both ours.
1. Every StorageBackup create was rejected by the CRD:
The StorageBackup "bck-snap1-aaqtbr" is invalid: spec: Invalid
value: "object": set exactly one of source (to request a backup)
or clusterRef (to record one the store holds)
We sent both. They are two different operations on one kind --
source asks the cluster to take a backup, clusterRef adopts one the
object store already holds -- so sending both can never be valid.
Now source only. Confirmed against the live CRD with a server-side
dry run: the old shape is rejected with that message, the new shape
is accepted. StorageBackupOps was checked the same way and its
restore shape is correct, so the bug is confined to this one call.
2. The lane deleted the cluster's default StoragePool, and every class
after the first then had nowhere to provision from:
10:15:45 class 1 teardown: Deleting pool CRD
'simplyblock-cluster-default'
10:21:51 class 2: CreateVolume failed: failed to resolve pool
"simplyblock-cluster-default" -> PVC never Bound
On k8s add_storage_pool often creates nothing: the operator
reconciles the request to a pool it already has and returns that
name. _add_pool_dual stored it in self.pool_name -- the comment
above it says the operator "may assign a pool name that differs" --
and teardown then deleted it as if it were ours. We asked for
'testpool' and deleted 'simplyblock-cluster-default', which is what
the StorageClass resolves against.
_add_pool_dual now records the pools that did not exist before the
call, and _delete_pool_dual refuses to delete anything outside that
set, saying so in the log. A pool the operator hands back is the
cluster's, not ours.
Cause 2 is why 25 of 26 classes could not even bind a PVC; cause 1 is
why the one class that could bind still took no backup.
Restored at Raunak's request. It was added on 2026-10-15 to match LblkRapidOutageStressDocker and reverted the same day, when the first run with it reached 52 FIO I/O errors in 1h03m on its second reboot. That failure is SFAM-2827 and is not fixed, so expect it to recur: rebooting one node leaves every peer unhealthy for about 50s, because UNREACHABLE counts as a state where peer connections must still work, and two reboots a few minutes apart overlap into a window with no healthy node and no usable path. Taking it anyway is defensible. Reboot is the only one of the four outages that reloads a node's lvstore from disk rather than restarting a process, and on lblk that path runs back through the kernel block layer. It is also the type that surfaced the 24-minute restart stall on k8s (SFAM-2825), which nothing else reaches. The NFS remount re-check added alongside it on 10-15 is still in the recovery branch and still needed: reboot_node only remounts if ssh returns inside its own 300s budget, while the recovery here allows 1200s.
Every backup in the 2026-10-16 lane failed the same way, with MinIO
healthy and the StorageBackup objects accepted:
control-plane returned 400: {"error":"Preconditions are not met",
"detail":"Cluster <uuid> has no backup configuration"}
Both backup lanes gave the cluster its store by patching
StorageCluster.spec.backup after the deployment config was approved.
That can never work. The operator sends backup_config to the control
plane from creationParams only -- the cluster-add call -- and the
control plane exposes no setter for it; /clusters/{id}/backup_config is
GET. A store patched on afterwards sits on the CR looking entirely
correct while the control plane has never been told about it, so the
cluster comes up healthy and cannot back anything up.
The ClusterDeploymentConfig carries spec.cluster.backup for exactly this
reason, and the CRD says so: the field is there so that "a store stated
on the document is present when the cluster is created rather than
patched in afterward by whoever remembers". The workflow comment next to
the patch asserted the opposite -- that backup "has no field" on the
deployment config -- which is how the patch got written.
So the store is stated on the draft now, before approval, and the bucket
is named for the run instead of the cluster uuid. The uuid is not
knowable at that point by construction: approval is what creates the
cluster that would carry it. The MinIO cleanup steps match the new name.
A half-stated store is refused where it is read rather than dropped
quietly, because the CRD requires all three fields and a cluster that
comes up perfectly and cannot back anything up is the expensive failure
here -- it costs a full lane to discover. The pre-approval summary
prints the bucket even when there is none, since that line is the last
look at the document before it becomes immutable.
Every class in the 2026-10-16 backup lane provisioned against encryption-pool and got: failed to resolve pool "encryption-pool": pool "encryption-pool" not found add_storage_pool picks a pool to reuse from `sbcli pool list`, positionally, and existing in the control plane is not the same as usable from k8s. The CSI driver resolves a pool through its StoragePool CR, so a backend pool with no CR in the namespace provisions nothing and every PVC stays Pending. Two pools on this lab are exactly that. encryption-pool is created by the workflow only when tls_enabled is true but survives in the control plane afterwards, so a later non-TLS run finds it listed with no CR behind it. simplyblock-cluster-default is the cluster's own pool and never had a CR at all. Both sort before simplyblock-pool, so the pick landed on the one that could not work. So reuse is narrowed to pools a StoragePool CR backs, and a CR is mapped to its pool through its status first -- the operator sets the backend name and it "may differ from the CRD metadata.name", which the surrounding code already says. An unreadable kubectl counts as no CRs, because guessing yes there puts the original bug straight back. The dedicated path had the same shape of error one level down: having matched a CR against the requested dhchap/allowedNodes, it returned next(iter(existing)) -- an arbitrary pool, discarding the match it had just made and handing back a pool with some other shape, which is the failure that matching exists to prevent. It returns the matched CR's pool now.
The encryption backup classes provision against encryption-pool, and that pool is created only on a TLS run -- it lives in the same branch as the vault settings, because encryption needs a key store. Without TLS those classes did not fail: they fell through to the plain pool and tested nothing, which is worse than failing. So a backup run is a TLS run now, whatever the form said, and the cross-cluster restore lane -- a backup lane end to end -- always is. Resolving it in one step also closes a gap that was costing TLS silently. Both workflows declare the input as default: true, but a default only applies to a workflow_dispatch that rendered the form; on a schedule or a workflow_call github.event.inputs is empty, so every `inputs.tls_enabled == 'true'` gate read "" and skipped cert-manager, the key store and both TLS_FLAGS blocks, while the declared default still said true. The steps read env.TLS_ENABLED now, which is resolved once. k8s-native-upgrade and monitoring-suite-k8s-native have the same empty-input gap. They are not backup lanes and turning TLS on changes what they exercise, so they are left as they are rather than changed in passing. One consequence to keep deterministic: with TLS on, encryption-pool exists and sorts first, so the blind-reuse path would hand it to every caller and the pool a test runs against would depend on whether TLS happened to be on. That path asks for the general-purpose pool by name instead.
Run 39013041668 brought the cluster up, activated it, created all six S3 bdevs at 07:36:44 -- and then sat UNREADY for 57 minutes, failing every retry with "Bdev already exists" until the force-activate step gave up. Two steps patch spec.template on every deployment to add regcred. Patching a pod template changes its hash, so that is a rollout of the whole control plane, and the deployments the loop reaches are the ones the operator has just created, which do not carry regcred yet -- so the patch was always a real change and always rolled them. webappapi, tasks and monitoring all went Terminating at 07:36:48, four seconds after the bdevs were created. Activation runs in a webappapi thread, so it died with the pod, leaving the cluster unready and six S3 bdevs behind. bdev_s3_create is not idempotent: every subsequent attempt failed on the first node, 108 times over the next hour. Nothing recovers from that on its own, and nothing had to before -- without a backup store, activation created no S3 bdevs and an interrupted attempt simply ran again. The deployment patch is redundant regardless. A pod inherits imagePullSecrets from its ServiceAccount at admission, which the service account loop already sets, and a pod admitted earlier that genuinely cannot pull shows up as ImagePullBackOff and is deleted by the following step -- one pod at a time, and only the ones actually broken. So both steps patch service accounts only now. The third copy of the loop, in the pre-CRD step, is left alone: it runs before the operator has created the control-plane deployments and before any cluster exists, so there is no activation for it to interrupt. That bdev_s3_create is not idempotent is a product matter and is not fixed here -- a control-plane restart mid-activation is an ordinary failure, and right now it wedges a cluster permanently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… primitives
Rebuilt directly on main now that #1356 has landed, so this carries only the work that PR did not: 17 files, all in e2e/ and .github/, no product code.
Contents:
Rebuilt rather than merged for the same reason #1356 was. feat/lblk-e2e still carries the R26.3-original commits that main holds as adapted copies, so a merge or rebase fights them rather than dropping them as already-applied. This applies only the delta between backup-rework's tip and lblk's, three-way, onto current main.
Verified: main's own TestObjectLimits registration and device-claim guard survive the patch, the two NameErrors #1356 fixed are not reintroduced, ruff reports nothing in e2e/ or .github/ (all 45 findings are in main's product test files), e2e compiles on 3.11, and discovery finds all six new classes.