Repository navigation
Sling node hardening for the v0.22 generation - #299
Merged
Merged
Conversation
….0-test3 The emulator release candidate bumps MARCHID to 22, rebuilds the uarch (a new pristine uarch hash) and moves stored machines to archive version 8, so every machine root moves: the test images, the devnet and every fixture are regenerated, after the engine's differentials passed against the new library and before any fixture was trusted. It is a new deployment generation for the state transition. - CartesiStateTransition publishes solidity-step's exported CM_MARCHID instead of a local literal; its bytecode, and every address built from it, moves. - The bindings take CMIO response lengths as u32 and refuse a longer payload rather than truncate it, mirror backing_store_config's new mark_pages_dirty, and expect version 0.22. - The reference CLI tests use the new computation-hash flags and input_file_index keys; the corpus pins the release's archive, its new mcycle-overflow case, and uarch-near-limit-tail as error-no-hash, whose template the node still refuses. - The seam-1 control test now pins the fixed collector: a rejection on the input budget's last cycle resets to the revert root. - CI installs the release candidate's package, the gas script pins the rebuilt yield image, and the constants benchmark accepts the 0.22 module.
v0.22 exports cm_get_uarch_pristine_hash and the uarch state's start address and log2 size, so the sys crate drops its hand-written declarations of the library's internal symbols (and the provider requirement they added), and the node and the bindings tests stop pinning the coordinates. The guard test now also holds the linked library's CM_MARCHID to solidity-step's export, so a mismatched emulator or step fails in unit tests rather than at node startup.
The v0.22 collector resets to the revert root when an input is rejected on its budget's last cycle, as the step does, so bulk collection takes that cycle instead of leaving it to stepping. The control test becomes a positive one: bulk collection returns the cycle and leaves the machine at the pre-input checkpoint. The ruler's residual assert now names the only input bulk collection still cannot take, one delivered at mcycle 2^64 - 1.
v0.22 opens the files it flocks with close-on-exec, so an anvil spawned while another test stores a machine no longer inherits its locks. The harness and the other anvil tests pass in parallel (three runs, 26 tests each, about 36 s against 110 s serially).
Close the upstream asks v0.22.0-test3 and solidity-step v0.16.0-test3 deliver (the seam fixes, close-on-exec, the pristine uarch API and range, the exported CM_MARCHID) and keep the open ones. Add the promotion step: both release candidates report 0.22.0, so the final tags need rebuilt images and fixtures and fresh data directories. The CLI exclusions narrow to the uarch cycle overflow and an unverified halted-template lead, the seam-1 guard leaves the glossary, the measurements stay labelled as v0.21 figures until re-measured, and version-specific wording in the justfiles, READMEs and test comments becomes pin-neutral.
ReactError and EpochManagerError were transparent wrappers that nothing
matched on except one arm handing its anyhow error to the reader. Every
caller formats them with `{:#}`, so plain anyhow keeps every message and
error chain while dropping the conversions at each call site, and it lets a
typed cause anywhere in a chain be found by downcasting.
A supervisor could not tell a terminal refusal from an RPC blip: every failure exited 1, and a panic or an unexpected worker stop reached only stderr. The exit status now follows the sequencer's contract (fatal.rs): 1 retry, 30 terminal, 40 rebuild the data directory, and 101 for a panic, restart and page, because a node restart is harmless and may clear a nondeterministic fault while a dispute clock runs. - Startup refusals of a flag, the template or the deployment are terminal; schema, semantics and emulator, consensus or geometry pin mismatches ask for a rebuild; transport failures stay retryable. - A root that starts from a state the node did not compute (a divergence it lost, or corrupt local state) stops the node with exit 30 instead of a panic loop; the Hero meets it only after the settled epoch's refunds. - A panic hook logs every panic at ERROR, main logs the final error once, and a worker failing during a signal-requested drain no longer exits 0.
Persistent failures that would lose a dispute stayed at WARN forever (a full disk, a dead or lagging provider, a relay holding a transaction, a failing dispute tick), while two narrow rules escalated after one repeat. One rule now covers them (escalation.rs): WARN while a failure lasts, ERROR once it has lasted G, the 5-minute response window past which a due response draws on the censorship budget; a transient blip never pages. - The reader, the runner and the manager's tick and dispute, settlement and refund planning use it, as does a sealed epoch the runner has not finished, which delays the root join. - Every call the manager plans is tracked across ticks: one still planned after G has not landed, whatever the lane's verdict, and the error carries the revert reason when it reverts at estimation. It replaces the consecutive-tick revert and log-check rules. - G is hardcoded, with the reasoning and when to revisit it: the node knows no block time to convert the table's budgets. - Ingestion waits silently until the deployment finalizes instead of failing every tick, which would now page every fresh deployment.
The planner already knew how the local commitment failed, but the node logged four missed-obligation waits at debug and folded every loss into one undifferentiated ERROR. - A wait that means an obligation was already missed (joins closed before the node joined, its clock expired, its match or its match's child is eliminable) logs ERROR every tick. - A loss names its case: elimination with its reason, another class's child win, a root won by a commitment with this node's final state (the correct state settles), or a divergence, with both final states and the guardian's remedy. - A sentry claims the state it computed once the root has a winner, its own or not; staging and acceptance still follow only its own win. A harness test has a node miss its root, lose it to a wrong commitment, and still claim.
The gap GC deleted runner-epoch boundaries that were not multiples of the gap. Since the runner publishes only whole-gap batch anchors it removed nothing in steady state, but after --snapshot-gap-inputs changed to a value that does not divide the last anchor, every later anchor was a non-multiple and each commit deleted the previous one, leaving the open epoch with its start and its newest boundary. A dispute on that epoch then replayed from the start, far past the documented gap - 1 inputs, on the response clock. Epoch GC still frees every boundary once the epoch completes.
aws-config bounds only the connect by default, so a KMS endpoint that accepted a sign call and never answered parked the epoch manager's tick forever: no action left the node, and nothing was logged. The client now sets a 20-second operation timeout, the RPC client's per-attempt bound, so the tick fails and the paging rule reports a persistent outage.
…agging head - A cold fold (a restart, or a dropped Solid) read every live child tournament from the root's creation block, though a clone has no logs before it. Behind a provider that caps eth_getLogs ranges, a restart late in a long dispute with many children issued hundreds of sequential range queries against a 5-minute response window. Each child is now read from the block of the NewInnerTournament that created it. - A finalized head behind the one Solid holds (backends behind a load balancer) failed the Hero's whole tick. Solid is still final, so the reader keeps it, as ingestion treats the same head as nothing new.
Working clones are mapped shared and grow as the guest writes, so a full filesystem would most likely kill the node by SIGBUS mid-batch, with no log line, and every respawn would replay into the same death. The runner now refuses a batch or a roll below 1 GiB free, about one stored machine, with an error the paging rule reports, while the Hero keeps defending computed epochs. The roll's disk line summed apparent sizes, which counts every reflinked or sparse clone at full length and never said how much room was left; it now logs the filesystem's free space.
A provider stuck at an old head (a node that lost its peers, a dead backend behind a load balancer) answers every read consistently, so nothing fails: ingestion sees no new blocks and the Hero waits on turns already taken while the clocks run. The reader now logs ERROR on each tick whose latest block is older than G by its timestamp. It is outside the trust model, so the node reports it and works around nothing.
The startup line now names the node, the linked Cartesi Machine version and its MARCHID, so an operator can tell which emulator a running binary links. Startup also says whether the signer is a sentry of the consensus, read once, since a sentry that silently casts no claim is the one surprise an operator would not see in the logs.
…rules Name what each exit status asks of a supervisor (and the systemd setting that stops a terminal loop), what to alert on beyond ERROR lines, why the data directory belongs on a reflink filesystem and stays put, and that the signer must accept ETH transfers within the payment gas.
- Log the first worker failure when it happens, not only after the drain, and keep its exit status if a second stop signal cuts the drain short. - Check a fatal error before the shutdown arm, so a divergence seen during a requested stop still exits 30. - Make only the template's deterministic refusals terminal; a transient load failure (memory, file descriptors) retries instead of holding the node down under RestartPreventExitStatus. - Page the runner wait only once the runner has reached the sealed epoch, so a fresh data directory's replay of history does not page for hours. - Vote as a sentry only once the root is decided, as documented. - Restart the runner's paging clock on each committed batch and roll. - Word the loss and stale-head errors accurately, check the claim value in the sentry test, and fix stale doc comments and the README's exit 40.
The node version never discriminated (the crate version is frozen) and the raw schema fingerprint refused stores over comment edits. node_metadata now holds STORE_FORMAT and COMMITMENT_SEMANTICS; a test pins schema.sql's hash so every schema edit decides whether to bump the format.
It only kept GC off the template's directory, which epoch 0's start row already does while anything can need it; a pinned directory never seeds again.
The database stored absolute snapshot paths, so a data directory could not move. A boundary row now names only its state hash; its directory is snapshots/0x<hash>, derived on every read and in GC. The CAS table that held the paths goes: GC releases boundary rows and, in the same transaction, keeps the states a remaining row still names. The plan and crossings derive their paths from the hash, the harness adversary links the node's snapshots into its private store, and the e2e seam derives the path the same way.
The trigger layer duplicated the writers' checks and turned two of them into string-matched panics. The writers now hold every class alone: the boundary row compares with its stored hash and panics on a disagreement, as settlements and quartet nodes already did, and completion refuses an epoch ingestion has not seen. escalate_tripwires goes with the triggers; raw writes stay unsupported.
…once The roll checked readiness twice, the window-root prefix three times and the final boundary's hash four times, and stored a machine that finishing an epoch never changes. It now loads the verified final boundary, folds the settlement root, and in one transaction re-checks that the cursor ends the sealed epoch, registers (e+1, 0) on the same snapshot and inserts the settlement. A settlement row decodes in one place with one panic; the tests that bypassed the schema's CHECKs to fake impossible rows go.
GC finds its orphans in one set-based query before the delete. The docs no longer claim the engine config panics on drift (startup refuses it with exit 30 or 40), the staging comment names the publishers that still race, and STORE_FORMAT says a number is never reused.
The actor prepares every intent on the context that planned it, so nine error variants re-checked the intent against itself and fired only on hand-built intents. Preparation now reads the engaged match from the context and panics if it disagrees; the engine tripwires stay errors.
Preparation no longer reads the match payloads the intents copied, so each intent keeps only the tournaments its call targets, and the large-variant allowance goes. A join still names its commitment: Solid must plan that exact join, which decision equality now checks.
LevelMaterial repeated the snapshot's descriptor and local commitment next to the engine coordinates, so preparation compared values assembly had taken from one variable. The context now keeps only the coordinates, and the dead root_tournament and levels accessors go.
…request TournamentResult was HeroTerminal plus Running, and a tick's wave never held more than one request. A tick now carries an optional terminal and an optional request, selecting a cleanup only in the arm where the Hero waits, so one request per tick holds by construction.
The tick already gates on the epoch's write-once settlement row, yet the sentry-claim, stage and accept planners and the loss report each read it again behind a None arm that could not run. They now take the tick's row. The sentry claim reads its epoch from canAcceptStagedTournamentResult, which it needed anyway, instead of getCurrentSealedEpoch.
…functions Every caller handed the arena sender the same provider the Hero's chain already holds, and the only other implementation was a test fake that ignored the bond read's pin. The builders now take the chain's provider, the join's bond read lives with the other pinned reads in the observer, and the test checks the real requests over a recording provider.
…moves to its test Production always set the region and endpoint, so the builder's us-east-1 defaults and its setters only obscured the one call path. Key creation is a LocalStack fixture, not something the node may do in an operator's account, so it moves into the test module, and lazy_static gives way to std's LazyLock.
Refusing a silent submit endpoint at boot makes eth_chainId a relay requirement, which the runbook now states and checks with cast. Its runs move to mainnet: the team found Flashbots' Sepolia endpoint does not include transactions. The README also prices the recovery walk of a template-seeded directory behind a provider that caps log ranges.
Cloning keeps each file's mode and hard-links one without write bits, so a read-only template seeded fine and then every advance batch failed to open its working clone (EACCES; as root, the emulator's read-only refusal), a liveness failure that paged only after 5 minutes. Startup now refuses such an image with exit 30 before anything is written, naming the file and the fix. Only files a shared load opens for writing need the bit, read from the stored config, so the PMAs and read-only drives may stay read-only.
A load takes the root hash from the image's stored hash tree and does not check it on pages marked clean, so a corrupt or forged image keeps a root its contents do not have; the docs ask operators to trust the image's source. The emulator team plans to support cm_verify_hash_tree in a later release (v0.22.0 documents it as for internal tests). The binding, the refusal (commented out in Template::inspect) and its ignored test land now, so that release is a bump, an uncomment and an un-ignore; todo.md tracks it, and the trust notes say they hold until then.
The line exists to identify a binary in bug reports, but it logged the workspace version 2.0.0, which no release carries in its name (v3.0.0-alpha.5 also built as 2.0.0). The release build now injects the tag through CARTESI_SLING_RELEASE, and any other build says it is unreleased. Cargo tracks the option_env! read, so a changed tag rebuilds the crate.
The anvil tests' reasons still blamed the emulator's leaked machine file locks, which v0.22 fixed by opening those files close-on-exec; the harness docs already dropped that paragraph. They now just say they spawn anvil and name the recipe, and the two corpus tests name theirs too. args.rs keeps two such reasons until another branch's edits there merge.
reviews/README.md requires unfinished review work to have a todo line or an owner doc. The small debts deferred until another branch's args.rs edits merge are now listed under node-architecture.md's Known debts, which no longer says none are open, with one todo line for their cleanup; the reader test for several seals in one chunk and the review's untaken decisions get todo lines too.
todo.md makes the Unreleased section this generation's release notes, yet it missed most of the node's operator-visible changes on this branch: the exit contract, the paging rule, the loss and missed-obligation ERRORs, the sentry's vote after a lost root, the disk floor, the KMS bound, the stale read endpoint, the startup line, the store format stamp, and the refusals of an unanswering endpoint and an unwritable template. The entries point at the README's operator notes rather than repeating them.
The node's overhead over the bare emulator holds on v0.22: a cold join at 1.01x and a deep proof at 1.05x. The cold join's disk delta rose from 18 to 34 MiB; it is a free-space delta on a shared machine and was not investigated further.
… out The emulator team asked that nothing call cm_verify_hash_tree on v0.22.0: it is there for the emulator's internal tests and unoptimized. The binding and its test now wait, commented out, beside the check in Template::inspect for the release that supports it.
Any local user can read a process's arguments (ps, /proc/<pid>/cmdline), and --auth-private-key put the signer's key there. The pk signer now takes its key only from CARTESI_SLING_AUTH_PRIVATE_KEY or --auth-private-key-file; the flag is gone, so passing the key on the command line is a usage error. clap cannot read an argument from the environment alone, so the signer reads the variable itself, and help never shows it. Neither or both sources set is terminal (exit 30). The e2e harness passes its key through the environment, and the Rust tests through a key file.
--aws-region defaulted to us-east-1 and the node always passed https://kms.{region}.amazonaws.com, overriding the SDK's own resolution: a validator that takes its region from a profile or instance metadata signed in us-east-1, and FIPS endpoints and other partitions (aws-cn) needed an explicit URL. Both flags are now optional overrides with no default, and without them the SDK resolves as for any AWS client. They no longer read AWS_REGION and AWS_ENDPOINT_URL themselves: the SDK does, with its own precedence (AWS_ENDPOINT_URL_KMS first). A start where no region resolves still fails, naming --aws-region. The operation timeout is unchanged.
--blockchain-http-endpoint defaulted to anvil's http://127.0.0.1:8545 and --blockchain-id to its 31337, values a production start never wants. A forgotten flag already failed loudly through the chain check, but only after the node had connected somewhere it was not told to. Both are now required, so a start without them is a usage error. The e2e harness passes anvil's chain id, and the argument tests pass both.
The config dump was one INFO record over a dozen lines, which a line-oriented log pipeline splits into lines without a level or a target. It is now one line of comma-separated fields under the same labels, with the endpoints redacted as before; the setup test pins both.
run took a ShutdownSignal that its one caller, main, always created fresh, and nothing requests shutdown before run does after the first exit. So worker_failure's graceful arm, for a worker that returned under an earlier request, could not be reached outside tests, and no test reached it. run now creates the signal, and the first exit is always a failure, exactly as before.
Chain::finalized_block_number read the same tag as finalized_head without its zero-hash check, so the reader's tick, the seed probe and startup sampled the durable boundary through a laxer path than the rest of ingestion. They, and the tests, now take finalized_head's number, and the duplicate is gone. The mocked heads carry a hash, as a provider's do.
Every Storage handle, the reader's, the manager's and the Hero's included, carried snapshot_gap_inputs, set by NodeConfig::storage, although only the runner's planning reads it. The runner now holds the gap and passes it to advance_plan, and its default lives beside it; --snapshot-gap-inputs and the runner's behavior are unchanged. Storage tests that set a gap their batch never read drop it, and the stopped-runner differential restarts at its own gap instead of the default a fresh handle carried.
run's startup hygiene promised to grow into another project's ritual, an aspiration no check backs. args.rs justified reading the linked pristine uarch hash outside an error context, but Machine::uarch_pristine_hash is infallible, so there is no local fault to misreport. sync.rs's module doc retold the signal's predecessor; the contract that errors do not travel through the signal stays.
harness/settled_start.rs drives the seeding gate through NodeConfig::setup_with and the seeded node's workers on the deterministic chain, but test-harness.md's inventory never named it, so a reader looking for where seeding is tested against the contracts would not find it.
The other anvil tests stopped blaming the emulator's leaked machine file locks, which v0.22 fixed, while another branch's edits to args.rs were pending; these two kept the stale reason until now.
Operators read the README's Run section and the help block, and the changelog is this generation's release notes. Both now say where the pk signer takes its key and why no flag does, that the endpoint and chain id have no default, how the KMS signer resolves its region and endpoint, and that the startup config is one log line. The help block is regenerated from the binary.
The deferred config-surface debts, the run signal, the finalized-head duplicate, the runner's gap, the comment noise, args.rs's stale ignore reasons and the harness inventory line have landed, so their todo line goes and Known debts keeps only the chain-name gate, which stays by decision.
A start with no private key, or with both sources, failed only after both endpoints had been asked their chain. The key source is now checked first, as configuration to fix before anything reaches the network.
It refuses nothing a supported chain needs, and removing it would touch several signatures for no change in behavior; it is a decision, not a debt.
The early check now parses the key, so a key that does not parse exits 30 before any endpoint is asked, instead of 1 after, which a supervisor restarts forever. The key file's path leaves its read error: clap suggests the file flag to an operator who still passes the key itself. The docs add exit 2 to RestartPreventExitStatus, since the required flags make a usage error the likely upgrade mistake.
GCdePaula
marked this pull request as ready for review
October 16, 2026 21:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves Dave to Cartesi Machine v0.22.0 and solidity-step v0.16.0 and hardens the sling node: louder operator signals, one storage generation, a simplification that removes about 4,600 lines (net about -2,700), and starting a node from the last settled state instead of replaying from genesis. Landed as granular commits; each phase was gated and adversarially reviewed before the next.
Generation impact. This is a new deployment generation for the state transition (
CM_MARCHID22, a new pristine uarch hash, machine archive version 8). Every node needs a fresh data directory: v0.22 refuses machines stored by v0.21, and the store format changed (older directories exit 40). Operators also change their invocation: the private key comes only fromCARTESI_SLING_AUTH_PRIVATE_KEYor--auth-private-key-file(never argv),--blockchain-http-endpointand--blockchain-idare required, and KMS leaves region and endpoint to the AWS SDK. Everything operator-facing is in CHANGELOG.md's Unreleased section and the node README's operator notes.Reading guide
The commits group into five phases; each can be reviewed on its own (the GitHub commit list shows them in order).
Emulator release candidate (from "build!: move to Cartesi Machine v0.22.0-test3 ..." through "docs: describe the v0.22 emulator and step pins"). The bump, the C-API reads that replace hand-declared uarch constants, and the harness running in parallel now that v0.22 opens files close-on-exec.
Phase A: operator signals and participation fixes (from "refactor(node): use anyhow in the hero and the epoch manager" through "fix(node): address the Phase A review"):
RestartPreventExitStatus=30 40.Phase B, storage generation (from "refactor(node): stamp the store with a hand-bumped format version" through "fix(node): address the storage-format review"):
STORE_FORMATwith a pinned schema hash, snapshot paths derived from state hashes (a stopped data directory can move), write-once checks in Rust instead of 26 triggers, and an epoch roll in one transaction.Phase B, simplification (from "refactor(node): prepare each Hero action from the planning context" through "docs(node): address the simplification review"): behavior-preserving: about +2,000 / -4,600 lines. The Hero prepares from its planning context; the
ArenaSendertrait, pass-through error enums and dead surface go; the tournament layer stops re-proving what the observer built (a delegated child is bounded by a pure check against its parent instead of a chain read; every validator in front of an engine assert stays); engine leftovers. No leaf, root or proof byte changes: the spec oracles, the prototype differentials, the seam witness fixture and the release corpus all pin that.Phase C (from "build: pin the final Cartesi Machine v0.22.0 and solidity-step v0.16.0" to the end):
--template-pathagain.docs/runbooks/submit-relay.md).cm_verify_hash_treeon v0.22.0, where it serves internal tests. Until the release that supports it, the docs say to trust a seed image's source like the template.snapshot_gap_inputslives in the machine runner; smaller tidying.Two commits cancel out and can be skipped: "fix(node): a settled epoch's recovery walk stops at its successor's seal" and its revert. They stay so the history matches what was reviewed.
For a contracts reviewer: the only production Solidity change is
CartesiStateTransitiontakingCM_MARCHIDfrom solidity-step; between step v0.15 and v0.16 onlyEmulatorConstants.solchanges.just measure-prt-gas, the authoritative gas gate, passed on the final pins (clean tree, official Forge 1.5.1): all 22 Tournament witnesses and all 13 full-stack leaf-proof witnesses, so no allocation moves and, per the calibration runbook, no new record is due.Validation
On macOS arm64 with the final releases (logs kept outside the repository):
just check; CI green on every push.just e2e-smoke5/5 with no ERROR lines.just measure-node-vs-emulatoron v0.22.0: the node stays at 1.01x the bare emulator for a cold join and 1.05x for a deep proof (docs/measurements/node-vs-emulator.md).just test-kmsagainst LocalStack.Open, tracked in docs/todo.md