Repository navigation
fix(node): robustness pass from the Go node and feature/prt comparison - #301
Merged
Merged
Conversation
…estart The seed's working clone is named after the process id and a per-process counter, so its first clone is .work-<pid>-0 on every start. The staging sweep ran in run(), after setup had already seeded. Where the PID repeats across restarts (a container's PID 1), a seed killed mid-clone left that name behind, the next seed's clone failed on it, and the node exited 1 on every restart until someone deleted the directory by hand. Storage now sweeps at open, under the directory lock and before the seed. The harness keeps its own call, since its restart reopens without initialize.
Both endpoints parsed as Url through clap's FromStr parser, whose error quotes the rejected value. A URL pasted without its scheme, or with stray quotes, therefore printed its API key or relay token to stderr, whether it came from the flag or from the environment: hide_env_values covers help only. A value parser now refuses it naming the flag and the parse error, withholding the value, and keeps exit 2. The README adds that keyed URLs belong in the environment variables, which local users cannot read.
The floor refused every batch and every roll below 1 GiB. The roll loads privately and writes two rows, so it cannot SIGBUS, and its floor only held back the settlement row and with it the root join. A sealed epoch's batches do need disk, but they are the clock path: an epoch seals only once the previous one is accepted, so nothing with a deadline runs beside them, and a refusal turned a batch that would fit into a certain stall of the root join. The floor now guards only open-epoch batches, which run beside the previous epoch's live dispute and can wait, and leaves the space to the work that carries a clock. Crossings stay unguarded for the same reason.
The chain delivers every input, terminal app or not, and the step's delivery to a terminal machine is a no-op. The node handled it (the ruler checks for a terminal state before feeding, and the proof logs the no-op delivery), but no test fed a later input to a terminal machine: the spec scripts ended at the halt, every terminal corpus case has one input, and the runner tests used echo and yield. Three layers now pin it: - the spec scripts gain a halt followed by an input, and the toy treats that delivery as the step does, so stepping, sampling, positioning and proofs at every slot cover it; - the seam witnesses gain two openings that feed a terminal machine (with and without a pending yield), replayed by NodeWitnessesTest. Only this catches a proof that drops the no-op log, since the state is unchanged and the node's post-state check passes; - the corpus gate runs the five terminal mcycle cases through the production runner with one more input, and the released root must hold, settled and served. Both divergence messages now also name COMMITMENT_SEMANTICS, the stamp a change to commitment construction must bump.
A call that reverts at latest is not sent, and the overdue ERROR quoted the raw RPC error, where a custom error is a hex selector. The lane now names it by its signature in the consensus, tournament and step bindings, with the raw data for its arguments, and keeps the RPC error when no binding knows it. The page that never clears, a won root the guardian then forecloses, now reads ApplicationForeclosed(address). A foreclosed epoch never settles, so it also never logged the point the README tells operators to wait for: recovery now logs once when every bond owed to the signer is recovered. This closes the to-do item.
run() merged the first exit with the drain's failures inline, and nothing tested it. That is where a worker failing during a signal-started drain once exited 0. The decision is now a small function with a test: a drain failure after a stop signal keeps its status (30 for a terminal one), a first failure keeps its status over a later drain failure, and a drain panic exits 101.
The panic hook logged its record across several lines, so a line-oriented pipeline kept only the location with the ERROR level and dropped the message to level-less lines. The record is now one line. The startup line also says whether the signer is a local key or an AWS KMS key, which a bug report needs and the line did not show.
An ERROR means an operator must act, and the smoke only counted each scenario's exit status, so a change that makes an honest run page would ship. The smoke now fails a scenario whose node log carries an ERROR line; the retained e2e logs of recent runs carry none. Wall-clock pages cannot fire in scenarios this short, so this pins the exact-state errors and panics.
The engine classified a yield (awaiting input, rejected, terminal) through cm_receive_cmio_request, which also reads the yield's data field and throws when its guest-declared length exceeds the 2 MiB TX buffer. The step reads only iflags.Y and tohost's device, command and reason. A guest bypassing libcmt with such a yield moved the chain on while every node errored on it each tick. The engine now reads the same header fields as the step. A Y flag under another device or command, which only a preset template can carry, is a fixed point the step neither feeds nor reverts, so it reads as terminal instead of panicking in the bindings. The trust-model caveat this needed goes from dimensioning.md and node-architecture.md.
- No input can overrun its window: delivery sets the step's per-input budget of 2^48 - 1 big cycles, so the window's last big cycle starts at a fixed point, and an input that exhausts the budget leaves a terminal app the node defends. The ruler's tripwire fires only on an emulator or engine defect. The docs listed it as an accepted input-reachable halt, which invites guards against a trigger that cannot exist. - A restart cannot change what the node does: every action and durable write derives from chain state and the store, and process state gates only logging, paging and read cost. - P1 rules out any operator flag that chooses the read block, disables a check or caps execution: such a need is a code change. - A join is the one stale call the contracts accept over a commitment the node no longer defends. That is why joins come from finalized state, and why the lane may leave an obsolete pending transaction to be mined or outbid instead of cancelling it. - A crash orphan that no publication re-adopts stays until a rebuild; the GC comment claimed otherwise. - A re-signed pending slot repeats its bytes only while the quote repeats, so the open question of whether KMS signs deterministically changes only a trace-level verdict, and its to-do item goes.
…verage reach - The settlement proof's register addresses come from the solidity-step copy bundled in cartesi-rollups-contracts, not from machine/step, so moving them needs a rollups-contracts release; two existing tests guard the copy. - The bulk collector also builds the height-30 quartets a descent rebuilds, not only the height-38 leaf, and the node never calls the mcycle collector. - No commitment test runs against the release binaries' own emulator build; the arm64 claim rests on upstream's native arm64 corpus run.
- The supervisor is in the liveness path. Startup exits 1 when an endpoint does not answer, and systemd's defaults stop a unit for good after five failed starts in 10 seconds, so a node started before its RPC after a reboot stayed down. The README now gives a configuration that never gives up (Restart=on-failure, RestartSec=5s, StartLimitIntervalSec=0), and says Docker and Kubernetes restart every failure status. - Signal deaths name their likely cause: 137 out of memory, 135 a full data filesystem under a shared mapping, 134 and 139 crashes below the panic path. A 101 loop at a snapshot-root assert means rebuilding the directory, never deleting one row or snapshot, which a replay adopts. - Memory sizing counts the catch-up chunk's input bytes, whose out-of-memory kill repeats on every restart. - A refused bond payment mines a no-op per tick, and its page shows a nonce that advances, unlike a relay hold. - An endpoint must omit `"error": null`, which alloy rejects. - The polling interval's help relates it to the 5-minute response window.
- Upstream: ask for EMLINK in the emulator's clone fallback. Every snapshot hard-links the machine's read-only files, so btrfs and ext4 stop live snapshots near 65k links per inode, where the runner and positioning stall. - Before the audited release: a review of the commitment and proof paths by someone who did not write them. The September budget-seam bug was shared by the Lua oracle, so the oracles share blind spots. - The validator-grade re-measurement also times one root response and one leaf build beside a busy runner, and counts a leaf proof's re-preparation. - The provider plan prices Sybil read load (about 0.72 ETH per live child, most of G at about 1,000), and records a smaller lever: one multi-address log read per generation.
The README recommended any filesystem with reflinks, btrfs included. But every snapshot hard-links the machine's read-only files, and btrfs caps a file at 65,535 links, which an application at gap 1 reaches at about 65k live accepted inputs; there clones fail and the node stops computing, and no disk provisioning lifts it. btrfs also exhausts metadata apart from the free space the node's floor reads. XFS with reflink=1 (the mkfs.xfs default since xfsprogs 5.1) has reflinks and a 2^31 link cap, so it is now the one production filesystem; ext4 (no reflinks: each input copies the machine) is labelled development and small applications, btrfs development only. The startup line names the data directory's filesystem, and on Linux the node warns once when it is not XFS, since many cloud volumes default to ext4. The disk note sizes snapshots for the two live epochs and the input payloads the database keeps, and says a copy to another filesystem expands every snapshot. The EMLINK upstream ask becomes low priority, and the measurement item gains the Linux XFS numbers every APFS figure stands for.
The read endpoint is trusted for truth and completeness, but its availability is not assumed: while it does not answer, each late response draws on the dispute's censorship budget. Operators are now told to put a failover proxy with lag detection in front of two or more independent upstreams, what each upstream must do (serve the chain, error rather than clamp a range past its head, no stale latest or finalized cache), and to prefer upstreams other validators do not share. The node keeps one endpoint on purpose: generic transport redundancy belongs in a proxy, and alloy's fallback layer either never fails over or fans every read out. The provider plan gains the measurement this leans on: whether providers clamp a log range past their head.
The emulator's prt branch holds claims in a frontier forest: every node above a bundle height, repetitions as fixed-point trees, bundles reopened by replay. The node already does the same with persisted quartet rows and run compression, at one 88-byte row per input against the forest's 4-35 KB, and it resumes after a restart. Porting the forest would grow the code and merge the node's container into the emulator's lineage, which the release corpus now checks independently. The docs record that reason, size the root composition's memory (about 250 bytes per epoch input), and note that the Lua sybil's interned hashes make a dense two-level leaf exceed its memory as well as the clock.
The reader logs ERROR every tick whose latest block is older than G, to report a read endpoint that stopped following the chain. Anvil mines only with a transaction, so on its chain an idle stretch ages the latest block with nothing wrong. The e2e smoke's new ERROR gate caught it on its first run, in a slow chaos scenario on an idle anvil. The check now skips anvil's chain, as the startup's missing-relay warning already does.
Positioning skipped a boundary whose loaded root differed from its row and replayed from an earlier one. The fallback never repaired anything: the crossing re-adopted the torn directory by its content-addressed name, so it became the next window's revert checkpoint (a rejected input there still panicked), and every window above it paid a full machine store, up to the snapshot gap minus one. It healed only sidecar damage at an accepted input, an out-of-model fault, since publication verifies before its rename. A torn dispute boundary now stops the node with the path and the remedy, as the runner's cursor already did, which settles the open to-do item. A vanished directory is still skipped and republished.
The emulator's prt recipe keeps a bounded number of checkpoints and widens their spacing as the epoch grows, which bounds disk. Here it would not: every input pays the rotation's close, clone and SHARING_ALL reload (about 140-200 ms on macOS) whatever the application does, so a crossing's cost grows with the number of inputs crossed, and a spacing of about n/K inputs lets the adversary choose it. The recipe affords it because its checkpoints are in-memory forks. Spacing by work alone fails the same way. The fixed gap bounds both clock-path costs it touches, the runner's throughput bounds disk, and on XFS the link limit never binds. The measurement item now names the per-input overhead and the runner's throughput on Linux XFS, from which the default gap and the aggregate-throughput price follow.
Ingestion, joins and epoch completion read finalized state, so finality that stops while the chain goes on stalled them at debug, with no signal until a join was already late. While latest is fresh, the reader now logs ERROR once the finalized block is over an hour old (Ethereum finalizes 13 to 19 minutes behind its head), naming the stalled duties and both causes, a finality incident or the endpoint's finalized tag. It reports and handles nothing, as for a stale latest block. Each committed batch also logs its duration, the time per input and the inputs still waiting. The runner's per-input time bounds the inputs it can keep up with and sizes the disk, and until now an operator could only infer it from the spacing of per-input lines.
The model spent its precision unevenly: seconds of relay hold and a heal for out-of-model corruption were priced, while realistic, cheaper failures went unstated. dimensioning.md now states three rules: - One delay ledger. Every delay draws on the same C, whatever its cause: censorship, a finality stall, a provider, relay or host outage, a restart, a page nobody answers. The node reports the delays it sees and works around none, nor compares its own latency with G, which closes that open to-do item. - One reference price. The cheapest accepted loss is the aggregate throughput attack. Every input, accepted or rejected, costs the runner a fixed overhead whatever the application does, so a few million inputs outrun it: 100-2,500 ETH on Ethereum, but 0.1-2.5 ETH on a cheap L2. Guards against costlier attacks are doors; cheaper walls are to close. The aggregate contract therefore binds for every application, not only heavy ones, and the 2^24 flood is beyond the reference price. - Evenly distributed effort, and an adversary who also chooses when. The per-input contract counts the node's overhead and the seal's open tail. The README prices floods on the experimental L2s, states the downtime budget, and sizes the host for the runner and a dispute together.
Heavy applications lower the snapshot gap to fit the per-input contract. At gap 1, though, every cheap input of a flood makes a snapshot, so disk fills at about a hundredth of the reference price, the one wall left below it once XFS removes the link cap. The to-do item records the design to build before such an application runs in production: publish a boundary at a work budget as well as at the gap's count, append-only and deterministic, so a heavy application keeps the default gap. A count cap that thins old boundaries also bounds disk, but adds a deletion path in a live epoch.
The floor stopped guarding every sealed epoch's batches, so that a sealed epoch's root join is never held by it. That also dropped it for a fresh data directory replaying settled history, where no clock waits: on a disk too small for the history, the runner died by SIGBUS in a silent restart loop where it had stopped with a paged error. The plan now knows whether a sealed epoch has settled (its successor is sealed, which happens only once it is accepted), and the floor exempts only the newest sealed epoch.
Its staging sweep removes working clones, which is safe only because no other process can hold one: every production start takes the lock first.
- The reference price is now derivable from the doc: a root window of about a week over the measured per-input overhead holds 3-4.4 million inputs, about 0.9-1.3e11 gas at 30k per batched input, so 90-1,300 ETH on Ethereum and 0.09-1.3 ETH on a cheap L2 (it read 1.3-2.5e11 before). - Disk sizing counts every input: one snapshot per gap inputs, shared only by a batch with no accepted input. - ext4 caps live snapshots near 65k links too, and every input clones, not only accepted ones. - alloy's fallback layer does fail over; it does not fit because it ranks by success and latency, never head freshness, and fans reads out. - A vanished dispute boundary is republished only when it is the disputed input's. - The ledger keeps G in its statement and has three rules, not two; the host bullet no longer repeats the provider one; the smoke's ERROR gate is in test-harness.md, with the reason; overlong lines are reflowed.
- The seed-staging test proves the sweep runs; its collision with the planted name, and so the ordering, holds only when it runs alone, because the checkout counter is process-wide. Its comment says so. - The finalized-age lines could not see the block tag the mock answered, so they only re-tested arithmetic; they go. - The corpus gate counts its terminal runner passes, so a renamed case in a future corpus fails instead of dropping its pass silently.
… node What the node has never done matters more now than more of what it has: the canonical table has never run node-against-chain, the production clone path (reflinks on XFS) has never run, no two sling nodes have defended one epoch, and the guardian's backstop has never been rehearsed. The staging item now runs on Linux XFS, from the README, by someone who did not write the node, after one harness run on the canonical devnet and with one disk-full check. A new "Before mainnet funds" item rehearses foreclosure inside the claim staging period on a throwaway app, the one defense that covers bugs nobody has found. The tests gain two honest nodes on one epoch.
A probe of the rotation on macOS put about 130 of its 141 ms in the reload's walk of the clone's 50 MB hash_tree.phtc: the emulator follows the cache's list one dependent page at a time, and a fresh clone is a new inode with an empty page cache. A warm reload of the same directory takes about 2 ms in every sharing mode, so SHARING_ALL was never the cost. The architecture doc says so, the Linux measurement now times the rotation with and without a sequential read of that file (a 15-line workaround that measured about 25 ms per input) along with the clone's writeback and reject mixes, and the cache read joins the upstream asks. The reference price records its ceiling: below about 36 ms per input the 2^24 input span binds first, so a faster runner raises the price by about four times at most.
Positioning skips a boundary row whose directory vanished and crosses from an earlier one. Only a storage unit test, through the dispute write-back that production no longer takes, showed the boundary coming back. The engine test removes a gap boundary, positions at it, and checks that the crossing serves the replayed root and republishes the boundary under its row as a clone of the start.
The README listed the restart directives and StartLimitIntervalSec in one breath and placed them all in [Unit]. Only the start limit belongs there; Restart=, RestartSec= and RestartPreventExitStatus= are [Service] settings. A small unit-file fragment says it unambiguously, and it adds the backoff the panic advice asks for where systemd supports it.
The emulator's next release fixes the cold read that made up most of the per-input overhead, at about 2.5 ms per input in a preliminary implementation, and adds validation on an untrusted load. It is not a blocker. The Linux measurement now times the overhead on that release instead of weighing a node-side pre-read, the upstream item becomes the bump and re-measure, and the throughput figures say they hold for v0.22.0 and stay provisional until the Linux run.
The delay ledger listed a halted chain among the delays that draw on C and said a week-long finality stall loses the dispute whatever the node does. Both overstate. Every contract deadline is a block.number delta, so on Ethereum a halt of block production freezes every clock and missed slots only stretch budgets in wall-clock time. A finality stall while blocks continue holds ingestion, joins, refunds and completion, but a joined match's responses and its settlement read latest and go on: the stall draws on C only past a join's own budget, and loses only a join it outlasts. The stale-head page no longer blames the provider when the chain itself stopped, the finality page says responses go on, and the provider plan notes the unfinalized tail a stall widens.
The fix that reads a yield from the HTIF header had one direct test, an accepted yield preset in registers and classified by a direct call. The new tests have the guest execute an accepted and a rejected yield whose declared length is one byte past the 2 MiB TX buffer, which the bindings refuse to read, and drive them through the runner's collect and both collectors. The step's reset verifier decides the expected root. Both tests fail with the old classification.
GCdePaula
marked this pull request as ready for review
October 19, 2026 17:12
stephenctw
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A robustness pass over the sling node, from a comparison with the Go
rollups node (next/2.0) and the emulator's
feature/prtbranch. Thecomparison found no divergence in commitments, proofs or machine use. What
it found is smaller: one guest-triggerable stall, a few operator footguns,
missing alarms, and trust-model prose that priced some guards and left
cheaper exposures unstated.
Reading guide
The commits are granular and each stands alone. Read them in order, or by
group:
Node behavior
classified a yield through
cm_receive_cmio_request, which throws whenthe guest-declared length exceeds the TX buffer. A guest bypassing libcmt
made every node error on each tick while the chain moved on. The engine
now reads the header fields the step reads.
boundary that does not hash to its row is local corruption; the node now
stops and names the rebuild instead of re-storing the boundary.
and endpoint URLs carry API keys.
epochs and replays of settled ones, never the newest sealed epoch.
the data directory's filesystem and warns on Linux when it is not XFS.
hour without finality progress, a per-batch INFO with ms per input, and no
head-age page on anvil's chain.
Tests
the release corpus's terminal cases);
false page on its first run);
storage unit test through the dispute write-back showed it, and
production no longer takes that path);
rejected, executed by the guest and driven through the runner's collect
and both collectors, with the step's reset verifier deciding the expected
root (both fail on the old classification);
Docs
the collector and corpus coverage reach;
fragment), sizing, XFS with reflink=1 as the one production filesystem,
provider redundancy outside the node;
in the blocks the contract clock counts, so an Ethereum halt freezes the
clocks and a finality stall holds joins but not a joined match's
responses) and one reference price (the aggregate-throughput attack), with
its ceiling: below about 36 ms per input the 2^24 input span binds first;
SHARING_ALL, but the reload's coldread of the fresh clone's 50 MB
hash_tree.phtc, which the emulator walksone dependent page at a time (about 130 of 141 ms per rotation on macOS;
2 ms warm), which the next emulator release fixes;
the spacing that would close it, and what the emulator's frontier forest
would and would not buy;
and proof code, and a guardian backstop drill.
Decisions this records
hard-linked inode near 65k, and every snapshot links the same PMA file.
the flood price follows the slower path, and skipping the per-input
checkpoint leaves every reject at a cold load plus a replay. The lever is
the cold load itself, which the next emulator release fixes (about 2.5 ms
per input in a preliminary implementation; not a blocker).
Follow-ups, not in this PR
production caller since the torn-boundary stop. It changes the dispute
feeder's checkpoint contract and the whole-epoch differential tests, so
it comes as its own PR on top of this one: refactor(node): feed a dispute ruler only the window it was positioned at #302.
upstream; EMLINK in the clone fallback and corpus vectors at the budget
seams are open.
Validation
On macOS with emulator v0.22.0, at the branch head:
just check: formatting, lints, clippy and 355 unit tests;just test-node-harness: 28 tests against anvil;just test-engine-machine: 22 real-machine tests, the new republish testincluded;
(
computation_hash_corpus_dave_matches_release_manifest);just e2e-smoke: 5/5 (echo simple, chaos, kill_catchup_batched, stf_all;yield stf_revert), with no ERROR logged by the honest node.
Not covered: nothing here has run on Linux; every timing is macOS APFS.