feat(archive): add state archive with historical query serving by bladehan1 · Pull Request #55 · bladehan1/java-tron

bladehan1 · 2026-09-20T07:17:22Z

What does this PR do?

Introduces the state archive feature: append-file state archive with catalog serving, geth-style archive index, checkpoint-bound recovery, path-state snapshot pipeline, historical query RPC/VM context wiring, and related performance and metrics attribution work. Rebuilt as a fully GPG-signed (Verified) history from feature/archive_block2; content is tree-identical to that branch.

Why are these changes required?

Provides bounded historical state storage and point-query serving for archive data, with recovery semantics aligned to checkpoints.

This PR has been tested by:

Unit Tests
Manual Testing

Follow up

Extra details

174 commits, all GPG signed. Branch is based on an older develop point and shows divergence from current develop; reconcile (merge/rebase) before merge if required.

Summary by cubic

Adds an opt-in state archive that persists bounded historical state and serves exact point-in-time and range queries, enabling historical RPC and VM execution without changing current chain processing.

Stores append-only history body, index, and commit records with checkpoint-bound crash recovery and WAL validation.
Publishes immutable serving generations with an incremental exact-27 serving index and a bounded range index from committed history.
Acquires native LevelDB/RocksDB snapshot leases and engine identities and request-owned historical read sessions for fail-closed query isolation.
Rebuilds a non-consensus path-state root on startup, tracks it through blocks and reorgs, and publishes it in block headers; the rebuild is streamed and parallelized to bound memory.
Historical VM execution rejects state-changing opcodes and unsupported precompiles via HistoricalExecutionGuard.
Adds offline block replay, resumable HTTP export tools, and archive startup diagnostics.
Archive is default-off; existing nodes need no migration.

Review notes

Branch is based on an older develop point and needs a rebase before merge.

^{Written for commit b76ddef. Summary will update on new commits.}

Capture canonical block metadata and materialize reverse diffs from completed block snapshots. Classify archive state stores explicitly and project account-asset companion mutations before root merge.

Add deterministic history body and index records, ordered durable commit markers, and bounded asynchronous persistence. Gate checkpoint advancement on committed archive history and cover crash, corruption, backpressure, and reorg behavior.

Persist reverse state history with bounded append-only body, index, and commit structures. Wire an opt-in archive writer into snapshot flush and add restart-safe account change indexing. Expose an experimental historical account balance endpoint for trace comparison.

Persist bounded restart checkpoints, truncation recovery, H/C/D/R progress authorities, and immutable serving generations. Pin committed history resources and fail closed on identity or durability drift.

Add durable engine identities and native LevelDB/RocksDB snapshot leases. Assemble exact Store generations under the SnapshotManager barrier and release partial acquisitions on authority drift.

Apply business mutations with participant progress in native sync batches. Freshly validate H/C/D authorities under the archive barrier and advance reader visibility only after exact convergence.

Unify LevelDB and RocksDB participants behind one recovery contract. Persist canonical mutation plans and bind checkpoint, participant, and reader progress to the same digest.

Collect exact ordinary and AccountAsset post-state without Store scans. Enforce target identity, bounded ownership, one-shot sealing, abort, and terminal payload release.

Drive an exact-27 capture batch through mixed native participants, inject a partial apply failure, and verify durable-plan replay, digest propagation, ownership, and restart convergence.

Apply two capture-produced targets through exact-27 participants. Verify per-target plan digests, overwrite and delete semantics, partial replay boundaries, reader publication, and restart convergence.

Prepare one target-bound Account and AccountAsset projection before snapshot merge, retain its forward payload through the revoking lifecycle, and seal it only against durable history markers. Keep the integration default-off and preserve the existing proposal 66 physical layout.

Classify ABI as mutable auxiliary metadata outside the versioned state participant set while retaining an explicit tombstone classification.

Exercise proposal-bound projection, layer-to-flush ownership, durable marker sealing, retry and reorg boundaries, exact participant coverage, and ABI exclusion.

Expose durable marker receipt creation through direct and asynchronous history sinks, then gate forward payload sealing on the exact committed flush range. Retain submitted frozen ranges across durability or receipt failures so retries do not resubmit history.

Verify direct and asynchronous receipt authority, disabled behavior, and retry ordering across durable wait and receipt failures without duplicate history submission.

coderabbitai · 2026-09-20T07:17:29Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 19725bf1-fe04-4a29-828a-05df43a1415b

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

cubic-dev-ai · 2026-09-20T07:17:43Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

# Conflicts: # actuator/src/main/java/org/tron/core/vm/repository/RepositoryImpl.java

github-actions · 2026-09-20T07:22:37Z

❌ Math Usage Detection Results

Found forbidden usage of java.lang.Math in the following files:

./chainbase/src/main/java/org/tron/core/db2/archive/HistoricalQueryControl.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveAppendCheckpointMaterializerV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveBlockFrameCodecV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveCataloglessFormatV4.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveCataloglessMetadataV4.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveCataloglessWriterV4.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveCommittedViewV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveFiveLaneWriterV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveLaneIndexV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchivePointReaderV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveProcessFdAdmissionV5.java
./chainbase/src/main/java/org/tron/core/db2/archive/StateArchiveTailV5.java
./chainbase/src/main/java/org/tron/core/db2/core/CommonCheckpointRuntimeOwner.java
./framework/src/main/java/org/tron/core/db/Manager.java

Please review if this usage is intended.

Caution

Note: You should use org.tron.common.math.StrictMathWrapper.
If you need to use java.lang.Math, please provide a justification.

cubic-dev-ai · 2026-09-20T07:44:10Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

Direct SnapshotRoot writes only occur while no revoking session exists, so the legacy eager Account-to-AccountAsset migration still applies to them; session-bound writes remain folded by the P66 materializer inside the block layer.

SEGMENT_TARGET_BYTES is bound into the v3 composite format identity and persisted CURRENT records; the v4 commit raised it to 2 GiB and broke frozen pre-change catalog compatibility. Keep v3 frozen at 2_000_000_000; v4/v5 carry their own constants.

A re-created Manager bean re-runs init within the same context lifecycle while the AccountAsset lane stays attached; skip the attach when the lane is already installed instead of failing the second initialization.

Exempt the deliberately cache-less account-asset snapshot participant from the second-cache check and attach the lane in the exact-27 startup scenario the way Manager.init does.

The format-v2 expectation was pinned before the commitment codec was deliberately reverted to format v1 for compatibility; restore the v1 root the code has always produced.

The v3 serving-index builder held the coordinator monitor across plan and RocksDB append I/O every 10-block batch, so query threads blocked for seconds in pinIndexed and 9.1% of historical-query requests spiked to 2.3-4.0s. Apply the P04 read-side pattern to the build side: the monitor only swaps the pending range into an in-flight capture and publishes the immutable generation, while plan/append I/O runs lock-free; MutableIndex pin captures its lease on volatile state and a request-owned snapshot. A concurrent close fails an in-flight flush closed instead of blocking.

cubic-dev-ai · 2026-09-20T11:39:46Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-20T12:32:25Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-20T13:49:40Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-20T15:38:23Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-20T16:14:26Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-21T05:05:39Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

cubic-dev-ai · 2026-09-21T06:10:30Z

This PR is large and would use a significant portion of your monthly review quota. Comment @cubic-dev-ai review this to confirm that you want cubic to review it.

bladehan1 added 30 commits September 20, 2026 14:50

feat(chainbase): add archive state collector

5f7c663

Capture canonical block metadata and materialize reverse diffs from completed block snapshots. Classify archive state stores explicitly and project account-asset companion mutations before root merge.

feat(chainbase): persist archive block history

86efea7

Add deterministic history body and index records, ordered durable commit markers, and bounded asynchronous persistence. Gate checkpoint advancement on committed archive history and cover crash, corruption, backpressure, and reorg behavior.

feat(block): add offline block file tools

b8a56b8

feat(block): add offline block replay

500a50b

fix(block): flush replay snapshot tail

78745bf

feat(block): add resumable HTTP export

a82975e

fix(block): parse large HTTP block ranges

303c28b

feat(block): replay consecutive block chunks

c12bb71

feat(chainbase): persist archive recovery state

d070a47

Persist bounded restart checkpoints, truncation recovery, H/C/D/R progress authorities, and immutable serving generations. Pin committed history resources and fail closed on identity or durability drift.

feat(chainbase): pin archive engine generations

5ddb88a

Add durable engine identities and native LevelDB/RocksDB snapshot leases. Assemble exact Store generations under the SnapshotManager barrier and release partial acquisitions on authority drift.

feat(chainbase): publish archive recovery state

5702e77

Apply business mutations with participant progress in native sync batches. Freshly validate H/C/D authorities under the archive barrier and advance reader visibility only after exact convergence.

feat(chainbase): coordinate archive recovery

e20ce4f

Unify LevelDB and RocksDB participants behind one recovery contract. Persist canonical mutation plans and bind checkpoint, participant, and reader progress to the same digest.

feat(chainbase): capture archive mutations

b45e7ea

Collect exact ordinary and AccountAsset post-state without Store scans. Enforce target identity, bounded ownership, one-shot sealing, abort, and terminal payload release.

test(chainbase): cover archive capture recovery

cb3dd60

Drive an exact-27 capture batch through mixed native participants, inject a partial apply failure, and verify durable-plan replay, digest propagation, ownership, and restart convergence.

test(chainbase): cover consecutive archive targets

6a64182

Apply two capture-produced targets through exact-27 participants. Verify per-target plan digests, overwrite and delete semantics, partial replay boundaries, reader publication, and restart convergence.

refactor(chainbase): exclude abi from archive state

cb7c2d0

Classify ABI as mutable auxiliary metadata outside the versioned state participant set while retaining an explicit tombstone classification.

test(chainbase): cover archive projection ownership

aaae524

Exercise proposal-bound projection, layer-to-flush ownership, durable marker sealing, retry and reorg boundaries, exact participant coverage, and ABI exclusion.

test(chainbase): cover durable flush sealing

966c216

Verify direct and asynchronous receipt authority, disabled behavior, and retry ordering across durable wait and receipt failures without duplicate history submission.

test(chainbase): assert legacy asset freeze

d0fd12e

feat(chainbase): bind archive exact-26 scope

ac6d5af

test(chainbase): reject stale archive scopes

fc70854

refactor(chainbase): centralize archive scope identity

f73a2cf

feat(chainbase): bind archive progress scope

dc9bee4

feat(chainbase): bind serving generation scope

87c2223

feat(chainbase): add P66 asset archive codec

4ade822

refactor(chainbase): unify P66 asset projection codec

acb5078

feat(chainbase): bind p66 phase to mutation plans

72223e4

perf(chainbase): pin path-state l0 index and isolate compaction io

c976d60

Merge remote-tracking branch 'origin/develop' into feature/state_archive

19dac1f

# Conflicts: # actuator/src/main/java/org/tron/core/vm/repository/RepositoryImpl.java

fix(chainbase): satisfy math and checkstyle checks

fe7347a

bladehan1 added 8 commits September 20, 2026 16:14

fix(vm): tolerate null store factory in repository root

c8199b7

fix(chainbase): tolerate repeated manager asset-lane attachment

ef7c09c

A re-created Manager bean re-runs init within the same context lifecycle while the AccountAsset lane stays attached; skip the attach when the lane is already installed instead of failing the second initialization.

test(chainbase): adapt snapshot checks for the account-asset lane

5e7d4e7

Exempt the deliberately cache-less account-asset snapshot participant from the second-cache check and attach the lane in the exact-27 startup scenario the way Manager.init does.

test(chainbase): restore format-v1 path engine root vector

a452676

The format-v2 expectation was pinned before the commitment codec was deliberately reverted to format v1 for compatibility; restore the v1 root the code has always produced.

test(net): verify pushblock with explicit sync source

7000391

test(chainbase): refreeze v3 format baselines at restored target

3835c1d

test(framework): log started test events for hang diagnosis

443bc1b

bladehan1 added 2 commits September 20, 2026 23:37

fix(chainbase): surface serving owner native errors to waiters

c5ab080

test(framework): gate leveldb-backed archive tests on arm64

539bdc2

test(chainbase): accept rocksdb 9 bloom filter options spelling

0721f34

refactor(chainbase): remove v4 and v3 append-file archive formats

fffec38

bladehan1 changed the title ~~feat(chainbase): add state archive with historical query serving~~ feat(archive): add state archive with historical query serving Sep 21, 2026

test(framework): align append file mapping assertion with v5 only format

b76ddef

Navigation Menu

feat(archive): add state archive with historical query serving - #55

feat(archive): add state archive with historical query serving#55
bladehan1 wants to merge 191 commits into
developfrom
feature/state_archive

bladehan1 commented Sep 20, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 20, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bladehan1 commented Sep 20, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

coderabbitai Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

github-actions Bot commented Sep 20, 2026

❌ Math Usage Detection Results

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 20, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bladehan1 commented Sep 20, 2026 •

edited by cubic-dev-ai Bot

Loading

coderabbitai Bot commented Sep 20, 2026 •

edited

Loading