Skip to content

perf(whitelist): per-address persistent membership with paged index - #1447

Open
Tekh134 wants to merge 14 commits into
CalloraOrg:mainfrom
Tekh134:perf/whitelist-persistent-membership
Open

Tekh134 wants to merge 14 commits into
CalloraOrg:mainfrom
Tekh134:perf/whitelist-persistent-membership

Conversation

@Tekh134

@Tekh134 Tekh134 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Closes #1421

Summary

Moves the whitelist from a single instance-storage Vec to per-address persistent entries, with paginated enumeration.

Changes

New storage

  • StorageKey::Member(Address) -> bool in persistent storage, with TTL extended on write and on is_whitelisted
  • StorageKey::MemberCount -> u32 in instance storage
  • StorageKey::ClearGeneration -> u64 in instance storage for O(1) clear-all invalidation
  • StorageKey::MemberGeneration(Address) -> u64 in persistent storage
  • StorageKey::WhitelistIndex -> Vec in instance storage for enumeration only

Paginated enumeration

  • get_whitelist_page(start, limit) backed by the WhitelistIndex enumeration index, with MAX_PAGE cap (100)
  • get_whitelist() kept but capped at MAX_PAGE and marked deprecated

clear_all (bounded)

  • clear_page(limit) removes up to limit members per call; repeated calls fully clear the whitelist
  • ClearGeneration counter invalidates all members in O(1) when empty

Migration

  • Admin-only migrate_list(limit) drains the legacy WhitelistList Vec into persistent entries in chunks
  • is_whitelisted consults both stores until migration completes
  • Legacy key removed once fully drained

Events

  • Dropped the duplicate legacy admin_nominated publish — set_admin now emits exactly one admin_nominated event
  • address_added and address_removed keep their current topic shape

Test plan coverage (all rows)

  1. Add then is_whitelisted -> true; count +1
  2. Add a duplicate -> AddressAlreadyInWhitelist
  3. Remove a non-member -> AddressNotInWhitelist
  4. 1 000 members, is_whitelisted CPU flat
  5. get_whitelist_page walk -> every member exactly once
  6. Legacy Vec 300 entries, migrate_list(100) x3 -> all migrated; legacy key removed
  7. clear_page -> is_whitelisted false for all; bounded cost
  8. set_admin -> exactly one admin_nominated event
  9. Member TTL extended on add and on read

Cost budget (is_whitelisted)

  • 1 member: 2 persistent reads + 1 instance read (O(1))
  • 100 members: same O(1) reads
  • 1,000 members: same O(1) reads

Previously: O(n) Vec scan over instance storage, growing linearly.

Migration docs

  • docs/WHITELIST_MIGRATION.md documents the storage diff, pre-migration checklist, migration procedure, dual-store reads, deprecated API, bounded clear-all, API reference, error codes, and security considerations

Move whitelist membership from a single instance-storage Vec to per-address
persistent entries for O(1) is_whitelisted, with paginated enumeration via
get_whitelist_page. Adds bounded clear_page, admin-only migrate_list to drain
the legacy WhitelistList Vec in chunks, and TTL extension on add/read.
Drops the duplicate legacy admin_nominated publish. Deprecates get_whitelist
in favor of get_whitelist_page.

Closes CalloraOrg#1421
@Tekh134
Tekh134 force-pushed the perf/whitelist-persistent-membership branch from 8e26964 to 50a6f72 Compare October 20, 2026 03:58
@drips-wave

drips-wave Bot commented Oct 20, 2026

Copy link
Copy Markdown

@Tekh134 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

jerrytekh55 added 13 commits October 20, 2026 05:57
Revert get_whitelist to original signature (no caller param, returns Vec<Address>,
reads from WhitelistList) to maintain backward compatibility with existing tests.
Mark as deprecated and cap at MAX_PAGE. get_whitelist_page remains the new
paginated API requiring auth and returning Result.
…n and clear

Fix clear_page_inner to keep WhitelistList in sync with WhitelistIndex on
partial clears so deprecated get_whitelist returns consistent data.

Fix migrate_list to rebuild WhitelistIndex from remaining entries after
partial migration, and to clear WhitelistIndex when migration fully drains
the legacy Vec.
Fix cargo fmt failure caused by inner attribute #![no_std] appearing
after outer doc comments. Also fix #[config(test)] typo -> #[cfg(test)]
which would break cargo test.
Fix compile error: enumerate() index is usize but to_clear is u32.
Cast i to u32 before comparison.
Fix compile error: get_ttl is on Storage, not PersistentStorage.
Add Eq to StorageKey derive for use as storage key map key.
Only sync WhitelistList from WhitelistIndex when entries were actually
cleared (to_clear > 0), to avoid wiping legacy-only entries.
Fix migrate_list to rebuild WhitelistIndex from existing index + remaining
entries + newly migrated addresses, so get_whitelist_page returns all
members after migration. Don't reset MemberCount to 0 on migration
completion.
Always set WhitelistList to remaining after clear_page_inner so clear_all
clears WhitelistList-only entries too (e.g. seed_member in tests).
Fix all compilation errors in whitelist contract:
- Address comparison: a != &address -> a != address
- usize/u32 comparison in clear_page_inner
- &remaining.len() as u32 -> &(remaining.len() as u32)
- Vec::new(env) -> Vec::new(&env) where env is owned
- MAX_PAGE as usize -> MAX_PAGE for u32 comparison
Fix checkpoint/src/lib.rs compilation errors blocking cargo test --workspace:

- validate_before_upgrade takes 6 args, removed extra probe keys argument
- finalize_migration takes 4 args, removed extra probe keys argument
- Added IntoVal import for StorageKey::Admin.into_val(env)
- Replaced Address::from_contract_id() with env.current_contract_address()
  (from_contract_id is private; current_contract_address() returns Address)
- Fixed Val::VOID -> Val::VOID.into() for try_into_val unwrap_or_else
…e_keys

try_into_val takes &self, so current_contract_address() result needs &.
This was the remaining compilation error blocking cargo test --workspace.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move the whitelist from a single instance-storage Vec to per-address persistent entries

1 participant