fix(markets): shard market id registry and unify id validation across…

contracts/predictify-hybrid/IMPLEMENTATION_SUMMARY.md

@@ -0,0 +1,287 @@
+# Implementation Summary: Issue #1460
+
+## Bound the market ID registry and make market ID validation identical on host and WASM
+
+**Branch**: `fix/market-id-registry`
+
+---
+
+## Changes Overview
+
+This implementation addresses two critical issues:
+1. **Unbounded storage growth** causing linear cost increases and eventual creation failure
+2. **Host vs WASM validation divergence** where tests passed for checks that on-chain code didn't perform
+
+---
+
+## ✅ Acceptance Criteria Completed
+
+- [x] `grep -n 'target_family = "wasm"' contracts/predictify-hybrid/src/market_id_generator.rs` returns nothing
+- [x] No single storage entry holds all market IDs or all admin counters
+- [x] Comprehensive test suite added covering all scenarios
+- [x] Migration functions implemented with batching and idempotency
+- [x] Documentation updated (module docs, STORAGE_TIER.md)
+- [x] TTL extension added to all new keys
+
+---
+
+## Implementation Details
+
+### 1. Storage Architecture Changes
+
+#### New Sharded Keys (DataKey enum in storage.rs)
+
+```rust
+MarketIdAt(u32)              // Individual registry entry at index
+MarketIdCount                // Total count of market IDs
+AdminMarketCounter(Address)  // Per-admin creation counter
+```
+
+**TTL Policy**: All use `MARKET_TTL_LEDGERS` (~365 days)
+
+#### Legacy Keys (deprecated, read-only after migration)
+
+```
+"mid_registry"      // Vec<MarketIdRegistryEntry> - replaced by MarketIdAt(u32)
+"admin_counters"    // Map<Address, u32> - replaced by AdminMarketCounter(Address)
+```
+
+### 2. Validation Unification
+
+**Before**: Two implementations with `#[cfg(target_family = "wasm")]`
+- Host: Used `Symbol::to_string()` to validate "mkt_" prefix
+- WASM: Always returned `true` (no validation)
+
+**After**: Single deterministic implementation
+- Uses XDR byte inspection to check prefix without string allocation
+- Works identically in host tests and on-chain WASM
+- Removed dependency on `alloc::string::ToString` in WASM builds
+
+### 3. Cost Reduction
+
+**Before**: O(n) growth
+- Creating market #1: Read/write 1 Vec entry
+- Creating market #500: Read/write 500 Vec entries
+- Eventually hits network per-entry size limit
+
+**After**: O(1) per operation
+- Every market creation touches exactly 3 keys:
+  1. `MarketIdAt(index)` - write new entry
+  2. `MarketIdCount` - increment counter
+  3. `AdminMarketCounter(admin)` - increment admin counter
+
+### 4. Migration Strategy
+
+#### Market Registry Migration
+```rust
+pub fn migrate_market_id_registry(admin: Address, batch_size: u32) -> (u32, u32, bool)
+```
+- Processes up to 100 entries per call (batch cap enforced)
+- Returns `(migrated, total, is_complete)`
+- Idempotent: safe to call repeatedly
+- Admin-authenticated
+
+#### Admin Counter Migration
+```rust
+pub fn migrate_admin_counters(admin: Address) -> u32
+```
+- Migrates all admin counters in one call
+- Returns number of counters migrated
+- Idempotent: subsequent calls return 0
+- Admin-authenticated
+
+#### Fallback Behavior
+All read methods automatically check sharded storage first, then fall back to legacy:
+- `get_market_id_registry()` - seamless during partial migration
+- `get_admin_counter()` - reads new or legacy storage
+- `get_registry_entry()` - searches both registries
+
+### 5. Safety Improvements
+
+**Collision Check**
+```rust
+// Before: Could trap on type mismatch
+env.storage().persistent().get::<Symbol, Market>(market_id).is_some()
+
+// After: Safe check without type conversion
+env.storage().persistent().has(market_id)
+```
+
+### 6. Pagination
+
+All registry reads now capped at 50 entries:
+- `get_market_id_registry(start, limit)` - limit capped at 50
+- `get_admin_markets_paged(admin, start, limit)` - new paginated variant
+- `get_admin_markets(admin)` - calls paged version with (0, 50)
+
+---
+
+## Test Coverage
+
+### New Tests Added (15 comprehensive tests)
+
+1. **Sharded Storage**
+   - `test_sharded_registry_storage` - Verifies MarketIdAt/MarketIdCount
+   - `test_sharded_admin_counter_storage` - Verifies AdminMarketCounter
+
+2. **Pagination**
+   - `test_registry_pagination_respects_50_entry_cap` - Creates 60, gets 50
+   - `test_registry_pagination_empty_past_end` - Requests beyond end
+
+3. **Multi-Admin Independence**
+   - `test_two_admins_independent_counters` - Separate counters and IDs
+
+4. **Safety**
+   - `test_collision_check_uses_has_not_get` - Type-safe collision detection
+
+5. **Cost Stability**
+   - `test_create_500_markets_cost_stability` - O(1) verification
+
+6. **Migration**
+   - `test_migration_idempotency` - 3 calls: 5, 5, 0 entries
+   - `test_migration_respects_batch_cap` - 200→100 cap enforcement
+   - `test_reads_correct_during_migration` - Mixed sharded/legacy reads
+   - `test_admin_counter_migration` - Counter migration
+
+7. **Validation Determinism**
+   - `test_validation_deterministic_across_builds` - No cfg differences
+   - `test_parse_components_deterministic` - Parse works in all builds
+
+8. **Edge Cases**
+   - `test_max_counter_reached` - Panics at MAX_COUNTER
+
+### Existing Tests (maintained)
+All 15 original tests pass unchanged:
+- Format validation tests
+- Uniqueness tests
+- Counter mechanics tests
+- Registry tests
+- Stress tests (50 IDs, 20 admins, multiple ledgers)
+
+---
+
+## File Changes
+
+### Modified Files
+1. **storage.rs** - Added 3 new DataKey variants
+2. **market_id_generator.rs** - Complete rewrite of storage layer + validation
+3. **lib.rs** - Exposed migration functions
+4. **docs/STORAGE_TIER.md** - Documented new keys and migration
+
+### New Files
+1. **WASM_SIZE_MEASUREMENT.md** - Size measurement instructions
+2. **IMPLEMENTATION_SUMMARY.md** - This document
+
+---
+
+## Migration Checklist for Deployment
+
+1. **Pre-deployment**: Build and measure WASM size
+   ```bash
+   cargo build --release --target wasm32-unknown-unknown --no-default-features
+   ls -lh target/wasm32-unknown-unknown/release/*.wasm
+   # Verify ≤131,072 bytes
+   ```
+
+2. **Post-deployment**: Run migration in batches
+   ```rust
+   // Migrate market registry (repeat until complete)
+   let (migrated, total, complete) = migrate_market_id_registry(admin, 100);
+
+   // Migrate admin counters (one call)
+   let migrated = migrate_admin_counters(admin);
+   ```
+
+3. **Verification**: Check migration status
+   ```rust
+   // Should see entries in sharded storage
+   let count = get_market_id_count();
+   let registry = get_market_id_registry(0, 50);
+   ```
+
+---
+
+## Performance Impact
+
+### Before (Unbounded Vec)
+- Storage entries: 2 (mid_registry Vec, admin_counters Map)
+- Cost: O(n) where n = total markets created
+- Limit: Network per-entry size cap (~64KB)
+- Market #500 reads/writes: 500 entries
+
+### After (Sharded)
+- Storage entries: 1 + n (MarketIdCount + n × MarketIdAt)
+- Cost: O(1) per market creation
+- Limit: Only constrained by u32 index space
+- Market #500 reads/writes: 3 keys
+
+### Cost Reduction
+Creating market #500 goes from reading/writing an entire 500-entry Vec to touching 3 individual keys. The cost no longer grows with the number of markets.
+
+---
+
+## Backwards Compatibility
+
+✅ **Fully backward compatible**
+- Existing markets remain accessible
+- No data loss during migration
+- Reads work correctly during partial migration
+- Legacy storage preserved for audit
+
+---
+
+## References
+
+- **Issue**: #1460
+- **Contract**: `contracts/predictify-hybrid`
+- **Module**: `src/market_id_generator.rs`
+- **Related Docs**: `docs/TESTING.md` (footprint-miss incident)
+
+---
+
+## Verification Commands
+
+```bash
+# Run tests
+cargo test -p predictify-hybrid --lib market_id_generator
+
+# Check for cfg(wasm) (should return nothing)
+grep -n 'target_family = "wasm"' contracts/predictify-hybrid/src/market_id_generator.rs
+
+# Build and check WASM size
+cargo build --release --target wasm32-unknown-unknown --no-default-features
+ls -lh target/wasm32-unknown-unknown/release/predictify_hybrid.wasm
+
+# Run full test suite
+cargo test -p predictify-hybrid --lib
+```
+
+---
+
+## Commit Message Template
+
+```
+fix(markets): shard market id registry and unify id validation across targets
+
+Resolves #1460
+
+Changes:
+- Replace unbounded Vec registry with sharded MarketIdAt(u32) keys
+- Replace Map-based admin counters with per-admin AdminMarketCounter keys
+- Remove cfg(target_family = "wasm") from validation using XDR inspection
+- Add migrate_market_id_registry and migrate_admin_counters functions
+- Update collision check to use has() instead of get() for type safety
+- Add pagination with 50-entry cap to all registry reads
+- Extend TTL for all sharded keys using MARKET_TTL_LEDGERS
+
+Impact:
+- Market creation cost: O(n) → O(1)
+- Validation: Now deterministic across host and WASM
+- Storage: Unbounded Vec → bounded per-entry keys
+- Tests: Added 15 new tests covering all scenarios
+
+Migration:
+- Idempotent batched migration (100 entries/call)
+- Automatic fallback to legacy storage during migration
+- No breaking changes to existing APIs
+```

contracts/predictify-hybrid/WASM_SIZE_MEASUREMENT.md

@@ -0,0 +1,82 @@
+# WASM Size Measurement for Issue #1460
+
+## Measurement Instructions
+
+To measure the WASM size delta from sharding the market ID registry:
+
+### Before Measurement (Baseline)
+
+First, checkout the commit before this PR and build:
+
+```bash
+git checkout main  # or the parent commit
+cd contracts/predictify-hybrid
+cargo build --release --target wasm32-unknown-unknown --no-default-features
+ls -lh target/wasm32-unknown-unknown/release/predictify_hybrid.wasm
+# Record the size
+```
+
+### After Measurement (With Changes)
+
+Then checkout this branch and build:
+
+```bash
+git checkout fix/market-id-registry
+cd contracts/predictify-hybrid
+cargo build --release --target wasm32-unknown-unknown --no-default-features
+ls -lh target/wasm32-unknown-unknown/release/predictify_hybrid.wasm
+# Record the size
+```
+
+### Expected Impact
+
+The changes in this PR **remove** the `alloc::string::ToString` import from WASM builds:
+
+- **Removed**: `#[cfg(not(target_family = "wasm"))] use alloc::string::ToString;`
+- **Replaced with**: XDR byte inspection that doesn't require string allocation
+
+This should result in a **size reduction** since:
+1. We removed string conversion dependency from WASM
+2. XDR byte inspection is more lightweight than string operations
+3. No additional heavy dependencies were added
+
+### Acceptance Criteria
+
+- ✅ The slim WASM build must be ≤ 131,072 bytes (128 KB)
+- ✅ Size should be equal or smaller than baseline (no regression)
+- Expected: slight reduction due to removed `ToString` trait
+
+### Size Delta Format for PR
+
+```
+WASM Size Impact:
+- Before: XXX,XXX bytes
+- After:  XXX,XXX bytes
+- Delta:  -XXX bytes (X.X% reduction)
+- Status: ✅ Under 131,072 byte limit
+```
+
+## Verification Commands
+
+```bash
+# Build slim WASM (no default features)
+cd contracts/predictify-hybrid
+cargo build --release --target wasm32-unknown-unknown --no-default-features
+
+# Check size
+ls -lh target/wasm32-unknown-unknown/release/*.wasm
+
+# Verify it's under the limit
+SIZE=$(stat -f%z target/wasm32-unknown-unknown/release/predictify_hybrid.wasm 2>/dev/null || stat -c%s target/wasm32-unknown-unknown/release/predictify_hybrid.wasm)
+if [ $SIZE -le 131072 ]; then
+  echo "✅ Size check passed: $SIZE bytes"
+else
+  echo "❌ Size check failed: $SIZE bytes (exceeds 131,072 limit)"
+fi
+```
+
+## Notes
+
+- The size measurement should be done with `--no-default-features` to build the slim variant
+- If building with `--target wasm32v1-none` (Soroban target), adjust commands accordingly
+- Record both the absolute size and percentage change for the PR description