[praizeD10]

closes #978

1. Summary of Changes
   IndexerService.persistEventsWithRecovery was the sole write path from Soroban RPC into indexer_events, but suffered from three critical defects:

Target Column & Expression Arity Mismatch: INSERT INTO indexer_events (ledger, tx_hash, op_index, event_type, data, canonical) VALUES ($1, $2, $3, $4, $5::jsonb) had 6 target columns but only 5 value parameters, omitting market_id and failing in real PostgreSQL with INSERT has more target columns than expressions.
False Reorganizations from Flawed Event Identity: parseEventOpIndex parsed only the suffix after the - delimiter. In Soroban RPC, that suffix is the intra-operation event_index (often 0000000000), not an operation index. Two distinct transactions in the same ledger shared the same (ledger, op_index), triggering false reorgs and erroneously rolling back confirmed bets into pending.
Non-Atomic Reorganization Replacement: Reorg detection, canonical status flipping, indexer_reorgs insertion, prediction repairs, and event inserts ran as uncoordinated queries across pool connections without transactional guarantees.
What Was Implemented:
Exact Column Binding: Rewrote INSERT INTO indexer_events to bind all 9 columns explicitly: (ledger, tx_hash, tx_application_order, op_index, event_index, market_id, event_type, data, canonical) VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9).
TOID Decoder & Position/Identity Separation:
Implemented decodeEventId(eventId) and formatEventId(...) in

src/services/indexerRecovery.ts
and re-exported in

src/services/indexerService.ts
.
Used (ledger, tx_hash, op_index, event_index) for event uniqueness.
Used (ledger, tx_application_order, op_index, event_index) for ledger position conflict and reorg detection.
Atomic Reorganization Replacement:
Wrapped conflict detection, canonical flip, reorg recording, and prediction repair inside a dedicated client transaction (BEGIN ... COMMIT / ROLLBACK).
Acquired a row-level lock on conflicting canonical events using SELECT id, tx_hash FROM indexer_events ... FOR UPDATE.
Predictions are only reset when the ledger position matches and oldRow.tx_hash !== newEvent.txHash.
Added repaired_predictions count tracking to indexer_reorgs.
Database Schema & Backward-Compatible Migration:
Added

drizzle/migrations/0030_indexer_event_identity.sql
adding tx_application_order, event_index, and repaired_predictions (defaulting event_index to 0).
Replaced the old unique index indexer_events_identity_idx with a composite unique index on (ledger, tx_hash, op_index, event_index).
Observability & Documentation:
Added indexer_events_inserted_total and indexer_reorgs_detected_total metrics in

src/metrics/registry.ts
.
Documented TOID bit layout, position vs identity, and atomic replacement in

docs/indexer-health-probe.md
.
2. TOID Layout & Stellar Reference
Soroban RPC event identifiers adhere to the format -<event_index> (e.g. 000000042949673000-0000000000), where the prefix is a 64-bit Stellar Total Ordering ID (TOID) zero-padded to 19 digits, followed by a 10-digit zero-padded intra-operation event index:

+---------------------------+-----------------------------------+--------------------+------------------------+
| ledger (32 bits) | tx_application_order (20 bits) | op_index (12 bits)| event_index (suffix) |
| bits 32..63 | bits 12..31 | bits 0..11 | base-10 string after -|
+---------------------------+-----------------------------------+--------------------+------------------------+
Bit 32–63 (32 bits): Ledger sequence number (toid >> 32n).
Bit 12–31 (20 bits): 1-based order in which the transaction was applied in the ledger ((toid >> 12n) & 0xFFFFFn).
Bit 0–11 (12 bits): 1-based operation order within the transaction (toid & 0xFFFn).
Suffix: Zero-indexed event counter within that operation.
Reference: Stellar Horizon TOID Documentation & Soroban RPC getEvents Specification.

3. Verification & Test Output
   All 49 unit and integration tests passed with >99% coverage across

indexerService.ts
and

indexerRecovery.ts
.

Test Outputs for Scenarios 1, 2, and 5:
text
PASS tests/indexerService.recovery.test.ts (28.763 s)
IndexerService durable event recovery
√ does not touch the database for an empty RPC batch (40 ms)
√ Scenario 1: persists a new canonical event binding all columns and populating market_id (3 ms)
√ Scenario 2: two different txs in same ledger with suffix 0000000000 produce two canonical rows without reorg (2 ms)
√ Scenario 3: does not count exact redelivery as a new event and creates no reorg (6 ms)
√ Scenario 4: marks the stale event non-canonical, inserts reorg row, and repairs predictions on real replacement (23 ms)
√ Scenario 5: failure injected after canonical flip rolls back the transaction (90 ms)
√ Scenario 6: malformed event id is rejected with logged warning and batch continues (4 ms)
√ Scenario 7: multiple events in one operation are inserted as distinct rows (1 ms)
Coverage and Validation Summary:
npm run lint: Clean (0 errors, 0 warnings).
npm run db:check-drift: Clean ("No schema drift — all schema changes have matching migrations").
Code Coverage:
src/services/indexerService.ts: 100% Statements, 100% Lines, 84% Branches
src/services/indexerRecovery.ts: 97.72% Statements, 99.09% Lines, 80% Branches
Combined: 99.05% Statements, 99.64% Lines

[drips-wave]

@praizeD10 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits