Skip to content

fix(settlement): add retry columns and bounded retry policy - #1547

Merged
greatest0fallt1me merged 1 commit into
CalloraOrg:mainfrom
0darkside0rbit-star:fix/settlement-retry-schema
Oct 21, 2026
Merged

greatest0fallt1me merged 1 commit into
CalloraOrg:mainfrom
0darkside0rbit-star:fix/settlement-retry-schema

Conversation

@0darkside0rbit-star

Copy link
Copy Markdown
Contributor

closes #1497

PR description

Fixes #1497 by adding the missing settlement retry schema and enforcing bounded retry transitions.

Summary

This change adds the missing retry_after, retry_count, and last_error columns to the settlements table, restores a retryable status in the CHECK constraint, and adds a partial index for the pending retry query path. It also moves retry exhaustion and retry scheduling into the store so the settlement state transition is atomic and cannot violate the ledger constraints.

Why this fixes the bug

Previously, the store code selected retry_after / retry_count and scheduled rows as retryable, but the database schema never created those columns or allowed the status. That caused PostgreSQL errors like:

  • column "retry_after" does not exist
  • new row for relation "settlements" violates check constraint

This patch makes the schema and runtime contract match again, so reconciliation can safely re-check tx_failed_too_early settlements without crashing.

Changes

  • Added migration: 0026_settlement_retry_policy.sql
  • Added rollback: 0026_settlement_retry_policy.down.sql
  • Added retry columns and partial index:
    • retry_after TIMESTAMP NULL
    • retry_count INTEGER NOT NULL DEFAULT 0
    • last_error TEXT NULL
    • status CHECK includes retryable
    • idx_settlements_retry_after on settlements(retry_after) where status = 'retryable'
  • Bounded retry policy:
    • default SETTLEMENT_MAX_RETRIES = 5
    • exponential backoff with equal jitter
    • cap at 30 minutes
    • when retry limit is reached, row transitions to failed with last_error set in the same SQL update
  • Store contract tightened:
    • scheduleRetry() only applies to pending / retryable rows
    • returns whether a row actually changed
  • Updated store docs and added a real Postgres schema-contract test covering:
    • migration compatibility
    • retry scheduling
    • retry eligibility
    • max-retry exhaustion
    • rollback behavior
    • pending query EXPLAIN plan

Validation

Verified with:

  • npx jest --runInBand --forceExit settlementStore.test.ts settlementStatusSyncJob.test.ts tests/integration/settlementStoreSchema.test.ts

Result:

  • 3 suites passed
  • 36 tests passed
  • 0 failed

Notes

This is a safe forward migration for populated tables and keeps the pending settlement query fast via the new partial retry index.

@drips-wave

drips-wave Bot commented Oct 20, 2026

Copy link
Copy Markdown

@0darkside0rbit-star Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@greatest0fallt1me
greatest0fallt1me merged commit 355f8ab into CalloraOrg:main Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the missing retry columns and 'retryable' status so settlement reconciliation stops crashing

2 participants