[abdulrahmanj383-creator]

Validates Horizon responses before JSON parsing, records the remaining-quota gauge, and retries 429 responses using Retry-After or bounded exponential backoff with jitter. Persistent 429 responses propagate as errors so they cannot be mistaken for an empty pool list.

Closes #348

[drips-wave]

@abdulrahmanj383-creator Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[abdulrahmanj383-creator]

@Miracle656 The three failing AMM ingestion tests now mock successful Horizon responses (ok: true, HTTP 200), matching the response contract used by the rate-limit helper. Please review and merge when ready.

[Miracle656]

Merged as 9e9a367. You diagnosed the right failure. A rate-limited Horizon response is not an error from the caller's point of view — it is a well-formed body with no _embedded.records — so the AMM adapter read it as "this pair has no pools" and the venue went quiet until a later cycle happened to fall outside the window. Nothing in the logs, nothing in the metrics. That is a worse bug than a crash.

Three things in the implementation are better than they had to be:

Retry-After wins over your own backoff. Horizon knows when its window resets; guessing is strictly worse. And you handle both forms the header can take — delta-seconds and an HTTP date — rather than only the common one.

Full jitter, not fixed backoff.

const exponential = Math.min(1_000 * 2 ** attempt, 30_000)
return Math.round(exponential * (0.75 + Math.random() * 0.5))

Four ingesters poll on the same interval. With a fixed backoff they get rate-limited together, sleep together, and retry together — rate-limiting each other again. Jitter is what breaks that, and it is the part people leave out.

A persistent 429 is raised, not logged. HorizonRateLimitError propagating up means the runner counts it in ingest_errors_total{category:'rate_limit'} and backs the whole cycle off, instead of the adapter swallowing it and the loop charging straight back in on the next tick. Exporting horizon_rate_limit_remaining from X-RateLimit-Remaining is the right complement: how close an instance is running to the limit is visible before anything starts failing.

Merged locally; two conflicts, both against work landed since you cut the branch.

---

The fix on top: the helper used a bare fetch

const response = await fetch(url)

main had moved both AMM call sites to:

const response = await fetchWithTimeout(url, { signal })

which supplies two things your version drops. fetchWithTimeout applies config.http.timeoutMs (HTTP_TIMEOUT_MS, default 10s) via AbortSignal.timeout, and — since #471 landed this afternoon — forwards the cycle deadline's abort signal so an abandoned ingest cycle actually stops.

Taking the branch verbatim would have reverted both: an unbounded request, inside a loop whose entire purpose is to have an upper bound, in the module that exists precisely because a slow Horizon is the failure being handled. fetchHorizonJson now takes an options object:

export async function fetchHorizonJson<T>(
  url: string,
  network: NetworkName,
  options: FetchHorizonJsonOptions = {},   // { attempts?, signal? }
): Promise<T>

and the signal goes to every request and to the backoff wait:

await sleep(retryDelay(response, attempt), signal)
// The deadline fired while we were waiting out the window: stop rather than
// spend another request the cycle no longer wants.
if (signal?.aborted) throw new HorizonRateLimitError(url)

That second part matters more than it looks. Your retry can wait up to 30 seconds; without an abortable sleep an abandoned cycle sits in setTimeout long after it has been told to stop, which is exactly the stall #471 was built to bound. The sleep helper removes its own abort listener so a long-lived signal does not accumulate them.

I also kept main's structured log.error({ pair: pair.pairKey, err }, …) over the branch's console.error — src/__tests__/logger.test.ts rejects console.* in non-test source, because the house logger pipes every line through redactSecrets.

Unrelated, found on the way

tests/readme.test.ts — which I merged a couple of hours ago in #475 — byte-compares README.md against generator output joined with \n. On any Windows checkout git supplies CRLF, so it fails on every line of an identical table:

- | AMM | `AMM` | Liquidity Pool | Stellar native AMM liquidity pools (Horizon) |
+ | AMM | `AMM` | Liquidity Pool | Stellar native AMM liquidity pools (Horizon) |

That is the fifth guard in this repo to hit the same thing, after openapi.json, *.yaml, *.csv and schema.graphql. Fixed both ways: README.md is pinned to LF in .gitattributes, and the test normalises line endings as it reads, so a contributor whose git config differs is not blocked by a diff that is only line endings. My bug, not yours — flagging it here because it is in the same commit.

129 test files / 1,553 tests pass. tsc --noEmit clean.