Skip to content

feat: ingest Blend lending pool state, rates, and derived utilisation - #453

Merged
Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
onajidavid87-web:Blend-lending-markets-have-no-price-or-rate-ingestion
Oct 23, 2026
Merged

Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
onajidavid87-web:Blend-lending-markets-have-no-price-or-rate-ingestion

Conversation

@onajidavid87-web

Copy link
Copy Markdown
Contributor

Summary

Adds rate and pool state ingestion for Stellar's lending protocol (Blend):

  • Rate Ingestion: Ingests each Blend pool's reserves, derived utilisation, supply APY, and borrow APY on a schedule per enabled network.
  • Dedicated Schema & Retention: Stores rates in a new lending_rates table via Prisma migration (rather than forcing into price_points), with a 30-day retention prune job.
  • Derived Utilisation: Utilisation is derived from liabilities and supply (calcUtilization), with edge cases ($U \le 0$, $U \ge 1$, BigInt scaling) pinned by unit tests.
  • Fault Isolation: Failed pool reads are skipped gracefully with a log and blend_pool_errors_total metric increment, preventing process crashes.
  • Dual-Network Isolation: Every row explicitly carries its network (mainnet or testnet) without cross-contamination.
  • Contract Addresses: Full 56-character pool and backstop addresses verified against Blend documentation.
  • API & OpenAPI: Added GET /rates and GET /rates/:asset endpoints returning venue: 'blend', documented in openapi.yaml and synced to openapi.json.

Related issue

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Docs
  • Tests
  • CI / tooling

Checklist

  • I have read CONTRIBUTING.md
  • npx tsc --noEmit passes
  • npm run build passes
  • I added / updated tests where relevant
  • I updated docs where relevant

Closes #212

@drips-wave

drips-wave Bot commented Oct 17, 2026

Copy link
Copy Markdown

@onajidavid87-web Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Miracle656 Miracle656 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a rebase. I tried to resolve this myself and stopped, because the conflicts are not the mechanical kind and guessing at them in config for a money-adjacent feature is not a trade I want to make.

What conflicts: .env.example (4 hunks), src/config.ts (3), src/index.ts (3), src/app.ts (2), prisma/schema.prisma (1), plus the two generated OpenAPI files.

Why keep-both does not work here. The config.ts hunks cut through object literals rather than between them. main adds a publisher: { block for the SEP-40 publisher; this PR adds a blend: { block. Both open a property inside the same object, and both closing braces fall outside the conflict region — so concatenating the two sides leaves one block unclosed:

src/config.ts(89,1):  error TS1131: Property or signature expected
src/config.ts(593,1): error TS1005: '}' expected
src/index.ts(308,3):  error TS1005: '}' expected

Resolving that correctly means knowing which block each tail brace belongs to, thirteen times. You have that in your head; I would be inferring it.

One thing I did find while trying: keeping both sides of .env.example produces a duplicate MCP_MAX_PAYMENT_PRICE= — there is a terse copy and a documented one. Keep the documented block when you rebase.

The work itself is wanted, and the timing is good. Blend lending rates are what the earn surface quotes, and a rate we compute from pool state rather than take on trust is the difference between a number we can defend and one we repeat. The shape looks right from the file list — a real migration, retention for the rate history, metrics, and tests for the ingester, the utilisation maths, retention and the route.

After rebasing, three things this repo will check and that have caught PRs today:

  • console.* is forbidden in non-test source; use createLogger
  • src/__tests__/envDrift.test.ts fails if a new process.env read is missing from .env.example — and it now covers scripts/ and examples/ too
  • tests/openapi.test.ts fails if a route is documented but not registered, or registered but not documented

Get npx tsc --noEmit and npx vitest run clean locally and I will take it.

…ingestion

main already ships a lending_rates table (0_init baseline) and
GET /rates/:asset reading it, so the Blend ingester now writes into that
schema instead of creating its own table:

- drop the branch's CREATE TABLE migration; add
  20261023130000_extend_lending_rates_for_blend, which adds the reserve
  state columns (APR, utilisation, totals, asset address) as nullable
  columns and a (network, pool_id, asset, timestamp) index
- insertLendingRates writes venue 'blend' and stores APYs in
  supply_rate / borrow_rate
- keep main's GET /rates/:asset unchanged; add the GET /rates pool
  listing to main's rates.ts and drop the duplicate route from the spec
- move the Blend ingester and rate retention job onto the structured
  logger, give the adapter its README type/description, regenerate
  openapi.json and the README tables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Miracle656 added a commit that referenced this pull request Oct 23, 2026
…closes #212)

A dedicated ingester reads Blend pool reserves over Soroban, derives
utilisation and supply/borrow APR, and stores them in `lending_rates` rather
than `price_points` — a lending rate is not a spot price, and the adapter's
`derive` returns `[]` so nothing leaks into the volume or VWAP paths.
`GET /rates/:asset` serves them, a retention job prunes old rows, and the new
columns ship as a migration.

What it got right, and why I am noting it: the schema change comes with
`20261023130000_extend_lending_rates_for_blend/migration.sql` rather than a
bare `schema.prisma` edit. Since `migrate deploy` replaced `db push` this
morning, a schema edit alone reaches no live database. The migration is
additive (every column nullable), leaves the baseline untouched, sorts after
`0_init`, and its index leads with `network`. The first PR today to need that
workflow got it wrong; this one did not.

Three changes on top of the contribution.

1. **`classification` was missing.** #455 made it a required field on
   `VenueAdapter` about ten minutes earlier, so this did not compile on merge.
   Set to `'primary'`, with a comment recording that it is informational here:
   Blend emits no price points, so it never reaches the net/gross split — and
   it is 'primary' because it holds its own reserves, being the venue a
   DeFindex vault allocates *into* rather than one re-reporting another venue's
   trades. Also registered in `VENUE_CLASSIFICATIONS`, which #455's new drift
   test requires — that guard earned its place on the very next adapter.

2. **The ingester loop was unbounded.** Blend runs outside
   `startVenueAdapterLoop` (it writes `lending_rates`, so the generic
   derive/write cycle does not fit), which meant it also missed the per-cycle
   deadline #471 added this afternoon. A `while (true)` around
   `await ingestBlendPools(network)` is the exact #351 shape: a slow Soroban
   RPC stretches one sweep indefinitely, the loop never crashes, so
   `restartIngester` never fires and the venue goes quiet with nothing raised
   anywhere. Now wrapped in `runCycleWithDeadline`, the same treatment #471
   gave the bespoke snapshot ingester, and `ingestBlendPools` takes the signal
   so an abandoned cycle stops at a pool boundary — one pool is one write unit,
   so nothing is left half-written.

3. Regenerated `clients/typescript/src/schema.d.ts` for the new `/rates`
   response shape; it is byte-compared in CI.

Note for later: `blendAdapter` is exported but deliberately not in
`venueAdapters`, so Blend does not appear in the README's generated venue
table. That is defensible — the table describes price venues — but worth
revisiting if more non-price venues arrive.

134 test files / 1,603 tests pass; tsc clean.

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
@Miracle656
Miracle656 merged commit 74bb7b1 into Divinity-Labs:main Oct 23, 2026
@Miracle656

Copy link
Copy Markdown
Collaborator

Merged as 4dc40a3 — this closes #212.

Nearly 2,000 lines on a path that writes to the money database, and it arrived with five test files, a migration, env documentation and the retention job already wired into src/index.ts. Four things I want to name specifically, because they are the ones that usually go wrong:

The schema change came with a migration. 20261023130000_extend_lending_rates_for_blend/migration.sql, not a bare schema.prisma edit. That matters more than it did last week: since migrate deploy replaced db push this morning, a schema edit on its own reaches no live database — the column simply never exists, every write fails, and the feature looks implemented. The migration is additive (every column nullable), leaves the baseline untouched, sorts after 0_init, and its index leads with network. The first PR today to need this workflow got it wrong; yours did not.

Lending rates go in their own table. derive returning [] with the comment "Blend rates are stored in lending_rates, not price_points" is the right call and the right place to say it. A supply APR is not a spot price, and letting it into price_points would quietly contaminate VWAP, the volume totals and the screener.

ingestBlendPools takes an injectable ingestPool, which is why blendIngester.test.ts can drive the sweep without a Soroban RPC.

New env vars are in .env.example and the envDrift allow-list. That test exists because a config change that works on the author's machine and nowhere else is a specific recurring failure here.

Three changes on top.

1. classification was missing

#455 landed about ten minutes before this and made classification a required field on VenueAdapter, so the merge did not compile:

src/ingest/venues/blend.ts(556,14): error TS2741: Property 'classification' is missing

Pure bad luck on ordering, not a defect. Set to 'primary', with a comment recording that it is informational here — Blend emits no price points, so it never reaches the net/gross split — and that 'primary' is still the honest value because Blend holds its own reserves: it is the venue a DeFindex vault allocates into, not one re-reporting another venue's trades.

I also added it to VENUE_CLASSIFICATIONS, which the drift test I added to #455 requires. That guard caught the very next adapter to arrive, which is about as good a demonstration of its worth as I could ask for.

2. The ingester loop had no upper bound

while (true) {
  try {
    await ingestBlendPools(network)

Blend runs outside startVenueAdapterLoop — reasonably, since it writes lending_rates and the generic derive/write cycle does not fit — but that also means it missed the per-cycle deadline #471 added this afternoon. And this is exactly the #351 shape: a slow Soroban RPC stretches one sweep indefinitely, the loop never crashes, so restartIngester never fires, and Blend goes quiet with nothing raised anywhere. It is the failure mode that is hardest to notice because there is no error to notice.

Wrapped in runCycleWithDeadline, the same treatment #471 gave the other bespoke ingester (src/ingesters/snapshot.ts), and ingestBlendPools now takes the signal so an abandoned cycle stops at a pool boundary rather than only at the end of the sweep — one pool is one write unit, so nothing is left half-written:

for (const poolId of netConfig.blend.pools) {
  if (signal?.aborted) break

Worth knowing as a general rule now: a bespoke ingester needs runCycleWithDeadline explicitly, because only startVenueAdapterLoop applies it for you.

3. Generated client

clients/typescript/src/schema.d.ts regenerated for the new /rates response shape — byte-compared in CI.


One note rather than a request: blendAdapter is exported but deliberately not in venueAdapters, so Blend will not appear in the README's generated venue table. That is defensible — the table describes price venues and Blend is not one — but if more non-price venues arrive it is worth deciding whether the table should grow a section rather than silently omit them.

134 test files / 1,603 tests pass; tsc --noEmit clean.

CHKM001 added a commit to CHKM001/Lens that referenced this pull request Oct 24, 2026
The /rates endpoint (added by Divinity-Labs#453) references
#/components/responses/BadRequest, but that response component was never
defined. openapi-typescript aborts with 'Can't resolve $ref at
#/paths/~1rates/get/responses/400', so 'npm run client:gen' — the first
step of CI's Typecheck & build job — fails before tsc runs, turning the
check red for every open PR.

Define BadRequest with the same minimal shape every other 400 in the
spec uses (description: Invalid parameters) and regenerate the committed
artifacts:

- openapi.json via openapi:gen
- clients/typescript/src/schema.d.ts via client:gen, which had gone
  stale (missing the whole /rates path) precisely because generation
  was aborting, so the 'git diff --exit-code -- clients/typescript'
  gate could never pass either.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
freebuff-web Bot pushed a commit to nonchalanttee22-lgtm/Lens that referenced this pull request Oct 24, 2026
…Labs#453/Divinity-Labs#455 batch

The second main-line merge (0742337) reproduced the exact defects of the
first (d6e8256) and added one more, so this is the same repair re-applied
on the new tip:

- src/metrics.ts: the active_subscriptions gauge is again left unclosed
  with the main-side counters spliced into its options bag — the file
  does not parse. Closed it: gauge with registers: [register], followed
  by the full set of counters.
- src/api/graphql.ts: both parents' import statements were kept again
  (metrics, mercurius, config) plus a new duplicate parseAssetStr /
  getActivePairs pair from this batch. Deduped to one statement each,
  keeping the richer variants.
- .env.example: document BENCH_MODE / BENCH_SUBSCRIBERS_PER_PAIR /
  BENCH_EVENTS_PER_PAIR so the env-drift guard passes on this branch's
  own benchmark script.

Verified on 0742337 + this commit: tsc clean, full suite 1606 passed /
1 skipped, check-dropped-tests reports no case missing against the
queue line (4dc40a3).

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blend lending markets have no price or rate ingestion

2 participants