Repository navigation
fix: classify venues and report net vs gross volume totals - #455
Conversation
|
@onajidavid87-web Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
There was a problem hiding this comment.
Needs a rebase, and two of the conflicts are design decisions rather than merge noise — so they are yours to make, not mine.
Conflicts: src/ingest/venues/types.ts, src/routes/volumes.ts (3 hunks), plus the two generated OpenAPI files.
1. Two names for the same concept. main now has:
/** Kind is descriptive metadata used by API projections, not the runner. */
readonly kind?: VenueKindand this PR introduces:
readonly classification: VenueClassificationSame idea, different name, different type, and yours is required where main's is optional. One of them should survive. I would keep one field and one vocabulary rather than have adapters carrying both — but which one is a call for whoever owns the venue model, and required versus optional is the part that actually changes behaviour: a required field forces every existing adapter to declare itself, which is probably what you want and is also the breaking half.
2. The volume query changed shape underneath you. main's handler reads a single row and derives trade_count, total_volume and a status of no_trades / no_positive_volume / ok. Yours loops over rows accumulating net and gross per venue. Those are different SQL and different response semantics, not two spellings of one thing.
I did not resolve this, deliberately. Picking a side would produce an endpoint that returns numbers — and a wrong volume total is the worst kind of wrong, because it looks entirely plausible and nothing fails. You know which query your net/gross split needs; please rebase onto the current handler and reconcile them.
The feature is worth having. Reporting net against gross is a real distinction: counting both sides of a trade, or the same flow across two venues, inflates a figure people compare across venues. Separating primary from derived venues is the right basis for it.
When you rebase, worth adding a test that pins the difference explicitly — one trade visible on two venues should contribute once to net and twice to gross. That is the case the whole change exists for, and it is the one that would silently regress.
…uoting-the-same-trade main moved /volumes/:asset to a single SQL aggregate (trade-size stats, HHI, status, provenance), so the net/gross split now sits on top of it: - venue_totals also returns a per-venue positive-volume trade count (trade_count_by_venue); net = gross minus aggregator venues - totalVolume/tradeCount stay the net figures; the trade-size statistics and provenance cover every venue - keep classification next to main's kind/type/description on each adapter; classify the new Phoenix (primary) and DeFindex (aggregator) adapters and document them in docs/venues.md - merge the VolumeResponse schema and regenerate openapi.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) `/volumes/:asset` summed every venue, so an aggregator re-reporting a trade that already settled on a primary market counted it twice. Venues now declare `classification: 'primary' | 'aggregator'` on the adapter, and the route reports `totals.net` (primaries only) beside `totals.gross` (everything observed), both labelled, with the per-venue breakdown unchanged. Aquarius and DeFindex are aggregators; SDEX, the native AMM, Soroswap and Phoenix are primary, each with its reasoning recorded. `classification` is a required field, so a new adapter cannot be registered without stating what it is — the same reason `type` and `description` are required. Distribution statistics (median, HHI, status) deliberately stay on gross, and `SUM(volume)::float8` casts outside the aggregate rather than in, which is the right form. Two changes on top of the contribution, both about the same risk: this decides whether a venue's volume reaches a published total, so two records of it must not be able to disagree quietly. 1. **The hand-maintained map outranked the adapter.** `getVenueClassification` consulted `VENUE_CLASSIFICATIONS` first and fell back to the adapter, so a stale map entry silently overrode the declaration the compiler enforces. Inverted: the adapter is the authority, and the map carries the prose. The precedence is now stated in the function's own comment, including why an unregistered source still defaults to `primary` — `price_points.source` carries CEX feeds with no adapter, and those are real independent markets. 2. **Nothing caught drift between the two.** The PR's tests assert named venues (`isAggregatorVenue('aquarius_amm')` and so on), so a new adapter added with no map entry, or with a mismatched one, failed nothing. Four tests now hold the invariant over `venueAdapters` as a whole: every adapter has a matching entry, every adapter has documented reasoning, resolution returns the adapter's own answer, and an unregistered source is never silently treated as an aggregator. Also regenerated `clients/typescript/src/schema.d.ts` for the new `totals` shape — it is byte-compared in CI. 130 test files / 1,571 tests pass; tsc clean. Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
|
Merged as The issue says "every ecosystem volume figure that has ever been wrong has been wrong for exactly this reason", and the fix treats it with that weight. Reporting both numbers and labelling them is better than silently picking one: a caller who wants gross can still have it, and nobody is guessing which they were given. Things that were already right and I left alone:
Two changes on top, both about the same risk. This field decides whether a venue's volume reaches a published total, so two records of it must not be able to disagree quietly. 1. The hand-maintained map outranked the adapterconst match = VENUE_CLASSIFICATIONS[sourceOrName]
if (match) return match.classification
// ...only then...
const adapter = venueAdapters.find(...)
if (adapter) return adapter.classificationThe map came first, so a stale entry would silently override the declaration the compiler enforces. That is the wrong way round: Inverted, with the precedence and the reason now written into the function. I also documented why an unregistered source still defaults to 2. Nothing caught drift between the twoThe new tests assert named venues: expect(isAggregatorVenue('aquarius_amm')).toBe(true)
expect(isPrimaryVenue('SDEX')).toBe(true)Those are good as regression tests for today's four venues, but they do not hold the invariant. A new adapter added next month with no map entry, or with a mismatched one, fails nothing — and the failure mode is a wrong number on a public endpoint, not an exception. Four tests now hold it over
The first is the one that matters: it fails the build on the disagreement instead of letting the runtime pick a winner. One housekeeping noteThe PR changes 130 test files / 1,571 tests pass; |
…closes #212) A dedicated ingester reads Blend pool reserves over Soroban, derives utilisation and supply/borrow APR, and stores them in `lending_rates` rather than `price_points` — a lending rate is not a spot price, and the adapter's `derive` returns `[]` so nothing leaks into the volume or VWAP paths. `GET /rates/:asset` serves them, a retention job prunes old rows, and the new columns ship as a migration. What it got right, and why I am noting it: the schema change comes with `20261023130000_extend_lending_rates_for_blend/migration.sql` rather than a bare `schema.prisma` edit. Since `migrate deploy` replaced `db push` this morning, a schema edit alone reaches no live database. The migration is additive (every column nullable), leaves the baseline untouched, sorts after `0_init`, and its index leads with `network`. The first PR today to need that workflow got it wrong; this one did not. Three changes on top of the contribution. 1. **`classification` was missing.** #455 made it a required field on `VenueAdapter` about ten minutes earlier, so this did not compile on merge. Set to `'primary'`, with a comment recording that it is informational here: Blend emits no price points, so it never reaches the net/gross split — and it is 'primary' because it holds its own reserves, being the venue a DeFindex vault allocates *into* rather than one re-reporting another venue's trades. Also registered in `VENUE_CLASSIFICATIONS`, which #455's new drift test requires — that guard earned its place on the very next adapter. 2. **The ingester loop was unbounded.** Blend runs outside `startVenueAdapterLoop` (it writes `lending_rates`, so the generic derive/write cycle does not fit), which meant it also missed the per-cycle deadline #471 added this afternoon. A `while (true)` around `await ingestBlendPools(network)` is the exact #351 shape: a slow Soroban RPC stretches one sweep indefinitely, the loop never crashes, so `restartIngester` never fires and the venue goes quiet with nothing raised anywhere. Now wrapped in `runCycleWithDeadline`, the same treatment #471 gave the bespoke snapshot ingester, and `ingestBlendPools` takes the signal so an abandoned cycle stops at a pool boundary — one pool is one write unit, so nothing is left half-written. 3. Regenerated `clients/typescript/src/schema.d.ts` for the new `/rates` response shape; it is byte-compared in CI. Note for later: `blendAdapter` is exported but deliberately not in `venueAdapters`, so Blend does not appear in the README's generated venue table. That is defensible — the table describes price venues — but worth revisiting if more non-price venues arrive. 134 test files / 1,603 tests pass; tsc clean. Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
…Labs#453/Divinity-Labs#455 batch The second main-line merge (0742337) reproduced the exact defects of the first (d6e8256) and added one more, so this is the same repair re-applied on the new tip: - src/metrics.ts: the active_subscriptions gauge is again left unclosed with the main-side counters spliced into its options bag — the file does not parse. Closed it: gauge with registers: [register], followed by the full set of counters. - src/api/graphql.ts: both parents' import statements were kept again (metrics, mercurius, config) plus a new duplicate parseAssetStr / getActivePairs pair from this batch. Deduped to one statement each, keeping the richer variants. - .env.example: document BENCH_MODE / BENCH_SUBSCRIBERS_PER_PAIR / BENCH_EVENTS_PER_PAIR so the env-drift guard passes on this branch's own benchmark script. Verified on 0742337 + this commit: tsc clean, full suite 1606 passed / 1 skipped, check-dropped-tests reports no case missing against the queue line (4dc40a3). Generated with Codebuff 🤖 Co-Authored-By: Codebuff <noreply@codebuff.com>
Summary
Prevents volume overstatement caused by double-counting trades routed through aggregators:
VenueClassification(primary|aggregator) toVenueAdapter. ClassifiedSDEX,AMM, andsoroswap_ammas primary markets, andaquarius_ammas an aggregator./volumes/:assetto report labelledtotals: { net, gross },netVolume,grossVolume, and trade counts (netTradeCount,grossTradeCount).totalVolumedefaults tonetVolumeto preserve backward compatibility without volume inflation.docs/venues.mdand referenced indocs/architecture.md.VolumeResponseinopenapi.yamland regeneratedopenapi.json.Related issue
Type of change
Checklist
npx tsc --noEmitpassesnpm run buildpassesCloses #244