Skip to content

fix: classify venues and report net vs gross volume totals - #455

Merged
Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
onajidavid87-web:Volume-is-summed-across-venues-that-may-be-quoting-the-same-trade
Oct 23, 2026
Merged

Miracle656 merged 2 commits into
Divinity-Labs:mainfrom
onajidavid87-web:Volume-is-summed-across-venues-that-may-be-quoting-the-same-trade

Conversation

@onajidavid87-web

Copy link
Copy Markdown
Contributor

Summary

Prevents volume overstatement caused by double-counting trades routed through aggregators:

  • Venue Registry Classification: Added VenueClassification (primary | aggregator) to VenueAdapter. Classified SDEX, AMM, and soroswap_amm as primary markets, and aquarius_amm as an aggregator.
  • Labelled Net & Gross Totals: Updated /volumes/:asset to report labelled totals: { net, gross }, netVolume, grossVolume, and trade counts (netTradeCount, grossTradeCount). totalVolume defaults to netVolume to preserve backward compatibility without volume inflation.
  • Documentation: Documented classification rationale in docs/venues.md and referenced in docs/architecture.md.
  • OpenAPI Schema: Updated VolumeResponse in openapi.yaml and regenerated openapi.json.
  • Test Coverage: Added fail-first test verifying volume separation for pairs traded on both primary and aggregator venues, alongside registry contract tests.

Related issue

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Docs
  • Tests
  • CI / tooling

Checklist

  • I have read CONTRIBUTING.md
  • npx tsc --noEmit passes
  • npm run build passes
  • I added / updated tests where relevant
  • I updated docs where relevant

Closes #244

@drips-wave

drips-wave Bot commented Oct 17, 2026

Copy link
Copy Markdown

@onajidavid87-web Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@Miracle656 Miracle656 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a rebase, and two of the conflicts are design decisions rather than merge noise — so they are yours to make, not mine.

Conflicts: src/ingest/venues/types.ts, src/routes/volumes.ts (3 hunks), plus the two generated OpenAPI files.

1. Two names for the same concept. main now has:

/** Kind is descriptive metadata used by API projections, not the runner. */
readonly kind?: VenueKind

and this PR introduces:

readonly classification: VenueClassification

Same idea, different name, different type, and yours is required where main's is optional. One of them should survive. I would keep one field and one vocabulary rather than have adapters carrying both — but which one is a call for whoever owns the venue model, and required versus optional is the part that actually changes behaviour: a required field forces every existing adapter to declare itself, which is probably what you want and is also the breaking half.

2. The volume query changed shape underneath you. main's handler reads a single row and derives trade_count, total_volume and a status of no_trades / no_positive_volume / ok. Yours loops over rows accumulating net and gross per venue. Those are different SQL and different response semantics, not two spellings of one thing.

I did not resolve this, deliberately. Picking a side would produce an endpoint that returns numbers — and a wrong volume total is the worst kind of wrong, because it looks entirely plausible and nothing fails. You know which query your net/gross split needs; please rebase onto the current handler and reconcile them.

The feature is worth having. Reporting net against gross is a real distinction: counting both sides of a trade, or the same flow across two venues, inflates a figure people compare across venues. Separating primary from derived venues is the right basis for it.

When you rebase, worth adding a test that pins the difference explicitly — one trade visible on two venues should contribute once to net and twice to gross. That is the case the whole change exists for, and it is the one that would silently regress.

…uoting-the-same-trade

main moved /volumes/:asset to a single SQL aggregate (trade-size stats,
HHI, status, provenance), so the net/gross split now sits on top of it:

- venue_totals also returns a per-venue positive-volume trade count
  (trade_count_by_venue); net = gross minus aggregator venues
- totalVolume/tradeCount stay the net figures; the trade-size statistics
  and provenance cover every venue
- keep classification next to main's kind/type/description on each
  adapter; classify the new Phoenix (primary) and DeFindex (aggregator)
  adapters and document them in docs/venues.md
- merge the VolumeResponse schema and regenerate openapi.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Miracle656 added a commit that referenced this pull request Oct 23, 2026
)

`/volumes/:asset` summed every venue, so an aggregator re-reporting a trade
that already settled on a primary market counted it twice. Venues now declare
`classification: 'primary' | 'aggregator'` on the adapter, and the route
reports `totals.net` (primaries only) beside `totals.gross` (everything
observed), both labelled, with the per-venue breakdown unchanged. Aquarius and
DeFindex are aggregators; SDEX, the native AMM, Soroswap and Phoenix are
primary, each with its reasoning recorded.

`classification` is a required field, so a new adapter cannot be registered
without stating what it is — the same reason `type` and `description` are
required. Distribution statistics (median, HHI, status) deliberately stay on
gross, and `SUM(volume)::float8` casts outside the aggregate rather than in,
which is the right form.

Two changes on top of the contribution, both about the same risk: this decides
whether a venue's volume reaches a published total, so two records of it must
not be able to disagree quietly.

1. **The hand-maintained map outranked the adapter.** `getVenueClassification`
   consulted `VENUE_CLASSIFICATIONS` first and fell back to the adapter, so a
   stale map entry silently overrode the declaration the compiler enforces.
   Inverted: the adapter is the authority, and the map carries the prose. The
   precedence is now stated in the function's own comment, including why an
   unregistered source still defaults to `primary` — `price_points.source`
   carries CEX feeds with no adapter, and those are real independent markets.

2. **Nothing caught drift between the two.** The PR's tests assert named
   venues (`isAggregatorVenue('aquarius_amm')` and so on), so a new adapter
   added with no map entry, or with a mismatched one, failed nothing. Four
   tests now hold the invariant over `venueAdapters` as a whole: every adapter
   has a matching entry, every adapter has documented reasoning, resolution
   returns the adapter's own answer, and an unregistered source is never
   silently treated as an aggregator.

Also regenerated `clients/typescript/src/schema.d.ts` for the new `totals`
shape — it is byte-compared in CI.

130 test files / 1,571 tests pass; tsc clean.

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
@Miracle656
Miracle656 merged commit c94be7c into Divinity-Labs:main Oct 23, 2026
@Miracle656

Copy link
Copy Markdown
Collaborator

Merged as 1a332bb — this closes #244.

The issue says "every ecosystem volume figure that has ever been wrong has been wrong for exactly this reason", and the fix treats it with that weight. Reporting both numbers and labelling them is better than silently picking one: a caller who wants gross can still have it, and nobody is guessing which they were given.

Things that were already right and I left alone:

  • classification is a required field on VenueAdapter. A new venue cannot be registered without stating what it is, which is the same reason type and description are required. The alternative — an optional field defaulting to primary — would make every forgotten aggregator a silent double-count.
  • Distribution statistics stay on gross. Median, HHI and the concentration status describe what was observed, and recomputing them on a filtered set would make them mean something else. The comment says so, which is why I did not have to ask.
  • SUM(volume)::float8 casts outside the aggregate. Casting inside is the Prices are stored to 18 decimal places and computed as floats #230 bug — every row to a double before summing, with order-dependent error. You got the form right; worth knowing it is wrong in seven other files, which fix: keep numeric through aggregation and cast once at output #470 is for.
  • docs/venues.md gives per-venue reasoning rather than a bare table, and the DeFindex entry ("holds no market of its own: deposits are allocated to underlying strategies... so it is never counted toward net volume") is the kind of thing that stops someone reclassifying it in six months.

Two changes on top, both about the same risk. This field decides whether a venue's volume reaches a published total, so two records of it must not be able to disagree quietly.

1. The hand-maintained map outranked the adapter

const match = VENUE_CLASSIFICATIONS[sourceOrName]
if (match) return match.classification
// ...only then...
const adapter = venueAdapters.find(...)
if (adapter) return adapter.classification

The map came first, so a stale entry would silently override the declaration the compiler enforces. That is the wrong way round: classification on the adapter is the fact a venue author cannot omit and the thing the ingester actually ran under; VENUE_CLASSIFICATIONS is prose maintained by hand beside it.

Inverted, with the precedence and the reason now written into the function. I also documented why an unregistered source still defaults to primary: price_points.source carries values with no adapter behind them (CEX feeds), and those are genuinely independent markets — so defaulting to aggregator would wrongly exclude real volume.

2. Nothing caught drift between the two

The new tests assert named venues:

expect(isAggregatorVenue('aquarius_amm')).toBe(true)
expect(isPrimaryVenue('SDEX')).toBe(true)

Those are good as regression tests for today's four venues, but they do not hold the invariant. A new adapter added next month with no map entry, or with a mismatched one, fails nothing — and the failure mode is a wrong number on a public endpoint, not an exception.

Four tests now hold it over venueAdapters as a whole:

  • every adapter has a VENUE_CLASSIFICATIONS entry whose classification matches, reported as a list of disagreements rather than a bare boolean;
  • every adapter has documented reasoning;
  • getVenueClassification returns the adapter's own answer for both its source and its name;
  • an unregistered source resolves to primary and never to aggregator.

The first is the one that matters: it fails the build on the disagreement instead of letting the runtime pick a winner.

One housekeeping note

The PR changes openapi.yaml for the new totals shape, so clients/typescript/src/schema.d.ts needed regenerating — it is byte-compared in CI. For next time, a spec change needs all three:

npm run openapi:gen
npx --yes openapi-typescript@7.13.0 openapi.yaml -o clients/typescript/src/schema.d.ts
npm run readme:gen

130 test files / 1,571 tests pass; tsc --noEmit clean.

Miracle656 added a commit that referenced this pull request Oct 23, 2026
…closes #212)

A dedicated ingester reads Blend pool reserves over Soroban, derives
utilisation and supply/borrow APR, and stores them in `lending_rates` rather
than `price_points` — a lending rate is not a spot price, and the adapter's
`derive` returns `[]` so nothing leaks into the volume or VWAP paths.
`GET /rates/:asset` serves them, a retention job prunes old rows, and the new
columns ship as a migration.

What it got right, and why I am noting it: the schema change comes with
`20261023130000_extend_lending_rates_for_blend/migration.sql` rather than a
bare `schema.prisma` edit. Since `migrate deploy` replaced `db push` this
morning, a schema edit alone reaches no live database. The migration is
additive (every column nullable), leaves the baseline untouched, sorts after
`0_init`, and its index leads with `network`. The first PR today to need that
workflow got it wrong; this one did not.

Three changes on top of the contribution.

1. **`classification` was missing.** #455 made it a required field on
   `VenueAdapter` about ten minutes earlier, so this did not compile on merge.
   Set to `'primary'`, with a comment recording that it is informational here:
   Blend emits no price points, so it never reaches the net/gross split — and
   it is 'primary' because it holds its own reserves, being the venue a
   DeFindex vault allocates *into* rather than one re-reporting another venue's
   trades. Also registered in `VENUE_CLASSIFICATIONS`, which #455's new drift
   test requires — that guard earned its place on the very next adapter.

2. **The ingester loop was unbounded.** Blend runs outside
   `startVenueAdapterLoop` (it writes `lending_rates`, so the generic
   derive/write cycle does not fit), which meant it also missed the per-cycle
   deadline #471 added this afternoon. A `while (true)` around
   `await ingestBlendPools(network)` is the exact #351 shape: a slow Soroban
   RPC stretches one sweep indefinitely, the loop never crashes, so
   `restartIngester` never fires and the venue goes quiet with nothing raised
   anywhere. Now wrapped in `runCycleWithDeadline`, the same treatment #471
   gave the bespoke snapshot ingester, and `ingestBlendPools` takes the signal
   so an abandoned cycle stops at a pool boundary — one pool is one write unit,
   so nothing is left half-written.

3. Regenerated `clients/typescript/src/schema.d.ts` for the new `/rates`
   response shape; it is byte-compared in CI.

Note for later: `blendAdapter` is exported but deliberately not in
`venueAdapters`, so Blend does not appear in the README's generated venue
table. That is defensible — the table describes price venues — but worth
revisiting if more non-price venues arrive.

134 test files / 1,603 tests pass; tsc clean.

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
freebuff-web Bot pushed a commit to nonchalanttee22-lgtm/Lens that referenced this pull request Oct 24, 2026
…Labs#453/Divinity-Labs#455 batch

The second main-line merge (0742337) reproduced the exact defects of the
first (d6e8256) and added one more, so this is the same repair re-applied
on the new tip:

- src/metrics.ts: the active_subscriptions gauge is again left unclosed
  with the main-side counters spliced into its options bag — the file
  does not parse. Closed it: gauge with registers: [register], followed
  by the full set of counters.
- src/api/graphql.ts: both parents' import statements were kept again
  (metrics, mercurius, config) plus a new duplicate parseAssetStr /
  getActivePairs pair from this batch. Deduped to one statement each,
  keeping the richer variants.
- .env.example: document BENCH_MODE / BENCH_SUBSCRIBERS_PER_PAIR /
  BENCH_EVENTS_PER_PAIR so the env-drift guard passes on this branch's
  own benchmark script.

Verified on 0742337 + this commit: tsc clean, full suite 1606 passed /
1 skipped, check-dropped-tests reports no case missing against the
queue line (4dc40a3).

Generated with Codebuff 🤖
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Volume is summed across venues that may be quoting the same trade

2 participants