diff --git a/docs/architecture.md b/docs/architecture.md index e8f6f6ef..cfa71607 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -62,11 +62,12 @@ Data is stored in a PostgreSQL database optimized with declarative range partiti * **`price_points`**: A unified table for all trade events across both SDEX and AMM, range-partitioned by `timestamp` into daily partitions (`price_points_yYYYYmMMdDD`). Automatic background workers pre-create future partitions ahead of need. Partition retention via metadata-only `DROP TABLE` operations is opt-in (`PRICE_POINTS_DROP_EXPIRED=true`, default: off) to safeguard raw historical records until rollups are completed (#231). Query execution plans on recent time windows leverage PostgreSQL partition pruning to scan only active partitions and skip all historical data. * **`pool_snapshots`**: Stores the reserve state of AMM liquidity pools, which is critical for slippage estimation and constant-product price calculations. -### 3. Aggregation & Routing (`bestRoute`) +### 3. Aggregation & Routing (`bestRoute` & Volume) When a price is requested, the `bestRoute` engine compares the available liquidity on both SDEX and AMM: * **SDEX Pricing**: Fetches real-time path payment quotes from Horizon to determine the effective rate for a specific trade size. * **AMM Pricing**: Uses the constant-product formula (`x * y = k`) against the latest pool snapshots to estimate the output and slippage. * **Optimal Routing**: Compares the rates and recommends the best execution path (SDEX, AMM, or a SPLIT for large orders) to minimize slippage for the user. +* **Venue Classification & Volume Accounting**: Venues in the venue registry are categorized as either primary markets (SDEX, native AMM, Soroswap) or aggregators (Aquarius). Aggregate volume metrics report labelled net (primary markets) and gross figures to eliminate trade double-counting. See [venues.md](./venues.md) for details. ### 4. API Layer & x402 Payment Gate The system exposes both REST and GraphQL interfaces. To monetize the high-fidelity data, Lens implements the **x402 protocol**. diff --git a/docs/venues.md b/docs/venues.md new file mode 100644 index 00000000..3352be33 --- /dev/null +++ b/docs/venues.md @@ -0,0 +1,72 @@ +# Venue Classification & Volume Accounting + +## Overview + +Lens ingests trade and pricing data across multiple venues on the Stellar network. When computing aggregate volume metrics (e.g. `GET /volumes/:asset`), summing reported volumes across all venues naively leads to severe double-counting. + +Aggregator venues that route through or quote underlying decentralized exchanges report the exact same underlying trades that are also ingested from the primary venue. If an aggregator routes an order through a Stellar native liquidity pool or SDEX orderbook, counting the volume on both venues overstates actual economic activity. + +To solve this, Lens classifies every venue in the venue registry as either a **primary market** or an **aggregator**. + +--- + +## Classification Taxonomy + +| Classification | Definition | Volume Accounting Impact | +| :--- | :--- | :--- | +| **`primary`** | An independent market holding its own resting liquidity (orderbook or isolated pool reserves) where trades execute natively. | Counted toward **net** and **gross** volume totals. | +| **`aggregator`** | A router, wrapper, or meta-venue that quotes and executes trades against underlying primary markets. | Excluded from **net** volume (to prevent double-counting); included in **gross** volume. | + +--- + +## Venue Registry & Classification Reasoning + +| Venue Name | Source Identifier | Classification | Technical Reasoning | +| :--- | :--- | :--- | :--- | +| **SDEX** | `SDEX` | `primary` | **Stellar Decentralized Exchange**: Stellar's native protocol-level Central Limit Order Book (CLOB). Resting limit orders are stored directly in the Stellar ledger state. Trades execute as native on-ledger operations matching resting buy and sell offers, establishing primary price discovery with independent orderbook liquidity. | +| **AMM** | `AMM` | `primary` | **Stellar Native Liquidity Pools (CAP-0038)**: Protocol-level constant-product liquidity pools. Liquidity providers deposit assets directly into on-chain pool reserves. Trades execute against pool reserves directly on the ledger. Each trade represents primary liquidity consumption in the pool. | +| **Soroswap** | `soroswap_amm` | `primary` | **Soroswap AMM**: Automated market maker smart contracts deployed on the Soroban smart contract platform. Pairs hold isolated pool reserves locked in Soroban contract instances deposited directly by LPs. Trades execute against the contracts' own reserves rather than routing to other venues. | +| **Aquarius** | `aquarius_amm` | `aggregator` | **Aquarius**: On classic Stellar, Aquarius acts as a liquidity aggregator and incentive layer. It quotes and routes trades through underlying Stellar native AMM liquidity pools (`pool_hash`) and SDEX orderbooks rather than settling against an isolated, independent liquidity source. Counting Aquarius volume alongside SDEX and native AMM trades double-counts the same underlying trades. | +| **Phoenix** | `phoenix_amm` | `primary` | **Phoenix AMM**: Constant-product liquidity pools on Soroban. Each pool contract holds its own reserves deposited by LPs, and trades execute against those reserves rather than being routed elsewhere. | +| **DeFindex** | `defindex_vault` | `aggregator` | **DeFindex Vaults**: Yield vaults on Soroban priced per share. A vault holds no market of its own: deposits are allocated to underlying strategies such as lending pools, so any activity it reports belongs to the venues underneath. It is excluded from net volume. | + +--- + +## API Response Format (`/volumes/:asset`) + +The aggregated volume endpoint (`GET /volumes/:asset`) reports both figures with explicit labels: + +```json +{ + "asset": "XLM", + "network": "testnet", + "window": "24h", + "startTime": "2026-10-16T12:00:00.000Z", + "endTime": "2026-10-17T12:00:00.000Z", + "totals": { + "net": 1000000.0, + "gross": 1500000.0 + }, + "netVolume": 1000000.0, + "grossVolume": 1500000.0, + "totalVolume": 1000000.0, + "byVenue": { + "SDEX": 600000.0, + "AMM": 400000.0, + "aquarius_amm": 500000.0 + }, + "venues": [ + "SDEX", + "AMM", + "aquarius_amm" + ], + "tradeCount": 1200, + "netTradeCount": 1200, + "grossTradeCount": 1750 +} +``` + +- **`totals.net` / `netVolume`**: Traded volume summed across primary markets only (`SDEX`, `AMM`, `soroswap_amm`, `phoenix_amm`). This is the true, non-duplicated ecosystem trading volume. +- **`totals.gross` / `grossVolume`**: Total volume across all reporting venues including aggregators (`aquarius_amm`, `defindex_vault`). +- **`totalVolume`**: Preserved for backward compatibility, reporting `netVolume` to ensure callers do not overstate real activity. +- **`byVenue`**: Granular breakdown of reported volume by venue source string. diff --git a/openapi.json b/openapi.json index 051e69a0..7b7f808a 100644 --- a/openapi.json +++ b/openapi.json @@ -1672,7 +1672,7 @@ "get": { "operationId": "getVolumes", "summary": "Get cross-venue traded volume for an asset", - "description": "Sums traded volume for `asset` across every pair and venue it appears in, measured in the asset's own units, with a per-venue breakdown and trade-size statistics. `volumeConcentration` is the Herfindahl-Hirschman Index (HHI), calculated as the sum of squared per-trade volume shares across the selected venues and window; it ranges from 0 to 1, with 1 meaning all volume came from one trade. A missing statistic is `null` when there are no trades or no positive volume. Provenance reports the venues, positive-volume trade count, observed sample count, most recent trade timestamp and its age at `endTime`.", + "description": "Sums traded volume for `asset` across every pair and venue it appears in, measured in the asset's own units, with a per-venue breakdown and trade-size statistics. `volumeConcentration` is the Herfindahl-Hirschman Index (HHI), calculated as the sum of squared per-trade volume shares across the selected venues and window; it ranges from 0 to 1, with 1 meaning all volume came from one trade. A missing statistic is `null` when there are no trades or no positive volume. Provenance reports the venues, positive-volume trade count, observed sample count, most recent trade timestamp and its age at `endTime`. Venues are classified as primary markets or aggregators in the registry. `totals.net` / `netVolume` sum primary markets only, while `totals.gross` / `grossVolume` include aggregator venues (e.g. Aquarius) that re-report trades settled elsewhere. `totalVolume` and `tradeCount` report the net figures; the trade-size statistics and provenance cover every venue (gross).", "security": [], "parameters": [ { @@ -4549,6 +4549,24 @@ } } }, + "VolumeTotals": { + "type": "object", + "required": [ + "net", + "gross" + ], + "additionalProperties": false, + "properties": { + "net": { + "type": "number", + "description": "Volume summed across primary markets only (excluding aggregator double-counts)" + }, + "gross": { + "type": "number", + "description": "Gross volume summed across all venues including aggregators" + } + } + }, "VolumeResponse": { "type": "object", "required": [ @@ -4558,9 +4576,14 @@ "startTime", "endTime", "totalVolume", + "netVolume", + "grossVolume", + "totals", "byVenue", "venues", "tradeCount", + "netTradeCount", + "grossTradeCount", "medianTradeSize", "topTradeVolumeShare", "volumeConcentration", @@ -4596,8 +4619,20 @@ "type": "string", "format": "date-time" }, + "totals": { + "$ref": "#/components/schemas/VolumeTotals" + }, + "netVolume": { + "type": "number", + "description": "Volume summed across primary markets only (excluding aggregator double-counts)" + }, + "grossVolume": { + "type": "number", + "description": "Gross volume summed across all venues including aggregators" + }, "totalVolume": { - "type": "number" + "type": "number", + "description": "Primary volume total (alias for netVolume for backward compatibility)" }, "byVenue": { "type": "object", @@ -4613,7 +4648,15 @@ }, "tradeCount": { "type": "integer", - "description": "Number of positive-volume price point samples in the selected window." + "description": "Number of positive-volume samples from primary markets (equals netTradeCount)." + }, + "netTradeCount": { + "type": "integer", + "description": "Number of positive-volume samples from primary markets." + }, + "grossTradeCount": { + "type": "integer", + "description": "Number of positive-volume samples across all venues, including aggregators." }, "medianTradeSize": { "type": "number", diff --git a/openapi.yaml b/openapi.yaml index ef8d2955..0dfd59e6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1147,6 +1147,12 @@ paths: when there are no trades or no positive volume. Provenance reports the venues, positive-volume trade count, observed sample count, most recent trade timestamp and its age at `endTime`. + Venues are classified as primary markets or aggregators in the registry. + `totals.net` / `netVolume` sum primary markets only, while + `totals.gross` / `grossVolume` include aggregator venues (e.g. Aquarius) + that re-report trades settled elsewhere. `totalVolume` and `tradeCount` + report the net figures; the trade-size statistics and provenance + cover every venue (gross). security: [] parameters: - name: asset @@ -2886,6 +2892,18 @@ components: type: string enum: [iqr, modified_zscore] + VolumeTotals: + type: object + required: [net, gross] + additionalProperties: false + properties: + net: + type: number + description: Volume summed across primary markets only (excluding aggregator double-counts) + gross: + type: number + description: Gross volume summed across all venues including aggregators + VolumeResponse: type: object required: @@ -2895,9 +2913,14 @@ components: - startTime - endTime - totalVolume + - netVolume + - grossVolume + - totals - byVenue - venues - tradeCount + - netTradeCount + - grossTradeCount - medianTradeSize - topTradeVolumeShare - volumeConcentration @@ -2920,8 +2943,17 @@ components: endTime: type: string format: date-time + totals: + $ref: '#/components/schemas/VolumeTotals' + netVolume: + type: number + description: Volume summed across primary markets only (excluding aggregator double-counts) + grossVolume: + type: number + description: Gross volume summed across all venues including aggregators totalVolume: type: number + description: Primary volume total (alias for netVolume for backward compatibility) byVenue: type: object additionalProperties: @@ -2932,7 +2964,13 @@ components: type: string tradeCount: type: integer - description: Number of positive-volume price point samples in the selected window. + description: Number of positive-volume samples from primary markets (equals netTradeCount). + netTradeCount: + type: integer + description: Number of positive-volume samples from primary markets. + grossTradeCount: + type: integer + description: Number of positive-volume samples across all venues, including aggregators. medianTradeSize: type: number nullable: true diff --git a/src/__tests__/venueAdapters.test.ts b/src/__tests__/venueAdapters.test.ts index 587f35e1..6ea3879f 100644 --- a/src/__tests__/venueAdapters.test.ts +++ b/src/__tests__/venueAdapters.test.ts @@ -13,6 +13,11 @@ import { getLastPrice, resetLastPrice, setLastPrice, + VENUE_CLASSIFICATIONS, + getVenueClassification, + isPrimaryVenue, + isAggregatorVenue, + getVenueReasoning, type VenueAdapter, } from '../ingest/venues' @@ -50,6 +55,7 @@ describe('VenueAdapter Interface and Runner (#218)', () => { for (const adapter of venueAdapters) { expect(typeof adapter.name).toBe('string') expect(typeof adapter.source).toBe('string') + expect(['primary', 'aggregator']).toContain(adapter.classification) expect(typeof adapter.isEnabled).toBe('function') expect(typeof adapter.getPollIntervalMs).toBe('function') expect(typeof adapter.discover).toBe('function') @@ -60,6 +66,29 @@ describe('VenueAdapter Interface and Runner (#218)', () => { } }) + it('classifies SDEX, AMM, and Soroswap as primary markets and Aquarius as an aggregator', () => { + expect(sdexAdapter.classification).toBe('primary') + expect(ammAdapter.classification).toBe('primary') + expect(soroswapAdapter.classification).toBe('primary') + expect(aquariusAdapter.classification).toBe('aggregator') + + expect(isPrimaryVenue('SDEX')).toBe(true) + expect(isPrimaryVenue('AMM')).toBe(true) + expect(isPrimaryVenue('soroswap_amm')).toBe(true) + expect(isPrimaryVenue('aquarius_amm')).toBe(false) + + expect(isAggregatorVenue('aquarius_amm')).toBe(true) + expect(isAggregatorVenue('Aquarius')).toBe(true) + expect(isAggregatorVenue('SDEX')).toBe(false) + expect(isAggregatorVenue('AMM')).toBe(false) + + // Documented reasoning must be present + expect(getVenueReasoning('SDEX')).toMatch(/order book|CLOB/i) + expect(getVenueReasoning('AMM')).toMatch(/liquidity pool|reserves/i) + expect(getVenueReasoning('soroswap_amm')).toMatch(/Soroban/i) + expect(getVenueReasoning('aquarius_amm')).toMatch(/aggregator|double-count/i) + }) + it('preserves exact source strings for backward compatibility', () => { expect(sdexAdapter.source).toBe('SDEX') expect(ammAdapter.source).toBe('AMM') @@ -92,6 +121,7 @@ describe('VenueAdapter Interface and Runner (#218)', () => { const customAdapter: VenueAdapter = { name: 'CustomDEX', source: 'custom_dex', + classification: 'primary', isEnabled: () => false, getPollIntervalMs: () => 1000, discover: async () => [], diff --git a/src/__tests__/volumes.test.ts b/src/__tests__/volumes.test.ts index cdf8aa62..1bf1973e 100644 --- a/src/__tests__/volumes.test.ts +++ b/src/__tests__/volumes.test.ts @@ -109,6 +109,107 @@ describe('GET /volumes/:asset', () => { await app.close() }) + it('separates net (primary markets) and gross (including aggregators) totals when traded on primary and aggregator venues', async () => { + mockQuery.mockResolvedValue({ + rows: [ + { + total_volume: '150.0', + by_venue: { SDEX: 100.0, aquarius_amm: 50.0 }, + trade_count_by_venue: { SDEX: 2, aquarius_amm: 1 }, + venues: ['SDEX', 'aquarius_amm'], + trade_count: 3, + observed_sample_count: 3, + }, + ], + }) + + const app = await buildApp() + const res = await app.inject({ method: 'GET', url: '/volumes/XLM?window=24h' }) + + expect(res.statusCode).toBe(200) + const body = res.json() + expect(body.asset).toBe('XLM') + expect(body.byVenue).toEqual({ SDEX: 100.0, aquarius_amm: 50.0 }) + expect(body.totals).toEqual({ + net: 100.0, + gross: 150.0, + }) + expect(body.netVolume).toBe(100.0) + expect(body.grossVolume).toBe(150.0) + // totalVolume defaults to net volume to avoid overstating real activity + expect(body.totalVolume).toBe(100.0) + expect(body.tradeCount).toBe(2) + expect(body.netTradeCount).toBe(2) + expect(body.grossTradeCount).toBe(3) + await app.close() + }) + + it('aggregates across multiple primaries (SDEX, AMM, Soroswap) alongside an aggregator (Aquarius)', async () => { + mockQuery.mockResolvedValue({ + rows: [ + { + total_volume: '430.0', + by_venue: { SDEX: 100.0, AMM: 200.0, soroswap_amm: 50.0, aquarius_amm: 80.0 }, + trade_count_by_venue: { SDEX: 5, AMM: 8, soroswap_amm: 2, aquarius_amm: 3 }, + venues: ['AMM', 'SDEX', 'aquarius_amm', 'soroswap_amm'], + trade_count: 18, + observed_sample_count: 18, + }, + ], + }) + + const app = await buildApp() + const res = await app.inject({ method: 'GET', url: '/volumes/USDC?window=24h' }) + + expect(res.statusCode).toBe(200) + const body = res.json() + expect(body.asset).toBe('USDC') + expect(body.totals.net).toBe(350.0) + expect(body.totals.gross).toBe(430.0) + expect(body.netVolume).toBe(350.0) + expect(body.grossVolume).toBe(430.0) + expect(body.totalVolume).toBe(350.0) + expect(body.tradeCount).toBe(15) + expect(body.netTradeCount).toBe(15) + expect(body.grossTradeCount).toBe(18) + expect(body.byVenue).toEqual({ + SDEX: 100.0, + AMM: 200.0, + soroswap_amm: 50.0, + aquarius_amm: 80.0, + }) + await app.close() + }) + + it('returns zero net volume if only aggregator trades exist', async () => { + mockQuery.mockResolvedValue({ + rows: [ + { + total_volume: '75.0', + by_venue: { aquarius_amm: 75.0 }, + trade_count_by_venue: { aquarius_amm: 4 }, + venues: ['aquarius_amm'], + trade_count: 4, + observed_sample_count: 4, + }, + ], + }) + + const app = await buildApp() + const res = await app.inject({ method: 'GET', url: '/volumes/XLM?window=24h' }) + + expect(res.statusCode).toBe(200) + const body = res.json() + expect(body.totals).toEqual({ net: 0, gross: 75.0 }) + expect(body.netVolume).toBe(0) + expect(body.grossVolume).toBe(75.0) + expect(body.totalVolume).toBe(0) + expect(body.tradeCount).toBe(0) + expect(body.netTradeCount).toBe(0) + expect(body.grossTradeCount).toBe(4) + await app.close() + }) + it('defaults to the 24h window', async () => { mockQuery.mockResolvedValue({ rows: [] }) const app = await buildApp() diff --git a/src/ingest/venues/amm.ts b/src/ingest/venues/amm.ts index 1335f2f9..ac1f79c5 100644 --- a/src/ingest/venues/amm.ts +++ b/src/ingest/venues/amm.ts @@ -280,6 +280,7 @@ export const ammAdapter: VenueAdapter = { name: 'AMM', kind: 'amm', source: 'AMM', + classification: 'primary', type: 'Liquidity Pool', description: 'Stellar native AMM liquidity pools (Horizon)', diff --git a/src/ingest/venues/aquarius.ts b/src/ingest/venues/aquarius.ts index bd77befb..fc8dfccf 100644 --- a/src/ingest/venues/aquarius.ts +++ b/src/ingest/venues/aquarius.ts @@ -126,6 +126,7 @@ export const aquariusAdapter: VenueAdapter = { name: 'Aquarius', kind: 'amm', source: 'aquarius_amm', + classification: 'aggregator', type: 'Liquidity Pool', description: 'Aquarius AMM liquidity pools', diff --git a/src/ingest/venues/defindex.ts b/src/ingest/venues/defindex.ts index 05a9e94b..12ce343e 100644 --- a/src/ingest/venues/defindex.ts +++ b/src/ingest/venues/defindex.ts @@ -420,6 +420,7 @@ export const defindexAdapter: VenueAdapter = { type: 'Vault', description: 'DeFindex yield vaults on Soroban, priced per share', source: DEFINDEX_SOURCE, + classification: 'aggregator', isEnabled(network: NetworkName): boolean { return getNetworkConfig(network).defindex.enabled diff --git a/src/ingest/venues/index.ts b/src/ingest/venues/index.ts index 77584eb4..2c4c08c8 100644 --- a/src/ingest/venues/index.ts +++ b/src/ingest/venues/index.ts @@ -28,3 +28,148 @@ export const venueAdapters: VenueAdapter[] = [ defindexAdapter, phoenixAdapter, ] + +export interface VenueClassificationInfo { + name: string + source: string + classification: import('./types').VenueClassification + reasoning: string +} + +/** + * Canonical registry of venue classifications and technical reasoning. + * + * Every venue ingested into Lens is classified as either: + * - 'primary': An original, independent execution venue holding its own resting + * liquidity (e.g. orderbook or isolated AMM pool reserves). + * - 'aggregator': A routing or wrapper venue that executes trades against underlying + * primary markets (such as classic SDEX orderbooks or native AMM pools). Counting + * aggregator volume alongside primary venue volume double-counts the same underlying trades. + */ +export const VENUE_CLASSIFICATIONS: Record = { + SDEX: { + name: 'SDEX', + source: 'SDEX', + classification: 'primary', + reasoning: + "Stellar's native protocol-level Central Limit Order Book (CLOB). Resting limit offers match natively on the ledger, establishing independent liquidity and primary price discovery.", + }, + AMM: { + name: 'AMM', + source: 'AMM', + classification: 'primary', + reasoning: + 'Stellar native on-chain liquidity pools (CAP-0038). Liquidity providers deposit assets directly into on-chain pools, and trades execute against constant-product pool reserves.', + }, + soroswap_amm: { + name: 'Soroswap', + source: 'soroswap_amm', + classification: 'primary', + reasoning: + 'Automated market maker smart contracts on Soroban. Pairs hold independent pool reserves deposited directly into Soroban contract instances.', + }, + Soroswap: { + name: 'Soroswap', + source: 'soroswap_amm', + classification: 'primary', + reasoning: + 'Automated market maker smart contracts on Soroban. Pairs hold independent pool reserves deposited directly into Soroban contract instances.', + }, + aquarius_amm: { + name: 'Aquarius', + source: 'aquarius_amm', + classification: 'aggregator', + reasoning: + 'Aquarius operates as an aggregator/liquidity layer on Stellar, quoting and routing through underlying Stellar native AMM pools and SDEX orderbooks. Counting its trades alongside primary venues double-counts the same underlying trade executions.', + }, + Aquarius: { + name: 'Aquarius', + source: 'aquarius_amm', + classification: 'aggregator', + reasoning: + 'Aquarius operates as an aggregator/liquidity layer on Stellar, quoting and routing through underlying Stellar native AMM pools and SDEX orderbooks. Counting its trades alongside primary venues double-counts the same underlying trade executions.', + }, + phoenix_amm: { + name: 'Phoenix', + source: 'phoenix_amm', + classification: 'primary', + reasoning: + 'Constant-product AMM pools on Soroban. Each pool contract holds its own reserves, so trades against it are independent executions.', + }, + Phoenix: { + name: 'Phoenix', + source: 'phoenix_amm', + classification: 'primary', + reasoning: + 'Constant-product AMM pools on Soroban. Each pool contract holds its own reserves, so trades against it are independent executions.', + }, + defindex_vault: { + name: 'DeFindex', + source: 'defindex_vault', + classification: 'aggregator', + reasoning: + 'Yield vaults on Soroban that hold no market of their own: deposits are allocated to underlying strategies (e.g. lending pools) and the vault is priced per share. Any activity it reports is activity of the venues underneath, so it is never counted toward net volume.', + }, + DeFindex: { + name: 'DeFindex', + source: 'defindex_vault', + classification: 'aggregator', + reasoning: + 'Yield vaults on Soroban that hold no market of their own: deposits are allocated to underlying strategies (e.g. lending pools) and the vault is priced per share. Any activity it reports is activity of the venues underneath, so it is never counted toward net volume.', + }, +} + +/** + * Returns the classification ('primary' | 'aggregator') for a venue by source or name. + * Defaults to 'primary' for unknown venues to ensure primary markets are not omitted. + */ +export function getVenueClassification(sourceOrName: string): import('./types').VenueClassification { + if (!sourceOrName) return 'primary' + const match = VENUE_CLASSIFICATIONS[sourceOrName] + if (match) return match.classification + + const lower = sourceOrName.toLowerCase() + for (const [key, info] of Object.entries(VENUE_CLASSIFICATIONS)) { + if (key.toLowerCase() === lower || info.name.toLowerCase() === lower || info.source.toLowerCase() === lower) { + return info.classification + } + } + + const adapter = venueAdapters.find( + (a) => a.source.toLowerCase() === lower || a.name.toLowerCase() === lower, + ) + if (adapter) return adapter.classification + + return 'primary' +} + +/** + * Determines whether a venue is a primary market. + */ +export function isPrimaryVenue(sourceOrName: string): boolean { + return getVenueClassification(sourceOrName) === 'primary' +} + +/** + * Determines whether a venue is an aggregator. + */ +export function isAggregatorVenue(sourceOrName: string): boolean { + return getVenueClassification(sourceOrName) === 'aggregator' +} + +/** + * Retrieves the documented classification reasoning for a given venue. + */ +export function getVenueReasoning(sourceOrName: string): string | undefined { + if (!sourceOrName) return undefined + const match = VENUE_CLASSIFICATIONS[sourceOrName] + if (match) return match.reasoning + + const lower = sourceOrName.toLowerCase() + for (const [key, info] of Object.entries(VENUE_CLASSIFICATIONS)) { + if (key.toLowerCase() === lower || info.name.toLowerCase() === lower || info.source.toLowerCase() === lower) { + return info.reasoning + } + } + return undefined +} diff --git a/src/ingest/venues/phoenix.ts b/src/ingest/venues/phoenix.ts index b6c3dea0..07a51aaf 100644 --- a/src/ingest/venues/phoenix.ts +++ b/src/ingest/venues/phoenix.ts @@ -532,6 +532,7 @@ export const phoenixAdapter: VenueAdapter = { type: 'Liquidity Pool', description: 'Phoenix AMM pools on Soroban', source: 'phoenix_amm', + classification: 'primary', isEnabled(network: NetworkName): boolean { return getNetworkConfig(network).phoenix.enabled diff --git a/src/ingest/venues/sdex.ts b/src/ingest/venues/sdex.ts index 7b228840..8f2bd787 100644 --- a/src/ingest/venues/sdex.ts +++ b/src/ingest/venues/sdex.ts @@ -190,6 +190,7 @@ export const sdexAdapter: VenueAdapter = { name: 'SDEX', kind: 'orderbook', source: 'SDEX', + classification: 'primary', type: 'Order Book', description: "Stellar's Classic Order Book (SDEX)", diff --git a/src/ingest/venues/soroswap.ts b/src/ingest/venues/soroswap.ts index 28363a9d..b60ad7ff 100644 --- a/src/ingest/venues/soroswap.ts +++ b/src/ingest/venues/soroswap.ts @@ -478,6 +478,7 @@ export const soroswapAdapter: VenueAdapter = { name: 'Soroswap', kind: 'amm', source: 'soroswap_amm', + classification: 'primary', type: 'AMM DEX', description: 'Soroswap constant-product automated market maker on Soroban', diff --git a/src/ingest/venues/types.ts b/src/ingest/venues/types.ts index 7f075b94..c5bfa763 100644 --- a/src/ingest/venues/types.ts +++ b/src/ingest/venues/types.ts @@ -15,6 +15,8 @@ export interface VenuePricePoint { eventId?: string } +export type VenueClassification = 'primary' | 'aggregator' + /** * Universal interface for all venue adapters (SDEX, Stellar AMM, Soroswap, Aquarius, Phoenix, etc.). * @@ -29,6 +31,7 @@ export type VenueKind = 'orderbook' | 'amm' | 'oracle' | 'unknown' export interface VenueAdapter { readonly name: string readonly source: string + readonly classification: VenueClassification /** Kind is descriptive metadata used by API projections, not the runner. */ readonly kind?: VenueKind diff --git a/src/routes/volumes.ts b/src/routes/volumes.ts index 2796ccd4..340ba7ba 100644 --- a/src/routes/volumes.ts +++ b/src/routes/volumes.ts @@ -2,6 +2,7 @@ import type { FastifyInstance } from 'fastify' import { pgPool } from '../db' import { activeNetwork, type NetworkName } from '../config' import { sendInternalError } from '../errors' +import { isAggregatorVenue } from '../ingest/venues' // Supported windows → lookback in hours. const WINDOW_HOURS = { @@ -20,10 +21,15 @@ const WINDOWS = Object.keys(WINDOW_HOURS) as VolumeWindow[] * GET /volumes/:asset?window=24h|7d|30d&network=testnet|mainnet * * Sums traded volume for `asset` across every pair it appears in and across all - * venues (SDEX, AMM, …) over the requested window. Volume is measured in the - * asset's own units: when it is the base asset of a pair we count `base_volume`, - * when it is the counter asset we count `counter_volume`. The response breaks the - * total down per venue and also returns the cross-venue sum. + * venues (SDEX, AMM, Soroswap, Aquarius, …) over the requested window. Volume is + * measured in the asset's own units: when it is the base asset of a pair we count + * `base_volume`, when it is the counter asset we count `counter_volume`. + * + * Venues are classified as either primary markets (e.g. SDEX, AMM, Soroswap) or + * aggregators (e.g. Aquarius) that route trades through underlying primary venues. + * To prevent double-counting of trades, the response reports labelled net totals + * (summed across primary venues) and gross totals (including aggregators), alongside + * a per-venue breakdown. * * Only one network's rows are summed: testnet and mainnet volume added together * is not a volume of anything. `network` defaults to this process's own network, @@ -73,7 +79,9 @@ export async function registerVolumeRoutes(app: FastifyInstance) { AND timestamp >= $3 ), venue_totals AS ( - SELECT source, SUM(volume)::float8 AS volume + SELECT source, + SUM(volume)::float8 AS volume, + (COUNT(*) FILTER (WHERE volume > 0))::int AS trade_count FROM asset_trades GROUP BY source ), @@ -92,10 +100,11 @@ export async function registerVolumeRoutes(app: FastifyInstance) { ), venue_summary AS ( SELECT COALESCE(jsonb_object_agg(source, volume) FILTER (WHERE volume > 0), '{}'::jsonb) AS by_venue, - COALESCE(jsonb_agg(source ORDER BY source) FILTER (WHERE volume > 0), '[]'::jsonb) AS venues + COALESCE(jsonb_agg(source ORDER BY source) FILTER (WHERE volume > 0), '[]'::jsonb) AS venues, + COALESCE(jsonb_object_agg(source, trade_count) FILTER (WHERE volume > 0), '{}'::jsonb) AS trade_count_by_venue FROM venue_totals ) - SELECT trade_stats.*, venue_summary.by_venue, venue_summary.venues + SELECT trade_stats.*, venue_summary.by_venue, venue_summary.venues, venue_summary.trade_count_by_venue FROM trade_stats CROSS JOIN venue_summary`, [asset, network, startTime], ) @@ -116,16 +125,44 @@ export async function registerVolumeRoutes(app: FastifyInstance) { ? 'no_positive_volume' : 'ok' + // Net excludes aggregator venues, which re-report trades that already + // settled on a primary market; gross is everything observed. The + // distribution statistics above (median, HHI, status) stay on gross. + const byVenue: Record = row.by_venue ?? {} + const tradesByVenue: Record = row.trade_count_by_venue ?? {} + let aggregatorVolume = 0 + let aggregatorTradeCount = 0 + for (const [source, volume] of Object.entries(byVenue)) { + if (!isAggregatorVenue(source)) continue + aggregatorVolume += Number(volume) || 0 + aggregatorTradeCount += Number(tradesByVenue[source]) || 0 + } + const grossVolume = totalVolume + const grossTradeCount = tradeCount + const netVolume = Math.max(0, grossVolume - aggregatorVolume) + const netTradeCount = Math.max(0, grossTradeCount - aggregatorTradeCount) + return { asset, network, window, startTime: startTime.toISOString(), endTime: endTime.toISOString(), - totalVolume, - byVenue: row.by_venue ?? {}, + // Labelled net and gross totals to eliminate aggregator double-counting. + totals: { + net: netVolume, + gross: grossVolume, + }, + netVolume, + grossVolume, + // Backward-compatible alias for net volume. + totalVolume: netVolume, + // Per-venue breakdown that the totals are summed from. + byVenue, venues: row.venues ?? [], - tradeCount, + tradeCount: netTradeCount, + netTradeCount, + grossTradeCount, medianTradeSize: row.median_trade_size == null ? null : Number(row.median_trade_size), topTradeVolumeShare: row.top_trade_volume_share == null ? null : Number(row.top_trade_volume_share), volumeConcentration: row.volume_concentration == null ? null : Number(row.volume_concentration), @@ -133,7 +170,7 @@ export async function registerVolumeRoutes(app: FastifyInstance) { status, provenance: { venues: row.venues ?? [], - sampleCount: tradeCount, + sampleCount: grossTradeCount, observedSampleCount, latestTradeAt, dataAgeSeconds,