fix: add stricter companion series to benches 004 and 008 - #2266
Merged
Conversation
Both benches score a proxy for the thing they claim to measure. Bench 004 scores the logo field as "non-empty string", which any provider wins by construction by rewriting logos onto its own CDN at a deterministic path. The current beneficiary is Mobula, at 100% logo on every chain against 22.9/37.8/78.9 for a provider returning upstream URLs. Adds logo_resolved: HEAD the URL, fall back to a ranged GET on 403/405, cache 6h so we do not hammer third-party CDNs. Bench 008 scores a hit as "any non-generic name", which cannot tell a curated entity from a personal name-service record. Measured: 25% of Serialized's hits and 25.4% of Mobula's named something other than the curated entity. The anchor list already carries a Hint for every address and the scoring path ignored it. Adds wallet_labels_accurate_total scored against that Hint. Both land as SEPARATE series, not as redefinitions, so the published leaderboards and their history stay intact while the stricter numbers build up alongside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both benches score a proxy for what they claim to measure. Neither is redefined here: each gets a separate, stricter series so the published leaderboards and their history stay intact while the new numbers build up alongside.
Bench 004:
logomeasures URL presence, not logo existenceMobula rewrites every logo onto
metadata.mobula.ioat a deterministic path derived from chain and address, so its logo field is non-empty for every token whether or not an image exists. Providers returning the upstream URL (ipfs.io, cdn.dexscreener.com, launchpad CDNs, twimg) are scored on whether the upstream actually has the asset.Measured: Mobula 100% logo on all three chains, against 22.9 / 37.8 / 78.9 for a provider returning upstream URLs. A HEAD sweep of 12 distinct Mobula logo URLs resolved 11 and 404'd one.
Any provider can win the current rule by rewriting to its own CDN, and the current beneficiary is our own product. New field
logo_resolved: HEAD the URL, fall back to a ranged GET on 403/405 (several CDNs reject HEAD but serve GET), 6h cache so a monitor does not hammer third-party hosts, empty Content-Type accepted because some IPFS gateways omit it on ranged responses.Bench 008: a hit is "any non-generic name"
That rule cannot tell a curated entity label from a personal name-service record.
dex.davywoodfi.ethon Permit2,jakie.base.ethon Base USDC andbonklanatoken.solon the Raydium authority all score as correct entity labels.Measured on the 100 anchors covered by both key-holding providers: 25% of Serialized's hits and 25.4% of Mobula's named something other than the curated entity. The bias is symmetric, so this is a property of the scoring rule, not of a vendor.
The anchor list has carried a curated
Hintfor every address from the start and the scoring path ignored it. New serieswallet_labels_accurate_totalscores against it.The matcher is permissive on form and strict on identity, with unit tests for both directions. Two real bugs it caught during development:
len(w) <= 3floor silently droppedOKX, soOKX 7did not matchOKX 1baseinUSDC (Base native)matchjakie.base.eth, scoring a personal Basename as a correct token-contract labelFixed with a 3-char floor, a generic/chain-word stoplist, and whole-word matching.
go test ./...green.Not in this PR
The bench 001 reference defect (spec claims archive nodes, harness uses each provider's own timestamp). That is a multi-day change to a live bench and gets its own PR.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CpArutAtXuBb1BVNUDXoYA