Repository navigation
Conversation
kentare
added a commit
that referenced
this pull request
Oct 28, 2026
…pages - benchmarks/benchlib and benchmarks/README.md come from #126, unchanged. benchmarks/kudos-pilot splits into engines, embeddings and indexes, each with a report.py that rebuilds its report from results/; engines and embeddings also keep their reference, per-side verdicts and run records. - New embedding models: gte-multilingual-base, pplx-embed-v1-0.6b, F2LLM-v2-0.6B, harrier-oss-v1-0.6b and text-embedding-3-small, plus Qwen3-Embedding-8B, 4B and Arctic shortened from their stored vectors (shorten.py). - Search steps: setup_models.py writes genki's search without the planner cut down to one step at a time (vector or fusion search, with and without ColBERT, and the whole search without ColBERT). chain_steps.sh runs them for text-embedding-3-large and pplx-embed, and the embeddings report shows which step loses the pages: the keyword fusion costs the most page MRR. - kudos-pilot-compare.ts writes per-question verdicts for the reports. The old flat results/ and its scripts go, and __pycache__ is no longer tracked. Part of #29, from the spike in #92.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
genki had to choose which text to chunk for Kudos: the text KUDOS serves, or Marker's markdown from the PDF. A service's test set is too small to show a few points lost in text extraction, so this adds the first component benchmark and the rules every benchmark follows.
benchmarks/, against a reference that never comes from a side, with paired differences and a margin set before the candidate runs.benchmarks/README.md: the shape of a benchmark folder and its report, and the list of benchmarks, includingkudos-pilotandknowledge-spikes, which stay on their branches (taggedevidence/kudos-pilotandevidence/knowledge-spikes).benchmarks/benchlib: shared code for run records, resampling and verdicts.benchmarks/ingestion: where KUDOS text and the indexed Marker text disagree, and the page data the page tests use.benchmarks/ingestion/opb: 1,933 tests on 221 pages, written from page images and checked blind. Marker ranks first, ahead of KUDOS text with or without its page breaks (results/report.md).docs/development.mdpoints to the benchmarks.ADR 0040 (#125) cites the report for its text-layer guard. Part of #29.
How it was tested
npm run checkin a clean worktree at the branch head (451 files, 4,710 tests pass).python -m unittest discover -s benchlib -t .(35 tests) andpython -m unittest test_reportinbenchmarks/ingestion/opb(2 tests), including "refuses_to_compare_records_with_different_references", "not_worse_needs_the_whole_interval_above_minus_margin" and "no_reference_test_is_derived_from_a_side_output".report.pyrebuildsresults/report.mdfrom the committed verdicts and reference with no difference.Main is merged in; the only conflict was the ADR index table, resolved with main's rows and the new 0003 status.
npm run checkpassesChecklist
packages/providers-*docs/architecture/modulesare updated if a contract, boundary or module changed