Repository navigation
Phase 11: test cues on facts, and close the phase - #41
Merged
Merged
Conversation
Add the extractor variant detail-cues, which asks for one to three questions that each fact answers and stores them as its cues, and the runner option --recall-ranking, which recalls a kept memory with another ranking than the one that captured it. bench/diagnose.py decides the cause of a miss with the recall ranking. On samples A and B, facts only, cues lifted the fused ranking by 4 and 2 questions, but no arm beat detail with the vector ranking (31 and 33 against 30 and 34). The configuration does not change. docs/benchmarks.md records the runs and the diagnosis of sample B.
rohan-hotdata
requested review from
shefeek-jinnah
and removed request for
a team
October 25, 2026 19:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a test of cues on facts to the benchmark, and closes phase 11. The configuration does not change.
How
bench/prompts.py: the extractor variantdetail-cues. It isdetail, plus one to three questions that each fact answers, returned in a schema withcues.Variantgainscuesand aschemaproperty.bench/e2e.py: the extractor puts the cues of the reply on eachFact.--recall-rankingrecalls with another ranking than the one that captured, so one kept memory is recalled with both rankings. The result file records it asmemory.recall_ranking.bench/diagnose.py: decides the cause of a miss with the recall ranking of the run.docs/benchmarks.md: the section "Cues on facts", and the variant table.bench/README.md: both options.No change to
src/hotmemory.Why
Until now the benchmark extractor wrote no cues, so the cue arm of every fused run was empty. The open question of phase 11 was whether cues make the fused ranking better than the vector ranking on facts.
detaildetaildetail-cuesdetail-cuesCues lift the fused ranking (+4, +2), but no arm beats
detailwith the vector ranking on both samples. The cue prompt writes about 22 percent fewer facts. The diagnosis of sample B found 0 of 20 and 1 of 22 misses ranked below k, so the ranking has little to gain on these small stores.Verification
make verify: ruff, mypy, 307 tests passed. New tests cover the cue variant,--recall-rankinginConfig, and thatrun_questionrecalls through the recall-ranking store.detailcapture replayed from the cache at $0.make integration, because no driver changed.Notes
recalluses one ranking for facts and chunks, and chunks have no cues, so the production configuration with the fused ranking was not tried.