Skip to content

Phase 11: test cues on facts, and close the phase - #41

Merged
rohan-hotdata merged 1 commit into
mainfrom
phase-11-cue-test
Oct 25, 2026
Merged

rohan-hotdata merged 1 commit into
mainfrom
phase-11-cue-test

Conversation

@rohan-hotdata

Copy link
Copy Markdown
Collaborator

This PR adds a test of cues on facts to the benchmark, and closes phase 11. The configuration does not change.

How

  • bench/prompts.py: the extractor variant detail-cues. It is detail, plus one to three questions that each fact answers, returned in a schema with cues. Variant gains cues and a schema property.
  • bench/e2e.py: the extractor puts the cues of the reply on each Fact. --recall-ranking recalls with another ranking than the one that captured, so one kept memory is recalled with both rankings. The result file records it as memory.recall_ranking.
  • bench/diagnose.py: decides the cause of a miss with the recall ranking of the run.
  • docs/benchmarks.md: the section "Cues on facts", and the variant table. bench/README.md: both options.

No change to src/hotmemory.

Why

Until now the benchmark extractor wrote no cues, so the cue arm of every fused run was empty. The open question of phase 11 was whether cues make the fused ranking better than the vector ranking on facts.

Memory Recall Sample A Sample B
detail vector 31 33
detail fused 26 32
detail-cues vector 28 32
detail-cues fused 30 34

Cues lift the fused ranking (+4, +2), but no arm beats detail with the vector ranking on both samples. The cue prompt writes about 22 percent fewer facts. The diagnosis of sample B found 0 of 20 and 1 of 22 misses ranked below k, so the ranking has little to gain on these small stores.

Verification

  • make verify: ruff, mypy, 307 tests passed. New tests cover the cue variant, --recall-ranking in Config, and that run_question recalls through the recall-ranking store.
  • The eight runs and two diagnoses above, on the local stack, $6.94 in total. The detail capture replayed from the cache at $0.
  • Not run: make integration, because no driver changed.

Notes

  • A store where the answer competes with thousands of facts, such as a BEAM conversation, can give cues and the fused ranking more to gain. Not measured.
  • recall uses one ranking for facts and chunks, and chunks have no cues, so the production configuration with the fused ranking was not tried.

Add the extractor variant detail-cues, which asks for one to three
questions that each fact answers and stores them as its cues, and the
runner option --recall-ranking, which recalls a kept memory with another
ranking than the one that captured it. bench/diagnose.py decides the cause
of a miss with the recall ranking.

On samples A and B, facts only, cues lifted the fused ranking by 4 and 2
questions, but no arm beat detail with the vector ranking (31 and 33
against 30 and 34). The configuration does not change. docs/benchmarks.md
records the runs and the diagnosis of sample B.
@rohan-hotdata
rohan-hotdata requested a review from a team as a code owner October 25, 2026 19:22
@rohan-hotdata
rohan-hotdata requested review from shefeek-jinnah and removed request for a team October 25, 2026 19:22
@rohan-hotdata
rohan-hotdata merged commit 6eacf0d into main Oct 25, 2026
7 checks passed
@rohan-hotdata
rohan-hotdata deleted the phase-11-cue-test branch October 25, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant