Skip to content

Fix MESSI exact search with adaptive SAX and strided queries - #81

Merged
MChatzakis merged 6 commits into
mainfrom
fix/messi-topk-duplicate-results
Oct 12, 2026
Merged

MChatzakis merged 6 commits into
mainfrom
fix/messi-topk-duplicate-results

Conversation

@sophisid

@sophisid sophisid commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

MESSI could return duplicate neighbors or partial squared-L2 distances, and raw SIFT exposed an additional adaptive-breakpoint error: a low positive PAA value was assigned to the highest SAX bin. That invalid lower bound pruned true nearest neighbors. On 20K SIFT vectors / 100 queries / k=10, the previous PR head returned 19–33 wrong top-k distances per run; one query returned a nearest distance of 27102 instead of 25379.

This fixes the affected exact-search paths:

  • Deduplicate queued positions, retain the best refinement, and accept only strictly improving top-k candidates.
  • Encode PAA values by searching the ordered cuts directly. Both tail bins, shifted cuts, repeated cuts and exact boundaries are handled without reading the previous cardinality's table or assuming Gaussian signs.
  • Copy strided Python MESSI queries to contiguous float32 storage and reject mismatched dimensions / unbuilt indexes. Both the integer-k and SearchConfig APIs are covered; ordinary fvecs views previously misread row headers as vector coordinates.
  • Preserve inclusive range pruning and continue refinement past partial sums equal to the radius, including radius zero.
  • Guard the optional Coconut Python binding so BUILD_COCONUT=OFF produces an importable module.

Regression tests compare every returned distance and ID/distance pair against exhaustive ground truth and reject duplicates. No recall threshold below 100% is accepted. Exact ties permit any distinct IDs at the correct distances. Coverage includes Gaussian/adaptive modes, one/four workers, k=1/10/100/1000, repeated queries, streaming inserts, second-index isolation, SAX lower bounds, NumPy layouts, and inclusive range boundaries. tests/check_messi_sift.py provides a reproducible optional audit against FAISS IndexFlatL2 and the published SIFT1M ground truth.

Validation completed locally: isolated Release C++/Python build; all 21 focused CTest cases passed; 8 adaptive/shared-index cases passed; 5 shared-SAX Astronomy checks passed; Coconut streaming and range checks passed with Coconut enabled. On the original 20K SIFT reproduction, wrong distances and record lower-bound violations both fell to zero. Twelve real-SIFT range checks also passed with exact comparisons.

The full 100-query SIFT1M static/streaming audit completed: all 160 cases passed, checking 4,444,000 returned ID/distance pairs, with zero wrong distances, invalid pairs or duplicate rows. Full-dataset references match the published ground truth. Coverage includes static 20K/1M builds and streaming 750K→875K→1M, k=1/10/100/1000, both breakpoint modes, one/four workers and two query repetitions, using unchanged raw descriptors and header-strided fvecs queries. Independent streaming groups ran in parallel. The original validator's final two checks were rerun in a fresh process after swapped-memory I/O waits; completed cases were preserved and no performance claims are made.

The regular C++ regressions are covered by Simple CI; the Python layout test additionally requires BUILD_PYTHON=ON and NumPy. GitHub Release/Debug, dry-run and CodeQL checks passed; the complete Simple CI suite is still running.

sophisid and others added 4 commits October 10, 2026 22:03
ts_euclidean_distance_SIMD stops once its partial sum reaches the bound and
returns that partial sum. The top-k scans accepted results with
`dist <= knn[k-1]`, so a series whose abandoned partial sum equalled the current
bound was inserted with that under-estimated distance, and its position was
then reported next to a distance that belongs to another series. With the
position-based de-duplication this wrong entry could also block the real
series, because it already occupied the same position slot.

Compare with `<` in the approximate scan and in the MESSI L2 leaf scan, so a
distance that is not strictly better than the current k-th is never inserted.

On 200k clustered 128-d vectors, k=10: returned (id, distance) pairs whose
distance does not match the series: 8 of 50000 before, 0 after; fraction of the
true top-10 returned: 0.9998 -> 1.0000. On 1M vectors the same: 1.0000.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: sophisid <55842445+sophisid@users.noreply.github.com>
pqueue_bsf_insert_no_dup_position returned as soon as the position was
already in the k-NN queue. An earlier entry for that position can hold a
larger, early-abandoned distance, so the correct smaller distance was dropped
and a true neighbour was missing from the result. test_Messi_L2Square failed
with "Indices mismatch" for the k=100 configs (e.g. query 22, rank 2: expected
label 9217, got 37688).

Keep only the best distance per position: if the position is queued and the
new distance is smaller, remove the old entry and insert the new one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…op-k fix

main reverted 3871490 (#79). Restore that change here so merging this PR brings
the fix back together with the strict comparison and the queue replacement.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sophisid
sophisid requested a review from MChatzakis October 11, 2026 23:22
@sophisid sophisid changed the title Fix/messi topk duplicate results Fix MESSI exact search with adaptive SAX and strided queries Oct 12, 2026
@MChatzakis
MChatzakis merged commit d98008d into main Oct 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants