Repository navigation
Fix MESSI exact search with adaptive SAX and strided queries - #81
Merged
Merged
Conversation
ts_euclidean_distance_SIMD stops once its partial sum reaches the bound and returns that partial sum. The top-k scans accepted results with `dist <= knn[k-1]`, so a series whose abandoned partial sum equalled the current bound was inserted with that under-estimated distance, and its position was then reported next to a distance that belongs to another series. With the position-based de-duplication this wrong entry could also block the real series, because it already occupied the same position slot. Compare with `<` in the approximate scan and in the MESSI L2 leaf scan, so a distance that is not strictly better than the current k-th is never inserted. On 200k clustered 128-d vectors, k=10: returned (id, distance) pairs whose distance does not match the series: 8 of 50000 before, 0 after; fraction of the true top-10 returned: 0.9998 -> 1.0000. On 1M vectors the same: 1.0000. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: sophisid <55842445+sophisid@users.noreply.github.com>
pqueue_bsf_insert_no_dup_position returned as soon as the position was already in the k-NN queue. An earlier entry for that position can hold a larger, early-abandoned distance, so the correct smaller distance was dropped and a true neighbour was missing from the result. test_Messi_L2Square failed with "Indices mismatch" for the k=100 configs (e.g. query 22, rank 2: expected label 9217, got 37688). Keep only the best distance per position: if the position is queued and the new distance is smaller, remove the old entry and insert the new one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MESSI could return duplicate neighbors or partial squared-L2 distances, and raw SIFT exposed an additional adaptive-breakpoint error: a low positive PAA value was assigned to the highest SAX bin. That invalid lower bound pruned true nearest neighbors. On 20K SIFT vectors / 100 queries / k=10, the previous PR head returned 19–33 wrong top-k distances per run; one query returned a nearest distance of 27102 instead of 25379.
This fixes the affected exact-search paths:
Regression tests compare every returned distance and ID/distance pair against exhaustive ground truth and reject duplicates. No recall threshold below 100% is accepted. Exact ties permit any distinct IDs at the correct distances. Coverage includes Gaussian/adaptive modes, one/four workers, k=1/10/100/1000, repeated queries, streaming inserts, second-index isolation, SAX lower bounds, NumPy layouts, and inclusive range boundaries.
tests/check_messi_sift.pyprovides a reproducible optional audit against FAISS IndexFlatL2 and the published SIFT1M ground truth.Validation completed locally: isolated Release C++/Python build; all 21 focused CTest cases passed; 8 adaptive/shared-index cases passed; 5 shared-SAX Astronomy checks passed; Coconut streaming and range checks passed with Coconut enabled. On the original 20K SIFT reproduction, wrong distances and record lower-bound violations both fell to zero. Twelve real-SIFT range checks also passed with exact comparisons.
The full 100-query SIFT1M static/streaming audit completed: all 160 cases passed, checking 4,444,000 returned ID/distance pairs, with zero wrong distances, invalid pairs or duplicate rows. Full-dataset references match the published ground truth. Coverage includes static 20K/1M builds and streaming 750K→875K→1M, k=1/10/100/1000, both breakpoint modes, one/four workers and two query repetitions, using unchanged raw descriptors and header-strided fvecs queries. Independent streaming groups ran in parallel. The original validator's final two checks were rerun in a fresh process after swapped-memory I/O waits; completed cases were preserved and no performance claims are made.
The regular C++ regressions are covered by Simple CI; the Python layout test additionally requires BUILD_PYTHON=ON and NumPy. GitHub Release/Debug, dry-run and CodeQL checks passed; the complete Simple CI suite is still running.