Skip to content

Fix Coconut disk streaming with bounded LSM updates - #82

Merged
sophisid merged 1 commit into
mainfrom
fix/coconut-disk-streaming
Oct 21, 2026
Merged

sophisid merged 1 commit into
mainfrom
fix/coconut-disk-streaming

Conversation

@sophisid

@sophisid sophisid commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Coconut writes its initial index to disk but keeps every subsequent insertion in an unbounded RAM buffer. This change persists bounded memtables and compacts immutable sorted runs, so streaming updates actually remain disk-backed.

The implementation follows the materialized Coconut-LSM-Full design: external-sort bulk loading, sortable-SAX run trees, Gaussian SAX pruning, asynchronous binary compaction, and exact squared-L2 top-k/range search with stable stream IDs. Independent queries share bounded candidate reads while retaining separate exact heaps. Recent-window top-k, disk/merge/query counters, Python bindings, demos, a streaming benchmark, and regression tests are included. Completed insertion timing includes waitForMerges().

This is DaiSy's paper-based implementation, not the authors' executable. Raw vectors live in disk runs; resident SAX summaries still require O(N) RAM. Indexes are temporary POSIX files, without reopening/WAL/recovery or concurrent caller inserts/searches. Adaptive breakpoints and DTW remain unsupported.

Validation on current main (d98008d):

  • Clean Release build with Python, tests, demos and benchmarks enabled; streaming benchmark smoke test passed.
  • Coconut CTest selection: 15 passed, 9 explicitly skipped invalid Astronomy range fixtures, 0 failures. Includes exact physical neighbors after compaction, scalar/batched equivalence, safe-bound parity, corruption and shape checks.
  • BUILD_COCONUT=OFF: separate library/Python build, successful import, no Coconut class or targets.
  • The same Coconut algorithm sources completed the frozen DEEP 100M→1B trajectory in 50M increments: all 76 independent query-batch validations passed with zero neighbor gaps.
  • The complete 295-case repository CTest suite is still running. Several ParIS cases reached the local 180-second per-test limit; those timeouts are not Coconut test failures. This PR stays draft pending review of the broader suite and CI.

Based directly on current main. Existing MESSI/SAX fixes are preserved; experiment scripts, datasets, logs and figures are excluded. /scripts/experiments/ is already ignored on main.

@sophisid
sophisid marked this pull request as ready for review October 21, 2026 10:57
@sophisid
sophisid merged commit c6cac82 into main Oct 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant