Solr Cell: index Tika 4.x tk:chunks into a dense_vector field - #4863
Draft
epugh wants to merge 5 commits into
Draft
Solr Cell: index Tika 4.x tk:chunks into a dense_vector field#4863epugh wants to merge 5 commits into
tk:chunks into a dense_vector field#4863epugh wants to merge 5 commits into
Conversation
…xml. This is mostly a "hey, look, I got us to tika 4", but it feels like its still trying to push Tika 4 into a tika 3 approach, when Tika wants to be different.
Adds a new tikaserver.chunks parameter to the ExtractingRequestHandler that reads Tika 4.x's tk:chunks -- text chunks with precomputed embedding vectors, produced by a server-side Tika Server embedding metadata filter such as openai-embedding-filter -- and indexes one Solr document per chunk into a dense_vector field, enabling kNN search over documents extracted with Solr Cell. Demonstrated end-to-end in a new BATS test (test_extraction.bats) using a small stdlib-only mock OpenAI-compatible embeddings server, and covered by new TikaServerExtractionBackendTest cases against a live Tika 4.0.0 container. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Tika 4.x's
tika-inferencemodule can split a parsed document into semantically-bounded chunks (splitting on Markdown headings/paragraphs) and attach a precomputed embedding vector to each, via a server-side embedding filter (e.g.openai-embedding-filter). This PR teaches Solr Cell (ExtractingRequestHandler) to read those chunks back out and index them directly.Solution
Mostly AI generated... I need to better understand what Tika 4 views as it's place in the world if we aren't just capturing attributes from a tree structure, and indeed, appear to have dumped the tree concept in favor of a markdown file!
Maybe these days, all that rich parsing and extraction we used to have has turned into "just give me markdown", but what about images? What about hOCR?
Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.