fix(knowledge): require the frontmatter delimiter to own its line - #331
Open
harsh-thakkar7 wants to merge 1 commit into
Open
harsh-thakkar7 wants to merge 1 commit into
harsh-thakkar7 wants to merge 1 commit into
Conversation
Both `extract_note_title` and `chunk_markdown_document` closed the frontmatter
block with `content.find("---", 3)`, which matches a run of dashes anywhere in
the text rather than at the start of a line. Any `---` inside the block ends it
early, and the remainder of the metadata is treated as body text.
A note with a dash run in a metadata value:
---
title: Groceries --- weekly
tags: [home]
---
# Shopping
Buy milk.
produced the title `Groceries` and indexed a chunk reading
`weekly\ntags: [home]\n---` — the rest of the metadata became searchable text in
the knowledge base, attributed to the wrong title.
The delimiters are now matched per line, so both must be a bare `---` in column
0. That also fixes three neighbouring cases the substring search got wrong:
- `----` is a CommonMark thematic break, not frontmatter. A note opening with
one had its real body swallowed as metadata.
- An unterminated block stays body text instead of silently splitting at EOF.
- CRLF notes now split, because the search no longer depends on where the
dashes happen to fall inside the line.
Both call sites share one `_split_frontmatter` helper so the rule cannot drift
apart again.
Adds 16 tests covering the dash-run cases, the thematic-break and unterminated
variants, CRLF, indented delimiters, and an end-to-end index/query check that
the stored title and chunk text are free of stray metadata. 8 of 8 mutants
killed; 14 of the new tests fail against the previous implementation.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Notes whose frontmatter contains a run of dashes had their title truncated mid-value and their remaining metadata indexed as searchable body text.
Root cause
Both frontmatter readers closed the block with a substring search:
finddoes not care where the dashes are. The closing delimiter has to be a line of its own, so a---sitting inside a line ends the block early — and whatever follows is treated as note content.Effect
GroceriesGroceries --- weeklyweekly\ntags: [home]\n---Buy milk.Buy milk.The truncated title is what the knowledge base attributes every chunk to. The discarded remainder becomes a chunk in its own right, so searching the vault for
tagsor[home]returns this note — content that is metadata, not prose.Three more cases the same line got wrong
The delimiter is now matched per line, so both must be a bare
---in column 0. That also corrects behaviour the substring search already had:----is a thematic break, not frontmatter. CommonMark reads it as a horizontal rule. A note opening with one had its real body swallowed as metadata:Before: title
Not frontmatter, body dropped. After: titleHeading,----line kept as content.An unterminated block stays body text. YAML needs both delimiters; without a closing one there is no frontmatter, and the note used to split at EOF and lose its tail.
CRLF notes now split.
findlocated the dashes regardless of position in the line; the per-line match requires the line to be---, so the\rno longer matters.Changes
backend/knowledge_logic.py— one shared helper, used by both call sites so the rule cannot drift apart again:rstrip()leaves the\rfrom\r\nendings, which is what keeps Windows notes working without a separate\rspecial case.extract_note_titleandchunk_markdown_documenteach lose their local block and call the helper. No other behaviour is touched — heading extraction, fenced-code handling, section paths and chunk sizing are untouched.Testing
16 new tests in two classes.
TestFrontmatterDelimitersMustOwnTheirLine— a dash run in a metadata value; several dash runs in one block; trailing metadata never reaching chunk text;----;--- text; unterminated block; CRLF; trailing whitespace on the closing delimiter; empty block; indented and tabbed rules; a rule partway down the note keeping the prose above it; and_split_frontmatterreturningNonefor every non-frontmatter shape.TestIndexingStoresCleanFrontmatterMetadata— end to end throughindex_knowledge_base: the stored title is intact, querying for the leaked metadata returns nothing, querying for real body text returns the note under the right title, and re-indexing a rewritten note updates the stored title.8 of 8 mutants killed, covering: reverting to the substring search, accepting any 3+ dash run, dropping the opening-delimiter requirement, an off-by-one on the remainder slice, splitting an unterminated block at EOF, tolerating an indented delimiter, starting the closing scan at line 0, and keeping the closing line inside the frontmatter.
14 of the new tests fail against the previous implementation, so the suite pins the defect rather than just the fix.
Two mutants were dropped as provably equivalent rather than papered over:
str.rstrip()already strips\r, and the only caller strips the remainder anyway. An earlier draft of the helper carried a redundant.rstrip("\r")for that reason; it is gone.Verification
pytest tests/test_knowledge_logic.py— 42 passed (26 onmain; this PR adds 16)pytest tests— unchanged failures, +16 passingruff check .— cleanruff format --check— clean