Skip to content

fix(knowledge): require the frontmatter delimiter to own its line - #331

Open
harsh-thakkar7 wants to merge 1 commit into
petertzy:mainfrom
harsh-thakkar7:fix/knowledge-frontmatter-line-delimiter
Open

harsh-thakkar7 wants to merge 1 commit into
petertzy:mainfrom
harsh-thakkar7:fix/knowledge-frontmatter-line-delimiter

Conversation

@harsh-thakkar7

Copy link
Copy Markdown
Contributor

Summary

Notes whose frontmatter contains a run of dashes had their title truncated mid-value and their remaining metadata indexed as searchable body text.

Root cause

Both frontmatter readers closed the block with a substring search:

if content.startswith("---"):
    end_idx = content.find("---", 3)

find does not care where the dashes are. The closing delimiter has to be a line of its own, so a --- sitting inside a line ends the block early — and whatever follows is treated as note content.

Effect

---
title: Groceries --- weekly
tags: [home]
---

# Shopping

Buy milk.
before after
stored title Groceries Groceries --- weekly
chunk 0 weekly\ntags: [home]\n--- —
chunk 1 Buy milk. Buy milk.

The truncated title is what the knowledge base attributes every chunk to. The discarded remainder becomes a chunk in its own right, so searching the vault for tags or [home] returns this note — content that is metadata, not prose.

Three more cases the same line got wrong

The delimiter is now matched per line, so both must be a bare --- in column 0. That also corrects behaviour the substring search already had:

---- is a thematic break, not frontmatter. CommonMark reads it as a horizontal rule. A note opening with one had its real body swallowed as metadata:

----
title: Not frontmatter
----

# Heading

Body.

Before: title Not frontmatter, body dropped. After: title Heading, ---- line kept as content.

An unterminated block stays body text. YAML needs both delimiters; without a closing one there is no frontmatter, and the note used to split at EOF and lose its tail.

CRLF notes now split. find located the dashes regardless of position in the line; the per-line match requires the line to be ---, so the \r no longer matters.

Changes

backend/knowledge_logic.py — one shared helper, used by both call sites so the rule cannot drift apart again:

def _is_frontmatter_delimiter(line: str) -> bool:
    return line.rstrip() == _FRONTMATTER_DELIMITER


def _split_frontmatter(text: str) -> tuple[str, str] | None:
    lines = text.split("\n")
    if not lines or not _is_frontmatter_delimiter(lines[0]):
        return None
    for index in range(1, len(lines)):
        if _is_frontmatter_delimiter(lines[index]):
            return "\n".join(lines[1:index]), "\n".join(lines[index + 1 :])
    return None

rstrip() leaves the \r from \r\n endings, which is what keeps Windows notes working without a separate \r special case.

extract_note_title and chunk_markdown_document each lose their local block and call the helper. No other behaviour is touched — heading extraction, fenced-code handling, section paths and chunk sizing are untouched.

Testing

16 new tests in two classes.

TestFrontmatterDelimitersMustOwnTheirLine — a dash run in a metadata value; several dash runs in one block; trailing metadata never reaching chunk text; ----; --- text; unterminated block; CRLF; trailing whitespace on the closing delimiter; empty block; indented and tabbed rules; a rule partway down the note keeping the prose above it; and _split_frontmatter returning None for every non-frontmatter shape.

TestIndexingStoresCleanFrontmatterMetadata — end to end through index_knowledge_base: the stored title is intact, querying for the leaked metadata returns nothing, querying for real body text returns the note under the right title, and re-indexing a rewritten note updates the stored title.

8 of 8 mutants killed, covering: reverting to the substring search, accepting any 3+ dash run, dropping the opening-delimiter requirement, an off-by-one on the remainder slice, splitting an unterminated block at EOF, tolerating an indented delimiter, starting the closing scan at line 0, and keeping the closing line inside the frontmatter.

14 of the new tests fail against the previous implementation, so the suite pins the defect rather than just the fix.

Two mutants were dropped as provably equivalent rather than papered over: str.rstrip() already strips \r, and the only caller strips the remainder anyway. An earlier draft of the helper carried a redundant .rstrip("\r") for that reason; it is gone.

Verification

  • pytest tests/test_knowledge_logic.py — 42 passed (26 on main; this PR adds 16)
  • pytest tests — unchanged failures, +16 passing
  • ruff check . — clean
  • ruff format --check — clean

Both `extract_note_title` and `chunk_markdown_document` closed the frontmatter
block with `content.find("---", 3)`, which matches a run of dashes anywhere in
the text rather than at the start of a line. Any `---` inside the block ends it
early, and the remainder of the metadata is treated as body text.

A note with a dash run in a metadata value:

    ---
    title: Groceries --- weekly
    tags: [home]
    ---
    # Shopping
    Buy milk.

produced the title `Groceries` and indexed a chunk reading
`weekly\ntags: [home]\n---` — the rest of the metadata became searchable text in
the knowledge base, attributed to the wrong title.

The delimiters are now matched per line, so both must be a bare `---` in column
0. That also fixes three neighbouring cases the substring search got wrong:

- `----` is a CommonMark thematic break, not frontmatter. A note opening with
  one had its real body swallowed as metadata.
- An unterminated block stays body text instead of silently splitting at EOF.
- CRLF notes now split, because the search no longer depends on where the
  dashes happen to fall inside the line.

Both call sites share one `_split_frontmatter` helper so the rule cannot drift
apart again.

Adds 16 tests covering the dash-run cases, the thematic-break and unterminated
variants, CRLF, indented delimiters, and an end-to-end index/query check that
the stored title and chunk text are free of stray metadata. 8 of 8 mutants
killed; 14 of the new tests fail against the previous implementation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant