Skip to content

fix(tag-migration): skip memories that are already fully tagged - #351

Merged
EyJunge1 merged 1 commit into
tickernelz:mainfrom
share121:fix/tag-migration-skip-tagged
Oct 2, 2026
Merged

EyJunge1 merged 1 commit into
tickernelz:mainfrom
share121:fix/tag-migration-skip-tagged

Conversation

@share121

@share121 share121 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

handleRunTagMigrationBatch re-computes the content vector and tags vector for every memory in the current window, even when a memory already has tags and a tags vector. This endpoint walks the whole project list one window at a time (the web UI drives it with batchSize: 3), so on a store with, say, 732 project memories and only 1 untagged memory, a full run still re-embeds all 732 memories — twice per memory (content + tags) — taking several minutes of local CPU to tag one memory.

Re-embedding an already-tagged memory cannot change anything: its tags are unchanged, so the tag vector is identical, and the content vector is identical. Skipping those memories makes the endpoint proportional to the number of memories that actually need work.

Change

In handleRunTagMigrationBatch, short-circuit a memory that is already fully migrated (has tags and a tags vector): count it as processed and move on without calling the embedding service or touching the shard.

// A memory that already has tags and a tags vector is fully migrated:
// re-embedding it produces the same vector and changes nothing. ...
if (currentTags.length > 0 && m.tags_vector != null) {
  migrationProgress.processed++;
  continue;
}

Memories that have tags but no tags vector are still processed, so a missing tags vector is still rebuilt. Untagged memories (and the tag-generation soft-failure path) are unchanged.

Why this is safe

  • The skipped memories' tags and both vectors are provably unchanged by the run, so their stored state is already correct.
  • The processed/cursor accounting is preserved (a skipped memory is counted as processed and the window still advances), so the progress UI and the "retry after completion" reset keep working.
  • Only the write path for memories that need work is exercised; nothing else in the batch changes.

Test plan

  • bun test tests/tag-migration-skip-tagged.test.ts — a fully-migrated memory is not re-vectorized (no embedWithTimeout call for its content, no updateVector), while the untagged memory still gets tagged + vectorized and both are counted as processed.
  • bun test tests/tag-migration-soft-failure.test.ts — existing soft-failure behavior unchanged.
  • bunx tsc --noEmit
  • eslint --max-warnings=0 on the changed file

Context

Follows #319 (soft-failure accounting) and #333 (force the save_tags tool call). Those made tagging correct; this makes a run proportional to the work left to do instead of to the whole store. On a 732-memory store with a single untagged memory this turns a multi-minute run into a few seconds.

(Also mentioned in #350 as a related annoyance — separate from the Windows engine-migration failure reported there.)

handleRunTagMigrationBatch re-computed the content vector and tags vector
for every memory in the current window, even when a memory already had
tags and a tags vector. Re-embedding an already-tagged memory cannot
change anything -- its tags are unchanged, so the tags vector is
identical, and the content vector is identical.

Because the endpoint walks the whole project list one window at a time
(the web UI drives it with batchSize: 3), a store with hundreds of
memories and a single untagged memory still re-embeds the whole store
twice per memory (content + tags), taking minutes of local CPU to tag one
memory.

Short-circuit a memory that already has tags AND a tags vector: count it
as processed and move on without calling the embedding service or
touching the shard. Memories with tags but no tags vector are still
processed, so a missing tags vector is still rebuilt; untagged memories
and the soft-failure path are unchanged.
@EyJunge1
EyJunge1 merged commit 5ad0b6d into tickernelz:main Oct 2, 2026
6 checks passed
@EyJunge1 EyJunge1 mentioned this pull request Oct 2, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants