Repository navigation
Fix tokenizer thread sleeping forever while grammar compiles on another thread (#131) - #132
Conversation
…#131) The tokenizer thread waits indefinitely while the grammar is compiling on another thread, and nothing signals it when the compilation ends, so the model lines are never tokenized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er thread (#131) - Poll with a short timeout while the grammar is compiling instead of waiting indefinitely, since the compiling thread never signals us. - Re-queue lines skipped in UpdateTokensInRange while the grammar is compiling so they get tokenized afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The concurrency test is nondeterministic, and the ForceTokenization requeue path lacks regression coverage.
2 open findings
What changed in this PR
Fixes tokenizer stalls while a shared grammar is compiling.
Changes:
- Polls compilation state every 10 ms.
- Requeues skipped lines and adds a regression test.
| File | Description |
|---|---|
src/TextMateSharp/Model/TMModel.cs |
Prevents indefinite waits and requeues lines. |
src/TextMateSharp.Tests/Model/TMModelTests.cs |
Tests tokenization after compilation. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| tmModel.SetGrammar(grammar); | ||
|
|
||
| // let the tokenizer thread see IsCompiling and go to sleep | ||
| Thread.Sleep(300); |
There was a problem hiding this comment.
Good catch, fixed in 0270b41. The wrapper now counts how many times IsCompiling was read while true, and the test waits for at least two reads (the first wait consumes the SetGrammar signal, the second one blocks) instead of sleeping. Verified it still fails against the unfixed TMModel.
| // Re-queue the line so it is tokenized once the compilation ends. | ||
| model.InvalidateLine(lineIndex); | ||
| return nextInvalidLineIndex; |
There was a problem hiding this comment.
Added TMModel_ForceTokenization_Should_Tokenize_Lines_Skipped_While_Grammar_Is_Compiling in 0270b41: it marks a tokenized line invalid without queuing it, calls ForceTokenization while compiling is flagged, clears the flag and checks the line gets tokenized. Fails against the unfixed TMModel, passes with the fix.
- Wait until the tokenizer thread has checked IsCompiling twice before clearing it, instead of sleeping, so the test always reaches the blocking wait on the broken implementation. - Add a test for lines skipped by ForceTokenization while the grammar is compiling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #131.
Problem
When a
TMModel's tokenizer thread starts while its grammar is being compiled by another thread, it callsWaitHandle.WaitAnywith no timeout. The pending_workAvailablesignal (anAutoResetEvent) is consumed on the first loop, the next wait blocks, and the compiling thread never signals the model when it finishes. The lines stay invalid until some unrelated invalidation happens, so in an editor the document stays uncolored.Also,
UpdateTokensInRangeskipped lines while the grammar was compiling without re-queuing them, soForceTokenizationduring a compilation left them invalid.Fix
UpdateTokensInRange(same path as the existing stale-result branch).Tests
TMModel_Should_Tokenize_Lines_After_Grammar_Finishes_Compiling_On_Another_Thread, which fails deterministically on master. It uses anIGrammarwrapper with a controllableIsCompilingflag.🤖 Generated with Claude Code