Make tokenization thread-safe when a grammar is shared by several threads

[danipen]

Related to #131 (follow-up to #132).

Problem

Grammar is not thread-safe, but the same IGrammar instance is often used from several threads: for example, several AvaloniaEdit editors with the same language, or the old and new TMModel tokenizer threads when the editor document is replaced (the scenario in #131).

• Lazy compilation: Tokenize compiles the grammar on first use (if (_rootId == null) GenerateRootId()). Two threads can compile at the same time, both registering rules into a plain Dictionary and calling SetId on the shared raw rules. A thread can then get a root id whose rule is not registered yet, and TokenizeLine returns null.
• Mutable pattern caches: rules build their RegExpSourceList / CompiledRule caches lazily, and BeginEndRule / BeginWhileRule call SetSource on the shared list on every scan with a back-referenced end pattern. Concurrent use throws InvalidOperationException: Collection was modified.

So a lock around GenerateRootId alone is not enough: with only that lock, the new test still fails with the second error.

Fix

Serialize Grammar.Tokenize per grammar instance with a private lock. The uncontended cost is negligible compared to tokenizing a line. The grammar never calls back into the caller while holding the lock (TMModel calls the tokenizer outside its own lock), so this does not introduce lock-ordering issues.

IsCompiling is kept for API compatibility; TMModel still uses it (with the polling fix from #132).

Tests

• First commit adds Concurrent_First_Tokenization_Should_Compile_Grammar_Once_And_Generate_Same_Tokens: 8 threads behind a Barrier tokenize the same line with a fresh C# grammar, 20 iterations, comparing against single-threaded tokens. It fails on master on the first iteration (null results).
• Second commit is the fix; the full suite passes.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The locking scope covers both tokenization paths, and the regression test exercises the reported race.

0 open findings

What changed in this PR

Serializes tokenization per grammar instance to prevent races during lazy compilation and mutable rule-cache access.

Changes:

• Adds a private tokenization lock around both tokenization APIs.
• Adds a concurrent first-tokenization regression test.

File	Description
src/​TextMateSharp/​Internal/​Grammars/​Grammar.cs	Serializes tokenization through TokenizeCore.
src/​TextMateSharp.Tests/​Internal/​Grammars/​GrammarTests.cs	Verifies concurrent tokenization produces consistent results without errors.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.