Add heap-based BPE merge path for large inputs (>128 bytes) - #7580

Merged

tarekgh merged 16 commits into

mainfrom

copilot/fix-byte-pair-encoding-performance

Feb 14, 2026

Commits on Feb 12, 2026

Commits on Feb 13, 2026

Remove timing-based performance test to prevent CI flakiness

Copilot
and
stephentoub
committed
604bc90
Add explanatory comments for threshold and heap capacity choices

Copilot
and
stephentoub
committed
8c0543d
Apply suggestion from @stephentoub
stephentoub
authored
058c10b
Use default capacity for PriorityQueue instead of pre-allocating to max

Copilot
and
stephentoub
committed
93697d0
Apply suggestion from @stephentoub
stephentoub
authored
507035a
Update src/Microsoft.ML.Tokenizers/Utils/BytePairEncoder.cs

stephentoub
and
Copilot
authored
394658e
Update src/Microsoft.ML.Tokenizers/Utils/BytePairEncoder.cs

stephentoub
and
Copilot
authored
f339af1
Fix CompareTo ordering, revert BOM changes, and update test comments

Copilot
and
stephentoub
committed
a43a57e
Add parameterless PriorityQueue constructor and use it in BytePairEncodeLarge

Copilot
and
tarekgh
committed
bd86b94
Add comment noting that CurRank assumes rank == token Id (Tiktoken-specific)

Copilot
and
tarekgh
committed
8984a3d
Remove stackalloc for State array; always use ArrayPool since method is only called for >128 bytes

Copilot
and
tarekgh
committed
f43f8ee
Replace List+ToArray with ArrayPool for result buffer in BytePairEncodeLarge

Copilot
and
tarekgh
committed
99c5107