Description
With default options, BertTokenizer deletes every character in the Unicode symbol categories (Sm, Sc, Sk, So) instead of tokenizing it. That covers the ASCII characters $ + < = > ^ ` | ~ and non-ASCII ones such as °, © and emoji. Nothing is emitted, not even [UNK], and nothing warns that input was lost.
Hugging Face's BERT basic tokenizer keeps all of them. It treats the ASCII symbols as punctuation — _is_punctuation counts every printable non-alphanumeric ASCII character — splits them into their own tokens, and passes any other character to WordPiece, which yields a vocabulary token such as °c, © or £10, or [UNK] when the character is not in the vocabulary, as with emoji.
That rule is not specific to HF. It comes from the original BERT release, whose tokenization.py names three of the affected characters outright:
# We treat all non-letter/number ASCII as punctuation.
# Characters such as "^", "$", and "`" are not in the Unicode
# Punctuation class but we treat them as punctuation anyways, for
# consistency.
BertTokenizer's own remarks state that its implementation "is based on the original Bert implementation in the Hugging Face Transformers library".
If the current pattern is deliberate — a decision that BertTokenizer's default should follow Unicode's \p{P} rather than BERT's own _is_punctuation rule — please share the reasoning, and I'll close this.
Versions
- Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, with identical results
- .NET 10.0.12, macOS arm64
- Vocabulary:
sentence-transformers/all-MiniLM-L6-v2 vocab.txt (uncased BERT)
- Reference: Hugging Face
tokenizers 0.21.1, BertWordPieceTokenizer(vocab, lowercase=True)
- Cosine column:
onnxruntime 1.22.0, mean pooling over the token ids
Reproduce
using Microsoft.ML.Tokenizers;
var vocab = File.ReadAllLines("vocab.txt");
var tokenizer = BertTokenizer.Create("vocab.txt", new BertOptions());
foreach (var input in new[]
{
"costs $5 + tax = $6", "a+b=c", "if (a < b && c > d) { x = y | z; }",
"The temperature was 5 °C", "\U0001F4AF great product \U0001F44D", "\U0001F680",
})
Console.WriteLine($"{input} -> {string.Join(' ', tokenizer.EncodeToIds(input).Select(id => vocab[id]))}");Expected vs actual
| input |
actual |
expected (HF tokenizers) |
cosine ¹ |
costs $5 + tax = $6 |
costs 5 tax 6 |
costs $ 5 + tax = $ 6 |
0.915 |
a+b=c |
a b c |
a + b = c |
0.649 |
if (a < b && c > d) { x = y | z; } |
if ( a b & & c d ) { x y z ; } |
if ( a < b & & c > d ) { x = y | z ; } |
0.912 |
The temperature was 5 °C |
the temperature was 5 c |
the temperature was 5 °c |
0.905 |
💯 great product 👍 |
great product |
[UNK] great product [UNK] |
0.701 |
🚀 |
(nothing) |
[UNK] |
— |
[CLS]/[SEP] omitted. ¹ Cosine between all-MiniLM-L6-v2 mean-pooled embeddings of the actual and expected token ids, run through ONNX Runtime.
No built-in BertOptions setting preserves BERT basic-tokenization semantics while correcting this behavior. BertOptions.PreTokenizer can be overridden with a custom pre-tokenizer, as shown below.
Cause
With ApplyBasicTokenization = true (the default), BertTokenizer uses PreTokenizer.CreateWordOrPunctuation, whose pattern is:
A symbol matches neither alternative, so the regex steps over it and it never reaches WordPiece.
A pattern that matches HF
Splitting off Unicode punctuation plus the ASCII symbols HF counts as punctuation, and keeping every other run of non-whitespace intact, reproduces HF's tokens:
[\p{P}$+<=>^`|~]|[^\s\p{P}$+<=>^`|~]+
Passing this through BertOptions.PreTokenizer = new RegexPreTokenizer(regex, specialTokens) gives tokens identical to HF tokenizers on 18 of the 21 inputs we checked (those above, plus capitalisation, punctuation, accents with RemoveNonSpacingMarks = true, CJK, a 109-token paragraph and a 323-token passage). The three that still differ contain a bare \n or \t, the separate issue #7724. That option is also the workaround until the default changes.
One caveat for anyone applying it as a workaround: HF's _clean_text deletes format characters (Cf) before tokenizing, while BertNormalizer deletes only Cc. The default pattern happens to split at a zero-width space and this one does not, so against HF "zero\u200bwidth" moves from 0.800 to 0.250 and "soft\u00adhyphen" from 0.554 to 0.091. Deleting Cf alongside brings both to 1.000.
BertTokenizer is the only caller of CreateWordOrPunctuation in this repository, but the method is public and documents its pattern. Changing only the default that BertTokenizer picks may be safer than changing that method.
Description
With default options,
BertTokenizerdeletes every character in the Unicode symbol categories (Sm,Sc,Sk,So) instead of tokenizing it. That covers the ASCII characters$ + < = > ^ ` | ~and non-ASCII ones such as°,©and emoji. Nothing is emitted, not even[UNK], and nothing warns that input was lost.Hugging Face's BERT basic tokenizer keeps all of them. It treats the ASCII symbols as punctuation —
_is_punctuationcounts every printable non-alphanumeric ASCII character — splits them into their own tokens, and passes any other character to WordPiece, which yields a vocabulary token such as°c,©or£10, or[UNK]when the character is not in the vocabulary, as with emoji.That rule is not specific to HF. It comes from the original BERT release, whose
tokenization.pynames three of the affected characters outright:BertTokenizer's own remarks state that its implementation "is based on the original Bert implementation in the Hugging Face Transformers library".If the current pattern is deliberate — a decision that
BertTokenizer's default should follow Unicode's\p{P}rather than BERT's own_is_punctuationrule — please share the reasoning, and I'll close this.Versions
sentence-transformers/all-MiniLM-L6-v2vocab.txt(uncased BERT)tokenizers0.21.1,BertWordPieceTokenizer(vocab, lowercase=True)onnxruntime1.22.0, mean pooling over the token idsReproduce
Expected vs actual
tokenizers)costs $5 + tax = $6costs 5 tax 6costs $ 5 + tax = $ 6a+b=ca b ca + b = cif (a < b && c > d) { x = y | z; }if ( a b & & c d ) { x y z ; }if ( a < b & & c > d ) { x = y | z ; }The temperature was 5 °Cthe temperature was 5 cthe temperature was 5 °c💯 great product 👍great product[UNK] great product [UNK]🚀[UNK][CLS]/[SEP]omitted. ¹ Cosine betweenall-MiniLM-L6-v2mean-pooled embeddings of the actual and expected token ids, run through ONNX Runtime.No built-in
BertOptionssetting preserves BERT basic-tokenization semantics while correcting this behavior.BertOptions.PreTokenizercan be overridden with a custom pre-tokenizer, as shown below.Cause
With
ApplyBasicTokenization = true(the default),BertTokenizerusesPreTokenizer.CreateWordOrPunctuation, whose pattern is:A symbol matches neither alternative, so the regex steps over it and it never reaches WordPiece.
A pattern that matches HF
Splitting off Unicode punctuation plus the ASCII symbols HF counts as punctuation, and keeping every other run of non-whitespace intact, reproduces HF's tokens:
Passing this through
BertOptions.PreTokenizer = new RegexPreTokenizer(regex, specialTokens)gives tokens identical to HFtokenizerson 18 of the 21 inputs we checked (those above, plus capitalisation, punctuation, accents withRemoveNonSpacingMarks = true, CJK, a 109-token paragraph and a 323-token passage). The three that still differ contain a bare\nor\t, the separate issue #7724. That option is also the workaround until the default changes.One caveat for anyone applying it as a workaround: HF's
_clean_textdeletes format characters (Cf) before tokenizing, whileBertNormalizerdeletes onlyCc. The default pattern happens to split at a zero-width space and this one does not, so against HF"zero\u200bwidth"moves from 0.800 to 0.250 and"soft\u00adhyphen"from 0.554 to 0.091. DeletingCfalongside brings both to 1.000.BertTokenizeris the only caller ofCreateWordOrPunctuationin this repository, but the method is public and documents its pattern. Changing only the default thatBertTokenizerpicks may be safer than changing that method.