You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Migrating a custom Hugging Face tokenizer to .NET requires the caller to know the tokenizer implementation, identify and load the correct vocabulary and merge files, interpret tokenizer configuration, and manually map those resources to the corresponding Microsoft.ML.Tokenizers type. In Python, AutoTokenizer.from_pretrained(...) handles this discovery and construction from a model directory or tokenizer configuration.
This gap makes migration harder and encourages application-specific loader code. #7384 can document the current process, but documentation alone does not provide an equivalent loading abstraction.
Describe the solution you'd like
Provide a factory API that creates the appropriate tokenizer from Hugging Face tokenizer artifacts, for example from a model directory or tokenizer.json file. The factory should:
Detect the supported tokenizer implementation and construct the corresponding .NET tokenizer.
Read the relevant tokenizer configuration, vocabulary, merges, special tokens, normalizer, pre-tokenizer, post-processor, and decoder settings when supported.
Report unsupported tokenizer types or configuration sections explicitly rather than silently changing behavior.
Allow callers to inspect the concrete tokenizer or its capabilities when implementation-specific behavior is needed.
The final API shape does not need to match Python, but the migration experience should be comparable to AutoTokenizer.from_pretrained(...) for supported formats.
Describe alternatives you've considered
Callers can identify the tokenizer type themselves and write adapters around individual tokenizer constructors. A migration guide can explain those steps, but every consuming library still has to reproduce the detection, configuration mapping, validation, and error handling.
Additional context
This request was identified while splitting the feedback in #7383, which originated in microsoft/semantic-kernel#9793. Related format-specific gaps should remain separately trackable, while this issue covers the complete discovery-and-construction experience.
Is your feature request related to a problem? Please describe.
Migrating a custom Hugging Face tokenizer to .NET requires the caller to know the tokenizer implementation, identify and load the correct vocabulary and merge files, interpret tokenizer configuration, and manually map those resources to the corresponding
Microsoft.ML.Tokenizerstype. In Python,AutoTokenizer.from_pretrained(...)handles this discovery and construction from a model directory or tokenizer configuration.This gap makes migration harder and encourages application-specific loader code. #7384 can document the current process, but documentation alone does not provide an equivalent loading abstraction.
Describe the solution you'd like
Provide a factory API that creates the appropriate tokenizer from Hugging Face tokenizer artifacts, for example from a model directory or
tokenizer.jsonfile. The factory should:The final API shape does not need to match Python, but the migration experience should be comparable to
AutoTokenizer.from_pretrained(...)for supported formats.Describe alternatives you've considered
Callers can identify the tokenizer type themselves and write adapters around individual tokenizer constructors. A migration guide can explain those steps, but every consuming library still has to reproduce the detection, configuration mapping, validation, and error handling.
Additional context
This request was identified while splitting the feedback in #7383, which originated in microsoft/semantic-kernel#9793. Related format-specific gaps should remain separately trackable, while this issue covers the complete discovery-and-construction experience.
Note
This issue was drafted with AI assistance.