Tokenizer
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
winget install --id Microsoft.Tokenizer --exact --source wingetLatest 1.3.3
Details
- Homepage
- https://github.com/microsoft/Tokenizer
- License
- MIT