Tokenizer
Microsoft Corporation
AI
0.0
|
338 Downloads
|
—
What's New
Version 1.3.3
No changelog available yet.
Description
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
Download
winget install Microsoft.Tokenizer
