rust-tokenizers

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Stars

273

Forks

26

Language

Rust

Last Updated

Apr 25, 2024

Similar Repos