AI & Machine Learning
Huggingface Tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
- Publisher / source
- Orchestra-Research/AI-Research-SKILLs (found on skills.sh, an awesome list)
- Upstream record
- Orchestra-Research/AI-Research-SKILLs
- Version
- Version 3
- Published
- Publication review
- Approved for publication
- Content fingerprint
d2ff3f9eed2cb8e15cb98e8e2398ca359325a32fd5d16ebfe0d115d7624583af