Software Development

Sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Publisher / source

Orchestra-Research/AI-Research-SKILLs (found on skills.sh, an awesome list)

Upstream record

Orchestra-Research/AI-Research-SKILLs

Version

Version 3

Published

2026-11-03

Publication review

Approved for publication

Content fingerprint

47b1d10f419a51a400010bcd80d99a8bdb5c4510fc3e57fe9bc9be7efa6a48f8