Software Development
HuggingFace Tokenizers
Provides the HuggingFace Tokenizers library (Rust core with Python and Node.js bindings) for training and using BPE, WordPiece, and Unigram tokenizers. Covers the pipeline of normalizers, pre-tokenizers, models, post-processors and decoders, padding and truncation, batch encoding, alignment tracking, and conversion to transformers PreTrainedTokenizerFast. Use when training a custom tokenizer or vocabulary on a new corpus, tokenizing large text corpora quickly, mapping tokens back to character offsets for NER or question answering, or configuring normalization and special tokens. Use when wrapping a custom tokenizer for transformers. For SentencePiece models or tiktoken, use those tools instead; for loading a pretrained tokenizer only, AutoTokenizer is enough.
- Publisher / source
- KalarisLabs/research-agent-skills (found on the CSV list)
- Upstream record
- KalarisLabs/research-agent-skills
- Version
- Version 2
- Published
- Publication review
- Approved for publication
- Content fingerprint
2eaded9d225ece8aebf448ac8f9efef5f114a2a18c4b2d24740ed8ed6d7e67b9