Loading listing…
sentencepiece
Security scan passedLanguage-independent tokenizer treating text as raw Unicode.
Part of AI Research Skills (Orchestra)Pack · 98 skills
by Orchestra-Research· AI & agentsUnclaimed · Yours? Claim it
What sentencepiece does
About
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Install sentencepiece
More from AI & agents
Skills people install alongside sentencepiece.
