For example, here's Command-R's: https://huggingface.co/CohereForAI/c4ai-command-r-v01/blob/main/tokenizer_config.json
#Please add more special tokens other than <s>, </s> and <unk>
fleet sparrow
molten dove
For example, here's Command-R's: https://huggingface.co/CohereForAI/c4ai-command...
a bit out of scope dont you think ? specially for the openweight stuff
and cmd-r is NC
fleet sparrow
i meant for future releases, pretty sure this is becoming a standard -- especially because of multimodality (gemma has a 256k vocab size as well, and many of those are unused)
tall pond
Which special tokens do you want specifically and why?