It could be useful to have an official way to count tokens, such as tiktoken alternative or a dedicated endpoint in La Plateforme.
#Token counting
stuck saddle
stuck saddle
I made some tests with transformers: https://gist.github.com/cbouvard/9a6ac5aa5d034030c4df856f6e8be966.
With LlamaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer")and AutoTokenizer.from_pretrained("mistralai/xxxxx") I have almost the same token count than the "prompt_tokens" returned by La Plateforme.