#Get tokens used after streaming a ChatCompletion

16 messages · Page 1 of 1 (latest)

cosmic lodge
#

Hello, I would like to know how I can get the usage of the prompt/completion tokens when I use a stream. As I can see in the examples, the returned output are not returning, even at the end, the amount of tokens used. Any way to do that? Thanks.

image.png
wraith tiger
#

So those will be your completion tokens

#

then for the prompt tokens you can tokenize the prompt using a local tokenizer

cosmic lodge
#

GPT itself told me this -- would this work you think?

image.png
#

Anyways I'm trying it out right now

wraith tiger
#

no

#

I don't think so

#

and also GPT isn't good at developing for OpenAI APIs

#

I don't think each chunk returns a total_tokens

cosmic lodge
#

I know I just tried lol

#

if that doesn't work ill just do it myself, like you told me

#

yep doesn't work, I'll do that

#

thanks

deep isle
#

The request has an overhead as well. If you read the OpenAI Tokenizer example, you will see it uses im_start, im_end, and extra newlines. I only use a single "user" message (no system). The token count I need to add for the prompt part is 7.