#Too many tokens being used

1 messages · Page 1 of 1 (latest)

north marsh
#

I am developing a chatbot with gpt 3.5 model. I have a instruction prompt (system message) what weights about 1k tokens. The problem is, each message that I send another 1k tokens is spent. It's almost like the system prompt is being passed through each message. Is that the case? If yes, how can I change this?

undone emberBOT
#

Hi @north marsh. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

Yes, your understanding is correct. When using the chat models of OpenAI, such as gpt-3.5-turbo, each message passed to the API consumes the number of tokens in the content, role, and other fields, plus a few extra for behind-the-scenes formatting. This includes the system message, user message, and assistant message. Therefore, if your system message is around 1k tokens, it will be counted towards the total tokens used in each API call.

To reduce the number of tokens used, you could consider the following strategies:

  1. Condense your system message: If possible, try to make your system message more concise without losing the necessary information. This could significantly reduce the number of tokens used in each API call.

  2. Break the text into smaller pieces: If your conversation has too many tokens to fit within a model’s maximum limit, you might need to truncate, omit, or otherwise shrink your text until it fits. However, be aware that if a message is removed from the messages input, the model will lose all knowledge of it.

  3. Use OpenAI’s tiktoken Python library: This library can help you see how many tokens are in a text string without making an API call, which could assist you in managing your token usage more effectively.

Remember, both input and output tokens count toward these quantities. For example, if your API call used 10 tokens in the message input and you received 20 tokens in the message output, you would be billed for 30 tokens.