#gpt 3.5 chat history

2 messages · Page 1 of 1 (latest)

hard shadow
#

So i coded a discord bot (discord.js library) with openai node.js library which is using gpt 3.5. Everytime a user sends a message to the bot, the message will be stored to be used for further recalls but the problem is depending on how many messages are stored, more tokens will be used for each request. Do we needs to reset the data time by time to optimize the token usage or there won't be such a huge difference?

rugged bloom
#

Yea there's a few ways you can reduce token count:

  • Keep the most recent X messages in memory and drop older ones
  • The above but instead of dropping them have the model generate a summary and include that in the history
  • Use text embeddings and store past messages, you then find the most relevant messages on each prompt and use them as the context