Token Truncation Issue

[surreal coral]

I'm trying to make an app that runs text-based adventure games using a GPT-3.5 chat bot as a storyteller. I've got it accepting sets of rules for games, characters, and creatures through JSON files so it can keep referencing these rules without taking up token space as "memory". However, I keep hitting the token limit. I want to be able to truncate old tokens as the conversation/game continues. So far, no attempt I've made on my own or any provided by ChatGPT seem to work correctly, and the games keep crashing as I hit the token cap.

I am happy to provide the code for troubleshooting purposes if needed.

[surreal rain]

Post the code 🙂 I’m hoping to do something similar…

[midnight rampart]

@surreal coral please provide a code in the discord code format not in files

[surreal coral]

It keeps wanting to paste the code as a text file anyway

[surreal rain]

    global conversation_history
    total_tokens_removed = 0

    while total_tokens_removed < target_tokens:
        removed_message = conversation_history.pop(0)
        total_tokens_removed += len(openai.api.encode(removed_message))

    return total_tokens_removed```

That looks like the relevant part. What happens if you try to keep the limit way smaller than the 4096 token limit? And display some information every turn like ’conversation history length’, ’number of tokens removed’, ’number of tokens used’. It might give a clue as to what part is not working.

[surreal coral]

I actually had a function to count total tokens in the HTML file, but it broke when I changed how the truncate tokens function worked. The previous solution ChatGPT gave me tried to approximate tokens by counting words, which obviously didn't work.