#```py
zinc isle
heya, you need to pass in token length initially - look at this example:
ai_response = openai.ChatCompletion.create(
model=model,
temperature=0.5,
max_tokens=max_tokens,
messages=messages
)
and count your tokens in on the way in as well and take that off the max
not pretty code but you get the idea
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
api_key = os.environ.get('OPENAI_API_KEY')
openai.api_key = api_key
def count_tokens(text):
tokens = len(encoding.encode(text))
return tokens
def count_total_tokens(messages):
total_tokens = 0
for message in messages:
total_tokens += count_tokens(message)
return total_tokens
def trimToMaxTokens(conversation, max_len_tokens):
conversation = conversation[::-1]
token_count = 0
trimmedConversation = []
for obj in conversation:
token_len = count_tokens(obj['content'])
if token_count + token_len < max_len_tokens:
trimmedConversation.append(obj)
token_count += token_len
return {'conversation': trimmedConversation[::-1], 'length': token_count}
not pretty but you get the idea
proud spoke
Problem is, Hugh, that omi farhans code apparently shouldn't get anywhere near the limit with that input 🤔
zinc isle
yeah but its probably the completion not your input
i've been there suggest trying that..
proud spoke
As far as my understanding goes, if you prompt with 3000 tokens, your response will be cutoff after 1000 completion tokens. GPT will give you a response, but the stop reason will indicate that you can out of context.
lucid peak
I did not mention any max token while requesting. What is the default value?
I also got proper responses couple of times. Probably while trying for the 5th time, it gave the error
proud spoke
The code you provided, do you run a loop around it?
The max amount of tokens that gpt-3.5-turbo can handle is 4097 as the error says. This includes both your input and the output from gpt.
lucid peak
The code you provided, do you run a loop around it?
The max amount of tokens tha...
No there is no loop.
I guess the output it provides takes more token
zinc isle
i make sure the tokens i pass in and specified total will not be > 4097
proud spoke
I guess the output it provides takes more token
Has to be that. But it's odd to me that the API errors on that.