#Max tokens not working

1 messages · Page 1 of 1 (latest)

silk shoal
# 
def calculate_token_usage(text, model="gpt-3.5-turbo"):
    encoding = tiktoken.encoding_for_model(model)
    token_count = len(encoding.encode(text))
    return token_count

  token_usage = calculate_token_usage(str(crafted_prompt))
  if token_usage < 4000:
      model = "gpt-3.5-turbo"
  else:
      model = "gpt-3.5-turbo-16k"

  response = openai.ChatCompletion.create(
      model=model,
      messages=crafted_prompt,
      temperature=0.95,
      max_tokens=700,
      top_p=1,
      frequency_penalty=0,
      presence_penalty=0.28
  )

will generate stuff bigger then 700 combined usage

golden orbit
#

which model are you using?

silk shoal
golden orbit
#

max token i think is only for the response

#

so your request has its own

#

I think

mystic mango
#

^ they are separate

silk shoal
#

from what i read i thought that they were combined and that max tokens = total token usage

deep burrow
#

max_tokens
integer or null
Optional
Defaults to 16

The maximum number of tokens to generate in the completion.

The token count of your prompt plus max_tokens cannot exceed the model's context length. Example Python code for counting tokens.