#max token

1 messages · Page 1 of 1 (latest)

glass lagoon
#

how to set max token

Using GPT-4 to understand the intent

chat_completion = openai.chat.completions.create(
messages=[
    {
        "role": "user",
        "content": f" what is the intention of this prompt '{user_prompt}' if it asks you t",

    }
],
model="gpt-4",
plucky sandBOT
#

Hi @glass lagoon. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

The max_tokens parameter is used to control the maximum number of tokens that can be generated in the completion of a request. The total number of tokens, which includes both the input and output, cannot exceed the model's context length. For example, if you're using the GPT-4 model and you want to limit the output to a maximum of 600 tokens, you can set the max_tokens parameter to 600 in your request like this:

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are gpt-4."},
        {"role": "user", "content": "How do I program the API?"}
    ],
    max_tokens=600
)

In this example, the messages parameter is a list of message objects. Each object has a role that can be 'system', 'user', or 'assistant', and content which is the text of the message from the role.

Please note that the max_tokens parameter is not a way to control the maximum tokens of the output only, it's the total tokens (input+output). Currently, there is no way to specify a maximum number of tokens the Assistant or tool should use, but support for this is coming soon according to a forum post (https://community.openai.com/t/534869) by a Support User from the Assistants API team.

Also, depending on the model used, requests can use up to 128,000 tokens shared between prompt and completion. Some models, like GPT-4 Turbo, have different limits on input and output tokens. There are often creative ways to solve problems within the limit, e.g. condensing your prompt, breaking the text into smaller pieces, etc. as mentioned in this help article (https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them#h_051eb08805).

#

For GPT-4 Turbo with vision, it currently sets a low max_tokens default which you can override as stated in this forum post (https://community.openai.com/t/500636).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@glass lagoon kapa.ai is still learning and improving, please let us know how it did by reacting below