I am using the assistant api with the gpt-4-1106-preview model and I am running into context limits when sending it messages in the playground, that seem too small compared to the context I undersstand the model can accept. Is this an issue with the input field in the playground or the api or the model?
#What is the maximum context limit that an assistant using the gpt-4-1106-preview can accept?
fair otter
inland wedgeBOT
Hi @fair otter. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
The gpt-4-1106-preview model, also known as GPT-4 Turbo, introduces a 128k context window, which is equivalent to about 300 pages of text in a single prompt [^1^]. However, the maximum number of output tokens for this model is 4096 [^1^].
When using the Assistants API, it automatically manages the context window such that you never exceed the model's context length. Once the size of the Messages in a Thread exceeds the context window of the model, the Thread will attempt to include as many messages as possible that fit in the context window and drop the oldest messages [^5^].
However, there are some limitations and issues that users have reported. For instance, some users have reported that they are unable to use the full 128k context window due to their usage tier limitations. For example, a Usage Tier 3 API account is limited to 40,000 tokens per minute, and only after hitting Usage Tier 4 (by spending a lifetime total of $250 with OpenAI) will the tokens per minute limit be raised to 150,000 tokens per minute [^3^].
There have also been reports of discrepancies between the Playground and the API, with some users being able to use the larger context window in the Playground but encountering errors when trying to do the same via the API [^2^].
OpenAI has acknowledged these issues and has stated that they are on the roadmap to be addressed in the coming months [^4^].
So, if you're encountering context limit issues, it could be due to a combination of factors including your usage tier, the specific way the Assistants API manages the context window, and potential discrepancies between the Playground and the API.
^1^]: [Help Article > En > Articles (https://help.openai.com/en/articles/8555510-gpt-4-turbo)
^2^]: [Forum Post: Playground works (128k) but not API > Original post (https://community.openai.com/t/526318)
^3^]: [Forum Post: Test new 128k window on gpt-4-1106-preview > Official Answer (https://community.openai.com/t/478424)
^4^]: [Forum Post: Assistant API: How to set limit of context window for GPT-4 turbo > Official Answer (https://community.openai.com/t/534869)
^5^]: [Docs > Assistants > How Assistants work Beta > Managing Threads and Messages > Context window management (https://platform.openai.com/docs/assistants/how-it-works)
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
fair otter
@frozen snow that doesnt answer my question
frozen snow
<@560940831775653888> that doesnt answer my question
You should be able to include 128k tokens of context, if not then that's a bug imo
fair otter
@frozen snow in the input field of the assistant playground?
frozen snow
OpenAI doesn't have any documented restrictions about the playground but you can use the API if the web UI is giving you a hard time
fair otter
1 validation error for Request body -> content ensure this value has at most 32768 characters (type=value_error.any_str.max_length; limit_value=32768)
I get this error from the assistant UI
frozen snow
I would try using API calls in that case