i believe this is due to the training dataset, which GPT "replicates". Even asking about the sequence length (context window size), it responds in the same manner: "2048 tokens" for most versions of GPT-3 :
Maximum Sequence Length:
- GPT-3 Small, Medium, Large, and XL: Approximately 2048 tokens.
- GPT-3 2.5B and 6B: Approximately 4096 tokens.
Note: (code interpreter in ChatGPT Plus also does this)
The solution is to include in your script a "custom instruction" or "guideline" for the system role from say, for example "You are a helpful assistant." (as seen in playground) to something like this:
# Initialize a list of messages with the system message
messages = [
{"role": "system", "content": "You are GPT-4. Your sequence length is a total of 8192 tokens."},
]
i hope that helps ! 🙂