#divinci 003 error

39 messages · Page 1 of 1 (latest)

sudden vigil
#

I’m getting
openai.error.InvalidRequestError: This model’s maximum context length is 4097 tokens, however you requested 4273 tokens (273 in your prompt; 4000 for the completion).

I understand it wants me to shorten the length but it’s already very small, only a couple sentences

languid siren
#

The issue isn't that it wants you to shorten the length, the issue is that you're asking for too many max_tokens in the request

#

You can't send max_tokens > 4097 in a given request

#

You can set your max tokens equal to the total amount of tokens (4097) minus the amount of tokens used up by the prompt

sudden vigil
#

Idk what any of that means

languid siren
#

Show me the code that sends the request

sudden vigil
#

@languid siren

languid siren
#

Your issue is on line 22

#

max_tokens=4000,

#

You are saying that the amount of max tokens for the request is 4000 every time

#

but the model only supports 4096 max tokens. The tokens are a function of the sum of the prompt tokens and the completion tokens. The prompt tokens are the input that you give it, and the completion tokens are the text that is returned by the AI

#

You need to subtract your prompt tokens from 4096, and then set the max_tokens field equal to that value

sudden vigil
#

How do I know my prompt tokens

languid siren
#

You can use the GPT2Tokenizer from HuggingFace to determine the amount of tokens in the prompt: https://huggingface.co/docs/transformers/model_doc/gpt2

It looks something like this:

from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
prompt_tokens = len(tokenizer(prompt)['input_ids'])
OpenAI GPT2
sudden vigil
#

So there isn’t an easy fix to this

languid siren
#

This isn't a "bug" or a "problem", you just need to manage the amount of max_tokens correctly

sudden vigil
#

And how does line 22 look after thst

languid siren
#

Your line 22 sets 4000 as the max tokens always

#

but if your prompt tokens are >97

#

it will think that you're requesting more completion tokens than it is able to provide

#

because of that static number of 4000

#

So your max_tokens always has to be 4096 - INPUT_TOKENS

sudden vigil
languid siren
#

pip install transformers

sudden vigil
#

I tried that and it says can’t find rust compiler

languid siren
#

You can also just really easily fix the issue by setting max_tokens to 3000

#

instead of 4000

#

and it will just work

#

but that's only a temporary fix

languid siren
sudden vigil
languid siren
#

I don't know how to fix that, I never had that issue

#

Try installing the rust compiler

sudden vigil
#

Is it trying to install the right thing?

languid siren
#

yeah