Making GPT outputs deterministic/reproducible

[drowsy star]

Hey all,

Is there any way to get same output from GPT everytime I send the same prompt?

[drowsy star]

Yeah I've tried with smaller temperature values ranging from 10^-50 to 10^-22 (apart from 0). It's not reproducible on big generations.

I cannot cache as it'll defeat my purpose.

I want something like do_sample=False as in huggingface to do greedy decoding

Previous documentation has the seed parameter but was in beta stage. It doesn't works.

[naive sonnet]

Set temperature to 0 like beefour said, and also set the seed param on the chat completions request

also, set top_p to 0

if you set all 3 of these you still might not always get the same response, but it's a lot more reliable than 0 temp alone is

[drowsy star]

Thanks a lot
Will look into this

Idk how conferences will react to this on their reproducibility checklist

Also, setting top p ane temperature is not advised, right?

[naive sonnet]

here's a demonstration of how temperature affects the chance of different tokens being generated:
https://www.baeldung.com/wp-content/uploads/sites/4/2023/05/softmax_animation.gif

you can see that as temperature increases, the likelihood of the most likely token (x5) goes down, and the likelihood of generating less likely tokens (x1-x4) goes up.
top_p determines what percentage of tokens are even considered. so in this example, if you set top_p to 0.4 (40%), no matter what temperature is set to, the model will never pick x1-x3, because those aren't in the top 40% of likely tokens. if you set top_p to 0, the only token that will be considered regardless of temperature is x5, because everything else is less likely than that. setting top_p to 0 is called greedy sampling.

there are some cases where the model can generate strange behavior, like repeating the same text over and over until it hits the generation limit, when using greedy sampling. personally I've never run into this problem with OpenAI's models but you'll have to test it for your own use case

[drowsy star]

Thanks for taking out the time for explaining.

My issue was the same that it'll keep on repeating if I alter both temperature and top p making it greedy.

I'll try this again. Thanks again

[violet shoal]

noo, please, that would be horrible and would beat the purpose and nature of new AI.