I have an app that queries OpenAI with n=4 for several reasons. I thought I could quickly modify Ooba and the OpenAI extension to return multiple choices. I implemented num_return_sequences in modules/text-generation.py and added that argument in completions.py when 'n' is passed in. But for some reason, Ooba doesn't return more than one sequence. Perhaps I am just doing it wrong, but are there any plans to support returning more than one sequence?
#supporting 'n' in the OpenAI extension
gray escarp
hoary dagger
Hi @gray escarp , multiple generations is a bit awkward if you're not doing it in the underlying model, you probably will get a lot of dups
I'd really like to get this working also, can you share the code changes?
gray escarp
I'd really like to get this working also, can you share the code changes?
I have a fork of Ooba doing this now. I'll share in a bit
hoary dagger
the 'n' parameter is used in a couple ways - which did you implement? completion variations?
gray escarp
I used it as num_return_sequences
Can you send me the reference that describes its other uses?
One oddball thing I found is that num_beams needs to be >= num_return_sequences
hoary dagger
Sorry, I'm a bit distracted, I'm currently on holidays and will be back the end of Sept and will be able to assist more when I get back. The difference is something in the OpenAI API reference docs with completion and chat afair.
It also has to do with how it works with 'best of', and how I did top_k
gray escarp
Yeah, I suspect they are doing something else entirely behind the scenes, because when you ask for multiple generations from openai, you get significantly different results. When I ask for multiple generations, I get nearly the same results. Going to play with some ideas to see if I can get this working closer to what openai generates.
hoary dagger
one thing to note, the completions API is now considered legacy. Some day it may be deprecated, like the edits endpoint.