Confidence score of completion API.

[tame topaz]

I want to know how do I generate confidence score on the result generated by openai completion api. #1037561751362863144 #dev-chat

[blazing kernel]

This isn't possible, the result generated by the API isn't one result with a single confidence rate, each token generated by the API has a log probability

[compact surge]

If you want a sort of confidence score, you can get an embedding vector for the question and the answer and do a dot product of the two vectors. It will tell you how close gpt thinks they are to each other in vector space. I find score over .8 are fairly good. Scores over .84 are really good and scores in the .7 range are generally not that closely related. It's not a confidence score, but it is somewhere to start