#How do I use logprob?

1 messages · Page 1 of 1 (latest)

ripe token
#

Hey @trim bolt, how do I access logprobs, for what models are they available and what is the best use that I can make for them in production?

trim boltBOT
#

Hi @ripe token. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

The logprobs parameter is available in the Chat Completions API and Legacy Completions API of OpenAI. When enabled, it returns the log probabilities of each output token, along with a limited number of the most likely tokens at each token position and their log probabilities. This can be useful to assess the confidence of the model in its output, or to examine alternative responses the model might have given source (https://platform.openai.com/docs/advanced-usage/advanced-usage).

However, it's important to note that the logprobs option is currently not available on the gpt-4-vision-preview model source (https://cookbook.openai.com/examples/using_logprobs).

In terms of usage, logprobs can be used in a variety of ways:

  1. Classification tasks: logprobs provide a probability associated with each class prediction, enabling users to set their own classification or confidence thresholds source (https://cookbook.openai.com/examples/using_logprobs).

  2. Retrieval (Q&A) evaluation: logprobs can assist with self-evaluation in retrieval applications. In the Q&A example, the model outputs a contrived has_sufficient_context_for_answer boolean, which can serve as a confidence score of whether the answer is contained in the retrieved content source (https://cookbook.openai.com/examples/using_logprobs).

  3. Autocomplete: logprobs could help us decide how to suggest words as a user is typing source (https://cookbook.openai.com/examples/using_logprobs).

#

  1. Token highlighting and outputting bytes: Users can easily create a token highlighter using the built-in tokenization that comes with enabling logprobs. Additionally, the bytes parameter includes the ASCII encoding of each output character, which is particularly useful for reproducing emojis and special characters source (https://cookbook.openai.com/examples/using_logprobs).

  2. Calculating perplexity: logprobs can be used to help us assess the model's overall confidence in a result and help us compare the confidence of results from different prompts source (https://cookbook.openai.com/examples/using_logprobs).

There are many other use cases for logprobs that are not covered in this cookbook. We can use logprobs for moderation, keyword selection, improving prompts and interpretability of outputs, token healing, and more source (https://cookbook.openai.com/examples/using_logprobs).

To access logprobs, you would need to set the logprobs parameter to True when making a request to the API. For example:

API_RESPONSE = get_completion(
    [
        {
            "role": "user",
            "content": PROMPT.format(
                article=ada_lovelace_article, question=question
            ),
        }
    ],
    model="gpt-4",
    logprobs=True,
    top_logprobs=3,
)

In this example, logprobs is set to True and top_logprobs is set to 3, which means the API will return the 3 most likely tokens at each token position source (https://cookbook.openai.com/examples/using_logprobs#2-retrieval-confidence-scoring-to-reduce-hallucinations).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: