logprobs

[oak chasm]

How can I use logprobs?

[scenic minnow]

Hi @oak chasm. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

The logprobs parameter is a feature available in OpenAI's Chat Completions API and Legacy Completions API. When enabled, it provides the log probabilities of each output token, and a limited number of the most likely tokens at each token position alongside their log probabilities. This can be useful to assess the confidence of the model in its output, or to examine alternative responses the model might have given [source (https://platform.openai.com/docs/guides/text-generation)].

Here's an example of how you can use logprobs in a function to get completions from the API:

from openai import OpenAI
from math import exp
import numpy as np
from IPython.display import display, HTML

client = OpenAI()

def get_completion(
 messages: list[dict[str, str]],
 model: str = "gpt-4",
 max_tokens=500,
 temperature=0,
 stop=None,
 seed=123,
 tools=None,
 logprobs=None, # whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message..
 top_logprobs=None,
) -> str:
 params = {
 "model": model,
 "messages": messages,
 "max_tokens": max_tokens,
 "temperature": temperature,
 "stop": stop,
 "seed": seed,
 "logprobs": logprobs,
 "top_logprobs": top_logprobs,
 }
 if tools:
 params["tools"] = tools

 completion = client.chat.completions.create(**params)
 return completion

[source (https://cookbook.openai.com/examples/using_logprobs#0-imports-and-utils)]

There are several use cases for logprobs:

1. Classification tasks: logprobs can be used to assess the model's confidence in its outputs, enabling users to set their own classification or confidence thresholds [source (https://cookbook.openai.com/examples/using_logprobs)].

2. Retrieval (Q&A) evaluation: logprobs can assist with self-evaluation in retrieval applications, reducing retrieval-based hallucinations and enhancing accuracy [source (https://cookbook.openai.com/examples/using_logprobs)].

3. Autocomplete: logprobs could help decide how to suggest words as a user is typing [source (https://cookbook.openai.com/examples/using_logprobs#3-autocomplete)].

4. Token highlighting and outputting bytes: Users can easily create a token highlighter using the built-in tokenization that comes with enabling logprobs [source (https://cookbook.openai.com/examples/using_logprobs)].

5. Possible extensions: logprobs can be used for evaluations (e.g., calculate perplexity of outputs), moderation, keyword selection, improving prompts and interpretability of outputs, token healing, and more [source (https://cookbook.openai.com/examples/using_logprobs#6-possible-extensions)].

---

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: