perplexity

[teal summit]

hi! does anyone know how to use the openai api to calculate perplexity of a sentence? thank u!!

[stuck peak]

How's this? Using GPT with the functions interface

...
sentence = "does anyone know how to use the openai api to calculate perplexity of a sentence?"
messages = [
    {
        "role": "system",
        "content": """You're an english wizard who understands how complex sentences are."""
    },
    {
        "role": "user",
        "content": f"""Remember that perplexity means the inability to deal with or understand something complicated or unaccountable.

Calculate the perplexity of this sentence:

{sentence}"""
    }
]


response = create_chat_completion(
    model="gpt-4-0613",
    messages=messages,
    max_tokens=1000,
    n=1,
    stop=None,
    temperature=0.0,
    top_p=1.0,
    frequency_penalty=0.0,
    presence_penalty=0.0,
    functions=[
        {
            "description": "shows the perplexity to the user",
            "name": "perplexity_display",
            "parameters": {
                "properties": {
                    "perplexity": {
                        "description": "the computed perplexity score on a scale from 1-10",
                        "type": "number"
                    }
                },
                "required": [
                    "perplexity"
                ],
                "type": "object"
            }
        }
    ],
    function_call={"name": "perplexity_display"},
)
...

Full code here: https://app.prompteditor.io/shared_prompt/VQ_X2j9hk2Qpf6pGiPQkbw/

[teal summit]

ty for ur response! unfort i need to use it with a model that is not a chat model so idt that will work cuz its giving me an error when i switch it to davinci

[stuck peak]

Why not use it with a chat model?

gpt-3.5 is recommended over davinci in almost all scenarios. Unless you're fine-tuning with your own dataset, you can save 10x the price and get better results with it

[teal summit]

yeah i am trying to evaluate a fine tuned model

[midnight dirge]

Is your model fine-tuned to evaluate perplexity?

If you want to evaluate your fine-tuned model output, you'd do what Jake suggested. You'd just use your fine-tuned model as the input to evaluate.