When I say make a simple prompt without the need for function calling, then I get back the response as a stream, but when I do function calling like so "Add milk to my todo list" then I get back the function name but I get back no response back from the API like "It is added" or anything, is that by design or is it supposed to send back a response even with function calling and stream?
#Open AI stream + function calling
jagged oriole
bold voidBOT
Hi @jagged oriole. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Yes, it is by design. When you use function calling with the OpenAI API, the model does not actually call the function. Instead, it generates a JSON object containing arguments to call the function. This JSON object can then be used in your code to call the function. The model does not provide a response like "It is added" or anything similar. The response you get is the JSON object that you can use to call the function in your code.
For example, if you have a function called add_to_todo_list(item: string), and you use function calling with the prompt "Add milk to my todo list", the model might output a JSON object like this:
{
"name": "add_to_todo_list",
"args": {
"item": "milk"
}
}
You can then use this JSON object in your code to call the add_to_todo_list function with "milk" as the argument.
When using function calling with streaming, you can manage both "normal" answers and function-calling by doing something like this on the generator object being received from API request with stream = True:
response_text = ""
function_call_detected = False
for response_chunk in response_generator:
if "choices" in response_chunk:
deltas = response_chunk["choices"][0][…]
However, there seems to be some confusion in the community about how to handle multiple function calls in one go when using streaming. This is something that might need further clarification from OpenAI.
For more information, you can refer to the Function Calling Guide (https://platform.openai.com/docs/guides/function-calling) and the OpenAI Community Forum (https://community.openai.com/t/594052) where similar issues have been discussed.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
jagged oriole
@bold void I am still not sure what you mean, let me clarify, would the json object when using function calling give me just this data
{
"name": "add_to_todo_list",
"args": {
"item": "milk"
}
}
Or would it also give me the response with that json data aka saying "Hey, I added it to the list" and the function name and arguments, etc?
bold voidBOT
Let me check the knowledge base for you. One moment! :books:
When you use function calling with the OpenAI API, the model generates a JSON object that contains the function name and arguments. It does not provide a conversational response like "Hey, I added it to the list".
For example, if you use function calling with the prompt "Add milk to my todo list", the model might output a JSON object like this:
{
"name": "add_to_todo_list",
"args": {
"item": "milk"
}
}
This JSON object can then be used in your code to call the add_to_todo_list function with "milk" as the argument.
If you want the model to generate a conversational response as well, you would need to call the model again by appending the function response as a new message, and let the model summarize the results back to the user. This is explained in the Function Calling Guide (https://platform.openai.com/docs/guides/function-calling) from OpenAI.
Here's an example of how you might do this:
# Call the function with the arguments from the model
response = add_to_todo_list(json.loads(function_call.arguments))
# Call the model again with the function response
new_prompt = "I added milk to the todo list."
new_response = model.generate(new_prompt)
# The new_response should contain a conversational response from the model
print(new_response)
In this example, new_response might be something like "Hey, I added it to the list". But remember, this second call to the model is separate from the function call. The function call itself does not generate a conversational response.
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
jagged oriole
@bold void Ah right, so basically I would pass the function name into a switch expression of sorts, run the function and if the function is successful pass that as context to the messages to OpenAI and then it would reply back with "Hey, I added so and so to your todo list" if it's an error I would pass that context to OpenAI again and then it would say something like "there has been an error, I could not add the todo to your list" Am I understanding this right or?
bold voidBOT
Let me check the knowledge base for you. One moment! :books:
Yes, you've got it right!
When the assistant suggests a function call, you should handle that function call in your own code. The result of that function call (whether it's a success message, an error, or some other data) should then be passed back to the assistant as a new message. The assistant can then generate a conversational response based on that new message.
Here's a simplified example:
// This is the function call suggested by the assistant
let functionCall = {
"name": "add_to_todo_list",
"args": {
"item": "milk"
}
};
// You handle the function call in your own code
let functionResult;
switch (functionCall.name) {
case "add_to_todo_list":
functionResult = add_to_todo_list(functionCall.args.item);
break;
// handle other functions...
}
// You pass the result of the function call back to the assistant as a new message
let newMessage = {
"role": "system",
"content": `The function ${functionCall.name} was called with the argument ${functionCall.args.item}. The result was: ${functionResult}.`
};
// You call the assistant again with the new message
let newResponse = await assistant(`Here is the result of the function call: ${newMessage.content}`);
// The newResponse should contain a conversational response from the assistant
console.log(newResponse);
In this example, newResponse might be something like "Hey, I added milk to your todo list" or "There has been an error, I could not add milk to your todo list", depending on the result of the function call.
This process is also described in the OpenAI Community Forum (https://community.openai.com/t/591372) and the OpenAI Cookbook (https://cookbook.openai.com/examples/how_to_build_an_agent_with_the_node_sdk#running-our-app-with-a-simple-input).
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
jagged oriole
@bold void So when I run a function there might be a delay right, is that standard in such applications?
bold voidBOT
Let me check the knowledge base for you. One moment! :books:
Yes, there might be a delay when you run a function, especially if the function involves complex computations or network requests. This is standard in most applications. The delay can vary depending on the specific function and the resources it requires.
For example, if the function is making a request to an external API, the delay might be longer due to network latency. If the function is performing a complex calculation, the delay might be longer if the server's CPU is under heavy load.
In the context of using the OpenAI API, there might also be a delay in receiving the response from the API. This delay can be affected by various factors, such as network latency and the complexity of the model's task.
For instance, in the OpenAI Cookbook's example on streaming completions (https://cookbook.openai.com/examples/how_to_stream_completions#3-how-much-time-is-saved-by-streaming-a-chat-completion), messages are received at different time intervals after the request, ranging from 1.59 seconds to 4.41 seconds. This shows that there can be a delay in receiving the response from the API, and the delay can vary depending on the specific request and the model's task.