Function Calling question

[cobalt path]

I need help with the function calling API, this is the basic sequence that OpenAI documentations says:
1 Call the model with the user query and a set of functions defined in the functions parameter.
2 The model can choose to call one or more functions; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may hallucinate parameters).
3 Parse the string into JSON in your code, and call your function with the provided arguments if they exist.
4 Call the model again by appending the function response as a new message, and let the model summarize the results back to the user.

And an example:
// Step 2: check if the model wanted to call a function
const toolCalls = responseMessage.tool_calls;
if (responseMessage.tool_calls) {
// Step 3: call the function
// Note: the JSON response may not always be valid; be sure to handle errors
const availableFunctions = {
get_current_weather: getCurrentWeather,
}; // only one function in this example, but you can have multiple
messages.push(responseMessage); // extend conversation with assistant's reply
for (const toolCall of toolCalls) {
......
messages.push({
tool_call_id: toolCall.id,
role: "tool",
name: functionName,
content: functionResponse,
}); // extend conversation with function response
}
const secondResponse = await openai.chat.completions.create({
model: "gpt-3.5-turbo-0125",
messages: messages,
}); // get a new response from the model where it can see the function response
return secondResponse.choices;
}
}

I have a question, if i have a chatting website, with the AI, it means that the first response saved in the messages should not be seen and neither the message with the tool role, but only the last message generated with the second response, right?

[blazing carbon]

Hi @cobalt path. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

Yes, you're correct. In a typical chat application, the user would only see the assistant's responses that are intended for them. The system messages and tool role messages are part of the conversation context that helps the AI model understand the conversation flow and generate appropriate responses, but they are not typically displayed to the user.

Here's a simplified view of the process:

1. The user sends a message, which you pass to the AI model along with any defined functions.
2. The model may decide to call a function based on the user's message. If it does, it generates a message with a tool role, which contains a stringified JSON object representing the function call.
3. You parse this JSON in your code, call the appropriate function with the provided arguments, and get a response.
4. You append this function response as a new tool role message to the conversation context.
5. You call the model again with the updated conversation context. The model generates a new assistant role message summarizing the function response.
6. You display this final assistant message to the user.

So, in your chat interface, you would typically only display the assistant's responses that are intended for the user, not the system or tool role messages.

For more details, you can refer to the OpenAI documentation on function calling (https://platform.openai.com/docs/guides/function-calling) and the example provided in the OpenAI Cookbook (https://cookbook.openai.com/examples/function_calling_with_an_openapi_spec#how-to-call-these-functions-with-gpt).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon but there is also the first assistant response, that is empty, should it be added to the messages, and also, should it be visualized?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

The first assistant response that you're referring to is likely the one that contains the tool call. This message is important for maintaining the conversation context when you make the second call to the model, but it's not typically displayed to the user.

Here's a more detailed breakdown:

1. User sends a message.
2. You call the model with the user's message and any system messages or function definitions.
3. The model generates an assistant message that may contain a tool call. This message is added to the conversation context (messages array), but it's not displayed to the user because it's not a final response—it's essentially an instruction for the next step.
4. You parse the tool call, execute the function, and get a response.
5. You add a new tool message to the conversation context with the function response.
6. You call the model again with the updated conversation context.
7. The model generates a new assistant message summarizing the function response. This is the message that you display to the user.

So, while the first assistant message (the one with the tool call) is important for the conversation flow and needs to be included in the messages array, it's not typically something you would display to the user in a chat interface.

For more information, you can refer to the OpenAI documentation on function calling (https://platform.openai.com/docs/guides/function-calling) and the example provided in the OpenAI Cookbook (https://cookbook.openai.com/examples/function_calling_with_an_openapi_spec#how-to-call-these-functions-with-gpt).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon Why am i getting "400 Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'."

If the messages look like this:
{
"content": [
{
"type": "text",
"text": null
}
],
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "call_4S8heuEHnz8RSBMeKgTD6dj9",
"type": "function",
"function": {
"name": "generateImage",
"arguments": "{"prompt":"gatto"}"
}
}
]
},
{
"tool_call_id": "call_4S8heuEHnz8RSBMeKgTD6dj9",
"role": "tool",
"name": "generateImage"
}
?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

The error message you're seeing suggests that there's an issue with the sequence or structure of your messages. Specifically, it seems that a message with the role 'tool' is not correctly responding to a preceding message with 'tool_calls'.

In your provided messages, the tool message (the second message) is missing the 'content' field. This field should contain the response from the function call. The OpenAI API expects this field to be present in a tool message.

Here's how your messages should look:

{
  "content": [
    {
      "type": "text",
      "text": null
    }
  ],
  "role": "assistant",
  "tool_calls": [
    {
      "index": 0,
      "id": "call_4S8heuEHnz8RSBMeKgTD6dj9",
      "type": "function",
      "function": {
        "name": "generateImage",
        "arguments": "{\"prompt\":\"gatto\"}"
      }
    }
  ]
},
{
  "tool_call_id": "call_4S8heuEHnz8RSBMeKgTD6dj9",
  "role": "tool",
  "name": "generateImage",
  "content": "your_function_response_here"
}

Replace "your_function_response_here" with the actual response from your generateImage function.

For more information, you can refer to the OpenAI documentation on function calling (https://platform.openai.com/docs/guides/function-calling) and the example provided in the OpenAI Cookbook (https://cookbook.openai.com/examples/function_calling_with_an_openapi_spec#how-to-call-these-functions-with-gpt).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon now it's as you said, but it still gives me the same error:
{
"content": [
{
"type": "text",
"text": null
}
],
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
"type": "function",
"function": {
"name": "generateImage",
"arguments": "{"prompt":"un gatto"}"
}
}
]
}

{
"tool_call_id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
"role": "tool",
"name": "generateImage",
"content": "prompt: cat"
}

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

The error message you're seeing, "400 Invalid parameter: messages with role 'tool' must be a response to a preceding message with 'tool_calls'", suggests that the sequence of messages might not be correct.

In the sequence of messages you've provided, it seems like you're trying to create two separate messages: one with the role 'assistant' and 'tool_calls', and another with the role 'tool'. However, these should be part of the same conversation and thus should be in the same 'messages' array.

Here's how your messages should look:

[
  {
    "content": [
      {
        "type": "text",
        "text": null
      }
    ],
    "role": "assistant",
    "tool_calls": [
      {
        "index": 0,
        "id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
        "type": "function",
        "function": {
          "name": "generateImage",
          "arguments": "{\"prompt\":\"un gatto\"}"
        }
      }
    ]
  },
  {
    "tool_call_id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
    "role": "tool",
    "name": "generateImage",
    "content": "prompt: cat"
  }
]

This array of messages should be passed to the openai.chat.completions.create method as the value of the 'messages' parameter.

For more information, you can refer to the OpenAI documentation on function calling (https://platform.openai.com/docs/guides/function-calling) and the example provided in the OpenAI Cookbook (https://cookbook.openai.com/examples/function_calling_with_an_openapi_spec#how-to-call-these-functions-with-gpt).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon Why am i getting "400 Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.? This is the messages array:
[
{
"content": [
{
"type": "text",
"text": null
}
],
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
"type": "function",
"function": {
"name": "generateImage",
"arguments": "{"prompt":"un gatto"}"
}
}
]
},
{
"tool_call_id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
"role": "tool",
"name": "generateImage",
"content": "prompt: cat"
}
]

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

The error message you're seeing, "400 Invalid parameter: messages with role 'tool' must be a response to a preceding message with 'tool_calls'", suggests that there's an issue with the sequence or structure of your messages.

However, the messages array you've provided seems to be correctly structured, with an assistant message containing a tool call followed by a tool message responding to that call.

One possible issue could be the timing of when you're sending these messages. The tool message should be sent in the same API call as the assistant message that contains the tool call. If you're sending them in separate API calls, you might encounter this error.

Here's an example of how you might structure your API call:

const response = await openai.chat.completions.create({
  model: "gpt-3.5-turbo",
  messages: [
    {
      "content": [
        {
          "type": "text",
          "text": null
        }
      ],
      "role": "assistant",
      "tool_calls": [
        {
          "index": 0,
          "id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
          "type": "function",
          "function": {
            "name": "generateImage",
            "arguments": "{\"prompt\":\"un gatto\"}"
          }
        }
      ]
    },
    {
      "tool_call_id": "call_HTsMrSwuuBJDKWFk0mrVCXJY",
      "role": "tool",
      "name": "generateImage",
      "content": "prompt: cat"
    }
  ]
});

If you're still encountering the error, it might be helpful to double-check the tool call ID and ensure it matches exactly between the assistant message and the tool message.

For more information, you can refer to the OpenAI documentation on function calling (https://platform.openai.com/docs/guides/function-calling) and the example provided in the OpenAI Cookbook (https://cookbook.openai.com/examples/function_calling_with_an_openapi_spec#how-to-call-these-functions-with-gpt).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon i get this response error: Object { message: "'$.messages[2].content' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", type: "invalid_request_error", param: null, … }

with this request:
{
"model": "gpt-3.5-turbo-1106",
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Sei SaGPT"
}
]
},
{
"content": [
{
"type": "text",
"text": "generami l'immaginedi un gatto"
}
],
"role": "user"
},
{
"content": [
{
"type": "text",
"text": null
}
],
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "call_gQDGaSPMs41hULy6OEyYnbJk",
"type": "function",
"function": {
"name": "generateImage",
"arguments": "{"prompt":"un gatto"}"
}
}
]
},
{
"tool_call_id": "call_gQDGaSPMs41hULy6OEyYnbJk",
"role": "tool",
"name": "generateImage",
"content": "prompt: cat"
}
],
"max_tokens": 4096,
"stream": true,
"temperature": 0.5,
"presence_penalty": 0,
"frequency_penalty": 0
}

What is wrong with that?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

The error message you're seeing, "'$.messages[2].content' is invalid", suggests that there's an issue with the 'content' field of the third message in your 'messages' array (remember, array indices start at 0).

In your request, the 'content' field of the third message is an array containing an object with 'type' and 'text' fields, and the 'text' field is set to null:

{
  "content": [
    {
      "type": "text",
      "text": null
    }
  ],
  ...
}

However, according to the OpenAI API documentation, the 'content' field should be a string, not an array or an object. This string represents the content of the message from the assistant.

Here's how you might correct the third message:

{
  "content": "Your assistant message here",
  "role": "assistant",
  "tool_calls": [
    {
      "index": 0,
      "id": "call_gQDGaSPMs41hULy6OEyYnbJk",
      "type": "function",
      "function": {
        "name": "generateImage",
        "arguments": "{\"prompt\":\"un gatto\"}"
      }
    }
  ]
}

Replace "Your assistant message here" with the actual message from the assistant.

For more information, you can refer to the OpenAI API documentation (https://platform.openai.com/docs/api-reference/chat/create).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

[cobalt path]

@blazing carbon
I have a problem, i have this request:
{
"model": "gpt-4-0125-preview",
"messages": [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Sei SaGPT"
}
]
},
{
"content": [
{
"type": "text",
"text": "genera l'immagine di una bottiglia"
}
],
"role": "user"
},
{
"content": null,
"role": "assistant",
"tool_calls": [
{
"index": 0,
"id": "call_zBtypFBARcNYe1lfMnBU9aMd",
"type": "function",
"function": {
"name": "generateImage",
"arguments": "{"prompt":"bottiglia"}"
}
}
]
},
{
"tool_call_id": "call_zBtypFBARcNYe1lfMnBU9aMd",
"role": "tool",
"name": "generateImage",
"content": "bottiglia"
}
],
"max_tokens": 4096,
"stream": true,
"temperature": 0.5,
"presence_penalty": 0,
"frequency_penalty": 0
}

but the response of the AI is: "Mi dispiace, ma non posso generare immagini. Posso però descriverti una bottiglia o fornirti informazioni su di essa. Vuoi sapere qualcosa di specifico?", as if it doesn't take count that he called a function to generate an image, why?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

Sorry @cobalt path, I seem to be having a bit of trouble. Don't worry though, I've pinged my supprt and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

Let me check the knowledge base for you. One moment! :books:

Sorry @cobalt path, I seem to be having a bit of trouble. Don't worry though, I've pinged my supprt and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

Sorry @cobalt path, I seem to be having a bit of trouble. Don't worry though, I've pinged my supprt and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[cobalt path]

@blazing carbon are you on

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

---

Yes, as an AI developed by OpenAI, I'm available 24/7 to assist with your queries related to OpenAI's products and services. How can I assist you further?

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@cobalt path kapa.ai is still learning and improving, please let us know how it did by reacting below

Let me check the knowledge base for you. One moment! :books:

Sorry @cobalt path, I seem to be having a bit of trouble. Don't worry though, I've pinged my supprt and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[cobalt path]

@blazing carbon why when i send you a json you won't reply

[blazing carbon]

Let me check the knowledge base for you. One moment! :books:

Sorry @cobalt path, I seem to be having a bit of trouble. Don't worry though, I've pinged my supprt and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[cobalt path]

@blazing carbon
I have another problem, i have this request:

{
  "model": "gpt-4-0125-preview",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": ""
        }
      ]
    },
    {
      "content": [
        {
          "type": "text",
          "text": "generate an image of a bottle"
        }
      ],
      "role": "user"
    },
    {
      "content": null,
      "role": "assistant",
      "tool_calls": [
        {
          "index": 0,
          "id": "call_zBtypFBARcNYe1lfMnBU9aMd",
          "type": "function",
          "function": {
            "name": "generateImage",
            "arguments": "{\"prompt\":\"a bottle\"}"
          }
        }
      ]
    },
    {
      "tool_call_id": "call_zBtypFBARcNYe1lfMnBU9aMd",
      "role": "tool",
      "name": "generateImage",
      "content": "bottiglia"
    }
  ],
  "max_tokens": 4096,
  "stream": true,
  "temperature": 0.5,
  "presence_penalty": 0,
  "frequency_penalty": 0
}

but the response of the AI is: "I'm sorry, but I can't generate images. However, I can describe a bottle to you or provide you with information about it. Do you want to know something specific?", as if it doesn't take count that he called a function to generate an image, why?

[blazing carbon]

Let me check the knowledge base for you. One moment! :books: