When using GPT-4 Vision, I have a tool call that is currently returning an image. I would like to pass that image into message contents (i.e. via image_url). However, it seems like image_url is only supported type-wise in the Typescript SDK when setting the message role to user and not tool.
#Can a tool return an image that is passed into Vision's messages as tool output?
wary ruin
green condorBOT
Hi @wary ruin. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Based on the knowledge sources provided, it appears that the image_url type is indeed used within the context of a user message when interacting with the GPT-4 Vision API. The examples provided in the sources show the image_url type being used in a user message to pass an image to the model for processing.
However, there is no explicit information available in the provided sources about the ability to return an image as tool output or about the support for image_url type in tool messages. The sources do not provide any examples or mention of a tool message containing an image_url type.
Here is an example of how image_url is used in a user message:
const response = await openai.chat.completions.create({
model: "gpt-4-vision-preview",
messages: [
{
role: "user",
content: [
{ type: "text", text: "What’s in this image?" },
{
type: "image_url",
image_url: {
"url": "<https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",>
},
},
],
},
],
});
(Source: Forum Post: Error OpenAI Node.js Overloads when using GPT-4V > Official Answer (https://community.openai.com/t/532712))