can we do image input or image url input in custom ai assistant api thread?
#image input in custom ai assistant api
cyan hatch
wind karmaBOT
Hi @cyan hatch. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Yes, you can include image input in the message content for the custom AI assistant API. The message content can contain either external image URLs or File IDs that have been uploaded via the File API. However, only models with Vision support can accept image input. The supported image content types include png, jpg, gif, and webp.
When creating image files, you should pass purpose="vision" to allow you to later download and display the input content. There is a 100GB limit per project for this purpose.
To pass image files to Code Interpreter, you need to add the file ID in the message attachments list. This allows the tool to read and analyze the input. However, please note that image URLs cannot be downloaded in Code Interpreter today.
Here is a Python code example of how to create image input content:
file = client.files.create(
file=open("myimage.png", "rb"),
purpose="vision"
)
thread = client.beta.threads.create(
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is the difference between these images?"
},
{
"type": "image_url",
"image_url": {"url": "<https://example.com/image.png"}>
},
{
"type": "image_file",
"image_file": {"file_id": file.id}
},
],
}
]
)