Help with image upload to assistants API

[spring rampart]

Hi guys, I'm trying to build a chrome extension that uses the openai api to send it images. I was able to setup the assistants api to handle chat interactions, but I can't seem to upload images to it, or when I do, the assistant doesn't seem to load them into context or even understand they're there.

Not sure why, perhaps I'm handling it wrong in the frontend. I let users upload image files in the chatbot (through an <input /> tag), and if an image has been uploaded, I load it to the AI as such

fileUpload ? await openai.files.create({
        file: new File([fileUpload], fileUpload.name, { type: fileUpload.type }),
        purpose: "vision",
      }) : null

Not sure if it's how I'm handling this or what, but I can clearly see that images are being uploaded to the AI (I can even see it in the storage tab of the openai dashboard) but for some odd reason the assistant keeps saying It cant see any images. When I try it in playground it works just fine.

[devout citrus]

Hi @spring rampart. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

To upload images to the OpenAI Assistants API and have them recognized in the context, you need to ensure that the images are correctly uploaded and referenced in the message content. Here’s a step-by-step guide based on the information from the knowledge sources:

1. Upload the Image with Purpose "Vision":
   Ensure that you are uploading the image with the correct purpose. The purpose should be set to "vision" when you create the file. This is crucial for the assistant to recognize the image as part of the input.

   const fileResponse = await openai.files.create({
     file: new File([fileUpload], fileUpload.name, { type: fileUpload.type }),
     purpose: "vision",
   });
   

2. Create a Message with the File ID:
   After uploading the image, you need to create a message that includes the file ID in the content. This is how you inform the assistant about the image.

   const thread = await openai.threads.create({
     messages: [
       {
         role: "user",
         content: [
           {
             type: "text",
             text: "Please analyze this image.",
           },
           {
             type: "image_file",
             image_file: { file_id: fileResponse.id },
           },
         ],
       },
     ],
   });
   

3. Ensure the Model Supports Vision:
   Make sure that the model you are using supports vision. Only models with vision capabilities can process image inputs.

4. Check for Errors:
   If the assistant still does not recognize the image, check for any errors in the API response or logs that might indicate what went wrong.

5. Refer to Documentation:
   For more detailed guidance, you can refer to the OpenAI API documentation on creating image input content.

If you follow these steps and the assistant still does not recognize the images, there might be an issue with how the message is being constructed or sent. Double-check the API calls and ensure that all parameters are correctly set. If the problem persists, consider reaching out to OpenAI support for further assistance.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@spring rampart Kapa.ai is still learning and improving, please let me know how I did by reacting below.