Vision syntax | OpenRouter | Page 1

modest crater Dec 21, 2023, 9:40 PM

#

<@&1094455453599137872>

topaz magnet Dec 22, 2023, 12:32 AM

#

That looks correct to me - but you will also need a human prompt as wel

modest crater Dec 22, 2023, 9:22 AM

#

[llm/error] [1:llm:ChatOpenAI] [5.26s] LLM run errored with error: "Unsupported MIME type: application/octet-stream"
[04:18:31.385] FATAL (icg/22039): "uncaughtException" detected
    Error: Unsupported MIME type: application/octet-stream

#

At first look, I think it's an issue with "streaming", does Gemini Pro Vision doesn't support streaming? Now I'm getting a different error (progress I guess x))

topaz magnet Dec 22, 2023, 9:26 AM

#

Gemini on OR supports streaming

#

Can you output the full JSON request that you're sending to OR?

modest crater Dec 22, 2023, 9:28 AM

#

[04:27:06.343] FATAL (icg/22990): "uncaughtException" detected
TypeError: Cannot read properties of undefined (reading 'message')
at ChatOpenAI.call (/node_modules/@langchain/core/dist/language_models/chat_models.cjs:298:34)
at processTicksAndRejections (node:internal/process/task_queues:95:5)

/Users/zakariamehbi/Code/InsertChatGPT/server/node_modules/@langchain/core/dist/language_models/chat_models.cjs:298
return generations[0][0].message;
^
TypeError: Cannot read properties of undefined (reading 'message')
at ChatOpenAI.call (/node_modules/@langchain/core/dist/language_models/chat_models.cjs:298:34)
at processTicksAndRejections (node:internal/process/task_queues:95:5)

topaz magnet Dec 22, 2023, 9:29 AM

#

Should look like this:

📎 message.txt

#

hmm..

#

Is the kwargs mapping correctly via the openAI client you're using?

modest crater Dec 22, 2023, 9:29 AM

#

Same / Same, no?

#

I'm using Langchain

modest crater Dec 22, 2023, 9:30 AM

#

topaz magnet Is the kwargs mapping correctly via the openAI client you're using?

Idk what that means

topaz magnet Dec 22, 2023, 9:31 AM

#

modest crater Idk what that means

basically whether the argument you have above maps correctly to the API request shape

#

If you try a public image from wiki for example, does it work?

#

Start from there, then debug your public URL

#

I suspect your CF R2 public URL doesn't have the correct mimetype

#

and is returning an octet stream (as specified in the error)

modest crater Dec 22, 2023, 9:31 AM

#

Actually it's working with GPT 4 that I use directly not via OpenRouter

#

Just not with the models of OpenRouter

topaz magnet Dec 22, 2023, 9:32 AM

#

hmm, so even the GPT4 on OR doesn' work??

modest crater Dec 22, 2023, 9:32 AM

#

MIME type: application/octet-stream I get only when I enable streaming, if disable I don't get it

modest crater Dec 22, 2023, 9:32 AM

#

topaz magnet hmm, so even the GPT4 on OR doesn' work??

Not sure, but I can try

topaz magnet Dec 22, 2023, 9:32 AM

#

I don't think we're cooking the URL if it's GPT4, only with Gemini and the OSS models since they don't understand public URLs

modest crater Dec 22, 2023, 9:33 AM

#

So you want me to send the blob/binary?

topaz magnet Dec 22, 2023, 9:34 AM

#

It should be a data-base64 string OR an URL that points to an image file with the correct content-type header (not octet stream, which is a raw binary stream)

#

We might consider supporting those octet links tho, but it would take a bit. Not sure if the octet binary contains any mime type identifier or would we need to parse the header in the binary header blob :d

modest crater Dec 22, 2023, 9:37 AM

#

I will try to make it work on my end. Now that's the issue with the generation being empty, no?

#

What about the streaming?

topaz magnet Dec 22, 2023, 9:37 AM

#

Streaming should work afaik - check the playground

#

topaz magnet Dec 22, 2023, 9:38 AM

#

modest crater Not sure, but I can try

lmk if this doesn't work - that'd be a P0 bug on our end

modest crater Dec 22, 2023, 9:39 AM

#

So, I tried openai/gpt-4-vision-preview, everything is working fine even with my signed url + streaming

#

Using gemini pro vision, the url isn't working, the streaming either

topaz magnet Dec 22, 2023, 9:41 AM

#

Yeah if the url points to an octet url, we are not parsing it down atm (since the underneath API requires the correct mimetype)

modest crater Dec 22, 2023, 9:42 AM

#

Oh, it's the same issue you're saying. So if I fix my URL, everything should be fine then

#

@topaz magnet Added contentType to my signed url with auto detection using the path, and I got the response from gemini vision with streaming. killer support!

topaz magnet Dec 22, 2023, 10:55 AM

#

We will eventually add support for raw octet stream, but likely will be after the holidays

rose ore May 28, 2024, 3:39 PM

#

Hey - I've a mp4 served from telegram which looks to come down as a octet stream from their servers. I wanted to send this to gemini vision but receive the same error as discussed in this thread. What was the fix in the end? I cant change the mime type their server sends.

#Vision syntax