Ideally this would be released in the same way Whisper was, but if that's not happening, I'm wondering if we can at least have custom setups with this new multimodal model with programmatic access to its hearing and sight through the API.
#Access multimodal capabilities outside of chatgpt
peak violet
crystal bearBOT
Hi @peak violet. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Based on the information provided in the knowledge sources, it appears that the multimodal capabilities of GPT-4, such as vision and voice, are currently available within the ChatGPT ecosystem. However, there is no indication that these capabilities are accessible outside of ChatGPT or through the API for custom setups source (https://community.openai.com/t/506972).
In terms of using vision and voice capabilities in the same conversation, it is possible within the ChatGPT environment source (https://help.openai.com/en/articles/8983381-can-i-use-chatgpt-vision-capabilities-and-voice-conversations-in-the-same-conversation).
As for the release of these capabilities in a manner similar to Whisper, there is no information available in the provided sources to suggest that this is planned or possible.