ChatGpt is Now Inside Game and It Talks

[oak talon]

https://youtube.com/shorts/m8kw7n3cMgM?si=orZYmeJrij3X_vTf

This is the first shorter video of a live, unedited demo of Andromeda AI, an experimental in-game assistant built using ChatGPT for dialogue and Speechmatics for speech recognition. It's been developed by a solo developer as a passion project!

🎮 Captured in one continuous take, this video demonstrates how AI can now:
• Understand natural language commands
• Respond with human-like awareness
• Control complex in-game systems — all through your voice

No scripts. No post-production. Just AI, conversation, and spaceflight.

💡 This is the future of gaming — where you don’t just play the game, you talk to it.

For more information check out Full video demo on my channel

[half pendant]

That’s actually really cool, is this running locally or through an API call?

[oak talon]

Glad you enjoyed it its using API Calls, the tech stack
• Speechmatics – Speech-to-text recognition
• OpenAI ChatGPT – Conversational intelligence
• ElevenLabs – Human-like voice synthesis
• Silero VAD – Voice activity detection
• Spotify API – Real-time in-game music control

[half pendant]

That’s awesome, that’s the second time that I’ve seen Elevenlabs being used. I wonder if this can be ran locally using OSS tied into workflows and using something like a local instance of Whisper…

[oak talon]

Full video here: https://www.youtube.com/watch?v=RBTL7NGLx40 in description you can find reddit post I answered tons questions

[oak talon]

It is certainly a possibility to run locally, however you will sacrifice accuracy and precision as there would be higher impact on GPU and some quality and speed reduction. For gaming generally cloud approach works better as you are online anyway. Both both approaches can be done.

The current speech provider is Speechmatics, it's the real magic there as their transcription is already quite optimised and sub-200ms latency. You could run Whisper locally, but it would be much slower unless you have serious GPU horsepower (RTX 4090 territory). I think my current latency was around 800ms-1.5s, and most of it just comes from LLM thinking time (and I use free tier on that ^^), not network. Would dedicated instance help? Maybe shave off 200ms on LLM but you would lose quality and still need Speechmatics cloud STT.

[half pendant]

Yeah, unfortunately I only have a 3090 so I’m guessing the latency and game performance would degrade. I think it’s an interesting concept as I see horror games starting to do things like this leaving me wondering how they were accomplishing this. I’ll have to look into Speechmatics as I am not familiar at all.

I’m curious, how long did it take you to think through and engineer this?

[oak talon]

Honestly the back story of this project is I always been huge believer that voice will be the next UI medium. The problem over years games have struggled with voice integration because its difficult to understand many voices with different dialect and accents. IN most games you say go left the character goes right or just dont understand. Now it got to the point with AI advancements its very good. Specifically Speechmatics tech allowed me to have great Speach to text ensuring that it knows exactly what I said.

The project started as small proof of concept for interview but I kinda took it to next level. Basic version was out in less then week. I always wanted to have ai assistant in games and it just fuled me to get this done.

[half pendant]

That’s really cool and very interesting, thanks for sharing!

[oak talon]

I needed great TTS, STT and LLM and find way to basically connect all of in game. Lots of debuging and hair pulling for sure

Its great use case not only from gaming perspective but also accessibility allowing people with vision impairment and mobility restriction to experience game