ChatGPT integration within Unity Game Engine as AI Agents

[true latch]

Hey everyone! 👋

I wanted to share with you a project I've been working on that explores the integration of OpenAI's ChatGPT model into the Unity game engine. It's all about empowering AI agents to comprehend and interact with the game world.

If you're interested in AI-driven experiences and game development, this project might catch your attention. It's still a work in progress, but I believe it could be both useful and interesting for those looking to incorporate AI into their Unity projects.

Feel free to check out the project - https://github.com/Michael97/ChatGPT-Unity-Demo

I'm open to any feedback or suggestions you might have!

Thanks all 🙂

[hollow nest]

Interesting. I'm implementing something very similar for my game.

I'm curious about how many tokens your average request+response contain. I figured costs are the main limiting factor for an actual production project.
Ideally you'd want to the agent to update at least every 10-20 seconds. And the amount of context it needs would depend on how immersive you want it's responses to be. It wouldn't be surprising to have several thousand tokens used per update and that's just only for one npc. Ideally we'd want several npcs in player range to be updated by the agent. That snowballs very fast up to tens of dollars per hour. And that's just for one user. Assuming an average user player 2 hours per day, it's gonna be several hundred dollars per user per month. And that's just the costs of the open ai API. There are probably not many people that would agree to a subscription plan of several hundred dollars per month.

Did you give that a thought? And if so, any ideas how to tackle that?

[true latch]

For tokens you're looking at about 70-140 tokens, with initial prompts needing 2-300 tokens, and I purge all tokens apart from the prompt tokens when we are reaching the token limit.

I think one solution to the API calls is to only make calls when the game deems necessary, and to have the majority of the AI Agent still be controlled by traditional methods, but decision making based on the LLM. Could expand this to use an LLM AI Director to control multiple agents in this way. https://voyager.minedojo.org/ is an interesting read on how they achieved a LLM controlling a AI Agent.

In the future I think we might see smaller LLM that are just as capable that could run locally, meaning no fees for API calls.

I haven't worked on the project for a few months so my memory might be a bit skewed on the token numbers.

To me this is just an experiment that would likely not head into prod at this current level 😅

[hollow nest]

I see. I was thinking about mixing with conventional ai. Specifically, I think goal oriented action planning would prove to be a good combo where the LLM agents would only define the final goals for the GOAP to follow. This way we would only need to update it when the goal is achieved.
This however could potentially limit the immwrsivenes of the ai character. For example, if the player talked with an NPC in natural language and managed to convince it to follow the player to certain location, the LLM would need to set a goal of following the player until reaching a certain location. In this case we would not need to make another request until the player reaches the destination. However, there's a whole window of opportunity for the ai to interact with the player or react to changes in environment during that travel. For example, the npc could give remarks about the locations they see on their way in natural language or change the goal entirely if it decided that it's reasonable to do so. This would add realism and immersiveness to the gameplay, but would require more frequent updates from the LLM agents, meaning higher API costs.

Also, from my experience, even employing all kinds of prompt and message history optimization techniques, I'm still using 300-1000 tokens per request.

I feel like collecting a good dataset with gpt3.5 API and then using it to fine-tune a cheaper open source model is the way to go, but I'm worried that it wouldn't be smart enough to produce desired and varied output.🤔

[true latch]

Using a combo of GOAP and LLM to drive the goals could be the next move. To handle how many requests we're making to the LLM, we need a smart system that figures out if a change in the game is big enough to require an update to the LLM. Like, if the LLM has chosen to follow the player, it should only get updates when there's a potential threat, a question from the player, or a significant narrative moment. You know, kind of like when characters in "The Last of Us" come across something and start talking. We could hard-code these moments (simple trigger colliders) to prompt the LLM to say something. But if we know there's going to be dialogue, maybe it's best to leave that to the general AI.

As for the token usage, I don't think we're ever going to get it really low for these kinds of scenarios. Our best bet is hoping for a drop in token cost or some serious optimizations. There's already been a 33% price drop, so we're moving in the right direction.

In the end, I'd like to create a demo AI that you can hook into any LLM API to test out these ideas easily 😄

Personally I'm hoping that we can run small LLM locally which would solve a lot of cost issues, but will probably raise gpu issues haha

[hollow nest]

I don't think LLMs would get small enough or consumer GPU strong enough to run them locally and produce good results any time soon. Even smaller LLMs might require to run on a server. But at least this way the pricing would be more predictable and probably cheaper.

Maybe in 10 years something like that would be possible locally.

[true latch]

Yeah, running LLMs on your own servers would likely be the cheaper option in the future. There is also a significant delay in request to response atm, so optimising the speed of responses would help in real-time scenario's.

[lofty vine]

ol