About Neuro with Games, sorry it's long

[jagged wagon]

Note: I'm new here so I don't know if this has already been suggested or has already been though of by Vedal, I also do not have a background in programming at all.

Can't Neuro be given a digital keyboard? And then just have her learn visual perspective when playing a FP 3d game and to identify herself in a third person/platformer game.

Not sure how it would work nor if it's easier or harder, but the idea is for her to hopefully learn the basics of a standard game like WASD or arrow keys with space bar through the use of the keyboard, that way through trail and error while playing or simply going through a tutorial or trail run she slowly becomes more competent play the game.

Again, do not know a thing about programming, and not sure if this is how she already plays games, cuz it sounds like Vedal hard codes her, as in merges her into the game itself thus making it a grind for her to be able to play, or that's not how it works?

[orchid mesa]

!!!possible missinfo!!!

If iirc vedal said it would require a lot of hardware to calculate her vision at more than 3fps. And I don't think you can just give a LLM keyboard and vision and say "play game".

For how she play games now (If i understand correctly):
Inscryption and sts(probably) fully translated to pure text that is inputed in llm and neuro makes a decision based on it

That is how I understand it. But I hope a real programmer will answer your question (not a first year computer science student like me)

[balmy oriole]

so couple of issues
what you are describing is actually a reinforcement model, that would be a totally separate model from neuro
when neuro played osu, she had one of those probably

now neuro's model is a multimodal llm which is totally different

about using a virtual keyboard, she could only play turn based or story based rpg games on a sunny day
the response time would be too slow and improper for anything else

[solid zodiac]

Go back and watch the minecraft streams. That would be what her playing FPS games based on a reinforcement model would look like

Except even that is being generous because the model likely wouldn't have any concepts of the game's goals, and if you train too much on a single game your model will only work on that game instead of being a general FPS playing model

And to get the reinforcement model to play the game accurately, a developer would need to define certain goals to reward it, which basically puts you back at square one

for osu its pretty easy - just click the circles 4Head. But for a game like minecraft, the goal is constantly changing based on context, which makes it very hard if not impossible to make a single reinforcement model play the game, let alone a general purpose FPS model.

[jagged wagon]

Ah, got it. For Minecraft I kinda expect her to be shit at it, with her being a literal bot and more or less kinda similar to a kid playing for the first time and she just slowly gets better at it.

I was looking at it more like how kids learn to play, where most games automatically have the same default controls and they just fiddle around till they understand the basics going by visual cues and perspective vision for 3d games. Then identifying where in the Hud of the game they can see where the missions are or what the Icons for objects mean and so on.

So Vedal would have to basically start from scratch each time regardless huh and she would have too slow of a reaction. Damn that sucks

[sonic sparrow]

Not sure if this would be relevant here but thought that this video was pretty interesting since it showed a way on how Neuro could be made to interact with games like minecraft. In the video, the guy creates a system where: you can prompt the AI with 'goals' to accomplish or have the AI generate its own goals indefinitely. Basically, he creates whenever a request is typed it dynamically calls to a command that tries to accomplish the request, and when the AI is done with the command, it is set up with an additional command stating that it needs to have a goal, creating an endless loop. I don't think that this would lead to Nuero playing more like an actual player, but it may allow for 1. endless gameplay where she is self-directed, and 2. better coop for collabs.

https://www.youtube.com/watch?v=IeXadWbvDiE

[shrewd dock]

I would like to see her play Postal 2