Touhou Integration

[upbeat wing]

This is a concept for a way for Neuro to play Touhou using the Godot Neuro SDK.

Because Touhou is closed-source, and I have no experience in modding it, I've opted to recreate Touhou in the Godot game engine instead. Not only am I more familiar with it, but the Neuro SDK solution for Godot already exists, and I'll have a lot more control over the actual game that Neuro will be playing.

When I have the time, and depending on how things go, I will be making this using one of the methods listed below (Or a different one if a better one comes up). This thread is currently mostly here to brainstorm and ask for help on how I could best go about doing this.

Here's a few methods I've brainstormed on how this could work

1. Turn Based
   Neuro is (from what I've seen and what Vedal's told us) not great at playing real-time games, so I could turn touhou into a bit more of a semi-turn-based game. The game pauses and asks Neuro where she wants to move, while giving her access to the location, hitbox size, and movement speed of every enemy, and enemy bullet. (Also her own hitbox size). Then the game runs for a short bit with what she decided, and then pauses again.

2. Precalculated
   The game is fully deterministic, meaning that future gamestates can be calculated. We ask Neuro in advance for every gamestate at the beginning of (or throughout) the run until she wins or dies, and then I run the game visually for the audience to view while sending Neuro information on what is happening so that she can react without having input.

3. Fuck it we ball
   We just ask Neuro whatever she wants to do on a specific occasion once every few frames, however fast her response time is, while giving her access to the entire gamestate at that frame.

4. Fake AI player
   Code an AI player that has variable stats (Not damage stats or health stats, CPU Intelligence stats) that determine how good it is at playing the game. We ask Neuro to allocate these stats to her liking, while also giving her more stat points to play with after every run (Or just over time as she plays? Could also allow her to change the stat distribution during the game), simulating her getting better at the game.

I would like some input on which of these options is most appealing to watch as a viewer and most feasible for me to complete, as I am only one guy who's still pretty new to Gamedev in general. Thanks for reading!

Edit: I've decided to go with option 4

[sweet goblet]

Why not just do the same thing that the osu ai did
(I know little about ai)

[upbeat wing]

That's sort of what I have planned with the 4th one, it's not really "Neuro" playing, but it looks like she's playing. While also giving her at least a little control over how things go

[zenith flax]

1 and 4 seem more practical ways of implementing it (so she has a good chance to win) , but I also like the idea behind 4 so maybe that? 1 might feel a bit awkward since that's kindof the opposite of the intended experience and defeats the point a bit

4 would make it strategy-based while also keeping it fast-paced and being a novel idea in general

[sweet goblet]

Even if it technically isn't Neuro's main llm "playing", it'd be nice to see her be good at a game for once

[red kindle]

Its boring

[upbeat wing]

Thanks for the input

I do already have some ideas for the "stats" she'd be able to control, but I would also like some feedback/input on those, being:

Input speed (How long of a delay is there between the AI Players' inputs)

Reaction speed (How long after a bullet or enemy appears does the AI Player know it's there)

Riskiness (How "afraid" the AI Player is of bullets or enemies, doesn't really change how well it plays (Could actually make it worse) but would work well for making the way they play more characteristic. Maybe Neuro decides that she wants to play incredibly risky, while Evil decides to do the opposite.)

General Intelligence(?) (Knowledge of certain game mechanics and using them effectively)

etc.

[sweet goblet]

I was imagining some sort of strategy-related modifier, if that would make sense for touhou

Like whether to be more risky with movement or smth

[upbeat wing]

Yeah those would also work, I should probably have tose separate from the actual "gameplay" stats so that how they decide to play is separate from how well they play, but yeah good shout

[sudden rain]

good to see you still exist

[signal coral]

1 seems kinda antithetic

to touhou

I'd say just 4

[upbeat wing]

Currently in the process of figuring out how the Neuro SDK works in general, I think I can manage this

[signal dirge]

I agree #4 is the best solution if Vedal doesn't want to train a new NN. I think she should be able to re-allocate stats at any point during the game (like a real player could change their strategy for a specific section). There should also be context for every spell card, probably non-spells and stage sections too (though the latter two don't have official names). It might also be helpful to describe the pattern to her so she can decide how to allocate stats based on that.

I fear that this will turn into a game of remembering a specific strategy for every section though, which while not dissimilar to actually playing Touhou, I don't think she will be good at. There should also be enough different stats to make a number of different strategies possible, and if taking the mid-game adjustment approach, there should not be one strategy that works for the entire game.

Also, I believe LOLK (Pointdevice mode) would work best for this since then we don't have to watch her be reset to stage 1 forever

[finite valley]

to be true its mostly depend how you plan to handle it to original gameplay
you can make 4 as neuro start with all and some basic settings like here try more defensive, or look out more on boss than close etc.
or do more lite-roguelike style somehow (like made her earn 'points' depend how well she go and them use points to 'upgrade' her stats)

[sweet goblet]

Any updates?

[calm spire]

Hey I just happened to see this thread. Touhou was actually one of the first games I wanted to integrate with the real-time API the community is building in #1457935008764399761 . It does pretty much exactly as you are describing (LLM tells a secondary smaller nn what to do). So I think we should merge our efforts on this together! (Plus I just love touhou)

We are collecting data to make this a reality. Perhaps you'd like to assist us in collecting data for the touhou games?

We are training the NN ourselves so Vedal wouldn't have to worry about it. We also ensured the l*tency would be low. The twins are currently set to play at 8hz which should be just good enough for touhou.

We train a general model which can load "game cartridges" which are extentions to the main model which make the AI specialized in a specific game without needing that much extra data

[sweet goblet]

Honestly, i'd love to see Neuro learn from scratch rather than being heavily pre-trained, kinda like with osu, but idk if that'd take too long with touhou

[upbeat wing]

Have been busy with other stuff in the meantime, I wouldn't be surprised if it'd take a while before I start actually working on this, but I'll notify this thread when that does eventually happen. So no updates as of now

[calm spire]

That'd be pretty difficult

I mean it could be done

but it's either "Learn to listen to instructions" or "Learn to play well"

it's really hard to do both. No company I know has been able to pull that off

[calm spire]

WRITE THAT DOWN WRITE THAT DOWN