FTL mod for Neuro integration

[lapis grotto]

Congratulations to Vedal and everyone on subathon end!

I've been working on this since the game jam start and now I'm ready to actually publish it here - TL;DR the link is https://github.com/chayleaf/neuro-ftl

IMPORTANT IMPORTANT IMPORTANT DISCLAIMER

It's almost fully integrated with the game (I'm finishing up minor parts), it may have some crashes for now, but i have 0 experience with AI agents and I'm 100% sure there are ways that make it easier for Neuro to read the game info. Nonetheless, it fully covers both the base game and advanced edition.

About the game
It's a roguelike about controlling your spaceship, focused on space combat
https://store.steampowered.com/app/212680/FTL_Faster_Than_Light/

It has an "advanced edition" toggle that adds more stuff, the base game has less features so I assume it's gonna be easier to play for Neuro.

Building
You'll have to install Rust using rustup and build it via something like cargo build --release --target i686-pc-windows-msvc. Also there are releases on GitHub but I haven't extensively tested the WIndows version yet.

Installing

See the readme

Steam

FTL: Faster Than Light

FTL is now available in 9 languages! English, German, Spanish, French, Italian, Polish, Brazilian Portuguese, Russian and Simplified Mandarin!************************The free expansion, FTL: Advanced Edition, is available now! Content additions include: new alien race, events, weapons, playable ships, drones, and more! Also adds additional mu...

Price

$319.95

Recommendations

58209

Metacritic

84

▶ Play video

[lapis grotto]

basically: i'd make claude play this to test how viable it is, but i can't pay for it, so all i can do is ask for feedback on what info should be more concise, currently i'm basically exposing everything you could possibly do or see in the game (even more because sensors are ignored at the moment)

current todo list:

• make sensors actually reduce how much info you get
• make beam targeting better (currently it starts on a room center while starting at a room edge is always the better option)
• test stuff, ideally
• make context more concise (idk how exactly but it's definitely a good idea)
• expose info about fired shots (when they are fired and when they hit) because it's important feedback for controlling weapons
• implement actions for resending parts of the context

[glass relic]

🧎‍➡️ I kneel

[lapis grotto]

today i:

• crossed off 2 items off the todo list
• added "tips" (basically prompts that give you an idea of what to do next; tips are only resent once every 5 seconds currently, and only if theres other context to be sent)
• worked on reducing context size
  there's more to be done but i'm slightly more confident about this potentially working now

[lapis grotto]

basically everything is done i guess i'll try to playtest this myself since i don't have an llm to do it for me, to figure out what areas could be improved

(other than crash testing, it probably still has crashes)

[manic rock]

this is awesome! i love FTL and also thought it could be a super cool game for neuro to play one day, thanks for putting in the time and effort to do this! maybe you could make a nice demo video for it after enough play testing?

[lapis grotto]

idk what there is to demo?

[manic rock]

hmm context/hints you would send to neuro i guess, though you did say you were just passing raw text from the game so far right

like for example, an AI understanding how to outrun the rebels and escape to the next sector in time would probably be an issue

[lapis grotto]

idk what you mean

[manic rock]

you know how the more time you spend in a sector in FTL, the Rebel fleet starts to take over, moving from left to right yea? Like the red zone

[lapis grotto]

i mean yeah, obviously the info about the danger zone gets sent to neuro

[manic rock]

mhm, is it just "danger here next turn" or is it danger in X turns for every node? I guess neuro could just compute it and play perfectly in theory but i thought it would be harder than that for her to just like... do it?

[lapis grotto]

yeah im triangulating a rough estimate of turns until danger zone reaches each star

of course the speed isnt constant, it's affected by nebulae and events so i can't predict it exactly

[manic rock]

well the speed kinda is constant, minus nebula which is half speed, but events are just offsets i think

[lapis grotto]

some events double pursue speed for N turns, some events set the speed to 0 for N turns

also nebulae in nebula sectors are faster than normal nebulae but slower than normal systems

[manic rock]

oh interesting, you could probably find whatever multiplier it is somewhere i guess

how easy is it to get like... the XY coords of the nodes?

[lapis grotto]

i mean yeah obviously i have it

[lapis grotto]

what do you mean by coordinates?

there are two pairs of coordinates

grid coordinates

because each sector is 6x4 stars

and, uh, "screen" coordinates

or idk how to call it

it affects which neighboring sectors are and aren't reachable

https://vxtwitter.com/subsetgames/status/1234309658854084608

[manic rock]

interesting, screen coords i think then, im just thinking about the rebel zone right, since its the same curved edge that sweeps the screen, so if you have a distance value for this edge and positions of each node, with a lil maths u should get a pretty accurate system i think

[lapis grotto]

as i said above im triangulating it

theres a point that starts at x=-959 and a random y, and it progresses by a certain amount of units to the right each turn, then everything within 767 units of that point turns into danger zone

if you're interested, the base progress per turn is 32 for nebulae/51 for nebulae in nebula sector/64 normally

i don't use any of 32/51/64 though, i calculate the average across the sector

now that i think of it i should be using median instead

or like max(median, current)

i guess max(average, current) is good enough

[manic rock]

gotcha, though why average? i mean it makes sense since its a lot easier but if you can count the minimun nodes to the exit, can you just add that unit increase for each of those nodes for an optimal route?

[lapis grotto]

minimum nodes to exit isnt the typical route though

[manic rock]

true i was thinking more later into exploring the sector there, so one doesnt fuck themselves over

[lapis grotto]

i guess i could only count the locations that arent already in danger zone for the count

[manic rock]

but i guess then you can go backwards kinda, and work out the maximum nodes without entering the rebel zone, idk

[manic rock]

yea that would be ideal

since you really dont wanna go there haha

and yea that skew from nodes you cant go would likely get neuro neck deep in the rebel zone lmao

[lapis grotto]

it wouldnt

the skew is minimal because the average cant be that much different from the actual value even if you include danger zone stars for the average calculation

[manic rock]

makes sense yea

[lapis grotto]

anyway changed it now

the biggest issue is what info to include when you update neuro of state changes, if you include too much info then its long but if you dont include enough then neuro might not react properly to stuff happening

well, i cant really predict how well she would be at playing in the current state, it seems manageable tho

i also noticed she often forgets about certain actions so i send periodic reminders to her about certain actions

[manic rock]

could you give an example of a state change with too much info? that youre concerned wouldnt work properly with much less info

[lapis grotto]

its not a single state change, its when they are rapid fire because of a bunch of stuff happening during combat like many crew members moving around or suffocating or whatever

i only send the things that changed, so for example when i say that "the crew member X now has 50/100 HP" i don't also mention their species

[manic rock]

Gotcha, what about when they move? Are you parsing positions per tick or crewmate moving from A to B

[lapis grotto]

i update neuro when the crewmember's current room changes

[manic rock]

Hmm, if neuro used pause well enough i think she'd be fine, its just a case of knowing when to use it without spamming it?

Or maybe there could be some threshold to autopause otherwise, idk

Since pausing is pretty useful for players anyway right, not like people play FTL in realtime

(tho idk how much stuff you have currently done with pausing)

[lapis grotto]

i mean its not like theres much to do

i just tell her to consider pausing while in combat and implement pause/unpause actions

[manic rock]

Well since youre only sending useful info, if she pauses more when stuff is hectic to process everything, wouldnt that still work pretty well?

[lapis grotto]

well, the problem is not whether she will be able to read it quickly enough (she will), the problem is whether she will be able to discern what is important and what isn't

and whether she will remember the old info if i send too much useless stuff

[blissful pier]

Nothing to do with anything here

I just find this hilarious xD

I highly support adding this to the twins roster of games she can auto pilot

[manic copper]

It would be neat to pair this with a mod that retextures characters.

[lapis grotto]

my mod only touches the code so the content can easily be modified separately, but also this is a super complex game for neuro and i think many adjustments will be needed before she can play it at all

[plain pecan]

@lapis grotto I tried running your FTL mod with Jippity, and it seems the OpenAI API doesn't like your JSON schemas. I've attached the log from Jippity to this message. For actions that don't take parameters, the Neuro Game API docs say to omit the schema field or set it to {}.

Let me know if you want me to test it again sometime

[lapis grotto]

thats fair, i'll change that

[lapis grotto]

@plain pecan done, fixed hopefully

[lapis grotto]

also your feedback is greatly appreciated because idk what kind of context would work best here

[plain pecan]

The only changes I see are to Cargo.toml and Cargo.lock. I re-built just in case I'm missing something, but I get the same JSON schema error

[plain pecan]

I was thinking about the approach of only communicating changed state. I bet it would work if Neuro was solely focused on the game, but idk if she'll be able to remember all the important bits of information when she's being flooded with messages from chat. It's hard to test with Jippity because I haven't implemented any way for it to intelligently retain important context. I'm only just now experimenting with retaining only the last 10 (or whatever number) messages

[lapis grotto]

that should've fixed it, i'll check in a bit

[plain pecan]

Never mind, it was a skill on my end. I was pointing to the wrong build target. I'm seeing a deserialization error on the mod's side now.

[agile ferry]

You have unlocked new role

[plain pecan]

oh nice I finally have embed perms on this account

[lapis grotto]

what kind of error?

[plain pecan]

I just sent the logs here. The Jippity logs are funny but probably not helpful - the AI sounds just like Neuro, complaining that their tools aren't working

The WebSocket error is from me killing the Jippity process, so don't worry about that part

[lapis grotto]

yeah i'm testing it

sorry for not doing it earlier lol

busy working on a more realistic game integration

[plain pecan]

no worries, I want to help

[lapis grotto]

oh i figured it out, stupid error

fixing

ok fixed please wait until i publish everything

done

it will spam logs im fixing that too

[plain pecan]

Jippity got stuck at the first map screen. I'm currently restricting his context to the past 10 messages, so he's probably just forgetting what's going on. I'll try again with a 25 message memory

holy shit it actually won its first battle

[lapis grotto]

from the logs, it looks like i don't give enough context on what is going on in error messages

i can work with that

(it opened the starmap but then tried opening it again and i said it cant open the starmap so it just gave up)

[plain pecan]

With a 25 message memory, it made it through the first battle but something went wrong on the battle results screen. It's possible that the issue is my fault in two different ways: the issue occurred right as I tabbed out, and Jippity is a little bit buggy when it comes to handling messages in sent in quick succession. I attached the FTL logs here, but I'll need to DM you the JIppity logs - you'll see why.

[plain pecan]

The behavior of kinda giving up when actions don't work seems to be consistent across games for this particular prompt and model. Neuro and Evil are a lot more persistent about doing what they want to do. I could adjust the prompt to try and make it more like them

[lapis grotto]

yeah thats fair

pushed out a new update where context is on a 1s cooldown

and reduced tips cooldown from 5s to 500ms so they're effectively always sent

(as long as there's any changes, that is)

also i think it's probably a good idea to disallow powering up/down the weapons system, instead the AI should explicitly pick which weapons to power up/down

wait i also noticed i sent invalid info about fired shots, oops

i told jippity that the player weapon "drone laser" fired a shot

or rather, it's correct but it's wrong for drones

good to know

[blissful pier]

You guys are doing amazing work! it's lovely reading and seeing the progress made!

[lapis grotto]

oh GOD it turns out that they are passing the wrong ship id to the drone projectile constructor, then they didnt notice that they're passing the wrong ID, but still had to fix it somehow so they MANUALLY set the ship ID for drone projectiles

and they're doing it after initialization

its fine i'll fix it

[plain pecan]

I don't know how that previous run went so well, but I'm having tons of new and exciting issues on Jippity's side. It even just stalled while outputting a message once - no idea how or why it did that.
One limitation of Jippity that Neuro doesn't have is that it loses its memory when restarted. I restarted it after it stalled and although it seems like all the actions were re-registered, the AI had no idea what to do and just kept roleplaying. That's a problem for me to figure out, though, not for you to worry about

I've considered removing Jippity's speech functionality entirely, but it's been really useful for troubleshooting. It's easy to notice a problem when the AI is complaining about it.

[lapis grotto]

i dont blame you, FTL is just barely possible to play for current LLMs, at least at this level of simplification (i.e. none, lol)

also noticed a weird thing with context that i'll fix

OH

i was wondering why it picked room id 0 as the target

like, i can imagine its less bloodthirsty than neuro but it still should target weapons or shields as a good strategist

but then i realized that as a part of the update that made sensors affect context, i removed all system data from the context if sensors arent of a high enough level

while i should only remove info about power and stuff but not about systems' locations

so basically the AI just doesnt know where enemy systems are and cant pick optimal targets

[lapis grotto]

well, changed up some stuff to hopefully fix some of the issues

[lapis grotto]

made it so it resends full ship info in every new location

[manic copper]

Is Neuro able to name the crewmembers? That would enhance the experience quite a bit.

You could make it a prompt at the start of a run or when you gain a new member

[lapis grotto]

she is

she can do anything a human player can do

which isn't necessarily good because sometimes you'd want to simplify stuff for AIs to be able to do it

i could write a ship control/crew control AI or just reuse parts of the one the game already has, and give neuro control over it, but i just want to see how well she does by herself first

anyway, she can name the crewmembers and the ship, these are actually the first features i implemented

[lapis grotto]

FTL mod for Neuro integration

[lapis grotto]

oh just realized i forgot to "make beam targeting better"

[manic copper]

It would still probably be a good idea to turn renaming into a prompt as soon as you get a crewmate both to make sure she actually does because it will be good content, and to reduce the number of options during normal gameplay.

[lapis grotto]

I currently advice her to do it but I don't force her to