Stack buffer overrun when saving

[lapis gorge]

Sometimes when the game saves it will "crash" (get terminated by windows due to buffer overrun) leaving no trace of the event except in the event viewer. This also causes the save file to be 0 bytes long.

Godots log is of no use since nothing catches the error before termination and writing to the log is interrupted mid write

Loading resource: res://assets/enemies/enemyPurple2.png
Loading resource: res://assets/enemies/enemyPurple4.png
Loading resource: res://assets/enemies/enemyPurple4.png
Loading resource: res://assets/enemies/enemyPurple2.png
Loading resource: res://assets/enemies/enemyPurple2.png
Loading resource: res://assets/enemies/enemyPurple4.png
Loading resource: res://assets/enemies/enemyPurple2.png
Loading resourc

The issue started occurring 3 days ago and i have spent that time looking for issues with my computer but no luck so far. SpaceIdle is the only known process to crash with this error at the moment.
I tried inspecting the savefile to look for anomalies but i dont have the tools to decode it.

[hollow harness]

Can you plop a save in here?

[lapis gorge]

Here. I ran another memory diagnostic just in case and got no faults.

I can debug more on my machine when i get back home if i have the necessary debug files.

[hollow harness]

I'll see of i can recreate when I get some time this evening

[lapis gorge]

Managed to decrypt it. I have 2 pairs of duplicate "save_name". OverdriveTier32 and Disable10000 achievements with each pair disagreeing if i have that achievement or not. They are also in a weird location having gotten inserted out of order just before the rest of the achievements on line 1194.

I have attached a valid json formatted version so it is easier to parse.

feels unlikely for that to be the cause but is probably at least related to it somehow

[hollow harness]

ya that wouldnt matter, if anything that might cause a load to be weird

I tried saving a bunch of times after loading your finle on both upcoing bpatch branch and live and it didnt even hesitate

[lapis gorge]

I think it is likely an interaction with something on my machine. But im still no closer to identifying what that may be. I with i had a more complete stacktrace but all i got is the fault offset. That is in the exe and i dont know if that contains any game code or if it is just the Godot game engine.

[lapis gorge]

may i request the debug symbols? i attatched windbg but i cant make much sense of the stack

[hollow harness]

uh this build should have them I think

[lapis gorge]

i was hoping for a pdb for the current build. ill see if i can reproduce the crash with the one you sent.

[hollow harness]

Oh I don't think i can get a separate pdb from godot

[lapis gorge]

00007ff7`767de620 godot_windows_opt_debug_64!Z13widechar_mainiPPw (<no parameter info>)
00007ff7`767eceb0 godot_windows_opt_debug_64!Z5_mainv (<no parameter info>)
00007ff7`78668160 godot_windows_opt_debug_64!main (<no parameter info>)
00007ff7`78677420 godot_windows_opt_debug_64!NvOptimusEnablement (<no parameter info>)
00007ff7`78677424 godot_windows_opt_debug_64!AmdPowerXpressRequestHighPerformance (<no parameter info>)

these are all the symbols available in that file

https://github.com/godotengine/godot/issues/37864
https://forum.godotengine.org/t/how-can-i-debug-the-engine-itself-on-visual-studio-2022/1241
these are the best leads i got so far. i have never used godot myself so i dont know much about the project structure. i get the impression the accompanying exe is just the engine without the editor, but im unsure if this gets buildt together with the pack file or if it is distributed with the engine.

https://docs.godotengine.org/en/stable/contributing/development/compiling/introduction_to_the_buildsystem.html#debugging-symbols

so im fairly sure the crash is in the engine, but parameters passed from gdScript may still be of relevance

[hollow harness]

im using a custome build of godot for steam realted stuff and can't get it to give me the .pdb file at all, I might have to rebuild the editor from source to do that it seems? Im not 100% sure

[lapis gorge]

that or maybe the export template? is the export template the same as the bundled executable?

[hollow harness]

no its a bit different, ill try a debug export emplate and see if it does it

hmmmm, nope

@pearl rock does any of this sound familiar for stuff you had to do to debug on mobile (obviously different but how you did it might work for this anyhow)

[pearl rock]

Will check in when I get back to normal dev setup tomorrow

[lapis gorge]

i swapped the executable to see if that makes a difference. the debug title and buttons are present but the pack file is the one from steam. so it either contains some flags or some game code and isnt purely the engine as i believed it might be.

its a bit laggy but stable so far. will be leaving it running for a while

[lapis gorge]

have to shut down now but its been running for 3.5h no crash

[hollow harness]

and it would crash with auto save triggering sometimes on live version?

hrm

[lapis gorge]

Yes that is correct. I will try to compare the executables tomorrow and see if there is a version difference on the engine.

[hollow harness]

shouldnt be

[lapis gorge]

left is the debug build you sent. right is the steam version.
its the same Godot version but one is labelled as a custom build.

[lapis gorge]

Another difference i see is the linker version. Steam: 13.37 Debug: 2.37

Debug file also has 12 instead of 8 sections

[pearl rock]

Nothing here rings any bells for me.
Just a few notes/random thoughts:

• No updates on our side for a while
• Save failing could potentially also be config file related. You say saving "sometimes" crashes -- could that be after making changes to window size/or similar? Is the config.cfg file protected or locked somehow? [Not super likely since you mention savegame becomes 0]
• Does a restart in any way help? 😅

I can probably poke the people over on GodotSteam discord and ask about debug symbols

Oh someone once had similar issue when they ran out of hard drive space 🤔 Might be unrelated in this case, though

[lapis gorge]

it has been working fine with the debug build.

still got 200GB

i have done several restarts. one after running sfc and two when i ran memory diagnostic twice

i too found it strange that no update had happened

though possibly a windows update

probably isnt worth spending much time on this issue given that so far it is only me it affects and is rather difficult to identify

[pearl rock]

I don't have Sylv's export settings for debug locally, so not sure on the details there.
The custom_build suffix is what is used for GodotSteam, as mentioned -- somewhat strange to me why it's not the same for the version Sylv sent -- I haven't really done a lot of PC exporting

Crashes during save like this has usually been because we have ended up with some funky states for something, such that when it tries to create the savegame dictionary something explodes, but then usually the save has then also crashed when we've tried to save. That's not the case here, the savegame loads & saves fine for me when I test it here as well 🤔

[lapis gorge]

loading is no problem and saving works most of the time. the crash seems to happen after opening the file but before writing

[pearl rock]

Checking the code, the error handling for saving game is... less than stellar 😅

[lapis gorge]

i think the best course of action is to see if you can get windows export to generate the pdb and if i still have the issue with that version i can do some proper debugging.

[hollow harness]

Ya im not sure why it seems different

[lapis gorge]

i see no reason GDScript should be able to cause a buffer overrun on its own, so it is most likely engine related.

[pearl rock]

Hm

Might be if save file fails to be opened for write correctly, then trying to plonk data into it --> buffer overrun

I'll try to ask GodotSteam people how to generate pdb.
I'll also see if I can sneak in some slight improvements to the save error handling -- at least try to print out what the error is if it encounters anything

[hollow harness]

ah if its not similar to what were doing for debug onandoird I can look into it more later

[pearl rock]

No that was some other funkyness

[lapis gorge]

at some point during my testing all crew data got wiped. got a save that isnt too old, but im curious. will the first crew member unlock on returning to the unlock sector? if not i would guess a prestige reset could enable that. either way it doesnt really matter.

[pearl rock]

I'm not sure on the details there, that's all Sylv territory

[hollow harness]

this has a little extra print logging/error logging for saves

[lapis gorge]

today the debug build decided to hang. on inspection it appears ALL threads are suspended. it is also seemingly unable to load crew data. unless it is failing to save them.

wacky stuff

[pearl rock]

If you use that new .zip file, does the log file contain anything new when it crashes? That might help track things down

In other news, I got a reply from the GodotSteam discord on how to create the debug symbols:

Download Godot source, check out stable commit, add GodotSteam to the modules folder, follow Godot docs for compilation commands, add debug_symbols=yes and separate_debug_symbols=yes.

Bonus points, add sentry-godot.

[hollow harness]

So you do have to rebuild from source ><

[lapis gorge]

figured out why the crew kept resetting.
"mastery_this_sleeve":8038102.146532,"xp":,"xp_needed":}}
no idea what causes it but it appears to be every save made with that first debug build.

[lapis gorge]

on another note i made a little tool in CyberChef
https://gchq.github.io/CyberChef/#recipe=Drop_bytes(0,8,false)Subsection('^(.{16})',false,false,false)To_Hex('None',16)To_Upper_case('All')Register('^(.{32})',false,false,false)Drop_bytes(0,32,false)Merge(true)Subsection('^(.{8})',true,true,false)Swap_endianness('Raw',8,false)To_Hex('None',0)From_Base(16)Register('([\\s\\S]*)',false,false,true)Drop_bytes(0,8,false)Merge(true)AES_Decrypt({'option':'Hex','string':'6265323137626430313630636130623139316464333463303433363235366533'},{'option':'Hex','string':''},'ECB/NoPadding','Raw','Raw',{'option':'Hex','string':''},{'option':'Hex','string':''})From_Base64('A-Za-z0-9%2B/%3D',false,false)Gunzip()Drop_bytes(0,8,false)Filter('CRLF','"save_name":"Crew"',false)Regular_expression('User defined','"save_name":("[^"]*")',true,true,false,false,false,false,'List capture groups'/disabled)JSON_Beautify(' ',false,false).
tried masking the ugly link but that broke it

[hollow harness]

youre crew saving that way is weird I dunno what would do that

[lapis gorge]

what can i say. im like a magnet to bugs and glitches.

[opal lagoon]

I'm starting to think your computer is cursed lol

[lapis gorge]

its not the computer. it is me

[opal lagoon]

ah fair enough. I have a friend who is known to have notoriously bad luck. it's unbelievable tbh, but it keeps happening

[lapis gorge]

the savelogging build just froze a few minutes ago so i checked logs.

Loading resource: res://assets/turrets/gatling_laser1.png
Loading resource: res://assets/turrets/missile_launcher1.png
Loading resource: res://assets/turrets/capital_burst_cannon.png
Loading resource: res://player/ships/BattleCruiserDualHangar.tscn
Loading resource: res://player/CapitalShieldBar.tres
ERROR: Error parsing JSON at line 0: Expected value, got ','.
   at: call (modules/gdscript/gdscript_functions.cpp:1215) - Error parsing JSON at line 0: Expected value, got ','.

this is from a regular save and exit and im now confident the json it tried to parse is the crew save.

Loading resource: res://enemies/GigaEnemy.tscn
Loading resource: res://assets/turrets/Big/turret_03b_mk3.png
Loading resource: res://enemies/GigaEnemy.tscn
Loading resource: res://enemies/GigaEnemy.tscn
Loading resource: res://assets/turrets/Big/turret_03b_mk3.png
Loading resource: res://enemies/GigaEnemy.tscn
Lo

this is the current freeze

[lapis gorge]

once i got a notif of someone managing to guess my appleId password (was old and outdated). so i went into settings on my phone to change it. the sections that contains security related settings ask for the lock code on the phone as a safety mechanism.
upon entering the correct pin another modal (same as the existing one) appeared on top of it asking for my pin...
upon entering the correct pin another modal (same as the existing one) appeared on top of it asking for my pin...
upon entering the correct pin another modal (same as the existing one) appeared on top of it asking for my pin...
upon entering the correct pin another modal (same as the existing one) appeared on top of it asking for my pin...
upon entering the correct pin another modal (same as the existing one) appeared on top of it asking for my pin... you get the idea
these were separate instances as i could press cancel and go back to the previous one.

i got an old photo album filled with various bugs but stopped collecting since there were so many. now i only save the unique ones.

[opal lagoon]

I had that happen to me on my Android lock screen. except if I put in the correct one, it says I've used up all my attempts and it will factory reset. but if I did it wrong, it just says it's wrong

fortunately restarting the phone fixed it

anywho, on topic, I'm amazed a corrupt JSON was allowed to be saved lol

well, maybe not corrupt. malformed?

[lapis gorge]

saving it isnt an issue since its just a stream of bytes. though whatever is generating the JSON is stringifying without checking for an empty string.

[hollow harness]

That is on the save right? I'll see if I can find the spot later today

[lapis gorge]

sorry about the late reply. i saw this byte code while debugging and am trying to figure out what a relative jump of 0 means

00007FF8E8103AC4 F8                   clc  
00007FF8E8103AC5 7F 00                jg          __imp_NtDelayExecution+7h (07FF8E8103AC7h)  
00007FF8E8103AC7 00 50 9D             add         byte ptr [rax-63h],dl  

eventually i end up googling x86 F8 7F 00 and along comes our new overlord trying to be helpful.

[hollow harness]

The core issue is the crew data being wonky tho

[lapis gorge]

that part i havent investigated further than it being malformed. dont know how to inspect the value with what i got here

[hollow harness]

your just sitting there doing "nothing" when this occurs?

thinking if I leave your save from a bit ago running maybe it will do the crew thing but not sure

[lapis gorge]

nah, the crew thing turned out to be every save with that debug build

[hollow harness]

errr as in its just the fault of that or as in it shoudl happen if I load one and save?

[lapis gorge]

all i can say really is that the saves made with the debug build you sent turned out to fail at saving all times it saved

[hollow harness]

mkay

[lapis gorge]

btw i found the function causing the freeze.

SpaceIdle.exe!00007ff7be5e244a()
it never returns, but i have no clue what it is.

[opal lagoon]

it's not even an empty string. it's literally nothing

[lapis gorge]

        SpaceIdle.exe!00007ff7be5dd4b0
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
    SpaceIdle.exe!00007ff7bf9fb042
    SpaceIdle.exe!00007ff7bf9fbd1f
    SpaceIdle.exe!00007ff7bf9df3da
    SpaceIdle.exe!00007ff7bf9f6884
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cb02 1111
     SpaceIdle.exe!00007ff7bf9f67b2 111
     SpaceIdle.exe!00007ff7bfab7a13 11
     SpaceIdle.exe!00007ff7be5e244a 1
    SpaceIdle.exe!00007ff7be58cbc4
    SpaceIdle.exe!00007ff7beb66d13
    SpaceIdle.exe!00007ff7bea40b7b
    SpaceIdle.exe!00007ff7bfeb5c26
    SpaceIdle.exe!00007ff7beb811fa
    SpaceIdle.exe!00007ff7beb883a8
    SpaceIdle.exe!00007ff7be345602
    SpaceIdle.exe!_Z13widechar_mainiPPw() + 5390 bytes
    SpaceIdle.exe!_Z5_mainv() + 48 bytes
    SpaceIdle.exe!00007ff7be2e13c1
    SpaceIdle.exe!00007ff7be2e14d6
    kernel32.dll!BaseThreadInitThunk
    ntdll.dll!RtlUserThreadStart

this is one of the call stacks. i added the ones to label the repeating calls. the recursive depth varies a bit between 10-13 and there is a little gap on the second layer. i suspect that in the steam version this depth wasnt held back hence the stack overrun.
still no closer to exactly what it is.

[hollow harness]

I dont think its the crew thing, I tried to emulate it from your save and if I force the data to nulls etc its fine on saving still, unless its like... something weird and not even null etc

[lapis gorge]

the stringifier probably handles nulls. i was suspecting some odd datatype with a bad toString

[hollow harness]

that blank spot is just a float so

[lapis gorge]

you dont happen to use split_floats in the vicinity of that?

[hollow harness]

nope not used at all

[lapis gorge]

i have identified part 2 of the recursion. it is _VariantCall::FuncData::call‎

so i see the possibility that there is an infinite loop possible in the GDScript

now i got to sleep since i forgot to do that while hyperfocusing

[hollow harness]

Is the crash fast
Or does it hang for like 6 seconds?

[lapis gorge]

instant with the steam version. the others have just frozen.

[lapis gorge]

@hollow harness i have an idea to test but dont have the tools. what is the highest number to_json can handle? i tried testing in the browser but godot uses JS built in JSON.stringify
or maybe null, nan or inf?

[hollow harness]

quick test shows about e240

[lapis gorge]

did it yield an empty string?

[hollow harness]

no it crashes above that and no save file at all

[pearl rock]

(That might be what happens, if it opens the savefile for writing and then never saves to it due to crashing before it starts writing anything)