A complete hang during a gauntlet run that does not recover (video)

[marsh zodiac]

We were running a gauntlet run with 3 players, and game hanged hard randomly during the trash clear phase between bosses. This seems to happen pretty consistently for us (or me) but it is isolated to Remnant 2. Normal Remnant 2 farming activities we can run for up to 6 hours without any issues, but gauntlet seems to have some kind of issue especially with multiple players that causes a hang or crash or desync.

Luckily, I was running OBS capture in the background, so I was able to capture the situation this happened. I tried to collect all available information to assist bug hunting, but unfortunately there is very little - only the hanged process memory dump.

Said memory dump from crashed process:

https://drive.google.com/file/d/1Ayvtz1VXPdywW6IkYdy2unc3FwSL4JC7/view

video showing the hang, with Discord banter sound track to prove that there should not have been a complete network disconnect (in elvish aka Finnish) https://youtu.be/iIuMdMQH-DA

as mentioned, this kind of thing does not happen with any other Remnant multiplayer mode or any other multiplayer game with same people. it seems to be isolated to gauntlet, and it seems to happen when the boss counter goes beyond 10 - we ran a two-player 7 boss run without any issues before.

trigger seems to be having 3 players, doing a gauntlet and being in the latter part of the run (around step 10-13 in the gauntlet run). with fewer players or shorter runs, this does not seem to happen.

to me this suggests some kind of memory or resource leak or maybe a concurrency issue that is amplified by player count.

for us, the hit rate seems to be 5/6 for our 3 person runs

additional details:

I was running a VPN software but it did not disconnect since the discord chat was continuing as usual. this may, however, cause subtle issues with network traffic including packet loss. but packet loss should always be accounted for if you are using UDP

hopefully this report is useful

[static raptor]

I asked a similar question about a week ago, someone had mentioned the fact that a known memory leak exists since the 3rd DLC. Apparently server related and nothing can be done.

Result of Loadout swapping, trait swapping, and clearly leaving the game open for longer than an hour.

After 55 minutes I checked the memory use and it was at 77%. Unfortunately myself and two mates crash in hardcore gauntlet more than we'd like.

It also creates several issues with the game creating weird bugs.

For example: just last night, my duo Loadout archetype switched mid fight and died wave 4. You can't Loadout swap in combat.

And secondly on wave 15 died to a sentinel elite flying eye mob and his prime perk did not activate.

It was not previously activated.

Game is wild 😭

[steel jungle]

It's not a server issue because this game doesn't have servers.

It's a problem with Unreal Engine. The Memory leak has existed since launch, and they managed to get it somewhat under control, but excessive loadout swapping will still crash you.

There are also many other external things that just don't play nice with this version of Unreal 5 anymore, including certain drivers or GPUs and some generations of CPUs.

or never have played nice with this version of the engine in the first place

Some Modifiers on Aberrations can also mess your machine up, but thats much rarer and somewhat inconsistent

[marsh zodiac]

yeah, my gut feel based on these issues is that there may be multiple triggers for these issues and they may not exist on gameplay layer but deeper inside the engine. it sucks, because 3 player gauntlet is probably the most fun you can have with your pants on 😉

hopefully during the potential Remnant 3 development (which I sincerely hope is in the cards, even if Remnant 2 will no longer receive any work) these kinds of issues can be ironed out since this game does have legs and having multiple end-game game modes (boss rush, survival and maybe mixture of both) would extend the tail of the game considerably.

regarding aberrations, I think clone modifier might be something that might be a trigger for many of my personal issues. this is absolutely anecdotal, but I have a strong gut feel that when we end up fighting an aberration with clone modifier, my game starts acting up somewhat and after a while it will crash. there might be some other modifiers that trigger this, of course, but this is something that I seemed to notice

[steel jungle]

Honestly one of the biggest pitfalls this game had for functionality was that they developed on one of the early versions of UE5, where there were still a lot of growing pains. Orb of Undoing/Loadout swap is probably just one consequence of that. I also see that PC players routinely have more issues than Console which likely stems from hardware/driver conflicts

You can try to clear your shader cache, maybe this will help mitigate it a bit, but at least the Orb/Loadout memory leak I'm afraid there's nothing you can do

[marsh zodiac]

agreed. and upgrading from pre-release chain to release chain is not painless, I unfortunately have experience for that from my day job. I also have had to manage the drivers somewhat, since at least Fortnite had pretty common crash issues with some releases of nVidia drivers

[steel jungle]

Yeah word
Another solution you can try is this:
#remnant2-discussion message

Both the Shader Cache clear and deleting that library executable have so far proven to fix crashing issues (even if they might have had different symptoms)

[marsh zodiac]

thanks! I'll try that out

[marsh zodiac]

oh yeah, and one note for any future games in case someone happens to read the feedback: this issue is complicated by the designed restriction of not being able to rejoin the session in case the game goes haywire. I absolutely understand the design goal for preventing the disconnect/reconnect cheesing but allowing some kind of recovery mechanism might be a good idea. gauntlet is considerable time investment and in this case the payoff is completely denied if something odd happens during the run - this could be softened somewhat without ruining the design goal.

maybe limiting the reconnects might be a good compromise, or allowing rejoin within some kind of grace period if you have exactly same spec running. dunno, there are some options available in that design space without hurting the "this is meant to be really hard and you need to bring your A game" design ideal.

this is something I encounter somewhat commonly with my day job (software dev working in various industries), and nowadays I try to always leave some kind of escape hatch to software I do for unexpected situations - they tend to always happen and you want to have some graceful way to handle them without pulling back your sleeves and diving directly to database or said machine/container/whatever.

I always try to find out ways for a person to do a self-recovery in case something goes bonk, but it is obviously harder to do if you have hard restrictions, either stemming from the design (you are not owner of the data and restricted by the upstream), external factors (cough, security, cough) or, in this case, gameplay design.

[marsh zodiac]

another sample for us: started with 3 players, one had to drop after first (or second) boss due to family stuff. we played duo till 15/19 at which point I had instant disconnect from host (my friend was hosting). my game was running fine, but during the transition period between bosses I just got the "Network disconnect" error. I did not even have the "pickup stutter"