#Server Crashing When certain clients connect?

1 messages · Page 1 of 1 (latest)

round crypt
#
1: '' (UR5DataKeeperForServer_Account::OnCoopAccountBLConnected):                                       Waits  27 ticks. Added at 12503. Character nullptr, PlayerState nullptr, Controller nullptr, Account R5DataKeeperForServer_Account_2147150989, LastIsNotReadyReport !DefaultPlayerView. (D:\Source\Build\work\gameRepoCheckoutDir\Source\R5DataKeepers\Private\R5DataKeeperForServer_Account.cpp:361)
 [D:\Source\Build\work\gameRepoCheckoutDir\Source\R5DataKeepers\Private\R5DkAccountTracker.cpp:558]
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: === Crash Stack Trace: ===
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: 
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000014ba5bc98 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000014ba59f6e WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000014bf890ef WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000007b627e49 kernel32.dll!BaseThreadInitThunk() [Z:\home\steam\server-files\R5\Binaries\Win64\dlls\kernel32\thread.c:61]
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000017005dca8 ntdll.dll!RtlUserThreadStart() [Z:\home\steam\server-files\R5\Binaries\Win64\dlls\ntdll\thread.c:285]
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x0000000000000000 UnknownFunction []
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x000000014bf89040 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.15-23.35.18:432][543]LogOutputDevice: Error: [Callstack] 0x00000000036eea70 UnknownFunction []
[2026.04.15-23.35.18:433][543]LogOutputDevice: Error: [Callstack] 0x0000000000000000 UnknownFunction []
[2026.04.15-23.35.18:433][543]LogOutputDevice: Error: 

Server ran fine for 8 of us last night playing for 5-6 hours, now it randomly crashes and reboots when certain clients/players connect. All players can get into single player games without issue. I really wish it wouldn't take the entire server down hah.

Any insights?

#

yeah this seems more unstable than ever.. looks like we won't be able to play the game as we all try to connect 1 by 1, someone will always bring the entire server down (different person every time) and we have to restart the entire process (docker container reboots)

round crypt
#

Is there anything I can provide to better help identify the situation causing the crash? The fact that it brings the server down for everyone is brutal 🙂

round crypt
#
[2026.04.16-01.52.15:404][345]R5LogCheck: Warning: [-1:205345]
!!! R5Ensure happens !!!
        Condition: 'false'
        Message:   Unexpected BL disconnect. AccountId 36586DC646DB5171A976A5956F24F36E. BLPlayerSessionId '79bfb251939d4fe3967f548a5625fb4a'
        Where:     UR5DataKeeperForServer::OnAccountBLDisconnected::<lambda_10>::operator () [D:\Source\Build\work\gameRepoCheckoutDir\Source\R5DataKeepers\Private\R5DataKeeperForServer.cpp:267]
[2026.04.16-01.52.15:404][345]LogOutputDevice: Warning: === FR5CheckDetails::PrintCallstackToLog ===
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: 
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: [Callstack] 0x0000000146d19311 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: [Callstack] 0x0000000144e7baec WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: [Callstack] 0x0000000144ef9166 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: [Callstack] 0x000000014445f928 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:405][345]LogOutputDevice: Warning: [Callstack] 0x000000014415fae5 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:406][345]LogOutputDevice: Warning: [Callstack] 0x000000014127ee2b WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:406][345]LogOutputDevice: Warning: [Callstack] 0x000000014128a631 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:406][345]LogOutputDevice: Warning: [Callstack] 0x0000000141284adf WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:406][345]LogOutputDevice: Warning: [Callstack] 0x0000000141284e51 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:406][345]LogOutputDevice: Warning: [Callstack] 0x0000000143e97bc8 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x0000000143e9ab25 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x0000000143a105c6 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x000000014385499f WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x0000000144101341 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x00000001441048bc WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:407][345]LogOutputDevice: Warning: [Callstack] 0x000000014410493a WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:408][345]LogOutputDevice: Warning: [Callstack] 0x00000001441053ad WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:408][345]LogOutputDevice: Warning: [Callstack] 0x000000014410a824 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:408][345]LogOutputDevice: Warning: [Callstack] 0x000000014bf2734e WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.16-01.52.15:408][345]LogOutputDevice: Warning: [Callstack] 0x00006fffffa21469 kernel32.dll!UnknownFunction []
[2026.04.16-01.52.15:408][345]LogOutputDevice: Warning: [Callstack] 0x00006fffffc00d2b ntdll.dll!UnknownFunction []
round crypt
#

we ended up having to log in every person individually to catch up to the same quest line (repair ship, recruit helpers) and the server stopped crashing.. it seems to be okay now, for now

sour canyon
#

Save 'Z:/home/steam/server-files/R5/Saved/SaveProfiles/Default/RocksDB/0.10.0/Worlds/8B6F961CEF384787D78212FE34593E3E/WorldDescription.json' is broken. WorldDescriptionFile 'WorldDescription.json' not found. [D:\Source\Build\work\gameRepoCheckoutDir\Source\BackendProxy\R5CoopProxy\Private\Data\R5CoopIslandCollection.cpp:43

#

We have the same issue when multiple users start connect. I had all the same nt.dll errors but tucked away I saw this too.

Looks like "R5CoopIslandCollection.cpp" line 43 is having issues maybe syncing all the co-op stuff and it borks out. Hope the dev teams sees this thread and specific error.

round thunder
#

@round crypt Can you share crashdumps?

#

From your server?

#

It should be either Crashes folder near logs or in \R5\ .sentry-native\reports

#

Also and full server log file from the time when crash occured

round crypt
#

We are in different timezones for sure, got this message at 4:59 am! haha.. one moment ill grab them

#

Do the logs contain sensitive information or PII? Should I refrain from publicly posting them here and send them directly?

round thunder
#

Yeah, you may add in discord and send via DMs if that's preferrable

round crypt
#

I have to add you as a friend to send them, but I have them archived and if you grep -i "error" * the logs provided you'll see a few error callstacks. The logs have rotated to the point that it's just barely still available.. good timing

#

Can't request either (naturally), but if you add me as a friend I can quickly accept and transfer them now

round thunder
#

Requested

round crypt
#

sent

round thunder
#

Yep, got 'em

round crypt
#

@sour canyon I don't get this same error, at least there are no similar references in my Logs to the world description being broken or not found.

#
~/docker/windrose/server-files/R5/Saved/Logs$ grep -i "broken" *

R5-backup-2026.04.15-23.35.18.log:[2026.04.15-23.29.41:661][341]R5LogNetCm: Warning:          [002341] ...tCmClient_GsStream_P2pClient::SetBrokenState    [0x7bbda51f4d90-   396] [a4c08ecdd19948a08407250fa4fdd4cc] GsStream P2pClient is broken. ErrorMessage 'Error on getting local ICE candidates for BL. Error '''. State WaitingBLLocalIceCandidates [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream_P2pClient.cpp:148]
R5-backup-2026.04.15-23.35.18.log:[2026.04.15-23.29.41:661][341]R5LogNetCm: Warning:          [002341] ...ient_GsStream_ClientReactor::SessionIsBroken    [Gs[0]-B9689AD649C840EFBFB920A203644422] Send session is broken. BLPlayerSessionId a4c08ecdd19948a08407250fa4fdd4cc. ErrorMessage 'Error on getting local ICE candidates for BL. Error ''' [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream_ClientReactor.cpp:155]
R5-backup-2026.04.15-23.35.18.log:[2026.04.15-23.29.41:689][341]R5LogNetCm:                   [002341] ...lient_GsStream_P2pClient::PlayerDisconnected    [0x7bbda51f4d90-   260] [a4c08ecdd19948a08407250fa4fdd4cc] PlayerDisconnected. State Broken. DisconnectReason Error on getting local ICE candidates for BL. Error ''. FarewellReason . [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream_P2pClient.cpp:83]
R5-backup-2026.04.15-23.35.18.log:[2026.04.15-23.29.41:689][341]R5LogNetCm: Verbose:          [002341] R5NetCmClient_GsStream_P2pClient::RemovePlayer     [0x7bbda51f4d90-   260] [a4c08ecdd19948a08407250fa4fdd4cc] Account already disconnected. State Broken [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream_P2pClient.cpp:101]
round thunder
#

We've tried to pinpoint the issue, but we couldn't from logs. If by any chance you will be able to repeat this crash on in windows environment - that would be great. Crash dumps from containers doesn't seem to reach our sentry :(

round crypt
#

depending on how the crash dumps are sent, that's not surprising. If it's just a built in http client sending to a remote endpoint it should work fine. If it's transferred as UDP traffic via a port mapping, the container will probably not have permission to send that through the host system without explicitly mapping that port. Nothing with this game (that I can find) uses configurable ports however.

It's a relatively large ask to try to repeat this in a windows environment. Theoretically, while I do have a self scripted game backup manager that keeps rolling save states, it does not have client side player save states and truely recreating that scenario is largely impossible even if I move the server to a windows box.

While I run the server on linux, and I play on linux (with one other), and 6 others in windows, we all triggered the error and crash at random.

We do believe it may be related to player state mismatches though.

For context, I had not repaired the ship but had free'd the last sailor to trigger completion for other people.

That was the last thing we did after the prior night of 6 hours stable.

The next day, chain logins were crashing the server with 1 new player (baseline 0 character, and 2 other characters).

We all made new characters with nothing done, and connected to the server (successfully).

We then began reconnecting leaving and reconnecting on our mains 1 by 1, from the perception of "least progressed" to "most progressed".

This did let us in over the course of staggered logins with loading over ~15 minutes.

We then caught and sync'd everyone on the same quests and were able to play.

We have not tried to play yet today since we're all at work.

round thunder
#

Yeah, I get that it's tall ask :( I've seen few reports of save issues, but they are not reliably reproduced so we're running on fumes trying to understand what's going on :(

round crypt
#

It's not a crazy ask, it's just not possible with the inability to reproduce client save states to their exact progression at the time of the crash since they aren't stored server side per server instance heh.

And given we were able to all connect fine with new characters, we speculate that it may be character/world state sync state related.

if/when it happens again, I can move the server snapshot to a windows machine and have everyone try to connect again.. my suspicion is that it will produce the same result based on other issues I've seen with dedicated servers on windows as well through this discord and internet in general.. but It's worth a try.

For what it's worth, I have zero interest in running a windows machine for a game server though as a permanent/long term solution 😅

round thunder
#

For what it's worth, I have zero interest in running a windows machine for a game server though as a permanent/long term solution 😅

Understandable, very valid take

#

We'll be improving our server along with EA as well as client I'm sure

sour canyon
round thunder
#

Is your server linux container hosted as well?

round crypt
sour canyon
#

I can also open a new thread but I think Gaudon and I are having similar issues. Played for 6+ hours but when more then 3-4 get on it crashes and I feel in my gut it has to do with co-op sync for quest. I turn this off today and see if it does any better.

If you would like a new thread I can do that too.

round crypt
# sour canyon I can also open a new thread but I think Gaudon and I are having similar issues....

we started with 8 people right out of the gate, but towards the end of a 6 hour sessions people had diverged a bit doing different things with clusters doing things at their own pace. The next day when we all logged in is when the instability hit.. any random combination of 2-3 of us at very different early stages of progression (up to and including when you get the ketch) was causing disconnects on join and crashing the entire server.

round thunder
#

If you can add rough timestamps on top so it's easier to skim through server logs (i.e. player X joined at 17:30 and we all crashed) - even better

round crypt
#

what happens if you build in the place the big boat that needs to be repaired spawns on when a player who hasn't done the quest connects?

round thunder
#

The big boat likely appears

#

Ignoring your building pieces

#

Or it doesn't

#

Both cases are equally not very fun

round crypt
#

Just tested it, it was the first.. and it did not crash the server.. which we speculated might be related to the issue but server did not crash in this case.

round crypt
#

our server just crashed without anyone connecting or disconnecting.

#
[2026.04.18-13.56.41:497][703]R5LogBLBiz:                   [782703] R5BLActor_Drop_EraseRule::Do_Impl                  Erased Actor_Drop F4C4FF98353341C293B2EB4789B851C3 from island D22AC937054148E4768F8E8D4F9FFC68     [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessRules\Source\R5BusinessRules\Cpp\R>
[2026.04.18-13.56.42:999][748]R5LogNetCm:                   [782748] R5NetGrpcClientReactorV2<>::OnDone                 [Gs[7]-30BF055DA67F44D4D38B3A719464892F] Disconnect with temporal error. error_code UNAVAILABLE. error_message 'Received http2 header with status: 502'. error_details ''. Self use count 3 [D:\So>
[2026.04.18-13.56.42:999][748]R5LogNetCm:                   [782748] R5NetCmClient_GsStream::OnDisconnect               [Gs-30BF055DA67F44D4D38B3A719464892F] Cm disconnected. Try to reconnect. Status 14 ('Received http2 header with status: 502') [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCor>
[2026.04.18-13.56.43:000][748]R5LogNetCm: Verbose:          [782748] R5NetCmClient_GsStream::ResetReactor               [Gs-30BF055DA67F44D4D38B3A719464892F] Resetting reactor                                             [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsSt>
[2026.04.18-13.56.43:000][748]R5LogNetCm: Verbose:          [782748] R5NetCmClient_GsStream::SetState                   [Gs-30BF055DA67F44D4D38B3A719464892F] GsStream. Change state. Registration => Initial               [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsSt>
[2026.04.18-13.56.43:100][751]R5LogBLAsync:                 [782751] TR5BLPromiseBase<>::~TR5BLPromiseBase              Incompleted TR5BLPromise was destroyed! DebugId 1DA0FC57229E42BA398C34D9420FDCCB                    [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5BLCommon\Cpp\R5>
[2026.04.18-13.56.43:145][752]R5LogNetCm:                   [782751] R5NetCmClient_GsStream::OnReconnected              [Gs-30BF055DA67F44D4D38B3A719464892F] Reconnect request has been finished. IsOk false. BLSessionId . ErrorMessage '' [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\>
[2026.04.18-13.56.43:471][762]LogWindows: Error: appError called: Fatal error: [File:D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream.cpp] [Line: 203]
           [782762] R5NetCmClient_GsStream::SetBrokenState             [Gs-30BF055DA67F44D4D38B3A719464892F] GsStream is broken. ErrorMessage 'Cannot reconnect to Cm'. PrevState Initial [D:\Source\Build\work\gameRepoCheckoutDir\Plugins\R5BusinessLogicCore\Source\R5Net\Cpp\Cm\GsStream\R5NetCmClient_GsStream.cpp:20>



[2026.04.18-13.56.43:472][762]LogWindows: Windows GetLastError: Success. (0)
#
[2026.04.18-13.56.43:472][762]LogWindows: Windows GetLastError: Success. (0)
[2026.04.18-13.56.43:539][764]LogOutputDevice: Error: === Crash Stack Trace: ===
[2026.04.18-13.56.43:618][766]LogOutputDevice: Error:
[2026.04.18-13.56.43:618][766]LogOutputDevice: Error: [Callstack] 0x0000000141474fe6 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:618][766]LogOutputDevice: Error: [Callstack] 0x000000014413b31d WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:619][766]LogOutputDevice: Error: [Callstack] 0x00000001414020b3 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:619][766]LogOutputDevice: Error: [Callstack] 0x000000014137095a WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:619][766]LogOutputDevice: Error: [Callstack] 0x0000000144eb2283 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:619][766]LogOutputDevice: Error: [Callstack] 0x0000000144e97584 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:619][766]LogOutputDevice: Error: [Callstack] 0x0000000144efdf97 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:620][766]LogOutputDevice: Error: [Callstack] 0x00000001444a841c WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:620][766]LogOutputDevice: Error: [Callstack] 0x0000000144361a42 WindroseServer-Win64-Shipping.exe!UnknownFunction []
[2026.04.18-13.56.43:620][766]LogOutputDevice: Error: [Callstack] 0x00006ffffea22dea ucrtbase.dll!_beginthreadex_trampoline() [Z:\usr\src\packages\BUILD\dlls\msvcrt\thread.c:201]
[2026.04.18-13.56.43:620][766]LogOutputDevice: Error: [Callstack] 0x00006fffffa21469 kernel32.dll!UnknownFunction []
[2026.04.18-13.56.43:621][766]LogOutputDevice: Error: [Callstack] 0x00006fffffc00d2b ntdll.dll!UnknownFunction []
[2026.04.18-13.56.43:621][766]LogOutputDevice: Error:
hollow yoke
#

Our server is down too. There are only two of us and no one else can connect. It just happened out of nowhere.

round crypt
#

yeah it looks like it's trying to phone home to a remote service or central service of sorts and it's failing to.. seems to be stuck in a boot loop because the central service might be down

hollow yoke
#

Mine restarted immediately. I started it manually. But I'm not with Nitrado. I'm self-hosted on a root server at Hetzner.

round crypt
#

I'm self hosted on my own linux box, 64gb ram, ryzen 5950x. has been pretty flawless minus the occasional crashes in this thread. Auto save backup manager I wrote for all 6 games it runs, auto reboot script every night.

#

I'm guessing there was an issue with the central server that keeps a listing of all the servers/invite codes

round thunder
#

Yes, we have it down for a few mins, investigating

hollow yoke
round thunder
#

It was not planned or expected :(

#

Sorry for that

hollow yoke