Game/server frequent crashes on map change event in 5.26

[blissful thicket]

Whether the server is using Metamod or not, it crashes more frequently than in 5.25. You can reproduce this issue by writing map hl_c09 in the server.cfg to create a map-changing loop; the server will shortly crash. Only tested with the server running Survival Mode. Haven't tested without, which maybe is relevant for troubleshooting.
Tested in both Windows and Linux.

[oak sapphire]

I've noticed the same as well (non-survival in my case). Server crashes 3-4 times a day on simple map changes which previously were fine.

[rancid cloud]

@timid mica 👆, both case, on survival/non-survival maps

[vocal goblet]

Do all of these maps crashing have an Osprey NPC?

[elder sequoia]

Hmm, c09 doesn't have an osprey, does it?

[vocal goblet]

Oh right Rick said it's for Windows too. It's not an issue we know about then with Osprey in Linux on map change when sound replacements are used.

How many map changes do I need to cause a crash?

On my 9th iteration of map hl_c09 so far. No crash yet.

Just tried with dedicated too. A dozen map hl_c09 commands issued, no crash.

[elder sequoia]

I'm guessing a bit more if it's only about 4 times a day Merryila? If their server is active that's at least like 30 between crashes I guess?
I'll try it out myself later to see if I can replicate it

What plugins is everyone running who is crashing?

Might help narrow it down if it happens with plugins and not without? @blissful thicket

[vocal goblet]

My instance is only running 4 standard ones: PlayerManagement, PlayerDecalTracking, EntityManager, and Teleport.

[elder sequoia]

I got a windows server with a few. I'll check to see if there are crash logs. It's idle but maps cycle slowly

[vocal goblet]

So far all I've encountered is an assertion failure:

ASSERT FAILED:
 m_vecPosition1 != m_vecPosition2

That's a map issue though, not a crash, because the start and end positions of 7 doors appear to be equal.

I was going to ask if this only applies to servers with people in them, but I just tried with a listen server, which I was definitely in while the map changed a dozen times.

[blissful thicket]

After some testing, I think I’ve discovered a few more problems that I didn't initially anticipate. I'm not sure if they’re connected to the frequent map change crashes we're experiencing.

1. No map scripts used. Only AFBase plugin in default_plugins.txt and map hl_c09 entered in server.cfg to make rapid map changes.
   Eventually it leads to:
   FATAL ERROR (shutting down): RegUserMsg: No room for user message 'ReqState'

With no plugins, it doesn't crash.

2. Started using a custom plugin to execute static.cfg on MapStart() event.
   It eventually (after somewhat 50+ map changes) starts throwing CASContextPool errors everytime the map loads (no crash):

ERROR: CASContextPool::Think: 4 contexts total, but 3 contexts in list!
ERROR: CASContextPool::Think: 1 contexts still in use!
ERROR: Active module was not set to null!
Server stopped logging Angelscript at: 2024-12-21 16:43:21
Angelscript log opened
Server started logging Angelscript on map 'hl_c09' at: 2024-12-21 16:43:21
ERROR: CASContextPool::Think: 4 contexts total, but 3 contexts in list!
ERROR: CASContextPool::Think: 1 contexts still in use!
ERROR: Active module was not set to null!
Server stopped logging Angelscript at: 2024-12-21 16:43:25
Angelscript log opened
Server started logging Angelscript on map 'hl_c09' at: 2024-12-21 16:43:25
ERROR: CASContextPool::Think: 4 contexts total, but 3 contexts in list!
ERROR: CASContextPool::Think: 1 contexts still in use!
ERROR: Active module was not set to null!
Server stopped logging Angelscript at: 2024-12-21 16:43:29
Angelscript log opened
Server started logging Angelscript on map 'hl_c09' at: 2024-12-21 16:43:29
ERROR: CASContextPool::Think: 4 contexts total, but 3 contexts in list!
ERROR: CASContextPool::Think: 1 contexts still in use!```

I'll do some more testing later.

[vocal goblet]

Does that only occur if you put a map line in your server config?
(That would by no means a reasonable use case, to have a server cycling around a single map instantly when it starts, infinitely.)

[blissful thicket]

No, it’s only for testing purposes in this case.

The servers didn’t crash as frequently in 5.25 as they do now in 5.26 during a MapChange() event. I first noticed this during the 5.26 RC when the server was empty most of the time. However, now that there are players and map changes occur more frequently, it’s much more noticeable.

[vocal goblet]

Do you have any crash dumps from when it happens with regular use?

[blissful thicket]

Nope. But I can activate it. Is it -debug to the command line?

[vocal goblet]

I would suspect that the map line may be causing a race condition. The same test without this but mp_timelimit 0.05 (3 seconds) would be a more reasonable test.
mp_timelimit mp_timelimit_empty -- for empty servers so that map defined time limits don't interfere.

While trying this myself I found an assertion problem when BlackMesaEPF is loaded because of an õ in the map's entity data. Didn't crash though.

Also when that map ended my console was totally flooded with

WARNING:  Master entity 'securitydr_open' not found.

...that entity name has no mention in its map data. (Map blunder.)

Same map by any chance? 😛

[blissful thicket]

It doesn't matter what map. It always crashes after ~20-40-ish maps changes and always during a MapChange() event.

[vocal goblet]

Is there a script involved listening for that event making use of the map string parameter?

[blissful thicket]

I use the MapChange() hook in several plugins, but I do not use that new parameter.

In SC 5.25, the server tended to crash during a MapChange() event after approximately ~140-180 map changes.

[vocal goblet]

Debugging this is somewhat slow as I get a waiting popup for this on every map change:

It's not even our module, we don't have symbols for it, and there's no error/exception written in the output. Currently not seeing a "I don't give a shit" option to skip it. (Just manually continuing each time is not causing any problem.)

Otherwise going through the whole standard map cycle, up to sc_persia currently, no crash.

Oooo found something with th_escape

CString.cpp : Expression: uiIndex < m_uiLength
Both values are 0.
Looks like a map script is trying to split an empty string.

This is just an assertion though, it won't cause a crash. Continued to toadsnatch fine.

Looped around to "-sp_campaign_portal"

Currently BlackMesaEPF is the only map that has caused a problem

62nd map in, no crash yet

I've gone around the whole official map cycle twice now, plus the server start map. No crash after 79 maps, running the 80th.
To look into this further we'll either need a crash dump or which map was running to cause it. Seems like it's going to be a map or plug-in problem.

[oak sapphire]

I've got a couple of Linux crash dumps on this, who is best to look at them? Is it still R4?

[vocal goblet]

Until I discover how to process them, yes. 😛

Oh apparently it's possible to debug Linux core dumps in Visual Studio. I could try that.

[oak sapphire]

I need to see if I can upload them first. I'm not at home at the moment and the internet here is trash 😦

[warped surge]

It's exactly how I do nowadays, no way I'm going back to manually playing with gdb commands 😂

that and WSL helps A LOT

[vocal goblet]

Cool. I've already got a Linux VM for compiling.

[vocal goblet]

You can probably upload it somewhere directly from your server? Or keep them there but make a password protected web vhost.

[oak sapphire]

Uploaded it directly from the server as you suggested, will DM you the link

[fervent vigil]

here statistics today, as you can see if server get some ppl then chances to crash is increased on map load or restart, if its empty or less palyers on 03:00 then less crashes
UPD: servers is running on linux

[austere adder]

Got any plugins running?

[fervent vigil]

idk, bcs im not server owner

[fervent vigil]

Long series map literally unplayable

[warped surge]

any crash dumps from these crashes?

[oak sapphire]

For anyone running on svends_i386 and suffering from MapChange crashes, I resolved this on S/v/en by reverting the libcurl.so.4 library to the one provided with 5.25

Not a dev so obviously can't guarantee this will fix things for other servers, but if you see lots of calls to libcurl in your debug.log at the point of crashing I'd say this is worth giving a go

[vocal goblet]

We've got a newer build of this library available, which was built in a Steam Runtime 3.0 (Sniper) schroot instead of a Debian 9 (Stretch) one.
This one built much more successfully on the first attempt and includes c-ares again. (Build 01 this day was produced but did not include c-ares.)
You're all welcome to try it. I had no crashes while connected to a Linux server running this over a few dozen map changes.
Simply a drop in replacement at path "/svencoop/dlls/libcurl.so.4".

(The version from 5.25 requires an OpenSSL 1.1 library available, which is becoming increasingly rare. As of 5.26 we've included OpenSSL 3.2 statically linked in.)

[blissful thicket]

After several days of testing, there was no improvement using libcurl.so.4 provided with 5.25.
Just to add more info, the client game process are crashing pretty frequent as well. Not only the server itself.

The server crash can occur anywhere between 2 to 40 map changes, most commonly between 6 and 22 changes with players.

Interestingly, I’ve observed three distinct scenarios:

1. The game server crashes randomly during a level change or shortly after a few players have joined onto the new map.
2. The game server crashes early after a level change has been made and while the clients are downloading the soundcache file. The download gets stuck for the client, and if they click "Cancel," their game process crashes. I'm unsure if this affects one, a few, or all clients.
3. The game server has been empty for a few hours and makes a level change. It randomly decide to crash.

[warped surge]

The click cancel crash during load is caused by the new camera feature

But for the server doesn't make sense for this to crash, what happens there is that a thread is spawned for donor queries and then libcurl does all the rest.

Unless if payload data is doing this too

But that would be a bug in libcurl as well and has to be fixed upstream

Bur if you used libcurl from 5.25 with same symptoms then the issue is somewhere else

Perhaps the payload/query data being sent to libcurl

It crashes very deeply into openssl

Note: I'm not denying your findings, this is just what I have also experienced later and based on coredumps I've got sent.

[blissful thicket]

As soon the next map started and 20+ players are connecting and spawning (about 5 successfully spawned before the server decided to crash), it throws malloc(): unaligned tcache chunk detected
I don't recall seeing this from earlier versions.

[unique mica]

I'm having this issue too, even with only around 5 players connected. It's not very often I play on my server but I definitely didn't have this issue on earlier versions as the OP has mentioned. It doesn't seem to matter which map is used, it just has that error randomly on map changes. I hope the issue can be found and fixed on the next update.

[winged sundial]

if they click "Cancel," their game process crashes.
Much more interestingly, if they press ESC instead of pressing Cancel they won't crash. But. There's a catch. If they try to open up the console, the game will crash (the crash happens in client.dll 👀)