B42.13.2 MP - Dedicated Server Randomly Becomes Unjoinable Until Restarted

[steel gyro]

We found some mod incompatibilities with 42.13.2 and removed them (CleanUI cripples servers that have more than 12-18 players) and wiped for the new update and our server has been running much better since.

However, within the last 24 hours we began having an error that can occur multiple times per hour or once every couple hours, and it keeps making our server unresponsive. Every time the server becomes unjoinable, players are unable to move or rejoin like a full desync, and we get the same shouldCollideWithCharacters errors spamming in the logs:

...
LOG  : General      f:13801, t:1769462358069, st:1,786,328> java.lang.StackOverflowError.
LOG  : General      f:13801, t:1769462358069, st:1,786,329>     at zombie.vehicles.BaseVehicle.shouldCollideWithCharacters(BaseVehicle.java:3150).
...

The only way to fix things after this occurs seems to be a hard kill of the server and a restart which results in some data loss based on the last time the server was saved (and how players exited the game).

The dedicated server host has 128GB RAM, and we allocate 111GB to the JAVA heap max.

Any ideas what might be causing this? We did have a hacker join last night, but we banned them within 2-3 minutes and performed a rollback. Attaching four debug logs from when the server entered this state for reference.

Thank you for your time and consideration! Any help would be greatly appreciated!

[steel gyro]

It happened another two times after posting this but I haven't seen it again since. Not sure if its just attacks caused by hackers who are jealous of our success, or what.

Had another confirmed hacker join around an hour and a half ago and spawn a bunch of items on 40+ players including katanas and 99 pipebombs. Really unfortunate how bad the anticheat is and how little we can do to defend ourselves

[errant dagger]

I replied to your other post as well, we cannot really help with issues on modded servers, using mods means you are on your own if you run into any issues, there are several warnings for that when enabling mods

[steel gyro]

Hey there Beard, appreciate your responses! This hasn't happened again for around 9 hours now. Seems like it may be related to hackers attacking, considering it has not happened again since. But its hard to say for certain.

Either way, even if we were to disable our mods, it doesn't fix the hackers coming in and spawning things and crashing the server and server economy. We are constantly one of the most stable modded servers with high pop, so we are a prime target for attackers to disrupt players.

We recently spent over 30 hours with 15-20+ helpers to test for desync and high load and we identified and removed several problematic mods in the last week (CleanUI for example was causing crippling desync once above 12-16 players).

We had already been using the majority of our mods throughout 42.13.1 for almost a month with no issues at all.

The hackers are a more important issue for us right now. Seems we dont really have much we can do there however due to no working lua checksum check or anticheat?

[steel gyro]

The last two nights hackers have attacked around the same time (11PM PST). The item logs make it pretty clear its a hacker because nobody can legitimately spawn katanas, or 99 pipebombs on 35 people. And yet all we can really do is ban them asap and offer a rollback where players lose progress.

Its far too easy for people to grief dozens of players

[errant dagger]

Lua checksum works fine, people have just been disabling it for odd reasons trying to get incompatible mods to work

So thats one security issue you have, I would never recommend disabling the checksum

Anticheats are disabled in the Unstable beta due to them not being compatible, the current anticheat is a lot of data being pushed server side

Cheaters can still work around it, so we always recommend having a whitelisted server, I am pretty sure all our current news and messages about B42 advise using that, and heavily recommend against using mods

[steel gyro]

Lua checksum works fine, people have just been disabling it for odd reasons trying to get incompatible mods to work
Literally never worked for us. Even tried with different combinations of only 1-2 mods and it was still not working. Sometimes you'd be able to join once, then after it bugged it you were never able to join again without the verification saying your files do not match. (Would still happen even if you deleted all workshop files locally and tried to redownload on server login)

As one of the people who have informing people from day 1 of B42 that disabling lua checksums is dangerous, I completely understand. But the game runs terribly at high pop without the patch mods. Removing the mods is not really an option for us.

Whitelist only works with steamID's right now, correct? No longer works on usernames like it did in B41? We can consider swapping to a whitelist but this does not fix the hackers

The mods are part of what makes our server run so well so having to remove them feels like shooting ourselves in both feet

[errant dagger]

See thats the problem, you misdiagnosed an issue with mods to be an issue with the checksum

Disabling the checksum isnt a fix, and it there isnt a problem with the checksum

There is a problem with mods, and disabling the checksum ignores the issue, thats all

The whitelist should work with Usernames as well, though there is a problem with the latest implementation of it, so Steam IDs are required at the moment

[steel gyro]

Apologies if i wasn't clear, but sounds like you understand my inital issue with the lua checksum is that it does not work with the current implementation mods.

Sounds like were SOL until we get some more stability. Apologies for wasting your time, greatly appreciate everything you do. Thank you Beard! Hope you have a great day

[errant dagger]

The main issue with it is that Linux currently loads mods incorrectly, the checksum catches it and basically doesnt allow anyone to join due to the mod problem

Windows works fine, though most servers are Linux based as far as I am aware

[steel gyro]

Is that related to the case sensitivity issue that happens with some folder names?

We run Linux as well so you are correct. That is good to know though

I couldn't imagine using windows to host a game server. Sounds like blasphemy. Why waste those precious resources on a GUI

[errant dagger]

Yes basically that

[steel gyro]

Greatly appreciate you Beard. Keep being awesome

[errant dagger]

If you routed it through Wine or something then you could get the checksum to run, possibly

[steel gyro]

Don't love the sound of that but will consider

[neat pivot]

how can we add them? from the command

or we have to manually edit the DB and restart to add whitelists in MP 42.13.2?

[errant dagger]

The command is /addsteamID STEAMID

You can manually edit the database as well

[neat pivot]

i tried doing that but it didn't work

addsteamid works

but it resets the admin everytime

like, in user list U see admin, BUT you don't see the admin icon

[errant dagger]

And what if you add the id and then /grantadmin after that?

Or remove the admin and grant it again

[neat pivot]

i didn't even know about thisd command

[neat pivot]

/grantadmin how?

like

/addsteamid ID /grantadmin or what

[errant dagger]

That command has to be used in game

If you want to use comands in the actual server console, you have to remove the slash

So its just "grantadmin username"

[neat pivot]

Of course i tried it without / from rcon

It doesn’t exist

[neat pivot]

It doesn't solve the issue

everytime you log back in you're not admin, i mean you're "admin" but you don't see the admin tool

[errant dagger]

Are you sure you wrote the add steam id command first?

[neat pivot]

Yes

[errant dagger]

In that case, can you privately share the database file of the server?