perf_prof_branch

[feral harness]

did you use alt syntax of event handler?

[heavy vortex]

No, just the plain "varname" addPublicEventHandler { code } one.

Not bothered if it doesn't work in preinit. Only wanted it for testing.

[whole cloud]

🥔

[plucky lichen]

perf v3 released?

[azure jewel]

same question

[ashen helm]

https://tenor.com/view/leş-gif-20880152

[whole cloud]

2.18.152272 new PROFILING branch with PERFORMANCE binaries, v3, server and client, windows 64-bit, linux server 64-bit
- Tweaked: Reduced server lag when sending a large JIP queue
- Tweaked: Reduced JIP queue being filled up due to disappearing magazine bug
- Tweaked: Reduced ability to duplicate items due to the server-side disappearing magazine hotfix

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[empty goblet]

server with KOTH stresstest and Warlords Redux on 2.18.152272

[deft oak]

This update fixed our dupe issue with containers but it added the issue to vehicle inventories

[empty goblet]

well performance wise those fixes did indeed help for now playable ~100(+-20) fps at 80 players for Redux

[indigo anvil]

sent a crash in DM.

[knotty wraith]

For the first time, the HC did not automatically connect to the server, it hung as if in the lobby - I had to restart it manually, perhaps an accident

[deft oak]

I may have observed this as well. I can confirm if it happens this restart as well. We have 3 running and 1 failed to initialize.

[deft oak]

Didn't happen on the last restart so may have been part of a patch I pushed earlier.

[shell lion]

Same problem hear HC has issues after last Perf.
HC connects but the initialization doesn't finish, it seems to be that he can't find the virtuell slot or something like this.
On my last 3-4 try's he doesn't crash, he only hangs in the initialization.

[empty goblet]

the HC is same version as server or older?

[knotty wraith]

#arma3_feedback_tracker message

[shell lion]

same version 2.18.152272

I can send you both rpt if you want (also before perf 2 and perf 3)

but you cant see much in it

no error message to this problem

i use one HC on the same machine for my mission (server) since 6-7 years.

[shell lion]

ahh you found the issue

[tribal pasture]

Just realized why people complain about bad performance with my new playable location in KotH. It has lots of rocks added. I'm all in for fixing performance issues with rocks!

[inland dew]

malden sucks, because of rocks

[silk pewter]

Malden rocks

(because of—)

[hybrid cradle]

Yeah the rock performance is pretty bad

Never had perf issues with it on Malden but have had ai walk through them etc

[tribal pasture]

I just realized I typed rockets

[indigo anvil]

strap rockets to the rocks, blow them up into smaller rocks 🙂 🪨 🚀 💣 🧨 🤯

[feral harness]

Iron Malden

[silk pewter]

I named a mission like that actually 😹

[feral harness]

Haha, nice

[median belfry]

Im using profiling exe with -networkDiagInterval=100 and using serverCommand "#shutdown"; but it dosent create an mpMessageDetailsServer.txt any solutions ?

[vivid reef]

new profiling build and crashes are back

[whole cloud]

Gib crash dumps

[vivid reef]

already sent should be in your dm

with first 2.18 profiling it crashed in second game, with the second version it lasted until game 4, now it crashes again in 2nd game

[whole cloud]

Ah oops, thank you

[deft oak]

I can confirm headless clients won't fully load in 2.18 perf v03. For some context, we run an exile server with three headless clients. 1 runs missions (all spawning, management, and brains). 1 runs zombies (all spawning, management, and brains). 1 runs patrolling AI (all spawning, management, and brains).

THE ISSUE

Instead of connecting as usual and beginning their startup, for example the zombie headless client creating and setting up the triggers for managing zombie spawning, they seem to connect but never fully load into the server to run their init scripts. The console still shows the usual loading information:

16:12:28 Dedicated client created.
16:12:37 Client connected: 127.0.0.1:2302
16:12:38  > Welcome to Arma 3 Exile Mod!
16:12:47  > Johhny uses modified data file
16:12:49  > Player Johhny connecting
16:12:49  > Player headlessclient connected
16:12:50 Loading profile "HC"
16:12:59  > Player Johhny connected
16:13:00  > www.exilemod.com
16:13:00  > Enjoy your stay!

RESOLVED BY

Downgrading to 2.18 perf v02 allows them to boot and operate as usual. In both cases the headless client and server were both running the same performance build versions.

[whole cloud]

My question there would be, does V2 on the HC work fine on a v3 server?
So is it a Serverside or client side issue

[deft oak]

SERVER STRUCTURE BACKGROUND

I run the headless clients directly out of my dedicated server directory and typically use the arma3server_x64.exe for the headless clients and the server. For these tests I instead renamed one of the exes to arma3server_x64_hc.exe and ran the headless clients off of those.

TEST CASE 1

SET UP

• Headless clients running V03
• Server running V02

RESULT

This works fine. Headless client boots fully.

TEST CASE 2

SET UP

• Headless clients running V02
• Server running V03

RESULT

Headless client appears to join but never executes init.

TEST CASE 3

SET UP

• Headless clients running V02
• Server running V02

RESULT

This works fine. Headless client boots fully.

TEST CASE 4

SET UP

• Headless clients running V03
• Server running V03

RESULT

Headless client appears to join but never executes init.

[inland dew]

so HC problem is server side, v03

[deft oak]

Agreed.

[sonic moat]

Currently running perf 2.18.152272 and I found a bug with fixed wing drones (Greyhawk and Ababil) in which after giving the drone a land command, the drone will land and become "bricked". It would ignore any waypoints given to it and would later default to flying at a 250m altitude if I manually control it to take off.

[heavy galleon]

Imo that is not only fixed wing, but happens on all uavs

[harsh grotto]

a very old bug

[shell lion]

workaround on my mission, despawn AI unit in drone when it land and create a new one, sync it to terminal and it work again.

[whole cloud]

found an issue that could be the HC one on prof v3.
But I'm confused why this doesn't break anything else 🤔

Server could sometimes re-send the same network messages multiple times, and for that skip some others.

Okey and just now, got a report of parts of the JIP queue going missing.
We're reverting back to v2, will do another attempt on monday

[deft oak]

sounds good, let me know if you want me to do any other headless client configuration testing with any new versions

[tribal pasture]

If its from Dwarden then it was a thing forever

Unless it happens in 100% cases, unlike randomly before

[feral harness]

I will look at it for 2.20

In the meanwhile post relevant FT tickets so I keep track

[uncut linden]

Sorry, but what happened with interaction with other players backpacks and AI corpses? Why 'container' is now a 'GroundWeaponHolder' instead of unit itself?

[knotty wraith]

they have Arma 4 in 2027 - they are busy with impressions which day

[silk pewter]

I mean Arma 4 is done already, we just wanted to have a guesstimate of Arma 5 so releases would be spaced out evenly

[full nova]

2.20 Already in the works?!?!?!?!

[hybrid cradle]

It was in the works before 2.18 released

[heavy galleon]

Did anyone report disappearing bodies already?

[harsh grotto]

as far as I know, yes

[feral harness]

remains collector runs in MP by default AFAIK

corpseManagerMode - 2 (Default in MP) = None_But_Respawned - Only units that can respawn are automatically managed by the manager
corpseLimit - Default: 15
corpseRemovalMinTime - Default: 10
minPlayerDistance - Default: 0 <- this means it will just delete the body without checking if player close

these are defaults, dont think they are different from 2.16, what changed is that GC is actually working now

You can always disable it from 3DEN or via mission config

Or configure with different params

https://community.bistudio.com/wiki/Description.ext#Corpse_and_Wreck_Management

@spiral pond You were very eager to fix remains collector, what are your setting, maybe we need to tweak defaults to make it less obvious?

[empty goblet]

one of the issues i see with the GC is that around spawns (if it's static) stuff starts to accumulate , on high player servers there is all the time someone in spawn zone

but not sure if there is way to tell GC to apply more aggressive culling around the spawn

[feral harness]

the maxcount takes care of it, the body is deleted after maxlimit no matter what

[empty goblet]

ye but it negates the use of it elsewhere (the max count limits) so in the end extra pass to clean spawn zones must be diy

[feral harness]

Monitored corpses that are dead for longer than corpseRemovalMaxTime will be considered for removal regardless of the corpseLimit

[empty goblet]

you misunderstood, the stuff accumulated at/near spawn(s) will decrease the stuff 'preserve rate' elsewhere because limits

[feral harness]

you misunderstood, you regulate spawn pileup by the max time, and up the limit

[empty goblet]

when the maxcount limit is hit , the FIFO will be applied and removed those reaching min timer

[feral harness]

yeah

[empty goblet]

which is why you don't want stuff to accumulate at spawn because it makes GC mostly useless on rest of terrain (everything affected is shrunk to checkperiod+maxcount+mintimer)

for me that's opposite for 'too much stuff problem' leading to stuff removed too fast

[feral harness]

you can just remove bodies at spawn from remains collector then periodically and manage manually

there is no universal solution with GC

[empty goblet]

not just bodies, everything (depends on gamemode) but usually corpse+holders will be there but for example with WL it's also wrecks as vehicles destroyed can happen there too

i know like i said, it still means there shall be manual DIY pass on cleaning spawn-area(s) asap

[feral harness]

killed EH -> check proximity-> removefromremainscollector->add to own array to manage

[empty goblet]

speaking of, don't forget about the ejected seats and cockpits (usually need manual cleanup solution too) and then mines

[empty goblet]

another option would be add script command which would change values for the GC for specific flagged object (that way it would be possible define zones with more or less aggressive GC behavior) by just adjusting the timers related to the corpse/wreck/holder w/e else GC covers)

[spiral pond]

corpseManagerMode = 1; // Default: 0 for SP, 2 for MP
corpseLimit = 30; // Default: 15
corpseRemovalMinTime = 60; // seconds. Default: 10
corpseRemovalMaxTime = 300; // seconds. Default: 3600

disposalCheckPeriod = 10; // seconds. Default: 5

weaponholderManagerMode = 1; // Default: 0 for SP, 2 for MP
weaponholderLimit = 30; // Default: 50
weaponholderRemovalMinTime = 60; // seconds. Default: 3600
weaponholderRemovalMaxTime = 300; // seconds. Default: 36000

wreckManagerMode = 1; // Default: 0 for SP, 2 for MP
wreckLimit = 10; // Default: 15
wreckRemovalMinTime = 60; // seconds. Default: 10
wreckRemovalMaxTime = 300; // seconds. Default: 3600

minPlayerDistance = 25; // meters. Default: 0
//minPlayerDistance[] = {50, 60, 70}; // meters. Default: {0, 0, 0}```

[feral harness]

thanks

[gritty wasp]

"working now"
That's make sense 😃

For TvT with "one mission - one life" we definitely need to tweak it.
We can benefit from GC too if it could replace high poly bodies with gear to simple models of graves or loot boxes.

I tried to implement such thing years ago but failed on deleting bodies while owner in spectator

[spiral pond]

GC setup requirements depends very much on the type of scenario (respawn or not/same playable area or advancing to new/bodies required for looting/number of players/number of AI/etc)

[feral harness]

we could definitely add 25m radius and some lesser checks

[fast hornet]

is there an ETA for a stable branch fix btw.? We've had the choice between basically all mags disappearing on reload or constantly duping mag item types from containers (which is even worse by far) for weeks now

[patent dome]

There was a staging update late on Friday, which will have to go through QA first before stable gets updated. So I don't think there is an ETA currently.
Dedmen also said he will have to do some more work to fix the JIP queue issues, so it is even more uncertain if this staging build will be the one to go to stable. #perf_prof_branch message

[empty goblet]

imho the check can be even less than 10, like 15 , 20 or 30 s ... there is not need to do it rapidly unless the count limits are very increased

[feral harness]

I dunno a lot can happen in 30 seconds

Fun fact, those who carry more gear deleted last

Goes for weaponholders too. Wrecks that are smoking also left to burn when there are other wrecks to remove

[light cargo]

generally experimental things stay in perf/prof, to graduate to stable they need to go through qa which is a hard sell for bi who is trying to reduce costs on a game that's on a community lifeline

[fast hornet]

The problem is there on stable as well though. At this point in time it would be better to roll back to 2.16 instead of leaving the game unplayable for weeks now

[light cargo]

players can rollback on their own, bi provides beta codes for older versions actually

[fast hornet]

Not an option for a public server, as anyone knows.

[light cargo]

it is an option for public servers too, ask the hoster and require players to downgrade

[fast hornet]

which makes it not an option.

[light cargo]

bi can issue a official statement with instructions to downgrade

sorry for reply pinging

[fast hornet]

yeah right .. most players can barely manage to add mods on steam and start the game. Getting players to do anything beyond that is not an option.

[analog acorn]

Lots of mods and missions have already been updated to use 2.18 stuff. Trying to do a rollback now would be an absolute disaster.

[hybrid cradle]

Yeah far too late

Hotfix needs to come as soon as possible really

[whole cloud]

More details?
I don't know what you mean.

[rose moth]

We are indeed working on a default branch hotfix ASAP. It was slated for today, but will likely have to wait until tomorrow or Thursday. That's always still pending QA review, but we know the current state is not ideal.

[whole cloud]

2.18.152293 new PROFILING branch with PERFORMANCE binaries, v4, server and client, windows 64-bit, linux server 64-bit
- Tweaked: Taking secondary control in a UAV is now limited to the same Group. Video feed and taking control of an empty UAV is available to everyone from the same Side.
- Fixed Excessive damage of various buildings by bullet fire - https://feedback.bistudio.com/T185327
- Fixed: Possible audio related freeze at mission start
- Fixed: Crash fixes

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[whole cloud]

The vehicle inventory duping thing is still in there.
Once the hotfix is out, I can remove the serverside fix attempt again

HC should be fine now

[knotty wraith]

Well, we can say that with this patch v4 - 2.18 has been released, congratulations to all of us 🙂

[inland dew]

fixed 0xC0000005 - STATUS_ACCESS_VIOLATION?

[whole cloud]

You just asked me "fixed crash" ?
How would I know what crash you are talking about

[inland dew]

the question is what kind of crashes were fixed, for exemple

[analog acorn]

I don't think it's possible to fully "fix" this type of crash because it's often caused by memory or disk problems, which can happen for reasons outside of Arma

[whole cloud]

"access violation" says basically nothing at all about why it crashed. So asking about or specifying that is totally meaningless

I forgot what crashes it was that were fixed there. Nothing specifically notable, otherwise I would've specifically noted it

[inland dew]

with v2/v3 on the server, as opposed to stable, we had perf/stable players having their game freezing/crashing to the desktop, all at the same time, at different times of the day, but always exactly at xx:50 minutes of whatever time,
server was still running.
then, when starting arma again, due to crash and trying to reconnect to the still running server, again immediate crash of the game for any of the clients (perf or stable), trying to reconnect.
so impossible to fully join the server, unless server fully restarted
and for them all it was crashing with status access violation
so we went back to stable and no more perf/stable client crashes since then

and when the crash of all the clients at the same time was happening, even if you joined the server only like 1 minute before your client crashed, it was generating ~124 mb mdmp

[orchid pine]

Clients regularly getting stuck on "Receiving Data" as they load into a ALIVE Mission - Sometimes it will even crash their game on starting of the gamemode. Very limited amount of game changing mods and only a handful of arsenal specific items. I can post the .rpt/.mdmp here if needed.

(on latest perf branch)

[inland dew]

btw, full vanilla

[orchid pine]

Audio related freezes still present pretty sure.

Clients crash out as soon as the mission starts.

[knotty wraith]

everything works fine for me

[whole cloud]

Did you send me the crash reports for it?

[whole cloud]

mdmp's are always needed, as a zip file in my DMs

[inland dew]

i have them, but decided to roll back to stable and see if it happens also on stable, before cotcacting you
but none of that on stable
from tomorrow morning we will have today's version running

[whole cloud]

How do you expect me to fix crashes that you're not telling me about and which I cannot see?

[inland dew]

we first wanted to make sure it's due to arma/perf, because we additionally to that rented another server and moved there + a3 patch + perf patch.
had to exclude as much as possible of that, before disturbing you for nothing

[whole cloud]

any crash should be reported.
Worst case i drag&drop it into my tool and say "already fixed" or

Even if its a mod, or a script or bad hardware causing the crash, I'd still want to know

Maybe I should put up a mod for automated crash reporting

[inland dew]

dm'ed

[odd siren]

Ive sent you them. I am one of the players he is referencing

[cloud sky]

Why not integrate it into the base game like in Reforger? Or would it be too much work or sth like that

[whole cloud]

because reasons

[woven loom]

Doesn't Arma 3 already report crashes like that? Where do those crash reports go? 😅

[fickle geyser]

nowhere, you can click to send them which opens feedback tracker.

[woven loom]

Ahh yeah I think i confused the part where it packs the data with it also reporting stuff 😄

[inland dew]

perf for hotfix when?

[whole cloud]

patience young padawan

[inland dew]

finally 307!

[whole cloud]

2.18.152307 new PROFILING branch with PERFORMANCE binaries, v5, server and client, windows 64-bit, linux server 64-bit
- Sync with 2.18 hotfix
- Fixed: Game could freeze when tabbing out while rendering - https://feedback.bistudio.com/T183720

Note: This should also fix the issue with mission files not downloading. We need feedback about that, please report both if the issue appears again or if it disappeared.

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[inland dew]

finally no need to use full screen window anymore!

i guess...

[vivid reef]

mission file download needs to be profiling serverside too? Or clientside sufficent for it?

[whole cloud]

Either hotfixed stable, or profiling on serverside.
Stable has the same "fix" for that

Just don't run v4 or older

[autumn timber]

Out of curiosity: what's the downside of fullscreen window? I always thought that this was the better choice, since I can then alt-tab and it keeps rendering, with nopause and AFAIR nopausesound.
The only "downside" i know is that you can't "gamma cheat" when in windowed mode 🤷‍♂️

[quaint flame]

I have probably never used full screen. Not able to tab out seamlessly is annoying

[inland dew]

what your screen refresh rate?

do refresh= your screen refresh rate in a3 documents config and no problem to alt+tab super quickly then

[somber plank]

or just run fullscreen windowed

¯_(ツ)_/¯

[inland dew]

works different than full screen

you can google the difference

[fickle geyser]

Isn't there barley any difference since w10, game mode, fullscreen optimizations and all that jazz?

[somber plank]

yeah

[fickle geyser]

It was a lot different win7 and before.

https://devblogs.microsoft.com/directx/demystifying-full-screen-optimizations/

[somber plank]

yeah it used to be a big difference, now its barely noticable
only issue I have run into with a very small number of games is them not locking the pointer correctly and it moving to my other monitor when you look around very fast (then clicking on a different application and switching focus by accident)

[inland dew]

anyway, why do full screen window if full screen works
just set the refresh in a3 config to that of your screen

[somber plank]

bc full screen requires you to change a config and borderless just works?

[inland dew]

it's not because arma is not cimpetitive cs, that i will use slower mode, even if it's barely slower

launching the launcher, then arma, then go to the subsettings of settings to do full screen window is not less work than go to user config and change screen refresh number to that of your screen

[fickle geyser]

Iirc active window swapping still works better in borderless

Not having refresh rate in settings is a bit if a joke tbh

[inland dew]

whatever, nevermind
do as you wish

[somber plank]

you change ur video settings anyway when u first play a game, nobody opens the config file before they start a game 😂

[quaint flame]

144

[inland dew]

then you can change refresh=; in user config in documents to 144 and alt+tab full screen as fast as windowed
but up to you, really

[quaint flame]

Arma 3 User config?

[rain moth]

main config

[inland dew]

yes

Arma3.cfg

[somber plank]

I love it when someone proves my point
C:\Users\username\Documents\Arma 3\Arma3.cfg

[rain moth]

Arma3.cfg in Documents/Arma 3, not the user

[quaint flame]

It's still slow 😄

[somber plank]

did u close arma before changing it? iirc it overwrites it on exit

could be wrong tho

[orchid pine]

Doesn't it automatically set that for you upon the detection of the video settings? Or maybe NVIDIA is the culprit for having that adjusted on my end.

It seems to already be set to the max for my screen which is 165hz.

[inland dew]

is it still 144 in the config?
you have to restart for that
or make the config read only, so it doesn't reset to whatever

[quaint flame]

Yes, still 144, and arma wasn't started

[inland dew]

what was the number when you opened the config?

[quaint flame]

60

[inland dew]

so you changed that to 144 and with ; after that?

should be able to alt+tab quickly then

or check current refresh rate in nvidia

[quaint flame]

Nah.

I'll just leave it 😄

Let Windows have the 0.53 FPS

[inland dew]

works for me no problem

do you maybe have several screens and not same refresh across all of them

[quaint flame]

1 screen

[inland dew]

or maybe your screen is 144, but not actually running 144

idk

imagine paying fpr a 144 hz monitor and using it at default 60 hz...

[vivid reef]

happens a lot

[quaint flame]

Sorry, to disappoint you.

[inland dew]

then it's just arma being arma

[void badger]

Don't act like you're getting above 60 fps in ARMA anyway

[orchid pine]

Honestly as of late been getting that and more

[void badger]

True, bless Dedmen and the BI team for perf and v2.18 🙏

[vivid reef]

far above 60

[whole cloud]

Taking a look at physx repo's provided by @cloud nacelle

The scenario is definitely way over-done. But the improvement will apply to real scenarios too. But it scales about quadratically. So a hundred physx objects not much impact, several hundreds, definitely.

The test scenario is a house completely built out of physx boxes that all have sim enabled.

Every frame. We update these boxes.
Updating a box in itself is not a problem. But we need to also poke all objects nearby to the box, in case the physx movement might make it collide with other things nearby.

So, for every physx vehicle, we find all objects around it (around 25 meter radius), and update them.

In the example, we have about 2000 objects, that are all in about a 20 meter radius.

So, for the first box, we find and trigger updates for the 1999 objects around it.
For the second box, we find and trigger updates for the 1999 objects around it.
For the third box...

Every object nearby, needs to be updated atleast once. But we are updating them hundreds of times.

One small tweak. Collect the areas that must be updated, and filter away duplicates/overlap.

Before: 29001 cells to be updated, 171ms per frame. ~4 fps
After: 16 cells to be updated, 0.77ms per frame. ~21 fps

Now, the physx simulation also takes a very long time, with so many active objects.
So much, that out of 48ms frametime, we spend 33ms just waiting for the physx simulation of the previous frame, to finish.

So if we now multithread phsyx. It runs so fast that physx is done, just after rendering of the current frame.
~63fps.
I didn't do the multithreaded physx correctly, when I do it correctly it won't be as good, because I need to balance it for other stuff that wants cpu cores.
There is also still the problem with simulating these boxes itself. physx objects are simulated every frame (non-physx get simulated based on time intervals, and importance, and distance-ish)
Even when all the physx objects have gone to sleep, so they don't simulate physx itself. The game's simulation still takes 3.7ms. This could be parallelized, and I can probably get that down to like.. 1.5ms. But that will take a while and the gain is rather meh.

But ~4fps to.. say 60ish in about an hour is probably good progress 🤣
But also, again. No-one really runs a mission with 2000 physx objects in a 20m square.

This stuff was also missing a profiling scope, it was just "hidden" in wSimA.
Short look at YAAB (when the "benchmarking started" text shows), 0.211ms, 0.453, 0.212 before
after: 0.161, 0.153, 0.203, 0.157

It just scales with number of physx objects, and YAAB only has like 300

[full nova]

But also, again. No-one really runs a mission with 2000 physx objects in a 20m square. -Yet

[restive pilot]

I'm curious if ragdoll performance is improved too
You can place like 20-30 soldiers and kill them all with setDamage at the same time

[full nova]

Wait how do you do the @silent message, dedmen? this looks like a new discord feature

[whole cloud]

type @silent at the start. its old feature

[full nova]

interesting

[whole cloud]

This nearby object updating, specifically filters out "Man" type objects, for the nearbys to be updated. So
But still, it scales with numbers. With dozens of ragdolls it won't be noticable.

[fickle geyser]

ragdolls/dead units are weird in general, in my unit we found out that disabling sim on them can gain a lot of frames. not sure why are they so costly.

[whole cloud]

I shall take a looker

[restive pilot]

No I mean with MT PhysX

[whole cloud]

I remember server bugs where hundreds of dead soldiers would be ragdolling at 0,0,0. And.. it was bad. But not so bad to notice it with a few dozen

[fickle geyser]

Was my repro with trucks near rocks any useful?

[whole cloud]

I didn't get to rocks yet

I put down 20 units, and ragdolled them at the same time.
No noticable drop.

If I make it about 60 units that all simultaniously dip over. I drop to 20 fps.
Performance cost are collision checks, about 0.3ms per unit. Theoretically multithreadable. But to do that I need to put simulation into coroutines, which is a big task.
And you usually don't have dozens of units ragdolling at the same time. It usually a handful or less, so the real impact doesn't seem that big to me

[inland dew]

the more players are on the server, the lower the fps of each client, also other players are just there, doing nothing...

[whole cloud]

Okey. And you're telling me that why?

[inland dew]

just an observation
maybe has something to do with physics/ragdoll? idk

[whole cloud]

The more AI units you place down, even if they stand around and do nothing. The lower your fps gets.
The more vehicles you place down, even if they just stand around. The lower your fps gets.
Oh surprise.

No it doesn't.

[whole cloud]

Actually my collision checks, are only so "bad", because all the units are standing close to eachother. So they end up checking collisions against eachother.
Separating them further so they don't actually collide with other objects while ragdolling, gets rid of even this

If I trigger ragdoll with
_object addForce [[0,0,5000], [1,0,0]]
to make all the units fly around. My game still runs with 260fps, even with 60 units ragdolling and flying through the air.

[full nova]

i remember that time if you are laggy deadbody's weapon is still a solid object after they died and can launch or arma'd ur vehicle if you ranover them

[whole cloud]

Multithreaded physx likely won't do anything in most situations, because physx is already async. It only does something if there is so much physx, that it cannot finish it before the next frame, where it then would have to wait for the previous frame to finish.

[deft oak]

Not sure how related to this discussion it is since this is more about scaling of physx objects but I noticed in some testing the other day that if you fall from a high enough height inside a vehicle and hit the ground hard enough your body will ragdoll right through the terrain as if the physx interaction with the terrain doesn't exist. It also happens if you're killed during certain animations. Your body just ragdolls through the ground with no physx interaction with the terrain.

[whole cloud]

60 units ragdolling through the air. 2.3ms for phsyx sim. While my total frametime is 3ms.
You'd need hundreds of units flying around

[inland dew]

a body can be comletely unarmed and it still is capable of disabling/destroying/catapulting your tank

talking about alive bodies, not dead

[restive pilot]

Do you still see this happening? It should be fixed (well except in MP)

[analog acorn]

(well except in MP)
That's a pretty big "well" :U

[deft oak]

I'm getting reports of mags still disappearing occasionally. Running perf v5 on the server and the clients are running the most recent 2.18 build. Let me know if you want any files and I'll DM you them. We run 4 hour restart cycles and all our logs get extracted and packed into a single archive if you just want everything from that session.

[whole cloud]

actually sure all clients are running the most recent build? Even a single one, not having updated yet could cause it

[deft oak]

what should I set the required build to? I imagine that's the best way to enforce the build.

[inland dew]

and drop the number of players on the server...

[whole cloud]

152302

[deft oak]

got it, I'll update that and let you know if I get any other reports

[knotty wraith]

after the update, players with high ping began to be kicked from the server
v5

21:22:56 BattlEye Server: (Side) Kazak: ky
21:23:14 Player glen: Wrong signature for file C:\Program Files (x86)\Steam\steamapps\common\Arma 3\jets\addons\ui_f_jets.pbo
21:23:14 Player glen disconnected.
21:23:14 BattlEye Server: Player #3 glen disconnected
21:23:17 Player Kazak: Wrong signature for file C:\Program Files (x86)\Steam\steamapps\common\Arma 3\!Workshop\@ace\addons\ace_headless.pbo
21:23:17 Player Kazak disconnected.

21:36:29 Player glen: Wrong signature for file C:\Program Files (x86)\Steam\steamapps\common\Arma 3\heli\addons\anims_f_heli.pbo
21:36:29 Player glen disconnected.
21:36:29 BattlEye Server: Player #3 glen disconnected
21:40:44 BattlEye Server: Ban check timed out, no response from BE Master

[knotty wraith]

In general, the problem is that these two players who play far from the server are thrown out instantly

22:06:11 Player Kazak connecting.
22:06:13 BattlEye Server: Player #2 Kazak - BE GUID: 6c7b740fc7802cf1e964fe42ce7143c4
22:06:14 Player Kazak connected (id=76561198152291623).
22:06:14 BattlEye Server: Verified GUID (6c7b740fc7802cf1e964fe42ce7143c4) of player #2 Kazak
22:06:38 BattlEye Server: Ban check timed out, no response from BE Master
22:07:22 Player Kazak: Wrong signature for file C:\Program Files (x86)\Steam\steamapps\common\Arma 3\!Workshop\@CUP Terrains - Core\addons\cup_terrains_music.pbo
22:07:22 Player Kazak disconnected.
22:07:22 BattlEye Server: Player #2 Kazak disconnected
22:07:28 BattlEye Server: Could not connect to BE Master
22:07:28 BattlEye Server: Disconnected from BE Master
22:09:34 BattlEye Server: Ban check timed out, no response from BE Master
22:09:44 BattlEye Server: Could not connect to BE Master
22:09:44 BattlEye Server: Disconnected from BE Master

[orchid pine]

Turn off Signature Checks on server and see if that helps

(if it is a private server and you don't mind people throwing in extra content)

[knotty wraith]

if this solves the problem, then I still won’t be able to do it further, public server

[deft oak]

We're getting a bit of this as well with some of the less stable connections but we've been experiencing things like this for some time now so I can't rule out the host/box.

[whole cloud]

I see you saying people are kicked for high ping.
I also see you posting logs that say they are kicked for not high ping.

Mhm.. Interesting. Surely.

[knotty wraith]

I’ll say it differently - players from Russia log in without problems, from Europe - they throw them out

Possibly a coincidence, but one is from France and the other is from Germany

that's why I came up with the ping theory

Let's wait until tomorrow - perhaps a special case between the server and steam

absolutely empty server - stable version 800 fps, v5 - 1000 fps
While I was testing the problem above, I accidentally discovered this

#arma3_feedback_tracker message

[empty goblet]

are you sure they don't have obsolete / corrupt data ?

[kindred radish]

High ping / long distance to server can indeed affect the signature system, don't ask me how but i also had many clients back then with similar issues.
It was always some random pbo.

[light cargo]

regarding fullscreen: actual fullscreen is exclusive fullscreen, fullscreen windowed/borderless is a non-exclusive fullscreen and its desirable for multi-tasking, the former is better for performance because its directly on the gpu

[deft oak]

yeah I can't say for certain it's ping related but they'll get kicked randomly for a signature verification error while they've been playing for a while and are regular players

[whole cloud]

The chance to get a atleast somewhat reliable repro for that, is probably pretty low huh?

[deft oak]

I can get client rpts/dumps when it happens, if that helps at all. And I can pair it with the server logs/dumps. It happens from time to time with one of my regulars who would send me that stuff if I asked him. But yeah, pretty much no way to reliably reproduce that one.

[woven loom]

I don't think i've seen a meaningful difference in Arma though

The only difference i've noticed is that no form of vsync seems to work with borderless windowed mode

(this has been an issue pretty much since launch i think)

[light cargo]

same with windowed, right?

the vsync issue might be related to dwm composition/aero forcing vsync itself

because in borderless and windowed, aero is active

whereas in exclusive fullscreen it is not

in windows vista/7 with aero disabled you can have apps on the desktop doing vsync on/off just fine

[woven loom]

at least with win 10, many (if not most) games (that i have played) work fine with borderless + vsync

[heavy vortex]

I always figured the random signature kicks were caused by unreliable hardware. It seemed to be specific people who get them repeatedly.

Russia -> Germany isn't particularly high ping, is it?

[woven loom]

These days i'm assuming it's not so simple 😬

[tribal pasture]

Been an issue since at least OA actually

[heavy vortex]

I wonder if you can replicate this sort of stuff with ping and/or packet loss simulators.

[whole cloud]

logs and dumps probably useless. I need to know what the networking d oes when it happens. and there are no logs for that

[kindred radish]

Maybe that info helps to narrow it down a bit.
Deleting AppData\Local\Arma 3 solved the "false signature kicks" for a while for this problematic clients.

[tribal pasture]

Hmm, could it be drive reading issues instead?

Not sure about the algorithm of it, but I assume it slowly reads PBOs, calcs their signature and asks the server to verify

[knotty wraith]

The player enters the server without problems, plays for a couple of minutes and then gets kicked out

[tribal pasture]

I think these signature checks are done during gameplay over time

[knotty wraith]

the list of such players has already increased today

a year ago this was a common problem for a random player. It was solved by restarting the server and the problem disappeared for this player.
then these problems became very rare.
after the last update - this became a clear problem

Is it possible to roll back the server to v4?

maybe there is a problem with the server settings

[empty goblet]

or the person has overheating cpu or memory generating errors or corrupt ssd

it's very simple, let them generate you sha-? or md5 hash of the file they have and compare it with what you have on client

[empty goblet]

server with KOTH and WL Redux and all the large WL servers are on 2.18.152307 ... feel free to ping me if there is some issue

[whole cloud]

Imagine you want to write a frame capture to a log file.

First you capture the text into a long string, its 2 megabytes.
Now, you want to write it to a file in 511 byte chunks.
For every chunk, you iterate over the whole long string, to calculate its length, TWICE. Once you have the 511 byte long chunk, you iterate over the whole chunk, to calculate its length, even though you know it should be 511 bytes.

To write one string to a file, iterate over the whole thing 123 thousand times.

The game freezes noticably while trying to do that.
🥔

[autumn timber]

I'm getting GTA V vibes

[inland dew]

and that's on your relatively recent 32 threads system
imagine something slower than your system 🫠

[autumn timber]

Context: https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/

[quaint flame]

Nice that RS actually fixed it.

[whole cloud]

Imagine spending 7ms to simulate streetlamps that do, literally nothing at all.

[fair ore]

couldn't be me

[spiral pond]

not meaningful to add some more logging when it happens to narrow it down more (ping, connection quality, specific reason (if multiple), etc) ?

[whole cloud]

no

[spiral pond]

like its a timeout, isnt it? ppl not able to compute, or more likely not send back the requested info, and thus server drops them (as security measure)

[whole cloud]

No

signature timeout is a different message

[spiral pond]

so its about data/ram/network corruption?

can only recall initially the game only calculated signatures at game start, and verified once on server connection or after some delay.
as you could delete pbos, or tamper by other means with it, it was made into a reoccurring check. was done in a2 or late a2:oa day i am pretty sure

initially the new/changed system had some issues from what i remember and thus Suma/Bebul provided some extra logging build (unless i misremember)

[whole cloud]

There are multiple signature checks

[spiral pond]

@knotty wraith it might be worth to ask them to verify files or check via rpt if they have the hotfix version

Used to be a problem in A2: https://forums.bohemia.net/forums/topic/122719-resolved-hotfix-arma-2-server-111-wrong-signature-for-file-dtalanguagecorepbo/
Unlikely with steam and may also be another case.
Running -checkSignatures could be also worth a try.

Its probably a different case, but better than nothing

[hybrid cradle]

I've never managed to get prof build to run with mods without crapping out, is there a trick I'm missing?

[whole cloud]

I've never had any issues with mods

what do you mean by crapping out?

[hybrid cradle]

Crashes, freezes etc

I can get something better descriptive later today

[whole cloud]

If crashes, then surely they were reported to me?

[hybrid cradle]

I was about 90% sure it was user error so didn't report

Granted I was just launching by swapping the names of the binaries 🫣

[spiral pond]

Here its suggested to delete the files manually: https://www.youtube.com/watch?v=rAUrOEb2jNQ
At times steam has problems to delete obsolete files at least. Cant tell if it can miss to detect faulty files. But again could be different type.

[82899] Fixed: Signature checking of BAF/PMC addons.
https://forums.bohemia.net/forums/topic/116230-answered-wrong-signature-v2-signatures-and-bafpmc/?page=3

again may have been a different case. unfortunately archive.org is either still bugged or no longer has http://www.dev-heaven.net/issues/21861

[knotty wraith]

what do you think about my server settings?
PvE server (150 AI (HC) vs 10 players)

MaxCustomFileSize=0;
MaxMsgSend=672;
MaxSizeGuaranteed=1324;
MaxSizeNonguaranteed=1324;
MinBandwidth=939524096;
MaxBandwidth=1024000000; 
MinErrorToSend=0.0023; 
MinErrorToSendNear=0.023; 
class sockets{
    maxPacketSize    =1200;
    initBandwidth    =160000;
    MinBandwidth    =12000;
    MaxBandwidth    =6400000; 
}; 

battleyeLicense      = 1;
allowedVoteCmds[] =            // Voting commands allowed to players
{
    // {command, preinit, postinit, threshold} - specifying a threshold value will override "voteThreshold" for that command
    {"admin", false, false}, // vote admin
    {"kick", false, true, 0.51}, // vote kick
    {"missions", false, false, false}, // mission change
    {"mission", false, false}, // mission selection
    {"restart", false, false, false}, // mission restart
    {"reassign", false, false} // mission restart with roles unassigned
};
 
BattlEye         = 1;
verifySignatures = 2;
kickDuplicate        = 1;
allowedFilePatching = 1;

allowedLoadFileExtensions[] = {"hpp","sqs","sqf","fsm","cpp","paa","txt","xml","inc","ext","sqm","ods","fxy","lip","csv","kb","bik","bikb","html","htm","biedi"};
allowedPreprocessFileExtensions[] = {"hpp","sqs","sqf","fsm","cpp","paa","txt","xml","inc","ext","sqm","ods","fxy","lip","csv","kb","bik","bikb","html","htm","biedi"};
allowedHTMLLoadExtensions[] = {"htm","html","xml","txt"};

doubleiddetected = "kick (_this select 0)";
onhackeddata = "kick (_this select 0)";
onunsigneddata = "kick (_this select 0)";
netlog=1;
logFile          = "server_console.log"; 

forcedDifficulty = "Custom";
steamProtocolMaxDataSize = 99999;
//requiredBuild = 152263;

[spiral pond]

i cant judge that. another thing to check could be how they join the server (via external launcher or internal) - if external is still running, may try the parameter to close the a launcher on game launch

back in the day also some poor routers could cause it

[whole cloud]

Buildings are normally simulated every 500ms.
But buildings that have lights, are simulated every 30 milliseconds.... Because, their light might move around..

Even though, the several thousand landposts on Fallujah, are all static objects with no animations that cannot move. But they are simulated as if they could move around

On Fallujah, every 30ms, 5000 static lamp posts, which cannot move. Are being simulated. Just because they might be moved around and would need to update the position of their light.
And also during daytime, when their light isn't even visible.

[hybrid cradle]

Standard arma moment

[light cargo]

you need network captures?

[whole cloud]

No. I need to reproduce.

[light cargo]

wouldn't those help to repro?

[whole cloud]

Networking is encrypted. No they wouldn't

[fickle geyser]

at this rate you will triple the framerate before we'll have arma 4

[fickle geyser]

how old is that code?

I wonder if it's a remnant of OFP light houses

they were rotating IIRC so maybe someone thought it's a good reason to update buildings with lights more often xD

[knotty wraith]

my RPT is flooded with these messages, what does this mean?

 In last 1000 miliseconds was lost another 49 these messages.
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1185) to 1475454602
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1186) to 1475454602
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1228) to 1475454602
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1239) to 1793543364
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1211) to 1793543364
16:31:05 NetServer: trying to send a too large non-guaranteed message (len=1148/1163) to 1475454602

[whole cloud]

yeah lighthouses from 2005

[light cargo]

a lot of RV a3 code is as old as the franchise itself unfortunately

[whole cloud]

some engine bug. Combined with you reducing the maxPacketSize

[light cargo]

what features are actually new code written for a3?

[fickle geyser]

a lot

[light cargo]

but a lot of the world/network seems dating from ofp/oa days

[fickle geyser]

ofc it does, all engines have stuff like this

[light cargo]

it depends on the engine but sure

TKOH and TKOM as well?

[fickle geyser]

tkoh is RV too

TKOM is enforce engine

[light cargo]

Ah ok

enfusion or enforce?

[fickle geyser]

enforce

[light cargo]

oh okay

thanks!

[fickle geyser]

enfusion is in some places on foundations from enforce, which also has some crust, AFAIK one of historic reasons the script files are in .c format

[whole cloud]

I tried to do multithreaded simulation with these lamp posts.
But even the simplest kind of simulation (Building), already has way too much stuff inside it.

When it has a flag on it, it needs to load the model, which can't be.
When it has a windsock, can't be.
When it has inventory, can't be.
When someone is doing some action on it currently, can't be.
When something is attachTo'd to it, can't be.
When its not static object, can't be, because it interacts with global AI map thing.
When it has marker lights, can't be.
When attached to a parachute, can't be.
When it is ambient animal, which is outside of camera area, can't be.

The time spent to filter on all these things to decide whether MT or not, would waste all the possible gain. Sad.

But, can probably atleast reduce the simulation speed dynamically.
A animated lamp might need to be simulated every frame. But certainly not when its kilometers away and not visible.

[hybrid cradle]

Old enough to buy arma 🙂

[spiral pond]

wasnt StreetLamp its own simulation? vs house or houseSimulated

[whole cloud]

There is a own "StreetLamp" simulation type, which simulates every 12 seconds and also has no multithreading blockers.

But, the lamps, both vanilla and CUP. Don't use it.

Land_LampShabby_F = "house"
Land_Lampa_sidl = "house"
Land_Lampa_ind = "house"

Fixing these, would be the proper fix for this issue.
Not only my hack getting them down to 500ms, but all the way down to 12000ms, without the need for hack fix

[light cargo]

wut

aw :/

[whole cloud]

smol tweak for non-animated, static lamps that cannot move..

That 2005 fix seems weird. If you need lighthouses to update alot. Why make all buildings with lights update alot, instead of just checking for lighthouses.
I assume streetlamps didn't exist back then, and no-one revisited it 😄

From 7+ms down to about 1ms.

Now, most of the remaining simulation time, is actually spent iterating over objects, and checking the timing and deciding that their time has not come yet.
Multithreading could help with that. But that will only be beneficial when there are alot of objects that are not simulated. And probably harm performance if there are many objects that are simulated that frame.

[spiral pond]

a3 seems to do it also via house simulation

class Lamps_base_F: House_Small_F
class Land_LampStreet_F: Lamps_base_F

[light cargo]

kek, weird

[whole cloud]

I can't see why you would do that.

StreetLamp simulation being simpler has a couple downsides.
Cannot attachTo objects (well you can.. but they won't be simulated at all)
Cannot have flag's on it (like flagpole)
Cannot have windsocks on it.
Cannot have animated textures.
Cannot have animated anything actually.
Cannot have inventory.

But all of these, aren't relevant for streetlamps really

[fickle geyser]

would anything break if we would just update the config?

[whole cloud]

Well the above listed things.
Someone attaching something to a streetlamp, might be unhappy about it. But that's all I can think of

[fickle geyser]

could config patch vanilla with a mod and ask cup to update

[whole cloud]

Could also make a patchmod that fixes both vanilla and cup

[fickle geyser]

@coral summit there's some configs to be updated in CUP

are you sure you can just change sim type to StreetLamp tho?

I recall the StreetLamp being in CfgNonAIVehicles

idk how that works

[spiral pond]

in CfgNonAIVehicles

 class StreetLamp {
  model = "";
  destrType = DestructTree;
  simulation = StreetLamp;
  colorDiffuse[] = {0.9,0.8,0.6};
  colorAmbient[] = {0.1,0.1,0.1};
  brightness = 1;
  armorBulb = 1;
 };
 class StreetLampWood: StreetLamp {
  scope = 1;
  model = lampadrevo;
 };
 class StreetLampMetal: StreetLamp {
  scope = 1;
  model = lampazel;
 };```

from OFP

you might want to have two variants

this more simple simulation variant for terrain placed, and the more complex in cfgVehicles for mission makers

[light cargo]

hmm, what about other things that would benefit from dedmen updating?

[spiral pond]

in A3 it has some more complexity too:

    class StreetLamp
    {
        model = "";
        destrType = "DestructTree";
        simulation = "StreetLamp";
        animated = 0;
        colorDiffuse[] = {0.9,0.8,0.6};
        colorAmbient[] = {0.1,0.1,0.1};
        brightness = 0.2;
        class HitPoints
        {
            class HitBulb
            {
                armor = 1;
                material = 60;
                name = "lampa";
                passThrough = 1;
                explosionShielding = 1;
            };
        };
        armorStructural = 1;
        class Reflectors
        {
            class LampLight
            {
                color[] = {0.9,0.8,0.6,1};
                ambient[] = {0.1,0.1,0.1,1};
                position = "Light";
                direction = "";
                hitpoint = "lampa";
                selection = "";
                size = 0.5;
                brightness = 0.2;
            };
        };
        aggregateReflectors[] = {};
        armorLights = 1;
    };```

[light cargo]

also, perf/prof has taken the place of community qa, right? or does it still need to go through bi qa department for stable graduation?

[whole cloud]

Good catch.
When creating CfgNonAIVehicles, then "simulation" can be "streetlamp".
When its CfgVehicles, then "streetlamp" is not actually a simulation type.
Its a class= property inside the p3d model file.

So you'd need to change the prop in the p3d

[fickle geyser]

it goes through QA when it's going into stable.

[fickle geyser]

so no place for simple perf gain

[whole cloud]

Well CUP can still fix the property in theirs 😄

[fickle geyser]

and BI in theirs

🏃‍♂️

|| no one wants to touch the models ||

[light cargo]

data updates without breaking compat with stable for perf/prof when?

[fickle geyser]

@silk pewter do we have any place this could be documented?

[fickle geyser]

never, not possible.

[light cargo]

how/why, is it because of data consistency/checksums?

[fickle geyser]

base game data works the same as addons, whole game is an addon, data must match.

[light cargo]

ahhh, but the base game data is like, separate addons for the core and the dlcs, right?

eg, addons, expansion, karts, heli, etc

[fickle geyser]

and?

[light cargo]

nevermind, i mean we could have a special addon added for testing data in perf/prof

we can control loading of individual pbos, right?

[whole cloud]

https://community.bistudio.com/wiki/Named_Properties#class
It says it also wants a CfgNonAIVehicles class for it, but on quick look I cannot see why that would be required

[light cargo]

so basically what im saying is provide a core mod pbo that only gets loaded when hosting rather than when joining as a client in servers that dont support it, but yeah i see how that would break compat with stable. but admins that want stable compatibility can just find that pbo and rename it to make it no longer load

[fickle geyser]

then the servers will refuse clients without that pbo

[light cargo]

see what i just said afterwards

the pbo will be optional and only provided in the perf/prof gdrive

[fickle geyser]

maybe it converts the CfgVehicle into simple one based on that other config?

wasn't it working somewhat like that?

[light cargo]

but admins that want stable compatibility can just find that pbo and rename it to make it no longer load

[fickle geyser]

what's the point then

[light cargo]

data fixes before stable without dev

[fickle geyser]

perf prof is there to be an easy drop in to test stable compatible performance changes

your data fixes basically turn it into dev as everyone needs to run the changes

you're making no sense

[light cargo]

right

[fickle geyser]

just have everyone playing on your server switch to dev then

[light cargo]

im not trying to force it to everyone but everyone who wants them, understanding the risk of breaking compatibility

sure i guess lol

dev isnt that bad of a switch/update, right?

well thanks for the discussion

[fickle geyser]

dev is basically perf prof without few bells and whistles

but with data changes... and new features

[light cargo]

ahhhhh hmmmmmm, i wonder, why some things are exclusive to perf/prof?

[hybrid cradle]

I even offered to do it for them just to fix a faulty viewpilot hidden selection in contact uniforms 😦

[fickle geyser]

They've even had some people send them fixed vanilla p3d models

and they were not implemented

[hybrid cradle]

I am going to demand a refund for contact from steam because of the shemagh only visible in viewpilot it's a game breaking bug

[empty goblet]

actually the person shall use -checkSignaturesFull and if that toss error the client got corrupted file or signature (or both)

[spiral pond]

checkSignaturesFull checks every byte of the file content, and therefore not only verifies signatures, but also verifies file integrity. It is not fully implemented yet, it fails on DLCs.

wasnt aware of that one yet

[quaint flame]

Is that a startup parameter for a3?

[whole cloud]

😢
Multithreading the filtering
But, I also waste about 0.12 (out of the total 0.4), just to set up the jobs and start them up -.-

[spiral pond]

added: https://community.bistudio.com/wiki?title=Arma_3%3A_Startup_Parameters&type=revision&diff=369984&oldid=369933

[heavy vortex]

Why is maxPacketSize smaller than MaxSizeGuaranteed & MaxSizeNonguaranteed?

[quaint flame]

I searched for it a minute before 😄

[spiral pond]

@knotty wraith ask the two ppl to run that please

[heavy vortex]

Unless it's coming up with the same files regularly I doubt it's on-disk corruption. IME with those errors they get random PBOs each time.

[spiral pond]

https://cdn.discordapp.com/attachments/1092300658738221106/1297521230294941706/image.png?ex=671b8050&is=671a2ed0&hm=5a2e2a99b447ab86f381daeeed1199d63b0065d7895c4f36b9eee2630271d310&

yeah its most likely not that. still have to rule out things step by step

or someone comes up with a magic repro for Dedmen (if not hardware/isp related)

[empty goblet]

yes, btw. i don't like the remake of the startup commands page #community_wiki message

[empty goblet]

it doesn't need to be on-disk corruption, it could be

• issue with controller (heat or firmware or faulty)
• issue with interface (bad cable, overclocked or EM interference beyond what CRC can deal with)
• software issue (firmware, driver, OS)
• overlock or heat issue causing any chip overheat (CPU, RAM, chipset i/o, device controller)
  and several other edge things (including malware and ransomware or even some crazy antiviruses / anticheats and cheats 🤣 )

[heavy vortex]

Yeah but -checkSignaturesFull doesn't necessarily trigger any of those.

[empty goblet]

hence why i asked for simple compare of the hashes of the .pbo and pbo.bisign files from client of the affected user vs whoever you sure has the correct data

[empty goblet]

it's good start, same as doing the simple verify files in STEAM

and because i got 0 answer about the hashes, my assume is the files or theirs processing are corrupt, until proved they ain't

ie ``` File: ui_f_jets.pbo.a3.bisign
MD5: 1ca7a8fbd890540ea8bd94b6dcc82a3b
SHA-256: e8378d8f8ebeb3a584713c5d8378776f0371a6610359a23678a8924a963cc6f0

File: ui_f_jets.pbo
 MD5: 4ada61d4c8259f880889a0a47d88426d

SHA-256: 80968483a1f2bbeb78e2d1cd760d9fc1e26f0a9e0a78b1fe41558bc9dc4bf4f3
and
File: anims_f_heli.pbo.a3.bisign
MD5: 698c97880bdf6713f0e8d2c795da9981
SHA-256: 813fd476aaa1e087388e1be7cbdaf432011f5e0fff517473c57d40ae6ff74434

File: anims_f_heli.pbo
 MD5: 2ea2254fd8e98e177470d3dce066e047

SHA-256: bbc7d858bfffdd9a843737ff3449217ded8abc5570e25706d7faabf547bbff1c

[knotty wraith]

and a new game day has come - yesterday's problem players came in and there are no problems, I have nothing to say - I wash my hands

I have no idea what these problems are

[whole cloud]

Well overhead down to 0.008ms

On empty terrain:
Simulating 70 / 4926 objects
Before: 0.670ms
After: 0.309ms

A mission on that terrain:
Simulating 542 / 6068 (Simulating number per frame is higher, because my fps is lower)
Before: 1.119ms
After: 0.676ms

That is better than I expected.

And I'm only doing this on wSimB. There's also wSimO and wSimR. But I don't know how high the filtering out ratio is on these.

It looks like from the total 14.9ms sim time this multithreading could at most get rid of another maybe 0.5ms
Instead, I found a thing I can (probably) cache in AI units (that are standing still, but in formation) simulation, which saves 4ms off that..

On another note. Why? They aren't even doing anything..

[spiral pond]

the lovely animals module 😄

[silk pewter]

the wiki? 🤭

but yeah we will see to that w/ Dedmen, thanks

[light cargo]

what's the difference? i thought -checkSignatures did it all?

[light cargo]

we should remove 👏 animals 👏 and 👏 zeus 👏 /hj

🥺
👉👈

[plain trout]

You can disable the animals.

https://community.bistudio.com/wiki/enableEnvironment

[hybrid cradle]

theres also

{deleteVehicle _x} forEach allMissionObjects ""```

does the same thing

[fair ore]

you were really cooking with this script

[loud garden]

deleteVehicle (allMissionObjects "");

But it's preferable to use dedicated command.

[fair ore]

wh

deleteVehicle doesn't take an array

[loud garden]

It takes, from v2.18.

[fair ore]

wiki wasn't updated, then

[loud garden]

It's updated.

[fair ore]

😦 !!!!!!!!

[woven loom]

Alt-F4 for peak performance

[hybrid cradle]

oneachframe{{private _og = _x; private _obj = (typeOf _x) createVehicle [0,0,0]; _obj setPosASL getPosASL _x; _obj setVectorDirAndUp [vectorDir _x, vectorUp _x]; {_obj setVariable [_x, _og getVariable _x, true]} forEach allVariables _x; _obj setVelocity velocity _x; deleteVehicle _x} forEach allMissionObjects "";}```
this script will triple your framerate especially in dedi

[loud garden]

Proofs?

[median eagle]

i wouldnt believe most things kjw says at face value.

[hybrid cradle]

it sends less network packages

[loud garden]

Provide FPS diagrams.

[hybrid cradle]

ok

give me a bit

https://cdn.discordapp.com/attachments/332936363340988416/1299122876770816051/image.png?ex=671c0df6&is=671abc76&hm=0dbac2225e4fdd0eb3cc8236a583a36bfb99db8ccbe911687e38c428389d69d9&

[silk pewter]

stahp trollin' 😄

[hybrid cradle]

😦

[loud garden]

Somehow it doesn't correspond with the fact that you are using createVehicle, which already implies synchronization over network. But OK, it was funny.

[analog acorn]

He wasn't being serious. The script is a deliberate joke.

[loud garden]

Sure, but I was interested to see "proofs".

[bright latch]

I do have a script somewhere that uses attachTo on lights on streetlights 😄

but you can break it, it's fine

[spiral pond]

possible to get diag_captureSlowFrame for the server before your MT and optimization work is ready please?

also to get diag_logSlowFrame in A3 finally would help a great deal #arma3_feedback_tracker message #perf_prof_branch message (or as parameter for diag_captureSlowFrame to log instead of visualize)

HC note potentially still relevant too: #server_linux message

finally a native solution to re-add diag_logSlowFrame/diag_captureSlowFrame after it was triggered (instead of the scripted loop)

PS: can you have multiple diag_captureSlowFrame with different "section" targets active at the same time, or just one diag_captureSlowFrame active at a time overall?

[whole cloud]

No. My fixes for it are too intertwined with the optimization things. But 🤞 that it comes next week
Thanks for reminding me how old I am, I'll take a look 😄

[whole cloud]

I would prefer log slow frame as parameter.
But we already have https://community.bistudio.com/wiki/diag_logSlowFrame so.. I guess I'll just fix it 🤣

Although.... That seems to have intended different functionality.
It says "log all frames" so sounds like it would continuously capture, even after it hit a slow frame.
But it really is only two words on the wiki that we need to change. So we'll probably make it only one frame.

CaptureSlowFrame to file, already exists. So I can just throw it in that command

Or I just keep logSlowFrame broken.
And instead add parameters for "toFile" and "continuousCapture" to captureSlowFrame command.
Mh. Putting it all into one command is easier, but we'll have that wasted logSlowFrame command thats kinda duplicated then 😄 But I'll just blame the past for it.

[whole cloud]

Actually instead of bool for continuous, I'll give you a number so you can say how many you want to capture. So just like 10 slow frames
and same for the #captureSlowFrame chat command

[whole cloud]

Ah the pain that every feature inflicts.

When doing multiple captures, all but the last one are corrupted. But it doesn't make any sense.
When it writes out the first capture, it doesn't even know if another will follow. Nor with the last one. 🤯

If I set a frameOffset of 1, it doesn't work. A frameOffset of 2 works. Well I guess that's the fix, if you're capturing slow frame you don't care if it only starts checking in 2 frames from now
But I have no idea why it happens.
If the first were fine, and the next ones were corrupted that'd atleast make sense

#captureslowframe total 0 0 4
and it captures every 3rd frame 😄

Just don't set a too high number..

Mh. seems captureSlowFrame total 0 0
Was always broken. If it captures the same frame, which also initiated the recording, it corrupts.
And even if you set offset to 1, it still immediately captures the current frame. I guess the minimum valid offset is 2...

You can now also do
diag_captureSlowFrame ["total", 0, 2, false, 3]
Which will open the capture ui 3 times

It always gets F'ed up when you start capture in a frame that gets captured. I think I have to hardcode that when you use continuous, the offset must be minimum 2.

yeah offset was bugged.
Offset of 0 means capture current frame.
Offset of 1 means capture current frame.
Offset of 2 means capture next frame.

Now I just need someone to do the wiki-ing for it 😄
diag_captureSlowFrame ["total", 0, 0, false, 3]

scope, threshold, offset, bool to file, scalar number of slow frames to capture

#captureSlowFrame server chat command
#captureSlowFrame total 0 0 3
scope, threshold, offset, number of slow frames to capture.

diag_logSlowFrame now also works, it infinitely captures, until you run captureFrame or captureSlowFrame which will stop the continuation.

I hope this'll be in next week

[quaint flame]

Can we already paste these in the chromium profiler?

[whole cloud]

exportJIPMessages doesn't log to RPT where the file is. (is in servers working directory)

logEntities Only has filename instead of Path and is also written to working directory instead of RPT directory.
It already logs to RPT where it is.

Both name doesn't conform to the standards (instead of using the function to generate a logfile name, someone re-implemented it by hand and its slightly different)
And the hand implementations for both, were copy pasted, and putting it into RPT directory part, was missed on linux.

[whole cloud]

You'd have to import it into the in-game UI, then ctrl-click export to clipboard to get chrome format.

I already have exporting both formats to file in my build. Not sure if I want to burden everyone with two files per captured frame.
And making it selectable is.. I guess command line parameter maybe? ugh
Could just say profiling is already a use case only for experts, and they can deal with having two files per frame 🤔

[restive pilot]

Speaking of multithreading simulations, aren't simulations of some objects dependent on other objects?

[whole cloud]

I don't think so.
Because we have no sorting/ordering of objects. Except for attachTo, but there the attached objects don't simulate themselves, they get simulated by their parent

[quaint flame]

Please give us two files.

[restive pilot]

What about, e.g. an object that's attached to another object?

[whole cloud]

[restive pilot]

BTW what about collision resolution? Was it done after simulation? I don't remember

[whole cloud]

its inside simulation

[restive pilot]

Well then that'd potentially break 😅

[whole cloud]

No it wouldn't. But also that would crash when multithreaded

[knotty wraith]

I'll try to do it

[whole cloud]

We had so many issues with JIP queue, both due to the magazine bug thing, and also previously it repeatedly came up.
If JIP queue is large, then updating it becomes expensive, dragging down the server FPS alot.
If your JIP queue has 20k public variables inside it. Then adding another 20, would take about 6ms and drop your server fps.

So... Async JIP queue!
Instead of message processing on main thread taking ~7ms. It now takes 0.05ms, and the rest runs asynchronously in parallel to everything else.
It'll even be spread across multiple frames, because it only needs to be ready when a JIP player joins (or some other manual triggers)

[fickle geyser]

I'm constantly impressed how much can be improved by having competent devs work on something without any deadlines forcing him to cut the corners.

Labour of love!

[whole cloud]

Throwing 1000 variables into it, used to freeze the server.
Here picture of before and after.
Before server freezes for 200ms while processing messages
After.. well.. Processing messages takes 0.5ms, and the JIP queue updating runs asynchronous, over the next ~7 frames.

If JIP queue is so overwhelmed that it cannot keep up, the server would now freeze when a JIP player joins, and wait till the queue is processed. Which is probably not so nice. But as it now has ALOT more available time to process it, I think it getting overwhelmed will be very rare, and the chance of a use JIP'ing in that moment should be low.

Ping @empty goblet so I don't need to copy this into DM's #perf_prof_branch message

When you have 150k variables in the JIP queue. Then adding a single variable could take around 3.5ms.
Update a handful variables regularly (like ACE medical does), and your server performance would go to its knees.
But now. Even when JIP queue is absolutely overwhelmed, server runs with smooth 50fps. But even 100 frames later, its still not done processing what I threw at it, but it doesn't matter. Until a JIP player joins, then you get the lag spike, that you would've gotten before (minus the things that were completed).

Run a #exportjipqueue during a inconvenient time...
whoops. 7 second server freeze

This also poses a danger I guess.
If your server runs at 1000fps, and every frame you send a publicVariable from a script running on server. And if that variable takes more than 1ms to insert into the jip queue.
It would gradually, put more and more tasks into the queue, because processing it takes more time than it takes for the next task to be added.

Server could kill itself on it.
Don't run your server at unreasonably high fps. Don't run bad scripts that push variables every frame.

I could detect when the queue is too full, and just add a sleep to force slow down the fps if it happens. But I don't want to 😢

[nocturne obsidian]

It sounds like the current setup is impressively optimized to maintain high server performance, even under the load of a heavy JIP queue. Given the sheer volume of variables, the delay for a single variable update makes sense, and it's great that regular updates from mod don’t immediately tank performance.

However, the trade-off seems to show up when a JIP player joins—since the system has to pause to catch up, resulting in that lag spike. This setup could potentially be improved wh some form of asynchronous queuing that prioritizes live players’ needs over new join-ins or perhaps by batching updates specifically for JIP events to spread the load. Alternatively, restructuring updates to selectively prioritize or delay certain variables could help balance out the demands. However, I know that might be tricky given the specific requirements of mods like ACE.

[whole cloud]

What?

Are you a bot?

[nocturne obsidian]

no

[whole cloud]

Sounds like ChatGPT

[nocturne obsidian]

😂 Thanks I guess

[whole cloud]

I can let the JIP player wait till the queue is empty enough that there won't be a big lag spike. But there is no guarantee that that will ever happen, if we don't halt the server (to prevent new insertions).
I could force the server fps down to a low number, to hopefully reduce insertions enough that the task list doesn't keep growing while server is running, but there is no guarantee that that will work. If it doesn't, the JIP user can never join in.

I cannot suspend the processing and do it later, because at the point where JIP queue is sent, it must be complete.

[nocturne obsidian]

You could look into managing priorities in the queue itself, I guess. but as you pointed out, without halting the server to block new insertions, there’s no guarantee that the queue will ever thin out.

[whole cloud]

And as I said above. The chance that a JIP user joins while JIP worker is overwhelmed is pretty low.

You could look into managing priorities in the queue itself, I guess.
I don't see how that would be a thing? There are no priorities here

[nocturne obsidian]

I was thinking batch processing. But as you said. a JIP user joins while JIP worker is low. Sorry for the confusion! I'm just starting to learn!

[vivid rune]

Can you lock the JIP variables so you can send it over more frames without changes?

[whole cloud]

I don't understand what you mean.
As I said, the JIP queue must be completely processed, to be able to be sent. I cannot pause it, if that's what you mean

[knotty wraith]

https://community.bistudio.com/wiki/deleteVehicle
point your finger - we don’t see)

[loud garden]

Alt syntax.

[vivid rune]

I think so. As I understand the processing of the the JIP queue is now parallel to the main thread. If JIP happens the JIP queue has to be finally processed for sending to the JIP player. What if this thread would in that case prioritized so in a multicore system it occupies one core for it (the rest are running on the remaining resources) and in a one core system it slows down the server because it has to be happen.

[whole cloud]

The thread will get highest priority while waiting on it. But that doesn't help that much
And one core systems don't exist anymore

[vivid rune]

What if that thread is sitting on his own CPU core only (maybe only when the queue is detected growing)? Is that possible?

[whole cloud]

Your operating system will decide that. And I don't see why that would make a difference

[spiral pond]

what means 150k variables in the JIP queue exactly? 150k different publicVariable/setVariable or what counts as "variable" here - also every state the engine has to sync (ie unit positions, damage state, etc)

[vivid rune]

Then I haven't understand what the system would slow down.

[whole cloud]

_ar = [];
_ar resize 150000;
{
missionNamespace setVariable [format["a%1", _forEachIndex], 1, true];
} forEach _ar;

[spiral pond]

if just "user/scripter" space, then there should be a BIG WARNING - "BAD CODING - FIX YOUR STUFF", no?

[whole cloud]

What warning? You want me to detect when someone sends too many variables?

[spiral pond]

rather the server if it get bogged down for too long for processing the JIP queue

time probably no good as you would need to measure it, yet size count of the jip queue should be rather cheap, no?

[fickle geyser]

wouldn't time be cheap too?

could print something to RPT when processing took too long

but what is too long

[whole cloud]

I can do "if size of to-be-processed tasks is higher than 1000 for N consequtive frames, then print warning"
But I'd rather not spend time coding that, for a case that should never happen

If you want to look for it, you can run #monitor and watch it

[full nova]

The path to 2.20 looking good so far

[vivid rune]

Make it sense to parallel JIP queue processing at some size is exceeded?

[whole cloud]

Not possible

Only one insert can be processed at a time.
If I split the queue into chunks, I can probably split that one insert into multiple tasks on every chunk.
But that would only really make sense if the queue is so large, that processing takes a long time. Which already should not happen, and does not happen on normal servers.

If your server today, runs at 5fps due to the jip queue being too full. Then it could cause a lagspike on JIP joining with the new system.
But if your server today runs at 5fps, you're doing something wrong already.

[heavy vortex]

I'm slightly puzzled that the cost of adding/replacing an entry in the JIP queue would be higher than the cost of the SQF that's doing it.

[vivid rune]

Yeah good point. I assume the processing of the variables itself would slows down the server first before the queue processing would take slow it down.

[heavy vortex]

I mean, it's not impossible, but that feels like it'd need to be a bad algorithm...

[silk pewter]

😄

[heavy vortex]

What also worried me about the JIP queue is the implication that inventories are always a sequence of changes from a baseline. Surely you just make a new baseline after X changes?

[whole cloud]

SQF is running on client.
You might have 50 clients, all spamming 100 public variables every frame, every client running at like 60fps.

[heavy vortex]

Ok, if people are doing that then they need some instrumentation to tell them to stop :P

[whole cloud]

In most cases, the server will not slow down at all.
It only needs to wait for the processing, if you #exportjipqueue, or a jip player joins. Which is rare

[vivid rune]

As I understand now if a player wants JIP the server "halts" for processing the JIP queue to be send to the player. I would say you can wait a player some time for the processing of the JIP queue. The only thing that have to be ensured is that the queue would be smaller over the time so that there is a chance that the player can join.

maybe that can be forced by gradually slow down the server.

[whole cloud]

That could be a pretty easy way yeah.
If player wants to JIP and the queue is not done. Make them wait. If the tasks get smaller over time, we can wait till its small enough.
If they keep getting bigger, we need to slow down the server or completely pause till its done

But that's what I already described here: #perf_prof_branch message

But, I think this issue will close to never happen anyway. So I'll not invest the time into such a system

[vivid rune]

Nice, then I have it understand it know . 🙂

[whole cloud]

Normal JIP queues have thousands of entries.
You only get into problem territory when you get to the hundred thousand
So that slowdown and lagspike on JIP, should never happen anyway. But it could. But if you get to that, you server would've been running with like 5fps the whole time.
Wheras now it runs with 50+ and only lag spikes on JIP

[vivid rune]

Does the server only process the JIP queue if a JIP player is requesting it or would it prepared all the time?

[whole cloud]

all the time

[spiral pond]

i guess gathering the network statistics log of any game mode running for couple of hours would give an idea if there is a problem of too much data sent. potentially also useful to double check with CBA/ACE/other more complex scripted suites

[vivid rune]

Does only changed variables put into the queue or does the unchanged variables filtered out before?

[whole cloud]

There is only one entry per variable in the queue. If the variable is sent again, the old entry is replaced. Not another added

[heavy vortex]

If you're only building the queue on demand, doesn't that mean that isn't true for the proto-queue?

[whole cloud]

I just said its not built on demand

[heavy vortex]

At the moment, yes.

But the plan is to change that?

[whole cloud]

no

That would just make it alot worse

[heavy vortex]

Hmm, don't get it then. I figured the cost of adding a JIP entry is basically figuring whether an entry already exists.

[whole cloud]

Correct

To do that, you need to run over the queue, to see if it (or something else relevant) is already there and needs to be updated.

on demand doing that would be stupid.
That would mean we collect all JIP relevant messages, let them lay around. And then when a JIP player joins, we let them wait for a few minutes while the whole queue is being built up?

Instead of just constantly keeping it up-to-date, so when the JIP player joins, its either complete, or almost complete.

[spiral pond]

queue is an ordered array, isnt it? what is the reason for that (vs a hash map)

[whole cloud]

The reason for that is that JIP messages must arrive in order.

You cannot set a variable on an object, before creating that object

[heavy vortex]

But you're making it async by essentially adding an additional queue of updates, right? So doesn't that queue inflate if the update thread can't keep up?

[whole cloud]

Yes it does. That's what I described above.

[spiral pond]

yeah thats clear then. so it contains also all object references (and more) next to sqf variables

[heavy vortex]

Isn't there a danger that the async queue gets absolutely huge?

[whole cloud]

Yes it is. That's what I described above.

[median belfry]

Im using profiling exe with -networkDiagInterval=100 and using serverCommand "#shutdown"; but it dosent create an mpMessageDetailsServer.txt , can someone help me ?

[whole cloud]

You can end the mission first, by runing #missions, and then running #shutdown.

[median belfry]

Thank you, I will try it out

[vivid rune]

What would take more time: write into a ordered array vs. write into a hash map and write it sorted into a array?

[heavy vortex]

Uh, neither of these are good :P

You'd write into an ordered tree (red/black, whatever) and then potentially convert that into an array.

[whole cloud]

What would take more time. Doing A, or doing A and B.

[vivid rune]

Doing A and B would give the chance to parallel it maybe

[whole cloud]

I don't see how

To update the queue, you need to iterate over every entry to check if that entry should be modified.
How would having a hashmap on the side help with that

[fickle geyser]

when you're iterating over the queue not all messages are relevant I guess? eg. you setdamage on a object, you need to iterate over everything until you find (or not) a setdamage message for object with that id?

[whole cloud]

Not all are relevant yes

We could create a couple hundred extra lists, of every message that is relevant to some type. So that we don't need to iterate over irrelevant messages.
But, now we have hundreds of lists to keep updated, in addition to the one big master list

[fickle geyser]

that was kinda my idea ;d

You push setDamage object1
You take a look at the "side structure"
It gives you a list of indices/pointers to all setDamage messages

etc.

[heavy vortex]

Also they still need a central order :P

[vivid rune]

The JIP player needs only all entries. Why have the server to sort it before? Do the server only have the information that is needed for sorting? If not, let do it from the JIP client.

[whole cloud]

There is no sorting. The list is not "sorted" by something. New entries go at the end. That's all the sorting there is

[vivid rune]

ok

[whole cloud]

There is a timestamp in network messages. That could work.
But the server would have to sort it, before sending it to the client. So I don't see the purpose in that

[whole cloud]

Its fixed in next profiling build (Well the one that has the async JIP queue in its changelog)

I added #monitor data about JIP queue.
Number of async tasks to still be processed, and length of queue currently.
Testing throwing 150k variables at the server.

Previously the server froze completely for 2 minutes, that would make most players disconnect from it.
Since I added the message processing time limit, it spams "Server can't keep up, too many incoming network messages." in RPT, and runs on quite low fps for 2 minutes until its done.
Now it runs at ~670 fps for ~2.5 minutes until its done. And if a JIP joins right at that moment, it would be back at the 2 minute freeze that we had previously.

Mh nevermind.
I actually could pause JIP processing when a player joins.

Just instead of sending them the completed JIP queue.
I need to pause it, send them the incomplete one, and then send them all messages that were not yet incorporated into the queue.

That means a JIP player will spawn a vehicle that had already been deleted, set variables onto it, set its position. Then delete the vehicle again when the rest of the messages comes in.

But that only would work, if we collect all messages that trigger JIP queue changes, which we don't.
On object deletion we only store the id of the object to remove from the queue. Not the message that triggered the deletion. And without being able to replay all the messages, it would be very likely to break something.
But I could probably store them for the places that are missing 🤔

[heavy vortex]

not sure if better

[whole cloud]

But still, I think none of that is worth the effort. Because it fixes a problem, that should never happen in the first place.

If you were overwhelmed on JIP inserts. Your server would already be basically unusable. So such cases already shouldn't exist.

[heavy vortex]

What if you just pause the main thread until the async queue drops below a threshold?

Still shouldn't be worse than before, because there's a whole thread doing JIP insertions now.

[whole cloud]

I can lower the server fps. But I don't see how that helps.

If 10 players each send 10 variables per second. That's 100 variables per second.
No matter if the server is running at 10 or 100 fps

I can freeze the server completely, then there won't be incoming messages. And thats what happens on JIP join.

[heavy vortex]

Yeah I dunno. Slow server vs queue inflation and freeze on JIP?

Results are gonna be bad either way

[whole cloud]

That is the overwhelmed scenario, which I would say is VERY rare again.

[heavy vortex]

Yeah, if you have a whole thread doing the JIP queue then it should be more tolerant than it was.

[whole cloud]

low jip load:
Before: High fps server, fast jip
After: High fps server, fast jip

medium jip load:
Before: Medium fps server, fast jip
After: High fps server, potentially small lagspike on jip

overloaded jip:
Before: very low server fps, maybe even freeze or repeated lag spikes, fast jip
After: High fps server, lagspike on jip

Short large spike in JIP load:
Before: server lag spike, fast jip
After: High fps, lag spike only if JIP happens in exactly that moment

[vivid rune]

Or you tank down the server fps to the point that the JIP is every lets say 5sec possible. So it is ensure that the JIP queue is not growing more and more in a specific time.

[whole cloud]

Again, lowering server FPS, doesn't reduce the JIP load

if you got 100 messages incoming per second. It doesn't matter what the server fps is.

If you now get a queue that keeps filling up.
That means previously you would've had a server that keeps getting slower until its so slow that clients loose connection

[high flume]

Hello mr mans

[vivid rune]

Can you force the client to lower frequency of send the variables?

[high flume]

so what is this channel about

[whole cloud]

No. I cannot tell clients how they run their scripts

If a bad mod spams variables, then I can't tell the bad mod to stop

[whole cloud]

Channel descriptions generally explain what channels are about

[analog acorn]

This channel is for discussion of the Performance/Profiling branch of Arma 3, which is typically used for experimental changes ahead of their deployment to the Stable branch in a game update.

[high flume]

shit my bad

I noticed just now

[vivid rune]

But you can control the sending in the engine.

[high flume]

So... bohemia does try to truly optmize Arma 3 someday?

[whole cloud]

I could make the clients collect the messages, and not send them to the server.
So instead of the queue bloating up on the server. We now have a queue bloating up on every client

And that again, is alot of work. To solve a problem that should not even come up...

[analog acorn]

That seems like a dangerous game to play. Clients holding vital JIP messages could disconnect at any time.

[whole cloud]

Ah well. I can also hold them on the server.
Which, is already happening. #perf_prof_branch message the middle scenario.
the message processing time limit. If they take too long, keep them in the receive queue.
A queue that keeps bloating up if the server cannot get rid of the data.

[vivid rune]

OK. New attempt: What if the server or engine on client side discard variable updates that have the same value?

[whole cloud]

Player A, cannot know what value the variable has on Player B's machine.
Nor does the server know

The server could skip the JIP insert, if the value is still the same.
For that it needs to know what the "current" value is. And for that it needs to find the message, in a completed JIP queue.
Because even if current value is 1, the incomplete JIP queue might have a pending message setting it to 2. So if I ignore a set to 1, the value would then be wrong.

I could make a separate hashmap of only variables, and keep that updated even before jip queue insertions. Then I can check against that to see if value is different.
That solves the problem for one type of message. Out of hundreds of types.

With solutions like that, I'd spend days implementing special case optimizations. To try to solve a problem, that should never happen anyway.

[sick arrow]

Why cant you drop some data? Like make it weighted so some stuff like movement can be droped server side and corrected in next frame.

[whole cloud]

We are only talkin about JIP queue relevant data, and that is all data that cannot be dropped.
Movement is already being dropped

[sick arrow]

Kk i missunderstood issue

[vivid rune]

From my point of view we only theorizing about the problem. 🙂

[whole cloud]

Well I'd prefer viable theories

[vivid rune]

Every one evaluate this from an other point. If they wouldn't communicated maybe someone wouldn't have a better idea. There are no dumb questions.

[whole cloud]

I shall stop wasting my time trying to explain/argue about theories to solve a problem that isn't looking for a solution.

[vivid rune]

Oh sorry. I was not recognizing that. My apologies.

[plain trout]

I created notoriously bad JIP situations by using the WH40k mod. Even if most objects were set to local only it was impossible to join if the player number went above ~30. I should still have those mission files.

Their space ship interiors had a large part of that.

[heavy vortex]

Trouble is that the engine can't know whether an equal-value network write is required or not. Often the script should know (because sensible scripts only write a variable from one machine), and should optimize it.

But a lot of scripting is lazy and just chucks true on the end of the setVariable.

[whole cloud]

Sending a few variables lots of times, is also not inherently bad.
If the jip queue is small, processing them is fast.

Only if the queue is very large, due to some bad reason. It gets bad.

[heavy vortex]

Can you clarify the inventory thing btw? Is every inventory item change a JIP entry, or is there some collecting there?

[whole cloud]

I know removeItem's are separate messages. Even if you repeatedly remove the same item even though its already not inside the container anymore

I don't know about the rest. But I'd assume every add/remove is a separate entry

Yes you could probably make a sync point.
But I'm not messing with the inventory system, it goes bad every time I try

[heavy vortex]

What about if an object with inventory is deleted? Does it at least remove all related inventory changes?

[whole cloud]

yes

[heavy vortex]

ok, shouldn't get too bad then.

[whole cloud]

except, in some missions where you have a box standing near spawn, and you use that box as arsenal, by having a script that continuously refills it every time a player takes something out.
Run that mission for hours, with dozens of players constantly taking items from it or putting them back.. it gets a bit bad

[heavy vortex]

yeah actually we might need to workaround one item there...

[whole cloud]

One thing we could do is, if you completely clear the cargo of an inventory, we could delete all previous add/remove's of it and just keep the complete clear remaining.
But afaik there is no command to clear all, its multiple commands and multiple messages.
And people who have such long living boxes like that, won't regularly clear it to reset its state. And if they wanted to clear it, they could delete and re-create it

[heavy vortex]

clear all would be a great command if it did that.

[opal hound]

Would it be possible to coalesce into one update on join or is that not possible with how JIP works?

[heavy vortex]

Order issues, I guess.

[whole cloud]

#perf_prof_branch message

[opal hound]

I see haha

[whole cloud]

exportjipqueue command just exports all messages.
It would be nice to have a way to count how many messages there are, about a certain object.

So you can see if you have some very bad object inside your mission, to specifically optimize that one.
But everything is mission specific, and are mission makers going to optimize for it? probably not

[heavy vortex]

From the description you gave I know we have one very bad object :P

[whole cloud]

Might be interesting to look at a jip export and see what's actually the big parts in it

[opal hound]

Which builds can run it, just profiling?

[whole cloud]

Don't know 🤔
Might be

I'd say if jip queue is less than 20k, you're totally fine.
Up to 50k you maybe should consider looking into it.
Above 50k you have a problem but its not a server killer.
Above 200k you're killing your server

[kindred radish]

I once had 100k when i checked due to many containers (Dayz Style game mode)
https://gyazo.com/a65249f84e5f8bf368ddf9b8fca70ae1
The export file was 11mb with containers and 3mb without containers
Don't remember any problems that i could connect to the JIP size back then.

Similar game modes with lots of player owned containers probably easily reach 200k+ message.

[plain trout]

I’ll rerun the W40K mission for us

[vivid rune]

Would it make sense to sort the JIP variables for the JIP player by nearest location and send the nearest container first and the fares away later by the next frame?

[whole cloud]

no

At the time of sending the JIP queue, you don't yet know where the player are.
They are most likely in respawn screen, and will soon spawn anywhere on the map

[hybrid cradle]

If it's webknight, don't use it. Well known problem with webknight mods

Same goes for most WH40k mods tbh

[plain trout]

there are extra webknight mods for it

ok

[spiral pond]

any good reason for the big block of dashes?

its like 80% of the file size

[whole cloud]

Yes. That prints details for every message. Which public builds don't know

[spiral pond]

ok. the numbers are counts, no size, right? is their size similar enough that count/amount is good enough of a measure?

[empty goblet]

be sure that happens in way too many missions/mods so , it will need to be 'covered'

WL Redux with 85 players _initMessages - Total: 71371 Type_360 : 224 Type_60 : 20057 Type_330 : 172 Type_346 : 2 Type_31 : 34 Type_451 : 103 Type_61 : 8 Type_391 : 11 Type_376 : 2 Type_62 : 35 Type_182 : 74 Type_377 : 21 Type_272 : 42 Type_393 : 1553 Type_183 : 270 Type_63 : 20 Type_273 : 27 Type_364 : 209 Type_439 : 4 Type_64 : 6889 Type_349 : 2 Type_65 : 132 Type_291 : 2056 Type_187 : 248 Type_248 : 12629 Type_53 : 952 Type_54 : 3657 Type_324 : 243 Type_294 : 2 Type_10 : 37 Type_385 : 239 Type_70 : 115 Type_400 : 714 Type_295 : 1 Type_267 : 32 Type_57 : 44 Type_328 : 172 Type_59 : 20167 Type_329 : 172 _initMessagesRE - Total: 159 Type_380 : 159 _jipMarkerInfos - Total: 159

[tribal pasture]

How bad is KotH if you have it populated?

[empty goblet]

well since your 'Tanoa' experiment 😁 it wasn't populated yet

so can't answer that question, need players first

80-110 fps with 80-85 players on WL Redux ... so definitely improvement vs state before hotfix

but, take in mind, i blocked any and all cluster ammunition ... 😁

[whole cloud]

size of messages is irrelevant (besides ram usage but we don't care about that)

[empty goblet]

captureFrame with ~85 players WL redux

[whole cloud]

I can't think of a solution to handle overflow.
Two options to fix it.

1. Less messages in queue
2. Less incoming messages.

And we can make processing faster, but that doesn't fix the issue, it just raises the ceiling a bit.

We can't remove anything from 1. without breaking JIP state, so there is no option there.

The number of incoming messages, is controlled by what the players are sending.

We can reduce server fps. But that doesn't change the rate of incoming messages, it will just be fewer frames, but more messages per frame.
We can freeze the server until queue is empty. But again, after we stop freezing the queued up messages are gonna be flooding in and overwhelming us again.
We can disconnect users, so they stop sending messages.

Currently the server does all 3 combined. Reduce the server fps, so low that its basically freezing (We had this on official servers, freezing so hard that all players would disconnect), and players disconnect because timeout. Due to the disconnected players, load is reduced to the point where server can run at some framerate again.

overflow is not a new problem, its already there.

With the new async stuff. Instead of lowering fps or freezing, we increase memory usage, server might run out of memory and crash.
Or if JIP happens, we freeze, disconnecting players with timeout again.
Difference now is, we are not lowering fps, and are not freezing and disconnecting players, when no-one is JIP joining.
Theoretically, I can very easily set up a "If too many tasks queued up and overwhelmed, don't allow JIP and kick every player trying to join in". That is a simple solution but its a bit user unfriendly?
With that, all we'd have left is bloating memory usage, at less than a KB per open task. We'd have no issue handling tens of thousands.

[empty goblet]

95 players ... server fps still between 80-120 , ok this is ... better than expected (but again i repeat i blocked use of cluster ammo 😁 )

[empty goblet]

Theoretically, I can very easily set up a "If too many tasks queued up and overwhelmed, don't allow JIP and kick every player trying to join in". That is a simple solution but its a bit user unfriendly?
we have server #lock ... why not just use it to lock server until the issue resolve (then #unlock ? (or the temporary lock is unable to resolve JIP problem?)

[whole cloud]

could work yeah

Would be nice to communicate to players why its locked. But that'd be a big new feature 😄

[empty goblet]

you can write it to RPT as part of the Warning message like
Warning: Server temporary #lock , due to JIP queue overload
Warning: Server #unlock , JIP queue processing resumed

[kindred radish]

And maybe a systemchat messages, so players online know whats up / can share the info in some cases.

[empty goblet]

nope, because that might spam, same reason i don't want it in server console atm.

[empty goblet]

i knew i jinx it, down to 50 fps dips , but still holding playable

[harsh grotto]

if there will be more money than 10,000 and raise the start level, then online I think it will increase again

[naive osprey]

Is there any way to figure out what each Type_xxx ex. Type_291 is with the JIP queue?
Our mission file has well over 1000 files due to it being a Life server with many niche things, which makes it difficult to pinpoint any issues.
Getting a lot of lag spikes with Server can't keep up, too many incoming network messages. Remaining in queue: 54 since the 2.18 update, so I need to fix it.

https://i.imgur.com/nbU9yUe.png

[heavy vortex]

Trial and error. Last time I tried I didn't figure that the JIP queue was also used for engine messages so I didn't find many.

[naive osprey]

RIP. Alright, thanks.

[heavy vortex]

hmm, possibly the numbers change too...

[naive osprey]

Doesnt seem likely, as our test server seems to increment at the same frequency on the same Type_291 on init and on players joining

Unless you mean between Arma version updates

[heavy vortex]

Do you have literally no publicVariable usage in this mission? :P

[naive osprey]

Very very little

I tried to reduce as much network processing as humanly possible multiple times

[heavy vortex]

ah never mind, that's type 10

[naive osprey]

Reduced RemoteExecs, PublicVars, public setVars etc

[heavy vortex]

But yeah, I think the trouble is going to be the messages that aren't directly SQF related.

[naive osprey]

Unfortunately I think the same

[heavy vortex]

curiously my last test had no 291s at all.

[naive osprey]

Very possible its different per Arma version then, which would make sense

[heavy vortex]

makes sense. 291 is highest in Antistasi, although only ~2000 with an airfield spawned in.

hah, most of those are from CBA :P

[naive osprey]

We do run CBA due to one of our server-side addons, I wonder if that's potentially where its coming from...

[heavy vortex]

Well, CBA's only 7 per unit here.

[naive osprey]

Hm, maybe not then. We don't use any AI at all, only players. Wouldnt explain the over 30k

[heavy vortex]

addItem + removeItem spam does not spam the queue, so I guess there's some sanity in there.

[naive osprey]

Possible that items and their weights aren't determined in the same message? ¯_(ツ)_/¯

[heavy vortex]

btw you can do allVariables on an object to sanity check what's set on it.

[naive osprey]

Yeah, I figured I'd check all objects on my client while connected;

_count = 0;  
for "_i" from 0 to 6 do{  
 {  
  _count = _count + count allVariables _x;  
 }forEach (-1 allObjects _i);  
}; 
{ 
 _count = _count + count allVariables _x 
}forEach allGroups; 
_count

Results in 5473 variables so

Kinda at a loss

[tribal pasture]

inb4 its a weapon that then gets modified into item with next message 🤪

or a magazine

[heavy vortex]

yeah I dunno. Not setting vars on map objects are you? It doesn't look like CBA does.

[naive osprey]

Nope, stayed away from that as much as I could

[heavy vortex]

Uh oh. 291, 53 and 54 do not appear to be reset when the related units are deleted.

[naive osprey]

Well thats..... fun

[spiral pond]

@naive osprey only BI devs know what the id is mapped too. afaik its obscured for network protection

[heavy vortex]

actually 291 might be correct there. 53 and 54 appear to be bugged.

[naive osprey]

Kinda what I figured

[spiral pond]

you could do trial and error in a very basic mission to figure out some

i guess the best you can do is to make a ticket with the log, attaching the mission/game mode, and basic steps that may replicate an usual jip situation

[tribal pasture]

So far its only obscuring it for legit developers as cheaters have no issue finding out network messages to exploit and fake to ruin the game

[empty goblet]

mpmessage log is also good to have

[tribal pasture]

At this point I see no reason not to disclose real network message type names

Been saying that for years actually, not just at this point

[spiral pond]

i'd assume it would help decrypt the network message and thus cheaters could manipulate it again

[empty goblet]

because then you can easily see that ie WL Redux has 6114585 total setVariable objects, total SetVariable Namespaces 12358777

[tribal pasture]

They already do exactly that, decrypt messages and create their own with cheats to ruin the game

[spiral pond]

how much that type of abuse is still a problem or would become again, is something only BI can really evaluate

[empty goblet]

it's not that simple ... if it was the game would be already dead on MP side and unplayable in public MP space

[tribal pasture]

Its just us who are in the dark having to figure what causes what

2024/10/04,  7:20:59 Client: Object 9:36 (type Type_463) not found.
2024/10/04,  7:20:59 Client: Object 9:36 (type Type_97) not found.
2024/10/04,  7:20:59 Client: Object 9:36 (type Type_96) not found.
```kind of RPT spam means and trying to guess what script could've lead to that

[spiral pond]

that said, there could be also ways to provide the network logs with more specific/useful information without exposing the encryption

[tribal pasture]

Competent cheats do exactly that, with source and explanations out in the open for years, we already live in that world for 10+ years.

I don't want message structure disclosed, only message names

[empty goblet]

again you talk nonsense, if it was that easy, anyone could ruin any game anytime w/o even being in that game

in short, it is not possible easily , period

[tribal pasture]

Speaking of RPT spam, I know that Object 9:36 might not even exist at this point of log, but I wish there was some kind of history hashmap so it could be better explained in the RPT

Maybe an optional debug setting as it obviously puts some load on to the server to keep history of all network objects

2024/10/04,  7:20:59 Client: Object 9:36 (EntityAgent) (type Type_463:UpdatePosition) not found.

Something like this

[spiral pond]

would it be possible to log the actual data in the jipqueue? this would be for debugging/analysis even more helpful, right?

[naive osprey]

Would solve my current issue for future people looking into it lmao

[heavy vortex]

huh, second run, spamming FAKs did nothing to the JIP queue :P

Something else just coincidentally added 2k JIP messages at the same time? Can't imagine what.

[whole cloud]

The types are intentionally obfuscated. And the numbers changed every game update.
You can send it to me and I can look at it.
The spikes will probably be fixed by async JIP queue

[whole cloud]

Internal builds that know the structure are doing that. That's all the dashes part.
Public builds have that information removed for security reasons. Same as the names

Sometimes sleeping over a problem is the fastest way to a solution.

When a new message comes in. We need to iterate over the whole queue, and check every entry for whether the new message replaces it.
We cannot run multiple messages in parallel, because they both might manipulate any part of the array and the second message may depend on the changes of the first one.

Iterating over a large array is slow. It could be split into multiple small arrays and then the replace operation could run in parallel on every small array. But that means everytime you want to iterate over the whole list, you need to jump through all the split arrays which is annoying to write up (while I don't have access to std::ranges).

But instead of splitting the array and parallelizing the whole process.
I can instead split the process apart from find&replace, into find everything to replace and then replace them all.

Can just take the whole array, section it up. And then run the find step in parallel over the sections. Now I have a list of all messages, that need to be replaced. And just need to shuffle around the parts in-between them.

That would be easy to implement and the change is only in a single place.
The MT overhead will make it not worth on small queues. But I can just make it do it if the queue gets larger than 20k or so.

[tribal pasture]

Why not just show names without IDs?

Type_UpdatePosition

Not very useful for cheaters, useful for debugging

Actually there will have to be a table in the game anyway which they can dump

[whole cloud]

Because then you have a mapping of ID to name in the binary.
Which we used to have, which cheaters were using.

[tribal pasture]

What do you think about that idea of netid history for debugging? To help with these Object XX:XX not found errors

Have a table which stores what the objects were before they were deleted to show in error messages

[whole cloud]

Yeah, as an opt in. Shouldn't be a problem. Just a bit of memory usage

[tribal pasture]

Would love to have it, might help quite a bit in trying to figure out where that spam comes from

Could also store some debug string alongside it (class=C_Man_1) for details

[feral harness]

Personally dunno about that, the netid is an int, so when message has arrived it is decoded into object id and it tries to find the object on the client, if it is not found it says not found. All is known at this point in what message it arrived. If you want to know what object it was, the message has to contain additional information about the object, dont think this is a good idea or even possible to redo a bunch of messages to have additional info

[tribal pasture]

Store the info the moment object is created. It could've existed at some point.

[feral harness]

Store where?

[tribal pasture]

On the client

Maybe object was created and then deleted in say JIP queue on join

[feral harness]

I can't even...

you want all objects to be added to some global hashmap on the client?

[tribal pasture]

Yes, all objects that client knew about at some point during its lifetime.

Yes its gonna add overhead but the idea is to have it as debug option

And on server too for server ... not found messages

[feral harness]

you need to manage this hashmap as well when object is deleted. All because you want to know what object that was - cost/benefit just doesnt add up

[tribal pasture]

Store it there only at the moment of creation, key=netid, value=debug string.

Even this little info could shine a light at what kind of objects are addressed after they're already deleted

[feral harness]

you probably think of objects that you can see, but there are thousands of objects that have id that you dont see, inventory is one of those convoluted systems, and those missing objects are most likely inventory

[tribal pasture]

Yeah, I guess this hashmap gonna have to be addressed from a lot of places at each kind of object creation

I understand your argument that netid in the message could've never ever existed on receiving client and we'd still see nothing but a netid in the log, but who knows maybe it will help us figuring it out for objects that did exist at some point.

[feral harness]

Even having object dumped in rpt with net id when it created so you can find it in -logs is a HUGE task

[tribal pasture]

You mean finding all object creation points to add the logging?

[feral harness]

yes

[tribal pasture]

Yeah, I guess its plenty of work.

[feral harness]

cost/benefit

work is fine ^^^^

[tribal pasture]

I'd also say log into separate log instead of RPT too

I wonder how many network objects there are in the game 🤔

[feral harness]

they are all network except map and decals and some local objects

map objects have own way of syncing

the moved out objects are still network objects

like when you disassemble a weapon of move player into a vehicle

inventory has multiple network objects for the same item

player has person and unit as separate network entities

[tribal pasture]

So far doesn't sound like much object kinds to implement this kind of logging

[feral harness]

you can see how many objects are there when you see debug name of the object it shows you id and it grows rapidly as game is constantly creates deletes objects from all over the engine. There is no central function, it can create object on the fly in some method and then destroy it soon after (or not), means you have to log that as well. HUGE HUGE undertaking

[severe bone]

Thanks I will have a look to fix this with next CUPdate

[whole cloud]

Nah thats easy.
The client already has ONE big hashmap of all netID's of all networked objects. That's where it looks it up in before it prints "object not found". And all we want to know are networked objects, and all networked objects were once inserted into that map.
And there is one "Add new entry" function into that hashmap. All that needs to be done is a couple lines of code in that function to add the entry into a second map.

[whole cloud]

I've fixed up ArmaScriptProfiler's engine profiling, shall also work on dedicated servers.
Oh ye jip queue, ya poor poor thing.

running my jip flood. You can literally see how the average execution time of handling network messages, keeps growing because the jip queue gets longer 😄

You can actually see, on the average time, how full the JIP queue is 😄
The jip queue is about "1.67ms per message" full

And, on a server. You can watch live how many messages, of what type come in, and how much processing time they take. Like when I trigger a set variable flood

And the best part of it all, you can record all this in a trace file and send it to me if you have a problem

And you can see my optimization of splitting preLd into 4 parts.
Instead of doing the heavy stuff every frame, it only happens once every 4 frames. Which isn't that great for frame-to-frame consistency, but more fps

And you can see how the game does literally nothing but waiting during the 3dSwp. Valuable processin time wasted.
But usually there would be AI and sound there, I'm just on VR terrain so there isn't anything going on

[whole cloud]

And the AI optimizations. Its not very good. Main thread is waiting on AI to get done.
AI is waiting on 3 path finding operations. But atleast the 3 path finding operations run in parallel now.

But overall, performance in YAAB is extremely bad, probably because I run profiler in debug mode :U
Its probably time to optimize the profiler, I can reduce the overhead by quite a bit.

I missed this profiling so much. Being able to see multiple frames in this detail, and have statistics between them is so nice 😄

[silk pewter]

https://tenor.com/view/dumb-and-dumber-comedy-jim-carrey-excited-happy-gif-10308926303665370357

[knotty wraith]

explain if I launch the game with the parameter -
-checkSignatureFull, what should I get?

Besides the fact that the game takes a long time to start, I don’t see anything else

17:22:15  > (8:21:39) cba_versioning - zen - Version Mismatch! (Machine: B Альфа 1-4:1 (Joker) (Joker) version: 0.0.0, serverVe
17:22:15  > rsion: 1.15.1.36, Level: 4)
17:25:08  > Player Joker: Wrong signature for file D:\Steam\steamapps\common\Arma 3\expansion\addons\data_f_oldman.pbo
17:25:08  > Player Joker disconnected
17:34:52  > Joker uses modified data file
17:34:55  > Player Joker connecting
17:36:29  > (8:35:54) cba_versioning - zen - Version Mismatch! (Machine: B Альфа 2-1:1 (Joker) (Joker) version: 0.0.0, serverVe
17:36:29  > rsion: 1.15.1.36, Level: 4)
17:38:30  > Player Joker: Wrong signature for file D:\Steam\steamapps\common\Arma 3\jets\addons\ui_f_jets.pbo
17:38:30  > Player Joker disconnected
17:52:32  > Player Joker connecting
17:52:35  > Player Joker connected
17:54:02  > (8:53:27) cba_versioning - zen - Version Mismatch! (Machine: B Альфа 1-4:1 (Joker) (Joker) version: 0.0.0, serverVe
17:54:02  > rsion: 1.15.1.36, Level: 4)
17:55:04  > Player Joker: Wrong signature for file a3\vegetation_f_enoch\tree\t_piceaabies_3f.p3d
17:55:04  > Player Joker disconnected

I asked the player to enable the parameter -checkSignatureFull, what next?

[analog acorn]

Is this actually a perf/prof-specific issue?
It looks like the player just needs to load ZEN and verify their game files.

[knotty wraith]

this is spam zen mode - no more

[empty goblet]

Dimon i told you what to do, ask the players to generate checksum of those files and compare it with checksum on your client

[knotty wraith]

you understand, they don’t even understand where the logs are, what checksum on your client we can talk about (

[empty goblet]

most of times it's corrupt files (pbo or signature) or processing the files gets corrupt (controller, bus, cpu, memory etc.)

networking issue is the one of last possibilities in the chain

[empty goblet]

install, adds new tab into file properties (right click on file), named Checksums
https://github.com/gurnec/HashCheck/releases/tag/v2.4.0

[spiral pond]

nothing if fine, error if not - result is logged to rpt

[uneven kernel]

ASP is currently hidden on WS, assuming its the EULA thing again

[whole cloud]

aaaa

It was! thank you 🫂

[empty goblet]

hmm, anyone else noticed that server console log simply vanish sometimes when server crashes ?

[whole cloud]

And looking at the code messes it up again.
There are two messages, where the new message to be added to the JIP queue, is changed while iterating over the old messages.

For example when someone does damage to an object, the new message accumulates the numbers of all old messages and combines it into one.
That means there always should only be one damage message per object in the JIP queue. Because as soon as a second one comes in, it gets combined and we end up with one. That's hopefully fine. 🤞?
If I split it, and it somehow ends up finding two messages, and produce different new accumulated numbers, we'd be messed up

And really nothing stops anyone from adding a new message in there, that actually accumulates multiple messages together, and then this just completely doesn't work

[whole cloud]

Also about the inventory and box stuff.

ClearItemCargo, removes all previous Add/RemoveItemCargo messages from JIP.
Same for ClearMagazineCargo/ClearBackpackCargo/ClearWeaponCargo

so if you're using add/removeXCargo on a box, clearing it from time to time can also clean up the JIP queue

[feral harness]

Cargo box JIP is not just current box content?

[whole cloud]

No there are no accumulated "these are the container contents" messages

[feral harness]

So whoever joins gets thousands of messages manipulating local container before it syncs?

[whole cloud]

yeah

[feral harness]

thats mad, but can be improved

[whole cloud]

Good luck messing with the inventory system.
As I wrote before, I don't want to attempt messing with it

[feral harness]

inventory changed event can just update JIP

I bet this is the major server hog on long running mission

[whole cloud]

I have data on long running missions that I will look at probably on friday

[feral harness]

when is the next prof?

[whole cloud]

Soon-ish probably

[whole cloud]

And multithreaded JIP queue handling is a go
Close to full CPU load on all threads 😄

ingesting 150k variables now takes 43.7 seconds, while the server runs smoothly at 500fps.

The last insert takes 900us

Here comparison to how it looked in 2.16
3 minute server freeze. All on one CPU core.
The last insert takes 3.42ms

And here is friday.
Async processing takes almost 4 minutes (The worker thread has "below normal" priority)

Also the "Total time" it shows, how much we in total spend on JIP queue updating.

2.16: 2m56s
Friday: 3m53s
Today: 43.4 seconds

I think that's enough optimizations for now. As long as this isn't broken in some way 🤞
I can see that the total count of JIP updates are different.
I throw 150k messages in, old had exactly 150k executions. But new is at 145k or 125k.. but I assume that's just the profiler messing up somehow..

[shrewd moss]

What are jip if I may ask? Are there some everyday user cases that can benefit this optimization?

[whole cloud]

"Join in progress". When a player joins into a already running mission.
Every server runs this stuff, all the time

[feral harness]

Well, luckily we also do not guarantee the order of jip calls The order of persistent remote execution for JIP players is not guaranteed, i.e. the order in which multiple calls are added is not necessarily the order they will be executed for joining player.

[analog acorn]

That seems.....bad

[feral harness]

This could be exclusively remoteexec related

[whole cloud]

RemoteExec has a separate queue

[silk pewter]

yet example 8 shows this

ah, but for JIP players

[feral harness]

yeah jip is not guaranteed

if it is the same message jip will be the last version of it. if different messages then no order

[heavy vortex]

My recent issue is more that guaranteed messages aren't guaranteed :P

Is there any order guarantee for other JIP (eg. publicVariable) vs remoteExec JIP?

[feral harness]

repro or didnt happen

[whole cloud]

main JIP queue is sent before remoteExec queue

[feral harness]

@silk pewter ^^^ useful to add on biki

[silk pewter]

ye, Multiplayer Scripting I think

[feral harness]

no into remoteExec note, there is one

[silk pewter]

Discord definitely lacks Slack's "save message" feature

[empty goblet]

they added bookmarks, save may come , ironically they can just re-use the interface for report message 🤣

[empty goblet]

is JIP queue stuff cleaned when inventories are removed (ie destroyed vehicles , deleted boxes, player leaving, dead AI via GC ops ?)

[whole cloud]

Yes

[whole cloud]

Bookmarks are nitro only feature

[spiral pond]

              LODShapeRoad: requested     26695200, allocated     27893760
      GameInstructionConst: requested     21688240, allocated     21991424
              GameDataCode: requested     18150424, allocated     18853888
   GameInstructionOperator: requested     12352512, allocated     12550144
   GameInstructionVariable: requested     11559488, allocated     11747328

GameInstructionNewExpression: requested 11260512, allocated 11444224

noticed this in rpt with profiling exe - is this from a #debug XXX mode, or done automatically?

PS: LODShapeRoad: requested 26695200, allocated 27893760 this huge amount of allocation cant be good/normal, is it?

[whole cloud]

This is printed at captureframe

[whole cloud]

You tell me.
Is 30MB huge?
Well I don't know how much RAM you have. But for me it's not.

[spiral pond]

well its the first/top item from that list. if its normal/within reasonable limits, then fine

SPE Normandy with low cell size, high object density is very demanding. maybe the road network/setup is part of that

[restive pilot]

Is it a buffer or something?
If not and it's per model, then 30 MBs can be huge if you consider 1000 models in a mission (or even just 100 of those huge models)
Tho I have no idea if this is even relevant because I have no idea where that message is coming form 😅

[spiral pond]

profiling.exe + #captureFrame in chat

[whole cloud]

Its a total, its the whole allocator

[median eagle]

Is there an announcement channel here that can be subscribed to when there is an update to the profiling branch?

[autumn timber]

Not really, but you can use from: dedmen. in: perf_prof_branch new profiling branch in the search box to find these

[opal hound]

Or pinned messages

[autumn timber]

Actually... yup, that would be the correct solution 😂
🤦

[feral harness]

unless ded forgets

[median eagle]

Hm. So my steam based arma3 client is currently on 302. is that the build for the 2.18 hotfix version or is that the 2.18 release version?

i noticed both my server.exe and server profiling.exe are on 307 on my dedicated box.

[light cargo]

validate the game if you're unsure

[median eagle]

I found the build numbers on the feedback tracker

[whole cloud]

You could subscribe to the forum thread.
I think forum might even have a per thread RSS feed

[median eagle]

I would need a forum account for that lol

[empty goblet]

i can see expectations are high 😁

[median eagle]

Other question, is the built version of a connecting client visible in the server rpt?

[whole cloud]

Not for the RSS feed

[whole cloud]

No

[gritty wasp]

Why not just use Steam?

Will we have ultra mega parallelized perf this week?

[autumn timber]

I'm assuming that it's for the server side, so you don't have the steam client there that is constantly running in the background + the server needs to then be restarted

[whole cloud]

Someone please post the :panik: emoji for me 🥺

[somber plank]

[knotty wraith]

https://media.istockphoto.com/id/164414495/vector/panic-emoticon.jpg?s=612x612&w=0&k=20&c=GCvh-tOWTbX1oHsyG4zX1414a9AC4wmc74hQ8BcJE5U=

[silk pewter]

😄

[whole cloud]

[restive pilot]

What'd find this time?! 😅

[autumn timber]

Which way are the eksploushyns?

[whole cloud]

2.18.152340 new PROFILING branch with PERFORMANCE binaries, v6, server and client, windows 64-bit, linux server 64-bit
- Added: (Profiling-only) Capture Frame dialog - Shift-clicking the export button now exports in Chromium tracing format
- Added: (Profiling-only) Continuous capture for diag_captureSlowFrame
- Added: (Profiling-only) SFRAME cheat code, which captures a frame longer than 20ms
- Added: (Profiling-only) Logging captureFrame to a file now also outputs a Chromium tracing format file
- Added: (Profiling-only) -networkDiagInterval now also logs size and count of publicVariables
- Added: Replaced Real Virtuality multithreading system with Enfusion's version
- Added: Asynchronous JIP queue processing
- Added: Multithreaded JIP queue processing
- Added: Multithreaded lights collection
- Added: More multithreading into rendering
- Added: Asynchronous AI and sound simulation
- Added: Partially multithreaded AI simulation
- Added: Multithreaded AI visibility calculations
- Tweaked: Improved PhysX simulation performance, if many PhysX objects are in close proximity
- Tweaked: Improved building simulation
- Tweaked: Buildings with lights no longer get faster simulation if they are not animated and static
- Tweaked: Visibility checks when applying explosion damage are now multithreaded
- Tweaked: (Profiling-only) Capture Frame dialog - Improved handling with many threads
- Tweaked: Improved server performance when sending a large JIP queue
- Changed: (Profiling-only) Capture Frame dialog - Tooltip now displays time with microsecond resolution
- Changed: (Profiling-only) Executing diag_captureSlowFrame on a dedicated client/server now writes the captured frame to a file
- Fixed: (Profiling-only) Capture Frame dialog - Horizontal scrollbar was unusable
- Fixed: (Profiling-only) mpMessageDetails would not be written on #shutdown shutdown

- Fixed: UAV would stop following waypoints after a land waypoint
- Fixed: supportInfo command did not list all commands on "u:" and "b:"

Note: while these changes have been tested internally, there are still many possible scenarios where things could break. They should be considered experimental. Please report all game crashes, new graphics glitches, and new cases of AI misbehavior.

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[whole cloud]

Usually changelog is late, today its early 🤣
Please consider that these are massive changes, you might want to only slowly adopt this. Don't throw this on all your servers immediately, I'm a bit scared of it.
I did test all but JIP in a big MP mission, and the JIP stuff by myself on a local server.. shooould be fine.
But especially also with mods there could be issues that I didn't find.
When you notice something sus, please report it quickly. We will consider reverting this before the weekend if its too flaky.

[silk pewter]

• Added: More multithreading into rendering
  see? just optimise!

[void badger]

Would many of these non-profiling changes work if only on the server and clients still on Stable? I.e. drone fix?

[whole cloud]

drone fix is where the drone is local/simulated. Which might be server, or some players machine
The performance stuff is also all local stuff

[void badger]

You rock

[whole cloud]

You meat

[void badger]

https://cdn.discordapp.com/attachments/1121145124592816140/1300945071989850185/homelander-angry.gif

[quaint flame]

My brain automatically read that

[loud garden]

• Added: Replaced Real Virtuality multithreading system with Enfusion's version
  Is A3 being migrated to the new engine? 😁

[whole cloud]

No

[knotty wraith]

disappearing magazines have made a revolution

[somber plank]

?

[knotty wraith]

https://drive.google.com/drive/folders/13eZ-AbXWm34noN2AY8D3JYLgxeoyV0az?usp=drive_link
v5

[somber plank]

"Usually changelog is late, today its early"

[whole cloud]

As I said. Changelog is early

Probably 20 more minutes

[knotty wraith]

v6 is not available

[somber plank]

no shit

[whole cloud]

Had the choice of posting 30 minutes early or 1.5h late. So here we are >:3

It's up now.
32 minutes 😄

[silk pewter]

https://tenor.com/view/the-lie-gif-11500229991946522721

[rose moth]

Someone is very lucky the pipelines didn't fail on them

[whole cloud]

https://tenor.com/view/believe-motivation-ted-lasso-jason-sudeikis-trendizisst-gif-7491964667140939589

[fickle geyser]

MULTITHREAD

[empty goblet]

⏱️ waiting for first benchmark compare pics / data🤣

[empty goblet]

• Tweaked: (Profiling-only) Capture Frame dialog - Improved handling with many threads
• Changed: (Profiling-only) Capture Frame dialog - Tooltip now displays time with microsecond resolution
  2x2 separate entries or 1x2 ?

[whole cloud]

One.
There were so many for that Dialog that I improvised a bit

[empty goblet]

TLDR changelog , so rare , totally legendary 😁

[harsh grotto]

https://tenor.com/view/yay-kitty-cat-happy-excited-gif-3804551669601991442

[tribal pasture]

[somber plank]

[fickle geyser]

now somebody put it on big public server and see what breaks

[heavy vortex]

new cases of AI misbehavior.

Good luck separating the new AI misbehaviour from all the old AI misbehaviour.

[rose moth]

That's how we get away with it 🕵️

[harsh grotto]

+13 FPS in YAAB

https://tenor.com/view/catshakira-cat-dance-catdance-catdancing-gif-22503034

[night mulch]

https://media.discordapp.net/attachments/694394891584667683/885935014058000384/image0.gif

[spiral pond]

tracy export froze my text editor as it has no spaces - aka a huge long line. make sure to have word wrap active when you paste

[harsh grotto]

it was 30-31, now 43 on the first try

[spiral pond]

basic test with SPE+DS+Combat Patrol

[whole cloud]

That oPrep is crap.
But the big part in it is probably sceAC and oSplt? Which I haven't yet found a way to deal with

I didn't test any of your SPE tickets yet btw. Maybe next week

[spiral pond]

aiGrp probably useful to check - two of those are logics 🤯 (~0.7 ms)

will be also interesting to compare client vs server computation for those

atm only one group is local to the client. rest on the server

[merry shuttle]

FPS are basically the same, my dedi server crashed with extDB3, without it it was loading fine and no problems.

[spiral pond]

the file is on SSD. so strange to have such long load time i guess

this is also curious. 0/0 vehicles but 1 ms on px3buoy

[somber plank]

crashes only on prof with extdb3?

[spiral pond]

overall child coverage by profiling scopes is pretty good. usually 95%+

[merry shuttle]

I've removed extDB3 from the server batch file and it loaded fine and worked ok with CUP, RF and GM loaded. On vanilla extDB3 works

[spiral pond]

only potentially relevant exceptions from a first check

[somber plank]

so extdb3 on prof = crash
extdb3 on default/vanilla = fine?

[spiral pond]

sframe cheat works 👍

[hallow lantern]

from 44 to 55 on my potato notebook

[vale shoal]

what extdb3 version do you use?

[silk pewter]

I must say: jackpot, and good job!!

[merry shuttle]

I'm happy for you and I hope it helps let's say less powerfull PCs

[inland dew]

actually no
perf/prof crashes with mods
perf/prof no crash without mods (vanilla)
vanilla = no mods, vanilla ≠ stable branch

[spiral pond]

bgD3D seems also interesting to look into (if possible) - what it does there. not covered by subscopes

[knotty wraith]

• 7 FPS

[whole cloud]

You can't really read these anymore. Because they are coroutines

[whole cloud]

Crash report into my DMs

[merry shuttle]

I think it is v1033

[somber plank]

@merry shuttle this is for you btw

[whole cloud]

Crash reports into my DMs

[whole cloud]

Not possible

[quaint flame]

holy cow

Game crashed. Dump file 14 gb

[knotty wraith]

18:54:31 "---------------------------------------------------------------------"
18:54:31 "---------------------------------------------------------------------"
18:54:31 CallExtension loaded: extDB3 (D:\Games\ArmA3\A3Master\@extDB3\extDB3_x64.dll) [1.0.3.1] [1.0.3.1]

18:54:31 "extDB3 Loaded"
18:54:31 "---------------------------------------------------------------------"
18:54:31 "---------------------------------------------------------------------"

```seems to work

[inland dew]

not me. we're on vanilla (no mods)
was commenting a response to @merry shuttle post that says it crashes with mods, when using extDB3

[merry shuttle]

The only mod which crashed the server was extDB3. But extDB3 uses CallExtension or is an extension.

[knotty wraith]

my processor was loaded from 35% to 50!
fps in mp confirmed + 7

• 15% on the processor - chic!

[spiral pond]

rendering seems a lot better MT now if i am not mistaken

can sceAC get multithreading (it seems not [yet]), or not possible?

Ger boots seem expensive there for some reason 😛