perf_prof_branch

[hearty haven]

if you need completely rewrite shitload of code just to allow some arma component to communicate between more than 256pl. thats understandable, im ok with that answer
if you can rewrite single line of code to allow some nerds to play according their preferable style of fetish, but you won't, i don't get it..

i thought that #perf_prof_branch is most suitable variant to test such features, and you can switch in-between just with replacing .exe file, no one in community will download 40gb+ dev branch arma just to test "small" event

[worldly badge]

Definitely not the latter case. If a Dev says no that's a no

If you don't get it, you're saying like "gib Arma 4" so that the answer is that

[hearty haven]

@worldly badge
sorry if im asked impolitely(eng is not my first language ofc),
im referring w/this context
https://i.imgur.com/rbmU8lf.png
#hardware_vs_arma message

MasterofBlub guesses there could be huge rework, also might be single line
Dwarden said there needed to be some work done like network optimization etc..,

New CPU's came out for datacenters, few arma patches, and so we could test it

[night mulch]

Your best bet would be to make request feature ticket at the feedback tracker

[whole cloud]

You asked if I can.
I said no.
Meaning no I can't do that.
What else do you want?

Pinging Dwarden just means Dwarden would then have to ping me, just so he can get a no from me again.
Thats why I told you to poke noone. Because it all ends up at me and I will say no.

I rather find it rude of you that you ask me a question and then don't accept no for an answer

[restive pilot]

I'm sure it's not a matter of "just a single line"
But even if it was, what's the point? Most people can't even play on a 50 player server without getting bad performance. Optimizing the game for those people is more worthwhile than increasing the player limit and dealing with nonsense feedback regarding performance and bugs at those limits, which is irrelevant to 99.9% of players that play Arma
Also let's not forget that Arma 3 is EOL. You can't expect major changes to gameplay

[whole cloud]

profiling at the top edge is actually very useful to find the bottlenecks.
But for that 150 or 200 players are already sufficient

[hearty haven]

just to be extra clear - perfomance we get on 245pl was extra surprising(for server and players),
everything worked just like 180 players mission, (even with BE+ace+rhs+community mods, trick is might be that no Ai bots or headless clients and:

params ["_x"];

if(_x in playableUnits) then
{
    _x setBehaviour "CARELESS";
    _x allowFleeing 0;
    _x disableAI "AUTOTARGET";
    _x disableAI "PATH";
    //_x setCombatMode "BLUE";
    _x disableAI "MOVE";
    // _x disableAI "CHECKVISIBLE";
    _x disableAI "COVER";
    _x disableAI "FSM";
    _x stop true;
    _x doWatch objNull;
    _x disableConversation true;
    _x setVariable ["BIS_noCoreConversations", true];
};

Server fps ~30 - 40, players limited view 2500, my rig 5600x+1080ti able to produce 70+fps

(we spread hostile sides around 1/2 of Malden ofc)
When 256 players appeared, game crashed in very unusual place
(loading from lobby to briefing)
which imo way less consuming than
(loading from briefing to live game and then de-sycning after)

[whole cloud]

The game shouldn't even let 256 players join. There should be a hardcoded player count limit to not let you have that many players in the first place

but I guess that limit is something that QA cannot easily test 😄

[hearty haven]

you, BIS devs, modmakers, whole community actually make a very huge leap forward👍🏼
Just look at how game performs
2017 250 test
https://youtu.be/hmAoAwWaMi4?t=32
2021 240 usual gameplay scenario https://www.youtube.com/watch?v=vxOip_-KLsE
https://www.youtube.com/watch?v=gmprMeOAUPI&t

also i remember back in ~2012 we first time crossed 200 players on arma 2, everyone was so excited. Times fly 😖

[whole cloud]

2.06.148670 new PROFILING branch with PERFORMANCE binaries, v7, server and client, windows 32/64-bit, linux server 32/64-bit

• Added: TimeLimiter for heavy AI tasks that could freeze servers
• Tweaked: Improved performance of triggers with Present/NotPresent activation
• Tweaked: Improved performance of network message encoding
• Fixed: Locations search in Eden Editor was not working as expected - https://feedback.bistudio.com/T161001
• Fixed: Sound occlusion from createSoundSource/say3D would not update when switching between 1st/3rd person while inside a vehicle - https://feedback.bistudio.com/T131341
• Fixed: say3D command would always disable sound attenuation (introduced in 2.04)
• Fixed: On-screen script errors were only displaying "Generic Error" messages
• Fixed: Script errors would be printed to RPT twice (once correct, and a second "Generic Error" message after it)
• Fixed: Eden Editor cut operation with an object under the cursor would copy all selected units, but only delete units under the cursor
• Fixed: Potential crash when using Regular Expressions in script

[empty goblet]

quite sure my answer was clear as it's way better choice to make the MP very enjoyable at values of in range of 64-255 players than trying pursue edge case of maxplayers past that boundary ...

[silk pewter]

0..65535 next step 😄

[rain moth]

is TimeLimiter something configurable or just a hard limit?

[whole cloud]

hard limit

[rain moth]

good, then people won't fuck that up 😄

[restive pilot]

heavy AI tasks
such as?

[empty goblet]

if it was custom (e.g. in advanced server config) it would need to be very sensibly defined from - to range but it was simpler to make it hardcoded, otherwise it would end as you said *up

[empty goblet]

it was discussed already before in the 148592 build i shared here some weeks ago #perf_prof_branch message

[fickle geyser]

if mission / mods got lot of triggers (both in sqf and fsm), there should be noticeable improvement too
not sure what that means. Triggers are "objects", how they're related to SQF/FSM.

[quaint flame]

@whole cloud Is it possible that the Eden search fix didn't make it?

I am on 2.06.148670 and the issue is still present.

Perhaps he means triggers created via fsm or sqf scripts

[whole cloud]

That was merged

but I think there might be a script fix part to that too?

yes its a combo engine+script fix. This is the engine part.
Shouldn't have mentioned that in changelog

[quaint flame]

Okay. 🙂

[stark falcon]

Still getting crash when using 32 bit profiling exe; is it just me?

[tawdry gazelle]

Strange... I got someone report 64 bit profiling crash on a 64 bit system, but I guess he messed up some system dlls. And he does not know how to find a rpt file, even I recorded a video for him...

[whole cloud]

get me crashdump

[plain trout]

dedmen I wonder if it would be possible to override the windows fullscreen optimization windows introduced with the 1709 creators update. I had 10+ people deactivating it and having massive performance gains like almost no stuttering anymore and way better and more stable fps

I recognised the bad_module_info in the windows event display and when I looked it up it recommended turning full screen optimisation off

[restive pilot]

I don't think it can be controlled by the program.

[mellow lark]

Or, make it so Arma runs fine with it

It should in theory benefit from it

Do you have any numbers though?

[plain trout]

one guy is reporting 10fps

and my game is way smoother

like way way

[mellow lark]

fps is meaningless

if we dont know baseline

so either give the baseline or use %

[plain trout]

I know, but the stuttering and experience feels better, it is KOTH

so thats 25 to 35

[mellow lark]

can't test for myself atm. but that's interesting

[plain trout]

we also played in my milsim unit I had fluid 30fps for 2 hours while I usually have frame drops

and some people that crashed had no more crashes

[mellow lark]

It depends quite a bit on what they have running in background though

[restive pilot]

not really. it either makes no difference or makes it slightly worse

[plain trout]

try it out.. you should "feel" it instantly..

[mellow lark]

idk I don't play that often anymore so not sure if I would tbf

[tawdry gazelle]

Do you mean force enable fullscreen optimization on Arma 3 gives much more better performance than leaving it disabled?

[plain trout]

I think windows forces it on everything ?

[restive pilot]

No. The opposite

[plain trout]

So you must force turn it off

[tawdry gazelle]

Yes, I remember windows force it on so we need to toggle it off...

okay that make sense... So it mean Arma 3 in Fullscreen gets more fps than Windowed?

[plain trout]

A MilSim mate said that the „optimization“ is achieved by increase the process priority for CPU and GPU. So that probably even works in windowed

And mirrors my experience because I only play in borderless window mode

[tawdry gazelle]

You means windowed with fullscreem optimization on has better fps than windowed without fullscreen optimization?

[plain trout]

No. Way worse and on top it gets unstable

Oh wait

I read that wrong. I usuall have the same or slightly lower fps in windowed, but it allowed me to tab out *faster so i prefer that

[tawdry gazelle]

I thought windowed mode wont gain anything from fullscreen optimization... As fullscreen optimization makes fullscreen games go through DWM composition, while windowed games always go through DWM...

[tawdry gazelle]

Ah, me the same 🤣

[plain trout]

I have no idea. It is just what I recognized after changing it. Haven’t run YAAB yet either but I might later today

General gaming was much much better

What I read was, that the bad_module_info affects a lot of steam games so maybe it creates some issues with the steam overlay

[tawdry gazelle]

I have never noticed a dialoug about bad_module_info... But DWM does cosumes lots of resources and may lead to fps drop. MS's typical "optimization"😅

[plain trout]

me too. I only found them in the event viewer because I check it if my overclocking is making any problems

[inland dew]

@plain trout what is your monitor refresh frequency?

[plain trout]

60Hz

I‘m just starting to do yaab comparisons and it doesn’t have any impact here

[inland dew]

Then put 60 in arma config in your documents and full screen (no window) will work just as fast to alt+tab and slightly better performance

Refresh

[restive pilot]

there's also the resolution. if the resolution is not the same as desktop alt+tab will still be slower than windowed

[plain trout]

Everything is at 4k

[inland dew]

Just change the refrrsh to 60 in arma config and enjoy full screen alt+tab

[plain trout]

So something did change my performance from 45 to around 60fps in yaab. But I cannot reproduce what creates the 45fps issues.

[restive pilot]

a background process?

[plain trout]

I’ve tried fso on and off, 4K vs 1080p (10-15%), modded (another 10% loss in both resolutions with rhs) VS vanilla

Looks like to many things changed since I recognized the changes… let’s see what happens if I enable onedrive again

[plucky lichen]

on 2.06.148670 64bit server
server cfg
{ LogObjectNotFound = false; SkipDescriptionParsing = false; ignoreMissionLoadErrors = true; };
server rpt
Error: Object(31 : 17085) not found x 1000's@whole cloud

[whole cloud]

https://community.bistudio.com/wiki/Arma_3:_Server_Config_File

LogObjectNotFound - False to skip logging "Server: Object not found" messages

Server:
@plucky lichen

[hexed gorge]

Can an option be made to turn the grass on and off that is independent of the quality of the terrain?

[whole cloud]

no, this is #perf_prof_branch

[hardy pollen]

Does the mod "Arma 3 Profiling Branch Companion Mod" affect servers that are not running it and not using the profiling branch in any way or is it strictly a client-side modification.

[autumn timber]

AFAIK it's 100% client-side

[rain moth]

should be some sqf(c) code used by server in there as well?

[autumn timber]

yes, but I read it as "you're only affecting yourself by using it, not anyone else, or the server you're playing on", which is true

[empty goblet]

the companion mod is on all zeus offical servers via servermod btw.

[hardy pollen]

Question, has anyone experienced constant desync on modded servers, (all players from 2 to 6 desync) and sever rubberbanding between player actions and AI, on servers running the performance branch server exe?

Or is it doubtful that the new exe is the cause of it?

[nocturne obsidian]

Hey just wondering is there a way to switch between the performance/profiling executable inside of the arma3 launcher when running the profiling branch on steam? I saw that there is an extra "Profiling" ribbon on the launcher but clicking it doesn't seem to switch between performance/profiling executable

[whole cloud]

no

rename the exe

[nocturne obsidian]

Yeah that's how I've been doing it so far, no worries just was wondering if there was another way

[whole cloud]

The joys of linux land

[silk pewter]

wait
you need to run -nosound for the server? isn't it a perf saver right here 😄

[whole cloud]

So for the people to ask if linux performs better than windows..
Some people say its roughly on par, others say its worse..

I say... I noticed yesterday after a performance fix that we had a linux build error. In code that I expected not to be compiled at all.. It was code for when multithreading is disabled.. that doesn't make sense does it?

Turns out, MicroJobs (our most important multithreading thing) was disabled on linux.
I set the 0 to a 1 (and added thread naming because nice to see) and boom MJob threads.
Coming to you on #perf_prof_branch early next week unless something goes wrong over my weekend testing

[bleak garden]

What would we do without you?

[quaint flame]

What have we done all the years without you ? 😄

[silk pewter]

what did you do all the years without us?

[whole cloud]

Well my linux server just crashed while testing.. I'mma fix that 😄

From recent profiling data I got from the RocketBeans Arma Nacht team (Big thanks! Its so hard to get good data) I saw a performance bottleneck (gonna be fixed next prof) when windows server had all its MicroJobs being busy (it uses that to calculate which network objects to send updates about, its just much faster multithreaded but even at that point it can still be a bottleneck if the server doesn't have enough threads assigned, Arma servers don't only use 2 CPU threads people! They use all of them).

And now I learn that linux does all this in a single thread, and still seems to be about as fast as windows builds despite this huge handicap.

[whole cloud]

All the names! I think its better for server owners to see whats going on. If physx thread is at 100% you know something is bad

[hexed verge]

fancy!

[whole cloud]

Daang, thats hot 😄

[hexed gorge]

What are these images showing?

[whole cloud]

High CPU load on worker threads.
Meaning the time that was previously spent in a single thread and slowing everything down, is now done in 11 threads on the side

[autumn timber]

Does that mean that this brings the linux server "just" on par with its windows counterpart or is there a chance to be able to boast something something linux master race?

[whole cloud]

Well people said it was already on par, now it should be much better than before.. so.. maybe it will already be better then? 😄

[autumn timber]

I'm definitely not complaining 😛

Especially that linux boxes are usually cheaper to rent 🤷‍♂️

[whole cloud]

With this enabled in my big AI test server runs at 7-14 fps. and the CPU load looks like above.
With this disabled it runs at 2-5fps and CPU load looks like

This is pretty huge..

[crude depot]

390% CPU usage instead of 101% - not bad

[whole cloud]

Why did I not see this before ┬─┬ ノ( ゜-゜ノ)

[autumn timber]

inb4 "with the latest perf prof branch, the linux server uses much more cpu, please fix!"

[whole cloud]

This is 390% cpu usage on a 6 core 12 thread system.
So it wouldn't get much higher than 600%

[crude depot]

the hosters will hate you 😄

[silk pewter]

"meh, not worth it!"
joke aside, it's heartwarming to see "old" A3 aspects still see some love 🙂

[autumn timber]

Previously, I could fit 6 instances of an arma server on one physical machine before maxing it out. Now it's only two instances; please fix!

[whole cloud]

use -cpuCount you stone age person !!111!1elf11

[autumn timber]

By the time Enfusion is out, it will not make sense to use Enforce script over SQF anymore ¯_(ツ)_/¯

[whole cloud]

I should do that too.
It thinks it has 12 threads, but its only a 6 core 12 thread system. I should pull that down to 6 probably 🤔

[hushed seal]

Only if you use the hoster provided windows license.

[autumn timber]

Especially that legal linux boxes are usually cheaper to rent 😁
There, fixed 😄

[empty goblet]

@whole cloud on linux (and other platforms) you could use
https://www.open-mpi.org/projects/hwloc/ api
to properly detect available native cores vs logical, numa groups and other architecture irks ,
ofc has github too https://github.com/open-mpi/hwloc with friendly BSD license

[hexed gorge]

I need to learn Linux! 😨

[whole cloud]

But that bottleneck was caused by having MANY objects on the terrain.
And @kindred radish's bugfix will potentially fix a large part of that too because all these objects also count into it.
But for that fix to work, the clients need to run profiling branch too, not only the server.

So if you run a server, consider asking your players to switch to profiling branch.

[restive pilot]

Won't that fix make it to 2.08?

[whole cloud]

no

oh wait

no yes, it will 🤣

been a long day

perfect time to fiddle with core networking performance stuff

[torpid river]

specifically profiling exe or just the profiling branch with performance exe? We could use steam api to force branch on clients with our launcher

[whole cloud]

performance exe, the default on profiling branch

┬─┬ ノ( ゜-゜ノ)

I was working on networking performance.
While profiling I found a kinda huge issue in vehicle sound simulation.
Made a one line change, and now my fps are from 15 up to 21...
And it messed up all my results from before so I have to rerun the benchmarks now!!! grrr

(╯°□°）╯︵ ┻━┻

[random isle]

what do rest of your team does 😄

[restive pilot]

What team?

[fickle geyser]

The question is.. what the team was doing when there was actually a team working on the game...

[whole cloud]

With 4180 (empty) quads in VR.
The sound simulation time per frame went from 43.2% to 23.7% with a single line change :wobcat:

https://cdn.discordapp.com/emojis/795600632500191243.gif?size=256&quality=lossless

[random isle]

i'm still hoping you will fix the remote object thing some day, the AI that walk with invisible gear / or when you hit them with a car it fully stops your car or sends it to moon

[whole cloud]

I don't know about that

[random isle]

hmm there are ton of posts about it thought you knew

it happens on HC in my case

they also refer this as naked bug

[whole cloud]

ah the naked bug I know. But afaik there is no repro for it

[random isle]

shit i will make you one

in 2 min

actually i already have one that was made for HC bug years ago i bet i can still use that one 😄

[fickle geyser]

Looking at how effective dedmen is I can't wait for the day he'll start looking into the AI

[restive pilot]

He did tho

[random isle]

it might be something as bumb as changing from 0 to 1 some where in engine lol

[whole cloud]

We can't do AI behaviour fixes anymore if you mean that

[fickle geyser]

;/

[quaint flame]

We need a new arma branch without the limitation of backwards compatibility 😄

[fickle geyser]

I only wish AI being able to drive properly ;D

could be hidden between a flag in AISteering component or whatever if you guys fear you'll break vanilla stuff.

[restive pilot]

I've already done that. All I need is a command for manual vehicle steering/acceleration. Which was refused

[fickle geyser]

|| https://streamable.com/bx8t2l ||

[whole cloud]

I don't know what you mean, looks good to me

[rain moth]

mating season?

[quaint flame]

The 2nd tank tried, really hard 😄

[fickle geyser]

I don't even want to bring up how much I died inside after making the whole campaign for SOG.

[quaint flame]

Turned out well though

[fickle geyser]

But yeah, I get it. It's old product and by making AI changes you could break everything.

[whole cloud]

Okey I may or may not have had a big idea how I can get rid of the majority of server network load (with many players and/or objects in the mission)...

Every frame the game iterates through all players and through all objects for each player.
And compares "This is the state I last sent to the player, and this is that objects state now, is the difference large enough that I need to send an update"

Crazy big idea, remember the time when the objects state last updated, and remember the time when the latest state was last sent to the player.

If object state didn't change since the last update was sent, we don't need to calculate the difference and check if its large enough, because we know player has most recent state...
What are the chances of this working and mostly eliminating the bottleneck we currently have in large playercount MP

This will only work if the owner of the object doesn't send updates when the object hasn't changed.
Theoretically there should be no updates for abandoned vehicles that aren't moving and aren't doing anything.... but this is still Arma

[restive pilot]

well if there are too many objects it could be slower, no?

[fickle geyser]

slower than checking the object state?

[restive pilot]

slower than what it's doing right now, because you're adding more check on top

[fickle geyser]

but it checks whole state of the object now

so it would get tiny bit more expensive when there's an object update happening

but should save quite a bit for objects that had no update happen.

That's how I understand that.

So the small performance loss for sending the state of updated objects should be outweighted by performance gain caused by no expensive checks on objects that had no update.

assuming all objects are not updating their state at once

[random isle]

this is still operation flashpoint, lets call things correctly 😄

[restive pilot]

what if the network sync wasn't the bottleneck for the client?
by adding more check, you're doing what you did before + more work to compare states, so slower in that case

[whole cloud]

No. All objects are always checked. What I'm doing is skipping the checks

[whole cloud]

Its one time check for skipping for 6 diff checks per object (don't ask why its 6..)

also its more cache friendly than even one of these 6 checks

The bottleneck right now is CPU cache

[fickle geyser]

So it replaces the current checks

not adds on top

[whole cloud]

its on top

by adding more check, you're doing what you did before + more work to compare states, so slower in that case
Yes for objects that did indeed update, we do one useless check.

But I would say that in a normal Arma mission, most placed objects are actually not changing

[fickle geyser]

So it's tiny bit slower in case object needs update.

But I would say that in a normal Arma mission, most placed objects are actually not changing

but that

so the gained performance for these objects should be bigger than lost one on objects that need updating.

[whole cloud]

yeah. But as our main bottleneck in there is cache.
This check is placed in a cache friendly way.
So the check is very cheap, compared to even one of the others

[hushed seal]

if that works that would be amazing

[whole cloud]

Looks like this during a real MP game. Utilization is great

And physx pulls quite a bit with its ~14%

[rain moth]

neat 👍

will be interesting to test on a 16 core 😄

[hushed seal]

is there any way of getting these details on windows?

[autumn timber]

Run WSL2? 😉

[whole cloud]

Yes, me enabling them. No I won't.. most likely

[rain moth]

but but process explorer 😄

[empty goblet]

counters with Process Hacker

[empty goblet]

note: at 85.190.148.52:2102 is running KOTH stress-test (to reach the limit of hw/engine) to get some pvp optimizations done

[mellow lark]

Which has it's own performance impact though

[autumn timber]

FYI, WSL2's impact is really minimal if you're using the native filesystem. What's usually the bottleneck is access to the NTFS Windows partition from WSL2

WSL1 doesn't have that issue, but it's generally much slower though so it's not recommended to use 1 over 2

I hope I'm not saying any really stupid things, but I seem to recall that WSL2 runs over HyperV and when HyperV hypervisor is enabled, your native Windows installation is also kind of running as a virtualized guest as well, so the speed difference between windows and WSL binaries should not be that much

[mellow lark]

yeah that's what I meant

accessing ntfs takes ages

it's incredible

[fickle geyser]

When you're using WSL2 you're kinda expected to run stuff everything in it. Accessing windows FS is kinda last resort.

Learned this hard way, believed in MS marketing how fast and great it is. Then found out that when using docker on windows on WSL I would have to have all my files inside WSL (which was not possible for me ATM) or suffer terrible performance.

[mellow lark]

Yep exactly the way I found out too

[night mulch]

yo dedmen will the pbo lookup fix be included in the next update?

[whole cloud]

ehhh

no

can you link me those mods?

[gritty wasp]

(Big thanks! Its so hard to get good data)
Say how and what to collect, and arma world will give it.

Well people said it was already on par
I think only those who did not try windows server or have oversized server for their load. For 150+ TvTs windows was always better

How did the story with traffic compression end? Will it be in 2.08 or only perf?

[queen owl]

My Autumn Livonia is here: https://steamcommunity.com/sharedfiles/filedetails/?id=2623774554 and this is the winter 2035: https://steamcommunity.com/sharedfiles/filedetails/?id=1195447305
Apparently the original lookup change didn't make it into the RC because my mod seems to work fine in it...?

[whole cloud]

The change is profiling branch only

[queen owl]

Ok, thanks. I thought it was meant to be in the next main update too.

[whole cloud]

running allMissionObjects every frame.
Instant 100% load

[silk pewter]

https://tenor.com/view/dieter-bohn-just-dont-dont-do-that-gif-24425067

[rain moth]

perfect, now you can say that the linux server will use all cores

no more complaints about arma not being multithreaded!

[autumn timber]

Writes an extension that creates 32 busy loops and markets it as "mod improving Arma 3 CPU utilization"

[empty goblet]

let's fill all the caches and make sure the prediction is overloaded with misses add some avx512 and you can melt the sockets

[rain moth]

always misalign all data and code, add lots of meaningless padding everywhere

[whole cloud]

[rain moth]

😎

[whole cloud]

2.06.148819 new PROFILING branch with PERFORMANCE binaries, v8, server and client, windows 32/64-bit, linux server 32/64-bit

• Added: The RPT Mod list now shows when a modfolder doesn't contain any PBOs to help with Linux servers
• Added: Automatically add CDLC tags to compositions that contain CDLC objects
• Tweaked: Limited length of script errors so that on-screen error message is always shown completely
• Tweaked: Use AvSetMmThreadCharacteristics to tell windows that we are a Game
• Tweaked: Improved multithreaded code performance and fixed bottleneck on dedicated servers (Thanks Kohbrax)
• Tweaked: Very experimental multithreading improvement on Linux
• Tweaked: Minor performance improvement if there are many vehicles on the map
• Fixed: Car no longer locks into extreme turn if driver opens map, changes seat or exits a vehicle after initiating turn - https://feedback.bistudio.com/T162644
• Fixed: Selection of groups and waypoints in Zeus was broken in SP
• Fixed: Erratic behaviour of Darter following waypoints when flying at high altitude - https://feedback.bistudio.com/T82570
• Fixed: Crash when shift click selecting item in a ListView without any previously selected items - https://feedback.bistudio.com/T161512
• Fixed: Rebreather facemask would show up as soon as feet touch water while diving goggles only show when swimming - https://feedback.bistudio.com/T161998
• Fixed: Stopping free fall by grabbing onto a ladder would still apply fall damage when leaving the ladder
• Fixed: "Suspending not allowed in this context" script error would not show in on-screen error message - https://feedback.bistudio.com/T151337
• Fixed: Potential game crash after loading savegame - https://feedback.bistudio.com/T162128
• Fixed: canAdd script command was creating stray entities (remotely in MP) (Thanks @kindred radish) - https://feedback.bistudio.com/T163205
• Removed: Network reports (networkDiagInterval, NetStrStatsExport) (still available on profiling build)

[whole cloud]

google drive failed for client, canAdd is still creating stray entities just not remotely anymore.
Next prof build will be later this week.
So if you find issues, tell me quick

[kindred radish]

Are you going to look into client strays or does it make sense to run a trash ent cleaner on the client ? 🤔

[pallid pebble]

• Fixed: Erratic behaviour of Darter following waypoints when flying at high altitude - https://feedback.bistudio.com/T82570
  

[hushed seal]

Network reports (networkDiagInterval,
oi!

[whole cloud]

already fixed today

[whole cloud]

85% of network string encoding was spent keeping track for NetStrStats, that 99% of users never use.
So don't even try to complain that 99% users should pay a high price for stuff they don't want.
If you still want it, you can get it on profiling.

[hushed seal]

oh wow, wasn't aware of that, no complaining from me 😄

[kindred radish]

Probably a coincidence but after using profiling build of the server for the first time it looks like Windows decided to change folder rights to ReadOnly and our log Extension stopped working till i realised it 😆
OS: Windows Server 2019 Standard

Maybe because of "- Tweaked: Use AvSetMmThreadCharacteristics to tell windows that we are a Game" ?!

[empty goblet]

and how you come on that absurd conclusion

[kindred radish]

Because i never had such a problem in years, i didn't change any folder rights and it was working 2 minutes before switching to profiling build and starting the server again.
And this is the most plausible change that could cause it 🤷‍♂️

[empty goblet]

the change has nothing to do with file system / folder rights so that's quite weird assumption

[kindred radish]

That's why i was asking, maybe anyone else comes here with the same problem.
Just found it weird so i thought doesn't hurt to share that thought.

[empty goblet]

it just tells OS how to classify the process https://en.wikipedia.org/wiki/Multimedia_Class_Scheduler_Service , https://docs.microsoft.com/en-us/windows/win32/procthread/multimedia-class-scheduler-service

[kindred radish]

Then it was something else, not saying i am sure it's the game/this change, but it would be a weird coincidence since the extension (A3Log) logged normally not even 1 minute before the change of the exe.
And as i said we never had this in years of running the server with A3Log.

[autumn timber]

OMG, please tell me that this is fixing what I think this is fixing! 😱

[pallid pebble]

I hope so 👀

[autumn timber]

i seem to recall that someone said that this is unfixable 🤞

@feral harness could you shed some light on what exactly was causing the darter to spin? I don't care if the answer will be overly specific. I just need to know! 😆

[feral harness]

pathfinding problem, doent like flying things

[autumn timber]

Still. May I ask for a more specific explanation? 😦
I'm seriously interested in knowing

[empty goblet]

i bet the Darter was like "do i love this path?" "or do i love that path?" "which cost me less?" "what i was thinking about?" "do i love this path?"... 😁

[autumn timber]

As a programmer, I'd be interested in knowing it, even if the answer would have been "it was increasing i even though it should have been increasing l" (assuming you explain what i and l were, beforehand 😛 )

[restive pilot]

well iirc the darter kept circling, so it probably couldn't finish a path node (condition never satisfied)
or maybe it kept thinking it was "off path" and regenerated new path constantly...

[gritty wasp]

• Fixed: canAdd script command was creating stray entities (remotely in MP) (Thanks @kindred radish) - https://feedback.bistudio.com/T163205
  Do we need both client and server to be on perf?

[kindred radish]

For the trash entity fix the client is the important part if you meant to ping me, so you don't need both on perf.

Quote from another admin "Looks like the fix is the client. Ran client and server, no entries, went to one of my live servers on V7 with my client on V8, no entries"

[feral harness]

In a nutshell it was pulling next move point from calculated path but because of high altitude it was rejecting it, basically what @empty goblet said

[gritty wasp]

Sorry for ping. I just copy n paste line from change log. I guess it depends where canAdd called.

[restive pilot]

• Tweaked: Use AvSetMmThreadCharacteristics to tell windows that we are a Game
  what's the priority for a game? 26 (max)?

[whole cloud]

whereever you execute canAdd command

[whole cloud]

We just set it to "Games"

so whatever priority your "Games" windows has by default

[empty goblet]

also this is just using very narrow bit of options e.g. https://docs.microsoft.com/en-us/windows/win32/medfound/media-foundation-work-queue-and-threading-improvements#enhancements-to-mmcss-support and there is https://docs.microsoft.com/en-us/windows/win32/procthread/platform-work-queue-api which we don't touch at all

[hexed gorge]

Is there a release date for version 2.10? I'm interested in setUnitFreefallHeight. Can it be brought forward to version 2.08?
(I accept "no" for an answer, but please don't give up on this function)

[worldly badge]

This ain't #dev_rc_branch but no, that will never be the thing. 2.08 is already right around the corner and the contents are already determined... I assume, but that's how it works

[hexed gorge]

No problem 🤜 🤛

[worldly badge]

Sit patience and we'll see 👍

[whole cloud]

yes, no

[willow thorn]

Something is wrong with the performance branch v7 and v8, I get this error when starting the mission, although everything is fine on the stable branch

15:02:23 Missing addons detected:
15:02:23   A3_Structures_F_Exp_Walls_Concrete
15:02:23   A3_Structures_F_Exp_Walls_Wired
15:02:23   A3_Structures_F_Exp_Walls_Tin
15:02:23   A3_Structures_F_Exp_Walls_BackAlleys
15:02:23 Warning Message: You cannot play/edit this mission; it is dependent on downloadable content that has been deleted.
A3_Structures_F_Exp_Walls_Concrete, A3_Structures_F_Exp_Walls_Wired, A3_Structures_F_Exp_Walls_Tin, A3_Structures_F_Exp_Walls_BackAlleys

[whole cloud]

Can you send me your modlist?

Only v7 and v8, or did you just switch to profiling branch after not having been on there for a long time?

[willow thorn]

I tried 7 and 8, didn't test others
We have our own mod pack, but the problem is that it only complains about addons, their apex

[whole cloud]

probably because some other addon is using same pboprefix to overwrite something

[whole cloud]

like these

[willow thorn]

I'm sorry, I didn't quite understand what I need to do

[rain moth]

try without other mods

[willow thorn]

ok

[whole cloud]

or don't use profiling branch till I fix that exocet bug which will not be this week, maybe next

[silk pewter]

I can ban that @night mulch if you want Ded'

[feral harness]

sweep under the Stratis

[night mulch]

9. Moderators are volunteers who moderate, intervene only when needed, by their best judgement in uncertain situations. If you feel that you have been wrongly handled by a Moderator please contact sysoporcm, wait till response (as soon as possible).

Time to execute rule 9

[silk pewter]

I had another execution in mind… 😈 😂

stop adding bugs! 😄

[night mulch]

*adds tons of bugs *😈

[sick arrow]

Sry for stupid question. But does this fix vehicle surround sound bug that has been in game forever. Crackling and massive fps drops.

[whole cloud]

crackling? no

[spiral pond]

from tanks at low RPM?

[sick arrow]

No in any vehicle. When you have surround sound will pop all the time and static crack with massive fps drops. I play on stereo because of that.

[random isle]

Error: EntityAI SubSkeleton index was not initialized properly (repeated 139x in the last 60sec)

[whole cloud]

Fix your broken mods I guess

[whole cloud]

That path issue with autumn livonia and winter 20335 and whatever FairyTale found there will be fixed in tomorrows build

[pale mica]

I've seen(listened) this issue only with RHS tanks.

[random isle]

that was on vanilla ai creation

[whole cloud]

Didn't touch any skeleton stuff though :u

[kindred radish]

This error was always present in our logs, also thought it's one of our mods. But if he has it in vanilla

[whole cloud]

2.06.148844 new PROFILING branch with PERFORMANCE binaries, v9, server and client, windows 32/64-bit, linux server 32/64-bit

• Added: New -noPauseAudio start parameter to keep audio running while the game is not in focus - https://feedback.bistudio.com/T163147
• Tweaked: Improved networking performance (update checking for players/objects is now spread over multiple frames)
• Tweaked: skipTime script command is now limited to max +/-1000 hours per skip - https://feedback.bistudio.com/T68952
• Fixed: canAdd/getContainerMaxLoad script commands were creating stray entities (locally) (Thanks AeoG | El'Rabito) - https://feedback.bistudio.com/T163205
• Fixed: File reading errors when there are multiple PBOs with the same prefix loaded
• Fixed: It was possible to see illegal game dates before 1970 - https://feedback.bistudio.com/T163269

[hushed seal]

Hey dedmem, is there any plan on changing this?
#perf_prof_branch message
(HC cache lock based on profile folder and not name)
Never created a ticket for it so not expecting anything, just wondering.

[whole cloud]

ahh there are two more stray entity leaks. When you add multiple uniform/vest into a vehicle inventory at once, and it seems inside setUnitLoadout

[whole cloud]

yes but its low prio

[hushed seal]

okay, thanks for your continued work!

[kindred radish]

Looks like something breaks certain light sources now (Perf V9).

Example with the "Land_TentLamp_01_suspended_F"
https://gyazo.com/0ada4cd42799928a87d28c3d3e2faad1
Ran this below to trigger it again after i saw it happening on our server and using skiptime -1.
skiptime 0.25
and it looks like this
https://gyazo.com/1f70ccf03c929c659100b58e1b402948

https://gyazo.com/485209ff076c7ea33664683e61f17ed3
https://gyazo.com/ebbff4f8c04c93ead99d727ec5a8b07c

So at a certain daytime they just break and behave weird. They cast weird/broken shadows and don't illuminate the surroundings like before.

Seems to be inconsistent tho
Here they work fine: https://gyazo.com/23e6bf40bea534e13cd6c4dfca1606d6
Here they don't: https://gyazo.com/86c0964d23ab738aa90ea870602b320e

• Even tho its the same time & type of lightsource. 🤔
  If the time goes towards sunrise again they seem to work again -> https://gyazo.com/7bafd01566c6494433ce16208942f94d

---

Another example with different light sources.
https://gyazo.com/1ceaa4b8e67150676344c31ce75ab67e
Only did one step forward
https://gyazo.com/c04fd3fa7fdbba6d39952d85ae260359

[whole cloud]

only on v9? v8 was fine?

[kindred radish]

Pretty sure it was, but iam going to test again to be 100 % sure.

[whole cloud]

My thought is the skipTime/dates stuff.
These lights look like they might be daytime off? Like during daytime lights turn off, these seem to just turn off at night

but it only happens for some lights, not all?

[kindred radish]

It started without using skiptime, i just used it to see/trigger the difference.

[whole cloud]

If you can confirm its v9 then I'll need to repro on monday

[median belfry]

What do you think about the idea to disable the time sync for one client, would it be difficult to add a command for it ?

[kindred radish]

Nvm its the same in v8 and even stable, not sure how i never saw that 🤔
Gonna debug it a bit more

[empty goblet]

time to twist it and lure Dedmen to replace A3 DX11v1 renderer with some older DX11v2 version from DayZ (2017) ....

denies he ever said that and hides from wrath of DD

[quaint flame]

screenshot

[empty goblet]

but i can dream about fixed lights, shadows, illumination not going thru objects/surface and renderer untied from simulation

[quaint flame]

You can dream about it, but dedmen can do it, right....right? 😄

[whole cloud]

https://cdn.discordapp.com/emojis/668897449439657986.gif?size=256

[empty goblet]

many claimed it could be done and fast, nobody did it so who knows ... maybe he can ... oh wait run he is there

[silk pewter]

Dwarden leaking the 2.08 changelog

[empty goblet]

2.08 is set in stone nearly

so you mistaken, it's .10

[silk pewter]

ah well, such a small change could be reviewed and added easily in the next profiling alright

[kindred radish]

Solved 🤣
I set dynamic lights to High a few days ago, with Ultra it looks fine
Probably the amount of light sources in this area which causes it to behave weird.

Dynamic lights high: https://gyazo.com/df567e10c031d96454627da37f1f0aea
Dynamic lights ultra: https://gyazo.com/9c7a8a0ad4836841d3a13a830ce8881d

Very high Dynamic lights also works properly
https://gyazo.com/5a3969f4ceb8d1a580627b73ac54f46a

[whole cloud]

Don't frighten me like that

[silk pewter]

good, more time to work on DX11v2 😄

[spiral pond]

would be clamping knowsAbout/reveal values to 2/3 digits, so you end up with less miniscule value difference, thus the comparison of old vs new is less often different/the value doesnt change as much - and doesnt have to be transmitted as often?

[autumn timber]

Were these people actual programmers or just Product Owners / Scrum Masters? 🤔
(because it won't surprise me if it's the latter - it's always like this 😄 )

[feral harness]

doubt it

[kindred radish]

Looks like the changes in v9 are
Much lower CPU load and more stable server FPS/CPS
https://gyazo.com/558f43e9a3c8147c14bd1e7bf97b5fb0

Great work dedmen 👌

[whole cloud]

Just wait for the unforseen consequences

[kindred radish]

24 hour stats: https://gyazo.com/c432b4282456501143b05a2952fc49a3
1 week stats: https://gyazo.com/ab572c702eadbe05aa0998c5a0936042
I swapped from V8 to V9 today at 15:00

Looks like a huge change in CPU load.
We have like 8k mission objects while the majority is not changing state since they are base building objects (walls, floors, containers etc).

[pale mica]

Am I missing something? the google drive folder is now private?

[whole cloud]

ohno

not for me

https://drive.google.com/drive/folders/1nOYRMZO4vs3vTkTN2I05XJ6objp8IjvQ this one

[pale mica]

thanks

The BI forum thread needs some updating

[whole cloud]

the posts at the end are correct

and the first post also has it correct

[pale mica]

yeah you are right, i had a bunch of tab open. I had open the archived ones by mistake

[kindred radish]

Running V9 since 24 hours now, no reports about any problems so far and still looking

8k mission objects / 300 vehicles / up to 150 AI / no HC / server fps cap 100
1 week stats: https://gyazo.com/84532742c5ff4cf7fcb550084433feed
24 hours stats: https://gyazo.com/6a907d7d996eb10c45a6f6c18babe341

dedmen needs a salary increase

[hushed seal]

the object heavy missions makers in my unit will love this

[night mulch]

is that improvement only for MP tho

[whole cloud]

it doesn't change your client fps

atleast I'm pretty sure it shouldn't

there aren't any less updates sent, there aren't any less objects around

[gritty wasp]

avg server fps 158 at 150 players. Insane.

[whole cloud]

I need more problem reports

[silk pewter]

…lights go through walls? 😬

[empty goblet]

casts magic renderer spell at @silk pewter ...

[whole cloud]

Any reports about spurious weapon firing even though you are not firing?
Or high "input lag" in zeus after hours when placing modules and stuff?

[upper wigeon]

For the improved networking performance to work properly, does the performance/profiling branch only need to be on the server or does it need to be on all the clients as well?

[strange badge]

yes, everyone's gun went off non-stop once the first person started shooting

and we had a GL get fired that didn't stop exploding, even after a mission restart and a map switch

[kindred radish]

game mode/scenario ? mods used ?

[strange badge]

@zis, @zen_compat_ace, @zen, @zei, @zec, @vet_unflipping, @urikimissionitems, @toh_characters, @theseus_services, @tembelan, @tac_mods_cup_vehicles_acre_racks, @tac_mods_a3_acre_racks, @tac_mods, @synixe, @suonkomarsahav, @spec4gear, @pulau, @napf_snow, @napf, @mu, @milgearpack_retexture, @milgearpack_ext, @milgearpack_aor, @milgearpack, @lambs_suppression, @lambs, @kujari, @kat, @hmd_cup, @hmd, @hellanmaa, @gt_armory, @grad_trenches, @grad_civs, @enhanced_movement, @edenobjects, @diwako_dui, @cup_weapons_ace_compat, @cup_weapons, @cup_vehicles_ace_compat, @cup_vehicles, @cup_units, @cup_terrains_maps, @cup_terrains_core, @cup_terrains_ace_compat, @commander-persistent-gear, @commander-core, @cfp, @cba_a3, @cartridge_cases, @bozcaada, @armaforces_friendly_tracker, @anizay, @acre2, @ace, @a3am, @3denenhanced

[hexed gorge]

😆

Need more problem to report? About the game?

[hexed gorge]

I mean, game is ok. I hope i can use V9 in my game host.

@strange badge 57 mods? They can be causing the problem.

[strange badge]

They are not, issue only occurred on profiling

[hexed gorge]

Ops, sorry to ping you now.

[strange badge]

That's fine

[boreal wigeon]

Same happened for my group yesterday wanted to test today what caused it. (the endless shooting)

[hushed seal]

Now that scares me....

[hexed gorge]

I congratulate Dedmen for trying new things. As we say here TMJ or in English WAT "We are together".
Long live Arma 3!

[empty goblet]

well i just seen video of it and i think i got it too , but i dismissed it as the usual alt-tab into fireclick loop

[hushed seal]

any ideas as to whats causing it? got a 70 player OP tonight and not that keen on 70 rifles going off non stop 😄

[rain moth]

V9 😛

[whole cloud]

I just played two days of missions and didn't get any random shooty thing
How do I find out what it is

[restive pilot]

play with these guys 😛

[whole cloud]

I'll try to push a v10 tomorrow with some things removed and then we'll see

[hexed gorge]

Hopefully later you can "roll back what was rolled back", that is, put the changes back.

[inland dew]

also no problem for few days playing official warlords

[whole cloud]

Sure. Its just annoying that we have to push more versions.
Though it would be nice to set a prof branch v number record 🤣

[quaint flame]

My Arma is crashing frequently on profiling branch in single player. Would you be interested in the crash reports?

[hexed gorge]

Prof branch v record? My record in Excel is v 65.

[whole cloud]

always

[empty goblet]

strange is, tons of Zeus/Wl servers run at 148844 and nobody reported that bug to me yet

[quaint flame]

https://feedback.bistudio.com/T163426 @whole cloud

I switched to stable and I've been playing for an hour now without crashing.

[silk pewter]

spam removed, thanks

[hushed seal]

75 players, 150AI and 2 HC, Server FPS at 80

with HC well above 100, all running great 🙂

[hexed gorge]

@hushed seal Can I enjoy this wonder just by upgrading the server to version 9?

[random isle]

25 people (count allunits) 750 server/hc at 15-20 fps, 400 units server/hc at 48fps

hc really dont do shit when overall units is over like 450 +, you can see cpu threads start to look like saw

and this is on i7 10700k @ 5.3ghz with fast ram

[hushed seal]

Yup

[thorny pumice]

Is the server FPS cap removed in the latest binaries or is there some startup param that ups the cap from 50?

[edgy comet]

-limitFPS= start parameter to adjust server FPS limit between 5-1000 FPS (default and previous limit 50)

[hushed seal]

Which is also in the main branch btw.

[whole cloud]

Same, noone reported it to me either.
I just randomly saw that one guy chatting about it in the other discord you saw too.

I reverted the main performance change.
If that wasn't it, I have one other idea (there was a change I made months ago, but that actually has been broken since then because I forgot a line... I fixed that together with the perf one, maybe its that)
And if neither of these, we'll keep digging 🙃

[kindred radish]

It was nice to have
https://gyazo.com/7340117afd2f5a204d0802334dfa0b0c

[whole cloud]

2.06.148854 new PROFILING branch with PERFORMANCE binaries, v10, server and client, windows 32/64-bit, linux server 32/64-bit

• Fixed: More stray entities bugs in addItemCargo and setUnitLoadout - https://feedback.bistudio.com/T163205
• Reverted: Update checking for players/objects being spread over multiple frames (due to spurious/repeated/unprovoked weapon discharge)

[whole cloud]

This is just a guess. The sooner someone tells me that weapons still keep going off, the sooner I can return that improvement 👀

[silk pewter]

that's all because of you @strange badge@boreal wigeon, I hope you're happy!! 😄

[hushed seal]

I'll stay on v9.... as a control group 👼

[gritty wasp]

Would be fun if unprovoked discharge is just ace_overheat

[whole cloud]

yeah 😄
Same thought, but KotH people reported it too on vanilla

[empty goblet]

and remember there is the stuck * fire bug which happens on alt-tab / switch window (the fire button gets triggered and stuck until you run out of ammo or push esc etc.

so lot of people disregard the 'new' bug as that 'old' bug

[boreal wigeon]

I saw it happen to 3 of my team members on Saturday. Server was running v9 including myself. Not sure about the other 3 but most likely not. I didn't have the issue. Even after putting the weapon on the back or dropping the weapon it kept firing. After encountering the issue I switched back to main branch and no issues happened afterwards. My though was that it could be a client -> server issue as server does some different stuff than the client due to the different version.

We were using a weather mod to create dust devils, mission maker decided not to use those incase that was the issue. But personally I don't see a connection between those and the firing.

However if wanted I can try to recreate the issue for a possible repro.

[strange badge]

I recreated the container running arma to switch branches, unfortunately losing server logs. I do have these though, https://clips.twitch.tv/SmokyGenerousWoodpeckerSaltBae-otqLh_HUmI0e_5Z9 https://clips.twitch.tv/ImpartialZealousGrassHeyGirl-h6cPfp0mkBTYN2AF

[boreal wigeon]

I have it happening to me right now, not really anything in the logs though... :/

[gritty wasp]

Did you update to v10?

[boreal wigeon]

No still running v9

Only thing in client logs idk about:

17:41:49 Client: Network message ad16 (update info b5146090) is pending
17:41:55 Client: Network message adee (update info b5146090) is pending```

[whole cloud]

ok thats hilarious.

Worth it :stamp:

[boreal wigeon]

Even if I go into Zeus my player keeps shooting...

[fickle geyser]

I guess some object/weapon state is not updated on the clients?

[whole cloud]

now try if v10 fixes it

[boreal wigeon]

will do

[fickle geyser]

https://tenor.com/bOEpv.gif

[whole cloud]

I don't understand it.
Firing a gun is a single guaranteed network message that is sent once.
It doesn't stay in some queue where it could be ....... mh.. 🤔

Ok well... That tells me v10 wouldn't fix it if its that 😁

[restive pilot]

also it doesn't happen every time you fire...

[whole cloud]

if the message doesn't get acknowledged by the client, it is resent.
But even if that happens the client should ignore duplicates

[strange badge]

It also doesn't create a projectile, it's just the firing animation on the weapon regardless of weapon (ace gpr) or state (will fire when put on your back with ace)

[fickle geyser]

if this is related they have different ids

[whole cloud]

and it makes the sound
and it spawns projectile tracer effects

[restive pilot]

tracer effects? isn't the effect part of the projectile model?

[strange badge]

I don't think I have a video, but we also had the spot of GL impact continually explode, effects and all that, but no damage being dealt from it

[boreal wigeon]

b5146090 bit is the same though, I tried checking if it has to do with the ammo id but it doesn't.
also there seems to be a pattern that gets "replayed" it's not total random.

[fickle geyser]

I would rather expect it to be weapon/player object id or smth.

[boreal wigeon]

Been shooting for the last 5 minutes on v10 no issues so far. On v9 I would have gotten the issue by now.

[boreal wigeon]

The messages also still appear so those are most likely something else.

[boreal wigeon]

Back on V09:
Added a Fired EH to the player, and it triggers.
It creates a new projectile for each "bugged" shot.
The weapon mode that's displayed is of the initial one, eg. after the bug appeared I switched the weapon mode the old one still shows up in the EH.

After disconnecting, restarting the game and reconnecting, the initial gun is still firing next to my (old) dead body.
Using a firedNear EH when standing next to the gun also triggers.

[boreal wigeon]

I've got a pretty convincing repro on vanilla & VR. You want a ticket dedmen or you have an idea already?

[whole cloud]

Well if its simple enough then I can try it without a ticket

[boreal wigeon]

Also got it with client on v9 :/ All previous test was with client on stable.

v10 on server (client on v9) is fine though, just did a final test to make sure.

v10 on client and v9 on server gives the issue. idk whats going on at this point.
Good luck finding the cause dedmen ❤️

[whole cloud]

Its the worst cause to find though :sad:

Diwako taught me that this same issue happens if you completely F' up your basic.cfg settings

The performance improvement shouldn't have any correlation with fired messages being resent repeatedly, but apparently it somehow does

But I just got report that stuff is still somewhat wrong.
So maybe the chance is that you are just lucky and are not getting the bug but its still there 🤞

[boreal wigeon]

I only swapped the exe on the client/server. So you would think that nothing else would matter. But you can never know for sure. If you want me to try and use a clean/new profile I can try that as well to rule it out.

[whole cloud]

client doesn't matter. pretty sure

[boreal wigeon]

pretty sure

[gritty wasp]

So, fact that not every community experienced this bug can be because of specific values in basic.cfg
I can check rpt for pending messages from Saturday games.

[boreal wigeon]

What OS do you use?

[gritty wasp]

Windows

[boreal wigeon]

I'm on Linux, that might also be a factor. Also makes sense why none of the official servers has issues as I'm pretty sure those run Windows as well.

Brett said he's using containers so I'm leaning that he might use Linux as well.

I'll do some tests tomorrow on a Windows server. As my repro on Linux was a 100%.

[fickle geyser]

Yes, Brett is using linux server in docker.

[rain moth]

how much difference is there between the threaded helper jobs on windows vs linux?

[empty goblet]

strange q, anyone can check if aren't some EHs like Killed / entityKilled etc. firing twice ? i seen some weird ... logs

[inland dew]

so far it's been all good for me, for few days already, on v9

[hushed seal]

Zero issues for us with v9, also on windows

[whole cloud]

eh.. stuff?

Windows uses windows threads, linux uses pthread
Windows uses windows events, linux uses pthread events
Windows uses windows mutex, linux uses pthread mutex

Not sure how to explain a "difference" to you

[rain moth]

I was thinking more if there is a difference in the distribution of workloads i.e. messages being handled differently on the platforms

[whole cloud]

no

Also network messages don't even go into that multithreading stuff that was changed recently

And "shot fired" messages are also nowhere near that huge performance improvement done in v9. That was about regular object updates, not one-shot messages.

[charred heart]

Hi everyone, tried to use latest performance binaries (v10) and on HC console instead of Arma 3 and version it says Model viewer:
Server: https://i.imgur.com/LByqECH.png
HC: https://i.imgur.com/jKVVOnf.png
happens only on Exile mod servers

[silk pewter]

what have you done 😄

[mellow lark]

cursed binaries

[violet elbow]

Oops.

[boreal wigeon]

Windows server had no issues.
Also cleared the basic.cfg on Linux server and issue still persists (I'm presuming default values should be safe).

[violet elbow]

Damn Linux servers taking away our performance improvements

[hexed gorge]

My game server host can't use prof branch 😨

@charred heart only on Exile mod? Linux or Windows server?

[charred heart]

Windows server 2019, yes, only on Exile mod, it crashes the server on load, before it reads the db completely. Server + HC.
Exile Escape server, and non Exile servers works fine however.

[inland dew]

I think that +- since V9 and also in V10, I experience for the first time in years, since A2, missiles frozen on the radard or teleporting back and front again and simply dead in my jet for many days and on different official Warlords servers
vehicles/air also teleporting/freezing

I have ping 6 and 500 mb internet and it was never a problem before. playing official warlords for years now

[charred heart]

who is your host, if I may ask?

[inland dew]

I also hear strange sound, from time to time, when playing on different official Warlords servers, like somebody is shooting with a MAAWS AT launcher (or similar) somewhere not far from me, one shot or few or more than that, when I'm in different places on the map, but there is 100% really nobody present

[empty goblet]

i can attest that ain't fluke in your dreams and i keep hearing that on profiling branch too

[random isle]

Make sure you have AVRT.DLL in your system32 folder

if not copy it over and it will work fine

[hexed gorge]

@charred heart GTX Gaming. I open a ticket about it and they can update it for me, I just need to make the EXE available 😀

A little too manual, but at least it's possible.

I've had a lot of great and "crazy" ideas, and they all took more work than I expected. Some were abandoned and others gave great results.

[charred heart]

it is there

[whole cloud]

Not me, blame exile

[kindred radish]

Don't think that's Exile related tho, otherwise we would have more people here complaining about the same thing.

[whole cloud]

Was able to repro the gun shooty on linux! thanks for the tips. Will investigate

And I tracked it down to that other change that I did months ago (but did wrong so it never worked) and just fixed on the side while working on the big stuff...
When I break that old stuff again like it has been, it doesn't spam fire my gun, even with the big performance improvement active.

But that means it should still bug around on v10.

When I reenable the old stuff I get the spam firing instantly, but only when I fire full auto.

OMG this is so funny.
This is the perfect random number generator

[boreal wigeon]

Glad you found the issue. And yes it's fun in testing but not when someone is deafening you with a MG next to you.

[whole cloud]

That goes into my top ranking of funniest bugs I caused

Sadly I'm not actually keeping track of that ranking, maybe I should

[empty goblet]

not funny, remember you need !perfect! random number generator for recoil generator to combat antirecoil scripts, sofware and 'hardware antirecoil' made by some mouse hardware makers 😉

so keep that perfect part for the future 🤣

[whole cloud]

HAH

We gotta keep this and only enable it for piracy 🤣

[kindred radish]

Not like there aren't already some "easter eggs" against piracy.
Falling leaflets, lightning strikes and a weird hold-action which caused me to search for cheaters for a couple of hours
Now i just ban everyone with DLC unlocker

[empty goblet]

add cheaters, griefers, exploiters and teamkillers to that group 🤤

[whole cloud]

Mh its a combo of both changes. Interesting
But the small change could also cause the desyncing that was reported so I'll just fix that one

[pallid island]

Does that bug actually fire a projectile?

[boreal wigeon]

If you put the gun down it should keep firing as well.

[whole cloud]

no

[boreal wigeon]

It creates one I believe as it has a tracer just no damage. It also creates particles effects on the ground.

[pallid island]

Huh. I think I experienced something like that several months ago in my group. I just chalked it up to desync or Arma at the time

It was a one time occurrence only though

I'll share the video when I get home tonight cause it was pretty funny

[whole cloud]

yes it can happen with desync/bad basic.cfg too

I think I have it fixed now, release tomorrow then

[empty goblet]

sighs with wish can't be tomorrow today ?

[gritty wasp]

Just fix or multithreading perf tuning too?

[empty goblet]

well from what i understand the fix means those reverted perf optimizations can be put back

[inland dew]

DX11 DayZ render when? 😏

[gritty wasp]

Also interesting why bug happened on Linux version

[empty goblet]

it happens on windows too, just DD was testing it on linux

[whole cloud]

Others reported on linux. Maybe it was more prominent there.
But I also played with it on linux two weekend missions and didn't hit it.
its weird ish

[autumn timber]

Better midrange textures when? 😏

[autumn timber]

That's obviously related to this message 🙄

[kindred radish]

Dying while getting out from a boat at the shore
Had it on multiple maps and even without any mods (KotH) ^^
It's impossible to replicate on purpose tho 😦

[inland dew]

Visuals improvement is one thing, but significant performance improvement is much more needed.
And DX11 render from DayZ is the solution to this

[whole cloud]

"And DX11 render from DayZ is the solution to this"
not according to my profiling

[autumn timber]

But i think that we can both agree that it's obvious none of the two things will be done, due the the scope size
I was writing mine as a joke

[whole cloud]

most performance issues below 40 fps are not rendering

[fickle geyser]

So is the weapons auto shooting issue not present on the v10? Having an op in few hours and not sure if I should leave the server on profiling branch

[hexed gorge]

@fickle geyser #perf_prof_branch message

"But that means it should still bug around on v10."

[whole cloud]

^

[inland dew]

Only partially, of course. Not a solution for all the problems

[inland dew]

Only one of man6 optimisations that can be done, but definitley worth it

[whole cloud]

From my numbers, not worth it

[minor silo]

Are those numbers available somewhere to study?

[silk pewter]

it's not a science common project 😂

[minor silo]

Arma is a fascinating phenomenon 🤔

Also it would be interesting to know these numbers even though these are for below 40 FPS which is the target of optimalisation. But not everyone has weak computer so what about upper range improvements?

[empty goblet]

puts the room at fire and runs with #enfusion_engine cake away

[whole cloud]

I don't care about upper range improvements

[minor silo]

Heh

s a d

[hushed seal]

In the eyes of they arma everyone has a weak computer

[minor silo]

I rarely go even under 60 FPS and many of my friends as well, so not everyone.

[hushed seal]

how big are your OPs? I've started tracking player FPS in ours and I don't think I've seen anyone average more than 60

[minor silo]

But I understand, that +1 FPS in range between 0-40 is more prioritised than +20 FPS in range 80-120.

[minor silo]

70+ GB modpack with a lot of poorly optimised mods, usually 30+ players, 100 AI(Scenarios curated by Zeus, so they are placed "Just in Time") and mostly without headless. Tho only few cross that 60 FPS barrier 😅, but the number of people is growing. But I guess that is necessarily not relevant to the perf_prof topic 😅.

[gray spade]

We stumbled upon a strange issue today. Our Linux server dropped instantly to 0 FPS with 2.06 performance and profiling x64 binaries v09 and v10 when loading the UMB Colombia map (https://steamcommunity.com/sharedfiles/filedetails/?id=2266710560). However loading it with the "vanilla" binary everything works at 100 FPS. No other mods or DLCs involved.

0 FPS with 2.06.148854 x64

100 FPS with 2.06.148470 x64

OS: Ubuntu 20.04.4 LTS

captureFrame on server with v09 profiling binary: https://gist.github.com/Schwaggot/8075a2a42cdf18c7eab29f827a93d126

If you need more info I will be online again tomorrow probably. The mission itself was just the map, 1 NATO Squad with all members set to playable.

[hexed gorge]

It's really just a few notes on a used napkin.

[pallid island]

if anyone wants to be humored https://streamable.com/bj7u29

[random isle]

Hehe saw this today as well on mi24

[whole cloud]

Thank you, I have successfully been humored

[rain moth]

profiling timeupdated: 1645695340 (24 February 2022 – 09:35:40 UTC)

[whole cloud]

Oh. I feared you were too early again but you are 10 minutes late and I just missed the ping

2.06.148877 new PROFILING branch with PERFORMANCE binaries, v11, server and client, windows 32/64-bit, linux server 32/64-bit

• Reverted: Reverted update checking for players/objects being spread over multiple frames (spurious weapon discharge fixed)
• Fixed: Several game crashes
• Fixed: Planes would start moving on their own when the user opened a map and placed a marker - https://feedback.bistudio.com/T163253
• Fixed: Incoming missile warnings were played through the radio instead of the effects channel - https://feedback.bistudio.com/T163245

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[silk pewter]

is the huge performance improvement back? 🙂

[whole cloud]

yes, first line

[restive pilot]

are these improvements available on the dev branch too?

[whole cloud]

not sure if todays

[tawdry gazelle]

double reverted! yeah! 🤣

[full nova]

wait new profiling is out?

[rain moth]

yes, v11

[inland dew]

• Tweaked: Improved networking performance (update checking for players/objects is now spread over multiple frames)
  have updated to v11 now and joined severs servers, also Warlords with more than 30 players and I was amazed by how low and stable/consistent the frametime is now. never seen it being so good until now. it's feels very very smooth to play

[inland dew]

in my very basic understanding (correct me if I'm wrong), checking stuff over multiple frames, instead of 1, is similar to having the engine doing a lot of stuff at the same time/in parallel on multiple cores/threads, instead of only on 1 core

[whole cloud]

If someone gets spurious weapon firing again, please start the siren first and spam ping me after 😄

[whole cloud]

Yes.
This code was already multithreaded over all available cores. But even that is too much load

Checking if all network objects need an update every frame is just too hard.
The bottleneck there is memory speed/cpu cache. I didn't find a viable way to improve that (I already made large improvements months ago there but it wasn't enough)

But instead, just check fewer objects.
The server knows when a objects state changes, so it doesn't need to check "does this object need to be updated" if it knows it didn't get a state change for a long time.

So now objects that don't change (houses and statics for example) are only checked every 5 frames, instead of every frame. And objects that did change always get checked right away so there will be no bad delays introduced.

I can probably increase that timerange higher than 5 frames but better safe for now

[inland dew]

look at the frametime with ultra + custom 1440p settings and max view distance and 30+ players on official Warlords

man... damn... so smooth now and so low...

[restive pilot]

can you get that framerate on the ground during gun fights?! I doubt it 😛

[gritty wasp]

This is desert. Try in towns

[whole cloud]

I don't think it would improve clientside performance though

[inland dew]

will go to Kavala now

it does, since frametime is super low and stable

never was before

[minor silo]

Many people reported to me that there is a FPS stutter when you shoot someone to the chest(cba + ace3 only). I do not experience such thing, but people with weak computer do.

[whole cloud]

Maybe it helps against burst updates of static objects 🤔

[restive pilot]

I've only noticed this near complex geometries. you sure it's related to perf branch?

[minor silo]

On stable, this is present as you say, only in close proximity to the character, but on perf, I've got reports that it was the case even across long distances. Maybe it is an ace3 issue, gotta instruct people to try pure vanilla.

[inland dew]

same "flat/low" frametime in Kavala and 60+ FPS

me very very happy

[inland dew]

where have you been all this time, Dedmen...

[whole cloud]

Getting a piece of paper that says I'm educated which I have never had to use since then

[inland dew]

I recorded in 1440p CBR 120000 kbps 48 KHz 320 mbps audio with my old GTX1070 Ti, so 10-15 FPS less on the vid

• Firefox with multiple tabs, Discord and other stuff. well, the real usage scenario
  https://www.youtube.com/watch?v=m2B0ZaBMK28

[worldly badge]

Well, looks like I'm regreting not to follow this topic very often

[inland dew]

but it's very very frustrating to have lower and lower FPS over time, after many hours off uninterrupted gameplay on official Warlords, with objects/textures flickering, with like 100+ ms frametime and later it reaches just few or even 0 FPS and Arma simply freezes/crashes/closes and that despite 8 GB GPU, 32 GB RAM and unlimited/auto paging file

and not only objects/textures flicker, but they also simply disappear
more distant objects disappear like for good and closer objects continue flickering heavily

and that despite me using -flush many many times during gameplay

almost all Warlords players have to restart Arma from time to time and transfer their $ to somebody else, to reset their FPS to playable/enjoyable, for hours and then restart Arma again.
Players with lesss vRAM, RAM and disabled or manually set paging file to a certain (conservative) value in GB have to completely restart Arma more often than me

https://www.youtube.com/watch?v=rP_12Qx5nPI

https://www.youtube.com/watch?v=iH4D-QlxHhw

whether it's due to memory leak(s) or something else, or different problems combined, I don't know. but it is how it is

[restive pilot]

or it could be related to your malloc?

[inland dew]

4 kb blocks default Intel malloc or 2 MB blocks CMA AVX(2) malloc lead both to exactly same result, just that with Intel malloc it takes longer than with CMA AVX(2) malloc

[hushed seal]

I can't state how much of a difference this makes on big servers, the (server) FPS difference is amazing.

[hexed gorge]

@hushed seal make difference for clients too?

[hushed seal]

My computer is shit (for arma) so can't really tell. And was using v10 last sunday so could't tell.

[empty goblet]

@inland dew the servers were on old version while you were testing new client, check now with the same version

[inland dew]

It's been like this forever, on all the stable and perf patches, even when server and client patches were of same build, until now

But of course I will test v11 as well

Just need to find time for a very very long and uninterrupted Warlords session

[hexed gorge]

Looks like something on your side.

Just my layman's opinion.

[kindred radish]

Since switching from v9 to v11 we dont have the low CPU load anymore 🤔
https://gyazo.com/327ee8de62376d3d042019de7064ddbd
https://gyazo.com/d1a895077d4db9bf8069e94099ebb301

[empty goblet]

... the real question is, is the load doing something worthwhile 🤣

anyway DD found something interesting last evening, i'm sure he will post more about it next week

[hushed seal]

Unlocked server FPS by accident?

[restive pilot]

I wonder if this only happens in scenarios where a lot of entities spawn and despawn
I've played lots of long missions and I've encountered this problem many times, but iirc only on those where a lot of entities kept spawning and despawning (dynamic missions)
this happens a lot in large MP servers where lots of people keep connecting and disconnecting, so maybe that's why

[last frost]

Hello everyone... I have a problem when playing warlords. i experience stutters with high FPS. is that normal?..I used to run it perfectly in the past. has anything changed since then?. I tried performance build and nothing changed. Thanks a lot.

[restive pilot]

has anything changed since then?
alot

I used to run it perfectly in the past
how long ago?

[last frost]

maybe 6 months ago

[hexed gorge]

@last frost
You play on official servers?
Server is using mods?
How much memory you have, you have SSD drive?

[last frost]

yes i play on official servers. i did notice i have better performance on warlords redux.. i have 16gb ram and ssd drive. windows handles the pagefile. I use cma malloc with large pages

[hexed gorge]

6 months ago you was using default malloc?

[last frost]

No i was using CMA x64... I just got done playing on a warlords server and it was pretty stable... I installed the performance branch exe through steam this time and i think it is working..there were minor stutters when FPS got in the 40s. so hopefully it stays that way. thank you for the feedback

[empty goblet]

redux has barely any AI (fraction of what's on classic)

[hexed gorge]

May be just a wrong feeling

[last frost]

ahh i see didnt know that

[fickle geyser]

When I hold alt to place composition in Zeus at original position my camera is moved somewhere far out of bounds
[328202,323342,100000]

[whole cloud]

well if the composition is way far 😄

[fickle geyser]

It's not tho.

where it should be moved? to center?

composition.sqe
center[]={11895.555,98.586548,7035.3706};

Running ZEN tho. I will check if it's at fault here

[whole cloud]

ye

[kindred radish]

nothing changed except the .exe files (not a single line of code or any config changes)

[bleak garden]

There has been severe performance issues on my server. Since the announcement for 2.08, both stable and perf would output 1 FPS at 400% CPU use. Any ideas on how to diagnose this?

[empty goblet]

with 2.06.148877?

[bleak garden]

Arma 3 Console version 2.06.148877 x64 : port 2302

[whole cloud]

We just ran a 80 player event and (last 4 times server died) and today it went really smooth and no issues

There must be something special on a server, a profiling capture frame might tell why

[full nova]

how/where to check this?

[bleak garden]

RPT logs, CTRL+F search

[full nova]

cheers

[bleak garden]

Do you think messing with basic.cfg could cause such perf. issues?

[bleak garden]

How would one do that?

[empty goblet]

if it's messed too much , yes

[bleak garden]

I managed to get the diag_frame. Lots of wDisp. I have no reference to tell if long wDisp (200ms) is normal

[whole cloud]

nothing long is normal

a normal frame should never be longer than 20ms

and if a single small part of a frame is already 200ms, thats way not normal

anything more detailed than just wDisp?

[pale mica]

Had a INT_DIVIDE_BY_ZERO crash on our server running perf v9

the HC crashed first, then the server.

less than a second later, we saw the HC disconnect message

https://file.io/TbarxFD4Ohjb mem dump

[gritty wasp]

It’s v11 already with critical fixes

[empty goblet]

ye, move to V11 ... save us analyzing already fixed crashes

but still thanks for sharing the dump (just in case it is something else unfixed)

[gritty wasp]

V9 was 150+ server fps
V11 around 50-60.
Feels like it's around +10-15fps than half year ago. Less impressive but better than nothing.
Also as reported previously, CPU load higher on v11.

[empty goblet]

like i said before v9 had bugs, dont use it as any metric

use v7/v8 when comparing to v11

v9/10 were rapid iterations with issues

and it seems v11 has issues too

[gritty wasp]

I understand. I just confirm that v11 load more and give less. Seems like due to bug some CPU intensive checks were skipped in v9

[inland dew]

I also could observe lower server FPS on v11 vs. previous builds (v9/10 not included)

[bleak garden]

https://ghostbin.com/4t6yz
Here's the entire main thread pasted. Hopefully, it is helpful.

[pale mica]

So, V9 has indeed higher FPS, but is it more resilient than V11?

in terms of performance when lots of things start happening

[whole cloud]

v9 is known bugged, so you probably just shouldn't run it

[whole cloud]

That dump seems to be from when you opened curator display.
Also that dump is from clientside
It just shows that opening curator display is slow, which is nothing new

[kindred radish]

We had no single visible problem with v9 tho. Was running it for a week and not a single report of new problems ^^

[icy hedge]

that happened to me when a composition was edited on another map. It would just freeze my game

[bleak garden]

Drats, how would I get it serverside? Global executing diag_captureSlowFrame ['total',0.003]; in debug console did not bring the debug window to appear

[whole cloud]

on server it dumps to a file

[quaint flame]

anyway DD found something interesting last evening, i'm sure he will post more about it next week
What did he find ? 😄

[gritty wasp]

Something interesting

[silk pewter]

but it was last evening

[bleak garden]

Thanks, diag_captureFrameToFile was more helpful.

here is the server's diag log: https://ghostbin.com/Nyveg

[whole cloud]

-1013.42902
The frame starts at time -1013?! wtf 🤣

ah no, thats its duration

[bleak garden]

actually, ghostbin only got part of the file. I can DM you the full file

[whole cloud]

Holy macaroni that log file is absolutely completely F'ed up

[bleak garden]

i'm humbled

[whole cloud]

I hope thats ghostbin and not Arma :u

[bleak garden]

DMed full log

[hexed gorge]

How many players?

[gritty wasp]

130-150 pvp no AI

[hexed gorge]

Holly Bolly!

[silk summit]

I wonder what made V9 so much faster...

[whole cloud]

Today in dev stories..
Looking at profiling data and noticing that 2% of frametime on server is spent in Voice over Net. On a server that has disableVoN=1; in config?!?!?!

Turns out. Even if you disable VoN. VoN is still on.
The server still opens the VoN network port, Still spends quite a bit of CPU time waiting for VoN messages, still updates which players can hear which other players every frame...
if you want to really disable VoN, you need to set the (undocumented!) voiceOverNet server.cfg parameter to 0! And even then it still opens the network port and waits for messages on it, you cannot even disable that part.
Wow

[fickle geyser]

I guess you will fix that?

[empty goblet]

can't we just get rid of the p2p VON and get server side one? will be "simpler" than try fix it

[whole cloud]

VoN is initialized before server.cfg is being parsed (in the main parsing thing). But luckily server.cfg is parsed in 6 places! ┬─┬ ノ( ゜-゜ノ)

[empty goblet]

only ...

[whole cloud]

Oh Mr. Dwarden, did you know there was a network command to reload server.cfg on the fly? I didn't.
But that only actually refreshes some config entry, and which it does is not documented.

Oh that is actually documented https://community.bistudio.com/wiki/Multiplayer_Server_Commands I just never heard of it

[quaint flame]

Sometimes when I read these posts I feel like Arma 3 is just some alien tech that you are trying to decipher 😄

[whole cloud]

That was fixed already in newer perf builds

[silk summit]

He keeps going deeper and deeper into 10+ years of code and systems so pretty sure some of it would feel alien to most modern coders 🙂

[random isle]

10+ years is a fake news

2001 is when it all started

[hushed seal]

I feel that way looking at my own code from a week ago, can't even imagine someone else's 10 year old code.

[random isle]

100% agree 😄

[whole cloud]

fake news 🤣

[silk pewter]

T H I S 🤣

[kindred radish]

we should have another branch with whatever made it so much faster included

[empty goblet]

maybe, i wanted it but it never was finalized, asked for server configs (settings) and mods /keys/ and some other stuff to be re-load(able) w/o restart

[empty goblet]

actually longer... some code was like 1999 if i remember some stories

[silk pewter]

Poseidonnn

[feral harness]

Now that you mentioned, I vaguely remember even trying it

[whole cloud]

Mh server spending 18% of its time waiting for dynamic simulation update to be done by a different thread.
How much time does that thread spend doing the actual update? 0.1%..

How can it wait for 18% of the time for a task to be done, when the task itself only runs for 0.1% of the time

[silk pewter]

https://tenor.com/view/magical-magic-gif-13543499

[whole cloud]

I checked the v9 improvement, and it still seems to be working as intended.
But I'll need to do a larger scale test to see if its really still goody

[hushed seal]

for what its worth we had two three hour OPs with ~70 players on v9 running fine.

[quaint flame]

One day I really need to join a OP of that size. The OPs I usually take part at have like 17 players and even lower FPS 🤣

[hushed seal]

FPS is usually alright, depends on how much stuff is going on and how bunched up these 70 players are 😄
I can send you a nice clip from a air assault we did last week if you'd like.

[restive pilot]

How can it wait for 18% of the time for a task to be done, when the task itself only runs for 0.1% of the time
thread scheduler delay...?

[hexed gorge]

He starts to wait before simulation thread started to run.

Like to wait for a bus when the bus driver is still sleeping in his home bed.

[gritty wasp]

Is it even editable? I think file is locked while server is running.

[tawdry gazelle]

, you should get used to it... My company has some code to retrive a system's cursor image. It seperates the cursor into 8 modes, each mode has very much different procedures with more than 2000 lines of codes in one single non-class cpp file and still not finished all it's work... And they use plenty of XML configurations to MANUALLY!!! determine what mode should it use and parse the XML almost everywhere in every LOOP!... Finally I deleted all of them and use about 800 lines of code. No manually configuration anymore. Done.

[whole cloud]

Yeah, thats a bug

[whole cloud]

I verified the performance improvements from v9/v11
11408 objects (static road signs), 1 player
server fps

Enabled: 115-140
Disabled: 80-100

It actually distributes it really well, with enabled it checks about 2000 objects per frame per player, with it disabled it checks 11408 objects per frame per player

Much better visible when simulating more players
4 players

Enabled: 95-130
Disabled: 75-90

16 players

Enabled: 90-110
Disabled: 30-40

For my test I placed 11k buildings (sim type House) in 3DEN inside the mission.

The server tries to keep all these static objects synchronized.

Even making them simpleObject doesn't change anything.
The proper way to improve this would be having static objects (simple objects, visual things, H-Barriers, some smartphone laying on some table for just visual content) be clientside only.
These kinds of objects that never change, never get damaged, never move are still synchronized by the server and its a huge waste.

Sadly this is not very easy to do, you basically have to spawn them via script with createSimpleObject local.
I'm thinking about if I can add a checkbox for that in 3DEN.. Not sure how doable that is but it would be good.

Also doing this moving them to local only objects only becomes worth it when you do it with hundreds of objects.

If you like building whole enemy bases, that may become worth it

But of course, local only objects really are local only, you cannot change their state and expect others to see them, you cannot store them in variables and share them over network.

[silk summit]

You sir are a godsend! We are talking 30% to 90% improvements is most scenarios, that is a game changer when it comes to Arma 3!

And about simpleObjects being local only it is a very elegant improvement. You can't really change them during a MP match, not even add actions to them. Of course the server should still be able to delete/move them via the global commands but that is all and it won't happen many times during a match in 99% of the missions. Most mission on the steam workshop feature hundreds if not thousand simpleObjects so the improvement will be felt by most players

An EDEN checkbox with "Local Only" or something would be optimal too, just like you said

[whole cloud]

I just tried to repro a mission I played last weekend.

It had 1905 disable sim static objects that never move.

1 player: 600-800fps
80 player: 200-300fps
80 player with spreading: 550-750fps

Its not that much. But this is 1905 times the same object, and my "simulated" players are 80 times the same player, so the branch predictor is very happy with this.
Which it normally is not with different players and different object types

[autumn timber]

I'm not fully sure if I understand this correctly.
Does that translate to better client performance? Or is this only better server performance? (which kinda trickles down onto clients because the server may be more responsive with the same hardware)

[restive pilot]

I'm wondering how you managed to get 800 FPS

or was it just based on CPU time?

[whole cloud]

only server

[whole cloud]

Just unlock fps and play on a server with no AIs?

[rain moth]

perfect for frying an egg on the CPU cooler

[whole cloud]

🥼

[hushed seal]

that would lead to only minor improvement on pref branch, correct? Since its already skipping all objects who haven't changed state.

[whole cloud]

its not skipping. Its spreading all objects over multiple framess (5 currently) it still has to check them all some time. With this it doesn't need to bother
And the owner of the objects also has to check if they should send an update for them to the server, with this they don't need to check.

[quaint flame]

So damage state and similar will never be propagated over the network right?

[whole cloud]

yes

Thats why it should usually be coupled with disableSim or simpleObject. If you want performance, you want that anyway

Sadly the simple option is not available on many objects. And I think changing that now is not viable. But disableSim also prevents damage impacts right?

[quaint flame]

Yes

[hexed gorge]

Yes, but if you enable simulation it get destroyed if it had take damage for that.

So damage is computed somewere.

Like on a Vehicle disabled, if you explode 10 C4 on it, it will not move, but when you enable simulation it will explode and fly away.

Or AI unit.

[whole cloud]

Well you won't be able to make non-simple vehicles or AI units local only, so no concern.
And if the mission maker wants to do something like that then.. all power to them!

[hexed gorge]

I made all my simple objects local to the players, to the HC, and nothing on server. Also Buildings with door are local with createVehicleLocal, on clients and HC. Server have no custom buildings. I was expecting huge gains in performance and this not happened.

[whole cloud]

Your Arma is either bottlenecked at error checking, or its not.
If its not, then all this won't make a difference

but the data I got from recent large missions with 80+ players were all bottlenecked by it, it will matter to them

[hexed gorge]

With 80 players, the bottleneck are the 1905 simple objects?

[hushed seal]

Is there any specific data you need? We're happy to send you some (~70 player missions)

[whole cloud]

Test repeated but with "Local only" checkbox checked on all these objects.

80 players Disabled: 300-400 fps.

[hexed gorge]

Oh sorry, i understand now.

[whole cloud]

Yes. But at that point the server still runs with over 100 fps so it doesn't matter.
But to that add all objects that are normally spawned in game.

Being able to get rid of 2000 objects does matter

[hexed gorge]

I sadly don't have much players to do proper tests. But i can configure my server to 100% synced buildings or 100% local buildings. So i will try do get a better comparison.

[whole cloud]

Having these objects be local also reduces the JIP load. They don't need to be JIP'ed.
And also when updating a JIP message, server doesn't need to iterate over these objects messages

[hexed gorge]

When HC is off server have it's local buildings, like clients.

Because AI is on server, so it need the buildings.

[whole cloud]

Non-local signs with improvement disabled at 80 players

Server fps: 9-15

But thats still 11k objects, quite alot.

In the local only one it takes me 45 seconds to JIP.
In the global one it takes me 58 seconds

I'd say it becomes important when you have 2000 or more static objects

[whole cloud]

I think this is a cool feature, and I'm already 80% done with implementing it so yeah.. why not 😄

Sadly I cannot make it work on profiling branch, unless all players are on profiling too

[silk pewter]

I see flying objects otherwise 😄

[restive pilot]

built in terrain objects are completely irrelevant right?

[hexed gorge]

Please find that Arma is syncing all map default buildings and trees all frames to all players 😂

+1000 fps fix.

[restive pilot]

well terrain objects are special afaik. they're a special form of local...

local + sync updates...

[random isle]

I remember OFP days, when your in KA50 and you unplug your modem for 30 sec and fly around kill bunch of stuff, then plug the modem back in and you get like 20 kills 😄

[fickle geyser]

they're local but their IDs are baked into terrain.

[whole cloud]

Yes they are local

[fickle geyser]

So you can send network message #ID123 was updated, and it will update the local copy on every client.

IIRC.

[whole cloud]

Mh.. the mission objects are built into mission, i could also give them ID.
Then they can be local only but stuff like setting variables on them would still work

[fickle geyser]

globally local as I like to call them. (or I probably heard that term from someone )

[whole cloud]

If I call the checkbox "Globally local only" people will be slightly confused

[fickle geyser]

that would be pretty neat.

[restive pilot]

I thought that actually meant they're local...
i.e "createVehicleLocal"led...

[fickle geyser]

It means now I guess.

[fickle geyser]

but if he does above.

They'll be the same as terrain objects. kinda

[hexed gorge]

This is a Zen Buddhist talking.

[restive pilot]

oh ok...
I haven't been following this convo closely...

[whole cloud]

Also deleting them would work, but it wouldn't be JIPed so.. badish

At least i think deletes are not recorded

[hexed gorge]

"Local Only" objects can be exported to a script like we do today with Eden missions?

[whole cloud]

Yeah. But that Eden export to script script won't know they are supposed to be local, it will just make them normal objects

[hexed gorge]

If we want to create those objects after mission start we just need to use createSimpleObject ["CLASS",[x,y,z],true] or createObjectLocal? Or use some new function?

Thats all thanks 😀

[whole cloud]

Same as before

[desert olive]

sorry for stupid question, does player view distance affect which objects are synced over the network ?

or everything is synced nevertheless ?

[random isle]

[gritty wasp]

Will it crash? I think closed communities can set required version and force players to match. Public servers will suffer, yes.

Eden checkbox may help mission makers to create detailed bases or houses but will give nothing to most scenarios where just triggers, units and vanilla environment.
Anyway better to have such stuff.

[hexed gorge]

I believe view distance don't affect what is synced and what is not, it just affect what in rendered and show on screen or not.

I allow my players to use viewdistance from 500 meters to 15000 meters.

[silk summit]

I set my servers VD from 1000 to 2000, seems like a good balance between FPSs and realism

[random isle]

Note if you set server VD to like 500, then AI that server own will be blind past 500m

We keep server at 4k and client at 3.5k

[tawdry gazelle]

Would it be possible to set max_objects_checked_per_frame via basic.cfg or somewhere? For some low-end server, it could always let more objects delay more frames.

for example I have 10000 objects and set it to 100, the check would spread over 10 frames. If I play for more time and objects increased to 20000, it would spread over 20 frames.

[whole cloud]

Afaik only distance to player, not View distance

[whole cloud]

It will just not spawn the objects

[merry shuttle]

What is the reason that planes cause such a huge FPS drop? For example when I place a Black Wasp and a pilot on the VR island I have about 550 FPS before entering the plane. When I'm inside and turn on the engine FPS drop to about 360. On Altis it's about 20 FPS. That's with the perofmance and profiling build, but I guess it is the same with the normal 2.06 patch.

[desert olive]

I am part of an Arma3 group (~200) and we have been running the perf build (x64 linux) for quite a while, its been great so far until recently where server performance dropped to single digits, and the CPU is always at 100% load. I noticed that there was a change in how threads work or am I wrong ?

regular server build and v7 run great 50-100 FPS most of the time, v9,v10,v11 run between 0-5 FPS while standing in base with 1 player on

we are running on docker

[whole cloud]

There were threading changes yes, but they were in v8

if your problem only starts in v9 then thats not them

[desert olive]

yea my bad, I just double checked and v7 is the one running good

everything after runs bad

this issue is also super inconsistent, heavy missions like KP Liberation seem to run better than simple missions like vanilla Combat Patrol

[whole cloud]

How many CPU cores does your server have?

[desert olive]

20

how do core pinning in docker

[whole cloud]

And all are available to your server inside docker?

[desert olive]

so we allocate between 4-6

[whole cloud]

You could try launching with -cpuCount=1 -exThreads=7

That would disable the multithreading (it was previously disabled)

[desert olive]

will try

oddly enough, the more cores we allocate in docker the worse it runs

with 2 cores its somewhat playable

[whole cloud]

on my server I didn't notice any issues, but maybe docker causes issues with multithreading 🤔

[desert olive]

I will try -cpuCount=1 -exThreads=7 and get back to you

[rain moth]

is it a VPS or dedicated box?

[desert olive]

dedicated box

i9-10900K

[rain moth]

that one does not have 20 cores, only 10

you're thinking about HT

[desert olive]

I mean, Linux sees HT threads as cores

[rain moth]

they will be flagged as ht

[whole cloud]

That would be oops 😄
I need to check that

[desert olive]

ok with -cpuCount=1 -exThreads=7 is better, getting ~60 FPS while standing at base, ~60 AI patrolling a town, no HC

fpsMin is a bit concerning though, ~15-20

[rain moth]

check for ht under flags in /proc/cpuinfo

[desert olive]

after 5 mins in the mission FPS is now ~90-100 (limitfps=100), fpsMin is ~50-60

[desert olive]

there is ht flag yes

[whole cloud]

So with -cpuCount=1 all is well then?

[desert olive]

seems so

[whole cloud]

Oki I'll see if I can fix the HT detection next week. Maybe thats good?
Though seeing "oddly enough, the more cores we allocate in docker the worse it runs" that can't be the only issue

[desert olive]

I see that its not maxing out all the cores anymore

I will revert this change to verify

[rain moth]

you could also restrict the amount of cores to the container

or pin the container to actual cores

for testing / workaround

[desert olive]

thats what we do, we pin to threads on same cores

foe example 4,14,2,12

4 and 14 are same physical core

ok I removed cpuCount=1 -exThreads=7 and its back to 9 fps

[whole cloud]

can you run profiling build and get a serverside diag_captureFrame of that?

[desert olive]

sure

btw this is an example
https://metrics.7cav.us/d/NED-gV3Mz/arma3?orgId=1&from=1646310759097&to=1646311540523&var-Server=training2

you can see the diff between the two secenarios

[desert olive]

I ran diag_captureFrame 1; on server

https://cloud.rtrd.io/index.php/s/faEjxCa9NxS4KAg

this is the log file

[whole cloud]

mtPmj; 17.72403; 48.26907;"Lock wait: 0.000 ms"
48ms in multithreading

[desert olive]

I assume thats bad

[whole cloud]

But why that one.

  visRU; 17.72341; 48.33646;""
   mtPmj; 17.72403; 48.26907;"Lock wait: 0.000 ms"
   mtPmj; 65.99440; 0.06384;"Lock wait: 0.000 ms"

There are two here, and the second one is just fine

reee

[desert olive]

what does cpuCount=1 do in context of Linux server ?

[whole cloud]

number of worker threads is cpuCount-1, so count of 1 turns off worker threads

[desert olive]

if I omit this parameter it autodetects I assume ?

[whole cloud]

ye

[desert olive]

is there a way to print out the autodetected number ?

I am just curious if it autodetects 20 cores or only the 4 allocated in docker

[whole cloud]

Don't think so

[desert olive]

so cpuCount dictates the amount of threads in your new multi-threaded system ? or am I completely misinterpreting this ?

[whole cloud]

yes

its not new, it has been there for a loooong time. It was just disabled on linux

[desert olive]

I set cpuCount=2 and it seems to run fine as well, could it be that it autodetects all the 20 cores and spawns too many threads for 4 allocated cores to handle ?

[whole cloud]

could be yeah

I guess I need to log number of CPU cores

next prof it will log it together with memory on top of RPT

[vivid rune]

I think the number of detected cores can be a standard RPT message (at least for perf and prof)

[rain moth]

Perhaps it just checks the number of cores for the CPU model as reported by cpuinfo? Instead of the number of cores available. Like Liber said 🙈

[whole cloud]

Well, it is now? Not sure what you mean

[whole cloud]

if (!HyperThreading)
  fp = popen("/bin/cat /proc/cpuinfo |grep -c '^physical id'","r");
else
  fp = popen("/bin/cat /proc/cpuinfo |grep -c '^processor'","r");

[vivid rune]

I use the performance exe on linux. You wrote that only the next prof exe becomes this message. I would have this message for the performance executable too. That's all. 🙂

[whole cloud]

what

perf and prof are the same except the profiling code

so if I add it to prof, its also in perf

[vivid rune]

Ah okay. Thx for clarify this for me.

[rain moth]

seems resonable, both reports 8 in my VM with a 16 core CPU allowed to use 8 cores. Let me check what docker says as well

docker reports the VM cores, both with cpu count limit and when setting specific cores

so when using containers you can run into issues there

you can check actual computation cores using nproc

maybe check if nproc is available and prefer that over cpuinfo?

works with both VM and container, reports correctly when setting specific cores

seems like it was by default part of (at least) debian 8 "jessie" which was released 2015

[wispy shuttle]

For information, we had a PvP mission yesterday, with 85 PAX (no AI at all). Usually, with "classic" or older perf binaries, we had around 10-20 FPS. With the latest V11, we never went under 75! So I'd really like to thanks Dedmen for his amazing work on the linux server !
For information, our currents specs are: Intel i7-4790K (4c/8th) - 32GB DDR3 1333MHz - 240 GB SSD, on Ubunutu 20.04, TS server is hosted on the same machine (at Soyoustart)

[rain moth]

for extra performance you should probably consider a more modern CPU and better RAM 😄

[wispy shuttle]

Yep, it's planned for next month 🙂

(we have the money now 😛 )

[rain moth]

hetzner has a sale with no setup costs atm so might want to consider that. offtopic for this channel though 😄

[whole cloud]

Go hetzner 😄 I went from Kimsufi to Hetzner with AMD, way more powah

[wispy shuttle]

Exactly what we are looking for (https://www.hetzner.com/dedicated-rootserver/ax51-nvme)

[rain moth]

yep, no complaints other than troublesome Intel NICs on some older machines

[hexed gorge]

Spawn 500 AI with the same uniform and same visible gear can give more server or client FPS than spawning 500 AI with random uniforms and random visible gear?

Most of the AI will be in the same area.

[gritty wasp]

I don’t think so. Maybe less GPU mem usage for model and textures.

[restive pilot]

well the cache misses will probably be fewer too, so it can improve the performance a bit

[rain moth]

there might also be differences in complexity, lods etc

[vivid rune]

Hi, this is maybe something small but if when i log in in my linux server as admin with "#login", the log in message fires two times in the logs

[whole cloud]

cant repro

[rain moth]

do you use list of allowed admin steam ids or password?

[desert olive]

is there anything in v11 client that's worth switching for ?

[silk pewter]

as far as I heard: perrrrrrrformance

[rain moth]

all fixes and performance boosts in v1-v11 not part of regular binary

[random isle]

https://www.youtube.com/watch?v=iMa6S7WCgyw we see this alot as well

[whole cloud]

Thats not new. Seen this happen for years with ace cookoff

[bleak garden]

My perf issues have been fixed. Changing to a competent server provider does wonders to your FPS.

[cold vale]

IIRC ACE cookoff uses setDamage 1 on the ammo projectiles it spawns and certain types (like the RHS BM-21s) just stay in the air forever. The fix I've tested is to spawn projectiles with a straight down trajectory so they immediately blow up to circumvent usage of setDamage 1

[restive pilot]

I asked Dedmen to improve triggerAmmo to work with everything, so you don't have to use setDamage anymore...
hopefully that fixes the issue

[whole cloud]

https://open.spotify.com/artist/0y1oZaJdi2jBS4AAZWvOF0?si=d7069831626d45bc

Mondkalb

[silk pewter]

yeees…?

[whole cloud]

Mondkalb!

[silk pewter]

Mondkalb?

[hexed gorge]

It not does that already? "Triggers a munition such as shells, bullets, missiles, rockets and bombs"

[random isle]

i dont use ACE

[hexed gorge]

From Arma 3 forum.

[silk pewter]

Arma 3?

[hexed gorge]

There is a Mondkalb in Arma 3 forum.

User.

[restive pilot]

it doesn't trigger for some stuff

[silk pewter]

I know, I met the guy 😉
I was asking about Ded's unusual reaction "Mondkalb" "Mondkalb!" 😃

[autumn timber]

As a programmer, it took me a few years before i realized it was Mondkalb and not Monkaldb 🙄

[silk pewter]

now that you mention it, he looked a bit like My Squirrel…

[autumn timber]

Not even joking 🤦‍♂️

Either he's really short or... you have a huge squirrel

[gritty wasp]

Addons in your pack can use same method for triggering ammo. Or it can be something else that cause them to stuck.

[vivid rune]

list of allowed admin steam ids (only one entry). Using 32bit linux performance server excutable (64bit doesn't run on Phenom II).

@whole cloud BTW: THX for the Linux server performance improvements: I played yesterday Dynamic Bulwarks (Many spawned and despawned enemy ai, lots of permanent building destruction between the waves/rounds) and my optical monitored thread distribution shows a nice equality over the cores. And the server/the ai was more responsive.

[vale shoal]

Our server just dies because of massive BattlEye issues - we're using the latest perf, here is the server RPT: https://avmg.eu/dl/a3altis_2022-03-05_20-01-33.rpt

Acually we're having massive issues with BattlEye in past weeks. Idk if it depends on perf/prof for smthing else

[regal ruin]

Moved to #arma3_troubleshooting

[whole cloud]

This is #perf_prof_branch ?

[regal ruin]

I was looking at this when deciding which channel to use, If it is offtopic I apologize

[knotty moth]

try #hardware_vs_arma

[whole cloud]

I'm not sure which.
You have trouble so.. probably #arma3_troubleshooting
But definitely not this ^^

[regal ruin]

I will move it, my bad guys 😅

[whole cloud]

But not many people use TrackIR so might take a while for an answer

[strange badge]

It does

[autumn timber]

But not many people use TrackIR so might take a while for an answer

Tbh I have been an avid FreeTrack user (instead of TrackIR because I am cheap 😛 ) since Arma 2 and I have been forced to stop using FT ever since I started using any optics over x2 because they were literally unusable.

Any minimal head movement while aiming down sights resulted in a HUUUUGE view shift, meaning I literally didn't know where I was aiming nor how much I should move my head to go back to the previous direction.

While there have been voices that said that I should use a shortcut to toggle FT on/off manually, that's yet another thing to think of in CQB and completely not worth it, and it made me ditch FT completely 😦

I would appreciate if it would be possible to, for example, check a "disable TrackIR while aiming down sights" box in settings

[cold vale]

Generally that is solved by adjusting your profile so there is a nice deadzone in the center so if you keep your head relatively in the same spot in the center it locks in like when TIR is disabled.

[silk pewter]

like with an optic, turn the cap backwards 😛

[regal ruin]

TrackIR software has a pause hotkey, if you turn toggle off in trackir and the set the key when aiming in arma 3 to hold it also works

Ive also found a really satisfying setup for the trackir, no neckbreaking deadzone adjustments needed. Dm me if you want the recipe

[feral harness]

why not just share it

[regal ruin]

I can do a write up. Give me some time

[pallid island]

Is there a list that explains each entry/tree of diag_captureFrame? Or only available to Bohemia?

[silk pewter]

https://community.bistudio.com/wiki/diag_captureFrame
takes to ↓
https://community.bistudio.com/wiki/diag_captureSlowFrame

[pallid island]

Thanks, but anything more detailed about the subtrees?

[silk pewter]

I have nothing more afaik

[pallid island]

understandably it would be a massive task to document it all I assume

[silk pewter]

maybe a dev can dig 👀

[pallid island]

[whole cloud]

Such a list will not be created no

[silk pewter]

too much internal info?

[whole cloud]

ju

[pallid island]

sad, but understandable

[whole cloud]

2.06 profiling branch had 257 commits! 🥳

[silk pewter]

don't break the repo! how many were fix fixes? 😄

[whole cloud]

😢

[silk pewter]

… oops?
nonetheless, this is but testifying to the love the team puts in it! :3

[whole cloud]

Profiling v0 tomorrow will remove the big stuff that was added since v8, it will be re-added in v1

[vivid rune]

I assume that in vanilla linux server 2.08 will not contain the MT stuff, yet?

[whole cloud]

correct

[rain moth]

v0 tomorrow

[whole cloud]

https://tenor.com/view/cat-light-computer-gif-21223179

[rain moth]

but no public branch update on steam yet...

[whole cloud]

https://cdn.discordapp.com/emojis/875891136848355409.gif?v=1

[hexed gorge]

2.08 anytime 🐣

Two more days because he needs to compile changelog.

[empty goblet]

takes tea time till v1 is out ... nothing else matters

[rain moth]

v0 release when? 😄

[whole cloud]

https://cdn.discordapp.com/emojis/742599803606925332.gif?v=1

[rain moth]

good thing our server update script can opt out of applying perf binaries at the end for now then 😄

[silk pewter]

https://tenor.com/view/cat-computer-typing-fast-gif-5368357

[rain moth]

poor cdlc_private_beta , it looks sad now

no love for depots there

[rain moth]

new profiling branch update

[whole cloud]

2.08.148925 new PROFILING branch with PERFORMANCE binaries, v0, server and client, windows 32/64-bit, linux server 32/64-bit

• Sync with 2.08 main branch update
• Previous Profiling branch exclusive improvements are still present
• Some Profiling branch features from after 2.06 v8 were removed, they will be re-added in v1

if you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

[rain moth]

Not sure if it's a propagation issue or not but switching to profiling added back EBOs

[restive pilot]

is 2.08 even out yet?

[patent dome]

[rain moth]

yes

[patent dome]

This is worrying.

[restive pilot]

I literally looked at dev.arma3 10 mins ago 😅
it's there now

[rain moth]

now it switched back to PBOs, propagation issue most def

[rain moth]

I looked at steam cdn metadata 😎

[rain moth]

https://dev.arma3.com/post/spotrep-00102

[patent dome]

https://tenor.com/view/sanford-fred-heart-attack-nervous-gif-24657978

[rain moth]

😄

Product Version: 2.8.0.148925 in exe metadata 😛

Version: 2.08.148925 in RPT

[patent dome]

Yea, likely a propogation issue. Manifest GUIDs are equal for both stable and profiling.

[rose moth]

This has reasons, which I'm sure a certain programmer will elaborate on 😉

[rain moth]

gief Windows 9!

[whole cloud]

I guess I missed one

[nocturne obsidian]

what was the diff between perf one and normal branch? because when i tried perf it didnt change a thing

[rain moth]

#perf_prof_branch message

[nocturne obsidian]

no like the actualy difference between the normal branch and the performance branch like what does it even do?

[rain moth]

that is the difference 😛

you can find the changelogs in the forums as well