perf_prof_branch

[spiral pond]

#captureFrame (from server)

tricky to understand the layering - seems one group has ~6-7 ms for some pathfinding 😐

[whole cloud]

Yeah it doesn't understand coroutines.
ai sim is basically like scheduled scripts. It might suspend and jump to another ai group. But the profiling doesn't understand it

[spiral pond]

would be good to get some #captureFrame from high populated PvP and PvE servers

to see how well MT works on servers

[whole cloud]

But thats a manually confirmed full dump right? and you also have a small dump too?

[quaint flame]

I have sent you everything that was in the log folder

and yes, it was manually confirmed one

[spiral pond]

the sleep block is to make sure the server remains within the set fps limit, right?

[whole cloud]

yes

Can anyone confirm invisible objects in Eden editor, virtual arsenal or AoW museum?
That might just be me?

[somber plank]

anywhere specifically?

[whole cloud]

Everywhere for me.
No-one else reported it yet, so it must be something specific to me

The extDB crash seems to be related to the terrain/mission. Not to extDB itself

[kindred radish]

Tested my old server with extdb3, no problem.

Not sure if it really matters: I still have fpslimit = 999 because i wanted to test my new CPU.
Before this perf build the server ran with 500 FPS+, now it moves between 150 and 200.

Client also runs on the same pc, but also did before when i got ~500fps.
Yaab test gave +8 FPS more (7800x3d)

[knotty wraith]

what are the downsides of using fpslimit = 999?

[whole cloud]

Higher limit, higher cpu usage.

My test server ran on 600fps 🤔
Would be useful to get a few captureFrame's of that. But if you run profiling you'd probably have even lower fps

Clients now use more CPU power, so that could influence the server.
And they also both use more threads, limiting cpuCount to half of the available cores could help. But would hamper performance too

[empty goblet]

i was advised to not deploy it until some 'fixes' 🤣

[whole cloud]

Nevermind I'm just stupid
My view distance is on 5m

[heavy vortex]

best performance setting :P

[knotty wraith]

the first crash went to PM

[silk pewter]

+10 FPS!

[harsh grotto]

[silk pewter]

I was talking about the 5m viewdistance
I did not test this perf yet 😄

[harsh grotto]

what if with each profiling update, he is simply lowers the viewdistance, posing as an improvement in performance?

[silk pewter]

:sus:

[whole cloud]

Xeno has some crash with attachTo. Some building was attached to something, or.. not attached to something. I'll need to repro that tomorrow
Anyone else tested complex MP yet? like Exile or koth or domination or liberation or smth like that?

Revo is the capture frame UI (tooltip drawing), so doesn't matter (:harold:)

Dimon is a crash inside the memory allocator? and the multithreaded AI stuff, that is interesting thank you!
It is probably related to a problem I saw before. I even have a fix for it but its commented out because it seemed to not be needed :harold:

[crisp wagon]

@kindred radish go check exile

[whole cloud]

All these I can quickly fix tomorrow, then we'll do a new build.
Give me more issues please :harold:

Also have reports of crashes in 3den, but no details yet and I didn't find any in short test. But that could be related to changes that are not on prof

[heavy galleon]

Do cheat codes work in editor?

[whole cloud]

everywhere

[heavy galleon]

trying to figure out why it freezes when I spawn in our d-20

[whole cloud]

Well maybe not loading screen 😄

[heavy galleon]

last time we solved it by removing fucton of triangles from hiden stuff, but not an option here

given it does not lag once you spawn it once, I assume it's a model load issue

also, does it work in perf, or only prof binary?

[whole cloud]

only prof has prof

[heavy galleon]

copy

will report back what I find, as I have no idea what the graphs mean

well, it does freeze for like 5s, but the sframe is never triggered

testing with frame, it does show it

[heavy galleon]

seems like I found a case where it does not trigger SFRAME even with freeze.
If I try to use FRAME during the freeze, also nothing.

Any ideas what else I might try?

open editor
enter SFRAME
spawn D-20
game freezes
nothing

[whole cloud]

In editor you could also just run diag_captureSlowFrame script

[heavy galleon]

will give it a go

[heavy galleon]

did not trigger

[whole cloud]

Interesting. 😄

[heavy galleon]

interesting with zero informational value unfortunately

[whole cloud]

for me, sframe triggers already by just opening debug console in eden

[heavy galleon]

that's why I changed the trigger threshold to 0.5

the freeze is several seconds

[spiral pond]

(local DS) once AI vehicles started to get simulated. then server crash soon after

[whole cloud]

seems model loading is shNew
A slow frame with "shNew" and threshold like 0.00001 should capture the model loading, if that is the problem

[heavy galleon]

on it

[whole cloud]

crash report 🥺

The low fps is weird

[heavy vortex]

Is that 3GB capped?

[brazen oyster]

Dedmen is the goat

[heavy galleon]

So yeah, that triggered it.
Would you be so kind and took a look at it? I have no idea how to orientate in it

[whole cloud]

send me logs of it and i look tomorro

[heavy galleon]

willco, thx

[knotty wraith]

Now the HC has fallen off - the server has survived as long as it works

[kindred radish]

Tested without client running, same FPS

[2024-10-30 18:48:25] [info] CWZ-ServerMetrics: Build:152340 | Runtime:3min | Players:0 | AI:10 | allMissionObjects:298 | AllVehicles:147 | ObjSimulated:72 | #soundonvehicle:0 | AllAliveEnt:171 | activeSQFScripts:6 | Player-SB/Vehicle-SB:0/0 | FPS:126
[2024-10-30 18:51:26] [info] CWZ-ServerMetrics: Build:152340 | Runtime:6min | Players:0 | AI:10 | allMissionObjects:298 | AllVehicles:147 | ObjSimulated:72 | #soundonvehicle:0 | AllAliveEnt:171 | activeSQFScripts:6 | Player-SB/Vehicle-SB:0/0 | FPS:112

bat params start "Arma3" /high /min /wait arma3server_x64.exe -enableHT -limitFPS=999 -debug -hugePages -bandwidthAlg=2 -loadMissionToMemory

[full nova]

just gonna wait for Dwarden to update the official server to use latest Perf_profiling

what should we do to kill time

[gritty wasp]

You probably hit GPU limit. I have +10fps in 1080p window and 0 in 4k fullscreen because of GPU.

[whole cloud]

Can you get me a captureFrame from a gpu limited scenario?

[fast hornet]

Looks like our server has no problems with v6 and extdb3 .. or at least the dev box started without a crash. Didn't test anything else yet.
18:57:55 CallExtension loaded: extDB3 (C:\SteamCMD\steamapps\common\Arma 3 Server\@ExileServer\extDB3_x64.dll) [1.0.3.2] [1.0.3.2]

[heavy vortex]

Testing locally, ordered some AIs to get in a car. One of them paced backwards and forwards for 10 seconds or so and then started to walk on the spot. Still doing it now.

The behaviour here was never good but I think that one's new :P

[vivid rune]

Vanilla +9.4 fps (+17%); heavy moded (simpl crater, graphics/sound/animation mods) +8fps (+25%)

[heavy vortex]

Moved car a bit. Second attempt did the weird pacing in random directions for even longer but then eventually figured it out and got in the car.

[spiral pond]

old hardware and DS on same machine - not our fast real DS

[heavy vortex]

subsequent attempts seem fine.

[whole cloud]

But I still wouldn't expect below 20 🤔

[spiral pond]

PZKW is many tanks and other vehicles

hence R3vo asked to check it 🙂

[knotty wraith]

the server is crashing - I'll roll back for now

[whole cloud]

Ah right, might still be physx then

[spiral pond]

doing another run with it now to see if crashes again or not. if not, will see to get some server capture with the low fps

[whole cloud]

If you get more crashes then send me all of them

[spiral pond]

ok crashed almost immediately 😄

[whole cloud]

But you were running other things.
What do you think is the cause? It being a dedicated server, or the specific mission?

[spiral pond]

bit weird also to get mpMessageDetailsServer-2024-10-30_19-11-38.txt before the mission started with just:

Server values
time = 0.000

[heady plinth]

is this only mp performance stuff or will it apply to sp?

[whole cloud]

Yeah. It is printed on cleanup. And before mission start it does a cleanup. Bit annoying, maybe fix someday

[whole cloud]

most of it is sp

[heady plinth]

better ai load spread?

[spiral pond]

will try another mission - i'd assume its scenario related

[whole cloud]

better many things. Changelog is above 😄

[heady plinth]

yeah, i went over it and have lots of fantasies in my mind, just trying to confirm that i am not making shit up in my head

old man scenario and contact (sandbox mission) are probably the no1 performance tests for sp scenarios. since even on my good pc i get max 35fps

[quaint flame]

Do we have a recommendation for treshold when using captureSlowFrame?

[heavy vortex]

Dropping a server-local crate (physx) is a bit odd.

Stops on ground initially and then does a wiggle after a second or so.

Like slight sink then rise back up to ground.

[hybrid fern]

It's all interesting, I suppose, but maybe you will try just for PvP servers, and disable the option "Super low texture settings", so that players do not abusive mechanics with transparent trees 🙏
About all sorts of other problems that have been hanging in the reports for 10 years, I don't even want to remember 😟

[whole cloud]

The AI optimization is basically

Group1: Run FSM, Create pathfinding grid, update targets. Continue with Group2.
Group2: Run FSM, Create pathfinding grid, update targets.

To

Group1: Run FSM. Create async task for pathfinding grid. Continue with Group2.
Group2: Run FSM, Create task for pathfinding. Continue with Group1
Group1: Create task to update targets. Continue with Group2
Group2: Create task for targets. Continue with Group1
Group1: Wait for targets to be done. Continue with group2
group2: wait for targets to be done.

So basically, the pathfindings and update target steps of AI groups, can now run in parallel, but FSM and scripts don't.
I already saw the pathfinding in parallel is an issue, if they try to do the same grid cell at the same time. Thought I fixed it but did it wrong

[whole cloud]

estimate how many fps you expect, convert it into frametime, and use that

[quaint flame]

Ok.

144 fps it is then 😬

[coral pumice]

The fabled optimization dlc is upon us

[fast hornet]

since you mentioned grid cells btw., my v6 client just crashed upon joining a v6 server as only player.. too many objects in a grid cell. Didn't have logging or anything enabled sadly

[whole cloud]

Was that a error popup, like a windows error message? or a RPT log entry?

[fast hornet]

error popup upon crash

ah wait maybe it's in the windows event log

[willow trail]

Hello everyone, can you explain in simple words how to install it, I don't really understand it

[fast hornet]

hm nope (for the eventlog thingie :D)

[whole cloud]

In steam, beta branch in the game properties.
But I suggest to wait till tomorrow

[restive pilot]

https://community.bistudio.com/wiki/Arma_3:_Steam_Branches

[whole cloud]

too many objects in grid cell, doesn't seem related to the changes here. It wouldn't do that

[quaint flame]

ded do you want all crash files via dm?

[fast hornet]

weird.. never saw that error before. Ah well. I'll enable logging and if it reappears i'll send you the memdumps as usual 😄

[whole cloud]

I want all crashes, I don't care how they get to me

[quaint flame]

Sending a bird right now

[heady plinth]

i have 60fps in old man while driving on the road with 6k view distance

will need more tests to see if this is actually real

[whole cloud]

What did you have before

[restive pilot]

Is that good or bad?!

[heady plinth]

between 40 and 50 at best

but i will revert to stable in a few mins and retest, just to make sure i'm not overhyping

[whole cloud]

That seems about what I saw in yaab

[heady plinth]

pretty sure i never had 56 fps in this spot

dont get me wrong, there are some big performance dips in the scenario. lowest i got was 45fps, but still. this is big

i have never played OM so smooth before

[whole cloud]

Hope its not one of the improvements I'll disable tomorrow 🤣

[heady plinth]

just setting view distance to 6k on ranoa was giving me less than 60fps

woop, game crashed in contact 😄

[whole cloud]

I'm eagerly awaiting your files 😄

[heady plinth]

i'll try to repro

hm. crash again right on mission start

[whole cloud]

which mission?

[heady plinth]

carpe noctem

first time i saw the cutscene, skipped it, went to map screen, then crash on the map screen with the funny heatmap

[whole cloud]

That's probably enough for me to repro

[heady plinth]

on 2nd time it crashed right after loading. sending files

3rd attempt, no crash so far

50fps in scenario, i cry

[elder viper]

Someone just reported that same error in a server I'm in

[heady plinth]

woop, scenario crashed again. was switching in the map screens again, same scenario. maybe this is a mission bug. will test with dev branch now 😄

[heavy vortex]

Perf is good here, I'm on the worst part of the map in Antistasi and client and server are both capped.

Had what looked like a hang for about a second on the client at one point though.

No further AI issues, maybe a freak.

[whole cloud]

AI in itself is a freak

But I already have one crash in the pathfinding stuff, so something is going wrong. That could break AI just like that, not finding its path.
Will be off tomorrow and we'll see

[heady plinth]

🤔 getting this when testing OM on dev branch

wrong channel. ignore, i guess 😄

[whole cloud]

known bug. We decided to not apply some fixes we had

[heady plinth]

damn. that's cluttering my debug log in OM right now 😄

just weird that it didnt happen on performance branch?

or is the fix applied there?

[whole cloud]

yes, no

[heady plinth]

well, if it's known then at least it's not my fault!

[gritty wasp]

I can not. Cheats does not work in YAAB.
Also I don't think that this is Arma fault. That is 4k on 1080ti.

[whole cloud]

cheats work everywhere

[gritty wasp]

not for me

It works in old man but not in yaab

[whole cloud]

Well you need profiling exe for it to work, not the performance one.
I know its not Arma's fault, but I want to see how Arma behaves in that case. It supposed to run AI and sound in parallel while its waiting for the GPU to get done, I don't know if thats working right

[gritty wasp]

I know. I run profiling. In Old Man I see black notification that cheat activated and after frame captures I have slow frame diag UI.
In YAAB I don't have notification and it goes till the end.

[whole cloud]

🤔 Mh weird. I shall investigate

[gritty wasp]

I also have flickering objects in 3den

[whole cloud]

While the camera is standing still, or while moving?

[gritty wasp]

still

tanoa placed man

[whole cloud]

Shadows enabled, do the shadows flicker the same, or differently?

[gritty wasp]

same

cursor also flicker

[knotty wraith]

I've had the sound in my headphones turn off suddenly several times already.

[gritty wasp]

cursor was flickering probably because I tried to record screen and alt tab. Now it is stable. But keep unit flickering

[whole cloud]

How many fps is your game running while flickering?
Above 100?

[gritty wasp]

59

[digital nimbus]

game crash after role select

[acoustic creek]

Game freezes with mod load up & then without mod load up

[whole cloud]

Please pack the crash report in a zip and send it to me

[whole cloud]

A freeze should also be creating crash report files

They are in

!rpt

[frozen tundra]

Arma RPT

Arma generates a .rpt log file each time it's run, which contains a lot of information like the loaded mods, or any errors that appear, this log file can be very useful for troubleshooting problems.

To get to your RPT files press Windows+R and enter %localappdata%/Arma 3

Additionally see the wiki page for more info: https://community.bistudio.com/wiki/Crash_Files

To share an rpt log here, please use a website like https://pastebin.com/ (Set expiry time to 1 week or less) to upload the full log, that way the people helping you can take a look at it and try to figure out the problem you're having together with you.
Note: RPT logs can hold personal information relevant to your system, the game or others.

[acoustic creek]

Should I send it in this chat or another one?

[whole cloud]

For now you can switch away from profiling branch, it'll be fixed tomorrow

[whole cloud]

my DM's would work too

[acoustic creek]

Okie dokie, will do after the operation I have now.

[vivid reef]

i just send you 2 recent crashdumps

[potent moss]

Can't even start up A3 with profiling

[carmine stump]

same here

[whole cloud]

We're reverting on Steam.
Make sure I got all the crash dumps if you had any.
v6 stays available on Google drive if you want it.

v7 tomorrow with fixes for all the things that I got crashdumps for

[opal hound]

Anything in particular causing them?

[inland dew]

read the thread man

[opal hound]

...which thread?

[loud garden]

The one where you're now.

[opal hound]

Nobody has said what causes them beyond starting the game/after role select?

[whole cloud]

I didn't check most yet

I know AI has a problem.
And that one would be more likely to hit when many AI units initialize, like they do at mission start

Could be that it's all that

[inland dew]

flickering (not my video)
https://youtu.be/z1_MGmFLPgI?feature=shared

[plain trout]

performance is superb on a map that was usally bad. 8k viewdistance no difference to 2k viewdistance

[ashen helm]

hey I recall this happening when player was moving to fast with a FOV too high (like while holding the zoom out button)

are you playing with custom FOV too?

[heavy vortex]

I saw a bit of that on non-custom 1080p, but not as bad.

Wasn't entirely sure whether I was hallucinating because it was very occasional.

Affected rendering of other units too, not just the player object.

Ah yeah, same thing later in that video.

[scarlet oak]

You people are doing the Lord's work.

I feel like in just the last couple of years there are so many things that have been fixed that I just accepted as permanently broken.

Keep up the excellent work.

[heavy galleon]

was the patch reverted or?
I got another update on steam

[fickle geyser]

yes.

round 2 tomorrow

[heavy galleon]

Ah, steam just being super late for me

[empty goblet]

you can always experiment with the binary from dropbox , just in case you want find more bugs before Dedmen wake up and start fixing them

[heavy galleon]

If I didn't have to wake up in 5 hours so I can spend 9 in uni, I would, gladly.

[empty goblet]

don't worry, i'm sure more bugs will be found anyway by someone else

[heavy galleon]

As long as they are AI related, I don't really care

[spiral pond]

profiling observations - dedicated server

[spiral pond]

profiling observerations - client

[vivid rune]

Hi, for what specific is this thread? Discussions/observation for the prof? bug reports? only for you? For all?

[spiral pond]

mostly for Dedmen, but also anyone else to learn/contribute with profiling = aka finding performance issues for Dedmen to look into for possible further optimizations

one has to identify such "a problem" in the first place, but then also narrow it down as much as possible to Dedmen can reproduce it or get a a really good idea whats going on

[empty plover]

sorry if this is wrong place to ask but has the lag issues been solved? I noticed one of my GUIs takes 2 seconds to load when it used to be instant

[whole cloud]

do a captureSlowFrame on your UI. "the lag issues" is too generic. I don't know what the problem is

Okey all catalogued.

• The improved building simulation, would try to load terrain objects on multiple threads which is bad. And I suspect there is also an issue with building destruction.
• AI multithreading has problem where a cell was prepared for pathfinding, and another AI tried to read from the same cell while it was still being written to.
• Another AI multithreading, There is some system where AI can lock cells, as far as I can see that is some unused old feature because I cannot find anything that actually checks for locked cells.. But when two AI's try to lock the same thing it went bad.
• Something in AI Visibility multithreading, but it seems to just be a problem with filtering of what things to check. Should be easy to fix later.
• We now do multithreaded lineIntersects checks, for which we need to animate entities. There was bug with wounded unit texture things, which might initialize in there which they can't do.
• The JIP queue thing, with accumulating damage. Which I already mentioned before and was worried about. I don't know why its bad yet. I'll just disable the multithreading only for that message type

[spiral pond]

https://ui.perfetto.dev/

the visualization tree(?) is quite nice to see what subparts/scope are "heavier" (in relation) but also provide some info on "special case" ("why" a vehicle is more demanding in the given situation)

[whole cloud]

Called flame graph. Because its like a (in this case upside down) flame

[spiral pond]

ie cLRSu for this vehicle extends the computation (whatever it is and for whatever reason)

[whole cloud]

its checking terrain height including roads
Its just doing some simulation that checks the height of something. That part doesn't seem to have its own scope

kju you posted a block of like 6 prof screenshots yesterday in here, please move them into one of your threads, that way I won't miss it

[nocturne obsidian]

Whats this about Enfusion Stuff?

[vivid rune]

This is Arma 3 stuff. Not Arma Reforger.

[nocturne obsidian]

Someone said That the Perfomrnace in Arma 3 has massively improved?

[vivid rune]

Yes but currently it is buggy. Bugfixing is ongoing for now.

[nocturne obsidian]

How Do I access it Perf Branch?

And Also how did they Improve the FPS? By how much?

[whole cloud]

If you want to read about it you can just scroll up. See last pinned message, and the messages posted below it

[hardy tree]

Woah, good stuff for A3. Can we expect more of the optimization passes in the coming years? Seeing that there is roughly a 10 FPS boost that is huge and makes Arma 3 even more enjoyable

[hybrid cradle]

Expect nothing

You won't be disappointed

[nocturne obsidian]

Wait

when does the Update go live?

[worldly badge]

See last pinned message, and the messages posted below it

[whole cloud]

The more we optimize, the less optimization opportunities remain.
Alot was done now, there might not be much more left to optimize.

[hardy tree]

Yeah that totally makes sense, appreciate all the work

[harsh grotto]

v7 today?

[heady plinth]

can always still swap the renderer with you know what :x

[ashen helm]

Godot

[empty goblet]

ic, so you want to support everything 😁 https://docs.godotengine.org/en/stable/contributing/development/core_and_modules/internal_rendering_architecture.html

[silk pewter]

vFinal!

[empty goblet]

vFreak ... cause Scary week 🎃

[spiral pond]

[request] ability to save .trace directly from diag capture dialog

[harsh grotto]

https://tenor.com/view/cat-pumpkin-kitty-kitten-cute-gif-19026347

[silk pewter]

mandatory PumpkinHead
https://forums.bohemia.net/forums/topic/220061-jboy_pumpkinhead/

[whole cloud]

with a windows save file dialog?
Or just save to file in RPT folder, like captureFrameToFile?

Ugh.. I would need to add UI buttons, which would be a config/data change

[whole cloud]

I thought about adding some halloween easter egg into the build today.
But I was too late with that idea 😢

[spiral pond]

what about alt+click on "copy all" button? just save to rpt with same naming schema as captureFrameToFile

[silk pewter]

I know the feeling

[whole cloud]

Well I can But doesn't it become too much modifier key chaos at that point?

[spiral pond]

dont think so. for one only very few ppl will use this anyway and those are mostly advanced users
atm the workflow with the cheats is very convenient - only exporting to trace is a few extra steps and gets you out of the game

another approach/option or addition would be to allow the cheats also to save traces directly

[whole cloud]

I have put it on my list

[fair ore]

spoooookyyy optimization

It's more frames but it also has fangs

[whole cloud]

The crashes were the easter egg.
No better way to scare/frighten Arma players, than making their game crash

[quaint flame]

we will document that, don't worry

[fair ore]

I SAW THAT

[whole cloud]

2.18.152344 new PROFILING branch with PERFORMANCE binaries, v7, server and client, windows 64-bit, linux server 64-bit
- Fixed: velocity script command would return 0 for PhysX objects - https://feedback.bistudio.com/T185618
- Fixed: customChat could fail if a player is removed and added in a certain way - https://feedback.bistudio.com/T185296#2690883
- Fixed: getObjectTextures/Materials would not work with Simple Objects - https://feedback.bistudio.com/T185884
- Fixed: Possible crash at game start
- Removed: Improved building simulation (caused instability, pending further investigation)
- Removed: Two parts of AI multithreading (caused instability, pending further investigation)
- Removed: Multithreaded AI visibility calculations (caused instability, pending further investigation)

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh
🎃

[heady plinth]

Removed no

[fair ore]

spooky!!!

[vivid rune]

Now I have roughly +7fps(+12%) in YAAB.

[knotty wraith]

oh where is my yesterday 50% on the processor... again 35 😥
Is it possible to return v6 on the client?

[whole cloud]

You can get v6 from google drive.
But its very unstable as we've seen

[loud garden]

Performance vs stability... The choice is yours! 😁

[knotty wraith]

Well, the problem was on the server version, I didn’t see any problems on the client version))
I'll close my eyes to this - as long as it works for me)

I, as a client, have nothing to do with this in theory:
- Removed: Improved building simulation (caused instability, pending further investigation)
- Removed: Two parts of AI multithreading (caused instability, pending further investigation)
- Removed: Multithreaded AI visibility calculations (caused instability, pending further investigation)
I don’t have AI on the client side

[whole cloud]

Yes you do have AI on client side. And buildings too.

[knotty wraith]

AI on the HC side. HC uses server build v7

[whole cloud]

You still have AI on your client even if you have no AI running on your client

[inland dew]

our body is AI that we command?

[knotty wraith]

I hope any other version of the client will not break the server

[whole cloud]

No. I don't want to explain it now

[silk pewter]

basically, AI is calculated on client as well and the result is ditched if the AI is not local, if I get things right

so, pure waste

[inland dew]

i remember there was an official singleplayer mission in a2, where you were leading ai infantry group and could switch into the body of any of them in real time

[silk pewter]

like, teamswitch?

[inland dew]

idk

[silk pewter]

the "don't calculate AI if not used" fix would bring wonders in matter of performance
but there were issues to fix iirc, so it is not simple

[empty goblet]

just calculate tiny bit in background when not used

[inland dew]

bi mining on players pcs 🤔

[knotty wraith]

the v7 server is working - no crashes yet, I’ll basically check v6 on the client right now, I hope no offense)

although it’s very strange - after this time, the processor jumps from 35 to 50 and the fps is at yesterday’s level

[harsh grotto]

the processor is warmed up and ready to work

[fair ore]

meow!! That would be cool

[silk pewter]

afaik it's in the pipes, or in the drawers at least ^^

[whole cloud]

That fix is already there and still active

But in most cases we just can't do that. And there is no fixing that.

[vivid rune]

Contact first mission no issues so far.

[spiral pond]

dedicated server is stable here so far

noticed some helmets flipping a bit up and down from both body dead and units in static weapons

[silk pewter]

wat, but I remember you told me it almost made it but was disabled because reasons (I believe local calculations were needed somewhere, etc)

[whole cloud]

It disables itself if someone needs the data

[spiral pond]

14:16:38 Game started.
14:24:11 Server load: FPS 0, memory used: 1974 MB, out: 0 Kbps, in: 0 Kbps, NG:0, G:0, BE-NG:0, BE-G:0, RQ:0, Players: 1 (L:0, R:0, B:0, G:1, D:0), JIP (T:0 Q:6882)
14:26:20 Server load: FPS 0, memory used: 1978 MB, out: 0 Kbps, in: 0 Kbps, NG:0, G:154, BE-NG:0, BE-G:0, RQ:0, Players: 1 (L:0, R:0, B:0, G:1, D:0), JIP (T:0 Q:6497)
15:04:12 Game finished.

had only #monitor 1 active - server console had these two. is this some automatic special case handling?

or done when server triggers captureFrame/captureSlowFrame

[knotty wraith]

my reasoning is as follows as I understand from my players: those who have very weak processors do not get an increase, if not worse, and judging by the Xeno report, those who have a very good processor also do not receive a bonus. As far as I understand, some do not have the ability to boost, while others already have boost by default. And these edits are specifically for processors like mine - i3-13100, which still have the opportunity to reveal their potential

[vivid rune]

To evaluate an increase in frames is hard to see in a dynamic situation like a multiplayer session. Use YAAB for that.
The new changes results in multiplayer more in a smoother gameplay.

The server should perform more stable in terms of constant server fps

[knotty wraith]

not to be verbose, these two links to videos of my processor running
on v5
#arma3_feedback_tracker message
and v6
#arma3_feedback_tracker message

[inland dew]

optimization doesn't automatically mean higher fps
can be that fps you already had, dips not as much and not as often and for not as long, thus more stable fps/frametime = smoother gameplay

[knotty wraith]

my FPS is in the positive only because the processor has become more loaded

[vivid rune]

What do you mean with "more loaded"?

[knotty wraith]

first crash

[knotty wraith]

35% >> 50%

watch two videos and CPU load

[vivid rune]

And what is bad on this? This is a wished behavior. More multithreading->more utilization

[knotty wraith]

Did I say somewhere that this is bad?

I visualized a real difference

[vivid rune]

The "only" in your post implies that you expect fps increase from another reason.

[silk pewter]

anyway, Trust in Dedmen® guys
if he removed some improvements, it's to improve stability while he fixes crash possibilities
I believe he know what he does 😎

[eternal mantle]

@whole cloud Hi, Does this mean that Arma 3 finally has multi-threading? Can you be more specific about what this item means?

Added: Replaced Real Virtuality multithreading system with Infusions version

[whole cloud]

Arma 3 always had multithreading

The new system from enfusion is more efficient, and allows us to also do asynchronous work
Makes it easier to multithread more things

[naive osprey]

Put latest profiling 2.18.152344 on our server, crashed within a minute of server starting.
We DO have intercept, and well as an addon called futurecept - Unsure if crash is due to them, or profiling itself.

Unfortunately cant upload the crash to discord as its 8MB - let me know if external links are not allowed.
https://www.mediafire.com/file/saz8p9uu4w3ggak/Profiling-2.18.152344.zip/file

[whole cloud]

If you made an extension, you can check by yourself if its your fault

[naive osprey]

I was not the one who made the extension, however I do have the symbols loaded and cannot see any relevant info to it in the stacktrace

[whole cloud]

Mh something with a player slotting in on role selection

Oh sending JIP queue to a player

[knotty wraith]

give me a channel so I can upload screenshots and post the link here

I don't understand the restrictions on screenshots

[orchid pine]

because people have abused it in the past sadly, one bad egg ruins it for everyone else\

but maybe you can inquire in #discord_server so that we aren't cloggin up perf

[knotty wraith]

the server is very bad - fps 8.9... the mission is balanced - I have never seen such drawdowns
I can't upload screenshots

The funny thing is that on HC the fps is normal, namely on the server it’s 2-3 times lower

but the server has no units

[spiral pond]

@knotty wraith cant post links to imgur or so either?

[knotty wraith]

I haven’t tried it - it’s confusing that I’m here to help, but they limit me

I can argue who is an ArmA veteran and provide evidence

[fickle geyser]

you're making a fuss out of everything and are so whiny

[heavy vortex]

The visibility check results aren't dumped, at least. It often maintains a full independent target list. There used to be some code on perf that disabled this under unknown conditions.

[fickle geyser]

there's also disableRemoteSensors command

[spiral pond]

and do #captureframe plus #captureSlowFrame total 0 0 0 from chat as admin
then drop capture frame log to this channel

[heavy vortex]

yeah but it's busted IIRC

[fickle geyser]

idk if it does the same as optimization but forcefully

[heavy vortex]

I forget whether disableAI "CHECKVISIBLE" acts locally.

[fickle geyser]

all disableAI is local

[analog acorn]

The restriction on posting images isn't about you personally. It's restricted for everyone, and it's to prevent people spamming irrelevant pictures. No one is trying to stop you helping.

[knotty wraith]

vanity - does it mean passionate? then I agree

and so your next jokes on me - to one place, free me from yourself

[spiral pond]

@knotty wraith please do this

[knotty wraith]

we need to start the server for profiling - people are playing, it’s only tomorrow

the server is running on performance ver

[spiral pond]

btw you can run DS on your own computer too next to your client (with enough RAM)

[knotty wraith]

but what does captureframe have to do with the server? If I run it as an admin, will it be a client screen?

[whole cloud]

It writes the captured data to a text file on the server

[vivid rune]

It does not capture images. It capture data from the server.

[knotty wraith]

Sorry, I already forgot this - I posted a text file here

I'll try harder tomorrow

[spiral pond]

info here: https://community.bistudio.com/wiki/Multiplayer_Server_Commands and on linked pages

with the data you can get this

and from that a better understanding why low fps happens

[analog acorn]

The server doesn't render graphical frames, but it still works on a "frame" basis for its simulation. Everything is processed in cycles. You might also hear these referred to as "ticks", when referring to servers.

[knotty wraith]

I’m already scared, I just let you know that what I run every day and monitor every day doesn’t work like that... I know how to write scripts, but I’m not ready to understand how an arma engine works

he told me what to do - I will do it, nothing more)

but while I was trying to write everything here, the server was released, everything returned to normal)
Do you also think that I’m causing a fuss?
I'm ready to leave

[vivid rune]

Wow.

[whole cloud]

I can only fix a problem if I know what it is.

[heady plinth]

can you fix my crippling anxiety?

[knotty wraith]

if there is a crash yes - otherwise not at all

just reality

[whole cloud]

Yes at all. That also applies to other problems.

"server running badly" doesn't tell me what's wrong.

[gritty wasp]

Discord have very useful personal blacklist feature exactly to hide such authors

[spiral pond]

anyone did get a mpMessageDetailsServer.log with actual content with the 2.18 prof yet?

i should have had a few sessions without server crashes and mission ends, but nothing logged

unless it happens only with #shutdown or endmission (mission not cheat)

[knotty wraith]

he has a personal grudge)

[whole cloud]

it happens when server cleans up objects, which happens at mission end or shutdown

[spiral pond]

like i used #missions, #reassign and such - maybe those dont trigger it

[whole cloud]

they should

[spiral pond]

ok weird. running SPE campaign now - will see it how it goes with those

[knotty wraith]

the second crash fell in PM

[knotty wraith]

what did you do during this time?

I'm rolling back to v 5, wait for information from veteran29, he should tell you what the problem is

[fickle geyser]

I did look into a feedback on upcoming CDLC. Hope your day was productive too ;)

[spiral pond]

a little bit of data corruption from sframe + copy-all export

vs old format

also not consistent for the same data

should a job subscope only happen in worker threads or fine also in main frame?

maybe i just missed it before, or something on the server is different?

now it added an entry - but the info cant be right

[knotty wraith]

The worst thing is to put your project in CDLC

[inland dew]

this topic is not about that, like at all

[knotty wraith]

I don’t see your messages about the server running on v7

Usually you can't be erased from here

[gritty wasp]

I had such json export issues tomorrow. Buffer overflow?

Perf with reverted AI stuff yaab gives +3fps on 12700k for me(+10 was on v6)

[spiral pond]

2nd mission completed and then in lobby for 3rd used #shutdown

[whole cloud]

main thread also works on jobs, if it has nothing else to do

[empty goblet]

very unstable server wise ...

[empty goblet]

acknowledge the #rules 😁

[knotty wraith]

Fans don't respect the rules

[analog acorn]

Some channels have stricter permissions. In here and #dev_rc_branch for example, you can't upload files even if you've acknowledged the rules.

[empty goblet]

true fans are rules bound...

[fair swan]

After this most recent update, audio is cutting in and out on the game. The audio makes a loud pop sound, and instantly after all game and VOIP goes dead.

Its happening multiple times without any input

[opal hound]

I was just about to ask about this aha. I just had issues with very delayed sound

[fair swan]

There is also a new bug where some players when respawning, are being killed again at respawn timer 00:00 and getting stuck in a loop of constant death and appearing at grid 00000. Only select players. Its become apparent this may only be happening to users on profiling branch since it has only been 4-5 players.

I will also add that the servers been up for one year, never had this type of issue. Server runs warlords, probably related since that game mode is a pile of open loops and scripts. But nothing has changed as far as server side mission or scripts etc.

[fair swan]

Yeah it was weird. A violent pop, sound died, then it popped, and came back on and there was a delay in the sounds from other players

Happened a few times now

Probably in my best interest to put server on stable now that the hotfix is out lol

[opal hound]

Did not get the loud pop

[silk summit]

I have noticed a 0.3 seconds delay in the sound of gunshots from the time I click to the time I hear the sound of the first shot, happens with all the weapons but only on the first shot, afterwards seems to work as intended

[fair swan]

Do you use a USB headset

[opal hound]

Technically? It's USB receiver to wireless

[woven loom]

Try connecting directly to analog out

I'll try if i remember tonight, I usually use a USB amp for my headphones but it can connect directly too

[thin wyvern]

Same bug have Reforger too.

[heavy galleon]

Not sure if performance related, but we encountered an issue when rvmat for our shell would load... Restart game, rvmat not loaded, restart game, rvmat loaded.
Afaik my model guy was on latest perf when he noticed.

[carmine stump]

AI became dumb

now my own AI rarely fire at enemies.

[heavy vortex]

Just got another video of the unit flickering bug. Let me know if you want a copy of it.

Doesn't appear to affect vehicles or thingX boxes, only units.

[spiral pond]

did A3 disable the speed limiter to turn as fast as possible as infantry as some point?

from what i recall dexterity is/was meant to control that (cfgWeapons)

I can do a 360 in <1 sec now

also AI in tanks/vehicles in WLs seems to have problems to advance next location (they arent blocked by anything. just dont move, turn in cycles or move back and forth). WLs + complex terrain is always tricky for AI, but seems different with perf branch atm

getting also very unstable fps on the DS. even more so strange as not much is happening. not that much AI, not much fighting if at all

server is not happy about PX 😬

[spiral pond]

it seems when you get close to AI vehicles, it changes the PX calc (intensity/type/or at all?)

getting also mini freezes for a couple of seconds when beaming close them

• server fps goes from 20-30 down to ~10 for that time

causes mini freeze

what this about?

[whole cloud]

Steam wants to tell us something and triggers the callback. It seems its telling us to refresh the stats/achivements? 🤔

[whole cloud]

Multithreaded phsyx might solve that, but its not there yet.
But this must not be a new problem then

[spiral pond]

not saying its new - just stating what i am observing with profiling

[spiral pond]

seems to repeat every x seconds. are there some WL specific Steam stats or just a general Steam API call to happen all the time?

[whole cloud]

steam would control when it calls us. So not gamemode specific. But I haven't seen that in any of my testing 🤔

[vivid rune]

@spiral pond Are you running mods or is this vanilla?

[spiral pond]

"-mod=!Workshop\@CBA_A3;!Workshop\@@WW2_SPE;F:\Workspace\A3Data\@SPE_Dev"

[quaint flame]

I have noticed steamCbK being quite high as well.

[whole cloud]

I got 4 crash reports, all JIP queue related.
And 1 report of a freeze due to AI but its unclear why it froze.

And a couple out of memory crashes, because AI tried to pathfind to -nan,-nan,-nan

Heh..
I forgot to cleanup a thing from a couple weeks a go.
a couple performance improvements were disabled. Including the rendering improvement I worked on first

[vivid reef]

its public holiday

do something relaxing

[whole cloud]

wha? in which country

[vivid reef]

almost whole europe

[sick arrow]

Only Catholic countries and Catholic parts of countries

[quaint flame]

Not here...

[digital nimbus]

great work with the v7 build it's really stable and i'm noticing an increase on fps

although the v6s' performance was even better, but there were sporadic crashes

[whole cloud]

Besides that one unclear freeze.
All crashes so far were server side only. So running it on client is probably fine.
But we will revert steam to v5 in a couple hours, just to make sure the weekend is fine. But you can still manually grab it from google drive.
We'll do the next iteration on monday

[hybrid cradle]

Appreciate you

[fair ore]

We love dedmen

[silk summit]

Dedmen for president

[knotty wraith]

Am I guessing correctly that the text file will be created if there is a problem on the server?

I run these commands - nothing happens

[silk pewter]

☁️ ☁️

🌄 🏞️

[whole cloud]

Found the jip queue bug Its a fun one.

jip queue has 4 elements A B C D We want to remove B, so we delete it A NULL C D Then we need to fill the gap, we do that by memmove'ing the last 2 elements, up by one. But. Memmove does not zero out the data that was moved away. It really just does a copy. ```
A
C
D
D

So we cut off the last one.
Oh, there is still an element in the last one, when removing an element from the array we need to delete it properly so we don't cause memory leaks.
Lets delete D ```
A
C
NULL
NULL
``` Okey and now adjust the array size  ```
A
C
NULL

oh. oh no.

[whole cloud]

It will create it in RPT folder. And also print to RPT that it created a file.
Only works if server is running profiling, And client must also be #perf_prof_branch v6 or newer (otherwise it executes a different command on the server due to a bug)

[knotty wraith]

on the server profiling 7, I have profiling 7

[whole cloud]

That should work then

[knotty wraith]

work immediately after running the command? or is the command waiting for the problem case to be reflected in the text file?

[whole cloud]

#captureFrame is immediately

[knotty wraith]

I launch - nothing

I understand that this is no longer relevant

While I'm playing on version 7 - until it crashes completely

[empty goblet]

in meantime, anyone has idea why Tracy from A3S profiler flat out instant crashes on Windows Server 2019 ? Visual C++ are installed so that shall not be it

[inland dew]

profiling means profiling, not performance...
it's normal that on perf nothing happens

[spiral pond]

send server rpt of that session via DM. on client you can use frame and sframe cheat (shift + minus on numpad)

[knotty wraith]

What is DM?

[spiral pond]

direct message

[knotty wraith]

I’ll repeat the question - is this still relevant?

[spiral pond]

yes

[knotty wraith]

wait - everything will be

[knotty wraith]

you turned out to be right - I seemed to be uploading a prof exe, but I checked the server status - generally perf at 5, where did I upload it... I appreciate you my friend)

Original output filename: Arma3RetailProfile_Server_x64

and I play like this, I think at 7 - how wonderful everything is)

[void badger]

So canon perf release is back to v5 for server and client? Any additional testing needed with any of the specific binaries?

[whole cloud]

v7 should be fine on client, if you want you can grab it from google drive and play with that.
Only thing I'm aware of on client now is flickering of units sometimes.

All other bugs I'm aware of are serverside only. So if you want to test something, keep running v7 on client

I also found a memory leak in the JIP stuff.

Memory leak:

struct IJIPTask
{
  virtual void Do() = 0;
};

No memory leak:

struct IJIPTask
{
  virtual ~IJIPTask() = default;
  virtual void Do() = 0;
};

[knotty wraith]

everything works). Should I run this when I see that the server is dying for unknown reasons?

[whole cloud]

When server is low fps. You run that. And it will write to the file what the reason for low fps is

Probably also run it multiple times, more data more better

[knotty wraith]

#captureSlowFrame total 0 0 0 Does it catch server latency itself or how does it work?

[whole cloud]

It is basically the same thing, it will also just capture one frame

It is
#captureSlowFrame
scope filter
time limit in seconds
offset

You can do
#captureSlowFrame total 0.1 0
Which means capture, if total frame time is longer than 100ms.
So if your fps drops below 10, it will capture one frame. That can also capture lagspikes that only last one frame

[knotty wraith]

I'm very sorry for the flood... but just because of yesterday I found this reminder of me
https://forums.bohemia.net/forums/topic/186695-a2a2oa-patch-18122015/
Thank you for not banning me - I promise in the future to be more informative without emotions

[heavy vortex]

Does it batch the queue removal to reduce the memmove spam on large queues, or are removal events too rare to bother?

[whole cloud]

Old thing would move every message separately. (If queue has 5k elements, and the first one is deleted, it would do 4999 single-element moves)
New thing now batches it around the gaps (one big move, for one removal). I just fumbled the end of that batching a bit

[heavy vortex]

oh, it didn't even use memmove before...

My followup point is that if you're removing 10 entries then you still only need to move each element once.

[opal hound]

I suppose you could use a linked list/queue instead but it's higher memory overhead

[heavy vortex]

nah, you just make (ordered) list of replacement points and rebuild the array in one pass.

It's only useful if there are enough removals to batch though.

I'd have thought that most JIP removals came from deleting objects, and so you'd get dozens at a time from all the entries related to that object.

[whole cloud]

Difference is moving 4 elements in 16 byte blocks each.
vs moving one 64byte block at once.

Fewer big blocks, instead of many small blocks

[heavy vortex]

Yeah, but we have 50k entry JIP queues?

[whole cloud]

If you set a public variable, it deletes the previous message of that variable

[heavy vortex]

oh, it adds it to the end?

[whole cloud]

The bottleneck is iterating over the list. Not removing entries from it.
Using a linked list instead, would make the bottleneck even worse, and fix the thing that isn't really a problem

[heavy vortex]

I actually figured that it overwrote public variable entries in-place due to a bug we had.

[cloud sky]

What about using linked list there? Or would it destroy performance e.g. with cache misses?

[heavy vortex]

but maybe that was unreliable guaranteed packets again.

[cloud sky]

Oops you just mentioned it lol

[opal hound]

true, I'm probably underestimating the size of the list

[feral harness]

do they have to be in order? you can just swap B with D and resize

[whole cloud]

Yes the jip queue needs to be in order

You can't just move adding an item to a vehicle inventory, to before clearing that vehicle inventory

[feral harness]

how long is the queue?

swap with one after until riches last then resize?

[whole cloud]

between 1k and about 300k before the game dies

[orchid pine]

https://tenor.com/view/skeleton-fall-skeleton-meme-gif-2635742470048290046

the game at 300k

[whole cloud]

Thats what it did before.
That's the bad way. Because it does thousands of swaps of single element. Instead of doing a bulk move

[feral harness]

if you have references to the queue mem move would destroy them no?

[whole cloud]

Not destroy them, it would put different things into the reference because the elements get moved around.
But there are no references

[vivid reef]

move last item in the gap

[whole cloud]

#perf_prof_branch message

[feral harness]

how often it needs to be processed, once per join?

[whole cloud]

Every incoming message that goes into JIP queue

[vivid reef]

wouldve been too easy

[feral harness]

make it a map?

priority map

[whole cloud]

I don't see how that solves anything

[vivid reef]

priority should be the oldest one if i understand correctly.

[feral harness]

easier to manipulate

[whole cloud]

I don't see how.
I know of priority queue, but map?

[feral harness]

yeah sorry queue

[whole cloud]

It still must be fast to iterate, because as said above, iterating over it is the main bottleneck. Removing elements is easy.
Making deletion faster and iteration slower is bad

[feral harness]

havent looked into stl containers for awhile

[cloud sky]

The priority queue is a container adaptor that provides constant time lookup of the largest (by default) element, at the expense of logarithmic insertion and extraction. Would the constant lookup be too slow in this context?

[whole cloud]

constant time for the largest element.
But we need to iterate all elements in order. Not just pick the largest

[vivid rune]

Do I understnad the mechanism of JIP right:

• mission starts
• player 1 (already in mission) take a item out of box into his inventory
  -> delete item in box
  -> add item to player 1 inventory
  goes into the JIP queue
• player 1 put this item into a car inventory
  -> delete item in Player 1 inventory
  -> add item into car inventory
  This action goes also into the JIP queue
• a new player 2 joins -> for him all is as the mission starts
  -> Player 2 receives the JIP queue to prozess all changes to be "up to date"

So correct?

[whole cloud]

yes

[gritty wasp]

Then we need to fill the gap, we do that by memmove'ing the last 2 elements, up by one. But. Memmove does not zero out the data that was moved away. It really just does a copy.
Classic C++.
func named memmove not moving but copy.
remove not removing but moving.

Committee spend years to write standard and lose naming game.

[vivid rune]

So JIP queue will always grow over time?

[whole cloud]

Generally yes

If you delete player1, and the car. All the inventory messages are also deleted. And the queue will shrink.
But deleting objects is rare

[gritty wasp]

Reported sound bugs server side related?

[vivid rune]

Have all Objects (player, car, item, etc.) a unique id? (I asume yes).

[whole cloud]

Have not investigated them yet

[whole cloud]

Entities yes. Items are not entities

[cloud sky]

"memmove() may be your best choice as it does not care about individual array elements. It just moves the bytes, and does so at the top speed your memory bus allows. You cannot get faster than the memmove() approach. Any native C++ vector operation would need to move the individual objects one by one, calling constructors and destructors along the way to ensure proper semantics, which can be significant overhead."
(from StackOverflow)

[whole cloud]

That perfectly sums up the optimization I did there

[heavy vortex]

Oddly enough magazines do have IDs, although I'm not sure they're global.

[analog acorn]

One day we may get the ability to actually do something with those IDs :U

[whole cloud]

The ID's are unreliable and the inventory system is chaos

[vivid rune]

If all objects in the queue can be identified correctly, a function can go through one of these objects per frame an make optimisations to the queue:
Let stick to my example. It can delete the

• adding to the inventory of player 1
• deleting to the inventory of player 1
  from the JIP queue
  So the JIP queue is almost optimal.

[whole cloud]

We don't know from which slot the item was removed.
And we don't know to which slot the item was added.

Say
Add flashlight
Add flashlight
Remove flashlight

Player now has flashlight on their handgun.

Okey we can merge the first and last and only keep the middle one.

Oh, now player has flashlight on their rifle

The inventory is a total mess.
If it were that easy, we would've already done it

[woven loom]

Damn, i do not envy you 😅

[vivid reef]

but how does it know wich flashlight to remove?

the flashlight got an ID but where it goes is priority? first flashlight to primary weapon, next flashlight to handgun?

next to inventory?

[whole cloud]

depending on what player currently has in inventory.

You could iterate the queue and simulate all inventory changes and at end try to find out what remains and remove the others but that is so super error prone

Spending weeks on trying to optimize this is not worth the time. Plus the risk of causing desync or duping bugs if you do something wrong

[heavy vortex]

Oh wait, this is why there's no removeItemCargoGlobal, isn't it.

You either add or you clear. So the JIP queue never inflates. Probably :P

[whole cloud]

https://community.bistudio.com/wiki/addItemCargoGlobal
negative number is remove

[heavy vortex]

ah shit

[vivid rune]

OK for my understanding:
if player 2 jips, and see player 1 , the position on which weapon player 1 has attachments is controled by the actions in the JIP queue? Not a dedicated JIP messages like "player 1 has attachements on main weapon and seconday..."?

[whole cloud]

Maybe better to understand example.

Add a bandage to player in any container that has space. (Player uniform is full, doesn't fit. Vest? Vest is also full. Backpack? Backpack has space, ok I'll put it there)
Remove bandage from players uniform.

You can't merge these. The first message knows an item is added but not where.
The second knows where it was removed from.

But in order to find out that the bandage that was added, was removed again. You need to know exactly where the add went to. And you don't know that.

[whole cloud]

yes

[vivid rune]

holy ...

[analog acorn]

That's such a horrible way of doing it

[whole cloud]

Reforger does it much better.
It just has a shared state on the server.
When you join, you ask the server "give me full current inventory" and you get it.

Such fixes are not feasible anymore.

[heavy vortex]

I assume the idea was that most units would still have an inventory close to the config-defined state.

So you save network traffic by storing the difference.

but now it's just horrible and the worst case is nasty.

[vivid rune]

JIP exists since OFP, right? So for 20 years the missions were not so extensive as today.

[whole cloud]

yeah

The inventory system is also (mostly) new since A3.
I don't know how it worked in A2

[analog acorn]

They built a new inventory system after a decade of experience and still decided to make it work like this!?!?!?

[vivid rune]

never change a running system

The whole server - client architecture of reforger was changed mid in development, as I understand. It had flaws that cannot fixed with the old architecture.

[inland dew]

well, Dedmen has to work with how it is
he's not the one who took decisions related to that back then, so

[heavy vortex]

Ok, tested and figured out script workarounds:

• For boxes, clearXXXcargoGlobal is confirmed to clear out the related single-item JIP commands.
• For units, _unit setUnitLoadout getUnitLoadout _unit will clear out the single-item JIP commands.

So if you have an arsenal system in your mission, do the setUnitLoadout on exit and the JIP queue shouldn't inflate.

[vivid rune]

The only (terrible) solution for optimisation I see is:
Let's say the container is a car with a lot of change messages to the inventory.
A function would check these, store the current state of this container, then delete it (so it is deleted from queue) and add a identical new one with all current items added.
The change can only happen if not in LOS of any player.

[heavy vortex]

clearXXXCargoGlobal & re-add works.

[vivid rune]

But what the implications is for scripts is, idk

[heavy vortex]

But note that addItemCargoGlobal ["FirstAidKit", 1000] adds 1000 JIP messages. So don't overfill boxes.

[whole cloud]

I even have the code for doing that https://github.com/WolfCorps/WolfLogistics/tree/master/addons/main/functions
Create preset from box (works with any inventory container), clear out the box, fill preset into box.

[analog acorn]

why in the world is that 1000 messages and not one message to add 1000 things

[whole cloud]

Mh actually mine ignores the fill level of magazines, and will fill them full. But not much work adjusting for that

[whole cloud]

Because in 99% of cases, a inventory interaction can only move one item. Only script has the ability to do multiple

[heavy vortex]

Similarly, running ACE will naturally result in a much larger JIP queue due to all the fucking bandages :P

[whole cloud]

Atleast now with the JIP queue changes, its much less of a problem. Atleast until the point where the server dies

[woven loom]

"The inventory knows where it is, because it knows where it isn't"

[whole cloud]

I want to add something to exportJIPQueue, that tells you the number of messages per object.
So you can see if you have some specific very heavy object in your mission to optimize away

[heavy vortex]

Turns out that the Antistasi arsenal box is ok because it gets cleared.

But players repeatedly using the arsenal to change loadouts may be worse.

[whole cloud]

But have to note when doing that.
You're generating hundreds of messages that need to be put into jip queue, all in a short timeframe.
Before profv6, that could freeze the server while it incorporates it into the queue, if the queue is already long

[heavy vortex]

As long as they die at some point it shouldn't be too bad :P

I don't think there would be much need to intentionally do a clear & rebuild on vehicle cargo, because they tend to be cleared naturally.

And yeah, spamming clear & rebuild "just in case" could easily be much worse.

[vivid rune]

Yeah it was a example. A Box would stay longer.

[orchid pine]

well sadly I am the Arma equivalent of Iron Man, so me and the JIP queue are gonna have to have some words

https://tenor.com/view/iron-man-tony-stark-robert-downey-jr-marvel-superhero-gif-16430080

[heavy vortex]

btw, when testing with exportJIPMessages, note that commands may not immediately enter the JIP queue. That tricked me initially.

which is silly given that it was the starting point of the topic :/

[whole cloud]

overfill boxes is also such a thing.

We don't know if a addItem network message actually succeeded.
It might fail.
So you might have
AddItem (now have 5)
AddItem (now have 5)
RemoveItem (now have 4)

You might think you can merge the last two. But then you'd have 5 items instead of 4.

[whole cloud]

flooding the jip queue

exporting a jip queue with 150k elements takes 90ms on my machine.
I guess I will also move it into async 🤣

Okey I did this now.
it prints the number of messages that would be deleted, if the object with that netId would be deleted.
And I also added debugname (which includes classname) to the object list at the top.

So now you can see that if you'd delete that ammobox, you'd get rid of 1000 messages in the queue

And I fixed the dashes spam

[gritty wasp]

It unveils Type codes for cheat makers?

[whole cloud]

uh.. no?

[analog acorn]

It's the same useful information that was already present, but the secret information is now censored by removal rather than being replaced with a big block of -. #perf_prof_branch message

[night mulch]

bro this is what i was telling you about, steam also does the infinite download loop thing that causes stutters all the time, this really ruined the game for me

i really hope it can be fixed now that it's been identified 😢

[orchid pine]

i also experience the infinite download loop often - and would be able to provide a repro if needed.

But it isn't to the point where it shuts my game down, but it does cause microstutters.

[night mulch]

Thank you very much, its an issue that i believe a lot of people has but is not talked about much 🙏

[night mulch]

It doesnt crash the game but can really ruin the experience when theres a stutter every 2 seconds, especially when you’re doing animation work/modding

[uneven kernel]

I have that bug, last time we discussed it I'm pretty sure we concluded it was something to do with Steam downloading compositions you're subscribed to?

#dev_rc_branch message

#dev_rc_branch message

I just try to keep the amount I'm subscribed to below like 30, above that it comes back. Seems like the more I'm subscribed to the longer it goes on for, so I usually boot my game then leave it for 5 minutes until the stutters stop 😆

[night mulch]

Yeah i also believe the download loop thing came after the introduction of steam compositions… but it would need to be investigated more

The thing is most players dont really subscribe to compositions, so majority of players probably dont have this infinite download loop bug which is why its not really much reported

But it can be really ruining the game for those of us that has this bug

Like for mission makers of course we would need to subscribe to a lot of compositions

[night mulch]

The loop will come back everytime a mission starts

Even in 3den

[plain trout]

I only have the loop when the game starts and all the other reports I read about this haven’t mentioned something else. You might be seeing a different issue.

Is it different between profiling and vanilla? because if not this isn’t the channel for it.

[night mulch]

well, you might be seeing a different issue as well

its not much reported so there might be some variations

[night mulch]

nevertheless its been identified theres something from steam that causes mini freezes in this channel, so it might be related to this bug

[gritty wasp]

I'm not about all info covered with dashes.
Type_194 - Weather objects
Type_92 - boxes
etc.

As much as I remember couple days ago someone asked for mapping Type to classnames and answer was - not possible because it is intentionally obfuscated because of cheaters.

[tribal pasture]

ClientCameraPositionObject
Do clients know about other clients camera positions? I know server does. I remember seeing a cheater video where they saw how admins were spectating them and stopped cheating to appear innocent, there were cameras floating around and I always wondered if it was global camera object or that engine camera position being available to clients.

[flat imp]

Just hearing of this effort, fantastic news. Wrt the queue, have you considered just using the moodycamel concurrent queue? It's basically industry standard at this point. Not as fun as writing your own I know but it's unfathomably fast and thoroughly tested. It's what Tracy uses internally

You'll want to check to see if its characteristics suit your workload since it has some caveats in its behavior with regard to the ordering of events, it's not a silver bullet

[spiral pond]

is there a (local) tool/software to load multiple (10s) / many (100-1000s) traces into it and flip through them quickly? or some filtering/aggregation?

was this in v6/v7 already? (client/server)

worth to review or already taken into account now?

[spiral pond]

https://github.com/jetelain/Arma3BasicScriptProfiler - was this ever shared? cant find here or on ACE discord.
as it says its quite basic and for runtime stats ASP should be better and way less overhead. still for CBA/ACE with macros in place already, one could generate some other stats with that

[whole cloud]

it has not been identified.
I know the reported steam lag (in main menu) with MANY subscribed missions/compositions is a thing. But we don't know why.
And I have no idea what the infinite download loop is

[whole cloud]

I only know the server does. Don't know if its synced to clients. Would probably be good to look it up

[whole cloud]

moodycamel is good when doing many concurrent inserts/extracts. But that is not the problem here.
We don't have many inserts and extracts, and especially not concurrent at all.

[whole cloud]

Tracy can combine two traces, and does aggregation/statistics over time and across two traces.

[whole cloud]

yea

[whole cloud]

Sound is now async, but not parallel to render.
Sound needs animation, and after sounds but before render, we are loading animations that will be needed in render step, which conflicts with sound using these animations

Funny that I talked about the same things 7 years ago 🤣 Amazing find

[spiral pond]

are frame and sframe MP only? cant get them to trigger in SP/have an effect

or something bugged with cheats in general? like it shown the confirmation message but nothing happens - even with endmission
cheatsenabled returns false, but not sure if thats relevant

["Arma 3","Arma3",218,152344,"Profile",true,"Windows","x64"]

entities worked tho; also fps

something is weird. neither the sqf work (properly)

diag_captureFrame 0 or 1 doesnt trigger at all

with >1 only empty

[dire wave]

Guys to test le last version i just need to opt in in perf branch on steam? And i get auto update? Or should i download updates manually somewhere?

[autumn timber]

The former

[light cargo]

you can do either. you would need to manually update if you want to run a dedicated server with creator dlcs branch for instance

[whole cloud]

No. I run them in SP yaab too and don't have an issue.

[night mulch]

this is the infinite download loop

https://gyazo.com/6432bcf3433d952d1e808e58a7c13380

the game stutters everytime it dos that (which is pretty much every 3 seconds)

this is a report describing the issue
https://www.reddit.com/r/arma/comments/qiog22/arma_constantly_trying_to_download_workshop/

[spiral pond]

ok found out what happened - launched by mistake with ASP (works with v7 now yay!)
seems like ASP disables the native profiling (UI) - launching without ASP, frame works again

[knotty wraith]

I confirm - I also suffer from this very much, I can only solve this problem by restarting Steam

the main thing is that you have everything downloaded and updated, then everything is ok
But as soon as something is in the download queue, Steam won’t let you play
I wanted to launch Arma from a laptop in my own Steam account, I have 200 gigs of mods signed up there and I realized that just downloading the ones needed for the server won’t work, Steam won’t let me play until it installs all 200 gigs

but this is not the thread to discuss this problem

[spiral pond]

do you have A3 launcher running while the game is active?

[whole cloud]

But I need a way to reproduce it

[dire wave]

Sorry to interrupt, newbie here, i was reading all this post, it seems that v7 is the one that provides best bechmark (for fps) client side? But it creates some problem still so you guys reverted the steam one to v6. Did i get all this right?

(How can i check the version that i have installed?)

From reactions should i understand that the best one is v5? Im a bit confused lol

[knotty wraith]

https://forums.bohemia.net/forums/topic/160288-arma-3-stable-server-218-profiling-performance-binary-feedback/?page=106&tab=comments#comment-3535767

[analog acorn]

The game shows a build number in one of the corners of the main menu, as well as in the Options menu in the Launcher. This number corresponds to one of the versions. You can see which versions have which build number by checking the pinned messages in this channel, or the changelog thread Dimon linked.

For example, prof v7 is 2.18.152344. Current stable is 2.18.152302.

If you want a specific version, you can download its .exe from the Google Drive link in the pinned messages. Doesn't matter which one you already have installed - only the .exe is different and you're about to overwrite it.
Verifying your game files through Steam will always revert you to the current main release for whichever beta (or not) you have selected in Steam.

[spiral pond]

@night mulch @knotty wraith did you check A3 launcher/steam logs ever for some potentially relevant info there?

[knotty wraith]

I think remove a few mods from steam, make sure they are removed, and then re-subscribe to them (preferably large mods so that they download for a long time) and launch the game. At such moments I can't even enter the server - the player freezes in front of the lobby at the mission loading stage.

manually removing the mod from the download does nothing - it will return to the download in a few seconds
But it seems to me that this is purely a problem of Steam and it doesn’t matter what game it is

[whole cloud]

steam is on v5. The one before the big performance things

[knotty wraith]

I didn't know that the launcher and steam have logs

Steam has a setting to disable background loading if the game is running, but for some reason it doesn't work
I remembered that I had the same problems in Dayz

[night mulch]

it makes no difference if launcher is on or off, the loop still happens, and dimon's issue seems to be different than the one im talking about

[dire wave]

Thank u so much for this you saved this thread a lot of question that i was going to ask. Thank you again ❤️❤️

[knotty wraith]

https://www.reddit.com/r/arma/comments/qiog22/arma_constantly_trying_to_download_workshop/ You gave this link - that's what I'm talking about

[dire wave]

Im gonna start testing now gonna report my findings here if anyone interested 💪💪

[night mulch]

yes, but the issue is not tied to having unfinished things in the download queue

it happens even when everything is verified and clear

basically all the time

so there seems to be some variations to the issue but its hard to tell since people who have this probably dont even bother to report it

but it can be a big deal to some people esp. modders

[knotty wraith]

the thing is that when steam tries to download something, you don't understand what it wants to download - it downloads and then breaks and it happens all the time, I can feel it right away in the game - the ping jumps.
If that's what you're talking about - that also happens

[night mulch]

yes, i even sent a profiling log to dedmen, but idk how far he got with it, the conversation kinda got forgotten

https://imgur.com/a/ttHwXm6

[spiral pond]

C:\Users\XXX\AppData\Local\Arma 3 Launcher\Logs
C:\Program Files (x86)\Steam\logs

[spiral pond]

i had a similar one - in the sense that if there were mod updates while the game was running, steam tried to update but couldnt as game was locked - but it tried every other moment. once i closed the game and steam realized it could update the mods, my issue was gone (till next time)

[night mulch]

hmm then your issue might be similar to dimon's but a different one from mine

[spiral pond]

if you can determine the scope label, you could share it here and ppl could run diag_captureSlowFrame on it

[night mulch]

the one reported in this case is most likely not tied to the mod update thing, as if it was then the guy wont really have this big of a problem with it since it happens only during mod updates which is not that often

[spiral pond]

this way potentially more logging data could shed some more light on it

[night mulch]

idk what tha means sadly 🥲

[night mulch]

i can send u the profiling logs i sent dedmen here

[analog acorn]

I've noticed Steam will usually have an A3 download in the completed section after I stop playing...with no download size given, and nothing having actually been updated

[spiral pond]

sure. that said unless its triggered from inside the game/some game-steam coms happens, nothing might turn up with ingame profiling

ie could be just steam/A3 launcher/the OS stalling the game

[heavy vortex]

@night mulch Can you still replicate without the A3 launcher?

[livid nymph]

For me, the problem was related to the number of Compositions I was subscribed. Once I unsubscribed from all of the Compositions, the issue went away.

[dire wave]

i get average 12 fps more in v5 compared to v6 (yaab standard setting, 5800x3d, 7900 GRE, 32 gb ram 3200, win 10)

[obsidian relic]

Same

[heavy vortex]

Sure it's the number rather than one bad/missing one?

[obsidian relic]

after removing around half, time of stuttering also lowered by ~50%

it looks like game is downloading and installing them with every game launch

[heavy vortex]

yeah, but the fact it's checking at all might be triggered by a single missing one.

And then the time is dependent on the quantity.

Not everyone who subscribes to compositions has this problem, right? Otherwise dedmen would have a replication case.

[obsidian relic]

Didn't had it in the beginning too, so you might be right

[slow dome]

Same here

[ashen helm]

now I think this is new as I have not seen it in previous builds, medium boulders are targettable for some reason

[woven loom]

KERRY! TARGET THAT ROCK, FRONT!

[cosmic solstice]

it's a sneak peek of the new minecraft gamemode.
You shot at the rocks to gather ores until you build a diamond armor and diamond gun (no swords because no mlee).
Then you can go to the end and slay the Miller Dragon.

[night mulch]

i can confirm that the stutters happen in-sync with the infinite download loop, and something from the game must have triggered it, it doesnt happen when you're still in the arma3 launcher before starting game

or it could be that steam requests something from the game but the game doesnt have a response to it, so the loop goes on and on

[night mulch]

you mean starting from the exe without the launcher? havent tried that, but the issue still happens even if you close the a3 launcher after game start

[night mulch]

I'm sure a lot of people actually have this problem but just thought of it as "arma being arma with the stutters" and didnt bother reporting it

[heavy vortex]

Arma actually has quite low frame variance until you spam unit spawns :P

Reports of stutter are relatively rare compared to low frame rates, but maybe a lot of those are the same cause.

[night mulch]

an important detail about this issue is that if you kill the internet connection, steam goes into "no connection" mode and the loop stops, and the game becomes smooth again, but as soon as you connect back to the internet, the loop starts again and so does the stuttering with it

[heavy vortex]

Is it possible for someone to make a modlist with all their subscribed compositions? I don't have any so I'm not sure how that works.

Wondering if it's possible to replicate with the same list.

[spiral pond]

@night mulch unless there is a scope currently not covered, it seems something external to halt the game. both times its between dlcSim and input - if that is consistent, it may give Dedmen some pointer where it could come frame

[night mulch]

Thankfully i was able to record a video to help everyone understand about this issue. it also includes hardware statistics like fps, cpu/gpu usage etc on the top right. Please check it out

https://imgur.com/a/OLYjtxF

each video has a description on the circumstances

if you play both videos at the same time you can see the stuttering is strikingly obvious

i also noticed that while in internet offline, steam non-loop state, the FPS is constantly at 60FPS.
on the other hand, in the infinite download loop state, the fps actually drops around +-10FPS + very obvious stuttering every second

[night mulch]

thats a great idea actually, i think i can do that

if others who have the same problem can also give their compositions list as a collection, it would be great so we can test it with multiple samples

[night mulch]

okay, is there something i can do to help

[spiral pond]

more captures to see if thats consistent - for you and others

with these two its now easier to mass catch or log (https://community.bistudio.com/wiki/diag_captureSlowFrame)

ie diag_captureSlowFrame ["total", 0, 0, true, 100];

other option should be via ASP and using tracy to get averages over a session

[night mulch]

do i have to run that from arma3profiling_x64 exe?

[spiral pond]

yep with v7

[night mulch]

ok will do when i get the chance

[night mulch]

sadly its not possible to automatically create a collection with all my subscribed compositions... they all have to be added manually :(

for reference i'm subscribed to 1755 compositions

but there have been reports of others having this issue with around 191 compositons
#dev_rc_branch message

if you want to help reproduce then you can try subscribing to a lot of random compositions, and try to include ones with many mods including ones you dont have installed (e.g HALO mod, WH40k mod, etc) and really old ones also

[knotty wraith]

By the way, I remembered one inconvenient thing: in the launcher, when searching for mods, tags are very necessary... having 200+ mods in the collection, when I start moving the campaign to a new island,
I have to manually go through all the mods and check them in the workshop in order to understand which mod is an island - in order to test only them to find a new territory

[spiral pond]

wasnt it possible to disable PiP before? now 200m seems the enforced minimum

[silk pewter]

You disable in in the Quality setting, not the slider

[quaint flame]

Can someone remind me what the correct way to add a % with format command?

Wasn' there a change recently where %% would result in % in formatted string?

[loud garden]

format ["Percentage sign: %1", "%"];

[quaint flame]

Yes, that's the old way

ctrlParentControlsGroup _ctrlCombo controlsGroupCtrl 106 ctrlSetText format ["%1%%", _colorRGB#2 * 100];
This works for me as well but I am not sure if it was intended

[heady plinth]

cant you use % or something? 😄

[quaint flame]

format ["%1%%", 100] // "100%"

Guess this is new and it's just hemtt and the biki that are not up-to-date

[gritty wasp]

https://feedback.bistudio.com/T81725

[quaint flame]

Cheers!

[feral harness]

could be @whole cloud ?

[quaint flame]

Yes, the ticket above has ded's comment

[gritty wasp]

So it is in 2.18 not in future 20 as you write on biki

[feral harness]

ah, yeah 👍

[quaint flame]

Yeah correct. I somehow thought we are on 2.20 😬

[gritty wasp]

also limit warning seems outdated

[fair swan]

Ive gotten out of memory crashes more than few times the last couple weeks. First time ive seen these since starting the server 1 year ago. Happens after a few hours and then the application terminates itself and doesnt let aynone join but the server is still on and writing RPT and responding to queries. I dont have any valuable technical info to provide along with this other than we use a lot of mods and a very script intensive mission. Nothing in RPT indicates any particular reason why the memory is loaded but the issue has become consistent and predictable.

[spiral pond]

send all mdmps as zip to Dedmen @fair swan

[fair swan]

Its a retail server if that makes any difference, I dont have access to local app data folder for example.

I cant locate those files on the server, I can only find BIDMP, RPT and Console logs

https://i.imgur.com/zZVQpKb.png

[spiral pond]

possibly the server provider deletes those automatically in some fashion as they get "fairly" big - if not, its not crash or the server process gets killed before it can store the crash info. potentially some clues are to be found at the end of the rpt of such session

[drifting swan]

is this still relevant? players on my server are losing half empty magazines when reloading even after hotfix in stable branch

[whole cloud]

A single player running a pre-hotfix version, is enough to mess it up for everyone.
The serverside check was removed.

[drifting swan]

requiredBuild = 152302; will it be enough to fix this? servers works in profiling branch

[whole cloud]

yes

[eternal kraken]

arma 3 profiling steam update downloaded and tried my CPU Test mission to check min FPS, before update it was at 23 min FPS now its at 8 FPS 😱

[whole cloud]

When?

[eternal kraken]

now

[whole cloud]

So this:

2.18.152368 new PROFILING branch with PERFORMANCE binaries, v8, server and client, windows 64-bit, linux server 64-bit
- Added: exportJIPQueue now includes the number of messages per object (how many would be removed if object were deleted)
- Added: Multithreaded AI visibility calculations
- Added: Some multithreading optimizations that were accidentally disabled (Async Sound processing, Render Preparation)
- Tweaked: Improved efficiency of inAreaArray
- Fixed: Server crashes related to JIP Queue
- Fixed: Inconsistency in height between InAreaArray and inAreaArrayIndexes when matching in 3D
- Fixed: enableReload would not work with static weapon
- Fixed: Removing magazine from static weapon would trigger reload even when enabledReload was false
- Fixed: removeMagazine was removing current magazine from static weapon first

Note: while these changes have been tested internally, there are still many possible scenarios where things could break. They should be considered experimental. Please report all game crashes, new graphics glitches, and new cases of AI misbehavior.

If you don't want to use the Steam branch, the files are also available for alternative download here:
https://drive.google.com/drive/folders/15p9j7C2nHUt6NoVfChX4YFuqzFXzblJh

KNOWN ISSUE: If your game is crashing at start or mission load, set -cpuCount to a value below 12, and turn off hyperthreading/-enableHT.

[whole cloud]

Send me your CPU test mission I guess?

Or profile it yourself

[eternal kraken]

i will send you, its on altis saltlake all vanilla

https://steamuserimages-a.akamaihd.net/ugc/2450614265062300594/3C094CE2C7B642304E4CE4B54AEFBFEB9D0F5988/

i make a 7zip of the mission

http://moerderhoschi.bplaced.net/public/tools/arma3/CPU_TEST_MISSION/

http://moerderhoschi.bplaced.net/public/tools/arma3/CPU_TEST_MISSION/2023-05-24_CPU_TEST.Altis.7z

before the min FPS was at 22 or 23

[ashen helm]

fix this too pls thx

[obsidian relic]

R5 3600 (4.0GHz, lost sillicon lottery), 5700XT, 32GB 3000MHz RAM.
I did fresh windows reinstall yesterday so arma is fresh too, no mods, YAAB was done today with my custom settings (high/ultra, 3000m VD).
V1 - first run, V2 - second run,
mem - mimmalloc 214
perf - one that was earlier today (7)
https://i.imgur.com/zU7qIGG.jpeg

[eternal kraken]

now i have a weird bug with the profiling build

https://www.youtube.com/watch?v=4EF_tCysjc4

[obsidian relic]

i am trying to replicate, it blinks for me but not as often

[eternal kraken]

i updated my CPU test mission now showing the version number in the hintsilent message

you can already see in the YT video

[whole cloud]

Once you know that its new and not just think it, I might take a look

[whole cloud]

The flicker bug, yeah I've heard about it

For now a minor bug 😄 Atleast no crash

[dire wave]

my game crashes when loadin after the last update

every time. even without mod

v8

5800x3d

[gusty viper]

Same here, no mods, tried without mimalloc, etc. 10700K, 32 GB

[whole cloud]

Send me the crash report please

[turbid mural]

Me too ,mine is server on i9-9900k 32GB Windows VM (have MDMP)

[dire wave]

sure thing! (where can i find it?)

[turbid mural]

DM?

[whole cloud]

When exactly, during start? when during start?

[whole cloud]

!rpt (The mdmp file there)

[frozen tundra]

Arma RPT

Arma generates a .rpt log file each time it's run, which contains a lot of information like the loaded mods, or any errors that appear, this log file can be very useful for troubleshooting problems.

To get to your RPT files press Windows+R and enter %localappdata%/Arma 3

Additionally see the wiki page for more info: https://community.bistudio.com/wiki/Crash_Files

To share an rpt log here, please use a website like https://pastebin.com/ (Set expiry time to 1 week or less) to upload the full log, that way the people helping you can take a look at it and try to figure out the problem you're having together with you.
Note: RPT logs can hold personal information relevant to your system, the game or others.

[whole cloud]

yes

[gusty viper]

Crash during loading 3D world

[knotty wraith]

I have seen flickering many times - this is some kind of twitching of an element of the unit's equipment - especially if you move towards the unit, I can take a video

[whole cloud]

I don't need more videos of the same thing that I've already seen

[whole cloud]

Can you send me your crash report?

[spiral pond]

SPE WL client + DS stable so far

just AI advancing not to next sector - could be just a WL problem tho

they did now (one group only tho)

[whole cloud]

I don't know what's going on.
I just launched server on latest version, all fine.
Run client, all fine.
Can play showcase missions, or official zeus server no problem.

The client crashes all seem to not generate mdmp's

[fast hornet]

hm yeah.. client crashes while loading the mission or even earlier it seems. No memdump. Nothing really interesting in the rpt as well imho.. you want it anyway?
Will test server in a few minutes

[whole cloud]

Does it only happen with mods?

[gusty viper]

Crash happens also without mods or dlcs

Cannot find mem dump

[fast hornet]

server crashes as well, also no memdump

huh, second time the server seems to run.. loaded mission and was online for a few seconds. then crashed again

time to revert i guess

[spiral pond]

["Arma 3","Arma3",218,152368,"Profile",true,"Windows","x64"] - no problems here so far. checking other missions now

PZKW with DS also no prob here

only strange thing was layer with simulation disabled didnt work - never had this before

[eternal kraken]

is there any documentation that i can understand what i see when using diag_captureFrame ?

[whole cloud]

no not really

Besides the basics rows are threads. And in there are segments of "tasks" that are done.

[quaint flame]

We are working on that.

[spiral pond]

do you mean mainly the labels (scopes) or "everything" about it?

[eternal kraken]

a shame i cant post pictures here. what is: aiMUp or cLVsS or visUU?

[spiral pond]

some of the scopes you can learn about by searching in this channel. Dedmen explained several now

[eternal kraken]

or pthAS or cLSAWNoW .... im lost

the Jobs/processes with the biggest single ms (cosuming the most time) are the showstoppers (making the game slow, killing fps) is this right?

[spiral pond]

for some you can get a general idea - ie vis = visibility check of some sort probably

[eternal kraken]

maybe my question was not that smart, there are so many processes in one frame thats really overwhelming. i think i need to test, see read and learn

[spiral pond]

if you just capture one frame, its just a single snapshot.
if you do via a slow frame capture, its says at least either overall low fps or the given scope is long enough.

from another angle:

• there are minifreezes - very long frames
• there is general low(er) fps phases - you want to know why. ie AI calcs take too long/too many, or PX for vehicles
• to get overall higher fps you need to reduce the frame lengths in general - this is what Dedmen mainly focused on with the MT. cutting all major parts into subparts (jobs) where possible, so as many main parts are shorter => higher fps

one sample

tank mission from R3vo - server side capture

most is PX3. server fps is 10-20 fps

so if its possible to either put each PX3 into its own thread, it would improve fps (reduce frame length), or other angle (harder) would be PX3 calc itself to be optimized

another sample

line 2-4 are the new worker threads Dedmen introduced with enfusion MT handler

left half is mainly general simulation (infantry, vehicles, etc) - these cant be MT either one may influence the next. so you cant run them in parallel

on the flipside right half is AI, sound, render, other calcs. most have white boxes (jobs) in line 2-4

aka main thread (first line here) is now shorter as these parts are no longer linear but ~1/3

[whole cloud]

@fast hornet Are you running server with -enableHT parameter?

[spiral pond]

going forward depends mainly what Dedmen still can do (technically and time investment), plus what situations can still be identified as problematic (be it minifreezes, or special situations like JIP, higher player/AI count, long mission duration may still have specific problems - some may or may not still be possible to improve. ie the long JIP queue)

[fast hornet]

yes.. want me to try without?

[whole cloud]

Yes. I'm quite sure thats it

[knotty wraith]

the flickering hasn't gone away

[whole cloud]

I know, I don't expect it to. It will be listed in changelog when we get to it

[eternal kraken]

thank you for your explanation kju

[whole cloud]

Seems cpu count higher than 12 is crashing.
-enableHT basically doubles the number.

[spiral pond]

is the number of worker threads dynamic based on cpu count available?

[fast hornet]

it just crashed without -enableHT, but i also have -cpuCount=16 set .. i'll try it without that as well

[whole cloud]

Yeah.
Its like cpuCount-1
with a max limit of 16

Yes.
cpuCount above 12 is crashing. That's it

Of course my PC has exactly 12

[spiral pond]

one more angle would be statistical analysis. meaning you identify from many many frames (and various missions and users) what are a) fps spikes (low) and b) what scopes in general are slow (ie >3 ms) - to learn what parts may be worth to optimize (further). yet that would be more complex optimization (so far Dedmen basically improved the "engine design/architecture" here with the MT work, while other problems are due to "bad coding")

[fast hornet]

hm yeah.. without both -enableHT and -cpuCount=16 the server seems to not crash.
But our test VM only has 4 cores and 8 threads, so the autodetect will never be above 12.

The Client still crashes, my desktop has 16 cores/32 threads. So the autodetect will be above 12 as well.
And the live server has 8 perf cores (16 threads) and 16 e-cores.. so also 32. Can't test it on that one but i'm pretty sure it would crash as well in this case.

[whole cloud]

We should get rid of -enableHT.
All it does is 2x the cpuCount

[fast hornet]

just launched the client without -enableHT and with -cpuCount=12 and it's running. So yeah, that's it.

[eternal kraken]

and -exThreads?

[whole cloud]

thats a different mechanism

exThreads any other than 7, only makes sense if you have like single or dual core cpu's. No-one does that anymore

And I found why the crash happens.
It ends up adding +4 to a thing.
But the hardcoded limit is 16. So it ends up doing 16+4 and blerps over

[fast hornet]

getting rid of exthreads and only having the 7 setting might be a good idea as well

a question apart from the bug.
When i set both enableHT and cpuCount, which one does the server use? Or does it do cpuCount * 2 in that case?
Or from another angle, when i have 16 threads and set cpuCount=16 i can just remove enableHT without any impact at all?

[whole cloud]

-enableHT is cpuCount*2

[fast hornet]

so for 8 cores / 16 threads the correct usage would be either -cpuCount=16 or -cpuCount=8 -enableHT with the same result of 16 threads?

[whole cloud]

yes

But i don't know if using threads, actually gives a performance boost or whether its bad for performance.
If you don't have 8 real cores, then throwing 16 threads at them might be worse, than just using the 8 cores fully.

[fast hornet]

'should' be marginally faster.. dunno. At least the server was able to 100% utilize all 16 threads when we had high load

(But that doesn't happen anymore since the scripting performance improvements, those helped a ton.)

[whole cloud]

Actually sorry, mixup

if you set -cpuCount=, then -enableHT is ignored.

If you do not set -cpuCount, then -enableHT would double it

[fast hornet]

ah ok.. then i can remove enableHT anyway. Thanks 🙂

[whole cloud]

What CPU Count the game, at the end, actually uses. Is listed in RPT at the top

PhysMem: 32 GiB, VirtMem : 131072 GiB, AvailPhys : 7.7 GiB, AvailVirt : 131068 GiB, AvailPage : 21 GiB, PageSize : 4.0 KiB/2.0 MiB/HasLockMemory, CPUCount : 16

[fast hornet]

i'll keep our dev server on v8 with cpuCount=12 for now.. i'll message you if it crashes. Not a perfect test with zero clients.. but maybe it helps.

[indigo anvil]

Hey! I've got an error on current perf/prof (2.18.152368):
16:52:31 Error position: <inAreaArray [player, 100, 100]) select {>
16:52:31 Error select: 3 elements provided, 7 expected
"inAreaArray [player, 100, 100])"
According to https://community.bistudio.com/wiki/inAreaArray the rest of the params are supposed to be optional.

[whole cloud]

@feral harness 😢

[whole cloud]

The

[center, a, b, angle, rect, (height), (isPosWorld)]
syntax right? in code it says the first 5 are required.
Or atleast, it says that now.

[indigo anvil]

Ah ok thx

[feral harness]

was it optional before, inArea has correct optional params

[fickle geyser]

In area array in current release works with 3 params

center, a, b

[feral harness]

damn

[indigo anvil]

yep

[feral harness]

ok will make inArea same

[whole cloud]

5 days ago (before the changes) it checked for minimum 5 arguments and threw error if less

[feral harness]

it said the same minus posisworld

[whole cloud]

As far as I can see, before the first change from 31st october.
it was 5 minimum parameters require.
[center, a, b, angle, rect, (height)]

And today, it still is 5

[feral harness]

was not intentional but now i guess will have to be

no problem defaulting a few more

[whole cloud]

Ah I found the 3.
Its good that the code was cleaned up now, the old one was harder to find stuff

[vivid rune]

70fps in YAAB. +26% against v5

[whole cloud]

I wonder what its gonna be against v7

Because now my "big" rendering thing is also in there

[vivid rune]

+12%

[vivid reef]

@whole cloud new performance is crashing but i dont get a dump

[whole cloud]

limit cpuCount to 12.
Known, fix incoming

[vivid reef]

and apparently we have another issue with something that did break in koth

[whole cloud]

and what is that thing?

[vivid reef]

im testing rn if the users were right

you know these team markers above players heads?

with profiling apparently they dont show, but stable they do

[whole cloud]

check rpt for errors?

[vivid reef]

oh yeah

tons of errors

[feral harness]

script?

[vivid reef]

17:46:41 File mpmissions__CUR_MP.Altis\client*c_fl_playerIcons.sqf..., line 4
17:46:41 Error in expression < _0bke;if(_0bkf)then{_06q deleteAt(_06q find focusOn);};private _0bkg=[];private>
17:46:41 Error position: <find focusOn);};private _0bkg=[];private>
17:46:41 Error find: Type Bool, expected Array,String
17:46:41 File mpmissions__CUR_MP.Altis\client*c_fl_playerIcons.sqf..., line 4
17:46:41 Error in expression <gyreh select{alive _x})}else{units _0xq}inAreaArray _0bke;if(_0bkf)then{_06q del>
17:46:41 Error position: <inAreaArray _0bke;if(_0bkf)then{_06q del>

reverting back to stable and cross check, but since we didnt update mission, it should work there

[whole cloud]

probably related to inAreaArray being broken. which means that is already found

[feral harness]

second yeah but first?

[vivid reef]

yep, stable it works and no error related to this in rpt

you want the full rpt?

[feral harness]

yes

[vivid reef]

you or dedmen?

[feral harness]

me

[vivid reef]

cant dm you

[feral harness]

those errors related so new prof should fix that

[vivid reef]

eta tomorow?

[whole cloud]

an hour hopefully, if nothing more goes wrong

[vivid reef]

[empty goblet]

erm, it was supposed to also affect microjobs and was specifically aimed on Intel HT and AMD SMT

[whole cloud]

That's what it does

That's also what cpuCount does

[empty goblet]

ye but it ain't just doubling the cpucount, i specifically inquired about that

[whole cloud]

Well on most CPU's it does, because threads is usually 2x cores

[empty goblet]

ye but what i mean it didn't act like -enableHT at 8C+8HT equals -cpuCount=16

[cloud sky]

Aren't the threads abstracted away at operating system level? Like that all "cores" are equal for the user

[empty goblet]

no, this was our decision how handle problems with non physical cores

and if you think you can trust windows kernel to do any reasonable scheduling and spreading of load or using correctly the right cores ... then you will be still waiting

that exists only in Linux

[empty goblet]

plus not all cores are equal and even then Windows messed it up, even if they got told which are more powerful, which are in what chiplet (like NUMA but not numa) and what quality or cache or latencies

[cloud sky]

But is it even possible? Like does Windows even differentiate between physical and "virtual" logical cores? I've always thought that the HT/SMT is indeed abstracted away to "just" 2 x cores in OS, but because OS is aware of the underlying architecture, it can utilize it efficiently (or that it happens automatically in any case)

[whole cloud]

But that's what it does.
It only changes the number returned by the "count how many cpu cores I have" function. It only affects a single number.
Which is then used to launch number of job threads

[empty goblet]

that's not what i mean, the old method just used cpuCount and multiplied by amount of cores

the newer method fixed core detection but if HT/SMT (logical) cores were detected then it was supposed to not use fully fledged threads on those but microjobs (lesser load)

[chilly geyser]

What happens in v7? Frametime in a game is insane. Its sooo smooth like never

[whole cloud]

microjobs are fully fledged threads.
The worker threads we call microjob threads

[whole cloud]

Rendering performance improvement (the first one I've done) was disabled until v8. v7 didn't have it, v8 has

[empty goblet]

well then something is wrong there because it was supposed to not use the same load threads on logical core as at on physical

because logical core under serious load slow downs the physical core

[heavy vortex]

You don't really get the choice on Windows.

[fickle geyser]

Is it even relevant now with enfusion workers?

[chilly geyser]

Actual version in steam is v7 or v8?

[heavy vortex]

You make four threads, Windows puts them where it likes.

Usually wrongly.

[whole cloud]

yeah that's what I expect too, which is why I think enableHT is useless/bad

[whole cloud]

it is.
But we don't differentiate between a "hyper" and a "real" core. We don't pin threads to cores, and never did previously.
So its as relevant as it was before.

enableHT just causes more threads to spawn

[whole cloud]

You can pin them to cores, if you wanted to.

[heavy vortex]

Normally for games the issue with excessive threading isn't execution resources but cache & memory thrashing.

[chilly geyser]

Found it. nvm

[whole cloud]

Steam isn ow v8

[heavy vortex]

So even using all the physical cores isn't necessarily beneficial.

[opal hound]

Is this still up to date with later windows versions & CPUs with P/E cores?

[chilly geyser]

@whole cloudfrom 2012 to this moment iv'e never seen better fps in arma

[whole cloud]

:wobcat:

[empty goblet]

core types aren't something we can deal with, that's job for OS kernel scheduler

[fast hornet]

it might be a bit different on Win11 since that has a newer scheduler.. but at least Windows Server 2022, which is based on Win10, has no clue. We set the CPU affinity to only p-cores manually. When you let windows decide it constantly shifts the threads between all cores and the performance is garbage

[chilly geyser]

btw, good question about physical cores. If i have 16 physical cores i need use only them for arma?

[whole cloud]

I don't know what you mean

[dire wave]

which cpu do you have?

[whole cloud]

we don't pin threads, so its up to the operating system to decide which cores it ends up using