(IN-37272) (IN-35144) (IN-33187) (IN-33155) (IN-35920) [1.0.0-pre.65] Performance issue case list

[livid cedar]

IN-35144, IN-33187, IN-33155, IN-35920 I submitted 4 cases for Entities Graphics performance issues for both CPU and GPU usage and also URP performance issue based on Entities Graphics

(IN-35144) (IN-33187) (IN-33155) [1.0.0-pre.44] Performance Issue Case List

(IN-35144) (IN-33187) (IN-33155) [1.0.0-pre.44] Performance issue case list

[livid cedar]

(IN-35144) (IN-33187) (IN-33155) (IN-35920) [1.0.0-pre.44] Performance issue case list

@violet epoch Will the performance improvement and gc spike fix be released at next Entities Graphics release? I see some of issues ady have issue tracker though.

[violet epoch]

I’ll ask again; they seemed like they were going to deal with the gc at least

[violet epoch]

it's not looking great, since it's apparently due to some engine limitations that will be hard to fix for 22.2. that said, with incremental gc on, i would be kind of surprised if you had big spikes due to that small amount of allocations. do you have incremental gc on?

[livid cedar]

Incremental GC is off

[violet epoch]

why is it off

[livid cedar]

Not a the fan of incremental GC and actually official also meantions that it's not performant for all the cases.

[violet epoch]

???

who said that?

afaik incremental gc is great

[livid cedar]

I guess is this thread https://forum.unity.com/threads/incremental-gc-feedback-thread.588664/

[violet epoch]

skimming that thread i don't see anything that says incremental gc sucks

[livid cedar]

🤔 Btw I guess I should move to untiy 2023.2 soon?

[violet epoch]

no, because the renderer likely won't work with it, and it's not tested at all

and anyway the gc graphics stuff hasn't been fixed in trunk even

[livid cedar]

Ya. I guess at least need to wait until 2022.3 lts and 1.0 out pre-release

[violet epoch]

at least. now where in that thread does it say incremental gc is bad

[livid cedar]

No but I see some users say not really working nicely as expected

[violet epoch]

they're confused. it's great.

it just smooths out spikes when the amount of garbage generated is small, which it is in your case

[livid cedar]

Ok. Just want to know most likely won't backport to 2022.2 cycle after it's fixed?

[violet epoch]

yeah i doubt it. graphics is pretty hair on fire with other stuff at the moment

[livid cedar]

Ya. It's just mitigate GC spike but it's still has GC lol

[violet epoch]

yes, but it's not enough to matter, i'm willing to bet

before incremental gc, gc was mainly bad because it caused unpredictable spikes. with incremental gc, gc is only bad once it becomes really a whole bunch, so that it's visible every frame and/or triggers a non-incremental collection

[livid cedar]

Currently GC spike is quite a lot I think. One is at deformation system and another I forget is which system

[violet epoch]

it's a lot because you don't have incremental on

so just try it please

[livid cedar]

Will let u know after I try it

🤔 Btw actually if keep having GC spike, will it keep increasing memory?

[violet epoch]

no, unless you don't let go of a reference to the objects you've created

[livid cedar]

I see. Just set the reference to null should be ok?

[violet epoch]

yep, assuming it's the only reference

but also, if you saw gc spikes before, that meant the gc actually found stuff to collect, usually

[violet epoch]

how'd it go?

[livid cedar]

Ticked use incremental gc but still emit gc spike every frame just exactly like before

[violet epoch]

It’s not a spike if it’s every frame 🙂 and do you mean there’s a collection every frame, or just some allocations? What is your overall frame time and what % is gc?

[livid cedar]

Profiler showing 0.6kb GC Alloc every frame

[violet epoch]

Ok but no actual time is spent collecting, it looks like

[livid cedar]

@stark ocean Will the performance be improved for GPU and CPU usage? And also is that gc issue won't be backported to unity 2022.2 cycle?

[stark ocean]

We are definitely trying to improve performance where we can, but we have to balance that with fixing bugs.

If you have GC allocs coming from core Entities Graphics code (i.e. not Deformations, which I think is still experimental) every frame even when leak tracking is disabled, then that could be a bug.

There are some expected GC allocs during "warmup" when certain managed collections grow to required capacity, but in steady state execution it should not GC alloc every frame.

[livid cedar]

Actually there's another system other than deformation also has gc spike every frame. I build development build to test so leak detection should be disabled.

[stark ocean]

Do you have a callstack for this GC alloc?

Or a profiler timeline screenshot to show roughly when it happens?

[livid cedar]

I think at least improve GPU usage performance. Currently the GPU usage never reduce even camera is not looking at 3D models

[stark ocean]

There should be some improvements to culling coming. Also, Entities Graphics should definitely frustum cull entities that are outside of the viewing area, unless those entities are somehow improperly configured (e.g. too large render bounds).

[livid cedar]

Let me check. Last time I see it's at EntitiesGraphicsSystem EndUpdate with 64 bytes gc spike every frame

[stark ocean]

Ahh, in that case I might know what it is actually.

[livid cedar]

🤔 Or actually 3d models with deformation can't support this frustum culling?

[stark ocean]

If you are seeing GC allocs from EndUpdate, I think there's a high chance it could be related to this leak bug: https://forum.unity.com/threads/reproducible-memory-leak-in-graphics-and-graphic-device-pool-when-instantiating-an-entity.1414770/

If you want, you could try the workarounds provided in this thread to see whether that also removes the GC alloc (which I think might be caused by a certain internal data structure growing unintentionally, which is also what causes the leak mentioned in the thread).

In general deformed models are more difficult to cull, because their bounds are more difficult to determine. I'm not sure how this is handled currently, but I suspect it might be possible for you to define custom render bounds for deformed models.

[livid cedar]

🥲 I will wait for official solution since I have no idea how to implement custom render bounds for deformed models. Hopefully can get the solution soon.

[stark ocean]

It shouldn't be difficult, it just means assigning a custom value to the RenderBounds component to replace the value placed in there by the default baking.

I don't know how it's set for deformed models right now, but I would guess the default is very conservative because in general it's difficult to estimate bounds for deformed models.

But you should be able to use a smaller bound if you want, for example by multiplying the Extents value inside the RenderBounds by some number that is smaller than 1.

The smaller the extents, the easier it is for Entities Graphics to cull the object.

But if the value is too small, then you might get rendering artifacts where the model is culled even though it should be visible.

I.e. if the bounding box in RenderBounds is too small to actually contain the model after it has been deformed.

[livid cedar]

🤔 It's default baked render bound value. I guess should be ok but not working?

[livid cedar]

\

[stark ocean]

I think it's very likely related to the leak I linked above.

And both the actual fix for the leak and the workaround for the leak should hopefully eliminate this GC alloc as well.

[noble herald]

I have the same problem with GC allocations

And I have Incremental GC enabled

this is my thread about the memory allocation: https://discord.com/channels/489222168727519232/1088473854562803742

oh @livid cedar, still have issues about this 😩

[stark ocean]

@noble herald Please check out the workaround in the thread I linked above in this discussion, I think it's likely to also fix the GC alloc.

[noble herald]

I was looking right now at this

theres some ETA for new release?

[stark ocean]

I think there will be an official announcement when there is an ETA.

[noble herald]

Thanks @stark ocean

[noble herald]

@stark ocean, I tried to apply the workaround, and kind works, but ASAP, occurs a crash for me.
Testing in Android.
Fatal error log in below.

[noble herald]

now its seems like other system has the problem

[violet epoch]

i have no idea if this applies to this case, but sometimes if it's impossible to reproduce an error in the editor, it can help to build a player with UNITY_DOTS_DEBUG in the scripting defines, which can enable some helpful checks

[noble herald]

this is a build error, on Android

I put UNITY_DOTS_DEBUG

but seems like the same in Android Logcat

theres another way to see these debug?

Result with UNITY_DOTS_DEBUG in scripting defines

[livid cedar]

@stark ocean Btw any plan to fix stalling and upgrade some of the systems to ISystem for Entities Graphics? I guess currently not all can upgrade to ISystem because of some BRG API can't be burst compile?
And also I see another bottleneck is at SRP side that RenderPipelineManager.DoRenderLopp_Internal() takes really long time to run. It seems like SRP core packages need to adopt Entities and rewrite it to give huge performance to really solve SRP performance issue.

[stark ocean]

What kind of stalling are you seeing?

[stark ocean]

This seems to be when running with a very light load. Do you have an issue when running a scene with a heavier workload?

[stark ocean]

In this latter screenshot there isn't that much time wasted due to the main thread, although there are certainly some small gaps in there.

But most of the time the worker threads are working properly.

The heavier the workload, the smaller the gaps should be in proportion to the overall time taken.

[livid cedar]

🤔 Ya I know but why Entities Graphics design it to be like that has multiple WaitForJobGroupID() spend so much time? From I what understand, it's caused by at somewhere calling complete job to force the job complete immediately

[stark ocean]

Entities Graphics starts a number of Burst jobs (the green bars in the profiler capture) to do the heavy lifting to update the changed entities to the GPU. The final part of that process is done with a compute shader, but the system needs to wait until the Burst jobs are done before it can start the compute shader. However, during that waiting time, the main thread will also participate in doing the work (you can see the green bar beneath the JobHandle.Complete), so the CPU time is not being wasted.

[livid cedar]

🥲 I see. So meaning until compute shader able to support job and burst, can't really completely solve the stalling issue

[stark ocean]

We do have plans to improve the system update time, and it should be possible to remove some of the overhead that is currently there.

[livid cedar]

I guess also can upgrade most of the systems to ISystem. Currently all are SystemBase

[stark ocean]

Currently Entities Graphics needs to interact with many APIs that are not Burst compatible, such as BatchRendererGroup, Mesh, and Material, so switching to ISystem is not a simple task.

[livid cedar]

@stark ocean I spotted at DeformationPresentation there are multiple jobs not burst compile yet and also there's something keep logging and emit gc every frame.

[stark ocean]

This seems like a bug.

[livid cedar]

Alright. Case IN-37272

(IN-37272) (IN-35144) (IN-33187) (IN-33155) (IN-35920) [1.0.0-pre.44] Performance issue case list

[livid cedar]

Alright. I get the slowest moment

[violet epoch]

i don't think i see any gc there?

[livid cedar]

Ya but it's still quite slow that there's UnsafeUtility.Free red bar takes 0.98ms. Currently when use skinned mesh u definitely will get gc. Another is using regular mesh at specific scenario only will get gc which I guess is the bug that mentioned by @stark ocean will fix at next release