languid saffron
See the screenshot. I'm not sure why this happens. I'm pretty sure this wasn't the case previously. I do a few schedule calls that in the past led to low schedule overhead and having the worker threads going to sleep only after doing a lot of jobs at once. Now Schedule overhead is consumes 25% of main thread and immediately triggering the worker threads.
Does anyone have any information about this. Might be related to a Unity update in the past or some theshold or buffer getting full?
I guess it's
Kernel: Re-enabled job batching to fix job system performance regression. (DOTS-9474)
from 2022.3.21f1 ...
deep cloud
This changed with the 2022 job rewrite, let me see if I can find a quote about it
languid saffron
Thanks tertle. I think the 2022 update allowed the main thread to fill local queries of running workers directly or something in that direction but it was still postponed until the ScheduleBatched call came. I'm currently checking the the Unity update above.
deep cloud
I was told ~9 months ago something like ScheduleBatchedJobs doesn't really do anything anymore
Just trying to find the source before I spread false info, though I never did test it myself
by default (unless you specify an arcane boot config option to re-enable batching) in 22 this does not do anything anymore post-jobsystem rewrite; iirc kevin found that after the rewrite batching didn't help on most workloads
languid saffron
Okay... not sure if that makes sense and all official information I found still talks about batching. Waking up threads is slow and Unity can't change that.
I still have hopes for the 2022.3.21f1 update. I will report.
rotund wadi
I'm also seeing some weird behavior related to this in 2022.3.21f1, seems like when I pass a job dependency it takes a much slower path where it constantly wakes up other threads which causes the main thread to slow down signficantly. @thorn vessel Can you tell what has changed in 2022.3.21?
thorn vessel
I think there was indeed something that dale landed pretty recently (which is where that release note came from); I'll ping him now (but he may or may not see it before tomorrow)
languid saffron
First Check: Same section after updating to 24f1 (from 16f1); schedule down from 700 to 100μs. Second image is Complete/ScheduleBatchedJobs call. So it looks okay again. Even with immediate complete this behaviour is around 3x faster in this case.
pale sonnet
First Check: Same section after updating to 24f1 (from 16f1); schedule down from...
unrelated but may I ask what you are profiling with? I don't recognize the UI
deep cloud
it's superluminal
languid saffron
https://superluminal.eu - game changer: sampling profiler, resolves unity's mixed callstacks, works great in mono, il2cpp, editor, standalone and is just super hassle free and fast
pale sonnet
nice, thanks! I didn't manage to get PIX working on PC so I was a little sad, this makes my day
languid saffron
@thorn vessel could you tell us the (unless you specify an arcane boot config option to re-enable batching) boot.config option for testing?
rotund wadi
First Check: Same section after updating to 24f1 (from 16f1); schedule down from...
So it's fixed again in .24?
lucid lynx
@languid saffron Hi Julian, I re-implemented the job batching support from the patch notes you mentioned. I'm having trouble understanding if you still have an issue with job batching now, but if you do you can enable/disable job batching with a command line argument -job-batching true or -job-batching false. With 2022.3.21f1 and later the default value of this setting is true which means job batching is enabled by default. For most people, I would expect that no one would have to change this setting since it yields higher performance but in the case that the new job batching implementation has game breaking bugs, I left the option in to disable it.
languid saffron
Thanks! I have a unrelated crash after updating when prewarming shaders that blocked me from testing in production but it looked good in my first tries.
@rotund wadi I guess as your issue started with 21, deactivating batching might be a workaround for you?
rotund wadi
@lucid lynx I'm running this test and it takes around 1.2ms of main thread time to schedule the jobs on a release development build on my 7950x, I tried toggling job batching but it didn't seem to do much, is this expected for ~1000 jobs? Do you have some recommendations to speed this up further? This is on Unity 2022.3.25f1 ```using Unity.Collections;
using Unity.Jobs;
using UnityEngine;
using Unity.Burst;
[BurstCompile]
public struct JobA : IJob
{
public void Execute()
{
}
}
[BurstCompile]
public struct JobB : IJob
{
public void Execute()
{
}
}
public class UpdateSystem : MonoBehaviour
{
private void Update()
{
NativeArray<JobHandle> dependencies = new(500, Allocator.Temp);
for (int i = 0; i < 500; i++)
{
JobHandle inputDep = new JobA().Schedule();
dependencies[i] = new JobB().Schedule(inputDep);
}
JobHandle.ScheduleBatchedJobs();
JobHandle.CompleteAll(dependencies);
}
}```
lucid lynx
<@1222959834375131251> I'm running this test and it takes around 1.2ms of main t...
It's not expected to be this bad, I'm digging in with a profiler now and at first glance it's a bit surprising to me how expensive some things are with the scheduling code path in this example. Basically what's going on is that the job system needs to do some work to manage the dependency JobB has on JobA, and most of the time the actual management of those dependencies are pretty fast. What's slow is after the management of dependencies, we might discover we need to wake threads and that waking threads part is expensive.
The part that's puzzling to me is that with batching, I expect almost no time to be spent in waking until after the batch is created and submitted for workers to start executing. Right now, I see 99% of time being spent in waking threads while creating the batch. I'll keep looking at this, it's looking like it might be a performance bug in the job system.
rotund wadi
It's not expected to be this bad, I'm digging in with a profiler now and at firs...
Thanks Dale! I'm being hit by this issue quite significantly, is there a place I can track this issue?
lucid lynx
Currently, no, we would have to create a new bug for this issue. I'm not sure what the correct process here is for creating a bug you can track, but I've already made some progress in making your example code faster. I'm pretty confident we can get a fix out for at least what I've already done since it is legitimately a performance bug that went out with the re-enabled job batching implementation.
rotund wadi
Currently, no, we would have to create a new bug for this issue. I'm not sure wh...
That's great news, will the fix be backported to the 2022 LTS?
lucid lynx
I can't promise that right now but my expectation is that it would because this is a bug in the batching code that I put into the LTS, and you're suffering from the bug. We tend to prioritize these as valid fixes and the code change is actually pretty small on my end so far to go from 1 ms on main thread to about 0.6 ms main thread. I think there may be more to gain, so we'll see where we ultimately end up.
rotund wadi
I can't promise that right now but my expectation is that it would because this ...
Ok, please let me know when the fix gets pushed into the 2022 LTS 👀
lucid lynx
I managed to create the bug and should be visible to you ☝️
I have a fix which is in the process of landing to various editor branches that provides about a 1.72x speedup in your example code. My test machine has an Intel Core i9-12900KF which has 16 cores/24 threads.
Hopefully you'll be able to get the fix in a future 2022.3 soon...