Crashes in editor with job system?

[earnest niche]

I am getting crashes that happen after getting some log spam of:

Assertion failed on expression: 'FenceHasBeenSynced(pendingCreationFence)'
Which I am wondering if it is something to do with the job system or is unrelated and a GPU thing.

What makes me think it's an issue with the job system is the editor crashes on one of the job threads with a call stack like this:

0x00007FF77B8D0F51 (Unity) profiler_begin
0x00007FF77B822669 (Unity) ForwardJobToManaged
0x00007FF77B81E4DA (Unity) ujob_execute_job
0x00007FF77B81D92D (Unity) lane_guts
0x00007FF77B820514 (Unity) worker_thread_routine
0x00007FF77BA4D876 (Unity) Thread::RunThreadWrapper
0x00007FFB1B857344 (KERNEL32) BaseThreadInitThunk
0x00007FFB1BB026B1 (ntdll) RtlUserThreadStart

I am currently working towards jobifying some code and am completing jobs on a scripting thread which I'm not sure is a supported scenario.

I also had a crash which I think output this as the managed stack (it's hard to tell as a few things are writing to the log at the same time)

[muted oyster]

Did you have a debugger attached when doing a domain reload (recompile)?

(for the first one)

[earnest niche]

it's certainly possible

[muted oyster]

using rider?

because this is something i often see and it's frequently changes a lot between rider and unity versions

detaching rider before you recompile avoids the crash

generally for me once this starts happening it's a near 100% repro

and mono debugger crashing is like the thing I am most looking forward to replacing in coreclr

[earnest niche]

visual studio

oh wait I'm getting them mixed up, the short one happens with nothing attached, eventually it was happening within seconds of opening unity

[muted oyster]

hmm i saw the profiler begin one recently

but i haven't seen it in maybe a week

maybe after b12 update

not sure

[earnest niche]

this is in 2022.3.21

[wintry mist]

completing jobs from a non-main thread is definitely a no-no

[earnest niche]

I'm a good boy now and don't do that, but I still get the crashes randomly

[wintry mist]

if you can turn this on to full dumps https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps and then send @serene valve the result, i bet he could figure it out. after you fixed the completing jobs from non-main thread thing, do you only get the short callstack or also the long one? and, do you have fast enter playmode enabled (so avoiding domain reloads on enter playmode)?

[earnest niche]

nope, full domain reloads

I should also note I haven't experienced the problem with 2022.3.11, it was only after upgrading to .21/.22 that I started getting it

[wintry mist]

ah could be connected to the recent batching thing. that said why do you have domain reloads on enter playmode?

[serene valve]

The fact that assertion failed is a bit concerning, it has to do with the job system and managed jobs. We need to have some structures allocated and initialized before managed jobs can execute and it's saying for some reason those structures haven't finished allocating/initing yet. I've seen other reports elsewhere that seem to suggest we have a bug in this area of code so if you manage to reliably hit this crash, getting a reproduction would be awesome.

[earnest niche]

I reproduced the crash but I didn't get a dump, despite just previously checking that dumps were being saved with ForceCrash

[modest karma]

are you sure its a crash then and not a sneaky app exit?

[earnest niche]

yes I get the unity bug reporter and it saves it's own dump

@wintry mist is there something special I have to do to get reliable crash dumps? Do I need to disable the unity bug reporter?

[wintry mist]

Yes, this: https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps set dumptype to 2 (meaning full), and then find it in the crashdumps folder in appdata it talks about. It will be large but it compresses well

Crash reporter does not get in the way though

[earnest niche]

I did that from before (though I set a custom dir) and I was able to get one when using ForceCrash but the crash reporter came up twice

[earnest niche]

whoops, I think the issue was I ran out of disk space after the first dump

(and ForceCrash(AccessViolation) doesn't trigger a WER dump)

this is how it's configured

I got another crash after making everything else was ok and no dump

[earnest niche]

ok I managed to get a full crash dump of this one

>    Unity.exe!profiler_begin(class profiling::Marker *)    Unknown
     Unity.exe!ForwardJobToManaged()    Unknown
     Unity.exe!ujob_execute_job()    Unknown
     Unity.exe!lane_guts()    Unknown
     Unity.exe!worker_thread_routine()    Unknown
     Unity.exe!Thread::RunThreadWrapper(void *)    Unknown
     kernel32.dll!BaseThreadInitThunk()    Unknown
     ntdll.dll!RtlUserThreadStart()    Unknown

[wintry mist]

upload somewhere for @serene valve ?

[earnest niche]

@serene valve can I dm you a link?