Problems debugging job allocations

[sudden lodge]

Hi, In my builds I get this message in the logs repeatedly:

To Debug, run app with -diag-job-temp-memory-leak-validation cmd line argument. This will output the callstacks of the leaked allocations.```
However, I've tried launching the .exe through cmd with this argument... and this message still appears instead of the full callstack.
I've also tried adding this to the launch options in unity hub but that doesn't help since these mostly occur in build.

Also, I've tried setting `NativeLeakDetectionMode` to `NativeLeakDetectionMode.EnabledWithStackTrace` via a script in Start() in case that has something to do with it, but this option doesn't seem to do anything either.

[sudden lodge]

Worth mentioning that my code doesn't use jobs, so it must be an asset that I'm using, or core unity code.
I've read some posts about similar issues with this and heard that the messages are often fairly safe to ignore, but I get a ton of them in build, alongside some crashes with no clear discernable cause. My assumption is that the crashes are caused by the memory leaks (since I have nothing else to go off of in determining their cause).

[left hornet]

Regarding the log message, if you can't get it to show callstacks, the only thing you can do is start disabling features and scripts until it's not there to confirm what the culprit is.

Regarding crashes, you should start by looking at the editor/player logs for the crash report info.

[dense bramble]

I've always enabled diagnostic flags through the Preferences menu

I haven't tried to use command line flags before

See if tha makes it start providing some information

[warped tundra]

Try reproducing it in editor first

1. Edit > Preferences > Jobs > Leak Detection Level, set to Enabled with StackTrace
2. Play your game in editor
3. Stop playing your game
4. You need to trigger a domain reload, right click any script and pick reimport
5. You should now see the leaks in the log

[dense bramble]

Oh, I'm a dingus. I missed that this was only appearing in builds.

Definitely try to reproduce it in the editor!

[sudden lodge]

the logs don't have anything meaningful in them, I'll post one of them: https://pastebin.com/M9qYb7jG
(the machine pistol thing was just a coincidence, it happens at random moments)

[left hornet]

UnityEngine.Rendering.ScriptableRenderContext:Internal_Cull_Injected (UnityEngine.Rendering.ScriptableCullingParameters&,UnityEngine.Rendering.ScriptableRenderContext&,intptr)
0x00000260FEF29963 (Mono JIT Code)

This looks like something that is doing work on your custom feature. And the next frame in the stack calls a managed function, but fails with invalid address:

ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FFC9DEF0270)
0x00007FFC9DEF0270 (UnityPlayer) (function-name not available)
0x00000260FEF29A28 (Mono JIT Code) (wrapper managed-to-native)

That might imply that a managed object is destroyed at that point.

[sudden lodge]

trying this but I can't replicate it in the editor at all

it didnt occur from anything specific in the playtests i did, seemingly random which makes reprod tricky

[left hornet]

Does your render feature handle culling in any way? Or provide a list of objects to render or something?

Ah, wait, your not the guy that had a custom render feature..?

[sudden lodge]

... this might actually be the wrong log, I was having an issue before with a corrupted lightmap that crashed 1 scene on load, and the logs for that had some stuff to do with the custom render feature in them

[left hornet]

Then what's the correct log?

[sudden lodge]

hang on, I'll contact my playtesters and try to get their logs, and also try to reprod it in on my pc

sorry about that

but yes, I do have a custom render feature and it is possible it has something to do with that. it gets objects with a specific renderlayer and creates a mask to mask them out of an effect

[left hornet]

It does sound like you have a custom render pass though. Which might be related:
RecordCustomRenderGraphPasses

[left hornet]

I'd double check that you don't have null or destroyed objects in that list.

Release builds might be stripping null checks in the engine code causing such invalid memory access exceptions.

[sudden lodge]

alright, got the logs from a playtester. (still couldn't repro on my pc). I had a look and couldn't find any mention of RecordCustomRenderGraphPasses (so those must have been from the corrupted lightmap issue I was having, since resolved).

https://pastebin.com/BEmRX4Uq
https://pastebin.com/4A6NFK1K
these are definitely the correct logs. interestingly, the first one has Invalid memory pointer was detected in ThreadsafeLinearAllocator::Deallocate! sprinkled everywhere, which I thought might be the culprit... but the second log I've posted (still from playtester) doesn't have this line anywhere and still crashed, so I'm not sure

it might be a hardware issue of some kind, that tester's pc is fairly old (still something I'd like to fix of course) because when I play the same build on my pc I still get the JobTempAlloc and sometimes the InvalidMemoryPointer logs, but it generally doesn't crash even after playing the game for a similar amount of time, and doing the same things in-game.
(this crash definitely still occured on my pc once, unless it truly is a hardware issue and that 1 time was something unrelated).

[left hornet]

Looking at the second crash stack trace, you can see it's transitioning from mesh clear to invalid memory access, and it's all called from some sort of third party asset:

  ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FFB36DD6A50)
0x00007FFB36DD6A50 (UnityPlayer) (function-name not available)
0x00007FFB37AE64E4 (UnityPlayer) UnityMain
0x00007FFB37ADDD8F (UnityPlayer) UnityMain
0x00000233FD0A0AFD (Mono JIT Code) (wrapper managed-to-native) UnityEngine.Mesh:ClearImpl_Injected (intptr,bool)
0x00000233FD0A0A3B (Mono JIT Code) UnityEngine.Mesh:ClearImpl (bool)
0x00000233FD0A09A3 (Mono JIT Code) UnityEngine.Mesh:Clear ()
0x00000234254C33B3 (Mono JIT Code) Coffee.UIExtensions.UIParticleRenderer:UpdateMesh (UnityEngine.Camera)

The first one stack trace is unreadable for some reason.

[left hornet]

There is something curious about the first crash though.

0x00007FFB37117E6B (UnityPlayer) UnityMain
  ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00007FF69C5911F2)
0x00007FF69C5911F2 (Cathode) (function-name not available)
0x00007FFB7A1E7374 (KERNEL32) BaseThreadInitThunk
0x00007FFB7C1FCC91 (ntdll) RtlUserThreadStart

Since a function in your assembly comes almost immediately after thread start, that might imply that it's not the main thread, but a thread created in your code. Do you know any instances of creating threads in your project(it could be an asset/plugin too)?

Actually, maybe that's normal, seeing how the other crash stack trace starts in the same way. Gotta check that to be sure.

I think the best course of action would be to open a crash dump in visual studio with your debug symbols outputted during the build and have a look around.

Yeah, actually. A lot of the invalid address stuff is probably due to lack of debug symbols rather than runtime errors.

[sudden lodge]

that's interesting, it only appears in that particular log (they also sent me another one, and I checked, no reference to that asset)

[sudden lodge]

I haven't touched jobs/multithreading personally, I'll have to check through my assets

my tween package I'm using supposedly doesn't use threads so it's not that, not sure off the top of my head what else it could be

[sudden lodge]

I think it's normal, checked a log from the lightmap corruption crash and it also had that, including the 'function-name not available' etc

[sudden lodge]

haven't done this before but I'll look into it

[sudden lodge]

ah, it looks like I have to do it through the command line in vsc, there's no gui for it

might have to look into installing visual studio then

only meaningful thing I could get from the command line was crashinfo printing ERROR: No crash info to display which is slightly worrying

[left hornet]

Yeah. I'm not sure how much you can see with vs code. With VS you should be able to see the state of the program at the moment of the crash. To a degree...

[sudden lodge]

all the logs look like this

[sudden lodge]

hmm, its late here but i'll look more into this tomorrow, can't seem to find anything more in the dump files

[copper lichen]

If this is a windows crash then you want to use vs to either read the minidump or attach the debugger to the native process to get more info on the crash.
Unity have a vs extension to repair native + managed stacks which can sometimes help

[left hornet]

That's just the crash reason. You should be able to attach the debugger to the crash dump to see more info.

[sudden lodge]

alright, I think I've found something

Animator::BuildJobs is in there... that aligns with what we've been seeing in the player.logs' mentions of JobTempAlloc

the first and third .dmps have this, but the second one has something else:

[sudden lodge]

research isn't finding anything, though I'll try updating burst and animation rigging package

though since it's a crash in Animator, which is core, maybe I should update unity version as well

[left hornet]

Don't quote me, but it feels like memory/heap corruption. In both cases invalid memory access seems to occur when trying to allocate new memory, which seems to trigger cleaning up unused pointers, which actually accesses the invalid memory.

It could technically be an engine bug, so updating the minor version is a good idea.
Othet than that, it could be caused by unsafe code that messes with pointers, native collections and such. I'd double check for such code in your project and third party assets and plugins.

Now I feel like this provides some evidence that the warning you were getting about JobTempAlloc could be somewhat related, so perhaps it's worth investigating it in details as well.

1. Try reproducing the crash with the program launched with -diag-job-temp-memory-leak-validation args.
2. Try enabling various job/leak detection/debugging options to see if you can get more details in the editor.
3. Try doing a global text search on your project(including package cache/library for packages code as well) looking for instances of Allocator.TempJob. Start by focusing on in project code and third party packages.

[sudden lodge]

alright, sure

I can barely replicate this crash on my machine, it's only ever happened for me once, I tried last night for ages and it didn't happen (there's seemingly no trigger in-game, I don't know of any way to force it to happen)

and when I play it in build with the diag-job-temp-memory-leak-validation argument nothing changes about the log... I'm assuming something would be different if it crashed but I can't reliably get it to happen (the JobTempAlloc logs don't change at all if they're supposed to)

hmm, I searched for all mentions of allocator.tempjob, and all of it is core unity, but interestingly the splines package uses them a ton, and my project has a lot of splines

I don't use unity splines directly but it is a dependency for the third party splines package so I have a feeling it might be that, I will try updating that as well

[left hornet]

How are you using the arg?

Also, what Unity version are you using?

[sudden lodge]

6000.2.3f1

[left hornet]

I wonder if that could be interpreted as several args.

Maybe try Cathode.exe "-diag-job-temp-memory-leak-validation"
Also make sure you're testing a dev build. And check the player logs. I think they might mention if the game islaunched with the arg or not.

[sudden lodge]

I've built a dev build, and also tried it with the arguments wrapped in quotations and without... neither of the logs from those sessions have any mention of jobs or invalidmemorypointers, except this bit at the start:
kGfxThreadingModeSplitJobs is not supported on Direct3D 11. Reverting to kGfxThreadingModeClientWorkerJobs instead.
GfxDevice: creating device client; kGfxThreadingModeClientWorkerJobs

no mention of launching with the arguments in either attempt

it is possible that updating my assets just fixed it though :P

[copper lichen]

From what i read, the editor diagnostic toggle should do the same thing. that build launch arg seems to be a weird undocumented feature thats for internal use so perhaps its no longer functional

[sudden lodge]

interesting to know, thank you

I've tried NativeLeakDetection = NativeLeakDetectionMode.EnabledWithStackTrace but that didn't seem to do anything, I will try again with the option in Editor Diagnostics -> Core

[sudden lodge]

slight development, got this in the editor: (notably not in play mode so it could be unrelated)