GPUskincache.cpp crash

[sinful prism]

I´m making a thread for anyone else searching:
I was able to reproduce the crash with a simplified blueprint actor. Need to do a few more tests to further pinpoint the origin.

[sinful prism]

Ok, quick update: I found a workaround for now.
I had to disable "Support compute skin cache" which is enabled by default if your project settings are using "Support hardware raytracing".
Metahumans use that setting for something, my metahuman character still works, so I´m not sure wether its save to just keep that off and what it even does.

[sinful prism]

@quasi karma This is the workaround I´ve found for now.

[quasi karma]

thank you! i am having hardware raytracing on, so i cannot turn this compute skin cache off. and i am using metahumans as well. seems this is a rendering problem then, i thought it was because of using skeletal merging in my character bp

[sinful prism]

Yeah, no, its confusing sometimes and I´m always blaming myself first, if I´m doing stuff I don´t really know much about...:)

Raytraced shadows also don´t work with nanite tesselated materials yet, so I´m currently going without it anyways.
I´m also using metahuman, I haven´t tested it properly yet, I think the biggest issue is the hair groom, if you have compute skin cache disabled.

[quasi karma]

also there're a bunch of warning about getSocketInfoByName of no skeletal mesh found for my charactermesh component

[sinful prism]

Ah, I didn´t check for that.
Are you using modular control rig?

I noticed that also didn´t work for my skeleton, as the naming convention wasn´t the same.

[quasi karma]

you meaning the animation rig section under the skeletal mesh?

no, i have no control rig set to my skeletal meshes, i'm just using the metahuman face and hands and combine with my own clothes assets, they clothes and the hands are all reassigned to ue5 default manny skeleton, only the face using the meta archetype skeleton

[sinful prism]

I´m still troubleshooting this, it ain´t that easy...
I reenabled skin cache and started with a clean map and both the metahuman and my custom blueprint actor didn´t crash the editor.
So I gotta narrow it down further.

[waxen grove]

We are digging through this too

Random groom crashes

all associated with DDC

[quasi karma]

hey, i did the similar as well. I use a new empty map as the start and default map, it won't crash again , and the project opening is way faster as well. I think this should be the way as a open menu map and i'll stick to it. Now when i open the previous crashed map, it will still crash for the same GpuSkinCache. Must be something in the map. But the thing is, i have other maps with the same character bp in them using some of the metahuman parts as well, and those maps didn't crash. And the crashed map is just a test map, and i can still play it while i'm using debug game and start debug to open project, so i'll just leave it for now

[sinful prism]

Yeah, I have to fix some other stuff and can´t troubleshoot this further rn now either, so I´ll leave it at the workaround disabling compute skin cache, as we don´t need grooms very often atm,

[quasi karma]

i'm thinking of a way to debug it, since we guess it's probably the characters' problem, maybe just copy the characters to a new map and see if it crashes. if it didn't , then copy some other stuff until crash... if it did, then deleted some and narrow down to maybe a few. i haven't got around to try it yet, but since my character doesn't have this problem in other maps, i think probably it's the character in this particular map has some issue.

[waxen grove]

we blew away all ddc cache for old cache, new cache, global cache, and local cache, and it fixed it, its 100% a cache issue

[quasi karma]

👍 how did you do that? the groom still works fine after fix?

[sinful prism]

I´d like to know as well...Is that the engines ddc folder?
Because there is nothing in my projects derived data cache folder and I´ve excluded that folder from source control.

[pliant bough]

C:\Users[UserName]\AppData\Local\UnrealEngine\Common\Zen

https://dev.epicgames.com/documentation/en-us/unreal-engine/using-derived-data-cache-in-unreal-engine?application_version=5.4

groom assets are broken with packaged games using UE5.4.1, the engine crashes when characters wearing groom assets start to ragdoll

https://forums.unrealengine.com/t/ue-5-4-1-groom-crash-with-package-buiids/1848334

[hard igloo]

is that the right place to clean DDC zen data? seems to be a whole separate thing in C:\ProgramData\Epic\Zen ?

[devout osprey]

@hard igloo I'd like to know as well

alot of people are reporting DDC issues after the UE5.4.2 hotfix if the oculus source engine is along side launcher

for whatever reason launcher is still using older broken ush file cache

[hard igloo]

I had the oculus branch so that maybe caused the issue ; do you have a link to any of the threads on it? I deleted both places and it didn't seem to put it back in C:\ProgramData\Epic\Zen anymore, but I probably haven't run the oculus branch or launcher version since deleting

[devout osprey]

I’ve seen post in the #cpp and #graphics on it

Also on VRE discord

A few things in forums and oculus start etc

@hard igloo

[sinful prism]

Not using the oculus source engine to my knowledge...or is that enabled somewhere by default and I have to disable it?

[hard igloo]

It's a special github branch, you most likely don't have it

[sinful prism]

Alright, so, since I need raytracing for the current project, I did some more troubleshootig.
Copied the whole project to a temp folder, then tried the skeletal meshes involved one by one, no crash.
Then tried the blueprint containing them, placed it in the level, no crash.
Hit PIE and boom, same crash as always, pointing to one specific line in the GPU skincache code.

So, I went through the blueprint and disconnected the animation setup function, where I use a lead pose component to control all skeletal animation data in one place.
No crash.
Reconnected that, crash.
So, I´m pretty sure its got something to do with that...
Currently no idea how to work around it, if its indeed a bug, no mention anywhere that lead pose component isn´t compatible with skin cache/raytracing.

I´m now creating a simplified blueprint containing only two skeletal meshes with one acting as lead component, see if that still crashes.

Jup, crashed again.
one more test, gonna try a different skeletal mesh/rig...

I´m currently not able to test this in a new empty project, I can only work remote and for some reason it seems like the epic games launcher doesn´t open correctly using remote desktop...

I´m gonna see if I can also recreate this in 5.3

Ok, seems like the issue isn´t there in 5.3.
Gonna do some more testing to confirm, but could really use some help with testing, because I can´t test in a new 5.4 project rn...

Also, seems I didn´t repost the error message thats the same for all crashes, so here it is once more:

[2024.07.02-13.07.00:108][933]LogWindows: Error: Assertion failed: DispatchData.PreviousPositionBuffer != DispatchData.PositionBuffer [File:D:\build++UE5\Sync\Engine\Source\Runtime\Engine\Private\GPUSkinCache.cpp] [Line: 2056]

[sinful prism]

Thanks, I just checked and tested with enabling that flag, still crashes.

[pliant bough]

Hi, I've reverted back to UE5.3 and upgraded VS2022 for the recent compiler bug:
https://community.gamedev.tv/t/compiling-issue-ue5-3/241963/2

The groom assets work as expected in 5.3 packaged game

If the groom assets stop simulating physics, add the groom plugin folder into the cooked assets
https://forums.unrealengine.com/t/groom-not-simulating-physics-but-only-in-packaged-game/484451/5

[flint forge]

Hi, we've been seeing this crash in our project primarily in 5.3 - but I think we also saw it in 5.2

I'm currently doing some investigating into it.
For us it's specifically in 1 level where we move static characters around by placing instances of them in different data layers, and then load in the relevant data layers based on the runtime state of the game save.

When we load in all these relevant data layers on level load, it's totally fine - however sometimes we will unload/load layers mid-gameplay, and for us a specific few days layers (unsure which yet) are causing this crash to happen.

I'm presuming that one of the characters in one of those layers is using a slightly different setup compared with all of the other characters, so I'm currently doing a full investigation to work out which one, and then which part of them is causing the issue to occur.
We've got a 100% repro for it in a packaged build, but cannot repro it in editor.

Just throwing my hat into this thread to flag another developer who is struggling with it, and also investigating the issue!

We have Skin Caching enabled explicitly on our character skeletal meshes - at least, for their heads which are metahuman, and for their body mesh which is custom. Their outfit skeletal meshes do not have skin caching enabled explicitly, but are set to auto which implies enabled if they support ray tracing, which they do.

The project is set to "Support Hardware Raytracing" and "Support compute Skin Cache".

We do not use Groom Assets, however we used to and they're still present in the project (but not the game). I'm currently auditing all of our blueprints to check whether any Groom Components remain, despite none of the groom assets or bindings being referenced by any of our game assets anymore.

[flint forge]

I've potentially found something - just doing the work to "fix it" and then will check out a packaged build this afternoon to confirm if it worked or not.

I've found that whilst we don't use Groom assets anymore, and we've removed all the groom components from the character blueprints a long time ago, the placed instances of some of those characters still reference the "GroomComponent" class as they haven't been resaved since the change to their blueprints.

I found this by first of all enabling "Show C++ Classes" and "Show Engine Content" in the content browser, and then navigating to "Engine".
I then searched for "GroomComponent" and right clicked on the C++ class and used the reference viewer.

This showed up various actor instances in levels that still referenced the class, despite the actual blueprint asset no longer holding the component type. Simply opening up the levels, and resaving the specific instances cleared out this reference upon a resave.

I'll let you know in a few hours time if this has actually resolved the issue for us at all

[flint forge]

I can confirm that this has fixed the issue for us.
Summary of the above: We used to use groom components, and then we removed them from character blueprints (months ago). However, instances that were already placed were holding onto a reference to the groom component as the instances had not been resaved since the change to the actual blueprint.
Simply resaving all of the instances so they no longer reference the groom component class fixed the issue by flushing out all rogue references to invalid/non-existent groom components.

It was previously a 100% repro on this issue, and now it is not happening.

[sinful prism]

Thanks for sharing.
I´ll look into this, but since I also tested this with a new blueprint actor using only two identical skeletal meshes, both from mixamo and neither using any groom components and was able to 100% reproduce this ONLY using "lead pose" component node, I´m pretty sure its not related to groom for me.
I only use Unreal for linear content production (aka rendering cinematics), so not dealing with any packaging here.

I´ve also made a post on the Unreal Engine forum:
https://forums.unrealengine.com/t/editor-crash-with-gpuskincache-cpp/1827647/13

[sinful prism]

Ok, just got another confirmed crash when using "lead pose component" in a blueprint actor.
Pretty sure thats the culprit, if anyone else can test this, please do and let me know!

Steps to reproduce:

1. Enable "support hardware raytracing" which should enable "use compute skin cache"
2. Create blueprint actor.
3. Add two skeletal meshes (name one leader, one follower)
4. In the constructionscript, add a “Set Leader pose component”.
5. Assign leader and follower.
6. Compile.
7. Hit “Play in editor”.
   You should crash.

[sinful prism]

1. Set up lead pose component with a skeletal mesh you are NOT also rendering (If you don´t have one, duplicate the original skeletal mesh, rename it and reimport it. I´ve started using a different skeletal mesh for importing the rig, if I use the same rig for several skeletal components).
2. In the skeletal mesh editors detail panel, disable "support raytracing". You can also disable "visible in raytracing".
3. Uncheck "visible" and check "hidden in game" tags.
4. Set it up as lead pose component.
5. I´m not actually sure if the next two steps are required, but I´m too lazy to test again...
6. In ALL skeletal meshes in your blueprint that use the same lead pose component, go into their details panel in the blueprint and check "Use Bounds from leader pose component" and "Ignore Leader pose component LOD".
7. Enable "use hardware raytracing", restart and enjoy!

EDIT:
...too soon. it actually crashed again when I hit PIE...
I´ll stop for today...

[sinful prism]

It may still be an improvement, as it means I can now at least use path tracing anywhere else in the project with that blueprint, if I add a switch to disable the lead pose component.
Or at least thats what I hope, as it doesn´t crash on startup anymore, when I have raytracing enabled.

[sinful prism]

So, this is my current workaround for the crash:

[sinful prism]

Update: Unfortunately its still happening in Unreal 5.5 and only "workaround" is to either not use raytracing OR not use Lead pose component.
I was hoping I could work around by using a control rig somehow, but turns out that would also require lead pose component to use ONE controlrig to control the whole modular character blueprint.
I´m out of ideas and I can´t figure out why my bug report isn´t submitting.
The bug isn´t listed anywhere in the bug database as far as I can see, I would really appreciate someone else confirming the crash....

[sinful prism]

Actually: Wasn´t able to reproduce this in a fresh project and did some more testing and it might actually be related to grooms after all.
Still doesnt make sense, since I´ve seen plenty of unreal pathtracer metahuman videos, which implies lead pose should work there as well.
Maybe its some weird combination of settings in my blueprint, but after deleting the groom assets, it started not crashing with hardware raytracing.
Looking into it some more to confirm

[sinful prism]

Damn it, nope, still crashing.
Something real funky going on here: It worked when I first created the level/added the blueprint and switching to path tracer.
But after the next restart, when I try to open the level I get an editor crash.
So, need to go back and recheck if that also happens with the empty project, if it doesn´t its gotta be somewhere in my current project:
Either some cache issue (but I´ve already deleted the saved/intermediate/derivedDataCache folders for the project) or something in the blueprint thats causing this and thats gonna be tough to track down.

[gusty lantern]

Let me know if I can test or help anyhow. Having the same issue.

[gusty lantern]

Thanks for the idea! I just tested it and confirm that simply replacing the "problematic" assets resolved the issue atleast for me. @sinful prism do you mind sharing a still problematic project of yours? I could test it.

[flint forge]

I've just upgraded our project to 5.4.4 from 5.3.2 and seen a regression of this issue. I simply decided to disable skin cache support in project settings as it's only really required for 2 things we don't currently use (Groom assets, and Hardware Ray Tracing).
Disabling it (quite obviously) fixed the issue for us in 5.4 😂

[gusty lantern]

Bummer, your tip didn't do the trick? It's really hard to investigate this bug.

[flint forge]

There's a potential fix in 5.5, I may have seen that the other day on UDN potentially 🤔

[gusty lantern]

hm can't find any relevant changes in the 5.5 branch/ main. But there has been some major changes from 5.3 -> 5.4.

[sinful prism]

Yeah? Because I tried 5.5 explicitly for this, hoping that the skeletal mesh nanite support would maybe fix the issue, but that was not the case unfortunately, not out of the box at least.

Damn, so much to do before my next project starts, but I´ll try to squeeze in one more test with the empty project to check if I can reproduce the error without the groom component. Its weird because afair the crash didn´t occur right away, it only occurred after closing and reloading the project.

I still don´t get why this wouldn´t be more widely reported, if all it took was lead pose component+hardware raytracing, again, metahuman uses lead pose component as well...

[gusty lantern]

My assumption is that it mainly happens with assets / projects migration from 5.3 -> 5.4

Atleast that's my hope 😄

[flint forge]

Have you attempted to resave all assets in the new engine version before hitting play? Wondering if they need fixup for baked data in their assets between engine versions (could be old invalid data baked in).

I haven't personally tried that

[gusty lantern]

That's exactly what I did.

[sinful prism]

Yeah, I deleted all intermediate/saved folders in the project, no difference.

[gusty lantern]

That might be not enough. I had to delete the actors in the levels and replace them.