Help with memory allocation issue

[slow rivet]

My windows build standalone game eats up 18GB of memory allocation going from one scene to another, after which it drops down to 10-12GB all the way through.

I am trying to troubleshoot what could be causing it, and here's some screenshots of profiling.

How do I go about figuring out what that could be? Thank you!

[shell fjord]

This is likely taken after the memory is freed, so it stays as reserved. Might want to use the regular profiler to see where the memory peak is and make a proper capture at that point to see the real culprit.

[slow rivet]

I did capture the regular profiler (first screenshot is of that). But not sure what I should be looking at. The allocated memory gradually grows to those levels

[shell fjord]

Not capture. Keep it recording while you play the game and monitor the memory module timeline to see when it goes up.

[slow rivet]

Oh, this is happening going from my title scene to lobby scene.

When the lobby scene shows up, the memory falls to 10-12

[shell fjord]

Well, you'll need to capture that timing. Or maybe the menu just before transitioning.

[shell fjord]

Does it show any details if you click the first object?

[slow rivet]

No details at all when I click the object. Just the <No Name>

[slow rivet]

Can you elaborate on this please? When you say capture the timing?

So the scene is a simple Title scene with a background, name, logo etc. and a click to begin text.

When a player clicks on it, it loads up all the game managers into DoNotDestroy, an object pooler/spawner, loads saved game slots from json files, and then opens up a new scene with 95% UI and barely any 3D objects

[shell fjord]

For diagnostics purposes, disable the new scene loading, trigger the transition, then make a memory profiler capture and have a look at it.

Basically you need to know what the state of the memory at the moment just before a new scene is loaded.

[slow rivet]

Ok, I am trying to understand this (sorry, this is pretty new to me) -

1. In the first scene, all I do is SceneManager.LoadSceneAsync(scene); on mouse click
2. In the second scene, all the managers, object pooler, etc. are in the hierarchy.
3. So if I disable the scene loading, how do I get to the issue?

[shell fjord]

First of all, if that's really all there is to the first scene, then yes, there's no point.
However, just to be safe I would still make a capture in the first scene.
Debugging is about confirming assumptions. Not building assumptions on top of unconfirmed assumptions.
Second, you should still monitor the regular profiler memory module in real time during that scene loading. It might provide clues as to what kind of systems are using that memory and at what point.
Third, once you have a grasp of what exact moment the allocations happen, you can trigger memory snapshot capture programmatically.
https://docs.unity3d.com/6000.3/Documentation/ScriptReference/Unity.Profiling.Memory.MemoryProfiler.html

[slow rivet]

Thank you, let me try that out.
"disable the new scene loading, trigger the transition" are you saying I should do SceneManager.LoadSceneAsync(scene) but with the loading screen hidden?

[shell fjord]

No. I was talking about commenting out SceneManager.LoadSceneAsync(scene) to see if it's something about your loading logic before the actual scene load.

[slow rivet]

I have 3 snapshots -

1. On Title scene
2. Halfway through the load of Lobby Scene
3. Almost before Lobby Scene loads

Here's a comparison of 1 and 2

[shell fjord]

Before any snapshots, as I said, you should be looking at the memory module timeline.

[slow rivet]

And this is comparing 2 to 3

[shell fjord]

In real time. Possible pause it if you see spike and drop in the metrics so that the frame does not escape outside the monitored range.

One extremely big bad red flag that you should pay attention to immediately is this.

You should be comparing editor to editor or build to build. Not mixing them. And, of course, editor would have a lot more memory allocated than a build.

[slow rivet]

All of those were captured same way (standalone player). I think that is just a warning, but thanks for pointing it out

[shell fjord]

What does it say if you expand it?

[shell fjord]

I'd double check that all the captures are really taken in build.

[slow rivet]

This is while I wait on the title scene without clicking

Anything I should check before proceeding?

[shell fjord]

Not really. Just keep an eye on the timeline/graph as you load the new scene.

[slow rivet]

The graph does not capture the whole transition, any way to capture longer duration?

It only shows last 300 frames or so

[shell fjord]

You can change that in the editor preferences. But shouldn't 300 frames be enough? that's like 5 second. Enough to stop recording.

[slow rivet]

It takes about 10 seconds from click to when memory goes up to 18GB and next scene loads

[shell fjord]

I see. Well, even capturing the last few sec of that 10 seconds climb should be enough imho.

[shell fjord]

You should seelct a frame that is more representative of the actual memory climb.

[shell fjord]

Ah okay I see.
The first thing I'd try is to disable deep profile.

[slow rivet]

Oh, I assumed that would help with deeper analysis. What's the reason to not use it?

[shell fjord]

It would only help with performance analysis. Not memory.

The reason is that it could be allocating quite a bit of memory on it's own. There's a chance that that unlabled native memory is coming from it.

Though, they have a dedicated category for it. So maybe that's not it. Still, worth a try.

[slow rivet]

This is without deep profiling

[shell fjord]

So, is that in the middle of the scene loading?

[slow rivet]

Yes

This is right before it finishes

And this right after. You can see the memory drop by around 8-10GB

[shell fjord]

So it's all coming from the "other" from the very start.🤔
Are you able to make a detailed memory profiler capture at that point?

I mean at the point before it drops.

[slow rivet]

Using memory snapshot?

Not sure how to time that. The capture is a point in time and I need to click that capture snapshot at exact instance

[shell fjord]

Doesn't have to be exactly that point of time. But somewhere during the loading. 10 sec should be enough time to alt tab to the memory profiler and press the capture button.

Also, what unity version are you using?

[slow rivet]

Yeah, so that's as close as it gets. Maybe a few seconds before scene loads

[slow rivet]

2022.3.62f3

[slow rivet]

Got the capture. What should I look at?

[shell fjord]

I see. Since it was an LTS I don't think it's an issue with the engine.🤔

[shell fjord]

Look at "all memory" tab breakdown sorted by memory consumption.

[shell fjord]

Okay. And what do you see in the details on the right if you click one of the objects(<No Name>)?

[shell fjord]

Right... So, something is allocating these huge objects. It could be native arrays or buffers for example.
The next thing to do is to try loading an empty new scene instead of the one you're loading and confirm that the huge memory allocation is gone.
When you can confirm that, the issue is likely somewhere in the scene - some code allocating a lot of native memory or maybe some kind of third party asset/plugin.

[slow rivet]

Is there a way to add debugging code to spit out names of objects?

I am loading all the managers on the next scene (characters, enemies, weapons, loadouts etc.) where I use a prefab and an image for each one.

But there's no way that adds up to even 1GB

It's all simple Synty models and textures

[shell fjord]

Yes, usually you can set name for all unity Objects via the .name property. But you need to know where you are creating them. 9GB objects is very likely some buffer, native array or a texture. It's very unlikely to be a normal data object.

[shell fjord]

My guess is this is coming from some allocation in code and not any specific asset.

Anyways, test with the empty scene first to confirm that it's an issue with that specific scene.

[slow rivet]

One thing I noticed in here

Not sure what's the cause and what's the effect, but same time the spike goes down, the files opened drops

I am assuming these are all the files I read for settings, enemy waves, saved games etc.

[shell fjord]

Are these big files? Are you using asset bundles or addressables perhaps? What's your build size?

Or are you reading files in your own code/third party assets somewhere?

[slow rivet]

Simple json files I read/save with Unity JsonUtility

[shell fjord]

Is this your custom file format? How are you opening them for read?

oh, it's json.

[slow rivet]

public static WeaponStates LoadWeaponDetails()
{
    string path = Application.persistentDataPath + "/GameWeapons.decayed";
    if (File.Exists(path))
    {
        string json = File.ReadAllText(path);
        WeaponStates weaponStates = JsonUtility.FromJson<WeaponStates>(json);

        return weaponStates;
    }
    else
    {
        return null;
    }
}

public static void SaveWeaponDetails(WeaponStates weaponStates)
{
    string path = Application.persistentDataPath + "/GameWeapons.decayed";

    string json = JsonUtility.ToJson(weaponStates);

    File.WriteAllText(path, json);
}

[shell fjord]

I feel like these are not the culprit. But you could try commenting this out and test to see if the allocations(and file reads) are still there.

[slow rivet]

So I tried using ChatGPT to help troubleshoot, and it led me to this -

Your culprit is 100 percent confirmed.

Your game is exploding because of this line:

Shader (80 Objects) — 2.74 GB native memory

That number should be:

5 MB

20 MB max for shader-heavy games

But yours is 2740 MB.
That is catastrophic and absolutely the reason Steam Deck crashes before title screen.

Nothing else in your snapshot is even remotely problematic.

I can never tell whether it's confidently spewing bs or not

[shell fjord]

It's not entirely wrong. Shaders should really not take that much memory, however, I'm not sure this is the real culprit behind the 17GB consumption.

Did you try any of the things I suggested yet?

[slow rivet]

Ever since I have switched to BetterLit shaders, this has been a pain

[slow rivet]

Yes, been testing out the save/load changes. But having to modify all my managers to account for not loading/saving from files

[shell fjord]

Sounds like an architecture issue. Your managers shouldn't explode if there's no data available.
But anyways, my higher priority suggestion was to try loading an empty scene.

[slow rivet]

If there's no files, I initialize them, save new files and my managers read from those

[slow rivet]

Bizzare, I took out references to 8 of the files (no save or load) those methods are commented out literally

But still shows 16 file handles open

[shell fjord]

Then this is not the files that were tracked. In the first place, it makes sense that unity wouldn't track regular(non unity api) file reads.

[slow rivet]

But good to get that tested and out of the way.

[shell fjord]

Yep

[slow rivet]

I am going to try and work the shader angle next unless you have other ideas I could try out?

[shell fjord]

Did you test with an empty scene yet?

[slow rivet]

Barely a blip when loading empty scene

[shell fjord]

Then yes, I'd consider shaders as one possible culprit.

[slow rivet]

Trying to mess with shader control (Kronnect). It shows a huge list, but I have no idea how to figure out which ones to exclude

[shell fjord]

Generally, unity should be handling it for you, but I'm not sure about how it works with this third party asset.

[slow rivet]

How does Unity handle it? I thought that was the whole reason for a 3rd party asset

[shell fjord]

I'm talking specifically about shader variants. It checks what features(keywords) are used in the scene, then strips shader variants that are not used at all.

That being said, I'm not sure why your shader file is several GB in the first place. That would imply that it doesn't use variants and just bundles everything as one shader..? 🤔

It's kinda hard to say anything without having my hands on that specific asset.

[slow rivet]

From what I am reading up, that is how shaders are compiled. Every variation and keyword?

[shell fjord]

Compiled, yes. But not how they're included in a build. There should be separate shader variant files a lot smaller than what you have, included in a build.

Ah, actually, you never expanded the shaders category. What does it look like if you do?

[shell fjord]

What do you see on the right bar when it's selected?

[slow rivet]

Sorry, was at work today. Here's the right side

[shell fjord]

That's crazy...
Are you using addressables in this project BTW? I found some unity forum posts about shader size growing with addressables in some scenarios.

[slow rivet]

I am not sure what that is, so probably no? Any way to check?

[shell fjord]

If you're not sure, you probably don't use it. It's a way to load assets at runtime and it's not enabled by default.

[slow rivet]

Unless some asset pack I have uses it?

[shell fjord]

Unlikely. It requires project wide config. An asset can't set it up silently without you not noticing.

Anyways, that shader is extremely sus.

[slow rivet]

Yeah, I am messing with excluding some variants, and it broke my build completely (everything pink only in build, not in unity)

Thankfully, I do a complete folder backup, so restoring from that

And this does not include the Library folder which I don't back up

[shell fjord]

Should probably just use version control.

[worn dew]

Just followed the convo and wanted to ask. I saw, that you have posted the code for the loading part and json serialize, but what you actually do with those files has not been shown, right? I rather guess, when loading files, you are creating objects out of it, which may stack up for example. maybe you can show, what your managers do or even isolate one manager that is giving the most spike in memory.

[slow rivet]

The charactermanager for example just hold holds an array of characters -

1. It takes the details from the json file
2. It takes a prefab (through inspector) - synty low poly
3. It takes a 256x256 image to show

Total 12 characters. Pretty bare stuff

[slow rivet]

I do for my game. But the assets stuff is just too much to source control.

So if I change a texture, sound, prefab etc. I move it to my game/prefab folder for example and is source controlled. But if I am not changing anything, I dont source control the whole synty series, or my mega sound packs etc.

Ok, I restored the whole backup overnight. Today I plan on unchecking one shader at a time, building a standalone, and testing it out

Maybe I should think about switching from this betterlit to a lightweight shader where I dont need to worry so much about all these variants

[shell fjord]

If you don't need it's features, sure. URP default shaders are usually more than enough.

[slow rivet]

I like the flat shaded look with outlines, and stuff. Which is why I went with it

[shell fjord]

I'm pretty sure there are way lighter ways to get a look like that. That asset seems to be more suitable for high definition rendering, rather than flat shading and outlines.

[slow rivet]

Yeah, this was years ago when I added that shader without fully understanding the implications

[slow rivet]

Oh the joy of compiling all shaders...

[slow rivet]

And still...

[shell fjord]

My guess is that you're using a lot of different shaders with different options in your scenes, so unity has to compile a lot of variants. You usally want to limit the amount of shaders as much as possible and disable graphics features you don't need.

[robust sequoia]

FYI, the Project Auditor package lets you see all shader variants in the project (and a bunch of other useful info)

Or it shows the variant count per shader at least

IIRC

[slow rivet]

Thank you. But in the project is different from in the build right?

[slow rivet]

I use Beautify for most of my prettifying - tonemapping, bloom, vignette, outline, sscc, etc.

Do I need BetterLit shader for anything if I mainly use Beautify? Could I switch back to URP/SimpleLit?

And is there an easy way to test it out without manually going through every prefab and finding its material and changing it and asjusting all settings?

[quiet crown]

I think there is some setting in beutify that will make it include all variants or something stupid

My collegue mentioned this to me recently

[slow rivet]

And that could be the issue also, since I use both BetterLit and Beautify. Gosh this is hard