Possible memory leak with addressables?

[radiant ice]

Making a thread, so that I won't forget.

[mental drift]

I should've done it, my bad

[deep palm]

I just tested your project. There is little memory allocation, but it seems to only stack up when loading the scene but not keeping stacking up, which makes me wonder, if this is just an editor thing keeping track. Did you ever profile the memory on a built version?

[mental drift]

how do I do it? i tested in the build project and took screenshot in the editor

[mental drift]

have you tested using memory profiler?

[deep palm]

Yes, I have

But I did not test build yet. you can assig nyour build to your editor profiler. did you do that?

[mental drift]

i took memory snapshots of the build version. this is one of the screenshots. first one before sceneLoad, second one after, last one after unload..but see after unloading, nothing much changes

[deep palm]

Gotta go now, maybe I can have a look later this afternoon on the project again

[mental drift]

thank you so much for your help

[radiant ice]

Well, here's my guess from playing around a bit.
The memory increase is likely from the reserved memory and potentially from memory fragmentation as well. When you load the scene, wait a while and unload it, most of the time, it goes back to the previously used memory size.
However, if you do it in quick succession or just many times, the amount of used memory would keep on going up.
On the first load/unload it seems to always go up a bit. The amount it goes up by seems to correlate with the size of the addressable package, so I'd assume that unity caches the bundle somewhere so that it could load it faster next time or something like that.
The other times that it goes up, is unclear though. It could be due to the garbage collector and memory fragmentation, but if tat was the case, it would go down eventually, yet it doesn't seem like it does.
There might be a bug with unity addressable or memory management in general, so the best you can do is wait for unity staff to reply. Alternatively, you could get an enterprise license. With the access to source code and debug symbols, you could investigate the engine thoroughly.

[deep palm]

I can have a look as soon as I got my new license and find some time. Maybe I get some insights, but cant guarantee when

[mental drift]

so there's no way right now achieve what i am trying to, right? and obviously I can't afford the enterprise license😅

[radiant ice]

Achieve what?

[mental drift]

load the scene, then when unloading, the memory would free up, i mean if i have 10 scenes, and every scene loading grows the cache size, then won't it be huge after loading and unloading all those scenes one after another?

[radiant ice]

Did you actually test it out or are you just making assumptions?
There's no guarantee that the unfreed memory increase is correlated linearly with your scene sizes or numbers.
You need to test and see for yourself. I'd also test using Scene manager loading/unloading to see if there's any difference.

[mental drift]

i tested with scenemanager, it works fine, the memory increase only a bit after unloading compared to the initial memory consumption. but i tend to keep the build size low, so i preferred using addressables

i will test again, i tried to solve this with so many things, i forgot what the results were for for each tries

[mental drift]

you were right, i tested again, this time, it seemed like the untracked memory is inreasing with scene size or scene numbers, rather based on the assets i am adding addressables group

thanks you for your help, i will dive a bit more tomorrow...will try to do a big test

[deep palm]

I suggest putting some more in to your scene to see, if it actually fills up faster. Also make a check if the scene is loaded already before destroying or enabling it

public async void LoadAssetReference()
        {
            if (!ValidAsset() || instantiatedReference || IsLoading)
                return;

            IsLoading = true;

            //Check if the addressable actually exists
            var locations = await Addressables.LoadResourceLocationsAsync(assetReference).Task;
            if (locations != null && locations.Count > 0)
            {
                AsyncOperationHandle<GameObject> assetLoader = Addressables.InstantiateAsync(assetReference, rootContent.transform, false);
                await assetLoader.Task;

                if (assetLoader.Status == AsyncOperationStatus.Succeeded)
                {
                    //Debug.Log("Load asset reference: " + assetLoader.Result);

                    instantiatedReference = assetLoader.Result;
                    AssetActive = true;
                    IsLoading = false;
                }
                else
                {
                    if (visibilityState == VisibilityState.FullyHidden)
                        UnloadAssetReference();
                }
            }
            else
                Debug.LogWarning("Error in loading asset: " + assetReference);
        }

public void UnloadAssetReference()
        {
            if (AssetActive)
            {
                //Debug.Log("Unload asset reference: " + assetLoader.Result);

                Addressables.ReleaseInstance(instantiatedReference);
                instantiatedReference = null;
                AssetActive = false;
            }
        }

Something like this, just copy pasted from my current project

[mental drift]

thank you so much, definitely I'll try it

[mental drift]

Well, since yesterday, i was trying to load and unload huge scene, so i got one from the asset store, a demo scene

now, it takes about 1gb of resident memory when loaded, but when unloading it takes about .80gb(out of my expectation), memory profiler shows, it takes most in the reserved section under native section. inside that in the alloc.gfx, it takes the most, what does these allocates in resident memory after unloading the scene? i could get anything clearly from google and forums

[radiant ice]

As has been said multiple time: reserved memory. Gfx is probably GPU memory.
Most engines would not return the used memory to the system, as reserving memory is expensive and the engine assumes it would be reused.

[mental drift]

oh, now i understand...but then exactly which part of the memory is freed up when unloading things?

[radiant ice]

It depends on the platform and implementation. Kinda hard to say for sure without seeing unity source code.

For example an engine I'm working with at work currently, would reserve the max possible memory on the consoles for it's internal allocator and use the memory from it. From the perspective of the system, that's like allocating ~11GB at the very start.

Then there are some allocators that might be reserving bigger blocks of memory each time. For example Mimalloc.

My suggestion is just to not pay too much attention to these numbers unless you have an actual problem of running out of memory.

[mental drift]

i got your point, i think i am overthinking about these, although my game is open world, but i guess these are much of deep subjects

anyway thank you for sharing these knowlegde

[radiant ice]

If you want to have full control of the engine, you should use one that provides access to source code. Or pay for a license.

Otherwise, you just have trust that the engine is doing things correctly.

[mental drift]

maybe, i will ask things later about memory optimization, if you don't mind....but obviously i wont get into these deep subjects

[mental drift]

Hi @radiant ice, when I am instantiating any object, it is loading two mesh renderers..
although in the help window of memory profiler, one says in scene, another says loaded because something is using it...same goes for the mesh filter....Is it by default like that? or any fix is there?

sorry if i have disturbed you

[radiant ice]

One is a prefab. One is an instance in the scene probably.

[mental drift]

so, both have to exist?

[radiant ice]

Yes. Though, you could probably unload the asset one if you are not planning to instantiate it's instances anymore.

[mental drift]

now i understand, thanks

[mental drift]

ahmm, @radiant ice, there is something i am again concerned about...its fine when memory is taking some of its spaces to cache things even after unloading...but every time i load and unload the space seems to widen itself, i mean, lets say initially i have 200mb resident on memory, then i load one asset that results the memory space in 300mb. Now when i unload, although it should be back to 200mb, but its fine that it results in 250mb . however, then when i am again loading and unloading it, at a inconsistent rate, the memory space seems to go up, like 40mb gain per load and unload...all the values are just to simplify, the ratios are similar....and all these are not from object memory, rather after unloading most space(about 80%) is taken by "Native subsystem, reserved and untracked(done being concerned about it)"...what should i in this case? as a open world system, all the assets must load and unload quiet a few time in one gameplay session....memory will be flooded if that goes on...and obviously i am using addressables

please tell me if you are annoyed or disturbed, i don't have much of of a connection with other devs, so i rely heavily on someone when feels reliable, but i must not disturb them

[radiant ice]

There's not much you can do in this case aside from filling a bug report and waiting for an explanation from unity staff.

That being said, you should test with an actual real life scenario to see if it actually causes you to run out of memory in a reasonable amount of time/gameplay.

[mental drift]

the thing is, i dont have my phone right now, thats why i have to wait for it and test in pc instead, and i don't know if i am getting right infos on pc compared to phone, so i am just trying to shape out the memory in best way to avoid issues later when i get my phone after 2-3 days

[radiant ice]

There shouldn't be a huge difference between a phone and a PC in this context. If there's an actual memory leak with addressables, it's a bug regardless of the platform. If it's not a leak, then unity should make it clear what that memory is used for.

[mental drift]

i submitted a request to unity around 4-5 days earlier, no reply yet

[radiant ice]

Yeah, it's gonna take some time.

[radiant ice]

What kind of request is it? Did you actually fill a big report?

[mental drift]

not bug report, it was her:
https://support.unity.com/hc/en-us/requests/new?ticket_form_id=65905

and in discussion

[mental drift]

i will fill a bug report then

[radiant ice]

Make sure to mention a memory leak in the title. Memory leak issues usually have a higher priority. And provide your example setup project that you shared here.

Btw, you are using the latest addressables package version, right?

[mental drift]

yes i am, right now i am using some huge aggregated prefabs from my main project

[mental drift]

alright

[mental drift]

@radiant ice i cant understand the bundle mode in addressables properly, seems confusing to me...do you know about it?

[radiant ice]

Nope. Never heared of it. What do the docs say?

[mental drift]

it doesn't explain anything clearly

or maybe, i can't get it correctly

[radiant ice]

Share the link

[mental drift]

https://docs.unity3d.com/Packages/com.unity.addressables@1.20/manual/PackingGroupsAsBundles.html

[radiant ice]

Seems like it's defining how the files are packed together

[mental drift]

yeah, but can't understand which one is used for what, i mean what are the difference?

[radiant ice]

The difference is how many bundles you'll have and what assets would be included in them. As for the purpose you can read bellow on the same page Dangers of too many bundles and Dangers of too few bundles

[mental drift]

alright, i'll look into it again

[mental drift]

@radiant ice i need your help again, as you know my game as a open world, has to load and unload alot of things, now to test addressables, i put all the things of every chunks in one gameobject and made it a prefab. but as usual, those huge gameobjects when instantiating creates huge cpu sudden drops(around 30-100ms for 2-3 frames when instantiating), how do i create a good system for instantiating after it loads via addressables?

every chunks consists around 2.5k to 3k objects(including trees, props etc)

first i tried to run coroutine and instantiate one per frame, but that is taking much time for these much asset, around 30sec, i enable the chunks about 5-6 sec before player is able to see it..

[radiant ice]

It seems like 2023(unity 6) has an async instantiation method. Maybe try it out if you can afford to switch to the newer version. If not, you'll need to rethink the way you instantiate your chunks. Instead all the objects being parented under one object, spread them out and instantiate smaller parts in a coroutine or something.

Another thing to take note of is that objects awake/start might be what's making it heavy. Try disabling your chunk object by default and see if instantiating it is any different.

To be honest, with a huge open world, you're probably better off with ECS and it's scene streaming option. You're gonna face many problems with the GameObject approach that are gonna limit what you can do.

[mental drift]

thank you for the deep insight, i'll look into the methods you mentioned

[mental drift]

right now, i have no option to break up the scene into multiple scenes, is there any way to use prefabs with ECS along with addressables?

[radiant ice]

Not sure. I don't have much experience with either actually, but if you go with ECS, you'll basically need to rework your project entirely.

[mental drift]

i wont do it either, i guess

I think instantiation with splitted framework is better

[radiant ice]

Did you try what I suggested with instantiating disabled object?

[mental drift]

i tried, still it has high ms on some frames

[mental drift]

in profiler, it says preload, instantiate(), create, addComponents...such calls

[radiant ice]

Yeah, that's too be expected

I guess you're best bet is to update to unity 6 and try out the async instantiation

The LTS should be out in a weak, so it's a decent solution imho

[mental drift]

alright, i will try it now...

[mental drift]

well, does this suppose to reduce cpu spike? or anything else?

[radiant ice]

Yes. It should run the instantiation on a background thread.

Though, Awake/Start and other init code would still run on the main thread, so you might need to address that as well.

[mental drift]

right now i am only judging by looking at cpu profiler, i tried to instantiate one of my huge chunks with a key pressed,
but both normal instantiate and async resulted the same in profiler graph, 4 frames of spike, with about same ms......
maybe i am doing anything wrong, i will just send 2-3 screenshots...i am on unity 2022.3(LTS)

[radiant ice]

Is there async instantiate on 2022??🤔

Share the code as well.

[mental drift]

please give me a bit, coming right after breakfast

[mental drift]

it has come to the LTS

[radiant ice]

I see. Interesting

[mental drift]

    public AssetReference asset;
    AsyncOperationHandle<GameObject> handle;
    AsyncOperationHandle<GameObject> handle_01;

    void Update()
    {
        /*if (Input.GetKeyDown(KeyCode.Q))
        {
            operations = Addressables.LoadAssetsAsync<GameObject>(assets, addressables =>
            {
                Instantiate<GameObject>(addressables);
            }, Addressables.MergeMode.Union);
        }*/

        if (Input.GetKeyDown(KeyCode.Z))
        {
            handle = asset.LoadAssetAsync<GameObject>();
            handle.Completed += AssistantManager_Completed;
        }
        
        if (Input.GetKeyDown(KeyCode.X))
        {
            handle_01 = asset.LoadAssetAsync<GameObject>();
            handle_01.Completed += AssistantManager_CompletedAsync;
        }
    }

    private void AssistantManager_Completed(AsyncOperationHandle<GameObject> handle)
    {
        Instantiate(handle.Result);
        Debug.Log("Sync");
    }
    
    private void AssistantManager_CompletedAsync(AsyncOperationHandle<GameObject> handle)
    {
        InstantiateAsync(handle.Result);
        Debug.Log("Async");
    }

[mental drift]

here is the code

[mental drift]

here are the screenshots

[radiant ice]

It seems like the spike is originating in your code, you should expand the hierarchy more.

[mental drift]

here it is

the "assistant Manager" is the script that contains the code i sent

i don't know what is going on here actually....

i will give a build

and profile it

[radiant ice]

Does the addressable include code that is not contained in the main build?🤔

[mental drift]

from the build

no, my code only contains what i sent you

this ms takes only when instantiating, for 4-5 frames

then comes back to normal

things going over my head, so there's no way but splitting these huge chunks? that will be too much of a work...

[radiant ice]

Expand the get_ChainOperation

[mental drift]

even a single sighting of this kind of hierarchy scares me to the core

[radiant ice]

Expand it more

[mental drift]

these are now divided into multiple segments, i cant expand all in one screenshot

[radiant ice]

Only the top one is fine. Expand a bit more.

[mental drift]

for your convenience, take a look if you find time, i am giving the screenshots though

[mental drift]

it seems, the more i expand, similar operations are there

i mean same labels over and over again

[radiant ice]

Is the script that you shared included in the loaded objects??

Actually, nvm. That's probably unrelated.

[mental drift]

no it is not

[mental drift]

i just checked again to ensure

and i am invoking the instantiation process on key pressed

loads one big chunk, with about 1k of objects in it

[radiant ice]

I mean, this looks like it gets the dependencies recursively. I'm not sure if it's normal or not. Logically thinking, this should be done on a separate thread.🤔

[mental drift]

😫 too much overwhelming

can't even understand what's happening, but it doesn't create screen freeze in mobile

[mental drift]

but drops fps

[radiant ice]

Are you not profiling a build?

[mental drift]

last 4 screenshots were from built

this huge chain stays for 1 frame, than it reduces, and get normal after 3-4 frames

[radiant ice]

How are you creating the addressables?

[mental drift]

i couldn't get it

what do you mean?

[mental drift]

i guess i am about to find out smthg important

if i am not wrong, although i will test some more tomorrow, every instantiation takes that same amount of ms for same amount of frame(approximately)

i test with an blank project and set only one cube addressable....when i instantiate it in build, profiler shows as huuge as 90ms in 1 frame and gradual reducing for next 3-4 frames before being normal

maybe this ms problem doesn't affect too much then...i'll look ito it for more a bit, do you have time to check it?

@radiant ice

[radiant ice]

I mean, how you have them setup. But nevermind that at the moment.

[radiant ice]

Sure, share the project and I can have a look when I have time.

[mental drift]

https://we.tl/t-3mSIvC4QkQ

here is a super simple project, almost empty, just one shader, one material, one cube model(exported from blender) and one script......shader and the mesh is set as addressables....press "Q" to instantiate and to release the handle.....Please make a development build and check the profiler when instantiating the object.....I will be very happy if I see a 60ms+ spike for one frame and gradual decrease for more 3-4 frames😅

[mental drift]

I found some rooms for improvement, i will try those out and let you know again

[mental drift]

did you test it?

[radiant ice]

Sorry, not yet. Was a bit busy yesterday.

[mental drift]

there's nothing to be sorry about it, just take a look whenever you get some spare time, that will be enough

[radiant ice]

Finally had some time to had a look.
I get a 60-70ms spike the very first time I load the package. It seems like this is caused by unity JIT compiling the relevant code. After the first time it doesn't take any CPU time on the main thread at all, no matter how many times I load/unload it.

In IL2CPP build the JIT spike is gone as expected. There's still a GC spike, which is to be expected though

There is no spike if the target frame rate is 60, since there's plenty of frame time for the GC to do it's thing.

Without target framerate, there's a 15 ms spike, which seems to be caused by the GPU. I'd assume that there's shader compilation going on.

Well, I guess it's too much to expect any better performance than that.

This is also only on the first load. The compiled shader (or maybe pipeline?) is probably cached, so it doesn't cause any spikes on the consecutive loads.

At this point it's a different issue, unrelated to addressables though.
You could probably fix it by preloading the shader cache during the game start or something like that.