Memory leak

[sleek dawn]

Does someone have an ideas or features how to find memory leak? I have a server, there are items spawning and enemies spawning systems. I found that my servers 4Gb ram gone with 3-4 days. Then i start to use network pool for players objects, because i have a fps drops when new clients connected. My ram starts to gone within 2-3 hours. Then i found a thread here when someone had same issue with memory leak because of butcher.cs mirror file. I changed that and now my memory gone within around 16 hours. So i still have memory leak but i cant find where exactly. Unity profiler (memory section) say nothing. Only that managedheap used and unused are increasing, very informative. I found that there is a HeapExplorer plugin, thats how i found that there is a byte[] increasing, but its impossible to find what code or files are increasing this byte[].

[rose plinth]

I can think of 2 possibilities,

• something is causing NetworkWriterPool to create more,
• you are not destroying old gameeobjects, and they are slowly building up in the scene.

you could have something log the number of gameobjects in the scene ever couple of mins (like 10 mins), and see if they grows

for the pool, you could add something that checks the count, and logs a warning if it goes over a certain number

[subtle ember]

Your main friend is server defines, wrap them around code, to strip it out of headless server builds.

#if !UNITY_SERVER
 Instantiate(specialEffect);
 Play audio
 Apply material and colour to player
 Download www unity web request image, and apply
 etc etc
#endif

Your server may be running code it doesn't need to. Defines may be a quick fix.
It matters less if clients leak after 4 days, but very important server does not.

[rose plinth]

if the leak is a byte[], then unlikely to be those things

[sleek dawn]

ya i did that and its fine, i did a console commands to check count of spawned items and enemies, its not increasing by the time. and pool have a max number of objects, so it cant go higher

i just have no ideas how to find, what increasing server memory. i found in the search that there is a memory leak with navmesh set destination method. maybe its one of leaks, but i think that there is not a one leak, i think there are few memory leaks, i will capture another snapshot by heapexplorer today to find what are increasing

but still, its hard to find out what cause it, because there are only types and references to objects. and object might have many scripts on it

[sleek dawn]

it definitely do, but it should not increase memory by the time

[rose plinth]

try modifying the pool is mirror to check the count in Return(), check if it goes over 10k or something ( i'm not sure how much mirror normally stores),

I think that is most likely thing in mirror to cause a leak, which will at the very least rule a lot of byte[] out

[sleek dawn]

you mean to check objects count in mirror network writer pool?

[rose plinth]

yes,

looks like there is a public property to show the count, so you could use thhat

or check in return to see iif it is over some limit

[sleek dawn]

yeah i found that. ill modify it and add a console command to show it. thanks

[rose plinth]

you might also be able to use these to get a full snapshot
https://docs.unity3d.com/Packages/[email protected]/manual/workflow-memory-leaks.html
https://docs.unity3d.com/ScriptReference/Profiling.Memory.Experimental.MemoryProfiler.TakeSnapshot.html

[sleek dawn]

I saw that unity memory profiler, to be honest it lookslike not so good. ive tested it long time ago. especially tile visualisation is a worst thing i ever seen. it seems that unitechs testing their tools in ideal situations like spherical horse in vacuum. but when i tried to use it in the big project this tile feature works terrible. anyway i will try it and i hope that it was improved and i can find it usefull to find my leaks

[sleek dawn]

im checking now snapshot by heapexplorer and its looklike that problem with enemies spawning. i have a lot of materials in memory which related only to enemies models, but it should not to be, because enemies destroying by networkserver.destroy. there are tons of materials and networkwriterpooled in the memory

i guess there is no need to check Return(). i think its 100% pool leak

[rose plinth]

if you modify materials sometimes create a copy i'm not sure if they need to be manually cleaned up 🤔

[sleek dawn]

hm. might be. but there are much more networkwriterpooled in the memory rather materials

i closed snapshot by incident, but i will capture it again and show how it looks like

[rose plinth]

probably taking 2 snapshots after a few mins apart and see what goes up

[sleek dawn]

i think now that if i will network pool enemies prefabs like i did before with players should fix this leak. but i dont understand why it happening

i have around 1.3gb ram to go before it crash, so i think it possible to compare them)

[rose plinth]

when I do pool stuff I always have a max capacity so it wont keep growing, and destroy extra objects or leave them for GC

[sleek dawn]

the thing is, that its not my pool, its a mirror pool

i did pool for players prefabs with max capacity as it should be, for now i have a 50 pool count at start and 60 max capacity

and i have an items and enemies spawning systems, but they working without pool, just networkserver.spawn and destroy, so it should have no leak i guess with that logic

well, new snapshot is interesting

there is a last memory sections. this is a materials

and this is a byte[] and networkwriterpooled

the rest are items spawned by items spawning system

both leaking. items and enemies. everything what destoyed by networkserver.destroy

but i still dont know what it cause. is it my bad or mirror

[rose plinth]

I think mirror store networkwriters instead of networkidentity, but I dont think they are the pooled version 🤔

[sleek dawn]

@prime gull sorry for ping, but can you please help with it?

[prime gull]

Can you make a memory profiler snapshot? Familiar with that and I can guide you through that better

Having a bunch of pooled writers lying around is completely normal and shouldnt be any concern

I'd also be happy to take a quick look at it myself if you want to send a snapshot over (eventually, I'll be out a bunch over the weekend - so no promises when, but I should get a few spare minutes to look at stuff in the next few days 😁 )

keeping in mind the usual caveats that come with sending over memory snapshots of course: they include all memory data, so if there's any sensitive passwords or user information you store they'll be in there

[sleek dawn]

ok. thanks. Should i send it in pm?

[prime gull]

yeah, unless you want to share it with everyone else 😁

[sleek dawn]

😅

[sleek dawn]

I sent a friend request, but I will also need some time to set up a profiler and collect data

[sleek dawn]

well. i did 2 snapshots. one from just started server on my machine, and one from almost ram gone server on linux. there is a huge difference even in snapshot size. 300mb vs 1.3gb

[sleek dawn]

Im trying to find a problem with snapshots, and i still have a leak. Ive made a network pool for players and enemies. for player 60 max capacity and for enemies 500. Both of them spawning and despawning by the time a lot. And now i see that i have 800 references to players and 3300 references to enemies in the memory, but it should not to be. My pool arrays tells me that pool count is normal, but memory not. Only one thing what unite it all its a networkidentity component. Is it possible that NetworkIdentity might present in the memory without release?

[plucky nebula]

hello,I also had this problem, as the server ran longer the memory increased until the server crashed.did you find a solution?

[sleek dawn]

hi, im trying to debug memory by unity memory profiler and heap explorer to find a leak, also i have an idea that it might be because of List. I will try to use init capacity for lists these days to check this theory.
Do you have often List usage? or Network pool? or often network spawn \ destroy in your project?

[plucky nebula]

I am not using List，just spawns a lot of test clients。I also added the Distance Custom Range component and Spatial Hashing Interest Management component.I don't know if it's because of this

[sleek dawn]

hm, it might be same issue, but for now idk how to find and fix it