How to delete an object from memory?

[peak wing]

Say we have an object instance that we know, but its being prevented from garbage collection due to being contained in a datastructure of an unknown program.

How can we delete that object, effectively causing all references where that object is being used to end up being null?

Is there are way to see all reference holders? Im guessing there is since the Gargabe Collector does.

So in other words a custom Garbage Collector.

[undone dagger]

<@&987246964494204979> please have a look, thanks.

[latent solstice]

when a process is created the OS creates a PCB (process control block), in this is the heap for the process, and these memory addresses which are allocated on the heap by programme A can not be accessed by programme B, that would be a memory access violation (segmentation fault).
so programme B can not have an object of programme A. you would have to explicitly set up shared memory or a pipe, so it would not be a random unkown programme.

[spring wolf]

why do you want to do that at the first place ?

[peak wing]

Im trying to make a software to fix memory leaks

[spring wolf]

in java ?

[peak wing]

Yes

Im sorry I thought this place is only Java

[latent solstice]

java has no memory leaks the jvm handles that

[spring wolf]

it has memory leaks, just not in classical sense

[latent solstice]

ok maybe with that MemorySegment and Arena

[spring wolf]

no

well

any Closeable could leak

but it could also be because you forgot to remove an object from a list

but that doesn't make sense, first of, how do you even notice them ? second of, if you set to null, what if it gets used ? third of, can't you just fix the bug ? @peak wing

[peak wing]

So I know the Object Type that leaks and I could track it, essentially getting the reference holders and clearing them

[spring wolf]

why don't you just fix the bug ?

[split breach]

If you observe leak, remove all static variables.

[peak wing]

I just dont know how, and it could potentially cause unexpected errors if something is null depending on how the datastructure is handled

[peak wing]

Because that would take an eternity and im looking for a more universal way

[spring wolf]

not nessessary, any obersable observer pattern can easily lead to leaks

[spring wolf]

???? why would it take an eternity ?

[latent solstice]

what is the point of the garbage collector then? then we could have manual memory managment in the first place when the jvm can not handle it

[peak wing]

I have already handled like 15 memory leaks in various softwares and it just doesnt stop happening. Its less but it doesnt stop, there keep popping up more

[spring wolf]

the problem is when you have listeners, you need to clear them when you don't use them

[spring wolf]

then fix the code

and no it wouldn't take ages

[peak wing]

It does

im looking for an alternative

[spring wolf]

if you notice a leak, you know how to fix it

[spring wolf]

that's not how it works

[peak wing]

Why?

[spring wolf]

you are basically noticing a bug, and instead of patching it, you are silencing it

not only that, but your way to silence it is like 100 times more complicated than fixing it

[split breach]

Maybe just list the pattern you saw that caused the leak, and do not do them anymore.

[spring wolf]

please answer, why would it takes ages to fix ? if you know where a leak in java is, then it's easy to fix. Why in your case, it would be different ?

[peak wing]

I already tried to fix it manually, but it keeps popping up at different static references.

[spring wolf]

so you are saying you do not know where the leaks are ?

also stop abusing static

[split breach]

Yeah, do not use static things, they will just keep a reference and never be taken by the garbage collector

[peak wing]

Its not my code that im fixing

All the manual fixes are reflection

[carmine dawn]

You've answered your own question. If you can find the item referencing it, you can't just remove the reference because null could be invalid for that data-structure.

You must resolve this at the root cause - there is no solution that can generally clean-up.

Unlike some languages, there are no orphaned objects to free - Java does that for you. And if they're not orphans you can't automate the cleaning up.

[spring wolf]

what ?

[split breach]

Static is really only for unchanged things that you absolutely need for the whole duration of your code.

Not a lot of thing are in this.

[spring wolf]

@peak wing what do you wish to do ? finding automatically the leaks, or fixing them automatically ?

because finding automatically the leaks is impossible

and fixing them automatically is either impossible or pointless depending if by "fixing" you mean correctly fixing the code or just deleting the object

[peak wing]

Well, the GC can see what holds the references, let me see it maybe?

[spring wolf]

sure, then what ?

[peak wing]

I dont need to "fix" it correctly, just avoid outofmemory is enough

[spring wolf]

what

[split breach]

No, that is not visible from java. At least from what I know.

[spring wolf]

please answer

what are you trying to do ?

are you trying to automatically find leaks ?

[peak wing]

Then I want to delete the object as the GC would if it doesnt find any, and maybe handle some know datastructure in the rederences to avoid unhandled NPEs

[split breach]

What? Just calculate the space needed beforehand from your algorithm.

[spring wolf]

but how do you know that you need to delete an object and not anotrher ?

and please answer my question

it is starting to be very annotying

are you trying to automatically find leaks ?

yes or not ?

[carmine dawn]

References of what? Every object. That can be millions to billions of objects. Objects of a specific type? Then just find out who references that type and evaluate the life-cycle of that reference (or the container).

You can't delete the object though, only the reference to it, and if null isn't valid then you have to delete references to the object that referenced it.

[spring wolf]

well you can just reflectively just search from all static fields since they have a problem with that, also visual vm is a thing

[carmine dawn]

Yes, using a heap analyser can help understand what's in memory, then you backtrack and see who creates and references those types.

[split breach]

Yes, but at that point it is just a hack job. No official java api control the garbage collector as far as I know. Gc is pretty much jvm internal.

[spring wolf]

indeed

and all of this seems to be a xy question

since i have absolutely no idea how you could end up with such problem

[peak wing]

I dont understand, I think I did?

[spring wolf]

no you didn't, just answer yes or no

[peak wing]

Not realy, I know the Object type that leaks, I can see where in visualvm too

[split breach]

What is you goal?
What did you try?
Why do you think it did not work?

Right now from what I understood, you want to delete an object and just by magic remove all references to it.

[peak wing]

It did work, but there are seemingly infinite leaks lol

[spring wolf]

but if you delete the object, it might just cause a npe and crash ? why does it leak ? why can't you fix it ? why do you need reflection to fix it ? this is a xy question, please explain your initial context

[peak wing]

Okay

[spring wolf]

there is no way that's true, it's extremely hard to cause a leak in java at the first place

[spring wolf]

also can you show a class with a leak you fixed or noticed please ?

[carmine dawn]

Outside of native resources, there is no such thing as a memory leak (in the classical sense) in Java.

Instead you may have flawed resource management, extending the lifespan of objects beyond their point of usefulness. Unfortunately there is no way to tell a useful reference from one that could have be released earlier.

So if you know the type, but some of that type are validly referenced and some are redundantly referenced, you can only identify the redundancy per use-case. That's not to say that 'patching' might be necessary in very unusual circumstances (as opposed to fixing the root cause), but the solution is still specific to the cause.

[peak wing]

I want to play Minecraft Modpacks, and I dont want to crash with OutOfMemory all the time, so i thought about a solution "one program to catch them all". Instead of fixing hundrets of memory leaks manually, sometimes in obfuscated code, and generally completely different code structures that i constantly have to switch between. And even if I fix all leaks that i can find in the heapdump's GC Roots, there pops up a anotger static reference holding it. Its like vaccuming in the desert.

The primary objects leaking are Worlds, these objects could be tracked so i could look specifically for them automatically in a program

[spring wolf]

there is also the problem that memory leaks in java are so rare, in order to get a out of memory, you have to really search for it

ah

huh

mc

[spring wolf]

So the main problem is how to find leaks, which I asked earlier
because it's impossible to automatically find leaks

Also, I played a ton of modpacks and never had such problem

[peak wing]

Most of the time they have smth like this:

public static final Map<Entity, SomeCustomModDataClass> entityData = new HashMap<>();

public void onEntityAdded(Entity entity) {
ebtityData.add(entity);
}

never being removed

[spring wolf]

oof

well

[peak wing]

And seemingly a lot of modders love todo this

[spring wolf]

problem is the following, how do you notice automatically such code, and how do you know you can delete an entity

the simplest way is to modify the source code

and fix it here

[peak wing]

I check if the Entity is added to the current loaded world

And I use a command to clean it up

[spring wolf]

btw where is this static field ? in an entity class ?

[carmine dawn]

So even patching it, it's a per-use-case problem. When has the map finished with an entry, so it can be removed from the map.

[spring wolf]

yea it's per use case

[peak wing]

In some Mods Handler class

[spring wolf]

you could patch it via ai tho

but well

again

the best and simplest way is to just edit the source

most mods are open source anyway

and even closed source, they shouldn't have heavy obfuscation so you should be able to easily decompile

and then if you want to go fast, you give the code to the ai, give directives, and ask it to fix it

either that or you make a tool that would fix source code

much simpler than to hack into java internals

@peak wing

[peak wing]

I have never seen AI being able to fix code respectively

[peak wing]

Im doubtfull of that

[spring wolf]

well you weren't here last week when chatgpt fixed a bug in my c project in seconds that I was debugging for 4H

[spring wolf]

try and see 🙂

[carmine dawn]

And even without changing source-code, you need to get a reference to the map. Then it's just an exercise to iterate over map.entrySet() deleting entries you don't want to keep (all the while hoping you don't trip over a ConcurrentModification with the owner).

AI is definitely getting more useful for very targeted purposes.

[spring wolf]

ahhhh true ConcurrentModification is also a problem

[peak wing]

But I have done it manually already, what am I supposed to learn from that point

[carmine dawn]

Whether that also means commercially viable is yet to be seen

[spring wolf]

wdym

you didn't hack into java internals manually

you also seem to not understand how hard it is for an algo to behave like your mind when you tried to fix it

you have to give strict directives to a code

it's not because that you solved it several times that code can do it

...unless AI is involved

...or again, just make a tool

that will patch the code

and well

the solution which is the closest to what you ask is what I said earlier
which is to reflectively check all static fields and try to somehow automatically find the problematic ones and somehow patch them
@peak wing

[peak wing]

rwflectively check all static fields?

[spring wolf]

ah and btw, let me remind you that you can also edit the bydecode, it's not like mc gives tools for that

[spring wolf]

yes, make a code, which will look on all static fields to find the fields that could have a leak

[peak wing]

I know using Mixins for this is better but thats even more work

[spring wolf]

it's like 100 times less work than trying to temper with teh GC 👀

[peak wing]

And I barely know these modpacks, i could break something without noticing

[spring wolf]

this is what you are asking tho

deleting objects that weren't supposed to be deleted will break things

this is what you asked for

[split breach]

Wait, is the solution all along not to get more ram?

xD

[spring wolf]

you literally said that it doesn't matter as long as you don't get the out of memory earlier

I tho you were aware of that

@peak wing

[spring wolf]

👀

Me and my 32Gb:

[peak wing]

I have already allocated a bit more than i can/should

And im at 85% system memory usage

[spring wolf]

how much GB ?

[peak wing]

3

[spring wolf]

huh

time to download more ram

[peak wing]

NO

[split breach]

3 at max?

[peak wing]

Yea

[split breach]

What?

[peak wing]

It uses 1200mb after startup

[split breach]

How is your server even working?

[spring wolf]

You shouldn't play modpacks with that little amount of ram

Also

well

[peak wing]

Its not a server, what do you mean?

[spring wolf]

Your out of memory is probably not even caused by a leak @peak wing

[split breach]

Client minecraft?

[peak wing]

If it works with 1200mb it should work after an hour still

[peak wing]

yes

[spring wolf]

it is very probable that it is caused just because you have too little ram to handle mc

[peak wing]

I can see that it is in the heapdump

That heapdump taking over an hour to load

[split breach]

Yes, maybe you just really are out of memory

[spring wolf]

so this map you shared above, how much memory does it cost after the crash ?

[spring wolf]

not quite, if you start to do many things in your world, it could cost more

[split breach]

Mine go up to 12GB

I just don't understand.

Modpack are pretty heavy.

[peak wing]

When i mean out of memory, it does not happen instantly, what first happens is that the GC runs more commonly, causing a lot more cpu and adding good amount of lag. Then secondly the timeframe in which the gc can run is too small, causing freezes. And at last these freezes become so common until it crashes.

[split breach]

And even if you removed static object loaded, you would still need to reload them later while playing.

[peak wing]

I dont see why at all

[peak wing]

What do you mean "reload them later while playing"?

[spring wolf]

yes that's normal, it simply mean you don't have enough ram for the GC to work

Anyway, I need to sleep, it's 4'30AM already 😭

[split breach]

Do you understand that ressources loaded by mods are used by the mod?

[peak wing]

Good night

[peak wing]

That expects that they are used by the mod

Currently there is roughtly 700mb used for worlds rn

So that means the application has just about 1gb for live objects, which means the GC has to run every few seconds, causing high cpu and lag

If it has 1,7gb to work with the lag is fairly better

[peak wing]

Those are 32 worlds btw

[peak wing]

But probably not required? How do you know thats not caused by world memory leaks?

Oh and another thing are texture memory leaks

They are common too

[split breach]

I got a log, and it is stable

[peak wing]

What does a log have todo with that

[split breach]

Until is get bigger non stop, it is not a leak for me, it is just a bad resource management.

Thing get preloaded so that the game mod is faster, but that take memory

[peak wing]

Every world is around 50mb on average only, you wont notice it like that if you allocate 12gb

[peak wing]

What exactly preloaded?

[carmine dawn]

So the solution remains the same. You need to find the source of the leak, and determine how it can be resolved. That might be null a reference or it might be that you need to delete an entry in a map.

In both cases, you're then waiting for the GC to do cleanup (and that takes a little memory too, and might wait until the system is almost out of memory first)

Note that the space on disk doesn't really reflect the space in memory. Firstly, the disk-representation can exclude padding and unused fields and secondly, they're usually compressed.

[peak wing]

Im not talking about disc space, why do you think that?

[carmine dawn]

Just making sure...

[peak wing]

I would send you some screenshots if i could

But im at 90% memory rn because analyzing the heap takes all my resources

[carmine dawn]

It's not an unusual misconception that the size on disk is some guide to memory (and it's not since it also doesn't all get loaded at once)

[peak wing]

I have heard of people thinking that reducing the world file size would help in performance, it shouldnt matter unless you are out of disc space

[carmine dawn]

There's a small cost in in-memory indexes of regions and chunks (unless they've improved that... my MC knowledge is very out of date).

[peak wing]

Only if we talk about chunks in a region file that is loaded I think

Can someone explains why loading my heapdump takes all memory?

[acoustic wraith]

weak reference

[undone dagger]

@peak wing

Your question has been closed due to inactivity.

If it was not resolved yet, feel free to just post a message below
to reopen it, or create a new thread.

Note that usually the reason for nobody calling back is that your
question may have been not well asked and hence no one felt confident
enough answering.

When you reopen the thread, try to use your time to improve the quality
of the question by elaborating, providing details, context, all relevant code
snippets, any errors you are getting, concrete examples and perhaps also some
screenshots. Share your attempt, explain the expected results and compare
them to the current results.

Also try to make the information easily accessible by sharing code
or assignment descriptions directly on Discord, not behind a link or
PDF-file; provide some guidance for long code snippets and ensure
the code is well formatted and has syntax highlighting. Kindly read through
https://stackoverflow.com/help/how-to-ask for more.

With enough info, someone knows the answer for sure 👍

[peak wing]

Now my game lags after i have fixed most of the memory leaks. I feel so stupid. I dont understand what i might have done wrong. But the memory usage is quite better now (useless?)

[carmine dawn]

Overheads from your logic scanning for things to clear references for? Or maybe you've undermined some caching mechanisms that now have to do extra work.

[spring wolf]

Time to do some profiling

[peak wing]

I did and I found no evidence of it being slower

Im feeling like paranoid about it

Its bad bc a game cant be easely profiled since scene changes and complexity

Even worse in modpacks

[spring wolf]

When does it lag ?

[peak wing]

But if i cause an reorcurring init or duplication of some data it should show up lol

[spring wolf]

How did you profile ?

[peak wing]

It does start stuttering without ahowing in the fps value after i do the memory cleanup, seemingly getting worse with every cleanup

[peak wing]

spark profiler

[spring wolf]

Wdym when you do a cleanup

[spring wolf]

Which Mc version?

[peak wing]

Instead of automatically running it, i made a command that also runs System.gc() so i see how much it saved

[peak wing]

1.12.2

[spring wolf]

Remove it and never call it again

[peak wing]

Why?

How do I see how much I saved

Do you think it messes up some kind of timer in the game?

[spring wolf]

Can't you just f3 ?

[spring wolf]

Not related to the game but java

[peak wing]

No because it doesnt do an agressive gc necessary

I would have to wait some time for it to show up

Also i was zsing System.gc all the time without the lag appearing after

So it doesnt realy make sense

[spring wolf]

If you don't use System.gc, does it lag ?

[peak wing]

mh? It should only freeze once during executatiob. Plus this command is a recommendation, java doesnt have todo gc

[peak wing]

Havnt tested, but my previous version did it with less memory leak fixes and it didnt lag after

[spring wolf]

How much % is used in f3 ?

[peak wing]

1450 of 3072MB around by now

[spring wolf]

Does it lag the moment this number goes down ?

[peak wing]

no

No the stutter is more like frametime lag

Like microlag

And general decreased performance

[spring wolf]

If it's since you changed the code, it might be because of it
Can you share the code ? I'll take a look later

[spring wolf]

How long does a tick last ?

[peak wing]

I have around avg fps 30 and min about 15 to 24

Same as before the clearup

[spring wolf]

Tps not fps

[peak wing]

I cant see client tick i think?

Server is affected too but thats not as visible ofc

[spring wolf]

In recent versions you can profile it from in-game
You can see how much processing each entity takes

Not sure about 1.12

But since you use a profiler, you should be able to get such info

[peak wing]

You mean the profiler chart in shift f3?

[spring wolf]

Hmm I don't remember in 1.12, maybe you can see tick duration and the tick graph yes

A tick should never be above 50ms

[peak wing]

Oh you mean the tick and frametime lagometer?

Yea thats not there

[spring wolf]

Then use spark

[peak wing]

But the profiler chart says I have fps-lmitywait takes around 50 to 75%

And the rest is relatively evenly tick and gamerenderer

[spring wolf]

Lmitywait ?

Find the tps please

And show your modified code

[peak wing]

Im worried the code is gonna be a bit context less

Im basically just checking if the Object is present in the current world and if not i remove it

[spring wolf]

Share if possible the whole code

In github

[spring wolf]

@peak wing so ?

[still slate]

Hi, as other people said you must fix all the issues of the mods you are using. I've seen the pattern quite often in minecraft plugins or mods where the developers stores temp objects in lists or maps and never clears them. This is what makes memory usage up and most of the time when there are too much objects the server either crash or has huge lag spikes. For performance issues I would recommend to use Spark (I've seen someone mentioned it). If you need deeper analysis on what happens in the JVM then I can only recommend to use a java profiler like JProfiler.