🔒 Actually clearing memory on demand

[gentle hill]

If I am not out of touch, even if you set a variable to None or use the del keyword, it only orphans it. Is there a way to actually remove it from memory and clear up the space?

[iron sable]

@gentle hill

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[vocal horizon]

no, afaik

-# From what I know, python uses an arena allocator, so for something to actually be freed memory, the whole arena has to be empty and that's quite rare

[ripe socket]

Might've found a lib that handles this. "gc" from the standard python c installation. Maybe use "gc.collect()" after "del {variable}"?

https://docs.python.org/3/library/gc.html

[vocal horizon]

no that's not the same

gc.collect attemps to deallocate objects that for example, is holding a reference to itself

like a list containing itself

[subtle ether]

Do you have a specific use case where this is required?

!e
In CPython, as soon as nobody if referencing an object, it is automatically destroyed:

class Banana:
    def __init__(self, length):
        self.length = length

    def __del__(self):
        print(f"Banana({self.length!r}) is dying")

def some_function():
    banana1 = Banana(420)
    banana2 = Banana(69)
    banana3 = banana2
    print("created the bananas...")
    banana2 = None
    print("set banana2 to None")
    banana3 = None
    print("set banana3 to None")

some_function()  

[charred light]

No, gc.collect does run the collection for everything it can. The docs for it literally start with "run the full collection"

[iron sable]

:white_check_mark: Your 3.14 eval job has completed with return code 0.

001 | created the bananas...
002 | set banana2 to None
003 | Banana(69) is dying
004 | set banana3 to None
005 | Banana(420) is dying

[vocal horizon]

for everything it can.
wdym? Python objects are deallocated whenever their refcounts have dropped to 0, so what I'm saying is running gc.collect won't help if it's refcount isn't 0, even after del 'x'. Where gc.collect can help is for 'unreachable' objects, e.g a list holding itself

[pallid bobcat]

python should handle cyclic references, gc is just for controlling that operation

done automatically specifically for types without an implemented __del__ function

iirc

gc is specifically for cyclic references + other things, ref counting is separate (?)

[gentle hill]

It's for a really sensetive app. Need to get the data out of memory as soon as the work is done

[charred light]

They are not deallocated exactly at that moment. They are only queued to be cleared and gc runs periodically.
If you create a lot of objects and then drop the reference, gc might not be able to keep up (you allocate new objects faster than gc is clearing). That's what gc.collect() is for - force that cleanup to complete fully before you do next command

[gentle hill]

already looked into it doesn't actually clear the data

[charred light]

How are you checking whether it "clear the data"? Because if you still have a reference to that object to check it, then of course it won't clear - you still have a reference!

[subtle ether]

Do you have the code somewhere?

[charred light]

And if you use something like weakref, you gotta remember not keep a strong reference to what weakref is referencing (don't save it as another variable, always check it via weakref)

[gentle hill]

oh does it actually clear it? I only read it and gave up. Didn't look far into it. I could be wrong

[gentle hill]

so if I get rid of all references, gc.collect will wipe the data from memory? My goal here is to make sure the physical mamory doesn't have any data on it. If it just marks it as empty tht won't do

[subtle ether]

If an object literally has no references to it, it will be immediately collected

[gentle hill]

I don't really have anything related to this particular issue written yet. It will just be object I need to get rid of as soon as work is done

[gentle hill]

so if I don't have it linked to anything else,

var = None
gc.collect()

will do the trick?

[subtle ether]

gc.collect() is unnecessary

[gentle hill]

garbage collector doesn't empty unreferenced memory instantly last I checked

[subtle ether]

As you can read in https://docs.python.org/3/library/gc.html:

This module provides an interface to the optional garbage collector. It provides the ability to disable the collector, tune the collection frequency, and set debugging options. It also provides access to unreachable objects that the collector found but cannot free. Since the collector supplements the reference counting already used in Python, you can disable the collector if you are sure your program does not create reference cycles.

Python uses two different mechanisms for removing unreachable objects. One is reference counting and the other is the cyclic garbage collector.
Every object has a counter that is incremented when a new reference to it is created, and decremented when that reference is no longer used. When decrementing the reference counter of an object, if it becomes 0, the object is immediately destroyed.

When you have e.g. two objects that store each other in an attribute (a.foo is b and b.foo is a), they will never reach a refcount of 0, even if you can't actually reach any of them outside of this cycle. This is what the second mechanism is for. The gc module is only for this kind of collection.

(If you are familiar with other GC languages, this is unusual, and also the worst way of doing garbage collection as far as performance goes apparently (having both refcounting and the cyclic collector))

[gentle hill]

yes but it only marks the space as free no? I need it to be overwritten actully. Like write 0s to physical memory.. (Do excuse me if I am mistaken)

[charred light]

Doesn't do it instantly, but does it fast enough for most programs not to have to care about it.
As I said above, it only matters if you're creating objects faster than gc can collect them.

[subtle ether]

See: https://docs.python.org/3/c-api/memory.html#object-allocators
There's an arena allocator which is used for small allocations (<512 bytes). It doesn't say how these arenas themselves are collected. It's theoretically possible for you to get unlucky and have a large number of these arenas hang around. If your objects are 512 bytes in size or less at some point, the best way to check if this is a problem would be to try it and see what happens.
For larger objects, the default allocator (malloc/free) is used.

If you do get issues with fragmentation or something like that, you can disable arenas at build time and potentially try enabling mimalloc instead of the default allocator.

[gentle hill]

I think I will just offload it to C++ and do the rest in python. I don't see any way to reliably claer up physical memory in python

[subtle ether]

If it's important to have as small of a memory footprint as possible, then doing it in C/C++/Rust/Go/etc. could be worth it.

[gentle hill]

it's not about how much memory I take up. Just need to get sensetive data outta there

[subtle ether]

Ah, I see what you mean now. You want to literally overwrite the bytes of memory so that they don't contain some sort of secret anymore.

In that case yes, it would be really difficult to control in Python

You could, theoretically, operate on a bytearray and then clear it out once you're done with it. But it's difficult to image what you'll do with it without creating temporary copies of some of the data when passing stuff to libraries.

[iron sable]

Python help channel closed using Discord native close action

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.