🔒 Interning of Integers in Python

[full vector]

I have to write a paper for college and therefore analyse the memory consumption of python and mojo lists with integers.
While experimenting I noticed that if I use the same integers Python always only generates one objects, while the others are just references to that object.

I read this is called interning, but I only find articles saying its just for the interval from -5 to 256.
I tested it in Python 3.10.13 and 3.12..7 :
Regardless if I use 1 or 1000000 as a value for my integer the interning behaviour is always done.

An example speaks louder than words:

>>> x = 1001
>>> y = 1001
>>> x is y
True

https://luminousmen.com/post/interning-in-cpython?utm_source=chatgpt.com

Also are there some papers or official python ressources explainig that, so I can quote them in my paper

[jade blade]

@full vector

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[ornate thorn]

!e

a = 1000
b = 1000
print(id(a))
print(id(b))
print(a is b)

[jade blade]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | 140441809539408
002 | 140441809539408
003 | True

[native mantle]

what's your question?

[cold iron]

Also are there some papers or official python ressources explainig that
I'd expect you'll find this in the source

[full vector]

Where do I find documentation for this feature?
Searching the web I only found articles regarding the interning for integer in the range -5 to 256

[tepid quail]

Are you sure you're running this code in the REPL?

If you run it in the REPL, you will get this: ```py

x = 1001
y = 1001
x is y
False

But if you run those lines in the same program, you will get a different result. That's because constants with the same value within one code block (e.g. a function, or within the global scope) are reused. See for example:

!e

def f(x):
    return (x, 420, 420, 69)
print(f"{f.__code__.co_consts=}")

def g(x):
    y = 419
    return (x, 420, y + 1, 69)
print(f"{g.__code__.co_consts=}")

a0, b0, c0, d0 = f(0)
assert b0 is c0

a1, b1, c1, d1 = g(0)
assert b1 is not c1
assert b0 is not c1

[jade blade]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | f.__code__.co_consts=(None, 420, 69)
002 | g.__code__.co_consts=(None, 419, 420, 1, 69)

[full vector]

I copied it from the link. Myself I run it as normal python file with

python test.py
```

[tepid quail]

Right, this example will only work in the REPL because of the above

[full vector]

Ah, I experiment with lists in Python and Mojo. So in my list implementation I just do this:

def test_large_list(size):
    large_list = [100000000 for _ in range(size)]
    # sleep(10)
    return large_list

if __name__ == '__main__':
    size = 100_000_000  
    listi = test_large_list(size)
    print(list[10])

The sleep is so I can use top to measssure the memory

I also noticed that I dont meassrue the same memory consumption each time.
It is fluctuating between 200 - 800 mb

[tepid quail]

There are definitely better ways to measure memory usage. For example by using psutil before and after creating the list: https://stackoverflow.com/a/21632554/10295729

[full vector]

I used top, becasue for mojo there is at least according to their discord no better method available.

[tepid quail]

wild

[full vector]

I will look into it and thanks for your explanation.
If you know where I can find that explanation in the official docs, it would be helpful

[tepid quail]

explanation of what in particular?

This page contains some details on integers: https://docs.python.org/3/c-api/long.html

The current implementation keeps an array of integer objects for all integers between -5 and 256. When you create an int in that range you actually just get back a reference to the existing object.

But why does this matter for your research? Maybe there's a bigger question you're asking?

[full vector]

That the integers are also reused in lists. At leasrt in the way I did it above.

size = 100000000
large_list = [100000000 for _ in range(size)]
print(all(large_list[0] is x for x in large_list))

This creates one int object that gets referenced in the list

[tepid quail]

Yes, it doesn't create a new integer from scratch every time. It's the same as this example: objects for literals (literal numbers, literal strings etc.) are created once when the program starts and are then reused.

[full vector]

Thanks, I was wondering why python just needs a quarter of the memory than mojo, when I add the same integrer in the list but more than double when adding different integers

[tepid quail]

Perhaps also take a look at memray for memory profiling

[full vector]

Will look into it

[jade blade]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.