#internals-and-peps

1 messages Β· Page 21 of 1

feral island
#

I think this is what the upcoming PEP 734 is about?

#

Sam has explicitly marked a few issues as up for grabs

feral island
cyan raven
#

I'm just looking for a more detailed answer, it sounds interesting.

#

For example this as well: https://github.com/python/cpython/issues/111968, or this https://github.com/python/cpython/issues/112050

GitHub
Issues Β· python/cpython

The Python programming language. Contribute to python/cpython development by creating an account on GitHub.

GitHub
Make collections.deque thread-safe in `--disable-gil` builds Β· Issu...

Feature or enhancement The collections.deque object has mutable internal state that would not be thread-safe without the GIL. We should use the critical section API to make it thread-safe. I think ...

jade raven
#

o.0 i thought deque was already thread safe

flat gazelle
#

it is thread safe with the GIL

cyan raven
#

I still don't know if I can work on them.

raven ridge
#

It's super common for libraries that use the C API to implement their thread safety in terms of the guarantees the GIL gives. As CPython exists today, if your C API function doesn't release the GIL or decrement any ref counts to 0 or call any functions written in Python (or call another function that does one of those things), it can't be preempted. That makes all sorts of multi threaded code much easier to write

raven ridge
quiet crane
#

Is there any pep for a "weak" lru_cache that can be used naively to memoize methods without keeping each instance of the class alive?

static hinge
#

You can set it in init

# 
def __init__(self):
  self.get_foo = lru_cache()(self.get_foo)
#

I don't think there's a pep, but it's definitely something I've wanted before. I usually just reimplement it so it supports methods

quiet crane
#

Oh, so each instance gets memoized version of the method at instance time? clever

#

Clever code should be avoided though πŸ˜›

merry bramble
quiet crane
#

Nice, we have some office-made decorators for cached_property and cached_method, but it would be nice to not need such things.
I read a nice blog post or website/article about this issue a year or two ago, but I can't seem to find it. I should save those good reads somewhere ...

dusk comet
#

#bot-commands message
does this look like a bug?
error range is 2 times longer than needed, i guess that's because these characters need 2 bytes in unicode to represent them

quick snow
radiant garden
#

probably caused by a different unicode library version or something similar, that doesn't properly count non-ascii char width

dusk comet
# 
>>> b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
  File "<stdin>", line 1
    b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
    ^^^^^^^^^
SyntaxError: bytes can only contain ASCII literal characters
``` tested on 3.12 + windows, there is no bug
#

weird

feral island
# 
  File "<string>", line 1
    b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
             ^
SyntaxError: bytes can only contain ASCII literal characters
% python3.12 -c "b'ΠΆΠΆΠΆΠΆΠΆΠΆ'"
  File "<string>", line 1
    b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
    ^^^^^^^^^
SyntaxError: bytes can only contain ASCII literal characters
dusk comet
# 
> py -2 -c "b'ΠΆΠΆΠΆ'"

> py -3.7 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
SyntaxError: bytes can only contain ASCII literal characters.

> py -3.8 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
SyntaxError: bytes can only contain ASCII literal characters.

> py -3.9 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
    b'ΠΆΠΆΠΆ'
             ^
SyntaxError: bytes can only contain ASCII literal characters.

> py -3.10 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
    b'ΠΆΠΆΠΆ'
          ^
SyntaxError: bytes can only contain ASCII literal characters

> py -3.11 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
    b'ΠΆΠΆΠΆ'
    ^^^^^^
SyntaxError: bytes can only contain ASCII literal characters

> py -3.12 -c "b'ΠΆΠΆΠΆ'"
  File "<string>", line 1
    b'ΠΆΠΆΠΆ'
    ^^^^^^
SyntaxError: bytes can only contain ASCII literal characters
#

<=3.8 dont have error locations (at least for stdin code)
3.9 is bugged

=3.10 are fine

#

at the same time snekbox running 3.12 has this bug too

#

!e b'ΠΆΠΆΠΆ'

fallen slateBOT
#

@dusk comet :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     b'ΠΆΠΆΠΆ'
003 |     ^^^^^^^^^
004 | SyntaxError: bytes can only contain ASCII literal characters
grave jolt
#

!e

b'This string has 4 instances of ΠΆ: ΠΆΠΆΠΆ'
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     b'This string has 4 instances of ΠΆ: ΠΆΠΆΠΆ'
003 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
004 | SyntaxError: bytes can only contain ASCII literal characters
grave jolt
#

!e

b'This string has 3 instances of ΠΆ: ΠΆΠΆ'
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     b'This string has 3 instances of ΠΆ: ΠΆΠΆ'
003 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
004 | SyntaxError: bytes can only contain ASCII literal characters
grave jolt
#

!e

import os
print(os.environ)
fallen slateBOT
#

@grave jolt :white_check_mark: Your 3.12 eval job has completed with return code 0.

environ({'LANG': 'en_US.UTF-8', 'OMP_NUM_THREADS': '5', 'OPENBLAS_NUM_THREADS': '5', 'MKL_NUM_THREADS': '5', 'VECLIB_MAXIMUM_THREADS': '5', 'NUMEXPR_NUM_THREADS': '5', 'PYTHONDONTWRITEBYTECODE': 'true', 'PYTHONIOENCODING': 'utf-8:strict', 'PYTHONUNBUFFERED': 'true', 'PYTHONUSERBASE': '/snekbox/user_base', 'HOME': 'home', 'LC_CTYPE': 'C.UTF-8'})
grave jolt
dusk comet
#

then why is snekbox broken?
it saves input to a file

grave jolt
#

Running it in docker doesn't reproduce the bug, so nothing in my system should impact it πŸ€”

#

!source e

fallen slateBOT
#
Command: eval

Run Python code and get the results.

Source Code

Go to GitHub

bot/exts/utils/snekbox/_cog.py:536
grave jolt
#

snekbox actually runs is in nsjail hmmm

dusk comet
#

!e ```py
import sys
print(sys.getdefaultencoding())

fallen slateBOT
#

@dusk comet :white_check_mark: Your 3.12 eval job has completed with return code 0.

utf-8
dusk comet
sour thistle
#

when reading from files in windows it uses a different one by default
edit; nvm guess that this is not what you're talking about

grave jolt
#

I'm on linux, and I get this on 3.9

>>> b'foo barΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆ'
  File "<stdin>", line 1
    b'foo barΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆ'
                          ^
SyntaxError: bytes can only contain ASCII literal characters.
#

the caret is always 1 char after the literal

dusk comet
# 
>py -3.9
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
  File "<stdin>", line 1
    b'ΠΆΠΆΠΆΠΆΠΆΠΆ'
             ^
SyntaxError: bytes can only contain ASCII literal characters.
``` πŸ€”
grave jolt
#

snekbox runs code in nsjail, maybe that's different

grave jolt
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 1, in <module>
003 |     exec('b"ΠΆΠΆΠΆ"')
004 |   File "<string>", line 1
005 |     b"ΠΆΠΆΠΆ"
006 |     ^^^^^^
007 | SyntaxError: bytes can only contain ASCII literal characters
grave jolt
#

? ? ? ? ?

#

if you run it in exec the bug is not bugging

#

wtf

#

OH

grave jolt
#

So there's probably a bug when printing the traceback

#

ΠΆ aside this code confused me for a moment

        self._line = line
        if lookup_line:
            self.line
grave jolt
#

oh wait, no no no

#

@dusk comet the SyntaxError already has the wrong end_offset

$ cat foo.py
b"ΠΆΠΆΠΆΠΆΠΆΠΆΠΆΠΆ"
$ cat main.py
try:
    import foo
except SyntaxError as exc:
    print(exc.offset, exc.end_offset)
$ python main.py
1 20
#

so the bug is probably somewhere in the C code

dusk comet
#

i guess offsets are calculated using byte position, not unicode-aware position

still sentinel
#

AAAAH

humble copper
#

guys i want to revers the stack how ?

image.png
unkempt rock
#

@round robin

#

@fossil pumice

wet ridge
#

O

fallen slateBOT
#

:incoming_envelope: :ok_hand: applied timeout to @unkempt rock until <t:1700715334:f> (10 minutes) (reason: mentions spam - sent 6 mentions).

The <@&831776746206265384> have been alerted for review.

#

:incoming_envelope: :ok_hand: applied timeout to @wet ridge until <t:1700715409:f> (10 minutes) (reason: chars spam - sent 5982 characters).

The <@&831776746206265384> have been alerted for review.

dusk comet
#

what is happening?
spam raid?

#

<@&831776746206265384> suspiciously high amount of off-topic messages

torpid bridge
#

!silence

fallen slateBOT
#

βœ… silenced current channel for 10 minute(s).

torpid bridge
#

!cban 962960473010102282 harassment

fallen slateBOT
#

failmail :ok_hand: applied ban to @round robin permanently.

grave jolt
#

!e
@dusk comet here's some fun confusion with your bug

ΠΎΠΎΠ°ΠΎΠ°Π°_ΠΌΠΌΠΌ = 42 + return + deploy
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     ΠΎΠΎΠ°ΠΎΠ°Π°_ΠΌΠΌΠΌ = 42 + return + deploy
003 |                                ^^^^^^
004 | SyntaxError: invalid syntax
grave jolt
#

if I got that error I might be scratching my head

neat delta
#

this triggers the fuzzy under return for other variable names. vsc warns of funny characters for the os and as, but are you hiding an RTL character somewhere in there?

grave jolt
#

it's a bug where a SyntaxError moves the caret one char too far for cyrillic letters

#

!e

the_bug = b"fooΠΆ"
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     the_bug = b"fooΠΆ"
003 |               ^^^^^^^^
004 | SyntaxError: bytes can only contain ASCII literal characters
grave jolt
#

!e

the_bug = b"fooΠΆΠΆΠΆ"
fallen slateBOT
#

@grave jolt :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     the_bug = b"fooΠΆΠΆΠΆ"
003 |               ^^^^^^^^^^^^
004 | SyntaxError: bytes can only contain ASCII literal characters
neat delta
#

well thats weird

dusk comet
#

it is not a SyntaxError's fault
col_offsets are assigned incorrectly during parsing, which causes Pablo's error messages to use wrong values

>>> import ast
>>> m = ast.parse('"ΠΆΠΆΠΆ"')
>>> e = m.body[0]
>>> e.col_offset, e.end_col_offset
(0, 8)

>>> m = ast.parse('"zzz"')
>>> e = m.body[0]
>>> e.col_offset, e.end_col_offset
(0, 5)
#

should we ping Pablo here? I think he should have solid understanding of what is happening, because he implemented this feature initially in 3.11

feral island
raven ridge
#

Pablo's not super active here, so you'd be more likely to catch his notice with a bug report than a ping here

boreal umbra
raven ridge
#

I pasted this example to him and said it was brought up here. We'll see if he takes the bait πŸ™‚

ember lintel
#

Hello

#

@raven ridge sry for ping, could you link me to the message?

raven ridge
ember lintel
#

I ll give an extra research cus im somehow lost

#

But thanks

raven ridge
#

if you have some questions, feel free to ask - though it'd be better in #python-discussion than here

ember lintel
#

Thanks

robust hamlet
#

Hello guys is there an app where your code is always on the cloud? Something like git but its is always saved up there so I can just start coding on my pc and the I can just jump to my laptop and instantly contiue working?

faint river
fallen slateBOT
onyx shadow
#

or gitlab or gitea or whichever you prefer

#

that is what those do

robust hamlet
winter cloak
#

ras

stoic horizon
cyan raven
feral island
#

should be in some .h file in Include/

cyan raven
fallen slateBOT
#

Include/object.h line 277

static inline Py_ssize_t Py_REFCNT(PyObject *ob) {```
cyan raven
fallen slateBOT
#

Include/object.h line 783

#  define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op))```
cyan raven
#

how would that increment the reference?

dusk comet
fallen slateBOT
#

Include/object.h line 734

static inline Py_ALWAYS_INLINE void Py_INCREF(PyObject *op)```
raven ridge
dusk comet
#

there is a weird cpp (not c++, but C-PP) feature: macros are not recursive, so if it expanded once in this place, it wont do it again there

#

so Py_INCREF expands once, but refuses to expand twice, so then compiler have to go and look for this name

cyan raven
#

or at least I can't see it.

supple geode
#

hello

dusk comet
fallen slateBOT
#

Include/object.h line 774

op->ob_refcnt++;```
supple geode
#

aq

cyan raven
dusk comet
#

<@&831776746206265384> he is sending random discord invite links in DMs

image.png
grave jolt
cyan raven
cyan raven
dusk comet
# cyan raven or at least I can't see it.

this particular function is very messy because of a lot of preprocessor things
i believe there are several "preprocessor"-code paths:

  • it calls something on lines 742/744 related to limited api
  • does nogil magic on lines 756/759
  • does something weird on line 768
  • does simple increment on line 774
cyan raven
grave jolt
cyan raven
normal spear
#

how do i unmute?

#

i'm suppressed

#

how do i remove that

grave jolt
normal spear
#

thx

dusk comet
#

!pep 554

fallen slateBOT
dusk comet
#

!pep 734

fallen slateBOT
dusk comet
#

i dont really understand the advantages of using subinterpreters in provided examples in pep 734

#

i dont see any advantages over just using threads

#

(assuming nogil is a thing)

naive saddle
#

the goal is that subinterpreters can run at the same thing independently of each other

#

subinterpreters-based parallelism would allow for better "IPC" between interpreters over multiprocessing

dusk comet
hasty turtle
dusk comet
#

yeah, kinda
GIL ensures that at any given moment only one thread can execute python code

hasty turtle
dusk comet
#

once per bytecode instruction
threads do roughly this:

  1. acquire GIL
  2. execute one bytecode instruction (or more - i dont remember)
  3. drop GIL
  4. repeat
hasty turtle
dusk comet
#

im not sure that it really drops/reacquires GIL after every instruction
maybe it does it once in 100 instructions or something like that

hasty turtle
#

but may i assume each line in dis is atomic

#

?

hexed mist
#

Whatchu coding

dusk comet
#

if there is only main thread, GIL is constantly reacquired (because nobody else can hold it)
if there are several of them - it probably will be reacquired by the same thread (because of reasons i dont quite understand), so you dont have "smooth" cpu utilization that is split equallt across all threads

dusk comet
hasty turtle
#

what is that 1% ?

dusk comet
#

CALL can call to C function that drops GIL, then acquires it again and calls other python function that breaks your invariants, and then it returns and you finish CALL instruction with different data you started with

#

or something crazy like that

dusk comet
hasty turtle
# dusk comet if there is GIL - in 99% cases yes

so, now we will have subinterpreter soon, may that change many assumption? for example, could i still thinks list is thread safe? before that, i can assume that because every read/write to list only need one byte code, which is atomic.

#

if 2 subinterpreters append some thing to the same list at the same time.....

feral island
#

but that's obviously an implementation detail that might change

raven ridge
#

!e import dis; dis.dis("a+1")

fallen slateBOT
#

@raven ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 |   0           0 RESUME                   0
002 | 
003 |   1           2 LOAD_NAME                0 (a)
004 |               4 LOAD_CONST               0 (1)
005 |               6 BINARY_OP                0 (+)
006 |              10 RETURN_VALUE
raven ridge
#

That BINARY_OP for example might call a __add__ method, with its own bytecode, and the GIL might be dropped between two of the __add__ method's bytecode instructions

raven ridge
quick snow
#

I still don't really get the point of subinterpreters. When it's usable from Python isn't it basically the same as multiprocessing, from a user-perspective? Perhaps with slightly less overhead? In which scenarios would I choose subinterpreters over multiprocessing?

merry bramble
#

I believe the dream of subinterpreters is that you can have the strong isolation of multiprocessing, meaning you have fewer hard-to-debug race conditions than with threading because of less shared state. But at the same time, the overhead of subinterpreters should be much lower than with multiprocessing, meaning (1) it's much cheaper to spawn new interpreters than new processes, and (2) you'd be able to run many subinterpreters at the same time even if you only had a few cores.

#

In theory it should be the best of all worlds for certain tasks

#

I'm not sure how close we are to that dream. I know we're close to being able to easily spawn subinterpreters from pure Python code, but I think the overhead of subinterpreters might still be a bit higher than it theoretically could be? I'm not an expert, though

peak spoke
#

I think the main wide appeal was single process paralellism, but that's not going to be that useful without the gil so you only get the separation from them as a benefit

hasty turtle
#

there are too many types of queueπŸ˜”

hasty turtle
dusk comet
feral island
urban sandal
#

There's a lot of cool potential for subinterpreters long term. A lot of the current view of it without looking at the plans for the future and other PEPs it will interact with could Easily make it seem like "just another multiprocessing", but once some things stabilize around this and the removal of the gil, this has a lot of potential with anything that would currently fork, and quite a few other things currently using process pools.

raven ridge
#

There's plans to make it an error to fork after spawning threads, which will make multiprocessing much harder to use on POSIX platforms than it is today. Also, multiprocessing is prone to breaking in hard to detect and recover from ways (one process from a process pool gets killed, let's say, or one fails to start, or the parent process dies while children are still running, etc). Also, multiprocessing is one of the few ways to truly leak resources with Python. You can easily leak memory that stays leaked even after every involved process is dead, and the only way to recover it is to reboot your computer. Even if the only thing we get from subinterpreters is something that's exactly like multiprocessing except with only one process, that actually solves a lot of problems

dusk comet
#

Also, multiprocessing is one of the few ways to truly leak resources with Python. You can easily leak memory that stays leaked even after every involved process is dead, and the only way to recover it is to reboot your computer.
can you explain more?
is it related to shared memory?

raven ridge
hasty turtle
#

and OS allows that?

raven ridge
#

Yes, that's how shared memory works

#

Python tries to clean up after itself, but that process can fail, and if it does you won't get some of those resources back until you reboot

dusk comet
#

so shared memory is kinda persistent between runs (if there is no cleanup/reboot)
you can create shared memory, die and then use the same memory in next process

hasty turtle
#

may i know the difference points and the relation between the new queue and the existing ones?

#

as said in that pep, queue.Queue/asyncio.Queue/multiprocessing.Queue

raven ridge
hasty turtle
#

could old queues be used here?

raven ridge
#

multiprocessing.Queue could, but then you'd lose most of the biggest advantages of multiple subinterpreters, and go back to a state where it's possible to leak system wide resources and need to pickle everything and pay a high overhead for passing objects around

hasty turtle
#

if i created a interpreters.Queue and put some thing in it, and get it back in the same interpreter, they only keeps the value instead of the object, too?

raven ridge
#

Yes

hasty turtle
#

in == out βœ…
in is out (unsure)

raven ridge
#

Yes

hasty turtle
#

they need to pickle to make it shareablle.

raven ridge
#

No, it can share underlying data structures

hasty turtle
#

tuple.

raven ridge
#

No, tuple needs to be recreated, but str and int might not

hasty turtle
#

could it share a tuple and one of the tuple s element is the tuple itself?

i created one last time. and any attept to hash it make the python crash....

#

!e

def f():
    import marshal
    tuplis=[]
    def ltot(inpl):
        if id(inpl) in tuplis:
            return b"r"+tuplis.index(id(inpl)).to_bytes(4,"little")
        else:
            tuplis.append(id(inpl))
            ret=b"\xa9"+len(inpl).to_bytes(1,"little")
            for i in inpl:
                ret+=ltot(i)
            return ret
    def conv(inpl):
        tuplis.clear()
        return marshal.loads(ltot(inpl))
    return conv
conv = f()# this function can create tuples with the same structure as the input nested lists.

a=[];a.append(a);
x=conv(a)
print(x)
fallen slateBOT
#

@hasty turtle :white_check_mark: Your 3.12 eval job has completed with return code 0.

((...),)
dusk comet
#

or internals

raven ridge
hasty turtle
#

!e

def f():
    import marshal
    tuplis=[]
    def ltot(inpl):
        if id(inpl) in tuplis:
            return b"r"+tuplis.index(id(inpl)).to_bytes(4,"little")
        else:
            tuplis.append(id(inpl))
            ret=b"\xa9"+len(inpl).to_bytes(1,"little")
            for i in inpl:
                ret+=ltot(i)
            return ret
    def conv(inpl):
        tuplis.clear()
        return marshal.loads(ltot(inpl))
    return conv
conv = f()# this function can create tuples with the same structure as the input nested lists.

a=[];a.append(a);
x=conv(a)
print({x:1})
fallen slateBOT
#

@hasty turtle :warning: Your 3.12 eval job has completed with return code 139 (SIGSEGV).

[No output]
hasty turtle
#

{x:1} makes it crash

#

hash(x) makes it crash too.

raven ridge
#

The new queue is different from all the existing ones in that it works differently (attempts to share underlying data rather than objects) and allows different types to be sent

hasty turtle
#

it is not serialization and deserialization.....

#

πŸ€”

hasty turtle
# dusk comet or internals

i wrote that code last time. it is a function that can makes nested lists becomes nested tuples with the same structure.
for example,

a=[];a.append(a);
x=conv(a)

makes a tuple of it self.

#

isnt that interesting?

hasty turtle
raven ridge
dusk comet
hasty turtle
#

ctypes?

dusk comet
#

that is boring

#

i dont remember how it is done exactly, but it works roughly like this:

  • make generator
  • pass this generator to tuple(...) constructor
  • firstly tuple object is created, and then it is populated by items from generator
  • now in your geenrator look into gc.get_objects(), find tuple that was just created and yield it
  • now tuple is appended to itself
# 
>>> def f():
...     yield "abc123"
...     yield next(x for x in gc.get_objects()[::-1] if x == ("abc123",))
...
>>> tuple(f())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in f
  File "<stdin>", line 3, in <genexpr>
  File "D:\Programs\Python\3.12\Lib\importlib\metadata\_text.py", line 74, in __eq__
    return self.lower() == other.lower()
                           ^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'lower'
``` i cant reproduce it :(
#

here it is πŸ‘†

hasty turtle
# dusk comet that is boring

mine is more easy and customizable.

if you want 2 tuple be in each other , you can
a=[];b=[]
a.append(b);b.append(a)
conv(a)

dusk comet
#

you are kinda changing type of lists to tuple
so your method can create all structures of tuples

hasty turtle
#

exactly.

pliant tusk
hasty turtle
#

?

#

didnot heard that before

pliant tusk
#

!e ```py
import gc

class magic:
def length_hint(self):
return 2

def __iter__(self):
    for obj in gc.get_objects():
        if isinstance(obj, tuple):
            try:
                0 in obj
            except SystemError:
                yield obj
                yield obj
                break

self_tuple = tuple(magic())
print(self_tuple)```

fallen slateBOT
#

@pliant tusk :white_check_mark: Your 3.12 eval job has completed with return code 0.

((...), (...))
pliant tusk
#

you can make it contain itself twice if you want

hasty turtle
#

i guess that it is because that function tuple() creates an tuple first, then append items into it. (that is not python code so it is doable.)

then you use gc.get_objects() to get the unfinished tuple and add itself into it.....@pliant tusk@dusk comet

raven ridge
#

Yes, exactly

hasty turtle
#

by the way, py byte code allows creating empty set directly. but py code syntax doesnt allow that...

dusk comet
#

{*()}

feral island
# 
  0           RESUME                   0

  1           BUILD_SET                0
              LOAD_CONST               0 (())
              SET_UPDATE               1
              RETURN_VALUE
dusk comet
#

:(

feral island
#

nah that's still multiple bytecodes

#

I think Serhyi suggested special casing that construct in the compiler but we didn't merge it

hasty turtle
#

could we remove them?

6374be02be2f25af.png
feral island
#

yes. I think the pushback was that this construct is too rare to special case

hasty turtle
#

i always avoid using set(). i fell it un-good-looking.

#

and it depends on the name "set"

raven ridge
#

Does {*()} feel good looking?

hasty turtle
#

much better

raven ridge
#

Interesting. I think it's much worse

hasty turtle
feral island
raven ridge
#

Yet another reason not to recommend that as the set constructor πŸ˜‚

feral island
#

I feel like there was a later attempt to special-case that construct in the compiler though

#

that's not mentioned in that thread

hasty turtle
#

another interesting thing of python is that although super is not a key word or soft key word, it is different from others.....
if i write ζ•΄ζ•°=int;字符串=str, i can use ζ•΄ζ•° or 字符串 exactly the same way as int/str.

but super is not. my_super=super does not makes my_super() works.....

#

but if that method has the name super inside, even if

a=1
super
b=2

makes my_super worksπŸ€ͺ

flat gazelle
#

Yeah, adding a __class__ cell which super uses is conditional on the name __class__ or super being used in the body

hasty turtle
#

who come up with this...

flat gazelle
#

Yeah, it is not great, but the practical benefits of not having to write super(Cls, self) just about everywhere are pretty significant.

dusk comet
#

compiler simply checks for super name in methods

hasty turtle
#

how could i write a function that gets the cell of the place it is called too.....

flat gazelle
#

you couldn't, it is special cased in the compiler

#

there is a function deep somewhere which just compares strings against __class__ and super

dusk comet
#

why isnt compiler adding __class__ cell to every method?

feral island
#

yeah in symtable.c

feral island
hasty turtle
flat gazelle
#

then its a breaking change

dusk comet
#

i believe it is a lot smaller than code object itself

#

the weirdest keyword-like thing in my opinion is __debug__
it is at the same time a global variable that you cant assign to, and it can be used as attr name, but you still cant assign to it

# 
>>> __debug__
True
>>> __debug__ = False
  File "<stdin>", line 1
SyntaxError: cannot assign to __debug__
>>>
>>> class X: ...
...
>>> x = X()
>>>
>>> x.__debug__ = False
  File "<stdin>", line 1
SyntaxError: cannot assign to __debug__
>>> x.__dict__['__debug__'] = False
>>> x.__debug__
False
feral island
#

hm that SyntaxError on attribute writes feels like a bug

#

possibly it's to prevent setting the attribute on modules?

hasty turtle
#

still thinks the super thing is 🀦

#

Explicit is better than implicit.

flat gazelle
#

Practicality beats purity

hasty turtle
#

what a hidden feature

flat gazelle
#

and golly gee was being more verbose than java due to needing to restate the typename 4 times per class not practical (slight hyperbole, but still)

hasty turtle
dusk comet
hasty turtle
#

i just...

#

i......

flat gazelle
#

I don't hate just doing the bare minimum to make the one pain point less painful, rather than trying to figure out how to make super a keyword without breaking everything

dusk comet
#

i think adding __class__ cell to every method makes most sense

#

yes, it is a bit memory-unfriendly, but we are doing python, so it is not a huge concern

hasty turtle
#

i noticed that my function s bytecode get a attribute
f.__code__._co_code_adaptive
after some using.
i want to store it into a file, and use it to replace
f.__code__.co_code
the next time i run that py file.

but it failed...it changed to non-adaptive version....

#

πŸ˜”

dusk comet
#

i guess code objects store more statistics information than just _co_code_adaptive

hasty turtle
dusk comet
#

you can try, it would be cool

hasty turtle
#

i m new to this channel. this channel is great.

dusk comet
#

i see it like this:

  • python loads regular .pyc file
  • does stuff that specializes functions
  • at shutdown gather all code objects and store them into .pyc file carefully

at next launch python should pick up specialized bytecodecode versions
(there is a lot of pitfalls, it is not a trivial thing to do)

feral island
hasty turtle
feral island
#

then this is definitely a good way to learn more about how the interpreter works πŸ™‚

#

the "adaptive" part is relatively new (Python 3.11); see

#

!pep 659

fallen slateBOT
feral island
#

for more

hasty turtle
#

emmmmm....πŸ€” before i read it, let me have a guess, maybe, loading a attribute then calling it can be reduce to 1 move, and in that move, if that attribute is a function object then directly call it with a self before args?

#

to skip creating the method object?

dusk comet
dusk comet
#

there was a LOAD_METHOD / PREPARE_CALL / CALL triplet (or something like that), that did almost the same thing as you described

feral island
grave jolt
hasty turtle
rose schooner
#

like just using {*()} and removing those 2 opcodes

raven ridge
feral island
#

(I only ever used 2.7)

raven ridge
#

Indeed, it's there in 2.7:

coerce(x, y)
Return a tuple consisting of the two numeric arguments converted to a common type, using the same rules as used by arithmetic operations.

Huh.

#

raw_input(): use sys.stdin.readline()

Glad that one didn't happen - I don't think that would've been beginner friendly at all

feral island
#

glad we don't have that any more

#

we ended up instead renaming raw_input() to input() which was probably the right call

flat gazelle
#

I do remember the main point for python being easy when I was like... 12-14 was that you didn't need to do string-to-int when reading user input, it just became int automatically. I remember being confused about having to do that when I started learning py3, but eventually I figured the situation out.

raven ridge
#

Yeah, definitely better that we got rid of the implicit eval - that was cute for ints, but terrible for strings, for instance

flat gazelle
#

Indeed

#

that I also ran into back in py2

hasty turtle
#

is that a undefined behavior?πŸ€”

pliant tusk
# hasty turtle is that a undefined behavior?πŸ€”

its defined that __length_hint__ is used to preallocate tuples, and its defined that tuples need to be in the GC before you start pulling from the iterator, (in case the iterator raises an exception, so that things don't get leaked). Grabbing the tuple as it is being built is not quite UB, but its not recommented (the gc module notes that you can get references to partial objects)

raven ridge
raven ridge
#

there wouldn't be anything incorrect about PySequence_Tuple creating a new tuple instance without putting it on the GC list, as long as it guarantees that it gets added to the GC list before a successful return. Until the function returns a reference is known to be on the stack, and the GC only "needs" to know about stuff that might be part of an unreachable cycle. While there's a reference on the stack, it's known that this tuple can't possibly be part of an unreachable cycle

pliant tusk
raven ridge
#

I think this bug could actually be fixed pretty easily

fallen slateBOT
#

Objects/abstract.c line 2056

result = PyTuple_New(n);```
`Objects/abstract.c` line 2099
```c
return result;```
hasty turtle
raven ridge
#

yes, you can use ctypes to break the isolation guarantees of the interpreters. If you do that, you might cause crashes or security vulnerabilities or any number of other issues

hasty turtle
#

so I will not do that.(unless for fun)

feral island
#

in general if you use ctypes any safety guarantees go out of the window

hasty turtle
#

I understand that. I just want to know if it is possible to get that because I can't test it as 3.13 has not came out

feral island
#

(with no stability guarantees)

hasty turtle
#

thank you. can't wait to play with itπŸ˜‹

dusk comet
#

im reading pep734
it says that builtin objects will be shared

does it mean that if i do dict.foo in two different subinterpreters i will get the same object?

what if foo method is fishhook'd, will i get the same function object?
if so, then i can share arbitrary data through foo.__dict__ which doesn't feel safe at all

Screenshot_2023-11-30-11-33-07-775-edit_com.android.chrome.jpg
quick snow
urban sandal
#

The interpreter assumes these are immutable, and unless fishhook goes out of it's way to change how builtins are looked up, this assumption would be broken and you'd have shared mutability that the interpreter thought was not mutable. This could lead to safety issues. I suspect that fishhook could adapt for this and only allow changing it for the current subinterpreter by changing how it operates, but I have gone out of my way to change apis rather than use fishhook, so I can't say I'm super well-versed in how it currently functions.

dusk comet
#

there is also a way to get underlying object dict by abusing mappingproxy's bug with | operator

#

@pliant tusk what do you think?

radiant garden
#

it's a known annoying issue

pliant tusk
onyx topaz
#

I was wondering, how do subinterpreters could potentially mix with nogil? I remember when the nogil PoC just dropped, the discussion was that subinterpreters can be considered as an alternative to nogil (despite the overhead). So, if Python will eventually move to nogil, subinterpreters will be obsolete?

dusk comet
#

yeah, i dont see any point in using subinterpreters if nogil is available

#

the only usecase i can think of is implementing some kind of REPL where UI and thing that executes commands are placed in two subinterpreters, so they dont affect each other too much

raven ridge
# onyx topaz I was wondering, how do subinterpreters could potentially mix with nogil? I reme...

off the top of my head:

  • it will be much easier for extension modules to support multiple interpreters than to support nogil, so it may be many years before important 3rd party libraries properly support nogil without race conditions
  • the plan is that there will be several CPython versions where the GIL is optional but enabled by default, so libraries that want to guarantee parallelism regardless of how the user runs the interpreter may want to use multiple interpreters since they're expected to be available everywhere much sooner
feral island
#

it will be much easier for extension modules to support multiple interpreters than to support nogil
it's not clear to me that this is true. Subinterpreter support requires modules to migrate to the new-style extension initialization API, while nogil requires better locking in places that are currently protected by the GIL. I believe Sam got numpy to work with nogil, while subinterpreter support for numpy sounds far off (https://github.com/numpy/numpy/issues/24755#issuecomment-1729061333)

raven ridge
#

that's interesting. I'd think that moving to the new-style extension initialization API would be much easier than finding and fixing all of the places where someone made assumptions about the GIL guaranteeing atomicity of some operation throughout the codebase

#

the latter seems much more open-ended. It's comparatively really easy to tell when you've successfully switched to the new initialization API and removed all your static variables

feral island
#

true, but to support a per-interpreter GIL, you similarly have to find all places where you were implicitly relying on the GIL to make your operations safe

onyx topaz
#

I mean, at the same time

raven ridge
raven ridge
feral island
#

that's true, there are some cases where a per-interpreter GIL is less work to support. However, if you're relying on the GIL to protect a C global, the per-interpreter GIL will still cause you trouble

raven ridge
#

yeah, that's fair. I haven't seen as much code trying to do that, but I'm sure there's some out there

#

virtually everything that uses borrowed references is a potential memory safety bug in a nogil world, right?

feral island
raven ridge
#

yeah, right now you have to be careful not to execute anything that could release the GIL, but in a nogil world I don't think it's possible to safely use any C API function that returns a borrowed reference to any object that another thread might have a reference to

#

the status quo is that it's difficult to do safely, the nogil state might be that it's literally impossible unless the borrowed reference is borrowed from a newly constructed object (and even that's not necessarily enough, given things like that self-referential tuple hack being discussed yesterday)

#

(though maybe we don't need to worry about that one, and people who abuse gc.get_objects() deserve what they get, heh)

feral island
raven ridge
#

that's interesting. I need to think about that a bit more to figure out how I feel about it. I guess it's useful that the reference to the type object can't be removed out from under you and leave you with a pointer to freed memory, but at the same time I'm struggling to think of a case where it's useful to hold a pointer to the type an object used to be, and that's fundamentally what this is enabling... It's nice that it's adding memory safety for code that holds onto the type object after the object's type has changed, but that code is probably still wrong, since using the old type for anything is probably incorrect, or at least suboptimal

#

but anyway, there's a whole lot more interesting cases for borrowed references than that. it'll be much more common that another thread modifies a dict after you've called PyDict_GetItem on it, or things like that

#

I think the end state has to be that (virtually?) everything returning borrowed references is replaced by something that returns strong references instead. Which means the deprecation of some things in the stable ABI

raven ridge
#

it seems to me that nearly every piece of C API code that iterates over a dict or list or tuple passed in by the caller will be memory unsafe in a nogil world

grave jolt
#

memory safety shmemory shmafety

regal glen
#

Just rewrite the entire CPython in Rust.

#

Should only take a weekend imo

raven ridge
#

cool, let us know when you're done

feral island
#

be sure to block off another weekend to rewrite all existing C extensions in Rust too

regal glen
#

Ah yes of course. This is a good timeline. See you guys back here in two weeks. Promise. πŸ‘€

grave jolt
# 
#![feature(rustc_private)] 

use embed_c::embed_c;

embed_c! {
    int add(int x, int y) {
        return x + y;
    }
}

fn main() {
    let x = unsafe { add(1, 2) };
    println!("{}", x);
}
#

just wrap every CPython module into this and rename the files to .rs

dusk comet
#

gradual typing was introduced to gradually annotate all the code
today we introduce gradual rustification

safe basalt
grave jolt
#

look behind you

onyx topaz
dusk comet
#

why is concurrent module empty
why wasn't concurrent.futures named concurrent_futures or something like that
what is the point in one redundant namespace?

raven ridge
hasty turtle
#

!e

import dis
dis.dis("frozenset({1, 2, 3})")
fallen slateBOT
#

@hasty turtle :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 |   0           0 RESUME                   0
002 | 
003 |   1           2 PUSH_NULL
004 |               4 LOAD_NAME                0 (frozenset)
005 |               6 BUILD_SET                0
006 |               8 LOAD_CONST               0 (frozenset({1, 2, 3}))
007 |              10 SET_UPDATE               1
008 |              12 CALL                     1
009 |              20 RETURN_VALUE
hasty turtle
#

it turns out that it goes like this:
frozenset->set->frozenset

#

now i fully agrees that they should add a frozenset literal....

grave jolt
#

You can do frozenset((1, 2, 3)) to avoid building the set

#

or better, if this really impacts your performance, create the frozenset once in a global constant

rose schooner
raven ridge
#

!cban 1148549789886201937 7d advertising spam across multiple channels

fallen slateBOT
#

:incoming_envelope: :ok_hand: applied ban to @frozen flicker until <t:1702027661:f> (7 days).

stone pivot
quick snow
spark magnet
grave jolt
#

Maybe it could work as a third-party package?

#

Like a module that exports its own patched logging/...

dusk comet
sour thistle
#

built-in classes have lowercase names, does that counts?

dusk comet
#

lets rename list to List

stone pivot
stone pivot
# quick snow In general: sure, why not. A few notes: - logging isn't the only stdlib module t...

logging isn't the only stdlib module that violates modern standards. Why only logging?

I wasnt aware of any others. Consider this PEP submission a more constructive way to approach the rant on Python logging I started a few days ago - 98 comments, and only 1 effective downvote πŸ™‚

Really, Python logging needs a high-level interface like what loguru provides so that 99% of the cases dont involve even seeing the things you see in the basic or advanced cookbook.

Reddit
From the Python community on Reddit

Explore this post and more from the Python community

stone pivot
stone pivot
hasty turtle
#

i m wondering if it is necessary that namedtuple needs eval/exec.

#

why it is used there and is there a way to not use it?

spark magnet
sour thistle
spark magnet
dusk comet
#

it is a lot more convenient and reliable than constructing function manually

hasty turtle
#

will it becomes slower?

dusk comet
#

why?
no matter how you create functions, if they are equivalent, they will have the same perfomance

#

im pretty sure doing eval/exec on string is faster than constructing ast + exec'ing it/constructing function and code object manually

stone pivot
# spark magnet Yes it will, and if you like loguru better than logging, why not just use loguru...
  1. fear of missing out... standard logging is widely used in many important modules such as Flask and SQLAlchemy. So The Next Great Thing had better have similar usage conventions
  2. This is not just about me. It's about saying one thing to people learning Python (obey PEP-8) and then they read standard docs and see blatant violation of PEP-8. This is a very stunning thing to deal with.
  3. We want a more appealing tutorial for Python logging. If it takes 5 lines of code to do what loguru can do in 1, then there is a problem. A different problem than this PEP addresses, but it still is an issue because it is not attractive to see a tutorial with so much boilerplate to do simple things.
urban sandal
# stone pivot 1. fear of missing out... standard logging is widely used in many important modu...

Anyone familiar with PEP8 would not see this as stunning. Prior to any of the actual conventions in PEP8, there's this whole spiel:

In particular: do not break backwards compatibility just to comply with this PEP!

Some other good reasons to ignore a particular guideline:

When applying the guideline would make the code less readable, even for someone who is used to reading code that follows this PEP.
To be consistent with surrounding code that also breaks it (maybe for historic reasons) – although this is also an opportunity to clean up someone else’s mess (in true XP style).
Because the code in question predates the introduction of the guideline and there is no other reason to be modifying that code.
When the code needs to remain compatible with older versions of Python that don’t support the feature recommended by the style guide.

#

So a PR to be a breaking change to fit with PEP8 ironically violates it

feral island
stone pivot
hasty turtle
feral island
#

I think we also need to generate all the self.x = x code, if I recall correctly

dusk comet
feral island
#

you're welcome to look at the code and see if you can get it to work without exec(), it might be a nice performance optimization if possible

hasty turtle
#

and in the end of the video, it also said that it is an implementation detail.

#

and dont rely on it.

cyan raven
#

@feral island (sorry for pinging you) I saw that you reviewed the pep(shorthand keyword arguments thingy). I know that it has not been accepted yet, but should I send a PR so that if it's accepted it can be just merged? It is more organised and can be linked on the Pep page.

fallen slateBOT
#
Rules

6. Do not post unapproved advertising.

feral island
cyan raven
raven ridge
#

And it seems pretty unreasonable to me to drop the documentation for the old, very widely used names. That will make existing legacy code far harder for random people to pick up and maintain. I think you'd need to keep both names in the documentation.

dusk comet
raven ridge
#

ultimately this seems like an all-around bad idea to me, but if you're going to do it, it should be done in a way that doesn't break things

dusk comet
#

oh, i forgot that this is the option
then yes, documenting both versions using same paragraph with double header is a lot better than what i proposed

pliant tusk
#

!e ```py
def ReadAfterFree(size, do):
b = bytearray(size)
class T:
def iter(self):
b.clear()
self.v = do()
yield b''
yield b''

c = b.join(t:=T())
return memoryview(c).cast('P'), t.v

leak, obj = ReadAfterFree(bytearray.basicsize, lambda: bytearray(8))
print('bytearray:', obj)
print('leaked memory of buffer:', leak.tolist())```

fallen slateBOT
#

@pliant tusk :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | bytearray: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00')
002 | leaked memory of buffer: [1, 139934072983744, 8, 9, 139934062675024, 139934062675024, 0]
pliant tusk
#

@feral island ^ you might think this is neat. I'm trying to think of an elegant fix for it, since stringlib is mostly dynamic and I don't think putting checks inside it would be a good idea, do you think it would be sufficent to just increment bytearray->ob_exports before calling stringlib join and then decrementing it after?

feral island
feral island
fallen slateBOT
#

Objects/stringlib/join.h line 25

seq = PySequence_Fast(iterable, "can only join an iterable");```
feral island
#

though I guess there is code later in that function that could also call into Python and therefore clear the bytearray

#

e.g. the PyObject_GetBuffer call on line 71

#

so your solution is safer

raven ridge
#

though the ob_exports option sounds simpler if it's sufficient

feral island
#

yes, the ob_exports option feels a little bad in the sense that we're throwing an error when we could do something broadly sensible instead that doesn't throw an error

#

but the failure case here is contrived enoug that I don't feel too bad about throwing an error

raven ridge
#

well, switching the implementation to essentially do bytes(the_bytearray).join would fix it, but at the cost of pessimizing every bytearray.join

#

though otoh, I'd wager that bytearray.join is really rare in the wild, anyway

#

I'm not sure I've ever seen it...

feral island
#

.join with a non-constant string/bytes is quite rare I think

raven ridge
#

yeah. some_bytes.join(some_bytearray) doesn't seem weird to me at all, but some_bytearray.join(anything) feels weird

#

there might be some special case where it's useful, but I can't easily think of one

pliant tusk
#

I think the best solution would be to adapt stringlib to use the buffer api for mutable strings

#

But that would be a lot more work, and way more to test

pliant tusk
fallen slateBOT
#

@pliant tusk :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 1, in <module>
003 |     b''.join(bytearray(8))
004 | TypeError: sequence item 0: expected a bytes-like object, int found
pliant tusk
#

!e py b''.join(b'ab') also fails

fallen slateBOT
#

@pliant tusk :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 1, in <module>
003 |     b''.join(b'ab')
004 | TypeError: sequence item 0: expected a bytes-like object, int found
grave jolt
#

we need a bytearrayarray

pliant tusk
#

I think you can do it with memoryview shapes?

raven ridge
#

... huh.

#

I bet that used to work in 2.7

pliant tusk
fallen slateBOT
#

@pliant tusk :white_check_mark: Your 3.12 eval job has completed with return code 0.

[[b'\x00', b'\x01'], [b'\x02', b'\x03'], [b'\x04', b'\x05'], [b'\x06', b'\x07']]
grave jolt
faint river
#

hey, I was wondering if this is some C error?

def f():
  try: f()
  finally: raise Exception
f()

object address  : 0000029FD73E9900
object refcount : 3
object type     : 00007FFC91FE9D10
object type name: Exception
object repr     : Exception()
lost sys.stderr
raven ridge
#

well, it's creating an exception chain that's sys.getrecursionlimit() deep. I'd suspect that it's then failing to print that exception, probably by hitting the recursion limit while trying...

faint river
mortal void
#

guys im new to python and i wanted to install scipy to use lambertw() but my editor (IDLE Python) refused to do so. So i switched to Visual Studio Code and now i think im getting scizo

#

i cant install it here either im confusion

white nexus
mortal void
#

ah ty

lunar wing
#

i have a feeling i didnt do this correctly eventhough the outputs are correct can someone help pls

image.png image.png
dusk comet
cyan raven
#

Does Cpython have a "token on fly" lexer? like it doesn't store the tokens in a dynamic array it just returns it back when the parser needs it or actually stores it?
Most compilers/interpreters use the former method.

flat gazelle
#

CPython uses the PEG parser now, so there isn't really a separate lexer at all AFAIK.

cyan raven
feral island
flat gazelle
#

ah, neat

wispy cloud
#

any one who have a good course for python ?

halcyon kernel
#

how do I resolve Cannot find the reference resize in (__init.py__) in cv2

#

in pycharm

spark magnet
little warren
#

Can anyone think of any material concerns or bad interactions running subprocess.run instances within an openMPI instance [within a SGE job] to call CPU heavy, specifically nonconcurrent, shell based / nix shell accessible commands?

if yes, what actually would be the "correct approach" within python, assuming the tool is nontrivial to import or recreate within python, but accessible via the shell.

When I implemented above it worked fine but I ran into an intermittent error and when I mentioned the issue to a more senior dev I got an unexpectedly strongly negative reaction ("subprocess is terrible and unreliable")

#

would like to understand what pieces are at play if anyone can throw me links to the right direction, I'm willing to do my own reading

rose schooner
#

oh wow
python/cpython#112732

neon troutBOT
olive marsh
#

where can i get design diagram of python argparse module , its very complex

grave jolt
#

Isn't doing filter(None.__ne__, things) bogus anyway? That's literally how equality dunders work

rose schooner
grave jolt
rose schooner
raven ridge
rose schooner
grave jolt
# raven ridge why?

__eq__, __ne__, __lt__, __gt__ etc. all have NotImplemented as a valid return value

#

don't they?

raven ridge
#

well, sure - that's not the issue

#

the issue is that None.__ne__(None) should be False

# 
filter(None.__ne__, things)
``` seems to be a less verbose version of ```py
(x for x in things if x != None)
``` Would you call the latter bogus?
grave jolt
#

The latter uses != which does the correct thing (check the other way around if the first comparison returned NotImplemented)

rose schooner
rose schooner
raven ridge
feral island
grave jolt
#

yeah, perhaps

#

maybe people should configure their linters to ban these calls πŸ™‚

#

I've only ween WPS prohibit direct dunder calls

#

!e

a = 0.5
b = 3
print(a.__add__(b))
print(b.__add__(a))
fallen slateBOT
#

@grave jolt :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | 3.5
002 | NotImplemented
grave jolt
#

!e

a = 0.5
b = 3
print(a.__eq__(b))
print(b.__eq__(a))
fallen slateBOT
#

@grave jolt :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | False
002 | NotImplemented
grave jolt
#

Actually, why would it ever be valid to return False while __eq__/__neq__ing values of different types?

#

doesn't that break commutativity? (a != b <=> b != a)

rose schooner
#

True != 1 also validly returns False

grave jolt
#

!e

class Foo(int):
    __hash__ = None

    def __eq__(self, other):
        return other is True

print(True == Foo())
fallen slateBOT
#

@grave jolt :white_check_mark: Your 3.12 eval job has completed with return code 0.

False
grave jolt
#

Copying stuff from IPython is unnecessarily complicated

rose schooner
#

isn't that also the same for any repl

grave jolt
#

Yes

#

seems like nobody thought of a decent solution

#

Oh wait, they did and I am a dummy

In [6]: %history 5
class Foo(int):
    __hash__ = None

    def __eq__(self, other):
        return other is True

print(True == Foo())
#

til

#

is that because of the common int parent?..

raven ridge
#

yeah

#

If the operands are of different types, and right operand’s type is a direct or indirect subclass of the left operand’s type, the reflected method of the right operand has priority, otherwise the left operand’s method has priority. Virtual subclassing is not considered.

limpid scarab
#

So I have recently opened an idea on the python ideas board (https://discuss.python.org/t/add-ability-to-force-alignment-of-ctypes-structure/39109), and also an issue on github (https://github.com/python/cpython/issues/112433)
There has been a little interest but not too much, but I am not sure if this is just because people have no opinion on this feature or what.
Are there any further steps I need to do to try and make this new feature a reality? I have already made some POC code changes, and have partially written some tests for it locally, so I'm definitely happy to do pretty much all the work for it, just need some direction on how to move forward I guess...

GitHub
Add ability to force alignment of ctypes.Structure Β· Issue #112433 ...

Feature or enhancement Proposal: When creating a ctypes.Structure to map data from c/c++ to python, I have been coming up against issues where I have a structure which has an alignment due to a #pr...

feral island
limpid scarab
#

ok sure, thanks. I assume to have a better chance it would be good to have all the documentation and test changes already done before making the PR...

hasty turtle
#

!e
import marshal
marshal.loads(b'\xa9\x01<\x01\x00\x00\x00r\x00\x00\x00\x00')

fallen slateBOT
#

@hasty turtle :warning: Your 3.12 eval job has completed with return code 139 (SIGSEGV).

[No output]
hasty turtle
#

!e
import marshal
print(marshal.loads(b'\xa9\x01r\x00\x00\x00\x00'))

fallen slateBOT
#

@hasty turtle :white_check_mark: Your 3.12 eval job has completed with return code 0.

((...),)
urban sandal
# grave jolt maybe people should configure their linters to ban these calls πŸ™‚

That would be a spurious lint, it's perfectly safe to do so within the same type, and only an issue for dunders across different types. I'd much prefer people use something like (0).__le__ than either bind a lambda (lambda i: i > 0) or have to do functools.partial(operator.le, 0) It would be cool if python could detect all of these and replace them with the same thing automatically so that there wasn't a reason to prefer not having that lambda, but limitations of interpreted languages and mutable functions...

umbral plume
#

there's plenty of things you can do in python which aren't necessarily unsafe to do, yet various linters will still have rules to flag up, including subjective style decisions

urban sandal
#

While that may be true, I don't think that's always a good design decision. We shouldn't have automated tools that are supposed to be good at detecting issues (linters) point people to a potentially worse way of doing something. If that pattern is something CPython devs don't want to see, then there should be efforts to make sure the alternatives are as good (They aren't).

spark magnet
#

i'm trying to decipher this discussion. What is the bad thing to do that linters complain about, and what is the worse way they point to?

urban sandal
#

using bound methods of a constant isn't inherently bad, a hypothetical lint to warn on that would lead to worse performance for no improvement in behavior in the cases it is fine in.

#

(see the thing I replied to not realizing at first how far back that discussion started)

raven ridge
#

the linter isn't complaining about using bound methods of a constant, it's complaining that using obj1.__ne__(obj2) isn't equivalent to obj1 != obj2

#

at least not in general, if obj1 and obj2 are different types. If you run obj1.__ne__(obj2) only type(obj1) gets a chance to decide whether they're unequal, if you run obj1 != obj2 then type(obj2) gets a chance to weigh in if type(obj1) says "I don't know"

#

(and that's a subtle bug; I didn't catch it right away even when @grave jolt was deliberately drawing my attention to it)

urban sandal
#

and yet the point remains. There are places where it is fine (when all the objects are of the same type) and there are real reasons to use that.

raven ridge
#

sure. But in this case - the case that started the conversation, filter(None.__ne__, iterable) - it would be nonsensical if all of the objects were the same type

#

it wouldn't make any sense to run that if you know that all of the elements of the iterable are None

#

I agree with you that linters are sometimes overly opinionated about stylistic stuff, but I don't think this is one of those cases. This is way more likely to be a bug than not be a bug, and it's reasonable for linters to flag things that are likely wrong even if they can't be certain it's wrong

urban sandal
#

the GH issue points out that that's a common pattern, been in use for a while without issue, and was only broken because of an underlying implementation change. I'd consider it more broken if any other type lied and had objects it said it equaled None

raven ridge
#

I think you're conflating two different things, though. Everyone agrees that the change to None.__ne__ was an unintentional breaking change that should be reverted. That's a separate question from whether I think you're conflating two different things, though. Everyone agrees that the change to None.__ne__ was an unintentional breaking change that should be reverted.

That's a totally separate question from whether obj1.__ne__(obj2) is something that linters should warn about

urban sandal
#

I'm not conflating things here, I think the only cases where obj1.__ne__(obj2) would be used and would break involve other design issues that should be considered the root issue.

#

requires an issue like mixed data types that dont know about eachother, but one adds the reverse comparison dunders

raven ridge
#

like int and float

urban sandal
#

I consider mixing ints and floats a much bigger issue than this. Even without this that can cause issues if people are unaware of it.

raven ridge
#

well - πŸ€·β€β™‚οΈ type checkers disagree with you, fwiw. They explicitly allow passing ints to functions documented as accepting floats

urban sandal
#

yes, but the other way around is problematic, and we're talking about just a pool of mixed objects here

raven ridge
#

!e no, not necessarily. ```py
from typing import Iterable
def drop_element(x: float, it: Iterable[float]) -> Iterable[float]:
return filter(x.ne, it)

print(list(drop_element(1, [1.0, 2.0, 3.0])))

flat gazelle
#

Python numeric types are designed in a way that allows an int to be a float/fraction/decimal/complex/etc

fallen slateBOT
#

@raven ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | /home/main.py:5: DeprecationWarning: NotImplemented should not be used in a boolean context
002 |   print(list(drop_element(1, [1.0, 2.0, 3.0])))
003 | [1.0, 2.0, 3.0]
urban sandal
#

using not equal on a float is already a design issue

raven ridge
#

no, it isn't. It's fine when two floats are equal

#

you don't need an epsilon if you know what the value is

#

you only need an epsilon if you want to treat two unequal things as "close enough", and here, we don't.

#

and anyway, note the DeprecationWarning raised there. Python is telling you that this will become an error in a future version

urban sandal
#

that's not what's becoming an error, lol

raven ridge
#

what do you mean?

#

it's telling you that using filter(x.__ne__, iterable) will raise an exception in the future when x is an int and iterable contains a float

#

(again, despite the fact that type checkers will not flag that bug)

languid crow
#

hello godlygeek

urban sandal
#

python type checkers are wrong about a lot of things

#

The implicit bool(NotImplemented) is the future error. The root cause is not necessarily x.__ne__, but the combination of using it in a situation where it could return NotImplemented.

raven ridge
#

yes, but that's any situation where the objects being compared are of different types

spark magnet
#

@urban sandal did you say None.__ne__(x) is a common pattern?

urban sandal
#

comes up with use of filter

feral island
#

it's common enough that people noticed when they upgraded to 3.12, but I doubt it's a common pattern in general

spark magnet
#

I had never seen it before, but searching GitHub turns up more than I would have expected.

feral island
dusk comet
raven ridge
#

pylint complains about py return x.__ne__(y) with C2801: Unnecessarily calls dunder method __ne__. Use != operator. (unnecessary-dunder-call) but it doesn't complain about ```py
return filter(x.ne, y)

#

and "unnecessary" isn't really the issue here, the issue is that __ne__ isn't semantically equivalent to !=

urban sandal
#

The alternative to that is is functools.partial(operator.ne, x). There's no point in doing that if you already know you don't need to consider it. lambda y, x=x: y != x is strictly worse, but hey, no imports, so such a hypothetical lint would (based on other similar cases) push people to that.

It not being semantically equivalent isn't necessarily a bug. (one could argue it a misfeature of the reverse dunders though)

merry bramble
#

Something I would quite like would be an operator.not_None function. Semantically, it would be exactly the same as not_None = partial(operator.is_not, None) -- but I feel like it's common enough to want to filter out Nones from a container that it might make sense for it to be provided directly in the operator module. It would be nice to be able to do for x in filter(not_None, iterable):

native flame
#

i know its not the same but filter(None, it) usually works for that

swift imp
dusk comet
#

filter(bool, it) does the same thing as filter(None, it) but is much more readable imo

#

filter(None, it) is rarely used, it uses weird default behaviour that is understandable only by reading docs and it also can be confused with something like filter(op.is_none, it) or something like that

merry bramble
dapper lily
#

I've deleted your message as part of rule 6
Please familiarize yourself with the #rules of this server.

cyan raven
#

I know that CPython is testing Python's syntax in python with unittest frameworks, but was wondering what it used before it could implement the tests in its "own" language. Any ideas?

spark magnet
spark magnet
frigid bison
#

1990 holy shit

spark magnet
dusk comet
#

GvR is older

faint river
#

The pyramids in Egypt are older

spark magnet
#

(also something is weird about that early history, because there are many commits that look like parts of "Initial Revision")

dusk comet
#

guido also wrote --Guido (last modified 10 Sep 90) in readme, but commit is dated "9 aug 1990"

#

guido is time traveler probably

safe basalt
grave jolt
feral island
#

worth noting that CPython also predates git. It was in Mercurial until a few years ago, and either CVS or SVN before that, and possibly others before that. So the old history has gone through several layers of translation from one protocol to another, some of that may not have worked well

feral island
#

I think unittest might originally have been a third-party project, but not 100% sure of that

dusk comet
fallen slateBOT
#

Lib/unittest/__init__.py lines 2 to 3

Python unit testing framework, based on Erich Gamma's JUnit and Kent Beck's
Smalltalk testing framework (used with permission).```
dusk comet
fallen slateBOT
#

Lib/unittest/__init__.py lines 29 to 30

Copyright (c) 1999-2003 Steve Purcell
Copyright (c) 2003-2010 Python Software Foundation```
feral island
#

Thanks, that means there were at least 9 years of Python without unittest πŸ™‚

grave jolt
#

unittest more like udontneedtest

cyan raven
feral island
#

in 1990? doubtful

#

anyway, not too much point arguing about what "should" have been done 30 years ago

cyan raven
hasty turtle
feral island
boreal umbra
ripe tinsel
#

Hypothetical question: Can an ownership-like model (inspired by Rust but different) solve the same problems that the GIL solves in the context of multi-threading memory management?

For example, lets say every time a main thread opens its first sub thread an additional "owner thread" is created to own the data.

  • The main thread and all other threads can then have a pointer to owner thread for reading purposes.
  • To synchronize data between threads, the Thread object has a method called commit() that can be called on any attribute (attributes of an object are objects too and can have methods inserted).
  • Committing an attribute to the owner thread can use a queue to request the owner to overwrite its data (or take ownership of attributes of that name associated with the thread pool and replace the values in the threads that have that attribute).
  • If a variable is changed without committing, those changes only exist for the local thread.
  • If only the main thread is still active, the owner thread transfers ownership to the main thread and closes.

I am not suggesting this as a feature, my question is hypothetical:
Can a system that uses rules, such as an ownership system, solve the same problems as the GIL for multiprocessing memory management?

PS. Practically, I would include this in the garbage collector rather than have an "owner thread". The example is only meant for illustration.

#

PPS. A garbage collector would only have to keep track of thread names, a commit keyword also makes more sense than a method

grave jolt
#

With this system you can only express "set X to 7"

#

And who calls commit and when?

ripe tinsel
quick snow
ripe tinsel
#

That's why I used a thread for my example, so memory management operations happen in parallel on the owner thread and other threads only need to submit changes to queue

grave jolt
quick snow
ripe tinsel
quick snow
ripe tinsel
# grave jolt Suppose I want to use this system for a reference counter. The count is now at 1...

In that case, when thread 1 commits +1 it gets added to the queue and if thread 2 commits -1 afterwards then that gets added to the queue. When the queue resolves it will make the changes in that order and the only problem you have left is if you read in between those operations (happens in a multiprocessing Queue in anyway if you request queue size so this is acceptable but not perfect)

ripe tinsel
ripe tinsel
grave jolt
#

Then the object will temporarily have a 0 refcount so it will die immediately

delicate blaze
#

I know this is a python server but can I get some Java help

#

It’s very simple im pretty sure im just new to coding

grave jolt
fallen slateBOT
umbral plume
delicate blaze
#

Oh I’m sorry

dapper ice
#

HI

pliant tusk
#

!e ```py
class Foo:
bar = 'value'
def func(self):
nonlocal bar
bar = 'another value'

def func2(self):
    nonlocal bar
    return bar
fallen slateBOT
#

@pliant tusk :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 4
002 |     nonlocal bar
003 |     ^^^^^^^^^^^^
004 | SyntaxError: no binding for nonlocal 'bar' found
pliant tusk
#

!e it works for wrapped functions, i dont see why it shouldnt work for class scopes ```py
def outer():
bar = 'value'
def func():
nonlocal bar
bar = 'another value'

def func2():
    nonlocal bar
    return bar```
fallen slateBOT
#

@pliant tusk :warning: Your 3.12 eval job has completed with return code 0.

[No output]
flat gazelle
#

You cannot have closures capture class variables at all

#

even just ```py
class X:
x = 1
def f(self): return x

urban sandal
#

self.x/bar works fine there in each broken example, what's the context for wanting this?

dusk comet
#

class scope doesnt exist after class is created
whatever scope is copied to dict and stored internally somewhere in the type object

#

metaclasses can replace scopes with their own fancy objects, that is cool during class creation, but doesn't make a lot of sense afterwards

pliant tusk
flat gazelle
#

I do not believe it ever did, fails in py2.7 and py3.5, and the more common example of this not working,

x = 1
class X:
    x = 2
    def f(self): return x
assert X().f() == 1
```has been going around for quite a while now.
pliant tusk
#

huh, i could have sworn i wrote some cursed code that relied on it at some point

#

guess not

unkempt rock
#

Hello

steel solstice
# 
def foo[T](): pass
class Bar[T]: pass
>>> statistics.fmean(timeit.repeat("foo[int]", globals=globals()))
0.0814359413983766
>>> statistics.fmean(timeit.repeat("Bar[int]", globals=globals()))
0.9336082836030982
```whys 2 so much slower than 1?
quick snow
#

!e

def foo[T](): pass
foo[int]
fallen slateBOT
#

@quick snow :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 2, in <module>
003 |     foo[int]
004 |     ~~~^^^^^
005 | TypeError: 'function' object is not subscriptable
steel solstice
#

it makes a generic alias like GenericAlias(foo, (int,))

# 
Python 3.13.0a0 (heads/function-subscript:97220ab6aa, Dec  5 2023, 10:50:30) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo[T](): pass
... 
>>> foo[int]
__main__.foo[int]```
quick snow
#

Oh, 3.13

steel solstice
#

its for a pep

#

!pep 718

fallen slateBOT
dusk comet
steel solstice
#

yeah just function.getitem

#

hmm, wonder why its so slow on a class

#

its probably got specialisation actually for the function case

dusk comet
# 
>>> class X:
...   def __getitem__(self, key):
...     return GenericAlias(self, key)
...
>>> x = X()
>>> GenericAlias = type(list[int])
>>> n = 10**6
>>> min(timeit.repeat('Foo[int]', globals=globals(), number=n)) / n * 10**9 # ns
634.6786000067368
>>> min(timeit.repeat('x[int]', globals=globals(), number=n)) / n * 10**9 # ns
124.98309998773038
``` i can kinda reproduce it on 3.12
difference is not so big i guess because `X.__getitem__` is pure python function
steel solstice
#

hmm, yeah its not specialisation, it must just be really slow for classes

dusk comet
#

where is ceval.c located in new versions? i forgot

steel solstice
# 
>>> def foo(): 
...  foo[int]()
>>> dis.dis(foo)
  1           0 RESUME                   0

  2           2 LOAD_GLOBAL              1 (NULL + foo)
             12 LOAD_GLOBAL              2 (int)
             22 BINARY_SUBSCR
             26 CALL                     0
             34 POP_TOP
             36 RETURN_CONST             0 (None)
>>> class Bar[T]: pass
>>> def bar():
...  Bar[int]()
... 
>>> dis.dis(bar)
  1           0 RESUME                   0

  2           2 LOAD_GLOBAL              1 (NULL + Bar)
             12 LOAD_GLOBAL              2 (int)
             22 BINARY_SUBSCR
             26 CALL                     0
             34 POP_TOP
             36 RETURN_CONST             0 (None)```they have the same bytecode
dusk comet
#

what is the specialized bytecode?

#

there is BINARY_SUBSCR_GETITEM, but im not sure what it does

steel solstice
#

its probably just cause its calling through to typing._generic_class_getitem

ebon wagon
#

Hello everyone, what should I do if my smart friend deleted all the files that have the word python and now he doesn’t have textures in many games?xD

steel solstice
#

oh its not even returning an instance of types.GenericAlias which is probably why

dusk comet
#

same on 3.12

# 
>>> X[int]
__main__.X[int]
>>> type(X[int])
typing._GenericAlias
steel solstice
# 
>>> my_cool_list = list
>>> statistics.fmean(timeit.repeat("my_cool_list[int]", globals=globals()))
0.16742044179700316```looks slightly better
#

its only 2x rather 10x

dusk comet
fallen slateBOT
#

Objects/abstract.c line 183

return Py_GenericAlias(o, key);```
`Objects/listobject.c` line 2918
```c
{"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, PyDoc_STR("See PEP 585")},```
`Objects/genericaliasobject.c` line 986
```c
Py_GenericAlias(PyObject *origin, PyObject *args)```
dusk comet
#

oh wait, i unconfused myself

#

type[T] and list[T] call the same thing - types.GenericAlias

steel solstice
#

ye

#

but if you make your own instance it goes into typevarobject.h or .c

#

so if you check that you see that it goes to typing._generic_class_getitem

dusk comet
# 
>>> type.__class_getitem__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'type' has no attribute '__class_getitem__'
``` huh
#

i'm officially confused

#

such a mess

quick snow
#

So it would be just __getitem__ on type

steel solstice
#

but types metaclass is type no?

quick snow
#

But type is cheating

#

And even if you disregard that: It doesn't need to use __class_getitem__ because it can just go via __getitem__.

dusk comet
#

there is no __class_getitem__ slot there: https://github.com/python/cpython/blob/main/Objects/typeobject.c#L5239
and this method doesnt exist in X.__dict__, but somehow attr access works: ```py

'class_getitem' in X.dict
False
X.dict
mappingproxy({ 'module': 'main',
'type_params': (T,),
'getitem': X.getitem,
'orig_bases': (typing.Generic[T],),
'dict': <slot X.dict>,
'weakref': <slot X.weakref>,
'doc': None,
'parameters': (T,)})
X.class_getitem
X.class_getitem

fallen slateBOT
#

Objects/typeobject.c line 5239

static PyMethodDef type_methods[] = {```
dusk comet
fallen slateBOT
#

Objects/typeobject.c lines 5359 to 5360

0,                                          /* tp_as_sequence */
0,                                          /* tp_as_mapping */```
dusk comet
fallen slateBOT
#

Objects/typeobject.c lines 5164 to 5166

/* We have no guarantee that bases is a real tuple */
Py_ssize_t i, n;
n = PySequence_Size(bases); /* This better be right */```
dusk comet
#

so i can provide instance of tuple subclass with bad behaviour?

# 
>>> X = type('bad_tuple',(tuple,),dict(__len__=lambda*_:-1))
>>> type('',X([int]),{}).__bases__.__class__
bad_tuple
>>> len(X([1,2,3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: __len__() should return >= 0
steel solstice
#

doesnt even do anything exciting for out of bounds +ve

dusk comet
#

sadly it is bypassing some overriden methods :(

# 
>>> X = type('bad_tuple',(tuple,),dict(__len__=lambda*_:10,__getitem__=lambda*_:'hi',__iter__=lambda*_:iter('world')))
>>> type('',X([int]),{}).__bases__
('w', 'o', 'r', 'l', 'd') # this is from pprint.pprint formatting
>>> str(type('',X([int]),{}).__bases__)
'(int,)' # this is what actually is stored
>>> type('',X([int]),{})('42')
42
limpid scarab
#

ok, so this is gonna be a weird one... So I am playing around with injecting python into a running process using pymem. This works well, but I have noticed an issue where certain keypressed don't work correctly. Long story short, I am reasonably certain that by injecting python the towupper function which resides in ucrtbase.dll is being somehow modified so that it doesn't return the correct values (or at least the values that the game expects...)
I am wondering if anyone know why this couild be happening. It definately happens when python is injected as I can make that happen on demand and there is no issue until I inject it.
I do see that python seems to have some checks around ucrtbase on windows so I wonder if it's doing something internally to make this function not operate the same way...

merry bramble
dusk comet
#

yeah, i know, thank you
i guess my wording was ambiguous πŸ₯², i was asking Gobot about bytecode of their function

mild cobalt
mortal lynx
#

Hello guys, this might be a stupid question but does anyone know that where can i find the source code of python3-devel, I would be more interested in how the whole process works, where you have the python installed on your pc so regardless of where you are you will always can access it, how does this whole process work?

fallen slateBOT
#

Cannot operate on silence internals-and-peps; it is currently locked and in use by another operation. Please wait for it to finish and try again later.

#

βœ… silenced current channel for 8 minute(s).

#

Cannot operate on silence internals-and-peps; it is currently locked and in use by another operation. Please wait for it to finish and try again later.

grave jolt
#

!silence

unkempt rock
#

!silence

fallen slateBOT
#

❌ current channel is already silenced.

#

Cannot operate on silence internals-and-peps; it is currently locked and in use by another operation. Please wait for it to finish and try again later.

#

βœ… unsilenced current channel.

limpid scarab
bright basin
raven ridge
#

it is true that CPython is a stack based VM

bright basin
#

Are there plans to refactor it into a more efficient model ? How much effort would it take ?

dusk comet
#

there were experiments on register-based VM in faster-python project
i dont know any additional details

rose schooner
#

there are also layouts preparing for the hybrid stack/register-based VM

bright basin
#

Uhmm, I want to start reading the CPython source code. Would this stuff be a good place to start ?

I'd imagine that the upgrade is not that convoluted if the code is well modularized. Tho full disclaimer, I know nothing about this stuff.

dusk comet
#

i dont know where are all parts of VM are located, but the part that does the bytecode executing is located in /Python/bytecodes.c file, iirc

bright basin
#

Got it, already have questions, but its basic stuff so I'm gonna see how for I can go with gpt4

dusk comet
#

TIL: on 3.12 there is literally no difference in perfomance between accesing __slots__'ed attribute vs normal attribute
there was little difference (~10-20%) on previous versions, but on 3.12 i can't even measure it

#

columns are: setup code, measured code, time spent in measured code
im using timeit for it

D:\>py -3.8 _.py
class X:__slots__=("a",)\nx=X();x.a=1    x.a                   20.21 ns
class X:...\nx=X();x.a=1                 x.a                   22.31 ns
class X:__slots__=("a",)\nx=X()          x.a = 1               25.09 ns
class X:...\nx=X()                       x.a = 1               31.11 ns
class X:__slots__=("a",)                 X()                   47.4 ns
class X:...                              X()                   48.29 ns

D:\>py -3.9 _.py
class X:__slots__=("a",)\nx=X();x.a=1    x.a                   20.13 ns
class X:...\nx=X();x.a=1                 x.a                   22.03 ns
class X:__slots__=("a",)\nx=X()          x.a = 1               23.7 ns
class X:...\nx=X()                       x.a = 1               30.57 ns
class X:__slots__=("a",)                 X()                   48.49 ns
class X:...                              X()                   50.09 ns

D:\>py -3.10 _.py
class X:__slots__=("a",)\nx=X();x.a=1    x.a                   23.41 ns
class X:...\nx=X();x.a=1                 x.a                   25.21 ns
class X:__slots__=("a",)\nx=X()          x.a = 1               27.02 ns
class X:...\nx=X()                       x.a = 1               31.86 ns
class X:__slots__=("a",)                 X()                   50.13 ns
class X:...                              X()                   55.4 ns

D:\>py -3.11 _.py
class X:__slots__=("a",)\nx=X();x.a=1    x.a                   14.99 ns
class X:...\nx=X();x.a=1                 x.a                   15.35 ns
class X:__slots__=("a",)\nx=X()          x.a = 1               13.69 ns
class X:...\nx=X()                       x.a = 1               13.86 ns
class X:__slots__=("a",)                 X()                   62.01 ns
class X:...                              X()                   71.82 ns
# 

D:\>py -3.12 _.py
class X:__slots__=("a",)\nx=X();x.a=1    x.a                   13.78 ns
class X:...\nx=X();x.a=1                 x.a                   13.77 ns
class X:__slots__=("a",)\nx=X()          x.a = 1               13.74 ns
class X:...\nx=X()                       x.a = 1               13.75 ns
class X:__slots__=("a",)                 X()                   61.98 ns
class X:...                              X()                   74.9 ns
rose schooner
#

good news i guess

dusk comet
#

also i noticed that '...%s...' % (x,) formatting is as fast as f'...{x}...', but i think this was a thing for a long time (a couple of years)

rose schooner
dusk comet
#

this is not optimized because there is no way to statically determine that x is not a tuple

#

so it defers to slow old method

rose schooner
#

ok

unkempt rock
#

hi i have a question
this are the structs for cpython's tuple and list objects

typedef struct {
    PyObject_VAR_HEAD
    PyObject *ob_item[1];
} PyTupleObject;

typedef struct {
    PyObject_VAR_HEAD
    PyObject **ob_item;
    Py_ssize_t allocated;
} PyListObject;

why the difference in how ob_item is declared?

rose schooner
unkempt rock
#

but tuples can also be resized

#

@rose schooner

rose schooner
fallen slateBOT
#

Objects/tupleobject.c line 899

_PyTuple_Resize(PyObject **pv, Py_ssize_t newsize)```
flat gazelle
#

it's a naughty C trick, you access a given element of a tuple like type.ob_item[10] for example - making sure to allocate more space than the PyTupleObject struct itself requires.

rose schooner
unkempt rock
#

but i thought that was to avoid breaking code that relied on immutable tuples

rose schooner
unkempt rock
#

i still don't see how ob_item plays into this

rose schooner
#

PyTupleObject.ob_item being declared as an array basically makes it part of the tuple structure itself

unkempt rock
rose schooner
#

so it can't be changed without reallocating the whole tuple structure

unkempt rock
#

what about lists then?

#

can you just point to a different vector?

#

is that the entire point

rose schooner
#

PyListObject.ob_item being declared as a pointer (to a pointer of an object) makes it reference another piece of memory separate from the list structure

#

it's reassignable

unkempt rock
#

ohh

rose schooner
#

unlike PyTupleObject.ob_item

unkempt rock
#

ok i get it

unkempt rock
#

makes sense tho

#

ish

#

wait no it doesn't how does that work

grave jolt
unkempt rock
#

i thought structs were just pointers with members

grave jolt
#

A struct describes a fixed-size value and names its members, and the members are stored contiguously in memory

rose schooner
quick snow
#

You can make the last element of a struct be variable-sized

grave jolt
#

Yeah well, that's kind of a hack

unkempt rock
#

for lists, the last element isn't ob_item

flat gazelle
grave jolt
quick snow
flat gazelle
#

And made optional in further standards along with VLAs AFAIK.

unkempt rock
quick snow
rose schooner
#

static arrays

unkempt rock
#

oh i am just really lost then

#

i don't think they're laid out how i guessed then

#

i based that guess on a very bad little test run

#

ok thanks guys that was a nice wakeup call

grave jolt
#

So it will literally occupy 16 consecutive memory cells, like an array

unkempt rock
#

no i got it now

#

didn’t even know that byte padding was a thing

#

which is where most of the confusion came from

south radish
#

hey does anyone know how to download Spotify playlists using python script

winged sphinx
lone gorge
dusk comet
#

what do you think about for x in y if z: statement syntax?

# imaginary syntax:
for var in iterable if condition:
    do_stuff()

# inspired by this:
[expr for var in iterable if condition]

# works like this:
for var in iterable:
    if condition:
        do_stuff()
quick snow
#

Kind of cute, but making it harder to skim code. Reminds me of Ruby a bit, where you can do stuff like

return 42 unless false
dusk comet
raven ridge
dusk comet
#

one line > two lines

#

also less indentation levels

raven ridge
#

I think there's no chance of that proposal ever getting accepted

#

things that allow more terse code just for the heck of it are incredibly unlikely to be accepted

quick snow
#

If you're worried about indentation levels you can also just do

for var in iterable:
    if not condition:
        continue
    ...
raven ridge
dusk comet
#

if you can write the same code in less lines without losing readability, i think it is a win
your get_area example joins two lines together, it is not simpler than "normal" code, it only hurts readability a bit
my proposal gets rid of for: if:/for:if not:continue boilerplate, and it adds new syntax, so there is no need to think about these compound statements - you have one simple statement that is baked into the syntax

spark magnet
raven ridge
#

in other words, we could have something today that gives both of your stated advantages (1 line instead of 2, no extra indent level) by just allowing an if statement on the same line as a for statement, but that's specifically and deliberately blocked as things stand today

urban sandal
#

people can already one line this and I've seen people do so, even at the cost of a pointless list:

[do_stuff() for var in iterable if condition]
#

might have better luck getting that specific pattern special cased in the interpreter to never create the list

spark magnet
quick snow
spark magnet
sly jay
unkempt rock
#

thanks

rose schooner
#

not sure where i found it but i did encounter it before

south pilot
raw jungle
#

One line is comfortable but not that much readable

urban sandal
# spark magnet might as well do: ```py for var in (var in iterable if condition): do_stuff(...

I agree that what you have there is the better version. I don't want to encourage pointless one-liners, writing more is fine, it's the same ideas split vertically instead of horizontally, all within a reasonable amount of total screenspace. Was pointing out that doesn't stop people from doing it, even when it comes with other reasons not to beyond the readability. (like the pointless list that could be too large to for in memory if iterable is large enough lazily generated...)

Edit: No, I meant the written out form and got a bit of it mixed up.

#

was more to point to the absurdity as a reminder that we don't actually need more terseness all the time

spark magnet
#

btw, I would not approve of the for-loop-over-generator that I suggested. Write it out.

ruby elm
boreal umbra
# 
import re

for _ in range(10):
    pattern = re.compile(r'\d+')

Does re optimize this such that the actual work of compiling \d+ is done only once, and any calls to compile a pattern represented by the same string retrieves the existing one from a cache?

steel solstice
#

yes, it has a small cache

fallen slateBOT
#

Lib/re/__init__.py lines 374 to 377

@functools.lru_cache(_MAXCACHE)
def _compile_template(pattern, repl):
    # internal: compile replacement pattern
    return _sre.template(pattern, _parser.parse_template(repl, pattern))```
steel solstice
#

ooh sorry maybe its not that one but the _compile above

merry bramble
#

You should be careful though β€” despite the internal cache, if your program uses a lot of regexes, you can often still get a large speedup from compiling the patterns once and reusing them. I got a 9% speedup for one of the tools CPython uses in CI just by doing that: https://github.com/sphinx-contrib/sphinx-lint/pull/82

raven ridge
#

oh hey, I use sphinx-lint - thanks for that, @merry bramble πŸ™‚

merry bramble
boreal umbra
raven ridge
#

that's from avoiding unnecessarily re-compiling the same pattern over and over

jade raven
raven ridge
#

the cache has a size limit, so if your program uses a lot of regexes, you might wind up evicting things from the cache that you actually need, and needing to re-compile them later

merry bramble
#

Yeah, the cache is an LRU cache: once it reaches its maximum size, the regexes that you used least recently are evicted from the cache

#

If you're dynamically generating lots of regexes (sphinx-lint is), then you can pretty quickly use up the cache, meaning the regexes that you actually are reusing over and over aren't actually present in the cache a lot of the time, and unnecessarily get recompiled over and over

raven ridge
#

Demo: ```
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(100)]'
2000 loops, best of 5: 221 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(100)]'
2000 loops, best of 5: 179 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(200)]'
1000 loops, best of 5: 375 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(300)]'
500 loops, best of 5: 565 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(400)]'
1 loop, best of 5: 737 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(500)]'
1 loop, best of 5: 908 usec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(600)]'
1 loop, best of 5: 313 msec per loop
$ python -m timeit -s 'import re' '[re.match("x" * i, "") for i in range(700)]'
1 loop, best of 5: 458 msec per loop

#

note the giant jump between creating 500 regexes and 600 - it leaps up from hundreds of microseconds per call to hundreds of milliseconds, 3 orders of magnitude slower

jade raven
raven ridge
#

and you know that the libraries you're using also don't use regexes internally

jade raven
#

ah never mind, i blinded over the

if your program uses a lot of regexes
in alexs message

raven ridge
#

if you don't know that, then you can't make any guarantees about whether the total number of distinct regexes used by the program exceeds the cache size or not, or the likelihood of your regex getting evicted from the cache to make room for regexes used by some library

jade raven
#

that's a very good point, never thought of that

jade raven
#

<@&831776746206265384>

cyan raven
#

Could you please send me this message 10 times in private?
If I don't respond increase it by 2 every minute.

raven ridge
#

@graceful oar Review our rules. We don't allow advertising on this server, so I'm deleting your message.

raven ridge
cyan raven
urban sandal
# merry bramble If you're dynamically generating lots of regexes (sphinx-lint is), then you can ...

I ran into a case with this a while back that made me think about if it was worth using a different eviction policy than LRU, but I ended up thinking it probably didnt occur too often and people can just not rely on the cache...

Could possibly be worth switching to an LFU eviction policy, would be less likely that people run into this kind of issue, but the cache semantics are a little more complex, may not be worth it

idle ravine
mild cobalt
errant oak
#

@summer lichen

mystic cargo
#

@graceful oarWe do not allow recruitment in this server, sorry.

#

!rule 9

fallen slateBOT
#
Rules

9. Do not offer or ask for paid work of any kind.

faint river
#

<@&831776746206265384> 1009175270613319802 asking for help with unethical/illegal topics (and also not topical to this channel πŸ˜„). This is the only thing they have mentioned in their 5 msgs in the server.

image.png
valid chasm
#

@unkempt rock, kindly refer to rule 5. We will not help with malicious topics.

#

!rule 5

fallen slateBOT
#
Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

flint steeple
#

i guess this guy really needs a python script that can crack wifi passwords

grave jolt
#

they should write a PEP for it then

unkempt rock
unkempt rock
modern pecan
#

Hello, I am working on a fully static build of python and I noticed that the "internal" static libs (libmpdec.a and libexpat.a) are not getting installed anywhere. Is there a reason for this? Could this be added? Maybe only for the case when the static lib is enabled and shared libs are disabled?

formal ore
rancid folio
#

Hi everyone, I have a question, from what I've observed ( not from here ) some people have a negative opinion on classes / OOP. why is that? Least from my experience it's been a fairly straight-forward. I've used Traits before in rust and that somehow made it unbareable.

sour thistle
boreal umbra
#

Or you fount traits unbearable?

rancid folio
inland acorn
# rancid folio Hi everyone, I have a question, from what I've observed ( not from here ) some...

This is specifically about inheritance, but it sums it up nicely: https://youtu.be/hxGOiiR9ZKg?si=Ehbnru9l7ZYMMNYw

YouTube
CodeAesthetic
The Flaws of Inheritance

Let's discuss the tradeoffs between Inheritance and Composition

Access to code examples, discord, song names and more at https://www.patreon.com/codeaesthetic

0:00 Introduction
0:25 Inheritance
3:32 Composition
5:22 Abstracting with Inheritance
6:52 Abstracting with Interfaces
8:20 When to use Inheritance

β–Ά Play video
#

It's less about it being confusing or whatever, but how easy it is to change the code

#

i.e coupling

inland acorn
swift imp
#

Or maybe even that encaspulation is pointless and really the crux of it all

regal glen
#

The bigger thing for all of that, is don’t do premature encapsulation or inheritance. Don’t start trying to solve problems that don’t exist yet just because they might exist later.

inland acorn
#

Oh yhe, people always talk about premature optimization, don't see much talk about premature abstraction

grave jolt
#

it's also not quite clear what the narrator means by OOP, because I have no clue what it is

urban sandal
grave jolt
#

It never defines encapsulation or objects, and those are very vague terms

ruby elm
#

I've never been able to take that video seriously (Object-Oriented Programming is Bad). That entire section of the video on not passing references is pointless since they later say themselves no-one codes like that in practice. To me the entire middle part of the video falls apart as just a rant on frustrations with the programing industry and not an actual critique or guide. Everything they complain about is something that just happens in large systems of code regardless of the paradigm used, since in the end dealing with state and io is hard. Even with pure functional code, you can still make a mess of state encapsulation once you start currying since that is still data, just stored differently.

urban sandal
#

and it relies on agreeing with many definitions that would be accepted in functional programming circles that werent agreed upon, or at least presented first

#

I do think people are too quick to encapsulate everything in a class rather than having a function which can operate on only what's needed, but there are better ways to make the argument than that video does.

urban sandal
#

A video that may make the argument slightly better (and more pragmatically) would be this one: https://www.youtube.com/watch?v=tD5NrevFtbU but it isn't even going after oop in entirety, but after certain patterns of factoring oop code.

People should keep in mind that this is from the perspective of someone teaching a course on performance and game dev, and that the performance reasons alone are not universally the important factor people should consider, but there's a few things in it that apply beyond, and a few of the things that make it easier for the compiler to do the right thing also help with locality of ideas and congitive reasoning about a program's design.

rancid folio
inland acorn
#

Are you familiar with ABCs (or Protocols for that matter) in Python?

rancid folio
silk slate
#

Sup fools.

At what stages does the GLEB behavior happen in interpretation and compilation?

When and how is the pycore_flowgraph.h used?

#

Python compiler and interpreter hit with the black magic stick or no?

jade raven
#

edit your message to be a bit more respectful

silk slate
#

Thats super respectful.

#

You just interpret it in an unintended context

jade raven
#

Sup fools.

silk slate
#

Irony

#

IRONY

slate sable
#

what's ironic about it?

silk slate
#

It's bold faced apparent irony, it's like calling you guys smart.

slate sable
#

.... yeah sure

silk slate
#

Cuz this is the place for the devs who know way more about python.

#

Smh

#

Which doesn't mean "squish my horse"

#

It means "shake my head"

slate sable
#

ok

silk slate
#

What's a furti?

#

Wait nvm that not why I'm here.

#

I'll wait for answers.

spark magnet
#

@silk slate what is an "unoptimized search" and where do you see one?

silk slate
#

I've never added anything to a stuff, so figured maybe it's a good opportunity.

spark magnet
silk slate
#

The search happens every time, yee?

#

Why?

#

The search doesn't even need to always happen.

spark magnet
silk slate
#

Can map the stuff and see if it can change per some branch

#

Wish I could send a voice message to quick say it all

#

But I'm too lazy to type it

spark magnet
#

you're going to have to use more words, and more specific words.

#

ok, never mind then.

silk slate
#

Yee

jade raven
#

you're not going to get an answer from the core devs without putting some more effort.

silk slate
#

I'll just read source then.

jade raven
#

they have more important things to do than try to read your mind and decipher what little you've said.

silk slate
#

Ok

spark magnet
#

@silk slate definitely also look into how dynamic python can get. names can come and go.

unkempt rock
#

core devs too busy copy pasting meta nogil code :/

silk slate
silk slate
#

or

#

idk theres all kinds of ways

#

point is i seen someone test it and show a 10% performance difference copying a global in.

#

and i thought

#

thats stupid

#

so i started reading source and learning more about the behavior.

#

im on my laptop now, so i can type faster.

spark magnet
#

it could be that you have a way to improve things, that will be great.

silk slate
spark magnet
#

but the core devs are not dumb.

silk slate
#

neither am i

spark magnet
silk slate
#

bleehhh gonna throw up

#

i hate ghidra

#

reverse engineering so slow

spark magnet
slate sable
#

Ghidra is a reverse engineering tool, not in anyway related to what you said.

silk slate
#

ghidra decompiles stuff and maybe sometimes can turn to source

jade raven
#

you dont need to decompile the bytecode

silk slate
#

it is, because what youre doing is the same essentially.

jade raven
#

!d dis

fallen slateBOT
#
dis

Source code: Lib/dis.py

The dis module supports the analysis of CPython bytecode by disassembling it. The CPython bytecode which this module takes as an input is defined in the file Include/opcode.h and used by the compiler and the interpreter.

spark magnet
silk slate
#

its reading the lowest stuff to infer some higher level pattern

#

thats backwards. ill just read source.

#

thanks

spark magnet