internals-and-peps

[static bluff]

I have a bit of a conundrum

I'm working on what you might call a wrapper around python's object model

To provide seamless, invisible privacy enforcement

As such, I have a base object class and a base type (metaclass) class

As with normal python, the metaclass needs to be a subclass of the base object class, and base object class needs to be an instance of the metaclass

Snake eating its own tail

So somewhere along the line, some surgery is going to have to happen to get everything aligned. The question is — dynamically change the Object class' type to Type, or, somehow inject the Object class into the mro of the Type class

[white nexus]

so i was lied to

#internals-and-peps message

!e ```py
class A:
...
class B:
...
a = A()
print(isinstance(a, B))
print(type(a))
print('-------')
a.class = B
print(isinstance(a, B))
print(type(a))

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

001 | False
002 | <class '__main__.A'>
003 | -------
004 | True
005 | <class '__main__.B'>

[white nexus]

type and isinstance are both in c but can still lie

[rose schooner]

delete the image

there's a bot token in there

[feral cedar]

your token is in that image, you don't want that on discord

you also want to regenerate the token

in case someone has your token

[deft horizon]

Such things are always at best heuristics, because https://docs.python.org/3/library/abc.html#abc.ABCMeta.__subclasshook__ exists and lets you do things like https://www.hillelwayne.com/negatypes/

[feral island]

they don't lie. you actually changed the class of that instance

[white nexus]

how does that work?

[feral island]

I guess it changes the ob_type field in the PyObject

[white nexus]

huh

this is unittest mock does it

https://github.com/python/cpython/blob/9d38120e335357a3b294277fd5eff0a10e46e043/Lib/unittest/mock.py#L542-L546

[fallen slate]

Lib/unittest/mock.py lines 542 to 546

@property
def __class__(self):
    if self._spec_class is None:
        return type(self)
    return self._spec_class```

[white nexus]

but how does this work

https://github.com/python/cpython/blob/9d38120e335357a3b294277fd5eff0a10e46e043/Lib/unittest/mock.py#L66-L69

[fallen slate]

Lib/unittest/mock.py lines 66 to 69

def _is_instance_mock(obj):
    # can't use isinstance on Mock objects because they override __class__
    # The base class for all mocks is NonCallableMock
    return issubclass(type(obj), NonCallableMock)```

[feral island]

That's a bit different. They don't actually set __class__, they just provide an override of the name.

That works because isinstance() internally does getattr(obj, "__class__")

(the C equivalent of that)

[white nexus]

but type() would return the actual class?

[feral island]

yes, that returns the ob_type field directly

[white nexus]

!e interesting ```py
class A:
@property
def class(self):
return int
a = A()
print(isinstance(a, int))
print(type(a))
print('-------')

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

001 | True
002 | <class '__main__.A'>
003 | -------

[feral island]

this is implemented here: https://github.com/python/cpython/blob/440332072706c5e422e6c54a2ec0ebb88e09c85c/Objects/abstract.c#L2587

[fallen slate]

Objects/abstract.c line 2587

retval = _PyObject_LookupAttr(inst, &_Py_ID(__class__), &icls);```

[candid grove]

y = (1, 2)
print(x == y)
print( 1, 2 == (1, 2))
print( (1, 2) == 1, 2)

[quick night]

all that is equivalent to this

x = (1, 2)
y = (1, 2)
print(x == y)
print(1, (2 == (1, 2))
print(((1, 2) == 1), 2)

@candid grove does this make sense?

[candid grove]

No

[quick night]

ok so when u do x = 1, 2 it creates a tuple (1, 2)
so x == y is (1, 2) == (1, 2) which is true

when u do (1, 2 == (1, 2)) this is equal to
( 1, (2 == (1, 2))) where the first element is 1 and second element is a condition (2 == (1, 2)) which is false

and when u do ( (1, 2) == 1, 2) this is equal to
( (1, 2) == 1, 2), here the first element is a condition (1, 2) == 1 which is false, and second element is 2

[candid grove]

Ok, now it makes, thanks

[quick night]

👍

[west tinsel]

Are soft keywords new in 3.10?

i.e. are match case and _ the only ones?

[white nexus]

no, yes

softkeywords were introduced in 3.9, but match and case are the only ones

[west tinsel]

Oh, how did that work?

[white nexus]

!d keyword

[fallen slate]

keyword

Source code: Lib/keyword.py

---

This module allows a Python program to determine if a string is a
keyword or soft keyword.

[white nexus]

!e import keyword ; print(keyword.softkwlist)

[fallen slate]

@white nexus :white_check_mark: Your eval job has completed with return code 0.

['_', 'case', 'match']

[white nexus]

python/cpython#20877

[neon trout]

GitHub

Include soft keywords in keyword.py

[west tinsel]

Oh interesting

What did support for soft keywords mean if there weren't any at the time?

[white nexus]

¯_(ツ)_/¯

[west tinsel]

Even still thanks for the answer and references, I learned a lot

[visual shadow]

I believe match isn't the first soft keyword, async await were introduced as soft keywords

[white nexus]

hm

[feral island]

true, but they became full keywords later, in either 3.7 or 3.8 I think

[white nexus]

match and case (and _) are supposed to stay soft keywords

[feral island]

yes. or we'll make a lot of re users very unhappy

[white nexus]

and would really break backwards compatibility

[visual shadow]

yeah 3.7. and also true regarding match being a default variable name for re users. guilty as charged 😛

[feral island]

not just that, it would break even calling re.match if we made match a full keyword

[visual shadow]

oh i hadn't thought of that, good point

[vernal plume]

👍

[west tinsel]

Has anyone used enums with nested classes or attempted to do so? If you have, what were the results and did you find any behaviour surprising or unexpected?
Here are two examples describing the situation I'm talking about

class Outer(Enum):
    a = 1
    b = 2
    class Inner(Enum):
        foo = 10
        bar = 11

class Outer(Enum):
    a = 1
    b = 2
    class Inner:
            c = None
            def __init__(self):
                ....

[elder blade]

Huh, what's your use-case for this?

That seems rather confusing 😅

[west tinsel]

I've been trying to get information about how people use the enum library to work out what is and isn't safe to modify https://github.com/python/cpython/issues/78157 https://github.com/python/cpython/pull/7950

[unkempt rock]

__import__('zipfile').ZipFile() doesnt work but from zipfile import ZipFile does

__import__ raises TypeError: 'ZipFile' object is not callable

>>> from dis import dis;dis("from zipfile import ZipFile")
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (('ZipFile',))
              4 IMPORT_NAME              0 (zipfile)
              6 IMPORT_FROM              1 (ZipFile)
              8 STORE_NAME               1 (ZipFile)
             10 POP_TOP
             12 LOAD_CONST               2 (None)
             14 RETURN_VALUE``` ```py
>>> from dis import dis;dis("__import__('zipfile').ZipFile")
  1           0 LOAD_NAME                0 (__import__)
              2 LOAD_CONST               0 ('zipfile')
              4 CALL_FUNCTION            1
              6 LOAD_ATTR                1 (ZipFile)
              8 RETURN_VALUE```

please ping me and send me the message link in dms after you reply because i wont be on

(lambda: (x:=__import__('importlib').import_module('zipfile.ZipFile')('temp.zip','w')(x.write(item) for item in __import__('os').listdir())))()``` heres my full code if ur interested

[rose schooner]

one what does that bytecode signify

[unkempt rock]

__import__ loads the attribute instead of importing from which is why its not working

[rose schooner]

two how?

>>> __import__('zipfile').ZipFile('temp.zip','w')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python310\lib\zipfile.py", line 1249, in __init__
    self.fp = io.open(file, filemode)
PermissionError: [Errno 13] Permission denied: 'temp.zip'
``` this indicates it works properly (`temp.zip` doesn't exist in my pc so that's probably the cause of the `PermissionError` which we ignore since it's not related to the problem)

[unkempt rock]

then why tf is mine not working

[feral island]

importlib.import_module('zipfile.ZipFile') doesn't make sense, ZipFile is a class and not a module

[unkempt rock]

OH

[rose schooner]

also can you call a ZipFile object?

[unkempt rock]

yea ik that was old code i pasted the wrong one

[rose schooner]

it doesn't seem to be related to __import__

[unkempt rock]

__import__('zipfile').ZipFile('temp.zip','w') this is what i have

[rose schooner]

ok well that works

[unkempt rock]

not for me

[rose schooner]

problem is you can't call a ZipFile instance

[unkempt rock]

i still dont get how i can fix it

[rose schooner]

don't call it

use it however it's supposed to be used

like a with statement or something

[unkempt rock]

how do i fit that in the lambda

[feral island]

why do you want to

[unkempt rock]

how do i fit that in the lambda

[rose schooner]

you can't do that simply

[feral island]

you could create a code object dynamically with bytecode for a with statement

(don't)

[rose schooner]

(longer answer: it's kinda complex don't try) ||(i tried doing it for different code and succeeded but don't do that)||

[pliant tusk]

or with contextlib.ContextDecorator. I once used that to implement try/except with only lambdas

[radiant garden]

manually call __enter__ and __exit__

[feral island]

that doesn't get you try: behavior

[radiant garden]

I know

[west tinsel]

Planning on going through the following, but does anyone have references the internal semantics of match? Also, does anyone know which file the implementation is in?

https://peps.python.org/pep-0622/
https://peps.python.org/pep-0636/

[fringe hedge]

@west tinsel https://github.com/python/cpython/commit/145bf269df

[west tinsel]

Thanks! I'll go through the commit alongside the peps

😐

[fringe hedge]

yeah it's a beast. consider cloning the repo and going through it using a tool (in my case, I can pipe a diff to vim and use code folding, for example)

[west tinsel]

Oh nice, I'll see if I can get that setup

[fringe hedge]

though I'm curious why you want to read the implementation

[west tinsel]

I think it's mostly out of interest. I'm pretty sure most of what I need to know will be available in the peps. I'm weakly anticipating that there might be edge cases though where side-effects could possibly happen, for example when calling a dunder method whose definition uses code that causes side-effects

[feral island]

I think it's scattered, a lot of it will live in the eval loop in ceval.c

[elder blade]

The majority really is in the autogenerated parser.c file

[prime estuary]

For pattern matching, actually the PEPs and language docs uniquely mention that code isn't supposed to rely on the side effects - reserving the right to have the implementation cache info for efficiency.

The language docs specify which magic methods are used: https://docs.python.org/3/reference/compound_stmts.html#the-match-statement

[rose schooner]

there are at least 4 files to be modified for the implementation of a grammar rule: Grammar/python.gram, Parser/Python.asdl, Python/ast.c, and Python/compile.c

then most of the grammar rules also need to be in Python/symtable.c and Python/ceval.c

and some also modify Python/ast_opt.c

most new operators then need to add implementations to Objects/*object.c

[marble lark]

Hello

One of the mods told me to come here so

why does this work ```py
from ctypes import c_void_p

class Duck:
def str(self):
return "Quack"
repr = str

c_void_p.from_address(id(True)+8).value = id(Duck)

But not this ```py
c_void_p.from_address(id(True)+8).value = id(False)

[peak spoke]

False is not a type

[marble lark]

Fuck

[marble lark]

Hello bird person that example was based of something of yours

Could you explain how yours worked

[feral island]

does that set the ob_type field on the True object to Duck?

[marble lark]

I have no clue

[feral island]

yeah looks like it

[marble lark]

Still on the journey on understanding why adding 8 to the id of True works

[feral island]

so this is how Python objects are defined internally ```struct _object {
Py_ssize_t ob_refcnt;
PyTypeObject *ob_type;
};

[marble lark]

As i have been told, yes

[feral island]

that plus some other stuff specific to the type

id(True) points to the beginning of that struct for True

so if you add 8 to it, you get to the ob_type pointer, which goes to the type object

[marble lark]

Why specifically 8

[feral island]

so for True, it points to the bool type object

[marble lark]

And how did you know it was 8

That's the most confusing part for me

[feral island]

on 64-bit systems (most of them) Py_ssize_t is 8 bytes

[marble lark]

Getting the number to add

[feral island]

on 32-bit systems you'd need to add 4 instead

[marble lark]

Fuck. Thanks

[marble lark]

How about numbers

c_int.from_address(id(69)+int.__basicsize__).value

[feral island]

https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_basicsize

ints I think are variable-length. their layout is a basic header (refcount+ob_type+size field), followed by a variable number of "digits"

that's needed because Python ints are unbounded

so I think what you posted will just get the value of the first digit

[marble lark]

But why does adding 24 work but not -232

On my phone

Adding both 24 and -232 gives the same value

Which is 4

But i can't actually change the 4 to something else like 5

24 works but -232 doesn't

[feral island]

you mean taking id(24)?

[rose schooner]

he means id(x)+24 and id(x)-232 i think

where'd you get the 232 from

[marble lark]

No

[feral island]

-232 would just go into whatever random memory is before the object, could be anything

[marble lark]

No i add + -232

Well same thing anyways

[rose schooner]

it's random memory

[marble lark]

God why does cpython implementation so damn annoying to learn

[rose schooner]

here's a map of all the offsets on a 64-bit architecture ```c
(PyLongObject){
.ob_base = (PyVarObject){
.ob_base = (PyObject){
.ob_refcnt = ..., // id(x)
.ob_type = &PyLong_Type // id(x)+8, &PyLong_Type aka <class 'int'>
},
.ob_size = 1 // id(x)+16 = id(x)+object.basicsize
},
.ob_digit = {x} // id(x)+24 = id(x)+int.basicsize
}

this kind of structure only applies when .ob_base.ob_size is 1

[marble lark]

Wtf

[feral island]

it's similar with a bigger ob_size, right? Just that the first digit will only be the value mod 2**31

[rose schooner]

first digit would be value mod 2**30

[feral island]

and for negative numbers the sign of ob_size just gets flipped

[feral island]

>>> c_int.from_address(id(x)+int.__basicsize__).value
0
>>> x = 2**31 + 1
>>> c_int.from_address(id(x)+int.__basicsize__).value
1

though I'm on 3.11, maybe this got changed? I also thought it would be 2**30

[rose schooner]

2**31 % 2**30 is 0

[marble lark]

Why do make myself have to understand this and not just accept that adding that does this

[feral island]

ah right

[rose schooner]

!e ```py
from ctypes import c_uint, c_ssize_t, py_object
print(f"""
.ob_base.ob_base.ob_refcnt = {c_ssize_t.from_address(id(4)).value}
.ob_base.ob_base.ob_type = {py_object.from_address(id(4)+8).value}
.ob_base.ob_size = {c_ssize_t.from_address(id(4)+16).value}
.ob_digit[0] = {c_uint.from_address(id(4)+24).value}
""", end='')

[fallen slate]

@rose schooner :white_check_mark: Your eval job has completed with return code 0.

001 | .ob_base.ob_base.ob_refcnt = 56
002 | .ob_base.ob_base.ob_type = <class 'int'>
003 | .ob_base.ob_size = 1
004 | .ob_digit[0] = 4

[marble lark]

How do you add a ob_type based on what cereal did

[feral island]

add? you mean how to get the function for adding?

[marble lark]

Like you said it's refcount+ob_type+size

How do you add 56 + <class 'int'> + 1

Unless int has a default value

[feral island]

I mean adding their sizes, not their values

[marble lark]

So 24

[feral island]

8 bytes for refcount, 8 bytes for the ob_type pointer, 8 for the size

(on most systems you're likely to encounter)

[marble lark]

Why is it 56

On what cereal showed

[feral island]

the refcount?

[marble lark]

Yeah

And 1 for the ob_size

[feral island]

I guess there are 56 references to that number

ob_size is 1 because it's one digit in base 2**30

[marble lark]

Where

[feral island]

somewhere in the program

[marble lark]

!e ```py
from ctypes import c_ssize_t
print(c_ssize_t.from_address(id(4)).value)

[fallen slate]

@marble lark :white_check_mark: Your eval job has completed with return code 0.

50

[marble lark]

On my phone it's 64

[feral island]

sure, depends on what you imported and what else you did in that Python session

[marble lark]

So at all times there's multiple references to numbers in the program even without more numbers?

[marble lark]

I just did the same code

Without adding anything

Hello hsp

[native flame]

yes, there's likely references to the number 4 internally too

[marble lark]

I always see you when it's something about internals

[native flame]

over 50 of them evidently

[marble lark]

Hmm so mine is over 63

[marble lark]

But by default it's just 8

Right

[native flame]

you're confusing two things

[marble lark]

I get confused on alot of things

[native flame]

8 is the number of bytes you need to store the refcount
56 is the value of this refcount, and those 8 bytes are being used to store this 56

those 8 bytes are capable of representing any of 2^64 values

[marble lark]

Hmm

So ints byte size is 8

Internally?

[marble lark]

And all of those bytes add up to python int byte size which is 24?

[native flame]

you use up 8 bytes for storing the refcount
you use another 8 bytes for storing a pointer to the type
then you use another 8 bytes for storing the size of this python int

which adds up to 24 bytes

after these 24 bytes you have the array representing the digits

the actual value of the python int

[marble lark]

Yeah

I was actually right for once

[marble lark]

Thanks i am slowly understanding this now

[marble lark]

How about booleans

[native flame]

how about them

the first two things are common to all python objects (refcount and type pointer)

[marble lark]

Aren't booleans 32 bytes

Oh wait not that

[marble lark]

!e ```py
print(bool.basicsize)

[fallen slate]

@marble lark :white_check_mark: Your eval job has completed with return code 0.

32

[marble lark]

So every refcount and type pointer byte size is always 8 bytes?

[native flame]

yes

!e which is why the minimum you can get to is 16, from sys import *; print(getsizeof(None))

[fallen slate]

@native flame :white_check_mark: Your eval job has completed with return code 0.

16

[marble lark]

What about bools

So what i'm assuming is that the bytes of bools are 16 bytes

[native flame]

dunno off-hand

[marble lark]

But i'm confused with that you can only add 8

[native flame]

the first 16 bytes are definitely for the refcount and type, idk how the remaining is laid out

[marble lark]

To the id of a bool

Like

!e ```py
from ctypes import c_void_p

print(c_void_p.from_address(id(True)+8).value)

[fallen slate]

@marble lark :white_check_mark: Your eval job has completed with return code 0.

139763916456512

[marble lark]

Like why does 8 work

[marble lark]

If you need 8 to get to the pointer to the type then why do i need to add 24 to the id of an int

Or do you not need 8

Like i'm confused how many members do you need to go over to get the actual thing you want

[native flame]

well, that depends on the thing you want

if you want to overwrite the type pointer then you just need to skip past the refcount which is 8 bytes

[marble lark]

How about ints

So i'm skipping both refcount and the pointer and the size?

Like what is left

[native flame]

yes

[marble lark]

If i'm just skipping all 3 of them

[native flame]

the digits of the int

[marble lark]

Ints have another thing in their struct?

[native flame]

yes, the digits of the int

[marble lark]

Judging by the example cereal gave it looks like a list

If the first index of that list is the digit

What's the second index

The second digit after the original digit?

i'm about to sleep hsp so answer it and i'll read it tomorrow

I'll continue this tomorrow

[native flame]

yes, its an array of the digits

i'm fairly sure digits here doesn't mean the decimal digits, but some other representation in terms of powers of 2

like, 567 isn't necessarily stored as [5, 6, 7]

[feral island]

yeah it's the base-2**30 representation

[boreal umbra]

According to the PyCon talk that just happened about python oddities, extending a list with += on a tuple that it's in is peak python oddity.

But I'm sure that together, we could find something even more odd

[swift imp]

I'd pycon live stream?

[boreal umbra]

The talks get posted as videos after. You have to pay to watch the live streams

[swift imp]

Did the speed improvements get talked about yet?

[feral island]

I'm in that room, hi 🙂

[boreal umbra]

I think that might be a keynote tomorrow

The keynote this morning was about not type hinting bad

Also people keep talking about nedbat, but idk if he's here

[swift imp]

Can you link to the videos

[boreal umbra]

They probably aren't posted yet, but wherever they end up, your guess is good as mine.

[swift imp]

Oh

[abstract badge]

I need help as soon as possible to solve this problem

https://tenor.com/view/cat-kitten-scare-scared-cat-scared-gif-24306961

[elder blade]

This isn't the channel for your type of question; see #❓｜how-to-get-help

[rose schooner]

hmm yeah in 3.11 it's gonna be 999,999,999

[sturdy timber]

Why's that? To allow optimisations somewhere (so you don't need to increment/decrement the refcount)? Not sure that would make any sense though

[elder blade]

It's called immortalized objects

Most of them are immutable, and no GC run or ref counting is necessary on them

[feral island]

I don't think that's in 3.11

[elder blade]

I don't think so either, but I think that's what @rose schooner is referring to. The ref-count on immortalized objects will be very high

[rose schooner]

?

[rose schooner]

so like this ```py

import sys
sys.version
'3.11.0a7 (main, Apr 5 2022, 21:27:39) [MSC v.1929 64 bit (AMD64)]'
from ctypes import c_ssize_t
c_ssize_t.from_address(id(4)).value
1000004110

in 3.10 it's something like this ```py

import sys
sys.version
'3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]'
from ctypes import c_ssize_t
c_ssize_t.from_address(id(4)).value
46

[feral island]

right. not sure what that's for. Immortal objects haven't been accepted into CPython yet

[elder blade]

Huh, if that's not immortalized objects then I have no idea what that is 🤔 Not seen any other PEP or post about modifying ref-counts

[rose schooner]

well there's this now https://github.com/python/cpython/blob/main/Include/internal/pycore_object.h#L17-L26

[fallen slate]

Include/internal/pycore_object.h lines 17 to 26

#define _PyObject_IMMORTAL_INIT(type) \
    { \
        .ob_refcnt = 999999999, \
        .ob_type = type, \
    }
#define _PyVarObject_IMMORTAL_INIT(type, size) \
    { \
        .ob_base = _PyObject_IMMORTAL_INIT(type), \
        .ob_size = size, \
    }```

[elder blade]

https://github.com/python/cpython/commit/bc02eac9d2cb36faffc5027b7ce09e6dd0922a7f

Seems kind of accidental?

[prime estuary]

That commit's just moving it to a different header.

It's to improve startup, instead of manually mallocing all the int objects, string singletons etc macros and code generation are used to define all of them as part of a massive struct. That way it can just be loaded off of disk directly.

[elder blade]

Ah yes you're right, I didn't look far enough. The macros above were first added here, yes: https://github.com/python/cpython/commit/121f1f893a39d0b58d3d2b5597505c154ecaac2a

[dusk comet]

Hi! I think this is a bug: a parenthesis is missing here

https://github.com/python/cpython/blob/main/Lib/logging/__init__.py#L453-L456

[fallen slate]

Lib/logging/__init__.py lines 453 to 456

class StrFormatStyle(PercentStyle):
    default_format = '{message}'
    asctime_format = '{asctime}'
    asctime_search = '{asctime'```

[peak spoke]

it can't have the closing brace or the search wouldn't find fields with format specifiers

[dusk comet]

https://github.com/python/cpython/blob/main/Lib/logging/__init__.py#L487-L490

[fallen slate]

Lib/logging/__init__.py lines 487 to 490

class StringTemplateStyle(PercentStyle):
    default_format = '${message}'
    asctime_format = '${asctime}'
    asctime_search = '${asctime}'```

[dusk comet]

it's confusing 😄

[peak spoke]

You can do "{asctime:10}" if you're using the format style, don't think template strings allow anything like that so the brace can be there

[marble lark]

Oh

[dusk comet]

@summer lichen

crosspost

[verbal escarp]

could callable be an alias for collections.abc.Callable?

[elder blade]

I don't think it should be

callable is a "helper function" that checks if something is callable

That's different from an ABC?

[verbal escarp]

isinstance(thing, Callable)?

[elder blade]

Yeah, I get that (and prefer it) but we need to be backwards compatible with if callable(thing)

[verbal escarp]

we could have it as class and have it backwards compatible with a little bit of cheat

then you could annotate properly with callable without from collections.abc import Callable and still be able to do if callable(thing):

[dusk comet]

def callable(x: object) -> TypeGuard[Callable]: ...

[feral island]

that's problematic because TypeGuard narrows strictly to that type

but we do have something like that in typeshed

[dusk comet]

T_co = TypeVar('T_co', covariant=True, bound=Callable)

def callable(x: object) -> TypeGuard[T_co]: ...
``` should this work?

[feral island]

no, because there's nothing to restrict the TypeVar

TypeGuard just isn't powerful enough

we may get a similar more powerful feature in the future

[grave jolt]

well, you can't really check if something is a Callable[[int], str] at runtime

that requires solving the halting problem more or less

[feral island]

the use case is if you want to narrow say Callable[[int], str] | None with a callable() check

[grave jolt]

ah

well in that case you can use is None or another "opposite" check

[feral island]

sure, but that's not always easy

[grave jolt]

hm... maybe we should have stuff from collections as protocols?

oh wait

they work as protocols already, right?

[feral island]

they work as ABCs

[grave jolt]

so you can do isinstance(x, Callable)?

[feral island]

only the really simple ones are protocols

but yes that works

[elder blade]

When is the feature freeze of Python 3.11? Where would I look up this information?

[sturdy timber]

The release schedule?

!pep 664

[fallen slate]

**PEP 664 - Python 3.11 Release Schedule**

Link

Status

Draft

Python-Version

3.11

Created

12-Jul-2021

Type

Informational

[feral island]

next week

[elder blade]

😔

[sturdy timber]

Anything you were hoping to get in before the feature freeze haha?

[elder blade]

I am so freaking frustrated bending my back backwards for Python compatibility across versions; I have an adrenaline rush to write a PEP to remove all type checks across typing.py

[feral island]

we removed a big one already 🙂

if there's specific ones you don't like, feel free to submit a PR

[elder blade]

I've had it easier writing compatible JavaScript and CSS 😩

[elder blade]

I don't enough about the typing code to know which one is causing me the most issues, I mostly want to remove all of them 😅

[west tinsel]

After beta freeze, which branch do feature additions go to?

[feral island]

I believe we create a 3.11 branch at that point, and then features go into main (=3.12)

[west tinsel]

Oh, makes sense!

[feral island]

documented at https://devguide.python.org/devcycle/#in-development-main-branch

[tawdry venture]

j

[boreal umbra]

I don't like that dict keys, values, and items are methods when they don't actually involve the creation of a new object. My understanding is that this is the case for backwards compatibility. Would anything potentially break if they were made to be attributes that return themselves when called?

[peak spoke]

what do you mean? They do create the view wrapper

[boreal umbra]

Oh

But if you call dict.items (for example) more than once on the same dict, aren't the views that you get the same object?

[peak spoke]

afaik no, but they both refer to the same dict underneath

[boreal umbra]

!e

d = {}
print(d.items() is d.items())

[fallen slate]

@boreal umbra :white_check_mark: Your eval job has completed with return code 0.

False

[boreal umbra]

And this concludes my hopes and dreams wrt this idea.

[fringe hedge]

@boreal umbra they can't be the same object because then you wouldn't be able to iterate through them independently

[boreal umbra]

Why not?

[fringe hedge]

because... the object keeps track of how far along you are in iteration

[feral island]

no, that's a separate object

[fringe hedge]

right, I'm saying it must be a separate object to do this

[feral island]

the iterator is separate from the items object

[fringe hedge]

oh, is it?

[feral island]

In [2]: it = d.items()

In [3]: iter1 = iter(it)

In [4]: iter2 = iter(it)

In [5]: next(iter1)
Out[5]: (1, 2)

In [6]: next(iter1)
Out[6]: (3, 4)

In [7]: next(iter2)
Out[7]: (1, 2)

[boreal umbra]

!e

d = {}
print(type( iter(d.items())))

[fallen slate]

@boreal umbra :white_check_mark: Your eval job has completed with return code 0.

<class 'dict_itemiterator'>

[boreal umbra]

Also @feral island send me a dm if you get a chance

[west tinsel]

is it worthwhile to ask for an issue to be closed as a duplicate?

Just because there's 5k+ issues already, it might not really make a difference

I've linked the existing issue for anyone who stumbles across it in the future

[rose schooner]

because changing dictionary
i mean it could also be a property but i'm not sure why it isn't

[feral island]

sure, I love closing issues

[feral island]

it likely predates descriptor support, and now there's little reason to change it

[rose schooner]

yeah that's what i thought

[west tinsel]

Here 😂 https://github.com/python/cpython/issues/92025

CVE-2016-8625 was also brought up in the original issue

[feral island]

wow that's quite a saga

but agree it's a dupe

[rose schooner]

while we're talking about dictviews is dictview indexing reasonable or not

[boreal umbra]

I know this is off topic, but I'm required to acknowledge that Jelle and I are next to each other right now.

[naive saddle]

damn.

[rose schooner]

what

[rose schooner]

.

[boreal umbra]

I'm not a fan of specified dict ordering, but I'm not going to say that no one can have view indexing

[west tinsel]

For typeshed, is it ok to add new 3.11 functions?

[prime estuary]

Stuff like that should be added with a sys.version_info guard, which type checkers understand.

[west tinsel]

Thanks!

Is the following ok then?

diff --git a/stdlib/zipfile.pyi b/stdlib/zipfile.pyi
index d5255b15..bf1ebf42 100644
--- a/stdlib/zipfile.pyi
+++ b/stdlib/zipfile.pyi
@@ -222,6 +222,8 @@ class ZipFile:
         ) -> None: ...
     else:
         def writestr(self, zinfo_or_arcname: str | ZipInfo, data: bytes | str, compress_type: int | None = ...) -> None: ...
+    if sys.version_info >= (3, 11):
+        def mkdir(self, zinfo_or_directory: str | ZipInfo, mode: int = ...): ...
 
 class PyZipFile(ZipFile):
     def __init__(

[west tinsel]

And what's happening with this change? https://github.com/python/cpython/commit/81c72044a181dbbfbf689d7a977d0d99090f26a8#diff-e4fd8b8ee6a147f86c0719ff122aca6dfca36edbd4812c87892698b3b72e40a1R2037-R2043

These lines specifically

@@ -2050,9 +2034,13 @@ static PyObject *
 builtin_input_impl(PyObject *module, PyObject *prompt)
 /*[clinic end generated code: output=83db5a191e7a0d60 input=5e8bb70c2908fe3c]*/
 {
-    PyObject *fin = _PySys_GetObjectId(&PyId_stdin);
-    PyObject *fout = _PySys_GetObjectId(&PyId_stdout);
-    PyObject *ferr = _PySys_GetObjectId(&PyId_stderr);
+    PyThreadState *tstate = _PyThreadState_GET();
+    PyObject *fin = _PySys_GetAttr(
+        tstate, &_Py_ID(stdin));
+    PyObject *fout = _PySys_GetAttr(
+        tstate, &_Py_ID(stdout));
+    PyObject *ferr = _PySys_GetAttr(
+        tstate, &_Py_ID(stderr));
     PyObject *tmp;
     long fd;
     int tty;

[spark magnet]

Anyone want to help with a coverage.py issue that involves CPython internals? I need to get this PR over the line: https://github.com/nedbat/coveragepy/pull/1353

[feral island]

yes, I've added a lot already

[feral island]

looks like a subinterpreters thing, moving stuff into state structs

[alpine tide]

hey, when would you guys say someone is not a beginner at coding anymore

[boreal umbra]

@alpine tide that's probably a question for #pedagogy, but my answer is that it's when you don't have to look up the syntax of the language to solve a problem. An intermediate will still be unable to solve some problems because they just don't know how to solve it in general. Even advanced programmers refer to documentation all the time.

[misty quarry]

Anyone here got an idea as to how the Python setup checks if I already have the version installed or not?

[spark magnet]

it's pip that does that.

[misty quarry]

Ah, so, I, uhhh did a small mistake. I manually deleted the Python folder and now when I try to reinstall it, the setup shows I still have the version

Even when I don't have it anymore

[rose schooner]

ok so when you open the python setup thingy is there a "repair" option or not

[misty quarry]

Yes, but it also errors out

[west tinsel]

So, in 3.11, dataclass has a new weakred_slot parameter, for typeshed how does this work and how would it be added?

if sys.version_info >= (3, 8):
    # cls argument is now positional-only
    @overload    
    def dataclass(__cls: type[_T]) -> type[_T]: ...
    @overload
    def dataclass(__cls: None) -> Callable[[type[_T]], type[_T]]: ...
    
else:
    @overload
    def dataclass(_cls: type[_T]) -> type[_T]: ...
    @overload
    def dataclass(_cls: None) -> Callable[[type[_T]], type[_T]]: ...
        
if sys.version_info >= (3, 10):
    @overload   
    def dataclass(
        *,    
        init: bool = ...,   
        repr: bool = ...,
        eq: bool = ...,  
        order: bool = ...,
        unsafe_hash: bool = ...,
        frozen: bool = ..., 
        match_args: bool = ...,
        kw_only: bool = ...,
        slots: bool = ...,
    ) -> Callable[[type[_T]], type[_T]]: ...
          
else:
    @overload
    def dataclass(
        *, init: bool = ..., repr: bool = ..., eq: bool = ..., order: bool = ..., unsafe_hash: bool = ..., frozen: bool = ...
    ) -> Callable[[type[_T]], type[_T]]: ...

How come there are multiple definitions inside a branch? And would 3.7 then have 3 definitions?

[feral island]

We'd need a new branch for 3.11

[feral island]

Those are conditional overloads, a recent mypy feature

[west tinsel]

Oh, I read the mypy documentation and this now makes sense

I don't think I understand the else clauses though

And 3.10 goes into multiple branches, I don't understand that either

[feral island]

The first if-else group is for writing just @dataclass, or @dataclass(None) if you really want to

That's different between 3.8 and earlier because the arg became positional-only

The second group is for @dataclass(some=arguments)

And that changed in 3.10 because slots= was added, and we'll change it again for 3.11 because of weakref_slots

[west tinsel]

Oh!

This makes sense now, thank you!

Is this all I add?

if sys.version_info >= (3, 11):
    @overload
    def dataclass(
        *,
        init: bool = ...,
        repr: bool = ...,
        eq: bool = ...,
        order: bool = ...,
        unsafe_hash: bool = ...,
        frozen: bool = ...,
        match_args: bool = ...,
        kw_only: bool = ...,
        slots: bool = ...,
        weakref_slot: bool = ...,
    ) -> Callable[[type[_T]], type[_T]]: ...   

[feral island]

I think so

[static bluff]

What is __itemsize__?

[jolly zealot]

100%

[feral island]

https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_itemsize

[opal canopy]

i need help with my code

[static bluff]

Oh goodness

XD I swear, I did a whole a bunch of googling before I came here and asked

[wide shuttle]

I can imagine. I just tried to google "__itemsize__" to see what you'd get, but you have to dig deep. There are some third-party pages that give some clues as to what it does, but you won't easily find the page linked (as it doesn't mention __itemsize__).

[sturdy timber]

!pep 690

[fallen slate]

**PEP 690 - Lazy Imports**

Link

Status

Draft

Python-Version

3.12

Created

29-Apr-2022

Type

Standards Track

[sturdy timber]

Thoughts on this pep? It feels like a pretty major change, I'm not sure how I feel about it. I did recently find myself annoyed at how slow something like pip --help is, so I can see the argument for it

[feral island]

I like it, it will also make some type checking use cases easier. (And personal note: First time I'm in the same room with the people submitting a PEP, which is an interesting experience.)

[misty quarry]

Since the Python setup use pip to detect Python, is there a way to force reinstall Python? I, uhhh, manually deleted the Python folder and deleted the partition it was in. So now the setup doesn't work for me now

[rose schooner]

it uses pip to detect python?

[misty quarry]

@rose schooner

[rose schooner]

ok

[rose schooner]

pin for future reference?

https://docs.python.org/3/c-api/typeobj.html

[boreal umbra]

the PEP addresses all the issues that immediately came to mind. if it's accepted, I think it should be made the default behavior with the -O flag

[honest perch]

how would it affect the antigravity module/easter egg? Since it has obvious side effects ||opening a webpage||, would this delay them until one of the module's attributes is accessed or something?

[dusk comet]

!e also __hello__ has side-effect

import __hello__

[fallen slate]

@dusk comet :white_check_mark: Your eval job has completed with return code 0.

Hello world!

[sturdy timber]

and import this haha.

It could be sort of nice if a module was able to mark itself as one that should not be lazily imported, but I don't know how that could be done without removing a bunch of the performance benefits.

[dusk comet]

register eager modules in some file, load this file on startup, use .set_eager_imports

.set_eager_import_static to write something to file containing list of eager modules

[raven ridge]

Or only allow it for packages, not modules, and handle it with a flag file inside the package - like the py.typed file that indicates whether or not a package provides type information

in practice that's not much of a restriction, since any module can be trivially turned into a package by just renaming foo.py to foo/__init__.py, at which point such flag files could be dropped into foo/

to me, this seems like an absolutely massive change to the semantics of the language. Possibly the biggest change since adding native coroutines. I think its utility is inherently limited if it requires a special command line flag to enable it - most people won't pass it (or even know about it), so any command line option would necessarily only be useful for power users. I think the environment variable is an outright bad idea, since environment variables propagate through the process tree, and this change isn't safe in general, so allowing it to happen for all Python runs seems unwise

[raven ridge]

I think that doesn't really work, though, thinking about it. Say an application does import a, and module a does import b. Say that module b has marked itself as "don't lazily import me". Well, tough, it gets lazily imported anyway, because a didn't mark itself as "don't lazily import me", so a gets lazily imported, and b gets eagerly imported only when a is lazily imported.

I think the only way this makes any sense to add is if the goal is for it to one day become the default, but I think that would be a huge breaking change several years down the line, or possibly for Python 4...

[sturdy timber]

Good point

[raven ridge]

I also think it's pretty weird, that, with this mode enabled, c = a + b can raise an ImportError. Maybe that's not a big deal, but it does seem weird.

[feral island]

it already can 🙂

though it's true that it makes it a lot more likely

[raven ridge]

I mean, sure, if __add__ calls __import__, but this could mean it happens even if a and b are (documented to be) int, or something.

[misty quarry]

anyone here knows any solution to my problem?

[raven ridge]

it also means that the (actual, non-lazy) import can run in an arbitrary thread, while currently nearly all module importing is done by the main thread. I'm not sure whether that's a big deal or not, but it does seem like a pretty significant change...

[feral island]

Pretty sure the import system already has thread-safety built in, but you're right that lazy imports make it much more likely this will happen in practice

[raven ridge]

yeah, I'm not worried about the interpreter so much - it can handle this, of course. I'm more worried about it from the PoV of application and library developers. Perhaps there's some race condition that occurs when module a is being imported by one thread while module b is being used by another thread given that both of them share a common module c, or something

and granted, if so, that was always a bug and always needed a lock. But this change will surface things like that, by making it much more likely that the import happens in a non-main thread.

[feral island]

@raven ridge I'd encourage you to bring up your concerns at https://discuss.python.org/t/pep-690-lazy-imports/15474/25

[raven ridge]

I was already talking to Pablo and Lukasz about it the other day, but... well, perhaps I should collect my thoughts and share them there...

[feral island]

oh, you were at PyCon? sorry I missed you!

[raven ridge]

Ditto, haha

[boreal umbra]

@feral island @raven ridge we were all near each other at the Mexican restaurant briefly

[feral island]

ah, during the Bloomberg invasion? 😄

[raven ridge]

yeah, I think I posed for a photo with Stel next to your table without recognizing you, @feral island 🙂

[elder blade]

Now that you mentioned it, we have to see this legendary photo 💯

[paper echo]

this channel is for discussion about the design of the python language itself. it sounds like you should open a help channel: #❓｜how-to-get-help

[misty quarry]

Ah okay my bad

[raven ridge]

Ah, and yeah - Pablo and I are teammates at Bloomberg, I'm his co-maintainer for Memray

[feral island]

nice, I need to try out memray 🙂

[raven ridge]

please do! We're very interested in bug reports, or usability case studies, or whatever. If you find something wrong or un-ergonomic, let us know.

[verbal escarp]

i don't see the core problem, actually. for big projects using an IDE or an editor with proper extensions like vscode is a given, which usually give you a heads up for unused imports

so changing the core logic of imports for a non-existent problem seems kind of.. weird, considering how many proposals are rejected due to maintenance alone

[feral island]

That only tells you if nothing in the file is using the import. But often many code paths never get taken at runtime

[verbal escarp]

what about optional imports?

[sturdy timber]

If you mean module level optional imports (e.g. if guarded) then that still requires you to know whether you'll use the module, which you often won't know for sure.

[verbal escarp]

import within the function/feature-entry-point?

[feral island]

quoting the PEP: "In an effort to improve startup time, some large Python CLIs tools make imports lazy by manually placing imports inline into functions to delay imports of expensive subsystems. This manual approach is labor-intensive and fragile; one misplaced import or refactor can easily undo painstaking optimization work."

[sturdy timber]

If you're using a module multiple times you either have to know where you'll use it first or copy paste the import to wherever you use it

[verbal escarp]

well, it would probably be guarded by a # don't move this or somesuch

@feral island

[sturdy timber]

It's not people moving inlined imports to the top because they don't know why they're there, it's because it's hard to get placing them right in the first place

[verbal escarp]

if you defer the import process up until the first use, it will become completely unpredictable

[sturdy timber]

If there are side effects from the import, possibly, yeah

[verbal escarp]

yeah, thinking of side-effects like setting up a database or some other file ops

couldn't you flag deferred imports with async import?

[radiant garden]

not with async definitely not

if python had a lazy keyword maybe

[feral island]

https://peps.python.org/pep-0690/#explicit-syntax-for-lazy-imports

[verbal escarp]

well formulated, but i disagree with the conclusion and i think the prize you'd have to pay is potential confusion about import order, non-executed imports because someone deleted a snippet..

you'd have to keep in mind that refactoring a local function could potentially change the execution order of the whole program

[peak spoke]

the vast majority of projects won't have side effects from imports, and can greatly benefit from the feature

[sturdy timber]

It is proposed as an opt in feature.

[peak spoke]

pretty much anything with user interactions currently either has to deal with the mess of having imports all over the code base instead of on top of modules, or having to load the whole application at startup

[verbal escarp]

and when a project does have side-effects it would make for a really fun bug hunt!

[peak spoke]

With it being opt in I really don't see how that could be a big of an issue

If you turn it on, you are working with the feature in mind

[verbal escarp]

as i understood, the idea was to hook it up into the optimizing flag

which means that when you think you're ready for production, the real fun only begins

[peak spoke]

It's a separate flag

[radiant garden]

That's not mentioned in the PEP.

[verbal escarp]

https://peps.python.org/pep-0690/#specification

[sturdy timber]

Lazy imports are opt-in, and globally enabled via a new -L flag to the Python interpreter, or a PYTHONLAZYIMPORTS environment variable

[radiant garden]

-O are different flags

[verbal escarp]

right, but who would expect different runtime behaviour with a flag that looks as if it's just an optimization?

and how to test against it?

[peak spoke]

someone who reads what flags do instead of using them arbitrarily

[radiant garden]

if a "lazy imports" looks like "speedier code" and you're not ready for it to be "lazy imports" then idk what to tell you as a programmer

[lusty scroll]

that exists ??

[sturdy timber]

Yeah lazy sounds like it's gonna make it slower lol

[verbal escarp]

it's a pep

[lusty scroll]

what if a module is imported for type hints .. I wonder if that counts as being used

[verbal escarp]

if a "lazy import" flag says THIS MIGHT CHANGE YOUR RUNTIME BEHAVIOUR NO GUARANTEES i probably would think thrice before enabling it

[sturdy timber]

What else would it do?

[lusty scroll]

I can't help wondering if this is a band-aid for over-eager imports

[verbal escarp]

i definitely don't like it as flag, it hides problems down the road and complicates testing

[lusty scroll]

it does kind of make importing in the scope something is needed seem a bit more justified

[verbal escarp]

what if it's a context manager, but the other way round?

with lazyness: import foobar

[lusty scroll]

I dont guess it would be made the default .. at least, probably not until such issues are sorted out

[verbal escarp]

explicit is better than implicit

[lusty scroll]

it says dynamic imports are never lazy so your use case shouldn't be affected right

how does it know if a module is ok to be imported lazily.. what happens when a name from the module gets referenced, etc.. seems likee it could add some significant complexities?

[sturdy timber]

It would load all modules lazily unless they're imported in a way that makes the import eager

[lusty scroll]

defer the execution of imported modules until the moment when an imported object is used
also wondering how does it know what objects come from which module without loading it

[lusty scroll]

maybe just an unresolved name would be enough to force eager imports

[sturdy timber]

It doesn't need to. You do module.abc, it sees module so knows that needs to be loaded. Star imports are always eagerly loaded

[lusty scroll]

that's true .. good point

so any from x import * is going to probably make every import eager up to that point then

[sturdy timber]

It would eagerly load x, but that wouldn't effect anything else (not even necessarily the things that x imports!)

Actually

If you do from x import * and x has import y, I'd assume x and y would be loaded

Although that probably depends on the implementation

[raven ridge]

the vast majority of projects won't have side effects from imports, and can greatly benefit from the feature
I'm quite skeptical of that claim - did the PEP authors devote any time to analyzing that and trying to see if it's true or false?

[feral island]

I presume it's based on their experience deploying the feature at Instagram

[raven ridge]

That doesn't seem to be the proposal, by my reading of the PEP. x would get imported greedily, y would be imported lazily, and there would be a reference to the lazy y placeholder in both the globals of __main__ and x

[raven ridge]

If so, that hardly seems like a representative sample...

[lusty scroll]

my experience is a great many packages have side effects

[raven ridge]

my intuition would have been that the overwhelming majority of applications do something with side effects at import time. I'd have guessed >90% if someone had asked me to put a number on it...

[sturdy timber]

I'm struggling to think of ways that side effect would be required to be executed at the start of the program, rather than just before you use that module

[lusty scroll]

maybe this would promote better module hygiene?

like instead of having an if __name__ == "__main__" they could have a __main__.py file instead

[raven ridge]

perhaps it checks an environment variable that the program might go on to change, or checks for the existence of a file that might later be created or removed

perhaps it registers a codec - that's the most insidious case I can think of off the top of my head

https://docs.python.org/3/library/codecs.html#codecs.register is often called by modules at import time - even the standard library itself calls it at import time when encodings is imported (though granted that occurs prior to __main__ executing). If that call is delayed, a call to "".encode("custom_codec") or b"".decode("custom_codec") can fail, because the codec wasn't registered yet

https://github.com/VDOMBoxGroup/vdomserver1.3/blob/b9df6bcbe3b78dbc3b48ab074150e1a0d31edab4/src/utils/codecs/__init__.py is an example of some code in the wild that's doing this.

https://github.com/maxpmaxp/pdfreader/blob/d94fa46cac1325f911818b7cb365aa91e07e1182/pdfreader/__init__.py#L5 is another...

[fallen slate]

pdfreader/__init__.py line 5

register_pdf_encodings()```

[raven ridge]

https://github.com/robertopauletto/pymotw3rel/blob/3c19059e33d1d434fc4cba0adad0be6f26e23e2a/dumpscripts/codecs_register.py#L17-L18

[fallen slate]

dumpscripts/codecs_register.py lines 17 to 18

codecs.register(search1)
codecs.register(search2)```

[raven ridge]

https://github.com/epigos/faust-demo/blob/f1ee44eca99fc01721c50b090cd42680b2eb603c/workers/avro_codecs/avro.py#L34-L36 - there's a pretty decent number of these

[fallen slate]

workers/avro_codecs/avro.py lines 34 to 36

codecs.register("avro_video", avro_video_codec())
codecs.register("avro_extraction_audit", avro_extraction_audit_codec())
codecs.register("avro_category_tag_audit", avro_category_tag_audit_codec())```

[raven ridge]

granted, this is a bit of a weird case - it's a module being imported explicitly because it has a side effect.

What I think is a lot more concerning are cases where modules have side effects that are less well known and well understood.

[raven ridge]

I bet it'd break pip: https://github.com/pypa/pip/blob/776238af48edd96719d4b32009067b734fd1b12b/src/pip/_vendor/six.py#L997-L998

[fallen slate]

src/pip/_vendor/six.py lines 997 to 998

# Finally, add the importer to the meta path import hook.
sys.meta_path.append(_importer)```

[raven ridge]

it'd probably break everything that leverages sys.meta_path, since they all rely on the meta path being set before some later import is performed, and lazy imports will make the relative order of those two things happening unpredictable

[raven ridge]

Notably, Instagram has a monorepo, which might make it a lot easier to handle libraries that perform work with side effects at import time...

and lazy imports are infectious - to know whether it's safe to lazily import a module, you need to know whether it's safe to import every module that module imports, and every module each of those modules import... And you need to re-evaluate that analysis every time one of your dependencies changes...

[icy lodge]

i learn only python so job is confirm for python progrmmaers

[west tinsel]

I'd be really concerned if it was opt-out (I initially thought this) I think it's less of a problem as an opt-in flag

I maintain a flask app that has code that's loaded outside of functions that I'm pretty sure produces side effects, how would the pep handle that?

I believe I have __main__ importing app.function which requires app.app which is defined globally using Flask, which I think produces side effects

I don't know if side effects are relevant here, but the code in the global context needs to run correctly

[elder blade]

I think there should be both. Honestly, a command line argument seems like it'll be used less than it could be if the option was made explicitly syntatic

[elder blade]

Well, someone won't enable it in production without first testing with it locally? If there's issues, they would have noticed it?

[white nexus]

lol you make some mighty assumptions

[sturdy timber]

Surely there would need to be some sort of way to enable it at runtime. For example with a CLI app it would be strange if a user needed to invoke it with the flag.

[white nexus]

that's what im thinking about

there needs to 100% be a way a package can declare "do not lazy import me"

tbh

that is actually entirely doable

although... hm

[sturdy timber]

Well... That's not what I'm talking about and also not necessarily easily doable :P

[raven ridge]

that is completely undoable

[white nexus]

why?

[raven ridge]

because in order for it to work, it would need to be recursive - if module C declares that it can't be lazily imported and module B imports module C, then module B can't be lazily imported either, even though it hasn't declared so. So when module A imports module B, there's nothing to tell it that module B can't be imported lazily

[white nexus]

ah

[raven ridge]

that is, before importing module B, you don't even know that it imports module C, so there's no way to know that B needs to be imported eagerly.

[white nexus]

didn't think of that

[raven ridge]

yeah - I wish there were a per-module opt-in or opt-out, but I'm pretty convinced there can't be one...

well, there could maybe be an opt-in at the module level - a flag file in the package or something - but I don't think there can be an opt-out

[white nexus]

ye

i was thinking of the metadata, how a package would store it

[quick night]

!e

class Foo:
    def foo(self): ...
    
f = Foo()
b = f.foo
id_f_foo = id(b)
del b
print(id_f_foo)
print(id(f.foo) == id_f_foo)

[fallen slate]

@quick night :white_check_mark: Your eval job has completed with return code 0.

001 | 140003272060096
002 | True

[quick night]

why is it false for me?

i think its PyCharm, its true in my terminal

[verbal escarp]

there could be, if you're willing to go down the crazy road.

i'm thinking of checking the ast for a certain token, and if it's in the ast, stop compile and execution

[raven ridge]

How does that solve the problem of it needing to be recursive?

[verbal escarp]

it'd be something like a "don't bother me"-lazyness

try to import, then back off

[raven ridge]

A imports B imports C, C has the token, but B doesn't so A's import of B is lazy, so C gets lazily imported even though it had the token - right?

[radiant garden]

that's addressed in the pep:
from __future__ import lazy_imports

[verbal escarp]

yeah, recursion is a bitch

[raven ridge]

The id() of an object can be reused once that object is destroyed. It's only guaranteed to be unique while the object is alive

[quick night]

ah ok

[verbal escarp]

wait, why would C be imported if it wasn't used by anything in A or B?

[white nexus]

what pep number is this again

[raven ridge]

Say B uses it.

[verbal escarp]

i'm rather thinking that A -> [B] -> C (with [] denoting lazy) would break the assumption that C would get imported

[raven ridge]

If C doesn't actually get imported until after A has accessed and resolved an attributed of B (causing it to be lazily imported), then C has been lazily imported despite its metadata saying it shouldn't be

[verbal escarp]

yes, that's a problem

which can't be fixed

except.. you'd have to parse everything (whether lazy or not) into an ast, extract the import statements, parse etc. recursively

[white nexus]

so effectively losing lazy imports yay

[verbal escarp]

no, execution would be delayed

[white nexus]

i mean

true

except that wouldn't always work, would it

[raven ridge]

You'd need to do all the module finding and parsing and compiling, which is the slow part that this PEP is trying to avoid

[verbal escarp]

yup

[verbal escarp]

also true

for example everything that fiddles with the import machinery at runtime could break

[white nexus]

ye

i have a import hook that makes any module importable as another

eg import trio would import asyncio, and replace itself with asyncio, so sys.modules['trio'] refers to asyncio

[verbal escarp]

yeah, case for the bin

[white nexus]

(note that this is an example, and I don't actually use it to make trio importable as asyncio, and that's not what this is about)

[verbal escarp]

justuse probably also

oh well ^^

[elder blade]

Wait do you not even get an import error? It checks for the module actually existing right? If it does, then it should be able to do some pre-emptive checking. A __future__ import would be perfect, but it's not an actual future so it doesn't make much sense.

[elder blade]

A dunder would be nice, but Python would need to run the entire file for that...

[raven ridge]

You do not get an import error until you try to use a lazily imported module, it does not check for the module existing.

[elder blade]

I guess it depends on whether you consider C lazily imported. Sure, compared to before C won't be initialized like it was earlier but B isn't even loaded yet so C hasn't directly gotten lazily imported.

[elder blade]

Ah good. I think this is something we can leave to static code analysis then

[raven ridge]

It was indirectly lazily imported, which is still lazy.

It declared that it wanted to actually be imported when a module thar imports it gets imported, and that didn't happen

[elder blade]

Modules can set a dunder, then the editor warns the user if they are lazily imported I guess? Although that won't work out nicely since the editor doesn't know if they are run with that flag

I'd really like new syntax for this, I like the explicitness. Not sure I like async import though, it doesn't have anything to do with coroutines.

Could re-use forward from the class definition PEP?

[raven ridge]

Modules can't set a dunder for it that the interpreter would respect, though I suppose they could set one that a static analysis tool would recognize

[elder blade]

Yes

[dusk comet]

Also we can flag modules as eager by using some characters in modulename (similar to .pyd filename)
For example:
mymodule.eager.py
mypackage.eager/init.py

[quick snow]

I'm not convinced there would be a need for recursive checks for eagerness flags of modules/packages

If a can be imported lazily, and b can't, and a imports b, why can't we just check for lazy-support of b once we actually import a for real?

[elder blade]

Because previously, if c had print('...') then it would print on startup, with that change it won't

c gets lazily imported because the code that imports it is delayed

[verbal escarp]

magic names? no way 😐

[quick snow]

c is b or a?

Okay, I'll rename: I run a, which imports b, which imports c. c is marked as eager, b isn't. Therefore, at the line import b no side effects happen, neither from c nor from b. But that's okay, because b is fine with being imported lazily. Once b is actually used, we hit import c, which, being marked as eager is run immediately. Nothing inside b happens out of order.

I'd be interested to see an actual example where this would be problematic.

In general, I think the os.stat argument in the "Rejected alternatives" part of the PEP is ridiculous

That's not where the slowness comes from, unless you import ten thousand modules, in which case: just don't do that.

Honestly, it reads a bit like the authors came up with their — admittedly clever — solution first, and didn't seriously consider the alternatives.

[verbal escarp]

and they formulated their text well to make it seem without alternative :p

at least respect for that

hm.. apropos alternative. if i understood correctly, the motivation for this PEP were huge cmdline tools that import everything but only take a certain codepath, yes?

what if the tool delegates to subprocesses instead?

if the tool grows too big and too slow with all the imports, it could be refactored into sub-processes, not just submodules

then you have a single entry point with a unified help and all, but if subsets of features are moved into a new lightweight process it could make things much simpler

[elder blade]

Now that we know that this is possible, I'd love to have this have other applications

[verbal escarp]

from subprocess import run
import sys

print(sys.argv)

if sys.argv[1] == "1":
    run(["python", "test1.py"])

if sys.argv[1] == "2":
    run(["python", "test2.py"])

print("end")

test1.py

import tensorflow
import matplotlib
...

test2.py

import nltk
...

easy as py

clearly separated code paths, no unnecessary imports

and those subprocesses still can import whatever they need and share modules in the same project

[radiant garden]

the overhead of spawning another python interpreter surely dwarfs any performance gains from just-in-time imports

[verbal escarp]

not at all, importing tensorflow takes much longer than just spawning a subprocess

[radiant garden]

how wonderful, pick your poison

[verbal escarp]

considering that many cmdline tools for python actually are wrappers of tools written in other languages, making extensive use of subprocess, i think applying the same logic for stuff written in python isn't that far fetched

[west tinsel]

Are there references for the evaluation of operators?

When a + b is called, is that always a.__add__(b) for example

[feral island]

it's in the language reference, though not necessarily in a very precise way

a + b may also call b.__radd__

[verbal escarp]

isn't there a shortcut for ints in bytecode?

[feral island]

maybe, but that shouldn't affect the semantics

[verbal escarp]

iirc a while ago we had a similar discussion in here about lots of shortcuts in bytecode preventing some optimization

[feral island]

it's generally the opposite, bytecode is easy to change

3.11 bytecode will be almost completely different from 3.10

[verbal escarp]

iirc the discussion was about generalized jit of some kind that only look at the dunders

but i can't recall the details

gist of it was that there were many bytecode optimizations that prevent such general approaches

[boreal umbra]

is it easy to change in the sense that it's easy to modify the code, or easy in the sense that it doesn't break compatibility (since the bytecode is unspecified)

[feral island]

Mostly the latter. I don't think I've ever actually changed the eval loop myself, it's pretty tricky code

[rose schooner]

how is the eval loop "pretty tricky"?

[feral island]

a 5000 line C function that has to deal with signals, tracing, profiling and is extremely performance sensitive

[raven ridge]

It got more complicated in 3.11 - now there's some automatic inlining of Python frames, so that a call from a Python function into a Python function doesn't need a new frame on the C call stack

[rose schooner]

so no more recursion limits 🤔

[raven ridge]

More like recursion limits may sometimes be bypassed

[elder blade]

Tail call optimization?

[boreal umbra]

Guido promised that Python will never have that. But he's not in charge anymore

[verbal escarp]

good thing, if only for that 😉

actually could be a good time to write a PEP for that?

[boreal umbra]

I think it's doubtful that they'd go for it. think about what they'd have to do to implement it while keeping the external behavior the same for things like exception handling.

I don't think Java uses it, for example

[flat gazelle]

Yeah, mangling stack traces is kind of painful to make work with stack based APIs. Also probably breaks at least some of stdlib since the caller frame may no longer be the actual caller.

[pliant tusk]

its difficult to do in python due to how function names can be rebound, so you could have code that looks tail-call optimizable, but isnt. py def foo(a, b): foo = print foo(a, b) and you could argue that looking for assignments is easy, but you could also use the globals() dict directly to make the assignment

[verbal escarp]

never claimed it would be easy ^^

just saying i'd love it

[pliant tusk]

python3 -i tco.py
>>> @tco
... def fact(n, acc=1):
...     if (n < 2):
...         return acc
...     return fact(n - 1, n * acc)
... 
>>> dis.dis(fact)
  3     >>    0 LOAD_FAST                0 (n)
              2 LOAD_CONST               1 (2)
              4 COMPARE_OP               0 (<)
              6 POP_JUMP_IF_FALSE        6 (to 12)

  4           8 LOAD_FAST                1 (acc)
             10 RETURN_VALUE

  5     >>   12 NOP
             14 LOAD_FAST                0 (n)
             16 LOAD_CONST               2 (1)
             18 BINARY_SUBTRACT
             20 LOAD_FAST                0 (n)
             22 LOAD_FAST                1 (acc)
             24 BINARY_MULTIPLY
             26 STORE_FAST               1 (acc)
             28 STORE_FAST               0 (n)
             30 JUMP_ABSOLUTE            0 (to 0)
>>> fact(99)
933262154439441526816992388562667004907159682643816214685929638952175999932299156089414639761565182862536979208272237582511852109168640000000000000000000000
>>> fact(5)
120
>>> def fib(n=1000, a=0, b=1):
...     if n == 0:
...         return a
...     if n == 1:
...         return b
...     return fib(n - 1, b, a + b)
... 
>>> fib()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in fib
  File "<stdin>", line 6, in fib
  File "<stdin>", line 6, in fib
  [Previous line repeated 995 more times]
  File "<stdin>", line 2, in fib
RecursionError: maximum recursion depth exceeded in comparison
>>> fib = tco(fib)
>>> fib()
43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875
>>> ```

i have a very naive implementation written that works on bytecode (def broken in 3.11)

[verbal escarp]

nifty

[rose schooner]

can i have a look at the implementation

i think i can customize it for 3.11.0a7+

oh it's on the pysnippets repo

[pliant tusk]

tbh with how the implementation is, it would be better to instead rewrite it

[raven ridge]

the naive fib() may just work on 3.11

actually, wait, probably not. The naive fact might just work on 3.11, though

[feral island]

I think you still have to do sys.setrecursionlimit()?

IIRC Python-to-Python calls no longer consume C stack, but we still count frames

[raven ridge]

I don't think you do - I think that's only counting C frames, not Python frames

oh. Well, then I guess I'm wrong 🙂

[feral island]

Python 3.11.0a7+ (heads/subprocess-issue38435:3df5b302b5, May  5 2022, 21:07:18) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def fact(n, acc=1):
...     if n < 2:
...             return acc
...     return fact(n -1, n * acc)
... 
>>> fact(1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in fact
  File "<stdin>", line 4, in fact
  File "<stdin>", line 4, in fact
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded
>>> import sys
>>> sys.getrecursionlimit()
1000
>>> sys.setrecursionlimit(100_000)
>>> fact(1000)
402387260<snip>
>>> x= fact(90_000)

[raven ridge]

well, fair enough.

with 3.10 does that segv for you? 🙂

[feral island]

probably, let me try

[raven ridge]

(I assume that's high enough, but just in case)

[elder blade]

Segv? Segmentation fault?!

[raven ridge]

yeah

as in SIGSEGV

[feral island]

yeah because you smash the C stack

[raven ridge]

"segmentation violation"

[elder blade]

OooOh, right. With the modification of the recursion limit

[feral island]

yes, segfault

[raven ridge]

https://github.com/python/cpython/blob/3f61db475692511a9676765e6f9f0bb204306e93/Python/ceval.c#L1778-L1782

[fallen slate]

Python/ceval.c lines 1778 to 1782

start_frame:
    if (_Py_EnterRecursiveCallTstate(tstate, "")) {
        tstate->recursion_remaining--;
        goto exit_unwind;
    }```

[raven ridge]

I think that's where it's counting Python frames in 3.11

so, I stand corrected

[feral island]

there was some talk at the language summit about changing this, I think for Pyston it was inefficient to actually count the frames

[raven ridge]

hm. It's tricky. If you don't count the Python frames, then sys.setrecursionlimit() ceases to be explainable - it counts the number of "C frames", where it's an implementation detail when (or if!) "C frames" get created

[feral island]

yes, @deft horizon had some concerns about keeping Hypothesis stack traces reproducible

[raven ridge]

the stack traces should be able to be maintained either way, as long as the tool is only tracking the Python stack - those PyFrameObject's are just heap objects that can be created and manipulated how ever we need to reflect the truth

tools that inspect the Python stack and the C stack together (like Memray!) are gonna have a bit of a bad time, though

[feral island]

right, I think the concern was about being able to get the exact same traceback on different machines or processes. Not sure I caught all the subtleties though

[raven ridge]

hm. Could be there's a subtlety there I'm not spotting.

Memray is gonna need to leverage https://github.com/python/cpython/blob/3f61db475692511a9676765e6f9f0bb204306e93/Objects/frameobject.c#L1113-L1117 in 3.11 to handle the C stack frame reuse, though

[fallen slate]

Objects/frameobject.c lines 1113 to 1117

int _PyFrame_IsEntryFrame(PyFrameObject *frame)
{
    assert(frame != NULL);
    return frame->f_frame->is_entry;
}```

[rich cradle]

I don't hit the recursion error, weird

[raven ridge]

also on 3.11? Same version?

[feral island]

your initial recursion limit might be different?

[rich cradle]

I take that back, it doesn't work in the standard repl, only ipython

[feral cedar]

ipython sets it at 10000

[raven ridge]

interesting that it works in IPython

[feral cedar]

oh nvm, only 3000

[raven ridge]

I wonder if they're deliberately creating a thread with a large stack and moving the processing to it to avoid having that segfault...

[feral island]

from @feral cedar sounds like they just do sys.setrecursionlimit

[rich cradle]

I was just running the fact(1000) example, so it's probably what psvm said

[raven ridge]

ah, OK.

[rich cradle]

yeah, fact(5000) errors

[raven ridge]

with a recursion limit exceeded?

[rich cradle]

mhm

[feral island]

IPython segfaults too if I set the limit to 100k and try fact(90k)

[raven ridge]

interesting. I'm surprised that the default recursion limit can be safely tripled

though I suppose it's set conservatively by default to account for platform differences

[feral island]

on my system on 3.10 it only segfaults after ~28k frames

[raven ridge]

wow, that's interesting

I wonder if a 32-bit build would have a much lower limit, and it's kept at 1000 across 32/64 for consistency

[feral island]

Honestly not sure how much thought went into that default. Maybe nobody wants to increase it because it's hard to prove it will work on all systems, and it's rarely a problem in practice

https://github.com/python/cpython/issues/47623#issuecomment-1093430390

"Why was 1000 chosen in the first place? If it's just an arbitrary value
then we can bump it to 4000 so that people don't get bad surprises when
upgrading their Python." And 14 years later 1000 is still the default 🙂

[raven ridge]

hah - well, that's pretty definitive

[rose schooner]

how would i make it work for JUMP_BACKWARD

[pliant tusk]

compute the length of the bytecode, and jump backward that length i assume

[rose schooner]

hmm yeah so i had problems with jmp_tbl so i just deleted the line in my modification that assigns to that and it works

[pliant tusk]

ok i guess, although that probably breaks jumps that occur in some places

jmp_tbl is critical for keeping track of jump destinations

[rose schooner]

this line https://github.com/chilaxan/pysnippets/blob/main/tco.py#L95

[fallen slate]

tco.py line 95

jmp_tbl[id(decomp[end])] = decomp[0]```

[pliant tusk]

yea that line sets the destination of the last item in decomp (a jump) to the first item in decomp

[rose schooner]

>>> import sys
>>> sys.version
'3.11.0a7+ (main, Apr 28 2022, 10:07:36) [MSC v.1931 64 bit (AMD64)]'
>>> @tco
... def fact(n=1000, acc=1):
...     if n < 2:
...             return acc
...     return fact(n - 1, n * acc)
... 
>>> fact()
40238726007709377...

made it work

[pliant tusk]

what did you change?

[rose schooner]

a lot

[pliant tusk]

pm me the edits?

[rose schooner]

the edited file?

[pliant tusk]

ya

if you don't mind sharing, im curious to see how it works

[rose schooner]

https://github.com/python/cpython/commit/93a666b5a56919a3633a3897dfdb9bddfb9614f0 goddamn it now it won't work for 3.11.0b1

i mean i just have to change how i get the inline cache entries

[pliant tusk]

ahh so your JUMP_BACKWARD works because jumps that arent in the jmp_tbl just get their arg passed through (although you should still set into the jmp_tbl and modify the recomp code to special case JUMP_BACKWARD)

[rose schooner]

well i don't know how the recompilation code works

[pliant tusk]

that would make it so that even if items are inserted before the jump_backward it would still recomp to the right place

im gonna wait till 3.11 byc is finallized and then im gonna rewrite the decomp and recomp code again

[rose schooner]

it seems like all jump bytecode is relative now

[pliant tusk]

oh sweet that makes the logic for decomp and recomp way simpler

would still need to special case JUMP_BACKWARD so that it understands that relative jump is negative but the rest of it will be the same for all jumps

[deft horizon]

Specifically that I don't want to break https://github.com/HypothesisWorks/hypothesis/pull/2506

[grave jolt]

and then you get an angry email from someone who actually relied on the limit for some feature in their program 😛

def _ensureNoLongerThan1000(things):
    if things:
        _ensureNoLongerThan1000(things[1:])

def ensureNoLongerThan1000(things):
    try:
        _ensureNoLongerThan1000(things)
    except RecursionError:
        return False
    else:
        return True

[west tinsel]

Had the same thought, some people don't deserve backwards compatibility 😂😂

[rose schooner]

damn python 3.12.0a0

[tardy sluice]

😰

[grave jolt]

3.14 is near

let's postpone the nuclear apocalypse just for that

[rose schooner]

https://discuss.python.org/t/walrus-fails-with/15606
https://mail.python.org/archives/list/[e-mail]/thread/5CWWY4EZKXLJZD47NSQA6TRD5SWMFGOJ/
i have a problem with the symtable when i'm trying to implement this

>>> ([a, b] := (1, 2))
(1, 2)
>>> a
1
>>> b
2
``` i can make it work with non-scoped things in the REPL but i think there's impending doom with scoped ones

[rose schooner]

ok i think i fixed the scoped ones

so overall seems easy enough to do

>>> a = lambda x: ((x, x[0]) := (x, x))[0]
>>> a([0])
[[...]]

[rose schooner]

just have to modify Grammar/python.gram and a little bit of Python/symtable.c and it's done (once you've regenerated the parser and rebuilt python)

[rich cradle]

quick let's discreetly add some weird bugs to the stdlib so that we can get 3.14.15

[grave jolt]

let's just switch to TeX version numbering

(look it up, it's cursed!)

[rich cradle]

I love it

[feral cedar]

it's much better than semver, that's for sure

[radiant garden]

Well here's my contribution to the field of versioning :>

https://github.com/RocketRace/setver

[radiant garden]

is it cursed compared to this?

[rose schooner]

>>> class a:
...     class b:
...             c = 3
...
>>> [((m, n, a.b.c) := (a.b.c, a.b.c + 5, 5)) for _ in [0]]
[(3, 8, 5)]
>>> m
3
>>> n
8
>>> a.b.c
5
``` more examples

[fossil blade]

what's the better version of this?

r"{}".format(string)

I remember something like
string:r

[feral island]

the better version of that code is string. But I think you're thinking of f"{string!r}"

[boreal umbra]

r is for raw strings. I think these are mostly used for regular expressions. But the point is that it causes the interpreter to not escape backslashed characters.

f strings are used in place of the format method in most circumstances these days. I sometimes use the format method to make a sort of template string, by not actually calling the method.

[fossil blade]

this is it!

!r

[fossil blade]

I know this stuff thanks though

[quick snow]

What does f"{string!r}" do that repr(string) doesn't?

[dusk comet]

it is a bit faster

>>> from dis import dis
>>> dis('f"{a!r}"')
  1           0 LOAD_NAME                0 (a)
              2 FORMAT_VALUE             2 (repr)
              4 RETURN_VALUE
>>> dis('repr(a)')
  1           0 LOAD_NAME                0 (repr)
              2 LOAD_NAME                1 (a)
              4 CALL_FUNCTION            1
              6 RETURN_VALUE

fstring dont lookup repr name in current scope

[prime estuary]

It's shorter to type also, given it's a common thing you'd expect to do.

[rose schooner]

>>> def quicken(f, args=(), kwargs={}, times=10000):
...     for _ in range(times): f(*args, **kwargs)
...
>>> def f(a):
...     return '%r'%a
...
>>> quicken(f, ('goboc',))
>>> from dis import dis
>>> dis(f,show_caches=True,adaptive=True)
              0 RESUME_QUICK                   0

  1           2 LOAD_CONST__LOAD_FAST          1 ('%r')
              4 LOAD_FAST                      0 (a)
              6 BINARY_OP                      6 (%)
              8 CACHE
             10 RETURN_VALUE
``` i was disappointed in it not having its own specialization in python 3.11
so i added a new specialization for it ```py
>>> # ...
>>> dis(f,show_caches=True,adaptive=True)
              0 RESUME_QUICK                   0

  1           2 LOAD_CONST__LOAD_FAST          1 ('%r')
              4 LOAD_FAST                      0 (a)
              6 BINARY_OP_STRING_FORMAT        6 (%)
              8 CACHE
             10 RETURN_VALUE

[rose schooner]

>>> from dis import dis
>>> dis("2 .__int__.__call__")
              0 RESUME                            0

  1           2 LOAD_CONST                        0 (<method-wrapper '__call__' of method-wrapper object at 0x00000260DE48AB70>)
              4 RETURN_VALUE
>>> dis("2 .__int__.__call__()")
              0 RESUME                            0

  1           2 LOAD_CONST                        0 (<method-wrapper '__int__' of int object at 0x00007FFCAFE5B348>)
              4 LOAD_METHOD                       0 (__call__)
             26 PRECALL                           0
             30 CALL                              0
             40 RETURN_VALUE
``` i got bored so i started optimizing random stuff

[next lion]

Question, I remember reading somewhere that Python 3.11 was going to make so new frames would not be created unless told to do so, does that mean that we will no longer be able to do fun things like checking where a function was called from or modify a caller's variable without explicitly passing it through in cpython?

I think I might have misinterpreted it though

[feral island]

it no longer creates frame objects until requested. But sys._getframe will continue to work, just lazily

[next lion]

Ahhh, I see, so we will still have access to frames

Thanks for clarifying

[safe hedge]

Is anyone able to expand and elucidate on the following line in the packaging docs:

However, namespace packages come with several caveats and are not appropriate in all cases

There's no discussion of it further in the docs that I can see

[feral island]

My experience is that they can lead to confusing behavior if you have a directory on your path that isn't meant for python packages

For example we have a big directory that contains a subdir called redis for the redis server code. But if you do import redis in Python code you might get a namespace package referring to that directory.

[paper echo]

basically you shouldn't use them unless you know that you need them

imo fully-implicit namespace packages might have been a mistake

i understand that pkg_resources.declare_namespace(__name__) is maybe a bit cumbersome, but is it bad?

[raven ridge]

Yes. The problem is that it necessitates a package that will call that registration method.

and that package may need to be installed multiple times in different directories on sys.path

by "package" I mean PyPI distribution, as opposed to a module that contains submodules, to be clear.

[elder blade]

Thing is, you'll only take advantage of namespace packages when you know what they are

[paper echo]

but a package which doesn't call that registration method can still "clobber" a namespace package anyway, right?

as in, two packages with ns/__init__.py can't both live inside the ns namespace package

[paper echo]

or you're a newbie wondering why import Projects.Scripts.CS101.Homework3 works, but import ..Scripts.CS101.Homework3 doesn't work

from the #pedagogy perspective, fully implicit namespace packages can really make things confusing

[elder blade]

Hmm, will that be discovered by Python though?

[paper echo]

as far as i know, yes? at least i recall seeing it in at least one help channel before

i might have even done it once or twice myself

[elder blade]

Like- there's two places Python can look: inside of site packages and inside of the project you're working on?

[paper echo]

let me test it, i might be wrong and/or remembering a different issue

this isn't about sys.path, it's about how packages are discovered therein

yeah, its a problem

mkdir -p a/b/c/
echo 'print(123)' > a/b/c/d.py
python -m a.b.c.d
# 123

tree a shows

tree a
a
└── b
    └── c
        ├── d.py
        └── __pycache__
            └── d.cpython-310.pyc

3 directories, 2 files

basically, the implicit namespaces are resolved within python's import and package discovery system

arguably they should have been handled by package installers (eg. setuptools) placing a dummy __init__.py file in the file tree, but then i guess you lose support for importing a "not yet installed" namespace package

its the same old tension in python between "python is an easy-to-use scripting language" and "python is an industrial-strength general-purpose programming language"

i think implicit namespace packages are also allowed to be installed in 2 different locations within pythonpath

here

.
├── lib1
│   └── a
│       └── b
│           └── c
│               └── d.py
├── lib2
│   └── a
│       └── b
│           └── c
│               └── e.py
├── main.py
└── main.sh

main.py:

from a.b.c.d import x
from a.b.c.e import y

print(x, y)

main.sh:

export PYTHONPATH=lib1:lib2
exec python ./main.py

and executing main.sh prints 1 2

so your only options really are: 1) status quo, fully implicit namespace packages 2) a special marker file (py.namespace?) that contains no content and precludes the use of __init__.py or anything other files in the same directory

[orchid karma]

So I'm super excited about this https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-f.html?m=1

[rose schooner]

if they tokenized non-indent whitespace and added a mechanism to ignore if not needed that'd make stuff a lot of easier

[raven ridge]

Yes, exactly. That's the problem, that __init__.py needs to be provided by exactly one distribution, and that distribution needs to be installed on a directory that's at least as early on sys.path as any sub package of the namespace package it's registering

[paper echo]

i guess i don't understand why a placeholder file like py.namespace (in all distributions using the same namespace) couldn't also do the same job. is the intention that theoretically one of the distributions could be unaware of the namespace and be allowed to use its __init__.py?

the rule could be either you have __init__.py or you have py.namespace, but not neither and not both

the contents of py.namespace would be ignored, analogous to py.typed

the problems w/ all the prior solutions described in pep 420 had to do with 1) reusing __init__.py, and 2) running code that dynamically modified the search path

the solution then had to be something in the space of "it needs to be something that can be statically determined, and it needs to not use __init__.py"

[raven ridge]

i guess i don't understand why a placeholder file like py.namespace (in all distributions using the same namespace) couldn't also do the same job
Because if you uninstalled any one of those distributions, it would break all the others.

the way wheels work doesn't allow for a file to be provided by multiple distributions such that it survives an uninstall of one of those distributions.

[paper echo]

does "uninstall" necessarily mean rm -rf?

[raven ridge]

yes

I mean, it could not, but then you're talking about changing how wheels are specified, and dragging this issue down into the packaging space.

[west tinsel]

I haven't followed the discussion and this is unrelated, but is there a reference for the specifics of how namespaces work? I've had to learn it through experience and really I'm working on intuition when I'm building things. I've seen guides in the past but they were surface level and didn't explain the internal workings or the reasons for lookup errors

Wrt importing and directory structure

[raven ridge]

What would work is a distribution-specific file that contains a list of all the namespace packages that distribution intends to register - if foo.bar requires foo to be a namespace package, there could be a foo.bar.namespaces file installed containing just the one line foo, or whatever. But that has a different problem: it's extremely expensive to resolve. Whenever you want to import any package, you'd need to stat every file on every directory on sys.path to see if it's one of those special .namespaces files

[raven ridge]

have you read PEP 420?

[paper echo]

right, i guess pep 420 was the lowest-impact solution then

[west tinsel]

My browser history says I've been on the page, but I don't remember reading it 😂

I'll go through it now, thank you!

[raven ridge]

it's dense, but it's the right starting point. If you read it and have questions, ask them here 🙂

[raven ridge]

Yeah. I don't like that PEP 420 makes it possible to accidentally create a namespace package, but the alternative - explicit registration - would have been a much more involved problem to solve. pkg_resources.declare_namespace and pkgutil.extend_path worked "well enough", even though they weren't perfect, so I suspect any attempt to require explicit registration would have collapsed under its own weight for not providing clear enough ROI given the amount of effort it would have involved

[safe hedge]

I just read all this and I feel like I still haven't fully grasped the negative of using a namespace package.
Is it basically that your package may get masked/superseded by a non-namespace package with the same name that has ended up in your path?

[feral island]

One downside is that directories that were never meant to represent Python packages can get picked up as namespace packages

[raven ridge]

Another downside is that some features only work on regular packages. ```shell-session
$ mkdir foo
$ echo 'hello' >foo/bar.txt
$ python -c 'from importlib.resources import read_text; print(read_text("foo", "bar.txt"))'
Traceback (most recent call last):
...
FileNotFoundError: 'bar.txt' resource not found in 'foo'
$ touch foo/init.py
$ python -c 'from importlib.resources import read_text; print(read_text("foo", "bar.txt"))'
hello

[safe hedge]

Well that’s interesting

[raven ridge]

It makes sense - importlib.resources.read_text is supposed to go to a package's directory and read a file from it, but namespace packages have many directories - how would it decide which of them to read the file from?

[safe hedge]

Slight aside but what is the purpose of that function more broadly? When would you want a .txt alongside your code like that?

[raven ridge]

storing non-Python data, generally. configuration, documentation, resources, sprites, etc

[safe hedge]

I get the issue, but thinking about it, isn’t having anything in the foo namespace dir a violation of the way it’s supposed to be done anyway?

[elder blade]

Well, it would be annoying if Python would error if there was literally any other file in a namespace directory

[raven ridge]

yes. This mostly comes up in the case that salt and I were talking about earlier: someone accidentally making their package a namespace package, when they meant to have a regular package, and having it Just Work (until it doesn't)

[safe hedge]

Sure, but the point is the code behaves differently because you implemented something unsupported rather than a real big imo

So is it safe to say that properly implemented there isn’t really an issue with them? And that the issues are mainly on the developer side rather than the user side?

[raven ridge]

the argument is that it's a weakness in the design that users can accidentally create a package that mostly works except in some contexts.

[safe hedge]

Sure, I guess it was more that my original question was more concerned as to whether the "caveats" mentioned in the docs meant that there was a good reason not to actively choose one over a normal package. If the only real downsides are in the potential to shoot yourself in the foot as opposed to actual functionality (I appreciate you can argue the example above with the txt file at the namespace level is just that, but I suppose there are ways around it like putting that file one level down?) then I think that's what I was trying to understand

[raven ridge]

but I suppose there are ways around it like putting that file one level down?)
Only if you put it into a regular package

if "one level down" is still a namespace package, you still have the same problem.

Hindsight being 20/20, I think it would have been better had they said that implicit namespace packages are allowed to contain other packages, but not other non-package modules. That would have cut down on the accidental creation of namespace packages by a lot...

then the case where someone has a foo/bar.py would fail if someone tried to import foo.bar, and if they really wanted that they'd need to instead create a foo/bar/__init__.py, which would prove that they actually know what an __init__.py is and that leaving it out of the foo directory wasn't accidental 😄

[safe hedge]

Yeah I think really I was probably only focussing on my intended implementation which would be a single parent namespace package containing traditional sub-packages spread over multiple locations (basically the example in the docs) without seeing how it can be jankier in other cases (multiple layers of namespace packages seems more confusing for example)

[paper echo]

also the weakness is mostly a pedagogical problem for newbies who are not aware of namespace packages, or people who stubbornly refuse to learn how python packaging works

[rose schooner]

what's the motive for using expression in lambdas instead of named_expression https://github.com/python/cpython/blob/main/Grammar/python.gram#L817-L819

[fallen slate]

Grammar/python.gram lines 817 to 819

lambdef[expr_ty]:
    | 'lambda' a=[lambda_params] ':' b=expression {
        _PyAST_Lambda((a) ? a : CHECK(arguments_ty, _PyPegen_empty_arguments(p)), b, EXTRA) }```

[quick night]

whats the diff?

[rose schooner]

named_expression allows the walrus

[quick night]

ah ok

[dusk comet]

from foo import (bar,) # valid syntax
from foo import () # invalid syntax

why the second line is invalid syntax?
why is it prohibited?
this can be useful in case i only need to import a module but not pollute the namespace

[feral cedar]

isn't that just import foo?

[boreal umbra]

from foo import (bar,) doesn't add foo to the namespace, only bar. what is from foo import () supposed to add to the namespace?

you could do import foo as _ if you wanted the import logic for foo to be executed for some reason. I guess.

[grave jolt]

If you don't want to bind a value to foo, __import__("foo")

[sturdy timber]

On python 3.11 is there as reason the enhanced error locations in tracebacks aren't enabled in the default repl/python shell?

[peak spoke]

Does the repl give you anything beyond the exception?

[sturdy timber]

Hmm I guess not, as line numbers wouldn't make much sense

Although it seems like something that should be perfectly possible

[boreal umbra]

@raven ridge ask Pablo to explain

[rose schooner]

https://github.com/python/cpython/pull/27117 there's a draft for it but it's only a draft

and also "Purely for design purposes"

[rose schooner]

it isn't a file

https://github.com/python/cpython/blob/main/Python/traceback.c#L454-L463

[fallen slate]

Python/traceback.c lines 454 to 463

binary = _PyObject_CallMethod(io, &_Py_ID(open), "Os", filename, "rb");
if (binary == NULL) {
    PyErr_Clear();

    binary = _Py_FindSourceFile(filename, buf, sizeof(buf), io);
    if (binary == NULL) {
        Py_DECREF(io);
        return -1;
    }
}```

[sturdy timber]

Ah interesting

I do wonder what "Purely for design purposes" is meant to mean

[rose schooner]

>>> 1/0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    1/0
    ~^~
ZeroDivisionError: division by zero
>>> def l(): 1/0
...
>>> l()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    l()
    ^^^
  File "<stdin>", line 1, in l
    l()

ZeroDivisionError: division by zero
``` i was able to implement it in a copy 3.11.0a7+ and reproduce the bug

i don't have an idea of how to fix it though

[sharp lodge]

I am trying to understand why SyntaxError is included on the Built-in Exceptions hierarchy https://docs.python.org/3/library/exceptions.html#bltin-exceptions if on Errors and Expections (https://docs.python.org/3/tutorial/errors.html) they distinguish syntax errors from exceptions? My understanding is that syntax errors are discovered during compile time and runtime errors during runtime. Am I correct in that only runtime errors (as opposed to syntax errors) raise an exception and consequently produce a traceback? And if so, why is SyntaxError then included in the hierarchy? And what is a SyntaxError print out called if not a traceback?

[rose schooner]

SyntaxErrors can also happen at runtime

!e ```py
print('a')
compile('2 +', '', 'exec')

[fallen slate]

@rose schooner :x: Your eval job has completed with return code 1.

001 | a
002 | Traceback (most recent call last):
003 |   File "<string>", line 2, in <module>
004 |   File "", line 1
005 |     2 +
006 |        ^
007 | SyntaxError: invalid syntax

[sharp lodge]

@rose schooner Thank you. That's good to know. Can you help me understand why some syntax errors aren't caught during compiling (other than your example with compile())? An internet search didn't reveal much about examples of syntax errors that are caught during runtime.

[pliant tusk]

the compile function is just passed a string. Python does not know that the string is going to be valid python code until runtime

[raven ridge]

the Python program isn't compiled as a whole - Python will compile a module when you import it, unless it detects that it has already compiled that module in the past and it hasn't changed since, in which case it'll reuse the already-compiled module from a cache. So, if you import a syntactically invalid module, that can only be caught at run time.

for instance:

/tmp>echo '"' >bar.py
/tmp>echo 'import bar' >foo.py
/tmp>python foo.py
Traceback (most recent call last):
  File "/tmp/foo.py", line 1, in <module>
    import bar
  File "/tmp/bar.py", line 1
    "
     ^
SyntaxError: EOL while scanning string literal

[lusty scroll]

anyone know the reason for this quirk?

As an implementation detail, most modules have the name __builtins__ made available as part of their globals. The value of __builtins__ is normally either this [builtins] module or the value of [the builtins] module’s __dict__ attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python.
from https://docs.python.org/3/library/builtins.html

it appears that if __name__ == "__main__", then __builtins__ is builtins, otherwise __builtins__ is builtins.__dict__ 🤔

and it was the same way in python 2

[rose schooner]

🤔

[rose schooner]

i fixed #27177 so this can work now ```py

def l(): 1/0
...
l()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
l()
^^^
File "<stdin>", line 1, in l
def l(): 1/0
~^~
ZeroDivisionError: division by zero

[spark magnet]

I like that 1/0 gets a dastardly mustache 🙂

[sturdy timber]

Nice

[true ridge]

just to play around that idea, i guess.

[sturdy timber]

I see, thanks!

[rose schooner]

https://github.com/python/cpython/pull/92827 i have a PR but for some reason it exits with code 1

[rose schooner]

oh i got it

it doesn't exit with code 1 now

[elder blade]

What the hell happened in that PR?

22 commits all named the same and you have opened/closed it like 4 times

[rose schooner]

i suck at contributing

most of the commits are file changes and the others are repairs

the closing/opening is because it kept segfaulting

and i guess i have to close it because #27177 is still open

[rose schooner]

this is the result of me not using git

[sacred quarry]

<class 'mercurial`>

[elder blade]

If you decide so. In the end, only one of the PRs should be merged if they fix the same thing.

If they have different stategies you have to decide which one would be better in the end. Otherwise, generally speaking you would honour the older PR unless that one has gone stale and the point of the new PR is to take over the old one's work.

[rose schooner]

older PR did go stale and has a bug

also older PR is not supposed to be reviewed/merged

[rose schooner]

sorry if i seem disrespectful but can i use your approach for fine-grained errors in interactive in the new PR

[true ridge]

No not at all. Though it is very important to document your work in the PR description (especially what are the differences compared to the initial PR, how did you overcome the existing problems etc.). From the initial point of view, I see a major problem (a new field for the code objects) in terms of the performance/memory consumption so you'll also have to run some benchmarks to ensure it doesn't degrade the existing performance. (I didn't review it yet, assuming everything else is fine).

[rose schooner]

ok

[rose schooner]

not sure what test cases i should do

[elder blade]

Try creating much more difficult nested expressions and maybe having multiple lines of code (to test that the correct one is picked)

[west tinsel]

Does the PR name have any significance?

https://github.com/python/cpython/pull/92819

The PR itself is fine, but I want to suggest that they change the name of the PR, does it matter?

[feral island]

It makes it easier to find the PR if the name is better

[west tinsel]

Ok, suggested!

[rose schooner]

update on this: until there can be a simpler way where the advantages outweigh the disadvantages then it won't happen

this is according to pablo

[lusty scroll]

how many "singleton" instances/types exist like None, NotImplemented, ..., etc?

!d NotImplemented - I was just noticing it mentions on here

[fallen slate]

NotImplemented

NotImplemented```
A special value which should be returned by the binary special methods
(e.g. `__eq__()`, `__lt__()`, `__add__()`, `__rsub__()`,
etc.) to indicate that the operation is not implemented with respect to
the other type; may be returned by the in-place binary special methods
(e.g. `__imul__()`, `__iand__()`, etc.) for the same purpose.
It should not be evaluated in a boolean context.
`NotImplemented` is the sole instance of the [`types.NotImplementedType`](https://docs.python.org/3/library/types.html#types.NotImplementedType "types.NotImplementedType") type.

[feral island]

Ellipsis is another, I think those three are the only builtin examples

[lusty scroll]

is it only those 3 ? oh, ok

interesting , good to know ✌️

also does NotImplementedonly have meaning in binary operators ?

[feral island]

https://github.com/python/cpython/blob/702e0da000bf28aa20cb7f3893b575d977506495/Python/bltinmodule.c#L3046

[fallen slate]

Python/bltinmodule.c line 3046

SETBUILTIN("None",                  Py_None);```

[feral island]

it's also meaningful for __length_hint__

from some quick grepping seems like otherwise the only place where it affects builtin behavior is binops though

[raven ridge]

Also True and False, right?

Or have I missed some subtlety of the question?

[lusty scroll]

oh had no idea

and thats not only for builtin types right?

[lusty scroll]

considering "fixed number of instances" True and False definitely fit

all-constant as well

[raven ridge]

That's how I interpreted it, but Jelle probably interpreted singleton as "the only instance of its type"

[lusty scroll]

is there another word besides singleton?

[raven ridge]

¯\_(ツ)_/¯

The bool constructor always returns one of two existing instances. That's not quite the singleton pattern, but there's no closer word... Doubleton? 🙂

I'd say that True and False are singleton instances of the bool type, but that's perhaps playing it a bit loose on the terminology front

[feral island]

yes, bools are conceptually similar but I wouldn't call them singletons. Enums are also similar (fixed number of instances), but those aren't builtin

[feral island]

you can return NotImplemented from a user-defined __length_hint__ too, yes. But that's also true for the binops

[native flame]

was looking through dir(builtins), just noticed there's an aiter and anext now

[lusty scroll]

is that for
async for ... ?

[native flame]

yeah, async equivalents of iter and next