internals-and-peps

[boreal umbra]

!resources

[fallen slate]

Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

[boreal umbra]

@tight phoenix if you have a follow-up question, please use #python-discussion

[peak spoke]

I was adding a handler for KeyboardInterrupt to go with a Qt app and noticed some strange behaviour, what could be the cause of it?
To let python handle the signal, I added a function that gets called periodically by Qt to step into python. If it called function is empty (pass), the function raises the keyboard interrupt as expected; but when I add a try except around it for KeyboardInterrupt, it suddenly isn't caught by python unless I do some operation like calling a function into its try block.
so

def f():
    pass  # KeyboardInterrupt gets raised

def f():
    # Never gets the interrupt, instead a future call to a different function will
    try:
        pass
    except KeyboardInterrupt:
        ...

def f():
    try:
        str()
    except KeyboardInterrupt:
        ... # works as expected

[feral island]

I guess the exceptions gets raised when the ceval loop checks for pending signals

and if the try block is empty, it never gets to do that

[peak spoke]

wouldn't that get skipped on in the empty func too though?

[feral island]

that func isn't empty, I bet it has LOAD_CONST and RETURN_VALUE in it

[peak spoke]

Yes, for the none return value, but the try except with pass does all of that too

I don't understand how adding the try except makes python completely skip on handling the keyboard interrupt

It's also skipped when I'm not calling a function but doing some operation that doesn't go through python like integer adding

https://paste.fuelrats.com/zahosaxofa.py fwiw

[elder blade]

There are a handful of opcodes which don't cause the interpreter to check pending signals

try is one of those.

This so that code such as the following is safe: ```python
acquire()
try:
...
finally:
release()

[peak spoke]

How does that propagate to the try's body which has its own instructions that do get the error in the empty function?

[elder blade]

I am not sure why case A got KeyboardInterrupt, but case B didn't get it inside of the body, either.

If you replace pass with just return, do you get the same behaviour?

[peak spoke]

Yes; they compile to the same bytecode

[quick snow]

Safe-ish, right? The signal could still occur before release() is called, but in the finally block. Or somewhere inside of release(), unless it's pure C..

[raven ridge]

This isn't about when a signal arrives, it's about when the interpreter calls the signal handler for a signal that has already arrived.

[quick snow]

Yes, and I understand it doesn't call it for some opcodes. But can't it call it before release is called; between LOAD_* and CALL_FUNCTION?

[raven ridge]

Yes

I'm with you now.

[flat gazelle]

In [14]: dis.dis('''
    ...: try:
    ...:     print(1)
    ...: finally:
    ...:     print(2)
    ...: ''')
  2           0 SETUP_FINALLY           12 (to 14)

  3           2 LOAD_NAME                0 (print)
              4 LOAD_CONST               1 (1)
              6 CALL_FUNCTION            1
              8 POP_TOP
             10 POP_BLOCK
             12 BEGIN_FINALLY

  5     >>   14 LOAD_NAME                0 (print)
             16 LOAD_CONST               0 (2)
             18 CALL_FUNCTION            1
             20 POP_TOP
             22 END_FINALLY
             24 LOAD_CONST               2 (None)
             26 RETURN_VALUE
```as I understand it, SETUP_FINALLY cannot be interrupted by a signal handler, since that would mean there would be no finally set up. But after that having an exception is ok, since the finally is there and will be run.

also, interesting that python doesn't put the BEGIN_FINALLY on line 4

since SETUP_FINALLY does get its own line

[rose schooner]

python 3.11 bytecode is so different

[quick snow]

Yes. But what if the signal gets processed inside the finally block (so yes, it did enter), but before release was called? Couldn't that happen? I remember reading an old blog post by njsmith about it (in the context of how Trio deals with it).

https://vorpus.org/blog/control-c-handling-in-python-and-trio/

[raven ridge]

Yes, that can happen. Functionally that's no different than if release was called and then immediately interrupted by an exception before it releases anything

[quick snow]

Yes. Unless release isn't written in Python and doesn't check for exceptions until it's done

[raven ridge]

Right. IIUC, the point of having certain bytecodes where the exception can't be raised or Python signal handler can't be called isn't to guarantee that the code in the finally block runs, it's just there to simplify the interpreter's job. There's some places where it would be hard to propagate an exception or call arbitrary code, so in those places, it just doesn't, and puts it off until a place where it's in a better state for it.

[rose schooner]

>>> import dis
>>> dis.dis('''
... try:
...     print(1)
... finally:
...     print(2)
... ''')
              0 RESUME                   0

  2           2 NOP

  3           4 PUSH_NULL
              6 LOAD_NAME                0 (print)
              8 LOAD_CONST               0 (1)
             10 PRECALL                  1
             14 CALL                     1
             24 POP_TOP

  5          26 PUSH_NULL
             28 LOAD_NAME                0 (print)
             30 LOAD_CONST               1 (2)
             32 PRECALL                  1
             36 CALL                     1
             46 POP_TOP
             48 LOAD_CONST               2 (None)
             50 RETURN_VALUE
        >>   52 PUSH_EXC_INFO
             54 PUSH_NULL
             56 LOAD_NAME                0 (print)
             58 LOAD_CONST               1 (2)
             60 PRECALL                  1
             64 CALL                     1
             74 POP_TOP
             76 RERAISE                  0
        >>   78 COPY                     3
             80 POP_EXCEPT
             82 RERAISE                  1
ExceptionTable:
  4 to 24 -> 52 [0]
  52 to 76 -> 78 [1] lasti
``` python 3.11

[quick snow]

Wow, okay. ExceptionTable instead of SETUP_FINALLY? I guess I have to read 3.11 bytecode docs..

[jovial flame]

Hello, I have removed your message as it goes against Rule 6 - don't post unapproved advertising. Thanks!

[dusty osprey]

its not paid tho?

[quick snow]

Which is why it only violates rule 6, not rule 9. It's also off-topic for this channel (the "internals" doesn't refer to "internal discussions on how to run and moderate this community").

[narrow kettle]

was making from __future__ import annotations the default removed from 3.10? allowing this code for annotations ```py
class Foo:
def bar(self) -> Foo:
...

[naive saddle]

yeah, it's been postponed twice, once for 3.10 and again for 3.11 https://docs.python.org/3/library/__future__.html#id1

see also https://mail.python.org/archives/list/[e-mail]/message/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/
https://mail.python.org/archives/list/[e-mail]/message/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/

[narrow kettle]

ahh this is exactly what i was looking for thank you very much

20 mins of googling was less then helpful lol

unfortunate, but the import works fine

[grave jolt]

it's like me waking up

"5 more minutes" x20, and then I'm barely on time

👀

[white nexus]

[quick snow]

I wish..

[raven ridge]

star imports are hard for tools that try to statically analyze a single module at a time to deal with, but they don't seem to make things any harder for tools that try to statically analyze the entire program - like static type checkers

[white nexus]

someone make a pep when

[raven ridge]

Serious question: why? What's the problem it would solve?

[radiant garden]

It would solve the problem of too much convenience

[boreal umbra]

I wouldn't let any code with a star import pass a code review, but I wouldn't make a backwards incompatible change just for that.

[rose schooner]

i'm gonna do that

[rose schooner]

just mixed it in my featureful modified cpython folder ```py

from future import no_star_imports
from sys import *
File "<stdin>", line 1
from sys import *
^
SyntaxError: star imports were removed from Python (PEP 997)

[white nexus]

lmao nice

[boreal umbra]

@minor karma take a look at #❓｜how-to-get-help. And remember not to ask to ask--you'll never get an answer if you don't say your actual question.

[charred wagon]

Really? I think it's fine to use them in init.py

I wouldn't want star imports removed

[boreal umbra]

I guess that's fine, but I've never seen that in a code review 😛

but if there's ever a circumstance where star imports create any amount of effort to read/understand the code, I wouldn't accept that.

[surreal sun]

This ^^ a lot of libraries from what I've seen rely on star imports for __init__.py and just removing them is gonna be a pain in the ass for a lot of libraries

[unkempt rock]

"explicit is better than implicit"

[astral gazelle]

and yet

[flat gazelle]

Star imports do sometimes make sense. No point in repeating tk. 7 times a line or turtle. once per line, certainly doesn't make the code more readable.

[rose schooner]

are you explicitly supposed to update __init__.py every time the module it's importing updates?

seems like too much work for something that can be done with one change and never again

[radiant garden]

definitely don't follow zen of python blindly

use common sense when it applies

[rich cradle]

I occasionally use star imports when the entire file is set around things from that module.

Like said above, if an entire file is just tkinter boilerplate, I'll use a * import.

[rose schooner]

!e why does using ast.unparse on a + b and a + b and ... and a + b parenthesize the latter few parts? ```py
import ast
c = ast.parse("a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b")
print(ast.unparse(c)) # i could've added more to show

[fallen slate]

@rose schooner :white_check_mark: Your 3.11 eval job has completed with return code 0.

a + b and a + b and a + b and a + b and a + b and a + b and a + b and (a + b)

[dusk comet]

!e

import ast
c = ast.parse("a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b and a + b")
print(ast.unparse(c))

[fallen slate]

@dusk comet :white_check_mark: Your 3.11 eval job has completed with return code 0.

a + b and a + b and a + b and a + b and a + b and a + b and a + b and (a + b) and (a + b) and (a + b) and (a + b) and (a + b) and (a + b) and (a + b)

[frigid bison]

Interesting

[boreal umbra]

is this just a quirk of the parser, or what

@raven ridge can you summon pablo?

[rose schooner]

it's a quirk of the unparser

for some reason they have this https://github.com/python/cpython/blob/main/Lib/ast.py#L1444-L1448

[fallen slate]

Lib/ast.py lines 1444 to 1448

def increasing_level_traverse(node):
    nonlocal operator_precedence
    operator_precedence = operator_precedence.next()
    self.set_precedence(operator_precedence, node)
    self.traverse(node)```

[fluid swallow]

Hello everyone I have a little bit particular question. I would like to know what are the asymptotic performances of dictionaries deletion in Python . I saw that python implement dictionaries using hash table and that it seems to use open adressing to solve collision but I read on the CLRS that deletion the open adressed hash tables is slower than deletion in hash table where collisions are solved with chaining

[feral cedar]

the asymptotics are the same, since the load factor is constant

[quick snow]

Do you know this page? https://wiki.python.org/moin/TimeComplexity#dict

[faint junco]

How important it is to know the time complexity of a function? 2 weeks ago I failed a coding test for a job because I had no idea what TimeComplexity was.

[faint junco]

And finally a good read about it.

[visual shadow]

This is probably not the right channel to ask, but it's very important. It gives you a way to figure out if the code you're writing is efficient or not.

[faint junco]

Apologize, I just took the chance because of his answer. And thanks for your answer too.

[neat delta]

i'm sure this has been covered before, so instead i'll ask: does anyone have a link to a discussion/talk/etc of why CPython exists at all? It's a general sign of maturity when a language is written in itself (aka bootstrapping), but python has never been this way. Why was CPython not abandoned for what is now considered PyPy at or before version 1?

[rose schooner]

so it's faster without requiring any more complex steps?

i don't really know the reason

[neat delta]

pure pypy is generally faster than cpython, and in some cases, (unoptimized, obviously) C

[native flame]

i don't have a source on this but iirc pypy is worse than cpython at interfacing with e.g. c extensions

[flat gazelle]

CPython IIRC is faster for one-off scripts and such where the JIT doesn't get to do much, and the extra warmup time is crippling.

[frigid bison]

I don't understand how pypy can be faster

Like how is it ran

Since python is interpreted

[feral cedar]

https://www.pypy.org/features.html

[visual shadow]

Pypy is not.

[flat gazelle]

the basic idea is you say "this + will only ever operate on ints that fit into 4 bytes", then you check that this is the case everytime before you run, and if it is, you do a fast addition using CPU instruction, rather than doing bigint math. Or you say "last 10 times, this called this specific function, it will probably call the same one again"

[feral island]

This is generally not true for dynamic "scripting" languages. JavaScript, Ruby, PHP implementations are also generally written in C

[flat gazelle]

even pypy isn't really python written in python, it's python written in a slang of python designed for writing interpreters

writing compilers in the language they are compiling makes sense, writing a metainterpreter as the canonical implementation is not anywhere near as common

even lisps generally aren't written in lisp, though most of their standard library is.

[rich cradle]

isn't it? I thought it was interpreted but some code is JITted.

[rich cradle]

You would need to maintain two implementations that way I believe, so it's significantly more complicated.

Because you need to maintain the code that makes up the original interpreter binary (probably in some other language), and run the second interpreter (which is in the language its interpreting) with that binary, which is kinda useless.

[visual shadow]

it is JIT compiled. making the distinction beyond that feels...hmm...awkward.

[raven ridge]

Interpreters with a JIT still don't compile every function. If a function is only called once, spending the time to compile it to machine code would just make things slower.

[visual shadow]

Indeed. All jit work that way though, so isn't saying jit enough?

Optimizing hot paths is one of the core trademarks of a jit compiler.

The implication being, cold paths don't get the same love

[raven ridge]

Right. But the statement "Pypy is not [interpreted]" is misleading, because functions that aren't on hot paths are interpreted in pypy.

[visual shadow]

In the same vein, pypy is interpreted is equally, if not more, incorrect.

[raven ridge]

sure, because both are absolutes. "Pypy is not always interpreted" or "pypy is sometimes interpreted" would be more correct than either

[visual shadow]

I suppose i can concede that all jit behave this way, but it feels awkward to call a jit anything but compiled. However, i'm not opposed to that version for sure, since it is correct.

[grave jolt]

!e

print(f"")

[fallen slate]

@grave jolt :warning: Your 3.11 eval job has completed with return code 0.

[No output]

[grave jolt]

Well that's interesting, empty f-strings are a thing

I mean, there's no reason they shouldn't be a thing

But damn... Maybe make it raw as well

[feral cedar]

they can be

[rose schooner]

nvm confused it for bytes string

[native flame]

would be weirder if it was forbidden tbh

[paper echo]

in general it's very rarely a good idea to abandon an old and well-established implementation in favor of an experimental and frankly somewhat radical one.

and pypy was until somewhat recently mostly an interesting experiment and a curiosity

by the time pypy came out, cpython was already a battle-tested piece of software with thousands of developer-hours poured into it. it would be madness to abandon such a codebase unless it was catastrophically unusable, which it is most certainly was not and will probably never be.

moreover, there is nothing wrong with cpython! why switch?

you could argue that the GIL is a problem, but pypy uses a GIL too because of how python is designed, it's not because the cpython devs suck and can't figure out how to do multithreading

python being used for "high performance" applications and/or "industrial scale" software development is a relatively new phenomenon as well, driven by factors like the increasing maturity of its web frameworks along with the industry souring on ruby, the ubiquity of python in machine learning which became very popular in the early '10s, and its increasing recognition as a good beginner/intro language

[rose schooner]

why is multithreading not allowed by default if it's for "high performance"

[paper echo]

it's not a matter of "not allowed by default". due to the highly-dynamic design of the python language, cpython has a "global interpreter lock" which prevents the bytecode interpreter from running more than one instruction simultaneously, even across threads. this is why historically you have always needed multi-processing, not multi-threading, for parallelism in python.

i put "high performance" in quotes because it's still relatively slow compared to languages with optimizing ahead-of-time compilers. but users have been placing greater and greater demands on python (and specifically cpython) over the years, and gradually cpython performance has improved to meet that demand.

i did a benchmark a while ago on a program that just read lines from stdin in streaming fashion and computed some running statistics; using idiomatic no-frills python code, cpython 3.10 was consistently faster than lua 5.4 and gnu awk (which admittedly is the "slow" awk), on par with luajit (!!), and only a bit slower than pypy. unsurprisingly, nim was a bit faster than all of the above.

i've been working gradually on re-doing that benchmark now that i have a fancy-pants m1 mac for work (and i'm now much better at all of the languages i tested)

it's also consistently faster than sbcl common lisp which i found very surprising, but that's case in point: cpython is pretty damn fast considering how ridiculously dynamic python is

loops and operations on basic data types are nearly instant, even on lower-spec commodity hardware. obviously there's a ton of overhead compared to e.g. c, but among dynamically typed "scripting" languages python is doing pretty well.

the only scripting language i expect should beat it is perl 5, but i would want to benchmark it

[rose schooner]

nim being superior 😔

[halcyon trail]

A benchmark like that could be very influenced by the implementation for reading from stdin

How much buffering it does and so on

Personally for a benchmark like that I'd probably read in all the input from a file and then benchmark the time just to compute the stats

I'm pretty surprised it was faster than the lisp, was the lisp compiled or interpreted?

[paper echo]

i'll reply more in detail in a bit, but part of my benchmark was "how fast does it go if i just write naive idiomatic code without aggressive optimization". and it was sbcl, ccl was even slower

and surprisingly in sbcl the overhead in profiling seems to be mostly from parsing floats. even calling out to stdlib strtod() only improved speed by about a third compared to the native lisp parse-float package that someone already wrote and is modestly optimized

[halcyon trail]

Gotcha. Idk then, my understanding is that compiled lisp is decently fast

But I don't have first hand experience here

Also I would expect JavaScript to beat python

I mean so much effort has gone into optimizing it

[raven ridge]

And money.

[radiant garden]

well V8 is JITted so you'd expect at least some performance gains from that

[paper echo]

indeed, node would probably beat python on that benchmark too (it's on the "todo" list)

[paper echo]

interestingly cpython 3.10 also totally smokes perl 5.34 on this task

[paper echo]

wait.. python is beating nim now on my m1

what the heck, now i'm skeptical

i'm not convinced that the bottleneck is even i/o here, python might just have a ridiculously fast float parser

[dusk comet]

is parsing float a hard task?

just do eval(s)

[radiant garden]

float parsing is utter bee soup

at least once you stray off the happy path

[paper echo]

fwiw i think it's a lot easier if you don't need to parse scientific notation, right? but even so, cpython is ridiculously fast at this, to the point where i'm suspicious that it isn't cheating somehow

[rose schooner]

wait that doesn't seem right

cpython uses a 600-line function to convert a string to a float

[flat gazelle]

The question is what do the other languages use

Beating nim is especially strange since I am pretty sure a nim string has a zero-cost cast to a C string, so they may just be using strtod from libc

[rose schooner]

how is a macro checked at runtime

does cpython have some way to get the macros that were defined

[deft pagoda]

is it cached parsing

[flat gazelle]

Macros don't exist at runtime in any form

[rose schooner]

if _PY_SHORT_FLOAT_REPR == 1, cpython uses https://github.com/python/cpython/blob/main/Python/dtoa.c#L1438-L2161

else it uses https://github.com/python/cpython/blob/main/Python/pystrtod.c#L155-L308

latter actually does use strtod

[rose schooner]

ok so my cpython defines it as 1

[fallen slate]

@rose schooner :white_check_mark: Your 3.11 eval job has completed with return code 0.

1

[paper echo]

i know that nim isn't using "raw" strtod because strtod permits leading whitespace while nim doesn't. idk what the diff is

[rose schooner]

!e wait this works too ```py
import sys
print(sys.float_repr_style == "short")

[fallen slate]

@rose schooner :white_check_mark: Your 3.11 eval job has completed with return code 0.

True

[paper echo]

tested on 80 million rows, apple m1 pro

cpython 3.10.5: 1.74 seconds.
https://git.sr.ht/~wintershadows/polyglot-benchmarks/tree/master/item/running-stats-cli/implementations/python/calc.py

nim v1.6.6: 2.14 seconds. compiled with nim compile --define:release --mm:orc calc.nim
https://git.sr.ht/~wintershadows/polyglot-benchmarks/tree/master/item/running-stats-cli/implementations/nim/calc.nim

perl v5.34.1: 6.74 seconds.
https://git.sr.ht/~wintershadows/polyglot-benchmarks/tree/master/item/running-stats-cli/implementations/perl/calc.pl

i honestly have no idea how to read lines from stdin in node and i have too much other stuff to do to figure it out rn

this is the data generation script: https://git.sr.ht/~wintershadows/polyglot-benchmarks/tree/master/item/running-stats-cli/generate-data.hy

and this is a small sample of the generated data: https://git.sr.ht/~wintershadows/polyglot-benchmarks/tree/master/item/running-stats-cli/data-sample.txt

[rose schooner]

why does nim look like python

[swift imp]

Macros are just text replacement by preprocessor

[paper echo]

by design. honestly it's a great language and i think it could get really popular if a handful of high-visibility startups and some big corp started promoting it, like what happened with go

[rose schooner]

there's usually a sys attribute for every important macro in cpython

[rose schooner]

the problem might be regex

[swift imp]

Those are preprocessor directives

[rose schooner]

aren't preprocessor directives #if, #else, #endif, ... those sort of stuff

[paper echo]

well i don't expect my perl to be optimized, it's not my main language by any means. also perl's implicit float conversion works differently from the other languages and i wanted to shoehorn it to have the same behavior as python float(). the bigger surprise is cpython vs. nim

when i had done this benchmark on my older macbook pro (2020, x86) nim was beating python consistently

[rose schooner]

have you checked sys.float_repr_style to see what function cpython uses for string-to-float conversion

[paper echo]

'short'

does that matter here?

[swift imp]

Can u give an example

[rose schooner]

yeah that means cpython uses the ~600-line function

[paper echo]

oh, as opposed to strtod(). well maybe that 600 lines is heavily optimized

e.g. how long is any given codepath in that function? might be a ton of special cases

OH SHIT hahahah i know why

i forgot i compiled a .so with mypyc

[rose schooner]

#define X 124 /* macro definition by #define directive */
#if X == 124 /* #if directive */
    ...
#endif /* X != 124 */ /* #endif directive */

[paper echo]

wait let me confirm that i'm running the .py and not the .so

[swift imp]

No of the macro you are talking about, sorry should have been more specific

[rose schooner]

https://github.com/python/cpython/blob/main/Python/sysmodule.c#L2977-L2982

[fallen slate]

Python/sysmodule.c lines 2977 to 2982

    /* float repr style: 0.03 (short) vs 0.029999999999999999 (legacy) */
#if _PY_SHORT_FLOAT_REPR == 1
    SET_SYS_FROM_STRING("float_repr_style", "short");
#else
    SET_SYS_FROM_STRING("float_repr_style", "legacy");
#endif```

[rose schooner]

can check from that

[paper echo]

holy smokes mypyc did it in 1.34 seconds

[swift imp]

_PY_SHORT_FLOAT_REPR has to be a macro itself no?

Like it's telling the preprocessor what code to include

[feral island]

right, likely defined by some ./configure magic

[paper echo]

if anyone here is good at reading c, here's the generated nim code: https://paste.pythondiscord.com/acoyexiziz.c i have no idea what i'm looking at

the "interesting" loop is line 266 to 324-ish

i find it very very hard to believe that this is actually slower, unless there's a ton of nim startup overhead and i just need to test on a bigger file

[rose schooner]

that looks like cython but a lot less messier

also no comments to say what line it is??

[paper echo]

there's a bunch of other metadata along w/ the generated c code

[rose schooner]

i thought it'd give the source code for parseFloat, probably not

[paper echo]

no it doesn't, and i can't actually find it in the nim source code either

[rose schooner]

there's https://github.com/nim-lang/Nim/blob/devel/lib/system/strmantle.nim#L58-L215 but i can't find anything else

[paper echo]

the main entry point seems to be here https://github.com/nim-lang/Nim/blob/8ccde68f132be4dba330eb6ec50f4679e564efac/lib/pure/parseutils.nim#L576-L598 but i can't find the source of nimParseBiggestFloat nor can i figure out how parseFloat* is wrapped into parseFloat

[rose schooner]

i just linked the source of nimParseBiggestFloat

[deft pagoda]

maybe compile flags matter

[paper echo]

oh thanks, let me see

i get 404 on that

[rose schooner]

change main to devel i guess

[deft pagoda]

did you use release flag? and maybe try the newer memory management -d:release --mm:orc

[paper echo]

i did both of those yeah

[paper echo]

it worked with v1.6.6 which is the version i have https://github.com/nim-lang/Nim/blob/v1.6.6/lib/system/strmantle.nim#L58-L215

i also don't know how to profile nim code

or c code for that matter

any newbie tips?

let me also see what the python profiler output is, i forget

is there a profiler for cpython bytecode?

[rose schooner]

C code is hard to profile

[raven ridge]

hm. perf or callgrind or even gprof can do it.

[paper echo]

https://paste.pythondiscord.com/japivujema.py interesting line profiler output

float parsing is only 12% of the total time

this if first_line check is a nontrivial bottleneck apparently, wow

i wonder if you could "optimize" this slightly by pulling out the first iteration. the if check seems to be quite expensive

[dusk comet]

what is this: @profile?
which module provides this function with such beautiful output?

[paper echo]

!pypi line-profiler

[fallen slate]

line-profiler v3.5.1

Line-by-line profiler.

[paper echo]

hm, i don't fully trust the output here though. it said that the if first_line line was 9% of run time, but refactoring it to avoid the if there didn't make it go any faster

[feral island]

depending on how that profiler works it may have significant per-opcode overhead

[rose schooner]

never tried those so 🤷

[ruby hare]

Good evening team!
I have an important question. I was wondering what everyone thinks of Mimo Dev. Should I apply for the course they offer?

[ancient flower]

Question: How often does the python garbage collector run
If I have the following code:

with open("file.txt", "r") as f:
  line = ""
  while True:
    last_line = line
    line = f.readline()
    if line == "":
      break
    

I assume that all the strings I read into line are allocated on the heap and only the pointer is overwritten. Is this True or is the string directly freed when the first pointer is discarded?
Thanks for help 🙂

This will run on 264kB ram and 16mB flash with micropython ( so not CPython)

[feral island]

on cpython, the garbage collector only comes into play when there are cyclical references involved. Objects without cyclical references, like your strings, get deallocated once their refcount reaches 0

I'm not familiar with micropython, but I would assume it behaves similarly. I wouldn't be surprised if it just doesn't do cyclical GC

[paper echo]

this channel is about discussion of the inner workings of the python language. it sounds like you meant to ask in #career-advice?

[paper echo]

i've been wondering about refcounting, does the de-allocation happen immediately when the refcount hits 0? or are de-allocations deferred at all?

[feral island]

immediately. But this is an implementation detail that may change

[paper echo]

right, okay

so if i write del x and there are no other references to x, the interpreter immediately needs to spend a hair of a second in order to free() that PyObject and its associated pointer?

[peak spoke]

what exactly it will do depends on the object but mostly yes

and runs its destructor if it has one

[paper echo]

but that's a pretty fast operation in general, right?

[feral island]

yes, it also doesn't necessarily call free(). For some types there is a "free list", basically a pool of allocatable chunks to use for creating PyObjects

[peak spoke]

it's not uncommon for it to do io like closing resources

if they were badly managed and weren't closed explicitly

[peak spoke]

small objects will also go to pre-allocated memory iirc

[paper echo]

oh that's interesting. is that like an "arena"?

[paper echo]

that's part of invoking __del__, right?

[peak spoke]

yes

[feral island]

not entirely sure here but I think an arena is basically a bunch of undifferentiated memory that you can allocate from, while a free list contains a fixed number of slots where you can put an object

[peak spoke]

the free lists keep around cleared objects to be reused when a new one is requested

[paper echo]

i see. but the concept is similar, of pre-allocating a bunch of memory and/or reusing allocated memory

[peak spoke]

so instead of freeing the memory, it only removes the data and saves the object to be reused. But cpython also does things through arenas for the small objects I mentioned - https://docs.python.org/3/c-api/memory.html#the-pymalloc-allocator

you can somewhat see it when you print ids of something like the number from a for range loop, as they'll most likely repeat themselves

!e

for i in range(1000, 1004):
    print(hex(id(i)))

[fallen slate]

@peak spoke :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | 0x7f86a519b0d0
002 | 0x7f86a519b0b0
003 | 0x7f86a519b0d0
004 | 0x7f86a519b0b0

[paper echo]

that makes sense, very cool

[raven ridge]

Right, even for types without a free list, freeing the object will typically return stuff to a pymalloc pool, rather than calling free(). Usually.

https://bloomberg.github.io/memray/python_allocators.html explains how that works in a hopefully approachable way

[paper echo]

does cpython adjust the sizes of its allocation pools based on available memory on the system? this stuff seems like it would require a lot of platform-specific tuning

[raven ridge]

it does not, no - pymalloc has a fixed 64 size classes, and arenas are dynamically added to or removed for each of them as needed based on memory pressure

[hybrid relic]

y'all I've been trying to understand https://bugs.python.org/issue44032 - From what I understand, it folds local variables and the operation stack into one single eval stack per interpreter thread, instead of allocating new ones every time a frame is pushed onto the call stack?

[daring kestrel]

When creating PEPs, does the author just pick a random number that hasn't yet been used or how does that process work?

[peak spoke]

I believe they're assigned to the pep by the people that approve them

[daring kestrel]

Approve as in the first draft?

[feral island]

Normally it's the lowest unused number, so the next would be 697 or so. The guidance is to first submit your PEP as "PEP 9999", and then one of the PEP editors (I'm one of them) will tell you what number to use

Occasionally people use special cute numbers (e.g., PEP 404 is about how there isn't going to be a Python 2.8; all the Python 3 PEPs are are PEP 3xxx) but that hasn't been common recently

[paper echo]

it's going to be interesting if we get to pep 2900 or so and people realize that we might be running out of room before a big name conflict 😆

[feral island]

that'll take a while :))

[paper echo]

i'm ready to call it official: entirely un-optimized python is roughly equal in performance with mildly-optimized nim on my benchmark with cpython, and compiled with mypyc it's beating nim by a full third

i guess that 600-line float parsing function was worth it

[frigid bison]

Can't Nim just steal the function and implement for their own language or would that be incompatible

[paper echo]

theoretically maybe you could rip it out and put it into a standalone library, but cpython source code tends to be very specific to cpython and it'd need to be adapted at least a little bit

[hybrid relic]

anyone?

[naive saddle]

While you're active, @feral island I presume there's a better place to open issues regarding python.org the website. Do you know what repository that would be? I just opened https://github.com/python/cpython/issues/95941 and I realize cpython probably isn't the best place for it.

[feral island]

https://github.com/python/pythondotorg maybe?

[naive saddle]

Ah that seems better. Do you happen to have the power to transfer it?

Although honestly I'm not entirely sure if it's worth moving it to pythondotorg.

[feral island]

no, I don't have commit access to the pythondotorg repo. I could transfer it to mypy though

[naive saddle]

haha, of course.

[sturdy timber]

Well if someone had permissions for mypy and pythondotorg but not cpython...

[fervent cargo]

what kind of data structure is used for context managers? Is it also a stack frame object?

[grave jolt]

The with statement is pretty much just syntax sugar for a try-finally block

maybe you could elaborate what you mean?

Any object can be made into a context manager if it has __enter__ and __exit__ methods

[raven ridge]

it's literally defined in terms of a try/finally block. There is a way to write a try/finally block that is exactly equivalent to what the with statement does.

[grave jolt]

well, a try-except-finally

because it can swallow exceptions

or rather, just try-except

</nitpick>

[raven ridge]

nah, try/except/finally indeed.

https://docs.python.org/3/reference/compound_stmts.html#index-15

The following code:

with EXPRESSION as TARGET:
    SUITE

is semantically equivalent to:

manager = (EXPRESSION)
enter = type(manager).__enter__
exit = type(manager).__exit__
value = enter(manager)
hit_except = False

try:
    TARGET = value
    SUITE
except:
    hit_except = True
    if not exit(manager, *sys.exc_info()):
        raise
finally:
    if not hit_except:
        exit(manager, None, None, None)

[fervent cargo]

I thought it was another special thing with a whole new data structure, after some documentation I learned it was just try except like you said but I thought there was more to it because they were talking about a runtime manager or something like that

So the protocol for context manager are just the enter and exit methods?

but how they made sure that it run those because there is no try/catch in C

[raven ridge]

there is in Python, though

whenever the Python eval loop calls a function, it checks whether the function returned successfully or raised an exception. If it returned successfully, the eval loop moves to the next instruction. If it raised an exception, then the eval loop either jumps to a relevant except or finally block or propagates the exception outwards to the next frame.

[fervent cargo]

ok I see but how internally in C they made sure to run enter and exit in any case, I looked in the cpython implementation but I cant find the right files that deal with that

[raven ridge]

!e ```py
def foo():
with x:
pass

import dis
dis.dis(foo)

[fallen slate]

@raven ridge :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 |   1           0 RESUME                   0
002 | 
003 |   2           2 LOAD_GLOBAL              0 (x)
004 |              14 BEFORE_WITH
005 |              16 POP_TOP
006 | 
007 |   3          18 NOP
008 | 
009 |   2          20 LOAD_CONST               0 (None)
010 |              22 LOAD_CONST               0 (None)
011 |              24 LOAD_CONST               0 (None)
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/apeqimukog.txt?noredirect

[raven ridge]

look for the implementation of the BEFORE_WITH opcode, if you're looking at the 3.11 sources

[feral island]

here it is (on main): https://github.com/python/cpython/blob/0b329f4f03b3e2d603cf81c39e7c9d83da123717/Python/ceval.c#L4593

[fallen slate]

Python/ceval.c line 4593

TARGET(BEFORE_WITH) {```

[fervent cargo]

okok I'll check it, thx Jelle for the link also

[icy rivet]

i need help

[sacred yew]

#editors-ides

also it would if you provided them with a description of the actual issue in text

[icy rivet]

When I run the code, it doesn't work. It needs a Python environment, even though I downloaded it

[pale flame]

Is specifying return types of functions mandated by pep8? Do I have to do this in my code for it to be good code?

def greeting(name: str) -> str:
    return 'Hello ' + name

[quick snow]

no, type hints are not required by PEP 8

[boreal umbra]

in this case, the function is simple enough that a knowledgeable python dev would know that name should be a str and that the whole thing returns a str. though as the complexity of the code increases, type hints are very helpful.

[pale flame]

Thanks.

[manic hearth]

Mandated? No. Desirable and helpful, absolutely.

The question is, "Why and how desirable and helpful?"

Actually, no, the question is, "What is the reason you are curious about PEP8's stance on type hinting?"

[paper echo]

!pep 8

[fallen slate]

**PEP 8 - Style Guide for Python Code**

Link

Status

Active

Created

05-Jul-2001

Type

Process

[paper echo]

doesn't say anything about type hints as far as i know

certainly wouldn't have when it was written

although a type hints style guide might be an interesting revision, or separate pep

[paper echo]

the function is simple enough that a knowledgeable python dev would know that name should be a str
the problem is that being knowledgeable in general does not scale in software development, neither across time nor space

i really do like dynamically typed languages for a lot of things, but i also have vanishingly little interest in writing production-grade applications without at least some kind of static type checking (or other forms of contract validation like clojure spec)

i've worked on too many applications and too many apis where the type of a parameter like user is completely opaque, making it difficult to use, let alone construct for test cases

not to mention that docs tend to have too few anchors and too few cross-links, making it extremely painful to search for such things

(see e.g. PyMongo)

the high cognitive load of an untyped python codebase is often a "death by a thousand cuts" situation

[swift imp]

Admittedly, static type checking is something I haven't integrated into the cicd pipeline for my works code base. It's a closed system and getting new pkgs is hard. We are still on 3.7 . I would like to have pyrite or mypy run on our code and start type annotating

I know some places where it would save me big time

[paper echo]

i was on a team that literally saved a codebase (and probably a company) by adding type hints, on python 3.5 around 2016

adding type hints was a major productivity multiplier in allowing us to refactor the code and start adding tests

otherwise i don't think we ever would have been able to do it

pycharm adopting static type checking early on helped too

i don't blame you, it's very hard to convince stubborn senior engineers that new tools are good. and i don't blame them either, up to a point, because their experience has taught them to be risk averse.

[swift imp]

That's the other thing. Pycharm is banned at my job bc it's a classified dod system and jetbrains has Russian ties

[paper echo]

hopefully mypy is not sanctioned!

[swift imp]

I think we'd have to get vscodium and pyright as the best we can do

[paper echo]

that's pretty good tbh

a lot of professionals use that now

you also don't need editor integration

[swift imp]

Yeah it's just lame that Spyder doesn't support type hints

[swift imp]

No but it's super nifty

[paper echo]

spyder doesn't? that's a bit surprising

i recently installed spyder for the first time in years, it seemed markedly improved over what it was, until it started crashing

is sublime text allowed? maybe uncle sam can fork out the $70 license, then you can use sublime-lsp and sublimerepl for a pretty decent experience

[feral island]

pyright in vscode is probably the best experience for typing in python at the moment

pycharm's typechecker isn't great

[paper echo]

yeah it's inconsistent w/ both mypy and pyright

[swift imp]

I would like vscode and pylance but neither are FOSS

So have to use vscodium and pyright

[feral island]

pylint is something totally different

and it's open-source

[swift imp]

I meant pylance

[feral island]

ah

[swift imp]

Apparently only difference between vscodium and vscode is the telemetry data gathering

But pylance isn't open source and doesn't support pylance

[deft horizon]

I looked into this a while ago, because I needed a function from variable name to pretty-sure-its-this-type (or don't-know, ofc). Big table of results at https://github.com/HypothesisWorks/hypothesis/issues/3311
The resulting function works for more than 10% of all frequency-weighted parameter names!

[paper echo]

heh that's pretty funny

fuzzily-deduplicated corpus of Python code
that's a hell of a dataset @deft horizon , is that publicly available / are you willing to publish it?

(im not sure what copyright law would have to say about it)

[deft horizon]

It's not public, sorry, but I'd recommend looking at ghtorrent; exact-dedupe is probably sufficient for most purposes.

[verbal escarp]

which is why adding type annotations is the first step when i start working on a code base of someone else base if it doesn't already have them - makes code much more maintainable in general

if i had to interview someone for a job, type annotations probably would be on my list of questions. it baffles me time and again that some people claim to be "python pro" and never having used or even heard of annotations

[grave jolt]

config: dict[str, Any] moment

or rather... Any moment 😛
although to be fair, people aren't evil and probably don't anyfy the code intentionally

[paper echo]

mypy json support when 😠

[feral island]

next release

for recursive types

I don't know what JSON support means

[paper echo]

recursive types. and yes!! that's fantastic news

[swift imp]

I think he means that. Imagine giant and nested TypedDict definitions

[radiant garden]

it means you hopefully have a big model encompassing little models

[rose scarab]

pycharm is russian?

😦

uhh probaly shouldnt tell us about your dod system though

[quick snow]

JetBrains is Czech, they did have R&D in Russia before February though

[dusk comet]

what is "dod system"?

[dusk comet]

jetbrains was founded by three russian devs

[feral island]

Department of Defense

[frigid bison]

Funnily enough the disabled all licenses bought by Russians

[uncut sierra]

why not having a constructor dict(iterable1, iterable2) instead of need to used dict(zip(iterable1,iterable2)) ...

[raven ridge]

because it's not a very common use case, I imagine. I don't think I've ever needed that.

[uncut sierra]

Im guess im weird then ... I combined many many times , 2 lists list of keys and list of value and want a dict from them.... lol

[quick snow]

I think if I would read dict(foo, bar) I might expect both of the arguments to be mappings themselves; for it to be equivalent to {**foo, **bar}/foo | bar. I think it's good that you have to be a tiny bit more explicit there.

[rose schooner]

imo dict(foo, bar) as keys-values arguments is what i'd expect out of all other alternatives

[quick snow]

I wouldn't be extremely surprised if it was keys and values, either. But I'd have to think about it.

I think the dict constructor is overloaded enough as it is..

[verbal escarp]

wait what? how?

[feral island]

Ivan came in and did it

[verbal escarp]

ivan?

[feral island]

Ivan Levkivskyi, mypy core dev who previously implemented Protocols among other things

[verbal escarp]

okay, nice

how are recursive types implemented then?

anything weird?

and is it compatible with fastapi and friends?

[feral island]

basically by making TypeAliases a first-class kind of type

previously mypy would always fully expand TypeAliases in its internal representation

which obviously doesn't work with recursive aliases

no idea about fastapi or what special typing needs it might have

[verbal escarp]

at least i know that deferred evalution was a problem

but TypeAliases could indeed be a solution to that problem, interesting

[paper echo]

wasn't there some talk about changing how deferred evaluation worked

[wet lava]

Hi! Is anyone interested in collaborating on https://discuss.python.org/t/simplify-lru-cache/18192 / https://github.com/faster-cpython/ideas/issues/450 ?

[wet lava]

I haven't really contributed to CPython before, so would appreciate anyone with a bit of experience. (Or just a willingness to also learn.)

[uncut sierra]

from { k:v for k,v in zip(ks,vs)} we optimise to (hopefully) to dict(zip(ks,vs)) ... maybe under the cover the zip object is not use at all and the dict constructor use directly ks and vs for building the mapping k:v ... that I dont know I never open the cover P

[frigid bison]

When/why does Python use absolute jumps over relative jumps

JUMP_FORWARD/JUMP_ABSOLUTE

[rose schooner]

pretty sure python uses more relative jumps now

anyways you can find out by looking at the source code in https://github.com/python/cpython/blob/main/Python/compile.c

nvm i can't find an example of absolute jumps

[feral island]

I think 3.11 may have gotten rid of them completely

[unkempt rock]

Is anyone learning data science from the IBM course?

[boreal umbra]

@unkempt rock your message is off-topic for this channel. try #data-science-and-ml

[naive saddle]

https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types never knew how complicated defining custom implementations of numeric operations could get was until I have to read this madness so I know what I'm doing to add support for __pow__ to mypyc 🥴

[hybrid relic]

is there a way to set py_tpflags_basetype on a particular class to make it uninheritable from python alone?

have tried @typing.final() but that's just syntax level unfortunately

[peak spoke]

you can raise an error in init_subclass to have an error when subclassed

[quick snow]

!e

from ctypes import c_int

flags_offset = 168
py_tpflags_basetype = 1024

def final(cls):
    c_int.from_address(id(cls) + flags_offset).value ^= py_tpflags_basetype
    return cls

@final
class Foo:
    pass

class Bar(Foo):
    pass

[fallen slate]

@quick snow :x: Your 3.11 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 14, in <module>
003 | TypeError: type 'Foo' is not an acceptable base type

[quick snow]

I think this is not part of the stable ABI though, so it may break in future Python versions.

[sour dagger]

Hello everyone, so Idk where is the right place to post this question. I just wanna ask everyone where or how can I find tech internships. I am looking for a company not too fancy, just enough to get my foot out the door.

[quick snow]

#career-advice, probably

[hybrid relic]

ah i see

thanks!

i don't mind it not being compatible across python versions, so this is an acceptable solution for me

[hybrid relic]

strange why the typing module doesn't already do this though

[flat gazelle]

because typing is about static checks, not about ensuring the types hold at runtime

[hybrid relic]

fair point

but this seems like a pretty useful thing to have regardless since it's already there

[quick snow]

Does it? What for?
I've never understood the need for this (other than technical reasons like how you couldn't subclass builtins in the past)

[quick snow]

So... if there is the sq_concat as well as the nb_add subslot, what happens if both are defined? Can you make an object that behaves differently when numerically added than when sequence-concatenated, and is there any way to enforce either of them from within Python (ignoring ctypes)?

[rose schooner]

to force nb_add, just normally add
to force sq_concat, assuming the arguments are self and other:
other's type must not be a subclass of self's type
nb_add has to return NotImplemented, and other must not have an nb_add slot

[rose schooner]

if both are defined nb_add is just used

source code here: https://github.com/python/cpython/blob/main/Objects/abstract.c#L1063-L1080

[grave jolt]

unless you use operator.concat I suppose

[quick snow]

Thanks!

[wet lava]

Well, even the standard library doesn't use them.

[flint mantle]

I love annotations — I use them every time on function arguments

[tiny whale]

what's the right way to check if a type hint is a union? Originally, I was using typing.get_origin(...) is Union. e.g:

>>> typing.get_origin(Union[int, str]) is Union
True

But this breaks in python 3.10 with the new union types, see the difference:

>>> typing.get_origin(Union[int, str])
typing.Union
>>> typing.get_origin(int | str)
<class 'types.UnionType'>

what's a predicate I can use that would work for both Union types and | without explicitly handling both cases?

by explicitly handling both cases, I mean this. It works, but I'm hoping there's some common base that both Union and | syntax can be checked against

def is_union(type_hint):
    return typing.get_origin(type_hint) in (Union, UnionType) 

[gleaming spoke]

Are there any linters that identify syntax and language tokens and provide links to documentation for that language feature?

[quick snow]

That doesn't sound like it's the job of a linter. But many editors/IDEs have that feature or plugins that provide it.

[boreal umbra]

in pycharm, you can control click on things to jump to where it's defined, even if that's in library code. but that's a conversation for #editors-ides.

[royal mason]

Hello lads, reviewing some proxy libs. any thoughts on interesting ones you have worked with?

[grave jolt]

what do you mean by proxy? the design pattern, or the network thingy?

[royal mason]

lol, the network thingy

[grave jolt]

I recently had the pleasure of writing a (somewhat dummy, but technically compliant) SOCKS5 server for testing, if that counts

[royal mason]

I am thinking of having an api cal being made in the middle of a request

yes, how was that compared to just using proxies with requests?

or better yet, you used the sockss server as your proxy? @grave jolt

[grave jolt]

Yeah, I had the real client (asyncio.ClientSession in my case) connect to it

although if you're doing something real, you should use some production-grade server

[royal mason]

yes, that already exists.

it's just it's a little funky, and want to replace it with something that's better to work with.

[south hearth]

here is the error

XXX lineno: 37, opcode: 0
Traceback (most recent call last):
  File "<frozen main>", line 46, in <module>
  File "<frozen incidentdetector>", line 37, in <module>
SystemError: unknown opcode

and here is the dis
https://paste.pythondiscord.com/imaxiwosus
where it went wrong?

[stable grail]

this is not the right c hannel for this

#❓｜how-to-get-help

[south hearth]

oh ok

[livid iron]

good

[glad bone]

He there 🙂 Do you think it would be a valid masters thesis to work on some pep? I study at Dtu (denmark copenhagen) and i would very much love to help out with python. However i never so far worked on a pep, just read bunch of them throughout the years. I now, that there are many peps which were deferred because of lack of funding. If you think some topic would be for approximately half year or have any suggestions please let me know 🙂 all feedback is more than welcome.

[paper echo]

possibly, this might be a better question for your advisor

[dusk crown]

is the more simple way then that ？

[quick snow]

!e not the right channel for this, but

lst = ["aa", "bb", "cc"]

print(" OR ".join(f"from:{element}" for element in lst))

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

from:aa OR from:bb OR from:cc

[dusk crown]

where is the good channel

[quick snow]

#❓｜how-to-get-help in general for normal questions

[dusk comet]

!e

lst = ['aa', 'bb', 'cc']
print(' OR '.join(map('from:'.__add__, lst)))

[fallen slate]

@dusk comet :white_check_mark: Your 3.11 eval job has completed with return code 0.

from:aa OR from:bb OR from:cc

[grave jolt]

!e
code golf moment

lst = ["aa", "bb", "cc"]

print(" OR ".join(map("from:%s".__mod__, lst)))
print(" OR ".join(map("from:{}".format, lst)))
print(" OR ".join(map("from:".__add__, lst)))

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | from:aa OR from:bb OR from:cc
002 | from:aa OR from:bb OR from:cc
003 | from:aa OR from:bb OR from:cc

[grave jolt]

!e
oh hey here's a thing

lst = ["aa", "bb", "cc"]
print(" OR ".join(["from:{}"]*len(lst)).format(*lst))

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

from:aa OR from:bb OR from:cc

[feral island]

surprisingly that's the fastest

2.4 µs ± 454 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [20]: %timeit " OR ".join(map("from:{}".format, lst))
1.26 µs ± 262 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [21]: %timeit " OR ".join(map("from:".__add__, lst))
875 ns ± 222 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [22]: %timeit " OR ".join(["from:{}"]*len(lst)).format(*lst)
804 ns ± 202 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

probably from avoiding the map() call

[quick snow]

What about mine?

[feral island]

though this one is even better ```In [23]: %timeit " OR ".join(["from:{}" for _ in lst]).format(*lst)
787 ns ± 6.09 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

(not endorsing this sort of micro-optimization when it doesn't actually matter though)

[grave jolt]

Wait... how?!

[feral island]

my guess is anything that needs to actually parse the string (format or %) pays some cost, and so does anything that needs to do map()

[feral island]

fastest yet ```In [25]: %timeit " OR ".join(f"from:{element}" for element in lst)
780 ns ± 3.28 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

[grave jolt]

try " OR ".join([f"from:{element!s}" for element in lst])

[feral island]

even faster with a list ```In [26]: %timeit " OR ".join([f"from:{element}" for element in lst])
557 ns ± 4.61 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

oh let me try !s too

[quick snow]

it skips __repr__ then?

[feral island]

!s is slower actually

[grave jolt]

yeah because if the argument is not a list or tuple, it gets consumed into an internal array

[feral island]

yes, and that's more indirection that you pay for

603 ns ± 5.43 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

even better: ```In [29]: %timeit "from:" + " OR from:".join(lst)
173 ns ± 0.176 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

[grave jolt]

that's different though innit?

[feral island]

from:aa OR from:bb OR from:cc

In [32]: print(" OR ".join([f"from:{element}" for element in lst]))
from:aa OR from:bb OR from:cc

nope

[grave jolt]

ahhh

yeah it's not immediately obvious how it works xD

[radiant garden]

i think it's pretty understandable why the one doing fewer string formatting calls is fastest

[feral island]

yes, it's also just doing the fewest operations

[grave jolt]

```
In [1]: %timeit "from:aa OR from:bb OR from:cc"
4.94 ns ± 0.0459 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)

[feral island]

ok you win

[rose schooner]

man why does everyone else load a constant with only ~5 ns runtime while i need ~7 ns

[dusk comet]

It takes 7ns on my pc

[grave jolt]

$ git gud
git: 'gud' is not a git command. See 'git --help'.

The most similar command is
    gui

[rose schooner]

man why does everyone else know how to use git while i have to edit on the website

[grave jolt]

I mostly use some gui nowadays

[dusk comet]

I love sublime merge, but also i know how to use git
I think you cant properly use guis if you dont understand git cli

[frigid bison]

I think it's the other way around

The GUI shows you features that I never would've discovered in the cli

It also makes it way less easy to fuck up

[south hearth]

why dont the python developers make a pyc decompiler themselves lol

[boreal umbra]

do you think there should be a pyc decompiler? what would you do with it?

python bytecode is unspecified, so you'd potentially need a different decompiler for every Python version (including bugfix releases), for every OS.

[feral cedar]

also if you produce the pyc you have the source

[south hearth]

most malwares dont come with the source?

bundled in exe

with pyc

anyways not the point

[south hearth]

yeah i agree on that

but the thing is its better to keep it up with themselves rather than any third party maintaining a decompiler and changing everything to keep up with newer versions of python

i mean think about it, if someone from the python developers makes a change they could automatically update the decompiler with that change as well

[rich cradle]

that's a lot of maintenance work for minimal gain

[rich cradle]

then don't run things you don't trust?

[south hearth]

again not the point im trying to discuss here, but well actually "that's a lot of maintenance work for minimal gain" makes sense tbh

[frigid bison]

I agree with the others, It's not python's responsibility to make a decompiler. Like with all the other languages that have a custom bytecode (.net, Java, lua) some other third party made a decompiler, just for python there hasnt been a really stable one. rocky did a good job with https://github.com/rocky/python-decompile3 but it still isn't nearly as stable as the ones for other languages, and it lacks support for 3.9 and higher. We'll just have to hope that someone makes a really good one, or update the one from rocky

[south hearth]

working on making it more stable currently tbh, thats why i thought of it

[rich cradle]

i honestly don't see the point in a decompiler in the first place

[frigid bison]

reverse engineering purposes

[rich cradle]

if someone's giving you a pyc without a source file, that seems like something to be suspicious about and not run

[frigid bison]

malware analysis, recovering of lost files, ... there are multiple reasons it could be useful

[raven ridge]

The question seems a lot like "why don't the authors of GCC also build a decompiler"

[south hearth]

well yep my bad, was just frustrated at that point xD

[frigid bison]

well no since it's way harder to decompile assembly than for example the bytecode python generates

[south hearth]

still it shouldnt be the language devs duty to make a decompiler, i agree with that

[frigid bison]

of course

[flat gazelle]

I don't think a standard decompiler could be both simple enough that is a thing that can reasonably be maintained with the release cycle, and complex enough that it doesn't take about 100 lines of code to rewrite the bytecode into something it cannot decompile.
In the end, a decompiler usable for most of the usecases of a decompiler has to be aware of various obfuscation methods et al, and a standard decompiler can't reasonably do that.

[frigid bison]

that's not the job of the decompiler, it should only take care of bytecode written by python. It will be the task of the deobfuscator to clean the obfuscation and generate bytecode that the decompiler can handle

[south hearth]

to be really honest that idea came to my mind cuz they were already maintaining dis

but well it is what it is

[paper echo]

why does Exception accept arbitrary args and kwargs? is there a historical reason for it? was there some intended use case for this that people don't really use now?

i was really surprised to see that it didn't just take a single message positional arg

[sour thistle]

if I try to pass in a keyword argument I get py TypeError: Exception() takes no keyword arguments besides that, I would expect for .args to be in order to to allow for users to pass back values, like in StopIteration?

[quick snow]

IIRC in Python 2 there was a message (which was available as an attribute).

[paper echo]

right, but wouldn't you think that those args should be somewhat specialized?

instead of just an arbitrary sequence of things that could be literally anything

[swift imp]

Is it like print? It'll report it all passed args

[rose schooner]

i mean you can do this ```py

raise Exception("a", "b", "c")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception: ('a', 'b', 'c')

like multiple messages at once maybe

[flat gazelle]

I would assume the idea is for subclasses to not have to write their own repr and value-storing logic. You just throw things into args and be happy

[frigid bison]

https://docs.python.org/3.9/library/ast.html#ast.Attribute

Here in the documentation it says that value is typically a Name node

what file can I check to see what the other options are?

[south hearth]

anyone knows what BEGIN_FINALLY, CALL_FINALLY and CALL_FUNCTION_EX does?
if so a docs would be helpful

[frigid bison]

https://docs.python.org/3/library/dis.html

its all listed in the documentation for dis

select the correct python version though

[rose schooner]

value can be any expression

[frigid bison]

yea theoretically but not in practice though?

[rose schooner]

wdym by "not in practice"

[frigid bison]

well since its an attribute it can't be a constant for example

since you cant do module.1

it also cant be an if statement etc

I dont really see what else it could be other than Name

[rose schooner]

that's a statement not an expression

[rose schooner]

you're looking at attr there

[frigid bison]

my bad

[rose schooner]

value is the object being attributed
attr is the attribute

[frigid bison]

yea

but so what else can value be other than Name

[sacred yew]

any expression

eg

In [4]: print(ast.dump(ast.parse('(2+2).b', mode='eval'), indent=4))
Expression(
    body=Attribute(
        value=BinOp(
            left=Constant(value=2),
            op=Add(),
            right=Constant(value=2)),
        attr='b',
        ctx=Load()))

[frigid bison]

ah I see

thanks

[frigid bison]

how is the walrus operator in the docs for 3.9?

[boreal umbra]

Well it came out in 3.8

[frigid bison]

it's giving me a syntax error though

[boreal umbra]

You might need the parentheses in that context. Not sure.

[frigid bison]

oh you were right

[boreal umbra]

!e (x := 4)

[fallen slate]

@boreal umbra :warning: Your 3.11 eval job has completed with return code 0.

[No output]

[frigid bison]

I thought it was a 3.10+ feature

nice to know

[boreal umbra]

Nope. Fun fact: the walrus operator pep is the reason Guido stepped down as the bdfl

[frigid bison]

lmao

[boreal umbra]

It was quite contentious, and he didn't want that much contention directed at him in the future

[frigid bison]

it became too flexible for him?

[boreal umbra]

No, he wanted it. Otherwise it wouldn't be part of the language now. At the time, he had the final say on all language changes.

[frigid bison]

oh I see

[boreal umbra]

Whereas now, there's a five member steering council

[rose schooner]

how was this created

[boreal umbra]

!pep 13

[fallen slate]

**PEP 13 - Python Language Governance**

Link

Status

Active

Created

16-Dec-2018

Type

Informational

[boreal umbra]

as for how everyone came to agree on it, I'm not sure.

[raven ridge]

A community discussion when Guido decided he was going to step down as BDFL to decide what the new process would be, followed by a yearly nomination process amongst the core devs to nominate people for steering council elections and a yearly vote amongst the core devs to select which nominees will become that year's steering council

[spark magnet]

there were a half-dozen or so different proposals about what the new governance model should be. There was a pep for each, with voting to decide which to use.

[boreal umbra]

Did any involve electing a new bdfl?

[unkempt rock]

anyone is there who can help me as a mentor

[radiant garden]

https://peps.python.org/pep-8010

All of them are under pep 8000

[wheat pike]

hi, I got directed here via one of the help channels: #help-croissant message

I'm somewhat new to python, and I'm just trying to figure out the ins and outs of structural pattern matching in the language. I was trying to understand how to match against str rather than say the literal "foo". Eventually found some documentation about the built in types here:

https://docs.python.org/3/reference/compound_stmts.html#class-patterns

It has this paragraph:

These classes accept a single positional argument, and the pattern there is matched against the whole object rather than an attribute. For example int(0|1) matches the value 0, but not the values 0.0 or False.

However, it would appear that this paragraph is incorrect. Counterexample:

!e

match False:
    case int(0):
        print("this is bad")
    case _:
        print("this is good")

[fallen slate]

@wheat pike :white_check_mark: Your 3.11 eval job has completed with return code 0.

this is bad

[wheat pike]

am I just looking at the wrong thing, or does python silently let 0 and 0.0 match against False?

under all circumstances

and the documentation for this feature is simply incorrect?

[quick snow]

On first glance: docs appear to be wrong. match first checks that False is an int, which it is, and then compares it with 0, which is True.

[wheat pike]

surely it should short-circuit?

its not an int... move on

[quick snow]

But is is an int.

[wheat pike]

i mean... yes in memory of course it is

[quick snow]

!e

print(isinstance(False, int))

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

True

[wheat pike]

oh really

sigh

ok, so the docs are just wrong here

[quick snow]

bool is a subclass of int

[swift imp]

bool is subclass of int

Dammit

How are the docs wrong?

[wheat pike]

and in fact even type checkers like pylance have it wrong

they incorrectly narrow to int when you do int(0) on a bool | int

the docs seem to explicitly say that int(0) won't match agianst 0.0 or False

and type checkers do the same

[swift imp]

How do you know it's not evaluating the cast and the case is actually 0

[wheat pike]

this is how pylance sees it

def foo(v: bool | int):

    match v:
        case str():
            v # Never
            pass
        case int(0):
            v # int
            pass
        case _:
            v # bool | int
            pass

i guess, this isn't unsafe

at least in terms of being able to get hold of relevant methods and such

because False is 0

but the docs suggest that I can make a runtime distinction between False and 0

[swift imp]

Match by value and put False first

[wheat pike]

sure, that is a pragmatic solution

the docs are still wrong though

[quick snow]

Then let's check if there's a bug ticket already and make one otherwise.

[swift imp]

Yeah, I see what u r saying

[wheat pike]

I'm not really interested in solving a specific problem here, I just want to understand the feature

[swift imp]

Def issue worthy

[wheat pike]

also, its pretty inconsistent with this:

!e

match 0:
    case False:
        print("this is bad")
    case _:
        print("this is good")

[fallen slate]

@wheat pike :white_check_mark: Your 3.11 eval job has completed with return code 0.

this is good

[wheat pike]

so apparently 0 is not False, but False is 0

so the distinction exists in the language, depending upon which side of the match expression you are on

so it feels like what was documented was what was intended

but the implementation is wrong

[quick snow]

This is probably just special-cased keywords in case

[wheat pike]

where do issues like this get raised?

I'm new to python - (ish) - I haven't never used to long enough to remember it

I'm guessing as its python is going to be some mailing list

[swift imp]

Github

https://github.com/python/cpython/issues

All issues are supposed to go through GitHub now iirc

[wheat pike]

oh ok, that's more familiar

[quick snow]

https://github.com/python/cpython/issues/96359

[dusk comet]

bool|int is the same as int

[dusk comet]

You are wrong
int(0) and 0 have different meanings in patma

[amber nexus]

!e

class SuperClass:
    def __init__(*we_shall_ignore):...

    def __call__(cls):
        print("Hello World!")

instance = SuperClass()

class Something(instance):
    ...

Something()
Something()

[fallen slate]

@amber nexus :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | Hello World!
002 | Hello World!

[amber nexus]

Can someone explain why this occurs

It uses the class of the instance as a superclass for Something

Why is this behaviour implemented instead of just being like "you can't subclass this"

nevermind all good now

I get it

#python-discussion message just in case anyone's curious, conversation between me and vivax about it starts here

With the tldr being

This isn't unusual behavior, this is just what always occurs, but most of the time you inhert from a class where the metaclass would be type, so nothing really special occurs, in this case our 'metaclass' of instance is just the SuperClass

[quick snow]

!e You can also inherit from anything that doesn't complain when being called with three arguments.

class wat(zip("wat", "wat")): ...

print(next(wat))

[fallen slate]

@quick snow :white_check_mark: Your 3.11 eval job has completed with return code 0.

('w', <zip object at 0x7f9e107bf700>, '__module__')

[quick snow]

(Same reason)

[amber nexus]

@hasty helm no thanks

[hasty helm]

Okay but am not forcing you guys beside it a legit stuff, if you ain't wanna do this online business yo I don't have problem with that

[amber nexus]

Anything that involves crypto and messaging a random person on telegram is just... no. And we don't allow advertising anyhow

[hasty helm]

Okay we are not advertising any how you might not know about the crypto trading but bear in mind that orders know about the crypto trading Investment so stop explaining you issues to me

[boreal umbra]

wtf

[quick snow]

More precisely, its type has to be callable with three arguments. The only builtins that I know that can do this are map and zip :)

[boreal umbra]

that is, a callable with three parameters. and what are they?

[dusk comet]

Slice

[boreal umbra]

(I had to be pedantic, because I almost said "what fills those three arguments?", and that's wrong.)

[dusk comet]

def type(typename: str, bases: tuple[type, ...], namespace: dict[str, Any]) -> type: ...

[quick snow]

Nice, I missed that.

[boreal umbra]

what's ns?

[quick snow]

Namespace

[boreal umbra]

so is that going to be globals() for the namespace in which it's called?

[quick snow]

The result of executing the class body, usually

Like {"__init__": ...} etc.

[dusk comet]

No, it is dict with all methods and class variables of your class

[boreal umbra]

oh okay

actually, now I remember, I've called type like this before to create some mock objects

but I didn't know what relationship it had with the class X(...): statement

[quick snow]

I think builtins.__build_class__ is responsible for doing that. You can have quite a bit of fun by overwriting that.

!pypi spelcheck

[fallen slate]

spelcheck v0.3.1

Automagically spellcheck your classes

[quick snow]

Like this ^

[dusk comet]

!d types.new_class

[fallen slate]

types.new\_class

types.new_class(name, bases=(), kwds=None, exec_body=None)```
Creates a class object dynamically using the appropriate metaclass.

The first three arguments are the components that make up a class definition header: the class name, the base classes (in order), the keyword arguments (such as `metaclass`).

The *exec\_body* argument is a callback that is used to populate the freshly created class namespace. It should accept the class namespace as its sole argument and update the namespace directly with the class contents. If no callback is provided, it has the same effect as passing in `lambda ns: None`.

New in version 3.3.

[quick snow]

That just looks like a convenience function which is not actually used by Python internally

[dusk comet]

I read somewhere that it implements all behaviour of class statement

class name(*bases, **kwds): exec_body

[quick snow]

At least patching it (or its __code__) doesn't seem to change anything. Patching __build_class__ does, and it would be weird to have two different hooks

[dusk comet]

Is *bases syntax allowed in class statement?

[grave jolt]

!e

xs = [str, object]
class Foo(*xs):
    pass

yeah

[fallen slate]

@grave jolt :warning: Your 3.11 eval job has completed with return code 0.

[No output]

[dusk comet]

Yes, but interpreter do some extra steps to create class, and these steps are not exposed as python functions

[dusk comet]

Cool, ty

[quick snow]

!e doesn't seem to be called at all

import types
types.new_class.__code__ = (lambda: "nope").__code__

class Works:...

[fallen slate]

@quick snow :warning: Your 3.11 eval job has completed with return code 0.

[No output]

[dusk comet]

If you have no metaclasses and bases with mro_entries, you can use type to create class
Instead you should use types.new_class

[quick snow]

Help on built-in function __build_class__ in module builtins:
                                                              __build_class__(...)
    __build_class__(func, name, /, *bases, [metaclass], **kwds) -> class

    Internal helper function used by the class statement.

[dusk comet]

Why new_class's argument exec_body is function? Isnt more convenient to pass mapping to populate namespace?

exec_body=lambda ns: ns.update(some_mapping) - i used this code several times to just populate namespace with some entries

[quick snow]

Because the metaclass could return an object (from __prepare__) that is not a dict

[dusk comet]

I propose: make exec_body argument of types.new_class either callable or mapping
If it is mapping, use it as ns.update(exec_body), if it is callable do exec_body(ns)

I think it is very small and not breaking change. It would make some code shorter and cleaner

[quick snow]

I still don't understand when you'd actually use this; like what advantage it has over just calling the metaclass

[dusk comet]

If you know metaclass you can call it
If you have arbitrary list of bases you should calculate metaclass and then call it. types.new_class does this automatically

[queen crane]

!code

[fallen slate]

Here's how to format Python code on Discord:

```py
print('Hello world!')
```

These are backticks, not quotes. Check this out if you can't find the backtick key.

[wheat pike]

@quick snow thanks for your patience earlier btw

[quick snow]

Thank you for finding the issue! It has been merged, by the way https://github.com/python/cpython/issues/96359

[wheat pike]

woop

[thorny lava]

Hello. I'm implementing hot reload as part of a utility (https://gitlab.com/skeledrew/fletil/-/blob/main/fletil/hot_reload_and_restart.py#L35). However I find that reloading indirect imports isn't working (I'm using Python 3.10.6). Example fails at 2nd to last assert:

from pathlib import Path

def test_indirect_reload():
    src = """
import tests.indirect

def callit():
    return tests.indirect.callee()
    """
    src1 = """
def callee():
    return 24
    """
    src2 = """
def callee():
    return 42
    """
    direct = Path(__file__).parent / "direct.py"
    direct.write_text(src)
    indirect = Path(__file__).parent / "indirect.py"
    indirect.write_text(src1)

    # import importlib
    import time
    import sys
    from types import ModuleType
    import tests.direct

    assert tests.direct.callit() == 24
    time.sleep(0.5)
    indirect.write_text(src2)
    # _ = importlib.reload(sys.modules["tests.indirect"])
    mod_name = "tests.indirect"
    mod = ModuleType(mod_name)
    ns = {}
    exec(src2, ns)
    sys.modules[mod_name] = mod
    assert ns["callee"]() == 42
    {setattr(mod, name, attr) for name, attr in ns.items()}
    assert sys.modules["tests.indirect"].callee() == 42
    setattr(sys.modules["tests.direct"], "tests.indirect", mod)
    mod = ModuleType("tests.direct")
    ns = {}
    exec(src, ns)
    {setattr(mod, name, attr) for name, attr in ns.items()}
    sys.modules["tests.direct"] = mod
    assert ns["callit"]() == 42
    assert sys.modules["tests.direct"].callit() == 42
    direct.unlink()
    indirect.unlink()
    return

I'm wondering if it's related to this question (https://stackoverflow.com/questions/71544388/understanding-pythons-importlib-reload) and connected bug issue, but it seems to have been resolved. Any ideas of where I may be going wrong or how I can resolve?

Previous discussion at #help-croissant message

[fallen slate]

fletil/hot_reload_and_restart.py line 35

# TODO: reload ALL modules at the path!```

[thorny lava]

Resolved: found and replaced the old instances via gc.get_referrers

[paper echo]

if you get this working... can you let me know? hot-reloading would be very useful in my workflow

[thorny lava]

Yep, I did. Still cleaning up a bit, but basically:

    old_i = sys.modules[mod_name]
    mod = ModuleType(mod_name)
    ns = {}
    exec(src, ns)
    {setattr(mod, name, attr) for name, attr in ns.items()}
    i_refs2 = gc.get_referrers(old_i)

    for ref in i_refs2:
        if isinstance(ref, dict):
            for key, val in ref.items():
                if val is old_i:
                    ref[key] = mod

[feral island]

I never looked into how gc.get_referrers works. Does it scan over the entire space of allocated objects?

seems like it does looking at gcmodule.c

[thorny lava]

Yep. And gets a list of anything pointing to the given objects

[paper echo]

this still doesn't cover from foo import x right?

[thorny lava]

No. Just import foo. For that a deeper search would be in order. It's the same core though

[paper echo]

cool, if this becomes a library or tool that you publish i will gladly be an early adopter

[thorny lava]

OK. I actually wasn't thinking of taking it too far at this time. For now I'm just working on some tools for https://flet.dev. But I do have another project that'll likely require a full impl

[quick snow]

It also doesn't handle the case where the referrer isn't a dict. It can be a list (that one is easy), tuple, method, ...

[thorny lava]

Nope. For my use case (hot reloading modules), relevant module objects wouldn't usually be found in anything other than a dict. But I got the original solution here, which covers more cases: https://gist.github.com/anonymous/74ae8e03a55a6a3ea533

[quick snow]

Thats.. an interesting way to replace it in tuples. I guess it's easier on the eyes than the ctypes version.

[thorny lava]

Maybe there was some Py2 reasoning behind it. Or could be that's what came to the person's mind at the time

[tacit hawk]

Does pip ignore python files on installation? I am trying to install from src distribution, the files are included in the package but excluded on installation. They are all files which include : character on their name.

[raven ridge]

are you on Windows? Filenames can't contain : on Windows, I believe

[tacit hawk]

Ah thats true

I built on linux

And tried installing on windows

[spark magnet]

why do the file names have colons in them?

[tacit hawk]

I tried to namespace django commands

namespace by app

[paper echo]

isn't that what python packages and namespace packages are for?

or are you talking about some non-python support files that are named with : in them?

[tacit hawk]

Django takes filename as command name

It must be unique in entire project

If 2 apps have same command name they will override each other

[paper echo]

why not . though?

[tacit hawk]

: was just the first that came in mind. I replaced by underscore.

[paper echo]

@tacit hawk you can also use a - since that's not a valid character in a python module name

[olive marsh]

Is metaclass type of type of an object (type(type(obj)))?

[native flame]

can you rephrase your question

[olive marsh]

class Meta(type):
  pass 

is metaclass type of a type of object? as type is also an object

[grave jolt]

a metaclass is the class of a class

type is an example of a metaclass.

When you create a class, by default the class will be an instance of type.

!e

class Foo:
    pass

print(Foo)
print(type(Foo))

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <class '__main__.Foo'>
002 | <class 'type'>

[grave jolt]

!e
But you can choose a different metaclass:

from abc import ABCMeta, abstractmethod

class Foo(metaclass=ABCMeta):
    @abstractmethod
    def bar(self, baz):
        raise NotImplementedError

print(Foo)
print(type(Foo))

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <class '__main__.Foo'>
002 | <class 'abc.ABCMeta'>

[grave jolt]

And yes, a class is also an object. And a metaclass is also an object. And a metaclass is also a class.

[olive marsh]

okay, so by default it uses type to prepare a class but we can change it by inheriting the type

[grave jolt]

yep

well, not inheriting

there's a special metaclass keyword argument.

[olive marsh]

but we make metaclass by inherting type right?

[grave jolt]

!e
In particular, type is its own metaclass

print(type(type))
print(type(type(type)))

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <class 'type'>
002 | <class 'type'>

[grave jolt]

yep

[prime estuary]

Note this type loop here is special, some C-API + special casing in the interpreter.

[native flame]

class A: ...
ctypes.py_object.from_address(id(A)+8).value = A

👀

[pliant tusk]

!e ```py
class Meta(type):
pass

class M(type, metaclass=Meta):
pass

M.class = M
print(M, M.class, M.class is M)```

[fallen slate]

@pliant tusk :white_check_mark: Your 3.11 eval job has completed with return code 0.

<class '__main__.M'> <class '__main__.M'> True

[pliant tusk]

In case you want to do it without ctypes ^

[pliant tusk]

how can i report something that i think is a legitimate security vulnerability in python?

[quick snow]

https://www.python.org/dev/security/

[pliant tusk]

thanks, reported it

[radiant garden]

How severe is the vuln?

[grave jolt]

counting 90 days until the big oofium

[pliant tusk]

potential for RCE if a webapp dev is particularly unlucky

its a bug that can be hit a few ways, but one of the codepaths is really trivial

@feral island you helped me out with a bug in the past, whats the typical process for a security report?

[feral island]

I think the link you were given above should be correct. Internally there's a team of coredevs who deal with security bugs, I'm not part of that group though

[pliant tusk]

ok sweet, i sent them a description of the bug

[feral island]

great, thank you!

[olive marsh]

can we access the values passed as an argument to the bound method <bound method A.a of <__main__.A object at 0x7fe6a0a11bd0>>
I am writing a decorator of the class

a=A()
a.a(45,'str')

I know the bound method is a reference to the function object of the class object(A) i.e __func__ attribute of the bound method object but it also has __self__ which may have instance-specific things like in this case 45 and str as positional args but how do I get this info from __self__ attribute of this bound method object, I tried to access the code object of this bound method object but I am not able to get this info.

[prime estuary]

Are you after the arguments passed in (45 and 'str'), or info about what parameters the method accepts? Both ways you probably want inspect.signature(). Pass in the method or function, and you'll get back a Signature object which has all the info about the parameters permitted. If you then swap the method with your own function taking (*args, **kwargs), you can call bind_args() on the signature to figure out which values go to what parameters, same as the interpreter does.

[olive marsh]

what parameters the method accepts
we can easily get things from definition, I am talking about 45 and str

[feral island]

the bound method doesn't know about that, it gets created before the arguments come into play

[olive marsh]

yes, i doubted

i think i can only fetch it from frame object

but to get to that frame where the bound method is called is tough for me

[pliant tusk]

if you have the frame you can just get the locals of that frame

[raven ridge]

Your example: py a=A() a.a(45,'str') is equivalent to: py a = A() method = a.a method(45,'str') The thing that is a "bound method" in that program is method, or a.a. It's the A.a method bound to a particular instance of the A class. The thing that's being bound is only self

[olive marsh]

I know, but it is tough to get to that place. No probs, i will try to find some other way. Maybe disassembly

[olive marsh]

Yes, i should know this. Thanks man

[prime estuary]

What you should do is replace the method with your own function, which receives the args, does what you want, then calls the original method.

[peak spoke]

Are improvements to python functions that are mostly overridden with the faster C versions accepted? I was looking at https://github.com/python/cpython/blob/615537e62f0a49f6888ac27046bd8de965512d9d/Lib/functools.py#L466-L477 and the code isn't being particularly efficient here with all the tuples

[fallen slate]

Lib/functools.py lines 466 to 477

key = args
if kwds:
    key += kwd_mark
    for item in kwds.items():
        key += item
if typed:
    key += tuple(type(v) for v in args)
    if kwds:
        key += tuple(type(v) for v in kwds.values())
elif len(key) == 1 and type(key[0]) in fasttypes:
    return key[0]
return _HashedSeq(key)```

[peak spoke]

...but builtins are reassigned to locals

[tacit hawk]

Is there a binding of BoringSSL (google's openssl fork) for python?

[paper echo]

does cpython have optimizations for appending to tuples like with strings?

[peak spoke]

I don't believe it does

[grave jolt]

@paper echo @peak spoke In CPython, the tuple's elements (the element pointers, that is) are located directly in the tuple object's layout. So if you realloc the tuple, you're going to change the address of the object in memory

which is fine if nobody refers to it, I guess? 🤔

[paper echo]

well copying and allocating a pointer is reasonably cheap right?

im wondering about += on tuples like in that code

as opposed to just using lists which (iirc) python pre-allocates space for

[rose schooner]

since when

[rose schooner]

python uses realloc on lists

[paper echo]

i guess it isn't 😆

[paper echo]

my understanding was that it also over-allocates its constitutent PyObject pointers by like 1/3

[pliant tusk]

new_allocated = ((size_t)newsize + (newsize >> 3) + 6) & ~(size_t)3``` it uses this to determine the real new size of a list

[paper echo]

what does that clever bit twiddling actually mean?

[rose schooner]

equivalent python code ```py
new_allocated = (newsize + (newsize >> 3) + 6) & 18446744073709551612

[paper echo]

lol i know what >> and & and ~ are, but i don't know intuitively what it means to >> 3 a number

[pliant tusk]

https://github.com/python/cpython/blob/main/Objects/listobject.c#L60-L70

[fallen slate]

Objects/listobject.c lines 60 to 70

/* This over-allocates proportional to the list size, making room
 * for additional growth.  The over-allocation is mild, but is
 * enough to give linear-time amortized behavior over a long
 * sequence of appends() in the presence of a poorly-performing
 * system realloc().
 * Add padding to make the allocated size multiple of 4.
 * The growth pattern is:  0, 4, 8, 16, 24, 32, 40, 52, 64, 76, ...
 * Note: new_allocated won't overflow because the largest possible value
 *       is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.
 */
new_allocated = ((size_t)newsize + (newsize >> 3) + 6) & ~(size_t)3;```

[paper echo]

ah okay. i figured it was some clever thing with powers of 2

[rose schooner]

0: 4
1: 4
2: 8
3: 8
4: 8
5: 8
6: 12
7: 12
8: 12
9: 16
10: 16
11: 16
12: 16
13: 20
14: 20
15: 20
16: 24
17: 24
18: 24
19: 24
``` list size for each `newsize` in `range(20)`

so there isn't a 0?

[paper echo]

kinda makes sense, i suppose you don't want some kind of allocation "bump" when you first add something to a list

that's milder pre-allocation than i expected

[rose schooner]

there's also extra stuff under it

[paper echo]

should still be a lot of re-allocations if you're in a tight loop appending lots of things

(unless there are additional optimizations)

[rose schooner]

so i'm gonna apply that hold on

nvm i don't think i need to

[pliant tusk]

https://github.com/python/cpython/blob/main/Objects/listobject.c#L77-L78

[fallen slate]

Objects/listobject.c lines 77 to 78

if (newsize == 0)
    new_allocated = 0;```

[pliant tusk]

there is this check, so new_size of zero is alloced as 0

[raven ridge]

The larger the list, the rarer reallocations become. That's what allows appends to be amortized constant time

[raven ridge]

new_allocated = ((size_t)newsize + (newsize >> 3) + 6) & ~(size_t)3
``` could be written in Python as: ```py
new_allocated = (new_size + (new_size // 8) + 6) // 4 * 4

[paper echo]

thanks 🙂

[raven ridge]

right shifts are power of two integer divides, left shifts are power of two multiplies, and &~3 unsets the 1s and 2s bit - so it's like x // 4 * 4 or like x - (x % 4)

[grave jolt]

Py_ssize_t is signed 64-bit on a 64-bit system, right?

[raven ridge]

yes, and size_t is unsigned 64-bit

[grave jolt]

why is it signed though? 🤔

if you first cast it to size_t the generated assembly will, hilariously, be the same

but with less bit twiddling 🙂

although I suppose 30 years ago compilers weren't quite the same

[rose schooner]

according to the blame, the logic was introduced 3 years ago and replaced the old logic (from 13 years ago) in the form of c new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6); that had a growth pattern of 0, 4, 8, 16, 25, 35, 46, 58, 72, 88

[rose schooner]

the old logic replaced an even older logic (from 18 years ago) which was similar but with an extra addition: ```c
new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6) + newsize;

[rose schooner]

this in turn replaced another logic (from 19 years ago) where newsize < 9 was newsize < 8
no change in anything else

[rose schooner]

there's also an older logic from the same time period with these values ```c
_new_size = (newsize >> 3) + (self->ob_size < 3 ? 1 : 6) + newsize;

then an older logic from 22 years ago https://github.com/python/cpython/blob/4a8d42f73f9767622f32468c02cefe616ccb34cf/Objects/listobject.c#L11-L53

then the oldest logic from 28 years ago https://github.com/python/cpython/blob/56a71eee656bb894c64cdd1105abee8813c92735/Objects/listobject.c#L12-L22

[fallen slate]

Objects/listobject.c lines 12 to 22

#define ROUNDUP(n, PyTryBlock) \
	((((n)+(PyTryBlock)-1)/(PyTryBlock))*(PyTryBlock))

static int
roundupsize(int n)
{
	if (n < 500)
		return ROUNDUP(n, 10);
	else
		return ROUNDUP(n, 100);
}```

[quick snow]

Please don't spam

[astral gazelle]

<@&831776746206265384> also some cleanup would be nice

[frigid bison]

Is there a way to disable parsing doc strings with the ast module? It returns an expression with a constant so its rather hard to distinguish it from real code

[flat gazelle]

I don't think so, since docstrings aren't handled in ast at all as far as I can tell

[raven ridge]

Docstrings are string literals, so I'd expect them to show up like any other string literal at the AST level

[radiant garden]

You can transform functions and classes to insert some other statement before the ast.Expr containing the string literal

[spark magnet]

you can skip the first thing in a module, class or function if it's a string constant.

[grave jolt]

hmm well, Python does pay some attention to docstrings, as they are stored at runtime

although I suppose this isn't part of the parsing

perhaps it's a "computed property" of a parsed def statement?

Maybe a docstring() function/method for modules, classes and functions might be helpful.

[naive saddle]

what would this function do? cause getting the docstring should just be a simple attribute lookup

[grave jolt]

I meant in the context of ASTs

[naive saddle]

ah like a special node/leaf?

[grave jolt]

yeah, basically find the docstring node of a node

[rose schooner]

docstrings show up as regular Constants

just do what nedbat mentioned above

[grave jolt]

I know, I meant that maybe it's worth having as part of ast

or is it "too simple" to warrant an addition?

[naive saddle]

A simple constant subclass could probably make sense

[grave jolt]

I haven't implemented it, maybe there's some can of edge cases

[grave jolt]

No I meant something like ```
fun_node: Def = ...
doc: ast.Constant | None = fun_node.docstring()

[rose schooner]

docstring (as a string) or None is implemented like ```py
e.value.value if isinstance(e := AST.body[0], ast.Expr) and isinstance(e.value, Constant) else None

[naive saddle]

ohhhh

[rose schooner]

which you can put in docstring = lambda AST: ... if you'd like

[grave jolt]

code golf moment

[rose schooner]

i don't see why the people at python-ideas would think this is complex/useful enough

[grave jolt]

won't work if the first node is a non-string constant like 42 but yeah, something like this

[rose schooner]

ok then ```py
docstring = lambda AST: (
s if isinstance(e := AST.body[0], ast.Expr)
and isinstance(e.value, Constant)
and isinstance(s := e.value.value, str)
else None
)

[raven ridge]

At runtime, the interpreter does, sure. But that's not part of the grammar or the language's syntax, so it wouldn't be reflected in an AST

[next frigate]

User: jaja143#5019 has been sending a known phishing scam, threatening to close a SteemApp account if a game purchased is not paid for.
The user will target members in a group chat that are new or recently made in app purchases up in the game and discussed strategies amongst other members.
This user is part of a group that then joins the server with the sole intent of trying to get users to click a link or say they need to pay bc another member used a steem account and the funds were never deposited in the other account.
jaja143#5019 Does not work for SteemApp or the game, and has no authority to ask for payment.
Please report any similar attempts or requests by members to discord and do NOT ever send money or click links

[hexed spire]

I think you should write that to @summer lichen

That's an easier way to contact the moderators

Though if you see anything that needs immediate action such as spam or scam in the server, ping the Moderators role in the chat

[boreal umbra]

You would want to send this to @summer lichen. but it's more important that you report that to Discord rather than us. And if that person isn't a member of our server, there's nothing we can do.

[vast saffron]

So just a quick question:

Regarding the __debug__ builtin-constant.

Does anyone know of a library/application/package, that,…

A. … relies on the value of __debug__?

AND/OR

B. …changes behaviour depending on the value of __debug__?

** AND/OR**

C. … actually has __debug__ builtin-constant mentioned anywhere in its source code (even if it does not do anything with it)

[peak spoke]

I like to use it for things like this https://github.com/Numerlor/Auto_Neutron/blob/master/auto_neutron/journal.py#L136-L141

[fallen slate]

auto_neutron/journal.py lines 136 to 141

if __debug__:
    week_before = float("-inf")
else:
    week_before = (
        datetime.datetime.now() - datetime.timedelta(weeks=1)
    ).timestamp()```

[peak spoke]

only downside is that pycharm kinda sucks around it as it think it's dead code

also is used a tiny bit in the stdlib

[vast saffron]

Hm, and you dont run into problems with it being auto True unless explicitly started via -O?

[peak spoke]

Not really, as I only want the not debug branch to run when built for distribution

I suppose code that uses the debug constant could be difficult to test, but I don't exactly have tests there so ¯_(ツ)_/¯

[spark magnet]

are you having a problem with __debug__?

[vast saffron]

No just recently had a discussion with a friend about python being in debug "mode" by default.

Initially I wanted to have a nice global available variable for development, that does not rely on setting it in each module, always checking env-vars or monkey patching it in. Thought __debug__ was it...
||I know I can query the dev mode via sys.flags, but this thing is mostly about trying to see the use in the __debug__ variable||

The biggest problem is that if you distribute via pypi and not compiling via pyinstaller for example, then all the logic that relies on __debug__ is run if the user runs it.

This whole thing seems very very backwards to me and almost useless in this state as unsetting the debug behavior is different depending on the way you distribute and if the user sets an extra flag.
|| Especially as there is PYTHONDEVMODE which has no effect on __debug__ and more confusingly sets PYTHONASYNCIODEBUG which also despite its name has no effect on __debug__.||

Mostly trying to understand all the rationale and trying to find possible faults with my logic, by seeing how other people use it.

[spark magnet]

i'm just curious: why did you use hidden text in this reply?

[peak spoke]

The biggest problem is that if you distribute via pypi and not compiling via pyinstaller for example, then all the logic that relies on __debug__ is run if the user runs it.
That's mostly the point, isn't it? As a library developer, you'd want users to be able to get things like useful asserts and debug information to ensure the logic is sound when developing, but then turn those off for actual users as they'd be doing checks that shouldn't ever be true if the api is used correctly

[vast saffron]

weird behavior I somehow acquired, where I use the hidden text instead of brackets for tangential stuff.

Actually never thought about why I do that and now see, that it is kind of weird probably

[spark magnet]

i see what you mean bout the strange defaulting for __debug__

[vast saffron]

if you add script entry points though it often is not a library for that part and still has __debug__ set. When looking at something like pipx, as far as I can tell they will also not run it automatically with the -O-flag.

[vast saffron]

As I sometimes am not that good with picking up some clues from written text (sorry for that), I rather ask directly:

Was this meant as a hint to pythons idiosyncratic behavior, by pointing to my idiosyncratic behavior, or was your answer general and not directly pointed at the message you replied to?

[spark magnet]

i think __debug__ isn't that useful. for production, you use logging levels.

[vast saffron]

If I could I would change that behavior, but it is really not worth breaking backwards compatibility for.

I thought about proposing an alternativ builtin-constant like __dev__ or __dev_debug__, that would only be True if PYTHONDEVMODE is set, but seeing how „python-ideas“ and ideas on github discussion and up makes me cautious.

Just out of curiosity, would anyone here see this as something useful?

[sacred yew]

Is there a point to having a new builtin when it can easily be replaced by a os.environ check?

[raven ridge]

An os.environ check wouldn't detect python -Xdev

The proposal is to have something equivalent to sys.flags.dev_mode, except presumably checked at compile time like __debug__ is.

[elfin moon]

is there general advice when to prefer Protocol over ABC?

[sour thistle]

it seems like the PEP which introduced protocols compares them quite frequently, https://peps.python.org/pep-0544/

The same problem appears with user-defined ABCs: they must be explicitly subclassed or registered. This is particularly difficult to do with library types as the type objects may be hidden deep in the implementation of the library. Also, extensive use of ABCs might impose additional runtime costs.
meanwhile Protocols do not enforce subclassing/registering
there might also be a few extra considerations related to how each of them interact with user-defined metaclasses

[quick snow]

TL;DR: you only need Protocol for static type checking, ABCs can do the same thing at runtime.

[grave jolt]

unless you use typing.runtime_checkable

[swift imp]

I thought protocols were typing for more functional approaches

Whereas ABC is oop approach

Like a protocol you care what the class can do rather than what the class is. So instead of typing an ABC that describes the methods any concrete class would have, you do a protocol where any class with these methods and properties will work

Like does mypy not check methods and props if you type an ABC for an argument?

The whole idea of protocol is only for typing seems bizarre to me

[grave jolt]

well, protocols have existed in python for a long time, they were described in documentation and such

whereas typing.Protocol is mostly for static type checkers

[swift imp]

This I understand but does mypy and the like, not do type checking at ABCs and concretes?

[grave jolt]

They do

[swift imp]

Ok

[deft pagoda]

easier to specify required attributes in a Protocol if that means anything --- you can do abstract properties in a ABC, but not generic attributes

[next frigate]

Thank you! Did it

[next frigate]

Thanks.

[obsidian echo]

could anyone help here https://discuss.python.org/t/pep-582-python-local-packages-directory/963/226 ?? I saw recently this PEP and I'm trying to see if it can move forward 😦

[boreal umbra]

Glad you like it! It's for PyWeek (see #pyweek-announcements). If you have other questions/comments about the server, those belong in #community-meta, not here.

[fossil fern]

idk if this is the right place, but is there more to building python from git than doing ./configure with some set of flags and then make? I keep getting errors during the make process

it's probably something silly

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './_bootstrap_python'
  isolated = 0
  environment = 0
  user site = 0
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/home/algmyr/python_at_head/lib/python3.12'
  sys._base_executable = '/usr/bin/_bootstrap_python'
  sys.base_prefix = '/home/algmyr/python_at_head'
  sys.base_exec_prefix = '/home/algmyr/python_at_head'
  sys.platlibdir = 'lib'
  sys.executable = '/home/algmyr/git/cpython/_bootstrap_python'
  sys.prefix = '/home/algmyr/python_at_head'
  sys.exec_prefix = '/home/algmyr/python_at_head'
  sys.path = [
    '/home/algmyr/python_at_head/lib/python312.zip',
    '/home/algmyr/python_at_head/lib/python3.12',
    '/home/algmyr/python_at_head/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

fwiw, no env vars starting with PYTHON is set

so no PYTHONPATH

[paper echo]

@fossil fern i've done it before, and it should more or less "just work" from what i remember. if you get an error like that, my guess is that one of the modules failed to build and somehow it didn't trigger an error soon enough in the process.

[fossil fern]

welp, time to tun make again and dump the make output somewhere

[feral island]

yeah this should just work sometimes make distclean/make clean helps

[fossil fern]

no obvious failures in the make output other than the one I shared

the fact that the python_at_head dir doesn't exist shouldn't matter right? if anything make is the one that should populate the installation dir with whatever

stdlib dir = '/home/algmyr/python_at_head/lib/python3.12'
sys.base_prefix = '/home/algmyr/python_at_head'
sys.prefix = '/home/algmyr/python_at_head'
sys.exec_prefix = '/home/algmyr/python_at_head'
sys.path = [
    '/home/algmyr/python_at_head/lib/python312.zip',
    '/home/algmyr/python_at_head/lib/python3.12',
    '/home/algmyr/python_at_head/lib/python3.12/lib-dynload',
]

building from the source release seems to work though :/

is there something special I need to do to build from the main branch from git?

tried regenerating configure with autoreconf, but same result

[feral cedar]

<@&831776746206265384>

[fossil fern]

so for some reason it works when I install the latest git version of python through the AUR on arch, but not when I compile manually

so for sure some missing config on my part, not sure what though

I guess I can do this via my package manager using my locally checked out repo, though I still have no clue why things fail even if I do the exact same commands

[steel solstice]

curious as to how many of you use __class__ assignment?

i just want to gauge how bad an idea a feature request for cpython is

[paper echo]

never even considered doing it

[quick snow]

I do.. but I don't remember doing it very often in "serious" code. What's the feature request?

[steel solstice]

a better error message if it fails

or maybe that it doesnt fail ;)

[swift imp]

Only with module customization

[boreal umbra]

I've used it to turn instances of a class that I created into a subclass of that same class. And I did this in a function on copies of the arguments. And that function doesn't return those copies.

That is to say, it's just an implementation detail

[paper echo]

I've used it to turn instances of a class that I created into a subclass of that same class.
that's actually pretty clever and i could imagine how that'd be useful (e.g. "patching" objects returned from a 3rd party library)

is it typical for integer division to always round down rather than round towards zero? i was surprised to see that -8 // 5 is -2, not -1

[dapper lily]

The floor division and modulo operators are connected by the following identity: x == (x//y)*y + (x%y). Floor division and modulo are also connected with the built-in function divmod(): divmod(x, y) == (x//y, x%y).
from the docs

[paper echo]

aha, there's a reason for it. thanks

[dapper lily]

interestingly,

If x is very close to an exact integer multiple of y, it’s possible for x//y to be one larger than (x-x%y)//y due to rounding. In such cases, Python returns the latter result, in order to preserve that divmod(x,y)[0] * y + x % y be very close to x.

[swift imp]

Why does the operator module have a setitem and attrgetter but no attrsetter

Now I have to make a closure

I means it's ez enough I'm just surprised

[grave jolt]

What's the usecase for attrsetter?

attrgetter is useful in map, filter and such

[paper echo]

same as setitem i suppose

[swift imp]

Exactly

[flat gazelle]

I mean, operator.setitem for attributes is the builtin setattr

and afaik nothing like operator.itemsetter as the pair to operator.itemgetter exists

[swift imp]

I would like that

Like personally I would like an operator.attrsetter that would take in multiple attrs as str then returned object you pass the obj and vals

[paper echo]

would it be totally silly for subclass LogRecord to use .format instead of %, and change globally in your app with setLogRecordFactory?

it seems like a supported use case in the docs, and i understand why it's clearly a bad idea for a library. but perhaps it's not so bad for an app?

i suppose you could go a step further and restructure LogRecord to not dump user-provided data into attributes, which i find infuriatingly "old school"

might as well set alias funcname for funcName while you're at it...

[raven ridge]

It's bad for an app because it constrains that app to not using any existing libraries that log.

You can do that and make it work, with the caveat that it only works if everyone who logs anything knows that you've done this and plays along

[peak spoke]

the cost to benefit ratio there doesn't seem particularly appealing

[paper echo]

ahh right. there's no way to use different log record classes for different loggers, is there

i suppose another solution is to check which logger is being used to generate the log record, but that's getting a bit hairy

[deft pagoda]

i thought there were log libraries already that used format

[paper echo]

you can use .format-style to configure a Formatter format string without affecting anything else

[deft pagoda]

was looking at loguru

[paper echo]

ah. loguru i think expects to take complete control over your root logger

i don't know how it works with other loggers in other packages, might be worth looking at if they attempt to handle this nicely

[deft pagoda]

dunno, but it does look extremely popular

[raven ridge]

It's just an entirely parallel logging ecosystem, with some helpers for shuffling records from one to the other if you need interoperability with logging

[paper echo]

ah, noted

eliot is like that too

and structlog to some extent, which i tried valiantly to use on my current project and am currently in the process of replacing with logging