internals-and-peps

[cyan raven]

so after the re-build where can I access the new executable?

[sharp plover]

Hello 👀 I have a question about PEP 561 (Distributing and Packaging Type Information). In bullet point (1) of the section linked below, what does "the path" refer to?
https://peps.python.org/pep-0561/#type-checker-module-resolution-order

[feral island]

the stub search path, e.g. mypy's MYPYPATH env variable

similar to sys.path for Python modules and $PATH for the shell

[sharp plover]

Oh right thanks! I thought that but was a bit unsure as there are a few things referred to as the path. Is there an equivalent for Pyright?

[feral island]

I think so, not too familiar with pyright configs though

[sharp plover]

Right ok, I'll check the docs. Thanks for your help!

[cyan raven]

or does that automatically update on windows?

[gray galleon]

.

[lone sun]

Several things are wrong with your timings. (1) Your timing loop creates a new stack frame for f with every iteration. This adds overhead; in theory it's a constant amount of overhead, but in practice there's a little noise, so it makes your timings less consistent. (2) Every iteration performs a global lookup of the names tup or dct. Again, in theory this is a constant amount of overhead. (3) Some computers will increase CPU frequency over the course of the test. To combat this you need a warmup period (you can do this by repeating the test several times and taking the minimum timing; taking the minimum also has the advantage of discarding test results that were screwed up by background activity). (4) You are only ever looking up tup[0] and dct[0]. This is a very specific memory access pattern which is not representative of all situations.

For comparison:

import timeit


tup = tuple(range(1000000))
dct = dict(enumerate(tup))

print(min(timeit.repeat("tup_local[0]", setup="tup_local = tup", globals=globals(), number=100000000)))
print(min(timeit.repeat("dct_local[0]", setup="dct_local = dct", globals=globals(), number=100000000)))
print(min(timeit.repeat("tup_local[500000]", setup="tup_local = tup", globals=globals(), number=100000000)))
print(min(timeit.repeat("dct_local[500000]", setup="dct_local = dct", globals=globals(), number=100000000)))

On my laptop, the results are:

2.26810464495793
2.493539751973003
2.2556535489857197
3.4669744869461283

[unkempt rock]

Does this code abide by PEP-8 standards?

class PEP8(object):
  def __init__(self, text: Optional[str] = None) -> None:
    self.text = text if text else generate_test()

[flat gazelle]

Inheriting from object wouldn't pass code review, but PEP8 doesn't talk about it. indentation should be 4 spaces, however. Other than that, looks ok from a style standpoint.

[unkempt rock]

Thank you
is self.text = text if text else generate_test() abiding by PEP8, specifically?

[flat gazelle]

yes, that part seems fine as far as I can tell.

[unkempt rock]

Ok thanks! 👍

[feral cedar]

you could also do self.text = text or generate_text()

[lone sun]

Both look wrong to me. What if the user calls the object with text=""?

I think it should be text if text is not None else generate_test().

[feral cedar]

it depends on what they want, i guess ¯_(ツ)_/¯

[gray galleon]

thanks for the feedback
can we conclude that dict lookup with integer keys (in this specific case of only int keys) is slower than tuple/list lookup
if so, why
given that the hash code of an integer is just itself and there is no collisions (because there are only integer keys and they all have different hashes), hash lookup should reduce to array lookup and they should have negligible difference

[feral island]

interestingly in Kyle's example it is much slower when the key is 500000 than when the key is 0. Perhaps some keys happen to produce hash collisions, so a few more pointers have to be checked

also dicts has to run __eq__ to verify the keys are actually the same even if the hash matches

[lone sun]

I believe we can conclude that dict lookup is slower in this case. I think there are a few reasons why. As Jelle said, there's the cost of __eq__, and there may be hash collisions. Even though the hashes are all distinct, the hash table may not be large enough to avoid all collisions. (I don't remember enough about dicts right now to remember if this will happen.) Also, there's extra indirection when looking up from the dict. With the tuple, the items all immediately follow the tuple object itself, so if you have the memory address of the tuple, you just do pointer arithmetic. With the dict, you need to look up the hash table (one pointer indirection) and then the list of items (which is stored separately since 3.6, if I remember right).

[cyan raven]

hmm any ideas?

[warm breach]

pretty consistent on my end

[naive saddle]

Wouldn't hash randomization also makes the results slightly inconsistent?

Whether that matters is beyond me

[white nexus]

plz let me know if you find something out

[sharp plover]

Oh right. Do you have a problem that requires this?

[white nexus]

not necessarily, but it might be useful

[raven ridge]

no, hash randomization only applies to strings, and the keys are are ints.

[raven ridge]

it would probably be helpful to enumerate the steps involved in a list lookup, vs the steps involved in a dict of ints lookup. A list lookup would be something like:

1. ensure the user provided index is a Python int
2. convert that int to a Py_ssize_t
3. if the index is negative, subtract it from the length of the list
4. if the final index is outside the list's bounds, raise an IndexError
5. look up the final index in the list's internal array (random access)

The dict lookup would be something like:

1. call __hash__ on the user-provided key (which internally returns a Py_ssize_t)
2. compute the bucket number that hash code corresponds to (hash code % num buckets in the dict)
3. look up that bucket number in the dict's internal indices array (random access)
4. if the index found in that bucket is the "unused" sentinel, raise a KeyError
5. if the index found in that bucket is the "deleted" sentinel, go to 9
6. look up that index in the dict's internal entries array (random access)
7. if the user-provided key is not __eq__ to the found entry's key, go to 9
8. return the found entry's value
9. we've encountered a collision: perturb the bucket number to find the next bucket number to check, then go to 3.

spelled out like that, it ought to be pretty obvious that the longer, more complex recipe with more random memory accesses would be slower.

granted, in this specific example there won't be any collisions, but the dict is still paying for two random memory accesses vs only one for the list, and the dict's internal indices array is a large sparse array that's less cache friendly than its dense entries array (or a list's dense array)

[cyan raven]

I don't know maybe I'm doing something wrong.

[deep nova]

star_targets_list_seq[asdl_expr_seq*]: a[asdl_expr_seq*]=','.star_target+ [','] { a }

star_targets_tuple_seq[asdl_expr_seq*]:
    | a=star_target b=(',' c=star_target { c })+ [','] { (asdl_expr_seq*) _PyPegen_seq_insert_in_front(p, a, b) }
    | a=star_target ',' { (asdl_expr_seq*) _PyPegen_singleton_seq(p, a) }

Any idea why these rules are a) implemented differently and b) separated in the first place? As best I can tell, they have identical behaviour and return types

[uncut fern]

Does anyone know how to zoom to one point (x,y) in matplotlib ?

[grave jolt]

You're in the wrong channel. Please see #❓｜how-to-get-help and open a help thread

[deep nova]

slicing      ::=  primary "[" slice_list "]"
slice_list   ::=  slice_item ("," slice_item)* [","]
slice_item   ::=  expression | proper_slice
proper_slice ::=  [lower_bound] ":" [upper_bound] [ ":" [stride] ]
lower_bound  ::=  expression
upper_bound  ::=  expression
stride       ::=  expression

What is this? slice_list ::= slice_item ("," slice_item)* [","]

It looks like [1, 2, 3, 4][1, 5, 10] or some such

[naive saddle]

!eval

class A:
    def __getitem__(self, key):
        print(key)

A()[1, 2, 3]

[fallen slate]

@naive saddle :white_check_mark: Your 3.11 eval job has completed with return code 0.

(1, 2, 3)

[naive saddle]

You can index/slice using arbitrary objects, like a tuple in the example above 😄

[deep nova]

Interesting...

So... You can't do this with lists (I checked, you get an error)

But in theory, anything within the square braces will be provided as the key to getitem

I suppose that's helpful XD

[naive saddle]

It's how dictionaries work, they need to support any hashable object as the key

[deep nova]

So, is there any real distinction then (as far as the grammar is concerned) between a subscription and a slicing?

[naive saddle]

Probably not? I can take a look at the grammar though.

[deep nova]

I'm reading here that...

There is ambiguity in the formal syntax here: anything that looks like an expression list also looks like a slice list, so any subscription can be interpreted as a slicing. Rather than further complicating the syntax, this is disambiguated by defining that in this case the interpretation as a subscription takes priority over the interpretation as a slicing (this is the case if the slice list contains no proper slice).

Slicing appears to accept any comma separated list of Expression | ProperSlice. A comma separated list of expressions is an expression list, is it not?

And the syntax specification for subscription already accepts an expression list

I suppose it doesn't really matter, because whatever is going between the braces will be supplied to __getitem__ regardless

[naive saddle]

Oh, are there separate rules for subscription and slices?

[deep nova]

You probably know better than I. I'm mostly working off the specification here: https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-slicing

[naive saddle]

I was looking at the (admittedly slimmed down) LL(1) grammar Black uses

[deep nova]

Because the grammar is... not entirely consistent

Ohhhhhhh send that my way!

I think I'll bypass a separation between slicing and subscription, and instead go with a sequence of subscription rules

[naive saddle]

Well, the goal is just parse valid Python. It's not really a good specification of the Python syntax especially after all of the hacks we added to it to support the PEG-only syntax

https://github.com/psf/black/blob/main/src/blib2to3/Grammar.txt

It's a rather lax grammar for Python

[deep nova]

Anything helps!

Man

Python is weird

[naive saddle]

Checking https://docs.python.org/3/reference/grammar.html I don't see a rule for a generic subscript node.

[deep nova]

Rockin

[naive saddle]

I don't really know much about Python's syntax or grammar anymore. My limited knowledge is several years out of date :)

[deep nova]

Well, you've been around I assume

Lemme ask you

slices:
    | slice !',' 
    | ','.(slice | starred_expression)+ [','] 

I don't see any reason this couldn't be merged into a single case

And I see a lot of this. Often, there will be splitting of things without even a different return type

What gives?

[naive saddle]

Presumably the list getitem dunder does an isinstance(key, (int, slice)) check before continuing

[deep nova]

Hmmmm

Can you tell me

In certain terms

What a starred_expression is vs a regular expression?

[naive saddle]

https://github.com/python/cpython/blob/8d0f09b1beafd95763a5da53acc58dac0bd63a53/Objects/listobject.c#L2880-L2891 well the mp_subscript slot does

[fallen slate]

Objects/listobject.c lines 2880 to 2891

list_subscript(PyListObject* self, PyObject* item)
{
    if (_PyIndex_Check(item)) {
        Py_ssize_t i;
        i = PyNumber_AsSsize_t(item, PyExc_IndexError);
        if (i == -1 && PyErr_Occurred())
            return NULL;
        if (i < 0)
            i += PyList_GET_SIZE(self);
        return list_item(self, i);
    }
    else if (PySlice_Check(item)) {```

[naive saddle]

_PyIndex_Check and PySlice_Check

[deep nova]

Presumably, though

These checks have to do with the actual implementation of certain concrete classes

I don't see how that should have much bearing on what gets parsed. Shouldn't the AST be more or less consumer agnostic?

[naive saddle]

It does not, you're correct. I was just explaining why list doesn't support a tuple as a key.

It's totally up to the object to decide what it accepts as a key.

[naive saddle]

No idea why it's laid out like this, sorry.

[naive saddle]

No idea. I'd guess that there are some places where a star isn't allowed in any expression, but looking at the PEG grammar, that doesn't seem like the reason?

star_expression:
| '*' bitwise_or
| expression

[deep nova]

Absolute madness!

XD

[naive saddle]

raise_stmt:
    | 'raise' expression ['from' expression ] 
    | 'raise' 
```hmm

!eval raise *[RuntimeError("hi")]

[fallen slate]

@naive saddle :x: Your 3.11 eval job has completed with return code 1.

001 |   File "<string>", line 1
002 |     raise *[RuntimeError("hi")]
003 |           ^
004 | SyntaxError: invalid syntax

[naive saddle]

I guess the '*' bitwise_or is to just prevent it from matching too much.

[grave jolt]

Ouch, not letting me multiply raise by a list

[feral island]

!e raise (*RuntimeError())

[fallen slate]

@feral island :x: Your 3.11 eval job has completed with return code 1.

001 |   File "<string>", line 1
002 |     raise (*RuntimeError())
003 |            ^^^^^^^^^^^^^^^
004 | SyntaxError: cannot use starred expression here

[feral island]

😦

[naive saddle]

wait what star_expression and starred_expression aren't the same thing

starred_expression:
    | '*' expression

my head hurts

[deep nova]

Bruh

Bruuuuhhhhhhhhhhh

Seriously. I'm trying to piece together the meaning from the actual PEG grammar, the modified "reader friendly" grammar, and the syntax specification

https://github.com/python/cpython/blob/3.11/Grammar/python.gram
https://docs.python.org/3/reference/grammar.html
https://docs.python.org/3/reference/simple_stmts.html#grammar-token-python-grammar-assignment_stmt

XD It's madness. I don't really have any idea what I'm doing — but I'm pretty sure I've found a few schizms which make me wonder what sort of mad opium induced fever dream the writers were suffering from

[naive saddle]

I've only written a simple (and still slightly broken) math expression parser, you're doing great I'm sure.

I tried rolling my own recursive-descent parser and failed miserably. I went for instead a packrat parser.

[deep nova]

Isn't packrat an augment as opposed to a method unto itself?

[naive saddle]

Probably, it's a mix of pure recursion and iteration

[deep nova]

I think

I've mostly got it

    |   lhs=( (t=target_list '=' {t}) + ) rhs=(yielded_expression | starred_expression)
        { Assignment(lhs, rhs) }

target_list ::=
    |   targets=( target ** ',' ) [','] { TargetList(targets) }

target ::=
    |   '(' t=target_list ')' { t }
    |   '[' t=target_list ']' { t }
    |       t=identifier      { Target(t) }
    |       t=attribute_of    { Target(t) }
    |       t=subscript_of    { Target(t) }
    |   '*' t=target          { StarTarget(t) }

primary ::=
    |       attribute_of
    |       subscript_of
    |       call_to             to do, once signatures are figured out
    |       atom

attribute_of ::=
    |   root=primary '.' attribute=identifier        { AttributeOf(root, attribute) }
subscript_of ::=
    |   root=primary '[' subscript=subscription ']'  { SubscriptOf(root, subscript) }
subscription ::=
    |   s=slice      !',' { s }
    |   e=expression !',' { e }
    |   t=((slice | expression) ** ',') [','] { Tuple(t) }
slice ::=
    |   a=[expression] ':' b=[expression] c=[ ':' d=[expression] {d} ] { Slice(a, b, c) }

atom ::=
    | i=IDENTIFIER  { Identifier(i) }
    |   literal         # to do
    |   enclosure       # to do

Who thought assignment would be so complicated O.O

Oh 😐 There's still a few things to do... lookaheads in primary, and there's some sort of distinction between regular targets and targets in multi-assignment

[deep nova]

Is it just me, or this is an inconsistency

slices[expr_ty]:
    | a=slice !',' { a }
    | a[asdl_expr_seq*]=','.(slice | starred_expression)+ [','] { _PyAST_Tuple(a, Load, EXTRA) }

slice[expr_ty]:
    | a=[expression] ':' b=[expression] c=[':' d=[expression] { d }] { _PyAST_Slice(a, b, c, EXTRA) }
    | a=named_expression { a }

In the top rule, its collecting starred expressions

In the bottom, named expressions

Meaning it can accept a comma separated list of slices and starred expressions, or a single slice, or a single named expression

I think I might get it...

something[a, b, c := d]``` is fine, but

something[a := b, c, d]

Is ambiguous. A named expression really only makes sense by itself

Though something[(a := b), c, d] works fine — and that's encapsulated nicely within the expression chain

[rose schooner]

nope

[rose schooner]

star unpacking in a slice doesn't make any sense
that's why it's only included in slices, e.g. py abc[*x, *d] but not something like py abc[*x:] python also allows named expressions in other places with commas, like py func(x := a + b, y) [a, x := c*z + b, c * b] so the slices rule isn't an exception, it's just following consistently with those other rules

[deep nova]

Interesting

[deep nova]

How does python handle error recovery? Or, I suppose

How does it implement smart error reporting?

In it's parser, I mean

[frigid bison]

What do you mean "smart"?

[deep nova]

Bad example

I'll get back to you — gotta disembark the bus

[grave jolt]

parse errors be like:

x = print("Hello, world"
def f():
 ^^^^^^

    Esteemed sire, perhaps your majesty may have forgotten a INTERNAL__TOKEN__COMMA token monad shlongbfucator.

[silent depot]

What's the difference between packaging and pkg_resources? I've stumbled across an issue where I'm comparing two Requirement objects from the two different modules, and it's causing me a bit of a headache :p

>>> type(a)
<class 'packaging.requirements.Requirement'>
>>> type(b)
<class 'pkg_resources.Requirement'>

[deep nova]

Hey Python people

Python Jedi, I should say

Anyone have a minute to talk about error recovery in parsing?

[boreal umbra]

You should always ask your actual question. People want to know what the actual question is, before they commit to helping. Take a look at #❓｜how-to-get-help

!e

import dataclasses

class NotADataClass:
    __dataclass_fields__ = 'LOL!'

print(dataclasses.is_dataclass(NotADataClass))

[fallen slate]

@boreal umbra :white_check_mark: Your 3.11 eval job has completed with return code 0.

True

[boreal umbra]

The implementation of dataclasses.is_dataclass requires users to respect that all dunder names are reserved for use by the language. Interesting.

[feral island]

that is a documented convention

[raven ridge]

https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers

[boreal umbra]

I'm aware of the convention. I was just interested to have found another instance where one could "break" built-in functionality by not following it.

[forest coral]

You can "break" inspect by specifying the right dunders too

!e

import inspect

def f(): ...

f.__text_signature__ = "(a,b)"

print(inspect.getfullargspec(f))

[fallen slate]

@forest coral :white_check_mark: Your 3.11 eval job has completed with return code 0.

FullArgSpec(args=['a', 'b'], varargs=None, varkw=None, defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})

[forest coral]

I used this a while back to generate a function that could take *args but masqueraded as taking a specific number of arguments so I could use scipy.optimize.curve_fit on it

[quick snow]

Re: PEP-709 (Inlined comprehensions):

Generator expressions are currently never inlined in the reference implementation of this PEP. In the future, some generator expressions may be inlined, where the returned generator object does not leak.
Anyone knows what these "some generator expressions" would be that might be inlined in the future, or is nothing planned yet? I hope stuff like PonyORM wouldn't be affected..

[feral island]

tuple(x for x in y)

any(x for x in y)

I hope stuff like PonyORM wouldn't be affected..
we will strongly try not to break any working code. PEP 709 is a PEP (as opposed to just a perf optimization) only because it changes some very obscure corners of observable behavior

[boreal umbra]

!pep 709

[fallen slate]

**PEP 709 - Inlined comprehensions**

Link

Status

Draft

Python-Version

3.12

Created

24-Feb-2023

Type

Standards Track

[boreal umbra]

def g():
    raise RuntimeError("boom")

💥

[raven ridge]

hm - that proposal would also affect profile and trace functions, right? They currently observe the entry into the artificial comprehension scope.

!e ```py
def profile_func(frame, event, arg):
print(event, frame)

def func():
return [x for x in "12345"]

import sys
sys.setprofile(profile_func)
func()
sys.setprofile(None)

[fallen slate]

@raven ridge :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | call <frame at 0x7f26bbe91c60, file '<string>', line 4, code func>
002 | call <frame at 0x7f26bbe08880, file '<string>', line 5, code <listcomp>>
003 | return <frame at 0x7f26bbe08880, file '<string>', line 5, code <listcomp>>
004 | return <frame at 0x7f26bbe91c60, file '<string>', line 5, code func>
005 | c_call <frame at 0x7f26bbe91c60, file '<string>', line 10, code <module>>

[raven ridge]

Seems like the PEP should mention that in the Backwards Compatibility section, too...

[radiant garden]

Is there thorough documentation about the intersection of the import system and custom codecs?
In particular, what is the program flow for an import statement that encounters a file with a custom #coding: ... declaration?
I'm asking this because my toying around with custom codecs yields strange results: the codec's decode() function is getting called multiple times per file, even if the initial call already consumes the whole source.

[radiant garden]

For example, this bare-bones example

# a.py
import codecs

def encode(input, errors = "strict"):
    raise NotImplementedError

def decode(input, errors = "strict"):
    print("decoding", bytes(input))
    return "x = 1", len(input)

codecs.register({"test": codecs.CodecInfo(encode, decode)}.get)

# b.py
# coding: test
x = 0

# c.py
import a
import b
print(b.x)

has the surprising result...

decoding b'# coding: test\nx = 0\n'
decoding b'# coding: test\n'
Traceback (most recent call last):
  File "/.../c.py", line 2, in <module>
    import b
  File "/.../b.py", line 1
    x = 1
SyntaxError: invalid syntax

Clearly there's some part of the process documented somewhere I haven't been able to find! Or is this perhaps a bug in cpython?

[gray galleon]

will python have a first party jit (not numba or pypy)?

[grave jolt]

Sounds lejit

[radiant garden]

Uh oh! This has wildly different behavior between the versions 3.8 and 3.11
Here's all the different results:

• 3.8.16

decoding b'# coding: test\nx = 0\n'
1

but sometimes

1

without the "decoding" print at all

• 3.9.16:

decoding b'# coding: test\nx = 0\n'
Traceback (most recent call last):
  File "/.../c.py", line 2, in <module>
    import b
  File "/.../b.py", line 1
    # coding: test
        ^
SyntaxError: unexpected EOF while parsing

• 3.10.10 and 3.11.2:

decoding b'# coding: test\nx = 0\n'
decoding b'# coding: test\n'
Traceback (most recent call last):
  File "/.../c.py", line 2, in <module>
    import b
  File "/.../b.py", line 1
    x = 1
SyntaxError: invalid syntax

Is this a regression in 3.9? 3.10? A regression with an unsuccessful fix? Or did the import mechanism just change noticeably in both versions?

If it is in fact a regression, it looks pretty gross -- it manifests differently in multiple versions, and it's existed for quite some time without unit tests picking it up

What a headache, lol.

[radiant garden]

The discrepancy within 3.8 seems to be that the module parsing is cached in __pycache__, meaning the side effects from the codec don't get run again.

Should I file a bug report?

[radiant garden]

https://github.com/python/cpython/issues/102353 for the time being

[gray galleon]

where is the type object implemented in python

[pliant tusk]

Objects/typeobject.c

[rose schooner]

searching Python/codecs.c i found these lines
https://github.com/python/cpython/blob/main/Python/codecs.c#L184-L189

[fallen slate]

Python/codecs.c lines 184 to 189

if (!PyTuple_Check(result) || PyTuple_GET_SIZE(result) != 4) {
    PyErr_SetString(PyExc_TypeError,
                    "codec search functions must return 4-tuples");
    Py_DECREF(result);
    goto onError;
}```

[radiant garden]

That shouldn't be getting triggered

I can probably trigger that by returning something other than a CodecInfo from the searcher function?

[rose schooner]

i'm still also searching the docs

[rose schooner]

yeah maybe

[swift imp]

Do you think there's a reason why commands don't work in a .pdbrc file? I honestly would really love it and am debating on making this my first contribution to cpython but am afraid there's a reason for it, that I am just ignorant of. Relevant links...

https://stackoverflow.com/questions/2013698/commands-for-breakpoints-in-a-pdbrc-file
https://github.com/python/cpython/issues/90095

[unkempt rock]

wow

i just relized how bad i am at python

[unkempt rock]

I realise python is the fundamental core of all programming languages, the father the son and the holy spirit.

[grave jolt]

what do you mean?

[unkempt rock]

without python you have nothing

[grave jolt]

🤔 how

programming existed before Python and will exist after Python dies

[neat delta]

python will never die

[grave jolt]

I hope that's not true

imagine living in 2100 and dealing with bad decisions from 110 years ago

We still haven't figured it out with greenhouse gases

[unkempt rock]

python is the future

or should i say the future is python heh

[grave jolt]

why?

and the future of what?

[feral island]

Programming languages come and go, but any language with massive usage (like Python) will take a very long time to fully go away. There's still COBOL and FORTRAN after all

[unkempt rock]

listen man, I'm just a prodigy

don't grill me here

it's in my personal opinion

[unkempt rock]

people still use them

?

[neat delta]

it's my understanding that fortran is king of certain applications as well, beating C et al for e.g. BLAS / LApack

[unkempt rock]

this is all very advanced stuff fellas 😓

it's all alien language to me really

[rich cradle]

there's tons of old software written in those languages that's still used today

banks are notorious for their usage of COBOL

[raven ridge]

Yeah. The design of the Fortran language makes stronger guarantees in some cases than C, which allows optimizing compilers to perform optimizations that can't be done in C.

[swift imp]

Like what

[rich cradle]

iirc fortran has rules about aliasing and restricting the number of pointers able to point to a particular location

[raven ridge]

yep

Fortran has an array type, C doesn't (or at least, in C, you can't pass an array to a function, only a pointer to its first element). Fortran guarantees that two function arguments can't refer to the same memory, C doesn't.

https://beza1e1.tuxen.de/articles/faster_than_C.html

[swift imp]

I don't see the significance of this

[raven ridge]

of which of them?

well, check the link. It explains both those points in some detail.

[swift imp]

Of not being able to pass an array by value and not allowing two arguments to both have different pointers to the same thing

[raven ridge]

they're related in that not being able to pass arrays by reference makes it virtually impossible to forbid another function argument from aliasing an array passed as an argument

and aliasing is relevant because there are many optimizations that look intuitively safe and obvious to do, but which would result in behavior changes if two different variables happen to refer to the same object

[rich cradle]

what whitespace characters does python allow? i'm trying to see if that info is in the reference, but i can't seem to find it here https://docs.python.org/3/reference/lexical_analysis.html

is it just cr, lf, tab, and space? or is it something more comprehensive like Pattern_White_Space (and if so, how do those play with indentation)?

[gray galleon]

C "arrays" doesn't even store length 😭

[rose schooner]

it's only an illusion in the compiler

[cyan raven]

Looks good.

[rose schooner]

yes

just CR, LF, tab, and space

actually FF is also included

[gray galleon]

wait so if i use nbsp python will think its not a whitespace

or is it normalized into normal whitespace

[rose schooner]

it's just normalized

unless i copied the wrong character

[rose schooner]

ok yeah it is normalized

(in the console)

[flat gazelle]

python doesn't normalise the whitespace though, both eval('1\N{NO-BREAK SPACE}') and eval('1\N{EM SPACE}') will error out

[rose schooner]

ok final #bot-commands message
it's not normalized

[rich cradle]

got it, thanks y'all!

[charred fulcrum]

h

[grave jolt]

h

[cyan raven]

h

[raven ridge]

an array has a length (known by the compiler, albeit not stored anywhere). But that information is lost when you pass it to a function (arrays degrade to pointers when passed as an argument)

[unkempt rock]

hey guys may someone help me with 5sim api

i get error when i run request

to buy phone number

[raven ridge]

!rule 5 7

[fallen slate]

Rules

5. Do not provide or request help on projects that may break laws, breach terms of services, or are malicious or inappropriate.

7. Keep discussions relevant to the channel topic. Each channel's description tells you the topic.

[unkempt rock]

wtf

it doesnt break any of rules

[raven ridge]

it breaks 2 rules. The two rules I just cited.

[unkempt rock]

oops wrong channel

but it doesnt break 5

its just me getting error while trying to do request in py

[raven ridge]

yes, it does. Bypassing captchas and phone number verifications is malicious. In general, bypassing any security protections put in place by a service provider is maliciuos.

[unkempt rock]

i dont bypass anycaptcha

and i dont use phone numbers for anything malicious

i only need help with request

so i dont see ur point

[raven ridge]

feel free to open a thread on @summer lichen if you think I'm making the wrong call, and other moderators will review. Otherwise, don't ask about this further.

[gray galleon]

so you have to pass the length

[raven ridge]

If you pass an array to a function, yeah - because the called function only receives a pointer to the array's first element. Within the function where the array was defined, though, the length is known and can be accessed.

But this has wandered well off topic, from questions of language design tradeoffs to quirks of the C language

[gray galleon]

why isn’t super a keyword but a function

[spark magnet]

because it doesn't need to be a keyword. same as print

[gray galleon]

there was a pep about this but it was superseded 😭
https://peps.python.org/pep-0367/

[gray galleon]

by super() of course

[lone sun]

My understanding of the history is that super was recognized as problematic during 2.x, but the right solution was hard to identify. If you've ever written Python 2 code with super(MyHardCodedClassName, self) calls everywhere, you'll appreciate how much super needed to change. The initial thinking was that super could become a keyword, and being a keyword meant that it would get parsed specially by the interpreter, hopefully at a time when it could be bound to the class. It turned out that this couldn't be done without introducing the __class__ cell, and once you do that, there's no reason to make it a keyword.

[gray galleon]

you could desugar super into __super__(type(<firstarg>), <firstarg>) (where __super__ is the original python super), no?

[lone sun]

Privately, I feel like super should actually be something that happens on the metaclass. In my head, super(cls, self) ought to translate to type(cls).__super__(cls, self). In reality it doesn't work like this; there's just one super function. I once considered proposing this, but I can't think of any use cases besides re-ordering the MRO. The only times I've seen people want to do that, something else had gone wrong earlier in their design.

[lone sun]

No, that doesn't work in the presence of inheritance.

For example, the following does not work:

class A:
    def f(self):
        return 1

class B(A):
    def f(self):
        return super(type(self), self).f() + 1

class C(B):
    def f(self):
        return super(type(self), self).f() + 2

c = C()
c.f()  # Infinite loop

It reaches B.f, which calls super(C, c).f(), which resolves to B.f again.

super needs to know the class in which it's defined in order to function correctly.

[rose schooner]

it'd more accurately desugar into __super__(__class__, self)
__class__ is implicitly defined by the symbol table generator

[rose schooner]

this is currently how super() (no arguments) works rn

[dusk comet]

why __class__ cell is not generated for every function?

[gray galleon]

why should it?

[dusk comet]

!e ```py
s = super
class X:
def f(self):
s()
X().f()

[fallen slate]

@dusk comet :x: Your 3.11 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 5, in <module>
003 |   File "<string>", line 4, in f
004 | RuntimeError: super(): __class__ cell not found

[dusk comet]

to allow super aliasing

[rose schooner]

‫it's implicitly generated when super exists as a (global) name or when __class__ is mentioned

[lone sun]

Right, this part is syntactic magic.

[rose schooner]

literally no other case

!e ```py
s = super
class X:
def f(self):
super
print(s())
X().f()

[fallen slate]

@rose schooner :white_check_mark: Your 3.11 eval job has completed with return code 0.

<super: <class 'X'>, <X object>>

[rose schooner]

!e ```py
s = super
class X:
def f(self):
class
print(s())
X().f()

[fallen slate]

@rose schooner :white_check_mark: Your 3.11 eval job has completed with return code 0.

<super: <class 'X'>, <X object>>

[lone sun]

When the class doesn't use super there's no reason to generate the __class__ cell. This optimizes the case where you generate lots of simple classes (e.g., classes just for storing stuff, like simple dataclasses or namedtuples).

[radiant garden]

Does it also run if the names are part of an unevalutated annotation?

Because I know things like yield` used to behave funny inside annotations

[dusk comet]

!e

s = super
class X:
    def f(self):
        if False: __class__
        print(s())
X().f()
__import__('dis').dis(X.f)

[fallen slate]

@dusk comet :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <super: <class 'X'>, <X object>>
002 |               0 COPY_FREE_VARS           1
003 | 
004 |   3           2 RESUME                   0
005 | 
006 |   4           4 NOP
007 | 
008 |   5           6 LOAD_GLOBAL              1 (NULL + print)
009 |              18 LOAD_GLOBAL              3 (NULL + s)
010 |              30 PRECALL                  0
011 |              34 CALL                     0
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/wujizebosi.txt?noredirect

[gray galleon]

there's no reason to make it a keyword.
super is special-cased, so i don't understand why don't make it a keyword

[dusk comet]

keywords are bad

[lone sun]

Keywords need special parsing support.

[dusk comet]

and special compiler support

[lone sun]

Remember that until recently, you couldn't assign to a keyword.

[radiant garden]

there's negative reason to change it now, but you could have argued for making it a keyword at release

not super convincingly maybe, but you could have

[gray galleon]

!e```py
s = super

class A:
def a(self):
print('a')

class B(A):
def a(self):
print('b')
s().a()

B().a()

[fallen slate]

@gray galleon :x: Your 3.11 eval job has completed with return code 1.

001 | b
002 | Traceback (most recent call last):
003 |   File "<string>", line 12, in <module>
004 |   File "<string>", line 10, in a
005 | RuntimeError: super(): __class__ cell not found

[gray galleon]

see, if you alias it, it no longer works

[lone sun]

Right.

That's part of how it works.

[gray galleon]

it doesn't work like a function

[lone sun]

At the time the class definition is parsed, super() is replaced by super(__class__, self).

(Really by self I mean the first method argument.)

I remember this being described as a "lexical" replacement. It's not part of the grammar, and super doesn't have default arguments.

[dusk comet]

it is not: ```py

class X:
... def f(self):
... super(); func()
...
dis(X.f)
0 COPY_FREE_VARS 1

2 2 RESUME 0

3 4 LOAD_GLOBAL 1 (NULL + super)
16 PRECALL 0
20 CALL 0
30 POP_TOP
32 LOAD_GLOBAL 3 (NULL + func)
44 PRECALL 0
48 CALL 0
58 POP_TOP
60 LOAD_CONST 0 (None)
62 RETURN_VALUE

it works like any other function, but inside it there is some frame magic

[lone sun]

Well, that's how I remember it being proposed.

[dusk comet]

python is too dynamic, so there is no way to ensure that super() is actually a builtins.super() call

[lone sun]

@gray galleon There's a good chance that you can find answers to your questions in the thread starting here: https://mail.python.org/pipermail/python-3000/2007-April/006667.html.

[raven ridge]

the best argument for keeping super() a function in the 2 to 3 transition is that it made it so existing super(StaticClassName, self) calls were still valid. Breaking those would have been another incredibly annoying backwards compatibility break between 2 and 3.

[flat gazelle]

it is also useful to be able to use super without the magic, as it does come up on occasion

[dusk comet]

is there any pure-python implementation of super (maybe without some C-level-related stuff)? i never fully understood how it exactly works

[cyan raven]

https://youtu.be/X1PQ7zzltz4?t=1102 maybe this helps?

https://github.com/mCodingLLC/VideosSampleCode/blob/master/videos/093_super_in_python/super_implementation.py

you can check how it was defined in pypy

[dusk comet]

that's cool

[dusk comet]

i have seen this, i know how super works from users perspective, but i dont know how it exactly works internally

[cyan raven]

then just try to play with it, and understand the base concept from the CPython code, as I said before the pypy implementation can be useful and then try to define it on your own.

[rose schooner]

this is so much better than what i did
‫https://github.com/thatbirdguythatuknownot/sniplections/blob/dc0dc84891110a9516a4d65ebd7a50eb658eb19e/pysuper.py

[rose schooner]

i'm gonna try to make a readable implementation in pure python that should be close to what the internals do

that doesn't use ctypes

[rose schooner]

https://github.com/thatbirdguythatuknownot/sniplections/blob/main/pysuper.py i think this works

[gray galleon]

so basically super is already a function in the first place and now we have to deal with the consequences

[raven ridge]

That's applies for every "why not change X to Y" question. There needs to be a lot of very clear advantages in order to justify breaking existing stuff.

[lone sun]

Also, what advantage is there to making super a keyword? As far as I can tell it just makes everything more complicated.

[feral island]

It fixes the awkwardness where the name super is special. For everything else, you can alias it and it still works the same

[raven ridge]

I'm not convinced it would be better, but it would need to be a lot better to justify breaking backwards compatibility. The downsides of being a popular language: there's lots of users, and they don't like it when you break their stuff.

[feral island]

Doesn't matter too much in practice, but it feels a little hacky to me that the name matters.

[lone sun]

But it does this by turning super into a keyword, thereby making it special...

[rose schooner]

super as a keyword only makes sense for no-argument super()
also that PEP was back in like 2008

[feral island]

sure, but keywords are supposed to be special

[rose schooner]

there's also super() with 2 arguments, super() with 1 argument
how do you justify a super keyword with those use cases

if we're gonna use parentheses with the super keyword in order to support those why not just make it a function

[raven ridge]

Sure, but "super doesn't work if you alias it, and we can fix that by preventing you from aliasing it" isn't very satisfying

[feral island]

"X doesn't work" is cleaner than "X seems like it should work, but blows up at runtime"

[raven ridge]

I'm not sure that super with optional parentheses could work. I think it would be ambiguous

[raven ridge]

A bit, yeah. But it's not terribly satisfying.

Making failures more explicit, and making things fail early, are definitely both wins, but I don't think they're huge wins, especially when it's a thing most people would never try to do

[lone sun]

Yeah, I view super-as-a-function being a kind of least bad compromise option.

It doesn't have every nice thing you could want, but I like it better than anything else I've seen proposed.

[gray galleon]

imagine if super is a keyword in the first place so we don't have to make this compromise 😭

[rose schooner]

how do you plan to support 2-argument/1-argument super()

[gray galleon]

__super__(cls, self) or self.__super__()
99.8% of the time you don't want 2 arg super anyways

[rose schooner]

ok good enough

i feel like it's a little inconsistent though

[gray galleon]

with python macros we might have super! keyword macro tho
super!.__init__()

it also free the interpreter of special casing super

[lone sun]

I don't think it makes sense to have self.__super__(). What super does is find where __class__ is in type(self).__mro__ and return a placeholder object that redirects you to the following class. This really has nothing to do with the instance; it's all about the class. (This is why earlier I said I mentally think of super(cls, self) as being something like type(cls).__super__(self). That's the only way I can remember the argument order.) (I think a __super__ function would be fine though.)

[rose schooner]

not in type(self).__mro__

it finds __class__ in the function's closures itself

that's the special case

[lone sun]

No, I mean that it has to look up what the next class in the MRO is.

__class__ is one of the items in type(self).__mro__.

[raven ridge]

it figures out the static class using __class__, and then finds the first class in the dynamic class's MRO - type(self).__mro__ - after that static class

[rose schooner]

ok fair

[gray galleon]

isn't it __class__.__mro__?

[raven ridge]

No, it's not. super() can delegate to a class on the same level of the inheritance tree of a type, not only to one on a higher level. If it used __class__.__mro__ it would never be able to do that

[gray galleon]

so what is __class__ for

[raven ridge]

Determining the type that a method was statically defined in

[gray galleon]

ok i still don't understand what does it do with __class__ (or first argument) exactly
do you have a pure python implementation of super

[lone sun]

Have you looked at examples of __mro__ before?

Like, when you have several levels deep, diamond inheritance, and so on?

[rose schooner]

@gray galleon

[raven ridge]

What super() does is find the next class in the MRO of the dynamic type of self after the class that the method currently calling super() was defined in

[gray galleon]

yeah, super delegates method calls to <class>.__mro__[1]

[lone sun]

No, it doesn't.

Consider:

class A: pass

class B(A): pass

class C(A): pass

class D(B, C): pass

Do you know, without evaluating the above code, what D.__mro__ is?

[gray galleon]

D, B, C, A, object

[lone sun]

Great. Now suppose each of those classes has a method f that simply calls super().f().

If you expand out the zero-argument form of these supers to two-argument forms, what do you get?

Next question: For each of those calls, which class's f is invoked?

[gray galleon]

so ```py
mro = type(self).mro
class_index = mro.index(class)
mro[class_index + 1]

[rose schooner]

the problem with that is that it uses ==, not is, to find the class

but yes that should be it

[raven ridge]

Yeah, that's basically right

[lone sun]

Right, so it's __mro__[1] when you're in D.f, but otherwise it's not.

[rose schooner]

!e while messing with this example i found this error with an interruptive newline in the middle of the message ```py
class A: ...

class B(A): ...

class C(B): ...

class D(B, C): ...

[fallen slate]

@rose schooner :x: Your 3.11 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 7, in <module>
003 | TypeError: Cannot create a consistent method resolution
004 | order (MRO) for bases B, C

[raven ridge]

The key is that the MRO we're looking at is dynamic (the MRO of the derived type of self). In order to figure out where to skip to in the mro, we need to know which class we are currently executing code inside of. So we need to know which class the currently executing method was statically defined in. That's what __class__ is for.

[lone sun]

Huh, seems you're right.

[gray galleon]

!e while messing with this example i found this error with an interruptive newline in the middle of the message ```py
class A: ...

class B(A): ...

class C(A): ...

class D(B, C): ...

print(D.mro())

[fallen slate]

@gray galleon :white_check_mark: Your 3.11 eval job has completed with return code 0.

[<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>]

[feral island]

https://github.com/python/cpython/blob/6716254e71eeb4666fd6d1a13857832caad7b19f/Objects/typeobject.c#L1996

[fallen slate]

Objects/typeobject.c line 1996

off = PyOS_snprintf(buf, sizeof(buf), "Cannot create a \```

[feral island]

not sure why that \n is there

[gray galleon]

what is the difference between A.mro() and A.__mro__

[rose schooner]

quickly sighted is that .mro() returns a list but .__mro__ is a tuple

deeper details i don't know yet

[raven ridge]

!e So: ```py
class A: ...

class B(A):
def foo(self):
print(class)
print(class.mro)
super().foo()

class C(A):
def foo(self):
print(class)
print(class.mro)

class D(B, C): ...

D().foo()
``` In this example, the foo defined in B delegates to the foo defined in C, even though C is not in B.__mro__

[fallen slate]

@raven ridge :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | <class '__main__.B'>
002 | (<class '__main__.B'>, <class '__main__.A'>, <class 'object'>)
003 | <class '__main__.C'>
004 | (<class '__main__.C'>, <class '__main__.A'>, <class 'object'>)

[raven ridge]

And that's because C is in D's MRO, and self is a D instance

[feral island]

From some looking at the code, __mro__ is an attribute that's actually stored on the type. .mro() recomputes the MRO from scratch. During type creation I think we essentially do tp.__mro__ = metaclass.mro(tp)

[gray galleon]

does that explain why .mro() returns a list?

[rose schooner]

you can append to a list without creating a new one

that's why

[gray galleon]

you can also append to a tuple in C layer?

[rose schooner]

doesn't that create a new object

a tuple has a fixed size

it's embedded into the allocated object memory itself

if a tuple is size 3 then there are exactly 3 objects that it's allocated for

[gray galleon]

hmm ok

the mro is built as a list then converted to tuple

as for why it is converted to a tuple
maybe to prevent modifying the mro

or to optimize space

[rose schooner]

probably both reasons

[cyan raven]

what is the reason that the asyncio library, is not using the coro.__name__, so with that, every task should have its own name(understandable name).

 _set_task_name(task, coro.__name__)

[flat gazelle]

You can have the same coroutine shared across multiple tasks. And IIRC the coroutine is already part of the repr, so it doesn't add new information

[cyan raven]

yes you are right

Task pending name='Task-3' coro=<coro1() 

[grave jolt]

You can set items, but not extend the tuple

because that would require a realloc, which would change the location of an object

[gray galleon]

so a tuple is just abstraction over a simple C array

[grave jolt]

kind of, yeah

a heap-allocated array of PyObject* pointers combined with its size

[warm breach]

but tp_mro is a tuple..?

isn't it only made a list on call of mro()

[warm breach]

It may not require realloc if using PyTuple_Extend though, depending on available memory

but it can only be used when ref-count is 1

[rose schooner]

tp_mro is assigned a tuple converted from a list by calling .mro()

[rose schooner]

in mro_implementation():
https://github.com/python/cpython/blob/main/Objects/typeobject.c#L2162-L2174
in mro_invoke():
https://github.com/python/cpython/blob/main/Objects/typeobject.c#L2246-L2263
assigning to .tp_mro:
‫https://github.com/python/cpython/blob/main/Objects/typeobject.c#L2314-L2326

[rose schooner]

mro_implementation() (or .mro()) returns a list, mro_invoke() converts that to a tuple, .tp_mro gets assigned what comes out of mro_invoke()
‫TL;DR .tp_mro comes from .mro(), not the other way around

[rose schooner]

also you probably mean _PyTuple_Resize() which still may use realloc()
there's no PyTuple_Extend()

[boreal umbra]

@dataclass
class Vec:
    x: int
    y: int

vec = Vec(1, 2)
vec.x + 3  # Vec(4, 2)

Can anyone think of a way to get this behavior? (doesn't have to involve dataclasses--I just used that to make the code as terse as possible.)

[feral cedar]

overwride getattr to return an object that implements __add__ to make a new vector

[umbral plume]

if Vec.x actually gave some new type of object that returns a Vec when an integer is added, then you could do something like that, but at the cost of being able to use Vec.x to directly get the integer value of x

(unless you then also had that custom type be a subclass of int, but then you've still got the issue of being unable to add another int without a vec being returned )

[warm breach]

well currently if that worked, the type hint for Vec.x would be wrong

[grave jolt]

You could make a custom "view-into-x", but then you wouldn't be able to work with it as a number

[grave jolt]

hmmm uhh

vec.x += 3

[warm breach]

could probably also make it a enum as well since type hints for those are overriden to work differently already

also yeah assuming if vec.x didn't actually return the int, how would you even get the int value

[deep nova]

Does python perform its own unicode decoding?

[grave jolt]

wdym

[deep nova]

Well, my understanding is that C itself support unicode

[grave jolt]

define 'support unicode' 🙂

[deep nova]

I guess maybe my question itself is a bit shaky, let me ask more generally

Through what mechanisms does python recognize unicode characters?

Lets say I include within a file's source code an emoji or else some nonstandard mathematical symbol. How do those bytes get recognized as a single character such that it can be handled appropriately by the lexer?

[feral island]

I believe Python implements its own codecs

[deep nova]

So, there is some machine somewhere in python's internals which comes bytes and produces "character" entities? Presumably, this machine feeds the characters it generates to the lexer?

As well, it might have to play a role in input or else file io operations?

I'm working on my own small compiler, partly as a school project and partly for fun. My intuition is telling me that if I want to go all out, I should have such a machine. Another part of me, the "keep it simple, stupid" part is telling me hand-coding such a thing even in C would be hideously imperformant

Which begs the question — how does an industry grade, mature language handle the problem?

[warm breach]

unicode decode is pretty fast

but also this is just for compiling bytecode, it doesn't affect runtime that much

[deep nova]

True

[warm breach]

looks like it happens somewhere around here

https://github.com/python/cpython/blob/main/Parser/tokenizer.c#L685

[fallen slate]

Parser/tokenizer.c line 685

decode_str(const char *input, int single, struct tok_state *tok)```

[deep nova]

Hmmmm

Quick follow up question

I'm imagining unicode-decoding to be an operation not entirely unlike lexing. Seems simple enough, at least until you get to the grapheme clusters phase (I havn't done that research yet)

But what reading I have done has mentioned error handling. Under which circumstances would input-bytes in a program's source code (or in input, or during file-io) be invalid?

After all, if it's typed by a human it's probably done so on a keyboard, and any sequence of bytes produced by a keyboard will be correct

Even mechanically generated text is unlikely to be garbled on that low a level

[grave jolt]

you mean UTF-8 decoding?

[deep nova]

Yeah

[grave jolt]

(because there are many ways to encode Unicode)

[deep nova]

Right — unicode is the abstract schema, UTF-8 is the actual encoding

[feral island]

most likely if it's not actually unicode

SyntaxError: Non-UTF-8 code starting with '\xca' in file /Library/Frameworks/Python.framework/Versions/3.11/bin/python3 on line 1, but no encoding declared; see https://peps.python.org/pep-0263/ for details

[deep nova]

I'm not sure I completely understand

[warm breach]

keyboards don't interact with files though, you're either directly entering bytes into a file (which can obviously be incorrect), or using some file editor, which encodes for you (and can corrupt files by saving with the wrong encoding or a million different ways)

[deep nova]

So, the TL;DR is that bad bytes happen

[warm breach]

ascii enjoys the luxury of having very few encoding conflicts, but errors like code page ambiguity are common in other languages https://en.wikipedia.org/wiki/Mojibake

UTF-8 and unicode (which, is still in "beta" support on windows) is a relatively new thing in general

there are also windows code pages, eu specific and japan specific encodings, etc.

[raven ridge]

I don't think "bad bytes" is the right way to think about it. The right way to think about it is just that the user told you to decode the file as UTF-8, and you tried, but UTF-8 decoding failed, because it wasn't actually UTF-8 encoded

[urban sandal]

If I were designing my own language, I'd constrain it slightly more than "valid utf-8".

Most notably, while I wouldnt want to outright forbid use of control characters such as right-to-left override, control characters in source code would require specific opt-in to be allowed.

[raven ridge]

why it wasn't UTF-8 encoded is a mystery to you. You can't possibly know, you just know that the user said "decode this file as UTF-8 and then execute it", and you failed on the first step, because it wasn't UTF-8 at all. You can tell the user what property of the binary data makes it invalid UTF-8 (that's easy, and that's what Python does), but you can't know what it was supposed to be

normally the reason will be that they tried to use the Python interpreter to run something that wasn't text at all (UTF-8 decoding a .png or a .exe is gonna fail), or they tried to run something that was text, but it was text encoded in an encoding other than UTF-8, so UTF-8 decoding it fails.

[urban sandal]

Sometimes when the issue is a direct result of encoding errors and the intent is that it actually should be utf-8, but something misbehaved or wasn't acounted for, tools like https://github.com/rspeer/python-ftfy can actually detect what caused it and correct for it, but I wouldn't use this as part of parsing source code.

[rich cradle]

typically, source files are valid UTF-8, but these are not allowed as their own tokens (besides strings and comments)

although, i suppose it is reasonable to restrict it for security purposes

[urban sandal]

Yeah, I'd go a step further than this typical case if I were designing a language. The current unfortunate reality is that many programs and editors don't handle control characters to specification, so to avoid issues, I'd have a way to mark files in which specific disallowed controlled characters are allowed, putting it on the developer to show they are aware and intending their use.

[rich cradle]

(the rust language considered bidirectional overrides a CVE and made it a hard error because it was causing code to render incorrectly and therefore obscured malicious code: https://blog.rust-lang.org/2021/11/01/cve-2021-42574.html)

[raven ridge]

I've definitely seen PoC exploits where RTL was used to hide executable code in what appeared to be comments

hah, that rust CVE may be exactly what I'm thinking of

[urban sandal]

There's also the thing where rtlo messes with user expectations rather than dev expectations, but that's a different place it should be banned. Take a look at something like

details_of_exe.txt and telling someone you are sending them a text file with metadata, but there's an rtlo in the filename and the actual name is details_of_{rtlo}txt.exe

This is something that has actively been used to make users less wary of what they are opening.

[deep nova]

You guys are smart

What I'm absorbing from all of this (and, it isn't much — I'm a bit under the weather today)

Is that bad bytes can happen for any number of reasons — attempting to read or run non-text files or files with non-utf-8 encodings as examples

Additionally, unrestricted decoding of all of the unicode specification may be confusing or insecure

Thus, some kind of machine must be built which consumes the bytes and produces characters, interprets and formats errors due to bad bytes, and safeguards against valid but disallowed characters

Does that all sound correct?

[raven ridge]

sounds about right. You can start by only accepting ASCII, and extend that to later support more of UTF-8, since ASCII is a subset of UTF-8

[deep nova]

Well, I'm currently using a hand-rolled lexer that is configured for ASCII

It's relying on python's internal read machinery though

[raven ridge]

it wouldn't be too much more complicated to preprocess the file by decoding the raw binary contents of the file as ASCII, and then iterate over that decoded string

[deep nova]

😄

[raven ridge]

Thus, some kind of machine must be built which consumes the bytes and produces characters, interprets and formats errors due to bad bytes, and safeguards against valid but disallowed characters
You can hand-roll your own ASCII parsing. It's strictly easier than hand-rolling your own UTF-8 parsing, since ASCII is a subset of UTF-8. Your hand-rolled ASCII parser would consume bytes and produce characters - you can use Unicode codepoints as the characters, since Unicode has the property that the Unicode codepoint of every ASCII character is the byte value of that character. You'd need to detect invalid ASCII (which is easy to detect, it's any byte with the high bit set - there are exactly 128 ASCII values). You could disallow some characters at this point if you wanted to - there's no particular reason to allow form-feed or vertical-tab, etc

[deep nova]

That's definitely a place to start. I'll be handing this in as a my final project for my OOP class at the end of term

My goal is to demonstrate an effective understanding of OO principals, and I'm intending to do that by building a decoder-lexer-parser pipeline, properly abstracted and contracted

So less important than the sophistication of the algorithm is the intelligent interconnection of the components. And ascii decoder seems like a good place to start

Though, unicode is the goal

Eventually

You guys would be horrified, horrified by this OO class

The teacher, bless his little heart, is too timid for the line of work. He's like a golden retriever puppy whose wandered into a den of honey badgers

[fallen slate]

Hey @deep nova!

It looks like you tried to attach file type(s) that we do not allow (.pdf). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a, .csv, .json.

Feel free to ask in #community-meta if you think this is a mistake.

[raven ridge]

one nice property of starting with ASCII is that, like I said, all ASCII is valid UTF-8. If you start with an ASCII decoder, then extending it to a UTF-8 decoder just requires turning some error cases into success cases - but the original ASCII decoding is still in there.

[deep nova]

Our midterm

[raven ridge]

this seems... not very on-topic

[deep nova]

Thus far the course (all 7 weeks of it) has been roughly equivalent to the first half of a four hour welcome-to-python tutorial

As for on-topic to this channel — sorry, yeah, not on topic. I just thought you might be interested

[thin anvil]

When compiling Python with ./configure --enable-optimizations --with-lto, why does link-time optimisation (LTO) gets triggered during the profiling phase of profile-guided optimisation (PGO)? I thought LTO only improves code locality, while PGO does most of the lifting by optimising hotspots in the source code?

I've never touched the original C family and compilers, a lot of what I know is handwaving through Wikipedia and blog posts

[gray galleon]

why is leetcode's runtime measurement so inconsistent
say these examples:
https://leetcode.com/problems/number-of-steps-to-reduce-a-number-to-zero/submissions/910916903/
https://leetcode.com/problems/number-of-steps-to-reduce-a-number-to-zero/submissions/910916694/
literally the same code
one is 20 ms
other is 40 ms
is it a leetcode problem or python problem?

[flat gazelle]

leetcode is just like that

actually benchmarking things properly and repeatably is way too expensive

[gray galleon]

why bother having a runtime tab if its so inaccurate 🤨

[grave jolt]

to make you feel better than others

Actually a more useful metric would be something like an "estimated complexity", with a chart or something like that

[sharp plover]

It's generally good enough to distinguish between e.g. an O(n) solution and an O(n^2) solution. The goal of these exercises is usually to find the lower time-complexity solution.

[hazy gorge]

@cobalt stream

[warm breach]

though to be fair wall time is important as well. Many theoretical good time complexity algorithms are slower in practice due to constant overheads

and things like cache locality aren't really considered by time complexity

and I think it's close to impossible to actually programmable determine time complexity for most turing complete languages, or even give a good estimate

[grave jolt]

Make a scatter plot of runtime vs N 🙂

[warm breach]

that's kind of the same issue though

with python you might get one going up and down for the whole thing

[swift imp]

@feral island Would you know who to best get in contact about getting an answer to my question on this issue? It looks like per the dev guide there is no official expert for pdb, and no one is replying to the issue.

https://devguide.python.org/core-developers/experts/index.html
https://github.com/python/cpython/issues/90095

[feral island]

I'm not sure there really is anyone. You can try posting at discuss.python.org I suppose.

[swift imp]

What section would be best bc I made a post in help, a few months ago, and its been crickets.

[feral island]

maybe core development. sorry, can't guarantee you'll get any response though

[unkempt rock]

!pep 701

[fallen slate]

**PEP 701 - Syntactic formalization of f-strings**

Link

Status

Draft

Python-Version

3.12

Created

15-Nov-2022

Type

Standards Track

[unkempt rock]

This PEP supersedes PEP 536 and aims for a formal grammar for f-strings.

[rose schooner]

still a draft so we don't know when that'll come

[feral island]

it's likely to be accepted very soon, the SC just asked for a minor change

[frank otter]

I'm going to stop using print debugging and start using the debugger

[warm breach]

is there an explanation somewhere of what deepfreeze modules are and what they do

PyCodeObject is also apparently defined with a macro due to something to do with deepfreeze?

[gray galleon]

should range have a specialized sum?
naively summing a range is O(n) where n is the number of elements
however, the sum of range is mathematically known and easy to calculate (have O(1) complexity)

[rose schooner]

at this point just use the formula then

[grave jolt]

yeah it seems like a pretty niche use case

also there's no __sum__ dunder so you'll have to add special case logic to sum

[native flame]

i guess you could make a case for more builtin functions to have customizable behaviour via dunders

apart from sum, max and min seem like they could be specialized in a couple of cases too

[swift imp]

I don't see why sum couldn't recognize it's a range object

[native flame]

i dont think this is important enough to get special cased

a __sum__ dunder would be better imo

(if we have this at all)

[quick snow]

I don't really see enough usecases for this to warrant a new dunder. That said, I could say the same about __pow__, which for dinner reason exists.

[grave jolt]

Why should it?

What's the benefit of doing so?

It would be kind of strange to define this special-case into the specification of sum. And if it's only an optimization specific to CPython, people shouldn't rely on it

So if someone wants to sum a lot of ranges for some reason, they're free to make their own function like sum_range : range -> int

[swift imp]

Summing an arbitrary iterable, there's no way of knowing before hand what the most optimal algorithm is but with a range, there exists a closed form solution, negating the need to actually iterate. Maybe it's an argument for the existence of a sum dunder

[grave jolt]

Sounds like Python needs type classes 🙂

[swift imp]

Then again most people don't have pure python for math anymore

[swift imp]

What is that

[grave jolt]

It's a polymorphism mechanism popular in Haskell, Rust and some other languages. The idea is that instead of "virtual dispatch" (i.e. adding a __sum__ method to every object that supports summation) or special-casing (adding an extra if to the built-in sum), you define a set of "pairs" (type, implementation). For example

trait Sum {
    type Total;
    
    fn sum(self) -> Self::Total;
}


struct Range {
    start: u64,
    end: u64,
}


impl Sum for Range {
    type Total = u64;
    
    fn sum(self) -> u64 {
        if self.start >= self.end {
            0
        } else {
            ((self.end - self.start) * (self.end - 1 + self.start)) / 2
        }
    }
}


impl<Ts: Iterator<Item=T>, T: Default + std::ops::AddAssign> Sum for Ts  {
    type Total = T;

    fn sum(self) -> T {
        let mut acc = T::default();
        for it in self {
            acc += it;
        }
        acc
    }
}

1. If you have some random iterable, you can do iterable.iter().sum()
2. If you have a range specifically, you can do my_range.sum()
   (Though I did add a 🙂 because this isn't really possible in Python, as it kinda needs static typing)

The difference is that you can implement this Sum for your own type, and if you're in a module that defines Sum, you can define summation for other people's types

[sour thistle]

isn't that just monkeypatching with extra steps?

[grave jolt]

huh?

how is that monkeypatching?

[sour thistle]

1. If you have some random iterable, you can do iterable.iter().sum()

[grave jolt]

Monkeypatching means mutating classes and such at runtime.
It's closer to extension methods in Kotlin/C# which are very different from monkeypatching

In other words, the particular function to call is resolved statically

(in Haskell it doesn't use method notation, doesn't really matter)

[warm breach]

Yeah this is nice. I miss extension methods from c# as well

wonder what the syntax might look like if we get it in python

[grave jolt]

aren't extension methods just hacks to avoid the pipe operator 🙂

Well, at least in Python, where there's no "auto-self"

before:

foo.bar().baz()

after:

foo |> bar |> baz

[warm breach]

not sure I like the idea of pipe operators, still functions you have to import and know about

[grave jolt]

You still have to import extension methods

[warm breach]

extension methods show up in IDE autosuggest

[native flame]

should be possible to have autosuggest for pipe operators too though right

[grave jolt]

Yeah I guess autosuggestions are one of those things...

[warm breach]

if the extensions class "runs" it should apply all of those methods

[feral island]

A nice feature that extension methods would give is that additional methods can come for free. Like an extension method for .length() might implicitly create a method for .is_empty() that you don't have to write yourself.

[grave jolt]

You could, of course, search for [functions that take type(x) as the first argument]

[native flame]

right

[grave jolt]

doesn't sound like rocket science, Hoogle already exists 🙂

[native flame]

as the only argument 🤔

[grave jolt]

Depends on how the piping is implemented as a feature. You could have something like

foo |> bar(?, 42) |> baz

[native flame]

mhm

[warm breach]

yeah and it'd be easier to add something like .first() .last() to all Iterables

I suppose extensions would have to go by methods and not types?

[native flame]

pipes have been rejected already havent they

what was the rationale

[grave jolt]

I guess there isn't a good way to hack them into Python 🤷

You could have your own thing like ```py
P(foo) >> (bar, 42) >> baz >>()

[native flame]

i like how you suggest a good way to hack them into python

[warm breach]

also c# extension methods has the capability over rust to override syntactic operators as well 🥴

[grave jolt]

(which would be the same as baz(bar(42, foo))

[grave jolt]

I mean, as a language feature. You could implement my snippet with no changes to the base language

[native flame]

why couldnt it be a language feature

[grave jolt]

What justifies the addition of such feature, if it can be replicated by a small class?

class P:
    def __init__(self, item):
        self._item = item

    def __rshift__(self, other):
        match other:
            case ():
                return self._item
            case (fn, *args):
                return P(fn(*args, item))
            case fn:
                return P(fn(self._item))

[warm breach]

I feel like it might just make calls more confusing, another way to do the same thing

[native flame]

arguably abusing >>, you dont get convenience features like foo |> bar(?, 42)

[grave jolt]

I guess Python is just not very well-suited for functional programming 🤷

[warm breach]

yeah it doesn't really push for that paradigm

[grave jolt]

Or another one: use intermediate variables:

x = foo
y = bar(x, 42)
z = baz(y)

[warm breach]

I don't think any non "functional" claiming languages have pipe operators either?

[grave jolt]

bash 🙂

[native flame]

idk does julia count

[grave jolt]

IIRC there was a proposal for JS but it got rejected

[warm breach]

what paradigm is bash even

it doesn't have objects at least

but functions allow side effects

[grave jolt]

dysfunctional programming

[warm breach]

unfortunately this does affect the lifetime of those assignments though

[grave jolt]

wdym

[warm breach]

if you chain those calls the intermediary would be GCd right after

assignments would last until end of scope

[grave jolt]

x = foo
x = bar(x, 42)
x = baz(x)

[warm breach]

mypy would be angry at that though 😔

[grave jolt]

sounds like a mypy problem 🙂

[warm breach]

and I think several linters

[grave jolt]

x = chain(foo, [lambda x: bar(x, 42), baz])

[warm breach]

but yeah that seems to be the recommendation python wants

break things up into intermediaries instead of giant chains

[grave jolt]

sometimes a chain is pretty good 🤷

Organizing your code as a pipeline is a valid decomposition strategy

[warm breach]

seems to make sense

though I haven't done much functional stuff

[grave jolt]

i have recently gotten into an Elm binge 🙂

[warm breach]

kind of want () => lambdas in python now 😔

map(f => f.name, seq)

instead of

map(lambda f: f.name, seq)

[grave jolt]

me when
the when the
when me is py map(new Function<Foo, String> { String run (Foo foo) { return foo.name; } })

[warm breach]

kind of annoyed that typeshed uses Callable to type functions

throws away a lot of commonly used attributes that functions have

like __name__

meanwhile if I type a function to accept types.FunctionType now mypy thinks functions are incompatible with FunctionType

wonderful

FunctionType isn't generic though, so maybe we can have a typing.Function?

[grave jolt]

and you haven't even started on how functions are also descriptors!

[warm breach]

why do non bound functions have to be descriptors again?

[hazy fiber]

any tips on a beginner to learn python, maybe a youtube channel or free course or something please

[quick snow]

So they can return a bound method when looked up on an instance.

[warm breach]

ah hm

[quick snow]

(and bound methods don't need to be/aren't descriptors)

[slow sequoia]

Im a newbie who wants to learn python at master level please tell me the best way to learn a language

[meager arch]

c++ kinda does in the stl

(ranges)

[neat delta]

So I'm compiling a .pyi of all the dunder methods in the order specified by the datamodel, and I've hit the metaclass methods: https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks .
Documentation gives the signature as class.__instancecheck__(self, instance), but pylint complains that the first argument of a metaclass method should be cls, not self, and pep 3119, which defines ABCs, agrees. So why does the documentation say self?

!pep 3119 (for reference)

[fallen slate]

**PEP 3119 - Introducing Abstract Base Classes**

Link

Status

Final

Python-Version

3.0

Created

18-Apr-2007

Type

Standards Track

[warm breach]

when the method is on an ABC the self argument is indeed an instance type so imo the self makes more sense

typeshed also uses self it seems https://github.com/python/typeshed/blob/main/stdlib/builtins.pyi#L181

[fallen slate]

stdlib/builtins.pyi line 181

def __instancecheck__(self, __instance: Any) -> bool: ...```

[warm breach]

though

typing uses self https://github.com/python/cpython/blob/main/Lib/typing.py#L464

[fallen slate]

Lib/typing.py line 464

def __instancecheck__(self, obj):```

[warm breach]

but abc here uses cls https://github.com/python/cpython/blob/main/Lib/abc.py#L117

[fallen slate]

Lib/abc.py line 117

def __instancecheck__(cls, instance):```

[neat delta]

and 3119, which introduced it, uses cls. more example of python stdlib's super-consistent nature i guess

[quick snow]

For metaclasses, the self is a class, so both seem fine to me..

[warped adder]

Has any progress been made on the macros pep?

[spice pecan]

If you're referring to PEP 638 "Syntactic Macros", it's still a draft

[burnt void]

is there any channel here for getting tips/help?

[grave jolt]

Have you seen #❓｜how-to-get-help?

[real copper]

What ide is that?

[sacred yew]

looks like pycharm

[cyan raven]

pycharm

[warm breach]

struct _frame {
    PyObject_HEAD
    PyFrameObject *f_back;      /* previous frame, or NULL */
    struct _PyInterpreterFrame *f_frame; /* points to the frame data */

what's the difference between

Type *name;
struct Type *name;

[feral island]

depends on how the types are defined

if it's defined as just struct X { T y: ...} then only struct X works

[warm breach]

here it's a

typedef struct _PyInterpreterFrame {
  ...
} _PyInterpreterFrame;

[feral island]

it's common to write typedef struct X { ... } X; and then both work

seems like that's what's done here

[warm breach]

both work? 👀

oh..

since it's ambiguous without struct whether it refers to the struct or another function or something?

unless it's typedef

[feral island]

it's just how the namespacing works in C I think

[pliant tusk]

in C you can use typedef to create an alias to any predefined or user type, and since struct T {} is a user type you can alias it directly to make it easier to use. for example you can also use typedef for a function type like this typedef void (*myfunc)(); where now myfunc is the type of a function pointer that takes no arguments and returns nothing

[flat gazelle]

you will generally see struct A *name if you need to use struct A before it is defined, e.g. during (mutually) recursive struct definitions

[warped slate]

hi

[silk belfry]

hello

[warm breach]

wait, int is no longer a PyVarObject in 3.12?

https://github.com/python/cpython/blob/main/Include/cpython/longintrepr.h#L82-L90

[fallen slate]

Include/cpython/longintrepr.h lines 82 to 90

typedef struct _PyLongValue {
    Py_ssize_t ob_size; /* Number of items in variable part */
    digit ob_digit[1];
} _PyLongValue;

struct _longobject {
    PyObject_HEAD
    _PyLongValue long_value;
};```

[warm breach]

python/cpython#101291 interesting, guess we're getting dynamic intrinsic ints like in python 2

[neon trout]

GitHub

[cpython] #101291 Restore (or beat) Python 2 performance for arithmetic operations on ints that fit into a single word

[ruby pilot]

Whats the point for the addition of sys.exception in python 3.11? It seems like the entirety of its functionality is already covered by just catching BaseException. Where can I find information about the motivation for this addition? Maybe an RFC?

If anything this seems like a step backwards, relying on the sys module to perform tasks that we have dedicated syntax for

[raven ridge]

https://bugs.python.org/issue46328 - it's a step towards getting rid of sys.exc_info, it seems

[ruby pilot]

Thanks for the link. Definitely makes more sense now, since exc_info is even worse and this is at least a step forward now.

Still seems like we would want to get rid of it eventually anyway since I think most people would agree that reading the global state of the program can lead towards confusing and difficult to maintain code, although that definitely won't be happening anytime soon

[raven ridge]

🤷‍♂️ It's unlikely to ever go away from the C API, since it's basically the way to check for an exception having been raised. Which means it's unlikely to ever go away from the Python API, too.

having the ability to read global state is useful, even if you shouldn't depend on it in most cases.

For what it's worth, C++ has https://en.cppreference.com/w/cpp/error/current_exception

[ruby pilot]

Makes sense, I was mostly worried about the docs promoting its usage here https://docs.python.org/3/reference/compound_stmts.html#except-clause

case study of it affecting someone #1085037552421052446 message

maybe not "promoting its usage" (rather just explaining how exceptions work) but still, it's in a pretty high traffic area

[gray galleon]

is StringIO implemented as a dynamic array similar to python lists

[royal panther]

@gray galleon the cpython impl can be read at https://github.com/python/cpython/blob/main/Modules/_io/stringio.c - it has multiple modes of operation but one can naively think of it as platform level array of UCS4 codepoints (there are optimizations)

[gray galleon]

yeah definitely dynamic array

[rose schooner]

well it technically still is

Py_SIZE(longobj) still works as it has a structure equivalent to the old one

[warm breach]

many people were calling ->ob_size though

and I'm not sure if this new struct may in fact become a single byte or some other data

it looks like that's the intention

[rose schooner]

and that is why use the python-provided API macros/functions

[warm breach]

to be fair not even CPython uses their own API

if you look at the diff

[rose schooner]

actually that should work

assuming it's casted to a PyVarObject *

that still works yeah

[warm breach]

I think usually it's casted to PyLongObject.

especially if you're accessing ob_item as well

[rose schooner]

you can't do ->ob_size in a PyLongObject

or more accurately a PyLongObject *

[warm breach]

looks like ob_digit is being accessed for some reason

though that might just be cython

[rose schooner]

there isn't an official API for the digits of PyLongObject * though

so probably optimizations

[warm breach]

I think they're looking to add some

especially if that field will be dynamically a single int or that array struct

looks like I'll need to add another override for 3.12 PyLongObject in einspect though 😔

[dusk comet]

i thought PyObject* is a pointer to dummy struct without fields, so users can't access any fields in it
to work with actual object users should use API functions or cast PyObject* to pointer to real struct

[rose schooner]

it contains GC, refcount, and type info
that's it

[rose schooner]

although the GC info is controlled by a macro and isn't required

[raven ridge]

My little cereal is all grown up

[sturdy fjord]

sus 👀

[uncut sage]

I had a help channel open for this question, but I think it might be suitable for here.

#1085243980423364678 message

I am setting up new projects using pyproject.toml to organize the project information and dependencies. I don't want to use dependencies.txt to hold the deps, so I figured out how to create both normal and additional (e.g. dev) dependencies in a pyproject.toml file and install them with pip install . and pip install .[dev]. However, this creates within the project, an install of the project itself -- that is, the project is listed in the pip list for the project's virtual environment, etc. Is this a behavior I should try to work around, ignore? Or is there another way I should be adding requirements to a new project that I want to use pyproject.toml on as a modernization step?

[warm breach]

it correctly installs into the environment, yes

since you'll need it for running tests without implicit imports and such

pip list doesn't especially exclude your own project

and wasn't really meant to be used as a lock file really

the only sane approach I've seen is poetry's poetry.lock

[uncut sage]

ok, I had not considered the angle about tests, can you elaborate on that @warm breach ?

[warm breach]

like when you have a src folder, it's usually because you want to test the actual package that ends up installed rather than the implicit import

if you just test the same directory import, your final package may well be empty and the tests still pass

so by installing and testing, it's the same "conditions" that a user would have, you're also testing that your package / wheel works

[uncut sage]

right! that makes sense, and I've built stuff of my own that had that exact behavior

This is all territory I hadn't explored until now, so it's useful to know.

[warm breach]

Anyone know what change caused PyTypeObject->tp_subclasses to be "May be invalid pointer" in 3.12 https://docs.python.org/3.12/c-api/typeobj.html#c.PyTypeObject.tp_subclasses

[rose schooner]

so i still may be doing some shenanigans that are somewhat like "not using the API"
https://github.com/thatbirdguythatuknownot/sniplections/blob/main/c-extensions/fast_itertools.c#L34-L37

[fallen slate]

c-extensions/fast_itertools.c lines 34 to 37

#define get_index_from_values_order(v, i) ((char *)v)[-3-i]
#define DK_ENTRIES(dk) (PyDictKeyEntry*)(&((int8_t*)((dk)->dk_indices))[(size_t)1 << (dk)->dk_log2_index_bytes])
#define DK_UNICODE_ENTRIES(dk) (PyDictUnicodeEntry*)(&((int8_t*)((dk)->dk_indices))[(size_t)1 << (dk)->dk_log2_index_bytes])
#define DK_IS_UNICODE(dk) ((dk)->dk_kind != DICT_KEYS_GENERAL)```

[warm breach]

aren't these just the same macros from pycore_dict.h

[rose schooner]

yes

i also include internals
https://github.com/thatbirdguythatuknownot/sniplections/blob/main/c-extensions/fast_itertools.c#L2-L8

[fallen slate]

c-extensions/fast_itertools.c lines 2 to 8

#define Py_BUILD_CORE
#include "Python.h"

#include "internal/pycore_pyerrors.h"
#include "internal/pycore_pystate.h"
#include "internal/pycore_long.h"
#undef Py_BUILD_CORE```

[rose schooner]

idk why i didn't think to just include internal/pycore_dict.h though

[warm breach]

well it's not always there

pycore_dict.h is new in 3.11

[rose schooner]

dictionary keys are also split in 2 types only in 3.11+

[warm breach]

hm?

seems like PyDictKeysObject exists in 3.10

can't find the definition though

[rose schooner]

i think we've gone through this before

all i remember is that it was in Objects/

oh yeah
https://github.com/python/cpython/blob/3.10/Objects/dict-common.h

[warm breach]

o right

[rose schooner]

normal entries and unicode entries

[warm breach]

btw making a docstring mini syntax for denoting source versions

class PyDictKeysObject(Struct):
    """
    Defines a DictKeysObject Structure.

    ..
        source: Include/internal/pycore_dict.h (struct _dictkeysobject) #[59827ad2d5]
        source[<3.11]: Objects/dict-common.h (struct _dictkeysobject) #[12e0897a69]
    """

[warped adder]

Why does PEP 0333 not support "hop-by-hop" headers?

I'm trying to implement websockets over WSGI and I really can't understand the logic here

[sturdy timber]

Ooooh nice, pep 701 was accepted! https://peps.python.org/pep-0701/

[rose schooner]

they removed the register stuff for 3.12
https://github.com/python/cpython/commit/675b97a6ab483573f07ce6e0b79b0bc370ab76c9
python/cpython#102738

[neon trout]

GitHub

[cpython] #102738 remove register instruction related code from the cases generator script

[rose schooner]

Since we won't have register instructions in 3.12, we should remove this from the generator for now.

[hard horizon]

!

[grave jolt]

?

@hard horizon Make sure to carefully read #voice-verification if you want to verify.

[slow sequoia]

I want someone to break my python codes anyone please

[warm breach]

what do you mean by break

[slow sequoia]

Make it full of errors

[slow sequoia]

By editing it

Im learning python from this book

And the authour tells me to break the code

So i need someone's help please

[limpid forum]

It's a task for you to break the code, so you don't need anyone's help. You're just supposed to play with the code and see what happens

[slow sequoia]

But i he said that a friend can also do that soi was just wondering is there's anyone wh can help me

[limpid forum]

No, it says you can take a friend's code to break. So you're still the one who does the breaking

[slow sequoia]

I still dont get how t break a code i can just put some wrong variable , some wrong signs

[limpid forum]

It says that you were already given tasks to break code before. What did they say? I assume it's more of "what happens if you change variable's value" etc rather than just making the code unusable. Because it's too easy to just make code unusable, you just need to break syntax and it won't even run

[slow sequoia]

Yeah i als want to say that

Its too easy

What's the point

When actually a code is broke

It's just takes 1h for me

But break a code with my own hands?

[feral island]

I think what they mean is probably along the lines of "don't edit the code, but pass unexpected inputs to it and see what happens"

[slow sequoia]

But he said to break the code

[slow sequoia]

@feral island still thanks i will do that too

[feral island]

As @limpid forum said, "breaking" the code by changing it so it doesn't run is so easy that it's not a useful exercise. So I think what the author must have meant is what I said above

[slow sequoia]

So

I just need to play with it fr a bit right?

[warm breach]

like if you made a

def some_math(a, b):
   return a + b / 2

what happens if you pass a string or other object into it? Is the error expected or handled

or a

num = input("enter number")

what happens if you enter a non-number?

[slow sequoia]

This one will works anyways this just wants an input

Wait sorry it was an example i guess

[sly fiber]

He shows you in the extra features on his free vids how to do it, You'll have to fil in a question to get those free extra videos per exercise, by filling first in the ISDN nr of the book. Then a question or couple are asked on which page and which row and side which word you see, is how get all his free material/vids from this book, also for the follow up "Learn More Python 3 the Hard Way"

[slow sequoia]

@sly fiber Link please

[slow sequoia]

Also should i consider taking notes ?

[sly fiber]

Sure you can, is what he told somewhere at beginning of the book if i remember it well, it's always good to take some notes if not understnad something immediately, and google is your best friend for it to find things out as he mentioned 😉

[slow sequoia]

Yes i can but the question is should i and do pepole take notes in this skill?

[sly fiber]

braking code is good to learn how to 'debug' it actually, so it's not that bad, on his free vids he shows how and stuff, i learned pythong (and still find my self beginner) on his 2 books, and also the other book ehm, Think Pything: How to think like a computer scientist (or something)

those 3 i used to learn myself, without before having no knowledge at all about coding stuff

is why i can recommend these 3 books, cause i don't know other ones, and it helped me pretty well

is just a hobby for me actually ^ ^

[slow sequoia]

Did you take notes at any time

[sly fiber]

if it's needed i do,

[slow sequoia]

Then its a yes

[sly fiber]

🙂

[slow sequoia]

I will also take notes for each chapter i guess

[slow sequoia]

I want to master it

Im going to master it

[sly fiber]

is a good options, when go later back to them, you'll find out when watch your notes they helped you in time

[slow sequoia]

Btw

Vids ?

What vids can you share them

[sly fiber]

is only to never give up,

he share for each exercise a video, can find it at beginning of his books the link for it and fill in ISBN nr from the book, even if it's a free pdf book 😉

[slow sequoia]

I pirate his book

I have a epub of his book

Actually i literally pirate everything

[sly fiber]

well, thhose also have the ISDN nr at front somewhere

so can use it to get his free extra videos that accompany his books

[feral cedar]

this seems slightly off topic (and also illegal)

[sly fiber]

i bought them, but i know some who did it that way, is all i can say psv

[sour thistle]

<@&831776746206265384>

[warm breach]

anyone know why python on windows can prevent null pointer access with an OSError and traceback while on linux it just segfaults?

is the windows way of access violations fundamentally impossible on linux or would affect performance?

[slim drum]

guys if i type somehow typing lets say

from random import *

t1 = Triangle()

can i do something like t1.randint

[warm breach]

where is Triangle defined

since I don't think random has a Triangle

[slim drum]

what object can i create with random

for turtle i can do something like t1 = Turtle() if i import it

[warm breach]

yeah, Turtle is a class

[slim drum]

does random have a class?

[warm breach]

there's random.Random or random.SystemRandom classes

but the module also has many functions

[slim drum]

im new to OOP and its pretty hard before i learned functions loops etc

[slim drum]

thanks a lot

[pliant hill]

is there a good reason for this being possible ```py

int = str
int
<class 'str'>
test = int("0")
test
'0'```

[sour thistle]

• adding special cases increase the complexity of the language overall, which has a lot of downsides
• that is not something you are likely to do by accident

where would you draw the line between "allow assigning any name" and "everything is a constant"?
there are some builtins you'll often want to overwrite such as id, max, min (even though that is not particularly recommende), and shadowing globals inside of functions is not too uncommon

[pliant hill]

I find these to be good reasons

I just happened to stumble across it and intrigued that it was possible

[sour thistle]

to be fair there are also a bunch of things that are the way they are just because someone thought it would be better that way for a short while in the past (even if they later changed their opinion), and/or because changing it would take an unreasonable amount of effort (oftentimes not only from the language developers, but also the language user's)

[grave jolt]

there are some builtins you'll often want to overwrite such as id, max, min
those are usually shadowed as locals though

though there's cursed stuff like LOAD_NAME...

[pliant hill]

I'm not even sure why I was even trying this in the first place tbh

[warm breach]

!e hi there, heard you talking about cursed stuff 👀

from einspect import view

view(print).swap(max)

max(print(1, 2))

[fallen slate]

@warm breach :x: Your 3.11 eval job timed out or ran out of memory.

2

[pliant hill]

this works effectively max,print = print,max

[naive stratus]

Hello. Can you suggest tutorials for a beginner like me?

[dusk comet]

What is the point of LOAD_NAME? Isnt it the same as LOAD_GLOBAL?

[grave jolt]

!e

import dis

@dis.dis
def f():
    class X:
        x = 1
        y = x

[fallen slate]

@grave jolt :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 |   3           0 RESUME                   0
002 | 
003 |   5           2 PUSH_NULL
004 |               4 LOAD_BUILD_CLASS
005 |               6 LOAD_CONST               1 (<code object X at 0x7feb45bb01d0, file "/home/main.py", line 5>)
006 |               8 MAKE_FUNCTION            0
007 |              10 LOAD_CONST               2 ('X')
008 |              12 PRECALL                  2
009 |              16 CALL                     2
010 |              26 STORE_FAST               0 (X)
011 |              28 LOAD_CONST               0 (None)
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/otizipenak.txt?noredirect

[grave jolt]

(flashbacks to that cursed example from guido's twitter)

[dusk comet]

So it is used in "dynamic" scopes where it is not possible to know it variable is local or global, right?

IIRC you can use custom mapping for class namespace, so LOAD_NAME should care about it

[grave jolt]

yeah it's a can of worms kinda...

oh right there's __prepare__ and stuff

[verbal escarp]

how to intercept arbitrary method calls in order to proxy a class similar to __getattribute__?

especially __call__?

[verbal escarp]

i found a solution in https://github.com/GrahamDumpleton/wrapt/blob/develop/src/wrapt/wrappers.py - ```py
def call(*args, **kwargs):
def _unpack_self(self, *args):
return self, args

    self, args = _unpack_self(*args)

    return self._Proxy__implementation(*args, **kwargs)```

heh

apparently he didn't find a more elegant solution either

[neat delta]

!pep 701 this was accepted a bit ago, but isn't in the what's new for 3.12 yet. Is it known if it's going to be (and the process is taking a while) or if it's getting postponed to 3.13? The current schedule (pep 693) only has a single alpha left before the beta freeze hits

[fallen slate]

**PEP 701 - Syntactic formalization of f-strings**

Link

Status

Accepted

Python-Version

3.12

Created

15-Nov-2022

Type

Standards Track

[spice pecan]

I'm kinda late to the party, but one important thing to note is that builtins exist in a scope below (or above, depending on how you look at it) global variables. By doing simply int = str you create a new global variable int that points to str, if you do del int, it will work as normal again, and it also will only affect the current module and the modules that star-import it. If simplified, name resolution boils down to

Is variable found in the current scope? Check the one above
Ran out of nested scopes? Check global variables
Not in globals either? Check builtins

So if you *really * wanna fuck around, you can do import builtins; builtins.int = str, which actually replaces int in that scope and will affect all modules and won't be fixed with a simple del int :)

[pliant hill]

good to know

so time to make a randomizer

[dusk comet]

im getting infinite error message after doing builtins.int = str, funny 😃

repl is so easy to break

[spice pecan]

i mean yeah that breaks basically every call to int

[feral island]

it will be in 3.12

[rose schooner]

i think the "What's New" part is only supposed to be done after the change

[lapis dew]

Anyone knows which line numbers contains errors?

I know is 3, 10 but that's not enough

[native flame]

please see #❓｜how-to-get-help and post your question in #1035199133436354600

[rose schooner]

https://github.com/python/cpython/commit/7559f5fda94ab568a1a910b17683ed81dc3426fb

i think i need to change some of my modules for this one

[south hearth]

def rev(n):
   return int(str(n)[::-1])

why wont it allow 0's as a string any idea?

[rose schooner]

wdym?

[south hearth]

try rev(200)

or 100

or any number with 0s

[rose schooner]

yeah that should be 002 AKA 2

i don't know what you're going for here

[south hearth]

oh right im converting back to int

my bad

[edgy stirrup]

Good morning, I’m new in here and working with Python, I used to work as software engineer but was 1996 using other languages that probably doesn’t existed right now like Cobol and C+. Well hope to get better ideas here and focus on this again. Thanks

[wary oyster]

be the greatest programmer ever

thats all you have to do

not that hard tbh

[languid pebble]

i have a question regarding the official docs and syntax:
https://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements

With the exception of assigning to tuples and multiple targets in a single statement, the assignment done by augmented assignment statements is handled the same way as normal assignments.
what does "assigning to tuples" mean?
i can't find any reference to this anywhere else in the docs, only assigning to targets, the 'and' implies that these are two different things

[raven ridge]

I think it's talking about doing (x, y) = z, and pointing out that augmented assignment is different because you cannot do (x, y) += z

[languid pebble]

what of the distinction between tuple assignment and (multiple) target assignment? is it a mistake?

[umbral plume]

I think multiple target assignment might actually be referring to x = y = z, since there's no x += y += z

[languid pebble]

it doesn't seem like it, if you look at the definition of the target_list
https://docs.python.org/3/reference/simple_stmts.html#assignment-statements
i'm really wondering about that "tuple assignment", it doesn't occur anywhere else in the docs that i'm aware of

[languid pebble]

i'm thinking it's a mistake because of the superficial similarity to tuples like here

[flat gazelle]

well, the syntactic element of (a, b) is called a tuple view, so it isn't wrong

[languid pebble]

i haven't heard that name before, is it an official one?

[flat gazelle]

ah nvm, it is a display

[languid pebble]

[a, b] = x, y is equally valid, so that opens the door to more questions

[flat gazelle]

https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries

though that does also include comprehensions

though the grammar in the docs does just explicitly list like so```target ::= identifier
| "(" [target_list] ")"
| "[" [target_list] "]"

but you are right, it is somewhat oddly said

[languid pebble]

reading further into the definitions... isn't it explicitly not a display?
the (x,y) of (x,y)=a,b is a target_list, which can be "(" [target_list] ")",
it's not a parenth_form which could in some contexts be considered a tuple display, which is defined as
"(" [starred_expression] ")"

they're defined separately, there's no connection that i can find

[flat gazelle]

it is a Tuple node in the AST, and error messages do call them displays

    {a} = 1
    ^
SyntaxError: cannot assign to set display

In [2]: ast.parse('(a,b) = c')
Out[2]: <_ast.Module at 0x1abdef96b50>

In [3]: ast.dump(_)
Out[3]: "Module(body=[Assign(targets=[Tuple(elts=[Name(id='a', ctx=Store()), Name(id='b', ctx=Store())], ctx=Store())], value=Name(id='c', ctx=Load()), type_comment=None)], type_ignores=[])"

[sturdy timber]

The parenthesised form isn't what makes a tuple a tuple (apart from empty tuple I guess), a tuple is created using an "expression list" https://docs.python.org/3/reference/expressions.html#expression-lists

[languid pebble]

that i know, there isn't anything officially called a tuple display either from what i can see

[languid pebble]

the ast node seems more like a convenience thing, i'm not sure i could consider that an authority, but the error message is definitely interesting

[sturdy timber]

I think the error message just comes from a "if parsing fails, try to parse any expression as the LHS and raise an error with the name of that expression" type logic

Rather than suggesting that what is on the left is a sort of expression

[languid pebble]

to summarise:
the language specification defines target lists and displays as separate concepts (the exception being this one odd mention of tuples in the docs),
the ast parser stores them into the tree using the same structure,
and the implementation uses some shared terminology/routines

in isolation, the syntactical unit of an expression list (or a display, if i understand it right) is a superset of the target list, so it makes perfect sense how you'd use shared means of dealing with them

[slow sequoia]

What is happening here at line 4

What type of variable assigment is this argv?

I know that much that argv picks up input from the cmd line only

But what does it does exactly?

[dusk comet]

!e ```py
x = [1, 2]
a, b = x
print(a, b)

[fallen slate]

@dusk comet :white_check_mark: Your 3.11 eval job has completed with return code 0.

1 2

[slow sequoia]

@dusk comet Still what' this argv for ?

[dusk comet]

please see #❓｜how-to-get-help and post your question in #1035199133436354600

[bitter kiln]

have you see CLI apps like ping or nmap
for example when you use ping
your syntax is like "ping 192.168.100.100"
the url part is one of arguments you send to app
and argv gather these argumants and return them as a list for you

[slow sequoia]

I get it

Its just a list of values passed in a cmd line

[bitter kiln]

yeh

[umbral plume]

I kinda wish python had rust-style enum support (i.e. typed unions, and similar enum Foo syntax for making them)

i feel like it'd also mesh pretty well with match statements, since you'd be able to create more fleshed out patterns than combinations of lists, tuples, and literals

[radiant garden]

type unions are technically already tagged unions

that plus dataclasses is quite ergonomic

Just as long as your union branches don't have a subclass relation you're good

[umbral plume]

I suppose so yeah, but my point with match statements still stands, since there's no built-in way to match by type (other than like case a if isinstance(a, int), but at that point you just want an if statement)

[native flame]

case int(): matches by type

[umbral plume]

wait since when??

[native flame]

since the beginning i believe lol

[umbral plume]

oooh, that's actually half of my complaints with that gone then :P

still, tagged unions let you have multiple enums wrapping the same type, whereas python type unions don't

e.g. there's no ```rs
enum Response {Yes(String), No(String)}

[uneven briar]

are we getting the ? operator

well I mean the whole ? thing, its not an operator

https://peps.python.org/pep-0505/

[sour thistle]

Status: Deferred and last modified over a year ago
probably not, or at least not anytime soon
you can probably look for the discussion around that PEP on mailing lists and such to see why it was deferred

[uneven briar]

where is that

oh

[sour thistle]

you can just throw python mailing list "pep" "505" and alike on google, but some examples of discussions around it:
https://discuss.python.org/t/pep-505-status/4612/13
https://groups.google.com/g/dev-python/c/eMNycEI5kao/m/wiGDkB11AgAJ
https://www.reddit.com/r/Python/comments/966onc/pep_505_noneaware_operators_and_considered_for_38/

[uneven briar]

https://pypi.org/project/pep505/
oh damn

well some people say it hurts readability

but dude x??default is way better than x or default, x if x is not None else default or alike

and its totally explicit, no magic involved

[swift imp]

null-coalescing is not exciting imo. I'd much rather other stuff

[swift imp]

I dislike that we have to do int() and int just doesn't work

[uneven briar]

looks implicit

[native flame]

honestly i think case name: catching everything is very bug prone

[uneven briar]

whats the difference between it and 0?

[native flame]

int() does not mean 0 in case expressions

more generally "calls" arent really calls

[uneven briar]

and thats a bad idea

also for those who say "if you dont know python you can still understand x if x is not None else default but not x??default, there are two things worth noting
1- come on dude, if you are a real python programmer you are gonna know the language's syntax. there is walrus operator which is (I'm almost sure) way more rare than the ? operator in languages, and nothing is wrong with it
2- even if you dont know what it is, you are just going to google "python ? operator" and learn about it. one time, its nothing too complex, and thats it

[native flame]

cant think of a better alternative for matching against type tbh

[uneven briar]

a special thing?

[swift imp]

If the sentinels pep gets added, then u cannot match on singletons

https://discuss.python.org/t/pep-661-sentinel-values/9126/123?u=melendowski

[native flame]

yeah i really wish we got the ? operators

big sad

[uneven briar]

dude they literally added a whole new syntax. why wont they add a special thing for type checking and make an implicit shit instead

[native flame]

well, i was asking what alternative syntax you think would be better

[uneven briar]

type[int] or smthng

[umbral plume]

should probably add the type pattern matching looks like: py match foo: case int(a): # <- int(a) checks whether foo is of type int, and assigns to variable a print(a, "is an int") so its not quite as implicit as case int(): print(...)

[uneven briar]

a bit better

[native flame]

!e

match 1:
    case int(a):
        print(a, "is an int")

[fallen slate]

@native flame :white_check_mark: Your 3.11 eval job has completed with return code 0.

1 is an int

[native flame]

interesting

[swift imp]

That's no good

[umbral plume]

gives me if let() vibes

[swift imp]

What if the match supports __int__

[uneven briar]

try string

[uneven briar]

as there is no real reason to use such a thing other than annotation so it would be ok

[umbral plume]

woah, i didn't actually realise this while testing

dataclasses actually deal with all the stuff regarding pattern matching automagically

[native flame]

how would the attribute matching part tie into that syntax
like case Point(1, y) checks isinstance(foo, Point), foo.x == 1 and assigns y = foo.y

[uneven briar]

if it finds type[something], it checks if x is instance of something

[native flame]

yes what about the pattern matching on attributes

[umbral plume]

i think my main takeaway today is that pattern matching is actually even deeper than I used to think it was 👀

and I may have thought up of a solution to tagged unions

[uneven briar]

?

[grave jolt]

I use unions of dataclasses

big W

[umbral plume]

the conclusion that i came to was pretty much to do the same thing lmao

[rose schooner]

walrus is probably a bad example since it was really debated and was the reason guido resigned from being the sole leader of python decisions

‫the same argument about readability was used against it

[halcyon trail]

I dislike walrus quite a bit more than ?? tbh