internals-and-peps

[verbal escarp]

there are certain cases where you can do a single check on multiple conditions, put that in a tuple and use that tuple as key

to a dict of lambdas

which can be useful if you have 3 or more boolean vars and you need to cover all different cases

[flat gazelle]

Yeah, that's more or less what pattern matching does in rust

[verbal escarp]

you can do that in python too, i've done it before

but it's rare

[flat gazelle]

Just with a lot faster hash than a full tuple

Yeah, in python it's only in isolated cases, and can't ever do it automatically

[verbal escarp]

i think most of the times cases are degenerate

like a list of "if bla: return" as a preamble of sorts and the "real" code is actually in the "end-else" if you'd write it all out

guess that's also what match/case boils down to

btw, i'm just working on a prototype for pip-auto-install.. mapping inherently recursive ideas to global singleton expressions with side-effects is pretty difficult 😐

it would all be so much easier if imports were only allowed on the module global level

if that was the case, i could just replace __import__ when a package is installed and imported and reset after, just like it's doable with the cwd

but instead, i basically need to keep track of the context in which something is imported, replace __import__ by a kind of decorated version which checks from where the import call is being made to be able to replace the call any time in the future by the appropriate version

only to be able to use pip and venv and simplify the heck out of the code

😵‍💫

@paper echo all your fault :p

[paper echo]

😂☠️

[verbal escarp]

xD

well, i'm kind of masochist in that regard, so..

[magic python]

some_list : tuple[str, int, str] = []

apparently this is not ok - having run with mypy --strict i get: Incompatible types in assignment (expression has type "List[<nothing>]", variable has type "Tuple[str, int, str]"), I was under the impression that type-hinting a data structure like lists / dicts at creation was an ok thing to do though.

some_list = [] shuts mypy up, but i lost information about what this list is going to contain.

clearly i've missed a point somewhere 🤔

[charred pilot]

maybe you wanted list[tuple[...]]?

[magic python]

🤦‍♂️ yes lol

i was thinking, i swear i've done this loads... thought something had been added to strict

[verbal escarp]

apropos AST and eval(), this presentation should be mandatory: https://www.youtube.com/watch?v=7e-2nUqnNqE

really cool and crazy what they came up with

[elder blade]

What a savage running arbitrary user code

Get it,, because-

[magic python]

how's one supposed to test / check whether a line has a trailing comment?

eg something like: line = "for i in 'this is # a test #': # pylint: disable=one,two" , i thought about using regex, but feel it could be pretty messy, and this sort of thing must be solved.

[verbal escarp]

read the line and split at '#'?

[magic python]

Have you looked at the example

I don't know how "read the line and split at #" makes sense here? If there can be multiple pound signs etc, after splitting, then what?

[verbal escarp]

either parse the line yourself or use a python parser and look at the CST

https://libcst.readthedocs.io/en/latest/why_libcst.html

[magic python]

Ok, idk how id write a parser

[magic python]

I'll check this, thanks.

What's the approach, read characters while the CST says it's a valid line?

Hmm no, BC it wouldn't be valid for the first few chars I guess

[ancient solstice]

hi guys, can anyone help me by reading my project idea and give me some feedback about the project ?

[magic python]

@verbal escarp do you have any experience with that module? It seems that It wont parse a single line of python, eg 'for i in 'this is # a test #': # pylint: disable=one,two' will throw an error from cst.parse_*, so i'm not sure if i can parse line by line, or have to evaluate a whole file at once etc.

i was intending to iterate over a file line by line, maybe this won't work tho 🤔

[main ginkgo]

is there any easier way to test out changes to cpython's grammar than to recompile it every time?

[brave badger]

libcst requires that the string that you pass to it is well-formed Python code

[magic python]

yeah seems that way, hm

[brave badger]

This is probably what you want:

See also: https://libcst.readthedocs.io/en/latest/nodes.html#libcst.IndentedBlock

>>> py = """
... for i in 'this is # a test #': # pylint: disable=one,two
...   pass
... """
>>> libcst.parse_module(py).body[0].body.header
TrailingWhitespace(
    whitespace=SimpleWhitespace(
        value=' ',
    ),
    comment=Comment(
        value='# pylint: disable=one,two',
    ),
    newline=Newline(
        value=None,
    ),
)

Although it might be better to use the builtin visitors instead. Likewise, this is straying a bit off-topic for the channel.

[magic python]

which channel is this on-topic for

[brave badger]

Help channels would be more appropriate

[magic python]

but of the channels, which is most appropriate?

obviously questions aren't strictly asked in help channels

[brave badger]

We don't have a topical channel strictly dedicated to parsing/static analysis, help channels serve as a catch-all

[magic python]

yeah i know there's a not a single one - but idk if it'd be more estoteric / software-design etc

[verbal escarp]

isn't "advanced-discussion" a catch-all for discussing advanced topics like parsing? ^^ not sure if a help-channel would be better suited tbh

but no, i haven't used libcst extensively

[sacred yew]

see channel topic

[verbal escarp]

"advanced language concepts" - parsing?

shrugs

[sacred yew]

in relation to python language features

not just general advanced topics

[verbal escarp]

would there be any benefit if modules were lazily loaded and any ideas how this could be achieved?

[tender goblet]

not sure whether this is the place to post, but...

Hello, I'm a novice Python programmer (have started learning the language about a year ago) and have recently found out about sets (the data structure). I've noticed that one do things with sets that can't otherwise be done with structures such as lists (like, for example, subtracting one set from another: set1 - set2 works, while list1 - list2 doesn't).

My questions are:
Aren't sets just strictly better? If not, what are the upsides/downsides?

I'm posting here in the hopes of gleaning some info that can't easily be found via a quick google search.

[verbal escarp]

they are different, and since they don't keep insertion order, they have some big drawbacks

[charred pilot]

but also upsides. sets have way faster membership tests

[verbal escarp]

yep.. but since you can't index them, otherwise nice features like random.sample don't work on sets

[charred pilot]

it works for now, but it's deprecated

[verbal escarp]

you could implement __sub__ for lists if you really need it yourself 😉

it's just not very well defined what list1 - list2 means in general

[main ginkgo]

what degree of laziness? load individual module members only when needed or load the whole module when needed?

[tender goblet]

oh, I meant that typing something like

list1 = [1, 2, 3, 4]
list2 = [3, 4]
print(list1 - list2)

doesn't work :P

thanks for the info, btw

[verbal escarp]

i'm wondering that myself. i'm guessing individual members would make most sense?

[peak spoke]

how would that deal with dynamic assignments?

(i.e. even things like changing an attr after the obj is created)

[verbal escarp]

class foo(list):
    def __sub__(self, other):
        return [x for x in self if x not in other]
        
            
L1 = foo([1,1,2,2,3,4,4,5])
L2 = [2,3]

print(L1 - L2)

[undone hare]

Looks like you'd want sets for that

Do you know what those are?

[verbal escarp]

seriously? 😉

[undone hare]

Actually, this channel is for discussion related to the development of the language itself, would you mind grabbing an help channel please? #❓｜how-to-get-help if you don't know how

[peak spoke]

sets were the start of their question

[undone hare]

Okay, I will learn to read

[main ginkgo]

hmm.. one way it could be done is using an import hook to rewrite modules' ast to use a toplevel getattr (see pep 562), or a module proxy with a getattribute and setattribute. the biggest challenge with this would probably be figuring out which members reference other members and inserting the correct stuff to handle that. as for benefits, one use that comes to mind is for monkey-patching a particularly huge module that you cant easily rewrite.

[undone hare]

That doesn't even seem help channel worthy

Please ignore me, thank you

[main ginkgo]

or modules that can use a lot of memory if loaded in all at once

[verbal escarp]

like tensorflow or numpy?

would it make any sense to load those lazily?

[main ginkgo]

not sure, would have to check how much they can be manipulated at the python level

[verbal escarp]

do you have an example where it might be beneficial so i can do some experiments maybe?

[main ginkgo]

hmm.. cant think of any good examples atm but if i do will let you know

[verbal escarp]

excellent

[main ginkgo]

iirc older versions of pyqt had a much larger memory footprint

there's one pyqt module that loads in absolutely every thing

they improved the laziness of that on their side i think

[nova iris]

when i saw that, it reminded me of set builder notation
and then i realized- comprehensions were literally inspired by set builder notation

[stuck valley]

hail satan

I reckon you could just replace the builtin list class.

Definitely.

[main ginkgo]

!e

import forbiddenfruit

def list_sub(self, other):
  other_set = set(other)
  return [
    i for i in self
    if i not in other_set
  ]

forbiddenfruit.curse(list, '__sub__', list_sub)

print([4, 3, 1, 2, 5] - [1, 2, 2, 3])

[fallen slate]

@main ginkgo :white_check_mark: Your eval job has completed with return code 0.

[4, 5]

[main ginkgo]

forbiddenfruit always ready for cursing needs

[dusty verge]

lmao, I didn't know you could do this

that is beyond scuffed

[main ginkgo]

theres also @pliant tusk 's fishhook module

[dusty verge]

I assume it works by mungling around with the c data?

[pliant tusk]

yea

[dusty verge]

Interesting stuff

[main lynx]

anybody have experience with writing/maintaining "safety-critical" python software? i'm hearing a lot of dogma about it being bad, but when i look around the reasons i see for not using Python don't really track

[visual shadow]

safety critical... safety in what sense? could you elaborate?

[flat gazelle]

Highly critical software uses formal verification and other related methods to ensure that the code is correct. This is much easier to do with C/java since there are advanced tools for it.

[verbal escarp]

lol.. I probably wouldn't use this approach myself to implement __sub__ with its O being n², but if the question is whether there might be a reasonable and doable definition/implementation - yeah, why not ^^

[stuck valley]

check fishhook in #esoteric-python

[verbal escarp]

i'm thinking that it might be rather easy to get there with python if only a subset of the language features is allowed, like going all-functional

[verbal escarp]

maybe do AST analysis to check

urgs.

sorry, wasn't for you :p

unhooks the message

[stuck valley]

helo

[verbal escarp]

yo

one could use the type annotations in the AST to define preconditions/postconditions in the sense of the hoare calculus

then it might be feasible to prove the correctness of individual modules before they are executed

https://en.wikipedia.org/wiki/Hoare_logic - something like that, yeah

it wouldn't be "gradual typing" anymore, but "gradual verification" 🙂

way cooler 😉

only drawback i can think of would be that the annotation used for verification would probably be incompatible with the established notation

oh, heh. not even

a: "int # 0 < a < 10"

nobody expects the comment in the annotation!

vscode happily evaluates a to be int

i loved hoare calculus in CS.. good times ^^

very nice

not as in-depth as i was suggesting, but it looks like a good place to start

[rigid vapor]

!e
print('Hello world!')

[fallen slate]

@rigid vapor :white_check_mark: Your eval job has completed with return code 0.

Hello world!

[rigid vapor]

[main lynx]

thanks, being able to prove correctness was the only valid-smelling reason, but sounds like there are solutions to that too

[verbal escarp]

what do you mean?

[main lynx]

wrt reasons to not use python in critical path of cars, robots, etc.

[magic python]

When running mypy it seems that it's possible to get a set of errors (mypy <path> --strict --show-error-codes). If you then add type: ignore[<code>] to these and rerun mypy there can be different errors, which weren't picked up on the first run.

I'm just wondering if this is expected? I'd prefer them to all appear on a single run, but perhaps that isn't possible I'm not sure.

An example of this happening is for this file: https://github.com/raphaelvallat/pingouin/blob/master/pingouin/utils.py

When first run the set of errors don't say anything about the missing annotations on the functions, but when the _initial_set of errors are ignored, the missing function annotation errors are raised by mypy, I don't really understand why it works like this 🤔

[verbal escarp]

ah. cool 🙂 well, iirc there's a lot going on with python in the robot space anyway already for some time now

[main lynx]

yes, that's where i'm encountering the dogma of python being a bad choice, despite it already being widely used, sometimes very poorly

[verbal escarp]

okay, makes sense

[main lynx]

if you're building a robot on linux, at the end of the day it's gonna be making the same system calls as a c++ implementation of the same task

[magic python]

here are examples of the initial errors generated, and then the errors generated having ignored the initial errors: https://zerobin.net/?04ff8bb17e473be0#ciK8p1zT29zWK4Cv7Y+YuSiVq1F5sDqo0+96+qbarGY=

this might just be how it works and the only way it can work, idk

[main lynx]

and the OS is gonna schedule it much the same

[verbal escarp]

well, i can think of a single big issue why you wouldn't want to use python in mission-critical systems

[little yoke]

hey can you do web related stuff in python (things like receiving https requests etc)?

[magic python]

@little yoke #web-development

[little yoke]

ty

[verbal escarp]

@faint quail it's the garbage collection, that might mess with real-time responsiveness

[main lynx]

ok, i've looked at this too, and it's possible to prove that a program has no circular references and turn off the gc

[verbal escarp]

TIL 🙂

[main lynx]

that is one of the touted reasons though

[verbal escarp]

then i couldn't think of any other reason why python couldn't work in that space

if you can make python gc-less without running into issues and prove its correctness.. holy crap

[magic python]

actually i get different mypy results depending on whether i do mypy . and mypy <path to single file>

[stiff crypt]

Idk if it's okay to place it here or in the help channels.

[main lynx]

i think more specific tools for proving the gc never removed anything could go a long way

esp. when compared to c++ which basically uses the same reference-counting approach without any standard gc

[verbal escarp]

you can use pyinstaller to turn your whole python project into an exe, yes

[main lynx]

writing application in c++ is akin to turning off python gc

(using shared_ptr etc.)

[verbal escarp]

@faint quail if you want to tackle that, i'll be rooting for ya ^^

[main lynx]

thanks for the chat 👋

[verbal escarp]

cya

[stiff crypt]

Ohhh I see. But I'm still curious about what professionals use when they release something made in python, like do they also use pyinstaller (which I guess not) or do they have their own way to do it. Or like they develop GUIs and other things in other languages and then just implement the python backend or something

[verbal escarp]

what professionals?

[stiff crypt]

Like python app devs/software engineers/etc

[verbal escarp]

that doesn't make any sense 😉

[stiff crypt]

I mean you write the code in .py right?

And then from a .py, how will you give it to the consumers/users when youre done making/developing it

[verbal escarp]

if you're talking about professionals, you need to think of the area of application, platform you want to deploy to etc

[stiff crypt]

Ohhhh, let's say it's a windows app

[verbal escarp]

like games, medical applications, web apps, data science, lab equipment..

windows app can be anything from ms excel manipulation, scientific applications..

data from the large hadron collider is evaluated in python just the same as F1 chassis design

and you probably wouldn't say those aren't professsionals

[stiff crypt]

Yaya, my terms are misleading but I get your point

[verbal escarp]

you had consumer software in mind, right?

[stiff crypt]

Yes yes

[verbal escarp]

well, you can use pyqt with pyinstaller to make an exe and nobody would guess they're running a python app 🙂

[stiff crypt]

these are in .py too right? are these executed via command prompt? or in an ide like pycharm?

[stiff crypt]

Ohhhhh, the gui is written from Python too?

[verbal escarp]

you can have it either way, yes

[verbal escarp]

pyqt is a hybrid, so yes, but not really ^^

the qt framework isn't written in python, but you as developer only write python

[stiff crypt]

the "qt" part is somewhat related to QT creator?

[verbal escarp]

yes

qt is the framework, qt creator is a tool for that framework

[stiff crypt]

Ohhhh, we've used qt creator as an ide for c++ but havent used it that much outside of writting/running codes

[stiff crypt]

So in short, you can just use straight up .py files when working on a company/something on private projects right? And then just deal with the .exes when you are working on something that you intend to give off as consumer software?

[verbal escarp]

sure, you can wrap up your python code as an exe and deploy it, no problem

[stiff crypt]

"deploy" is the technical term?

[verbal escarp]

sure

[stiff crypt]

sorry I'm an incoming 2nd year comsci student but still a veryvery noobie

thank you!

[verbal escarp]

np ^^

you can ask for more specifics in #tools-and-devops i guess that's the more appropriate channel for that

that channel is dedicated to questions on deployment

[stiff crypt]

I've seen some of our seniors' works, thesis, machine problems, etc. Some of them are made in python, their source codes are included in their papers but I cannot understand them at the moment, I started to wonder how can they make a .py file into something more consumer-friendly, as our homework and projects the last schoolyear were just text-based/console-based and some file manipulation. Thank you for clearing up some things in my mind!

[verbal escarp]

well, most thesis code isn't meant to be user friendly, just sits on github and is run via terminal - and isn't well designed python code either

most of the times you'd probably want something like a web-app or like the bot here

evaluate code and run it against some tests from a web form

really depends on who's the consumer ^^

you might be interested to check out some low/no-code environment, just to get an idea of what's possible

[stiff crypt]

I've also read somewhere that python isnt really optimized for like py-to-exe stuffs

[verbal escarp]

python isn't optimized for anything other than cobbling readable code together, quickly

[magic python]

I'm still wondering about this: #internals-and-peps message

presumably there's a good reason, I don't understand why though

[main lynx]

is there a config file in . it might be reading?

[verbal escarp]

no idea.. but i did have another thought on why "gradual verification" would be a big win to python

pharmaceutical production and other medical applications that are close to patients need to be as error-proof as possible. usually that would mean rigorous development and excessive testing, but verification could change that game

(ironically, most pharmaceutical production machines run windows..)

if you could validate the correctness of critical modules (even if they are function-only), that would open python up for use in another very important industrial sector

[gleaming rover]

I don't think this is possible

without restricting Python's functionality

AFAIK

reflection screws up formal proofs

but I would be happy to be corrected by someone who knows what they're talking about...

[verbal escarp]

add pre-/post conditions in annotations, restrict python to a subset, like functions only and check the AST

i don't see why that couldn't work

[gleaming rover]

yeah, it could

[verbal escarp]

i wished i had more experience in AST manipulation, it seems to be key to a lot of very useful areas

[swift imp]

I've recently used it for a large code refactor at work. Its amazing. Especially if using 3.9 which has ast.unparse to return it back to valid python code. We did a huge refactor of like 20k lines.

[verbal escarp]

cool 🙂

why was the refactor that large and how did AST manipulation help?

[swift imp]

The refactor was huge because when the library was first written the original creator was really allergic to OOP. We refactored it by making classes that encapsulated a lot of the procedural code (which was very patterned and cookie cutter copy and pastes).

Basically we defined "check lists" of code that fit a certain pattern per their AST traversal. When identified we transformed it and used it to create classes through AST. Then we dumped the ast of the classes into valid code and wrote to a file.

Each ast.AST object has a body attribute that is a list holding more ast.AST objects. You can add or remove to that list.

[verbal escarp]

that sounds scary and awesome

how did you figure out how to work with the AST in that way?

are there tutorials for those kinds of manipulations?

[swift imp]

I would dump an object using ast.parse and then snoop around the resultant object's help/dict. After sometime when I felt comfortable I started looking more at the documentation.

[verbal escarp]

hehe

so basically the "dig through undocumented xml-tree" approach

[swift imp]

Yeah. I learn the best through experimentation and then going back and reading the docs

[main ginkgo]

not all of them, you can see which ones do by checking the grammer https://docs.python.org/3/library/ast.html#abstract-grammar

and in some like Expression body is a single node

[fickle osprey]

hey guys just a general question about hardware for ML. So if i have a basic LSTM with attention layers and beam search algorithm which i want to train and evaluate on multiple datasets ranging between size 500mb to 4gb size (before pre-processing) whats the hardware i would need? For example within cloud how much ram and what kind of gpus i would need for quick training (ideally within 4/5 hours)
2) For fine tuning GPT Models (124M layer) on a 2.6gb dataset (before pre-processing) what kind of gpu + ram i would need. Goal is to finetune and evaluate within 10 hrs.
3) GPT NEO Model. For this how much ram and computational power i would need considering dataset size is 8GB. Goal is to train within 24 hrs.
*sorry fine tune not train

[sacred yew]

#data-science-and-ml ?

[fickle osprey]

thanks posted there.

[cyan elbow]

Hi I keep getting this in arch when running startx

Yes I got 3d enabled

[raven ridge]

This server is for the Python programming language. You can ask about that in an off-topic channel, or on a different server.

!ot

[fallen slate]

Off-topic channels

There are three off-topic channels:
• #ot0-psvm’s-eternal-disapproval
• #ot1-perplexing-regexing
• #ot2-never-nester’s-nightmare

Their names change randomly every 24 hours, but you can always find them under the OFF-TOPIC/GENERAL category in the channel list.

Please read our off-topic etiquette before participating in conversations.

[surreal sun]

Why is it that today of all days I’m finding a crap ton of usecases for metaclasses

I used to have zero uses for them

[deft pagoda]

you seen the cool enum metaclass?

[peak spoke]

what usecases did you find? I've used a metaclass once so farm in normal code

[deft pagoda]

!e

from itertools import count

class IncrDict(dict):
    def __init__(self, start=0, step=1):
        self.counter = count(start, step)

    def __missing__(self, key):
        if key.startswith('__'):
            raise KeyError(key)

        self[key] = next(self.counter)
        return self[key]

class EnumMetaclass(type):
    def __prepare__(*args, **kwargs):
        start = kwargs.get('start', 0)
        step = kwargs.get('step', 1)
        return IncrDict(start, step)

class Enum(metaclass=EnumMetaclass):
    def __init_subclass__(cls, **kwargs):
        """Pull kwargs from class def"""

class Fruit(Enum, start=10, step=15):
    APPLE
    BANANA
    KIWI

print(Fruit.APPLE, Fruit.BANANA, Fruit.KIWI)

[fallen slate]

@deft pagoda :white_check_mark: Your eval job has completed with return code 0.

10 25 40

[deft pagoda]

this is my favorite use of metaclasses --- taking over the namespace with __prepare__

you can do neat tricks, like add in a class-specific method decorator in prepare and then remove it in __new__ for some real magic class definitions

[surreal sun]

Well, I thought I had something but I already messed up lmao

i was trying to get the first value defined in an init using a metaclass but that failed

Since I was trying to have the user have the first value defined in an init be the one they want to use for most dunder methods, then implement dunder methods in the metaclass itself, but I could just use inheritance for that too

[surreal sun]

So __prepare__ would override any previous classes with the same namespace?

[deft pagoda]

i don't know what you mean about previous classes

in the above example, the names APPLE, BANANA, KIWI aren't even defined, but the namespace is built out of this weird IncrDict instead of a normal dict, so name errors are caught and assigned some default value

[surreal sun]

Ah

Ohh, now I understand. That's actually a pretty cool usage of metaclasses tbh

[deft pagoda]

you can build weird alternatives to dataclasses with this method:

In [37]: from q import q

In [38]: class Point(q):
    ...:     x, y
    ...: 

In [39]: Point(1, 2)
Out[39]: Point(x=1, y=2)

same thing, x and y aren't even defined, but they're assigned default values anyway -- in this case they're made part of the __init__ and __repr__

[surreal sun]

!e ```py
class Attrs(dict):
def init(self, dct: dict):
self.values_dct = dct

def __missing__(self, key):
    if key not in self.values_dct.keys():
        raise KeyError

    self[key] = self.values_dct.get(key)
    return self[key]

class AttrDefinition(type):
def prepare(*args, **kwargs):
return Attrs(kwargs)

class Dataclass(metaclass=AttrDefinition):
def init_subclass(cls, **kwargs):
pass

class Foo(Dataclass, x=1, y=2):
x
y

print(Foo.x)

tried this, and it works

[fallen slate]

@surreal sun :white_check_mark: Your eval job has completed with return code 0.

1

[paper echo]

TIL about prepare

[surreal sun]

this is amazing lmao

[deft pagoda]

this is the source for q weirdness: https://github.com/salt-die/q/blob/main/q/q.py

[main ginkgo]

this sounds interesting, what would be a concrete example?

[deft pagoda]

i experimented with a class-specific decorator here: https://github.com/salt-die/nurses/blob/master/nurses/widgets/widget.py#L51-L62
But I don't do this anymore my current tui library

[fallen slate]

nurses/widgets/widget.py lines 51 to 62

class Observer(type):
    """
    This metaclass simply drops the `​bind_to`​ decorator into the class dict.
    `​bind_to`​ allows one to quickly bind functions to attributes in the class body - these attributes
    will be turned into Observables by the decorator.
    """
    def __prepare__(name, bases):
        return { "bind_to": BindMagic }

    def __new__(meta, name, bases, methods):
        del methods["bind_to"]
        return super().__new__(meta, name, bases, methods)```

[deft pagoda]

which looks like this in a class definition: https://github.com/salt-die/nurses/blob/master/nurses/widgets/widget.py#L152-L158

[fallen slate]

nurses/widgets/widget.py lines 152 to 158

@bind_to("top")
def _set_pos_hint_y(self):
    self.pos_hint = None, self.pos_hint[1]

@bind_to("left")
def _set_pos_hint_x(self):
    self.pos_hint = self.pos_hint[0], None```

[deft pagoda]

so now, class instance attributes top and left will be converted to properties that call these methods when __set__

it's super magical and was mostly an experiment, but maybe for some other framework it could be useful as a time saver

[verbal escarp]

i'd prefer attrs with annotations, if i need a class that only holds x and z i take namedtuple, which is magical enough

[deft pagoda]

attr.s i.s a bi.t un.readabl.e

but i don't think namedtuples are magical, i think that should be the preference whenever possible

[verbal escarp]

namedtuples are extremely magical if you have a closer look

[deft pagoda]

i've looked at the source

[peak spoke]

yeah I'd definitely consider a dynamically created class magical

[deft pagoda]

i mean, it's just a tuple with properties that are itemgetters

[peak spoke]

the class it creates is simple, but I wouldn't say the process used to get there is

[flat gazelle]

The concept of eval for dynamically created types is as old as eval, I would hesitate to call it magic. It's more just an application of dynamic typing

[nova iris]

ah, see, you did the same thing as i did - creating a superclass solely for overwriting default object.__init_subclass__ behaviour so kwargs can be inserted
i was always searching for a solution to this, it seemed as if this quick and dirty fix was too un-elegant

but i guess it iz what it iz

[deft pagoda]

you can do it with an __init__ instead in the metaclass, but i also wanted a normal class wrapper

[nova iris]

>>> wait()
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    wait()
TypeError: cannot create 'generator' instances
```aww

[white nexus]

I've been thinking about it, and in comparison to other languages, I think the fact that python requires a pass statement since it doesn't have brackets helps make code more readable.

Requiring pass means that its impossible to write a class without intentionally not putting anything in it.

[grave jolt]

I don't really know where pass would be useful

In an empty class you can replace pass with a docstring

or a no-op expression like ...

[verbal escarp]

... is not nothing

[white nexus]

(What does ... do?)

[verbal escarp]

it's an object

[spice pecan]

It's Ellipsis, one of the most notable uses is skipping array dimensions in numpy

[grave jolt]

... is understood as "missing implementation" in protocols/abstract classes by type checkers

[spice pecan]

IMO pass is a fairly explicit way of saying "This is, and should remain, empty"

[verbal escarp]

i think it's the only no-op code that isn't an object

[spice pecan]

I'm pretty sure it's the only legitimate no-op, yeah

[white nexus]

An exemption of that if there is a comment above it that starts with TODO... or similar.

[unkempt rock]

Anyone good at packages and imports #help-apple

[spice pecan]

I'd use ... in that case

As suggested by @fix error#1642 earlier

[white nexus]

I have a bunch of raise NotImplementedError while I work on a file, what is the difference?

[spice pecan]

... is missing implementation, pass is intentionally empty

[grave jolt]

I think pass is just a syntactic limitation of Python, I haven't seen it in other languages

[spice pecan]

raise NotImplementedError would be ideal

[south raft]

... returns class ellipsis

[verbal escarp]

NotImplemented is an alternative

[white nexus]

yeah, that was implied/stated in my original statement 😄

[verbal escarp]

but it's also an object, which may or may not have performance implications

[spice pecan]

NotImplemented has a very specific meaning

[grave jolt]

If you're worried that ... is more expensive than pass, it's not

[spice pecan]

It's what you return from a dunder if you don't know how to interact with a certain type and want to let it decide

[grave jolt]

!e

import dis

def foo():
    pass

def bar():
    ...

dis.dis(foo)
dis.dis(bar)

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

001 |   4           0 LOAD_CONST               0 (None)
002 |               2 RETURN_VALUE
003 |   7           0 LOAD_CONST               0 (None)
004 |               2 RETURN_VALUE

[white nexus]

What about methods/functions that don't have meaning yet, but exist as a sort of todo?

[spice pecan]

Ideally, raise NotImplementedError()

[white nexus]

ah, so what I already do 😛

[spice pecan]

With an appropriate docstring whenever possible, or a TODO comment

[grave jolt]

yeah, if you put ... or pass while you're working on it, you might get a sneaky None somewhere

[white nexus]

right

[spice pecan]

... is typically used in stubs, pass generally seems to be used as an intentional no-op that's supposed to stay that way

[deft pagoda]

i've written a few while loops where the important bit was the condition and so the body of the while was pass

[white nexus]

I have one file that has pass since it works how it is, but still needs additional features (think subclass, that doesn't have anything to improve on yet)

[white nexus]

so just waiting for the condition to become true?

[deft pagoda]

yep, usually this involves conditions that have side effects

[paper echo]

:= ftw

[deft pagoda]

yep, but also other things like: https://github.com/salt-die/nurses_2/blob/main/examples/tetris/tetris/tetris.py#L462-L467

[fallen slate]

examples/tetris/tetris/tetris.py lines 462 to 467

def drop_current_piece(self):
    """
    Drop piece.
    """
    while self.move_current_piece(dy=1):
        pass```

[deft pagoda]

i also have elif chains with pass!

[paper echo]

Oh that's interesting

[deft pagoda]

https://github.com/salt-die/nurses_2/blob/main/examples/tetris/tetris/tetris.py#L351-L357

[fallen slate]

examples/tetris/tetris/tetris.py lines 351 to 357

if self._lock_down_task.done():
    pass
elif self.move_reset < MOVE_RESET:
    self._lock_down_task.cancel()
    self.move_reset += 1
else:
    return False```

[deft pagoda]

this is so you can handle most cases with an else

[main ginkgo]

similar idea with for too, sometimes you might only care about the last item of an iterable so could get it like this:

for item in some_iterable:
  pass

# item is now set to the last item in the iterable

[deft pagoda]

i've use this before too

[main ginkgo]

really clean way to find an item in an iterable that fulfills a condition too e.g.

for item in some_iterable:
  if meets_some_condition(item):
    break

[paper echo]

I would prefer filter and next for that

[astral gazelle]

That looks like what any does

[deft pagoda]

well, if you only need one item, then filter is more work

the for loops that search for something often have an else block for me

for item in some_iterable:
  if meets_some_condition(item):
    break
else:
  item = None

[peak spoke]

I'd really dislike if I stumbled upon a while loop like that, and would have to look into the condition to see if it's not busy waiting for no reason

[main ginkgo]

any will always return a bool rather than the item itself

[peak spoke]

for the other things just use itertools/more-itertools

[deft pagoda]

what do you mean, waiting for no reason

[paper echo]

reduce(lambda x,y: x or y, ...)

[paper echo]

Same, i prefer next((x for x in stuff if foo(x, y)), None) but ideally wrapped in a function. Clearer intent (to me) than a for loop

[deft pagoda]

but it's a while loop

[peak spoke]

a tight loop like that will eat up the thread so waiting for a condition like that can be costly without knowing what exactly is going on, and at least from my view the condition should be doing simple work

[deft pagoda]

i don't have any threads

i'm confused

[peak spoke]

if you're running you have a thread

[paper echo]

Oh sorry i meant to respond to the other one

[deft pagoda]

are we being really pedantic, or is there some point i'm missing

[paper echo]

The idea is that the function in the while does nontrivial work

[peak spoke]

and a tight loop will consume all of the cpu time made available to it, which I'd suspect the while ...: pass to be

[paper echo]

Like it might be literally an event loop

It's just tidier than while True/break

[deft pagoda]

^

[peak spoke]

I think I'd prefer the break

[paper echo]

Yeah, i would probably write it that way if i wrote it for work

"At home" i like salt's idea

[deft pagoda]

while True:
    if not self.move_current_piece(dy=1):
        break

seems worse to me

[paper echo]

Or make the name very very clear

[deft pagoda]

but w/e

[paper echo]

Instead of move_current_piece call it game_loop or something

And the latter function would call the former

[deft pagoda]

it's not a game_loop though, it literally tries to move the piece downwards, but returns False if the movement fails

[paper echo]

Fair

As long as you put a comment in there I think it's fine

while self.move_current_piece(dy=1):
    # Loop ends if no moves are left
    pass
...

[deft pagoda]

dy

i mean the entire function is called drop_piece i hope that's clear what it's doing

HAS NO ONE PLAYED TETRIS

[nova iris]

no, __prepare__ initializes the class namespace with optional initial values, and it could be populated later on
you can check out my class creation flowchart to see :P https://github.com/Objectivitix/ccc-flowchart it has some information on that

[nova iris]

OH WAIT I JUST REALIZED - since the class body execution happens before type.__new__ eventually yeets the original namespace made by __prepare__ and keeps a mappingproxy copy, you can do these cool tricks! i was confused for a sec lol

[deft pagoda]

yeah, it all gets converted to a dict in the end

or mappingproxy

[nova iris]

but damn, that hack is nice!

love to see some good uses of metaclasses and stuff lol

[nova iris]

does anyone know why generators have a function call overhead, but stuff like lists don't?```py

import cProfile
cProfile.run('sum([i * 2 for i in range(10000)])')
5 function calls in 0.001 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 0.001 0.001 <string>:1(<listcomp>)
1 0.000 0.000 0.001 0.001 <string>:1(<module>)
1 0.000 0.000 0.001 0.001 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {built-in method builtins.sum}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}

cProfile.run('sum((i * 2 for i in range(10000)))')
10005 function calls in 0.003 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
10001 0.002 0.000 0.002 0.000 <string>:1(<genexpr>)
1 0.000 0.000 0.003 0.003 <string>:1(<module>)
1 0.000 0.000 0.003 0.003 {built-in method builtins.exec}
1 0.001 0.001 0.003 0.003 {built-in method builtins.sum}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
```as i understand it, the list would be transformed into an iterator and next() will be called on it 10000 times as well, so i'm a bit confused

here's stack overflow's answer, but this only explains generators' inefficiency

why don't list comps have this overhead as well? they're implicitly transformed into list_iterators, no?

[proud anvil]

The number of calls are different though 1(LC) compared to 5000001 in generator expression, this is most because sum is consuming the iterator hence has to call its next method 500000 + 1 times(last 1 is probably for StopIteration to end the iteration). For a list comprehension all the magic happens inside its code object where the LIST_APPEND helps it in appending items one by one to the list, i.e no visible calls for cProfile. @nova iris

https://stackoverflow.com/questions/30112326/generators-vs-list-comprehension-performance-in-python

[paper echo]

Ironic considering your username!

Python has a lot of "fast paths" for special cases

These usually bypass the object and method lookup system for exactly this purpose, eliminating or reducing the overhead of a function call

https://youtu.be/qCGofLIzX6g fun talk that explains some of these cases

https://youtu.be/s5_5XmmwMx8 I haven't watched this one but the title looks fun

[nova iris]

oh... okay, thank you!

[white nexus]

is this a python bug?

in 3.9 and below Union[T, None] turns into Optional[T]
in 3.10 T | None does not get turned into Optional[T]

[boreal umbra]

Where is this from

[grave jolt]

!e

from typing import Union, Optional
print(Union[int, None])
print(Optional[int])

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

001 | typing.Optional[int]
002 | typing.Optional[int]

[grave jolt]

]e

from typing import Union, Optional
print(Union[int, None])
print(Optional[int])
print(int | None)

[fiery pilot]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

001 | typing.Optional[int]
002 | typing.Optional[int]
003 | int | None

[grave jolt]

hmmm

[grave jolt]

[peak spoke]

should it be changed to Optional? The union types are their own type separate from typing's Unions, even if they are practically the same thing

[boreal umbra]

@peak spoke why is that?

[grave jolt]

But Union does result in an Optional

[peak spoke]

Yes, but a | None is not an Union, it's an instance of UnionType; I don't think it'd make sense to convert that to a generic alias or whatever the internal class typing uses is. With Optional it's "converted" because Optional is actually an Union with None and it's just specialcased in the string representation

[grave jolt]

ah, it's specialized in the representation

that's so confusing

[white nexus]

this came across when using discord.py, and wondering what if I used from __future__ import annotations, then used the new | operator.
So a command definition goes from the first to the second below:
async def edit(self, ctx: Context, message: Union[discord.Message, None] = None, *, content: str) -> None:
async def edit(self, ctx: Context, message: discord.Message | None = None, *, content: str) -> None:

The problem is that it results in this error:
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

[grave jolt]

Are you on Python 3.10?

[white nexus]

no, was using 3.8, with from __future__ import annotations

The code is valid, but that is an error with discord.py internally caused by the Union

[peak spoke]

The feature doesn't exist before 3.10, postponing the annotation evaluation isn't going to change that

[white nexus]

what?

but, it... does exist.

[white nexus]

yeah, it exists

or, partially

wat

[gleaming rover]

isn’t that an issue with | on types

that isn’t in 3.8

[white nexus]

uh

[grave jolt]

@white nexus from __future__ import annotations just defers the evaluation of annotations, it doesn't add | to types

[white nexus]

then why--

!pypi pyupgrade

[fallen slate]

pyupgrade v2.24.0

A tool to automatically upgrade syntax for newer versions.

[white nexus]

guess its technically a bug with this ¯_(ツ)_/¯

[grave jolt]

@white nexus ```py
''.join([str(i) for i in range(3)]) # ''.join(str(i) for i in range(3))

Why change for a slower variant 😔

[charred pilot]

half of these changes aren't even necessary though, this'll just mess with git blame

[white nexus]

why is it slower?

[grave jolt]

Internally, CPython does this (pseudocode):

if is_list(arg) or is_tuple(arg):
    items = arg.internal_array
else:
    items = collect_items_from_iterable_into_array(arg)

[naive saddle]

I'd assume that the generator protocol has some overhead + list comprehensions are implemented directly via opcodes internally skipping the iteration protocol

[grave jolt]

the generator will be immediately exhausted, so it's just extra overhead with no benefit

[naive saddle]

!e

import dis
def f(): return [str(i) for i in range(5)]
print(dis.dis(f))

[fallen slate]

@naive saddle :white_check_mark: Your eval job has completed with return code 0.

001 |   2           0 LOAD_CONST               1 (<code object <listcomp> at 0x7f73b91337c0, file "<string>", line 2>)
002 |               2 LOAD_CONST               2 ('f.<locals>.<listcomp>')
003 |               4 MAKE_FUNCTION            0
004 |               6 LOAD_GLOBAL              0 (range)
005 |               8 LOAD_CONST               3 (5)
006 |              10 CALL_FUNCTION            1
007 |              12 GET_ITER
008 |              14 CALL_FUNCTION            1
009 |              16 RETURN_VALUE
010 | 
011 | Disassembly of <code object <listcomp> at 0x7f73b91337c0, file "<string>", line 2>:
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/azulexetih.txt?noredirect

[sturdy timber]

In cases where I wouldn't care about speed (most cases) I'd consider it similar to something like print(("hello world")), less brackets just makes it look cleaner imo.

[grave jolt]

I guess

Actually, CPython can perform an optimization if join is used on a literal

but it doesn't

[verbal escarp]

I'd say that Optional should be deprecated in favor of Union

because None is not the one-and-only default

we also got the PEP suggesting user-defined sentinels

then we'd get stuff like int | sentinel("nothing") or something like that

https://www.python.org/dev/peps/pep-0661/

[lament sinew]

It's by far the most common one, easily enough to justify having Option as a short hand for Union

[verbal escarp]

what happened to the suggestion to add ? as syntax shorthand for Optional?

[lament sinew]

never heard of it, but I hate it already.

[verbal escarp]

iirc it came from GvR

[lament sinew]

wouldn't be the first hateful thing about python to come from him.

[verbal escarp]

if Union with None is so special to warrant an Optional, you can make it an extra syntax as well

not really much of a difference

[lament sinew]

do you feel like adding an extra type to a package and adding new syntax to the whole language are ways to solve a problem that differ widely in scale?

might as well ask for special syntax to make http requests

or regexes

[verbal escarp]

typing isn't just any package

[deft pagoda]

this annoys me, i created my own sentinel function a few months ago

i didn't know there was a reference implementation

[verbal escarp]

i implemented my own sentinel as an Enum.. welcome in the club 😉

[deft pagoda]

mine's a function that builds out a functionality based on abcs or passed in namespace

[verbal escarp]

sounds convoluted :p

[deft pagoda]

well, a general sentinel building function has a lot to do

https://github.com/salt-die/sacks/blob/main/sacks/primitives/sentinel.py#L4-L11

[fallen slate]

sacks/primitives/sentinel.py lines 4 to 11

def sentinel(name='', default=None, repr='SENTINEL', abc=None, methods=None, attrs=None):
    """
    Build a better sentinel!
        * Modifying attributes of this object does nothing (with no errors).
        * Abstract methods in `​abc`​ (an abstract base class) not provided in `​methods`​ will return `​default`​.
        * If a value in `​methods`​ is `​"identity"`​ a property that returns the sentinel will be provided.
        * If a value in `​methods`​ is `​"default_iter"`​ a default iterator will be provided.
        * If a value in `​methods`​ is a subclass of `​Exception`​ a default function that raises will be provided.```

[deft pagoda]

alternatively a class decorator @sentinel would make sense --- and would reduce the complexity of this function signature

[magic python]

I feel as though it could be made more clear that a Sequence in mypy should be something that contains order (which is why a set isn't a sequence)

[prime estuary]

I'm not sure if that's MyPy's job exactly, it's a generic name used in the docs to refer to things with a similar API to lists: https://docs.python.org/3/glossary.html#term-sequence
The Sequence ABC does more directly imply that indexing is require by implementing index(), count() etc.

[prime estuary]

If you want to wrap your head around the ABC taxonomy, there's a nice table at the top of collections.abc with the required methods for each and the inheritance:
https://docs.python.org/3/library/collections.abc.html
For your example, note the common class for both sets , lists and dicts is Collection.

[magic python]

@prime estuary oh, good points I wasn't aware of this 🙂

[solar haven]

Hello! Quick question: Where do you put annotations in init? In the method signature like def __init__(self, x: int) or in the body like def __init__(self, x): self.x: int = x?

[grave jolt]

Those have different meaning. If you don't put an annotation in __init__, the type checker will think that it accepts Any as that parameter.

[solar haven]

Oh, __init__.__annotations__ is empty. Can I ask how should a class be annotated? Is the first one enough?

[elder blade]

I looked at it while making mine 😉.

That said, it's literally just an object subclass with a repr and all Final typings out there

I did have to type it as Any though, otherwise all types would be x: Union[MissingType, str] = MISSING

[nova iris]

This is very interesting, I've never used much sentinel values and usually people just go with an object object, but this could be quite useful

[elder blade]

It is indeed very useful
https://c.tenor.com/FrZln65w7JAAAAAM/snob-rich.gif

[candid pelican]

hello friends i need some help with this class...

class RequireToken:
    def __init__(self, function):
        self.function = function

    def __call__(self, *args, **kwargs):
        if request.args.get('token') == SECRET_KEY:
            return self.function(self, *args, **kwargs)
        return {'message': None}
        

how can i pass as parameter to this decorator?

[paper echo]

@candid pelican pass the parameters in __init__ and rewrite __call__ to be a decorator.

i assume this is for flask (because of the request global)

from functools import wraps

import flask

class RequireToken:
    def __init__(self, secret_key):
        self.secret_key = secret_key

    def __call__(self, function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            if flask.request.args.get('token') == SECRET_KEY:
                return function(self, *args, **kwargs)
            return {'message': None}
        return wrapper


app = flask.Flask(__name__)

@app.route("/users", list-u)
@RequireToken(SECRET_KEY)
def list_users():
    ...

(you might want to return 401 if the secret key is missing or invalid)

[candid pelican]

ohh thanks hehe

i searched... but i did not find smth like that

[paper echo]

yeah, this is hard to search for and there probably isn't a single authoritative guide or howto doc for it

i can't remember where i first learned it

there might be a good stackoverflow question on class-based decorators

[candid pelican]

i see

can you explain for which reason you are using that decorator wraps i deleted it and nothing changed

@wraps(function)
def wrapper(*args, **kwargs):
    if flask.request.args.get('token') == SECRET_KEY:
        return function(self, *args, **kwargs)
    return {'message': None}
return wrapper

[paper echo]

https://docs.python.org/3/library/functools.html#functools.wraps

[candid pelican]

thanks hehe

a quicky question can i make multiples decorators in a single class???

do you have some article abt it or smth??

[paper echo]

no, a class instance can only have one __call__ method. but you can create other methods/properties that return other decorators!

[candid pelican]

oh i c

[paper echo]

!eval ```python
class Fancy:
def init(self, val1, val2):
self.val1 = val1
self.val2 = val2

def deco1(self, func):
    def wrapper(*args, **kwargs):
        print('Value:', self.val1)
        result = func(*args, **kwargs)
        print('Result:', result)
        return result
    return wrapper

def deco2(self, func):
    def wrapper(*args, **kwargs):
        print('Value:', self.val2)
        result = func(*args, **kwargs)
        print('Result:', result)
        return result
    return wrapper

@Fancy(5, 10).deco1
def f(x, y):
return x + y

@Fancy(5, 10).deco2
def g(x, y):
return x - y

f(5, 10)
g(5, 10)

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

001 | Value: 5
002 | Result: 15
003 | Value: 10
004 | Result: -5

[candid pelican]

ohhh great

i got it

could i use the staticmethod to do it?

[paper echo]

i don't see much value in staticmethods in general

[candid pelican]

im doing a class ....

when i end it i send to you

[candid pelican]

look

class Security:
    def __init__(self, secret_key):
        Securitykey = secret_key

    @staticmethod
    def require_token(function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            if request.args.get('token') == Security.key:
                return function(*args, **kwargs)
            return {'message': None}
        return wrapper

now i can pass my key just a single time and use the decorator like:

security = Security(SECRET_KEY)

@security.require_token
def foobar():
  pass

[undone hare]

That's a weird way of doing it, why using a class attribute (which btw is never set)?

class Security:
    def __init__(self, secret_key):
        self.key = secret_key

    def __call__(self, function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            if request.args.get('token') == Security.key:
                return function(*args, **kwargs)
            return {'message': 'Invalid authentication'}
        return wrapper
```That's how I'd do it

[static hinge]

The security key should probably be hashed too

With salt

[undone hare]

Well, it depends

[candid pelican]

the class is not completely done

i will set new methods you know

[static hinge]

You should also return 403

[candid pelican]

yeh

i will do it

i don't wanna return a invalid message i wanna return wrong data

......

[paper echo]

you can still return the wrong data, but with a 401 or 403

[candid pelican]

okay

im a beginner lol

like why i should not return 200??

[undone hare]

Http status codes indicates how the request went

200 means "OK", everything went fine

[candid pelican]

but theoretically it was successful

[undone hare]

401 is "Unauthorized" for example, or 404 "Not found"

[undone hare]

It isn't, you didn't get the expected data because your authorisation was wrong

[candid pelican]

ic

[paper echo]

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

[undone hare]

The Wikipedia list is quite nice, I would recommend you to read through them at some point https://en.m.wikipedia.org/wiki/List_of_HTTP_status_codes

[paper echo]

this is also #web-development material now

[undone hare]

I guess MDN also works but is more technical

[candid pelican]

class Security:
    def __init__(self, secret_key):
        self.key = secret_key

    def __call__(self, function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            if request.args.get('token') == self.key:
                return function(*args, **kwargs)
            return {'message': 'Invalid.'}, 401
        return wrapper

like that???

[paper echo]

self.key, not Security.key

Security.key would be an attribute on the Security class itself, self.key is an attribute on the self object, i.e. the current instance of the Security class

[candid pelican]

i forgot it lol now is right???

[undone hare]

looks about right

wow, learn to check the timestamp akarys

[verbal escarp]

it's all relative 😉

[unkempt rock]

Is there a way to use contextlib.nullcontext in 3.6? Importing future or something?

https://docs.python.org/3/library/contextlib.html#contextlib.nullcontext

[peak spoke]

define your own

there's probably a backport somewhere but for something as simple as that I wouldn't bother

[unkempt rock]

I guess it would be as simple as ```py
class mynullcontext:
def init(self, file):
self.file = file

def __enter__(self):
    return self.file

def __exit__(self, *args):
    pass

[flat gazelle]

Why not just do

@contextmanager
def nullctx(a):
    yield a

[unkempt rock]

Or I could copy the original https://github.com/python/cpython/blob/7f1c330da31c54e028dceaf3610877914c2a4497/Lib/contextlib.py#L732

[fallen slate]

Lib/contextlib.py line 732

class nullcontext(AbstractContextManager, AbstractAsyncContextManager):```

[unkempt rock]

didn't know about that decorator

[paper echo]

Need this for sys.stdin

Too lazy to set up Click...

[elder blade]

Create a Click app that creates a Click app

[unkempt rock]

Mypy complains about no enter or exit unless I cast this

[boreal umbra]

did you tell mypy to FUCK OFF?

[unkempt rock]

I used cast, is there a better way

[boreal umbra]

I think you can actually instruct mypy to ignore certain things via comments, but I don't use mypy explicitly. Just whatever pycharm and our linters tell me to do.

[paper echo]

Honestly yes

[paper echo]

Might as well post the play link

https://mypy-play.net

[south monolith]

I'm trying to import stuff from flask , for example render_template

but if I do from flask import render_template the compiler can't find it

the only thing I can do is from flask.templating import render_template

This applies to most other packages, like Blueprint

Does anybody know how to fix this?

[paper echo]

I've had that issue with some packages at various times (eg scipy), and I never really knew what caused it

[south monolith]

so did you have to import all packages by including the module name?

@paper echo

[quiet python]

yes, because the functions / classes are located within the submodules for your specific case so you'll always have to do it, which is true for other libraries as well such as tensorflow, sklearn, and scipy

[south monolith]

in the video I'm following, the youtuber didn't have to change his import statements like I do

[raven ridge]

that sounds like there's an incorrect flask/__init__.py

[south monolith]

In the flask __init__.py file, it has from werkzeug.utils import redirect as redirect

[raven ridge]

what do you see if you do:

import flask
print(flask)

[south monolith]

I'm not able to do from flask import redirect, I have to type from werkzeug.utils import redirect

that's unbelievably redundant

ok I'll try that

>>> print(flask) <module 'flask' from 'C:\\Users\\Sean\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\flask\\__init__.py'>

[raven ridge]

ok, that seems reasonable - is that __init__.py the one you were just reading from?

what error do you get if you do from flask import render_template, exactly?

[south monolith]

PyCharm has a message that says Unresolved reference 'render_template'

I haven't been able to run my project for a while today. All of my code is unorganized, because I'm following a tutorial and I'm at the stage where I implement blueprints

[raven ridge]

well, forget what pycharm says - if you run the code, does it work?

[south monolith]

Hmm, let me try that

This will take a few minutes, since there are a lot of imports

[surreal sun]

So I was working with asyncio, and I used await asyncio.gather, though even though all the coroutines finished their handling, the event loop was just idling and taking its time even after all the coroutines were done running, so it adds a second or two to the execution of my code, anyone know why?

And it's only when I change the parameters to add a new function to the asyncio.gather

[south monolith]

sorry @surreal sun I'm not done yet

[surreal sun]

My bad

[quiet python]

they might be running a different version of flask

check the video release date

[grave jolt]

what happened on python-dev?

[south monolith]

@raven ridge after a long time of tedious editing

I finally ran my code

And it works...

PyCharm has been lying straight to my face

I can't believe this

[raven ridge]

That makes more sense than the imports not working, frankly 😄

That means it's a question for #editors-ides though

I've never used pycharm and can't help much about what it's getting wrong.

[south monolith]

This is the most annoying issue I've ever experienced

In my short career of coding

[raven ridge]

Maybe it's using the wrong site-packages directory or something

[south monolith]

That... is an issue I will deal with later

So frustrating lmfao

That's enough for today

Thank you SO MUCH @raven ridge I never would've considered that PyCharm was lying to me... I thought I could trust him

[surreal sun]

#internals-and-peps message

^ For this, it said initially that it was asyncio._overlapped.GetQueuedCompletionStatus but now it says something else entirely

You're good!

[grave jolt]

I read through it, it was rather entertaining and absurd.

[paper echo]

in my case i always assumed it was related to c extensions

geez is this sulla guy ok?

[verbal escarp]

link?

[grave jolt]

https://mail.python.org/archives/list/[e-mail]/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/

[verbal escarp]

heh.. i haven't scrolled down far enough to see the popcorn-worthy parts of that link at first

wow

a single mod who doesn't exercise their power to keep people from jumping at each other's throats..

[jovial flame]

Don't piss of Terry I think, I'm not too far down the link yet

[paper echo]

It does look like that user was eventually banned

[verbal escarp]

really? i thought the mod said they didn't want to do anything.. oh well 🙂

@paper echo hey btw 🙂

just to let you know, we're currently in the process of cleaning up the justuse code, refactoring and removing everything that doesn't contribute to functionality

i hope in the end we'll have a pip-auto-install that "just works" as was the idea

(although at first pip wasn't part of the vision, but oh well)

after that we can hopefully release a new version that actually keeps its promise regarding auto-installation, and then we can finally tackle P2P distribution of packages 🙂

[paper echo]

cool!

you're moving right along

[nova iris]

today i finally understood the meaning of "flat is better than nested"

so, in my quest to completely understand generators and explore some advanced features, i stumbled upon the "delegating to a subgenerator" feature

and this is the semantically equivalent code that was documented in the pep: https://paste.pythondiscord.com/zeyofudeyo.py

so RESULT = yield from EXPR is equivalent to all of that convoluted crap

understanding absolutely nothing, i yeeted it into an IDE and finally made the code tons more readable, with comments everywhere https://paste.pythondiscord.com/agicizoduv.py

but still, HOLY CRAP, this INSANELY nested logic with a COPIOUS amount of try/except clauses really made me think lmao

it's like trying to make your brain wrap around a tesseract

[verbal escarp]

and thus, "syntactic sugar" was invented

[nova iris]

lmao 🤣

yeah, that one line of code is syntactic sugar for an absolute monstrosity

everyone prefers r = yield from iterable than this https://paste.pythondiscord.com/agicizoduv.py

[verbal escarp]

just what i was meditating over during a shower today

[nova iris]

you were meditating subgenerators during a shower? respect

[verbal escarp]

"oh no, more syntax to learn! damn, lua is so much better because it's so little syntax to learn! oh crap, now i need to invent constructs for everything! python's approach is actually not that bad - there's greatness in mediocrity!"

that's basically the gist of my meditation

[nova iris]

lol

[verbal escarp]

less exclamation marks, more water

well, i was pondering how an influx of people from diverse backgrounds will change python, what it'll mean in terms of programming language paradigms, how it's a good thing that python never claimed to be the fastest, smallest, whatever language..

i think there's a lot of potential for python, even growing the more people are getting involved early on, which will inevitably lead to more complex syntax, but i think the way python is going, it won't necessarily lead to a steeper learning curve

which is quite exceptional

[flat gazelle]

practicality over purity is a great line

yes, .get() is unpythonic and you should use exceptions, no, exceptions aren't always the correct solution.

[paper echo]

.get() is unpythonic and you should use exceptions
who says

imo, use .get if you expect the thing to not be there sometimes, use try if the thing really is supposed to be there but someone might have f'ed it up

[flat gazelle]

unpythonic may be a strong word

but python does use exceptions for a lot more things than the name would suggest

StopIteration is not what I would call an exception in any sense of the word

[paper echo]

heh, StopIteration

imagine if python had lisp-style conditions!

import os, sys

def main():
    use_debug = bool(os.environ.get('DEBUG', '').strip())
    try:
        api_key = os.environ['API_KEY']
    except KeyError;
        print('API_KEY is not set!', file=sys.stderr)
        return 1

if __name__ == '__main__':
    sys.exit(main())

i would say that both of these are pythonic in their place

[flat gazelle]

yeah, exceptions just aren't readable enough to be useful for simple things like alternatives

but well,

try:
    debug = bool(os.environ['DEBUG'].strip())
except KeyError:
    debug = False
```is also an option, it's just kind of worse

you could definitely find python purists arguing for it

[paper echo]

yep that's definitely worse imo

side note thing that i'm still upset about: when StopIteration is raised from calling .send() on a generator, it doesn't contain the generator's return value the way it would if you call next() on it

[flat gazelle]

oh wow, that seems bug report worthy

[paper echo]

wait, i might be wrong

i am pretty sure i had this happen somewhere

!e ```python
def gen():
yield 5
yield 10
return 'hi'

g1 = gen()
next(g1)
next(g1)
try:
next(g1)
except StopIteration as exc:
print(exc)

g2 = gen()
g2.send(None)
g2.send(None)
try:
g2.send(None)
except StopIteration as exc:
print(exc)

[fallen slate]

@paper echo :white_check_mark: Your eval job has completed with return code 0.

001 | hi
002 | hi

[paper echo]

that works fine, maybe the issue was elsewhere

however it would be nice if you could tell next to return the StopIteration value

[nova iris]

omg i keep on finding the most obscure objects imaginable lmao

interesting: the iterator of a dictionary has the same type as the iterator of a dictionary key view

i mean, i guess they should both have the same behaviour so it's not surprising

interesting anyways

[obsidian tapir]

yes that's why

for key in dictionary: ...

is same as

for key in dictionary.keys(): ...

[elder blade]

Yeah, it was the new mod who did it 😅

[nova iris]

am i the only one who thinks python is a little inconsistent with how it handles unreachable code? for example, in py3.10, a pattern like this is invalid:py match 1: case x: pass case y: pass and raises SyntaxError, because case y is unreachable. BUT, let's now look at functions:py def foo(): return 1 return 2 and```py
def foo2():
raise Exception
raise ZeroDivisionError

but it might be implementation detail, though, since errors like these in functions perhaps can't be detected during compile time?

[flat gazelle]

I don't think a syntax error can be an impl detail, but ye, unreachable code in pretty much all other non-trivial cases in non-deterministic, whereas it can be easy to make a mistake in patterns

it is somewhat inconsistent though, that I won't contradict

[nova iris]

i agree, unreachable patterns is probably way easier and more efficient to detect than the other cases

[stone field]

There's actually a use case for unreachable code in Python, believe it or not. The simplest way to implement a generator that immediately raises StopIteration (or the async equivalent of it, AsyncStopIteration) is the following:

def generator():
    return
    yield

[verbal escarp]

wtf ^^

[nova iris]

hm, that is interesting - so the interpreter would parse it as a generator, and it would have the raising mechanism as well, super cool!

i never thought of that

[pliant tusk]

isnt py generator = lambda:(yield from ()) or py def generator(): yield from () simpler?

[stone field]

yield from cannot be put inside an async generator

[pliant tusk]

ah didnt know that

[spark magnet]

also, yield from is newer than yield, so there was a time when that wasn't available for making an empty generator.

[terse hill]

When it comes to standard libraries for python, how come the most optimal data structure isnt used like binomial or fibonannci heap for heapq module:, both can insert in O(1) time (amortized and not) as opposed to O(log n) for a binary heap

i get that the delete-min is amortized for fibonacci

but it seems even bionmial heap would be better then binary

[paper echo]

there used to be a special case in the parser so that any code inside if False: wouldn't even be parsed. i remember when they took it out there was some debate as to whether it should be removed or not

[nova iris]

oh wow. why did they take it out?

[paper echo]

consistency i guess

[nova iris]

ah, i see

[elder blade]

It was used for static type checking for example

Which has now been superseded by if TYPE_CHECKING

[dusty verge]

Wait what, is python getting switch statements

[elder blade]

Pattern matching*

[nova iris]

match statements, and yes, they're coming in 3.10

some people dislike the double indentation

but i actually like it lol

python: hey guys, you can now parenthesize context managers! check this out:```py
with (
CtxManager1() as example1,
CtxManager2() as example2,
CtxManager3() as example3,
):
...

also python: ```py
match [1, 2, (3, 4, "hi")]:
    case (
        [int(x), y, (3, _, z)]
        if 1 <= y <= 6
    ):
        
SyntaxError: invalid syntax

[paper echo]

with () is a blessing

match is... not

[candid pelican]

gosh i never thought about it.

[dusty verge]

Yeah I agree, the case ( ) stuff feels kinda obfuscating*

[nova iris]

lol

[nova iris]

how is it obfuscating? if you have a case line that is too long, wouldn't you want to separate the pattern and the guard onto different lines?

[stone field]

Because the fibonacci heap is only sound on paper. When you start considering real programs, other forms of heaps beat them in terms of performance due to many factors. The major one is that other implementations can be expressed in contiguous memory, i.e. arrays, while the fibonacci heap has to go around chasing pointers. The memory use is also larger and sparse. A heap implemented in terms of an array could fit snugly inside the L1 cache of a CPU, and referencing it is hundreds of times faster than seeking to main memory to find pointers. Though since heapq module is implemented in pure python regardless, it would be an interesting case study.

[dusty verge]

Oh, wait the syntaxerror is due to the newline?

I thought it was due to the unpacking

[peak spoke]

doesn't heapq have a C implementation with python only used for some parts of the interface and as a fallback like with json etc.?

[nova iris]

no lol, (expression if expression) is not valid in python

[nova iris]

no, it's due to the parentheses

[raven ridge]

Yes

[deft pagoda]

i remember theres some cool private functions in heapq

oh, max heap functions

secret tech

[candid pelican]

i saw a so interesting code in javascript ....

const keys = {
  ArrowUp() {
    console.log('ArrowUp')
  },
  
  ArrowDown() {
    console.log('ArrowDown')  
  }
}

document.addEventListener('keydown', (event) => {
  if (keys[event.key])
    keys[event.key]()
})

it's useful to avoid repeat some ifs statements... there's some way to do it in python???

[charred pilot]

yes, you can do basically the exact same thing

[candid pelican]

but how to declare a function inside a dict??

[charred pilot]

you can't make a def, but you can either use a lambda or just reference a previously defined function

[candid pelican]

i can but to complex stuff??

[charred wagon]

That isn't valid syntax in JS. But I see what you're getting at. Python has anonymous functions and first-class functions so yes, it can be done in Python

[candid pelican]

why not??

[charred wagon]

Because objects need to have key value pairs, but that code isn't doing that

[astral gazelle]

its

ArrowDown: () => {}
ArrowDown: function() {}

[charred pilot]

in python it'd be something like

def go_up():
  print("going up !")

def go_left():
  print("going left !")

commands = dict(up=go_up, left=go_left)

I = get_next_event() # or something

commands[I]()

[candid pelican]

no

i just forgot the ","

[nova iris]

>>> foo(*(1,), 2, d=4, *(3,), e=5, f=6)
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6}
```what happened to posargs before kwargs 😔

python's call grammar is really weird

[candid pelican]

i think that i could use an class

class Keys:
    def __init__(self):
        self.keys = {
            'arrow_up': self.arrow_up,
            'arrow_down':self.arrow_down
        }

    def arrow_up(self):
        print('arrow_up')

    def arrow_down(self):
        print('arrow_down')

allowed_keys = Keys()
allowed_keys.keys['arrow_up']()

look it

but it is so big

[paper echo]

this looks alphabetical

keys = dict(
    arrow_up=lambda: print('arrow_up'),
    arrow_down=lambda: print('arrow_down'),
)

[candid pelican]

yes i know

but to complex things???

just use ifs statements lol

[deft pagoda]

def complex_function():
    ...

keys = dict(
    arrow_up=complex_function,
    ...
)

[candid pelican]

its a good one....

i did smth similar to do a CLI

[candid pelican]

i found this way to:

class Keys:
    def __getitem__(self, key):
        return Keys.__dict__[key]

    def arrow_up():
        print('arrow_up')

    def arrow_down():
        print('arrow_down')

allowed_keys = Keys()
allowed_keys['arrow_up']()

the best way i think

[sacred yew]

nope that's valid

[candid pelican]

yes it is

[sacred yew]

it's a es6 feature

[candid pelican]

i didn't know that

[sacred yew]

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Method_definitions

[candid pelican]

i used object literals

to call the funtions

[nova iris]

well, i did order the parameters in alphabetical order

but the point is python's grammar for calls have this section:```ebnf
starred_and_keywords ::= ("" expression | keyword_item)
("," "" expression | "," keyword_item)*

which means that you can put single-asterisk iterable unpacking after keyword arguments

so the actual call order is:```
func(<posargs or *something>, <kwargs or *something>, <**something or kwargs>)

i wonder why this particular feature exists

[static bluff]

I concur. I'm sure some of the jedi masters on here will have something to say on the matter, but this does seem like an oversight

Maybe it was included as a matter of flexibility

Are you finished? I popped on here to ask about my own thing, but I don't want to derail your discussion

[nova iris]

hm, i'm really trying to think of a scenario where we would want to insert a single-asterisk unpacking AFTER a keyword argument

[nova iris]

sure! ask away, i just wanted to share my thoughts :D

[static bluff]

Please feel free to weigh in

A while back I built what I refer to in my mind as a totally-overkill lexer for Python code

Really, it was a meta-lexer that built lexers from grammars. It was highly configurable, state-based and worked using callbacks written directly into the grammar

Later on I spoke to the guy who actually wrote the tokenize.py module in the standard library. He chuckled when he saw my code, because it was totally over-engineered. The lexer he wrote was an off-the-cuff quick and dirty support module for Python's fancy new PEG parser

[paper echo]

TIL you could do this

[nova iris]

i can't even build lexers, so you making a meta-lexer is already super impressive :P

[paper echo]

!e ```python
def f(a, b=1, *args, **kwargs):
return a, b, args, kwargs
print(f(1, b='oops', *(9,9,9), **{'x':5}))

[fallen slate]

@paper echo :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 3, in <module>
003 | TypeError: f() got multiple values for argument 'b'

[nova iris]

lol, the first 9 likely went into the b slot

wait no

i'm pure confusion what

[static bluff]

Thanks! Though the hardest part was actually dissecting the ridiculously unsemantic code that most python lexers are built with. Its really just a big, tightly sequenced regex pattern

[nova iris]

yeah, reading other code is sometimes quite hard, especially if it's something similar to regex lol

[paper echo]

well the point is, look at the signature for f

[nova iris]

today i dissected delegating subgenerator code

[paper echo]

i suspect this was just to allow people to put *args at the end for cosmetic reasons

[nova iris]

yeah, b is a pos-or-kwarg

[nova iris]

*args? but that's the function param

wait, i'm going to show you something

[paper echo]

i mean in the signature

[nova iris]

>>> def foo(a, b, c, *, d):
        print(a, b, c, d)

>>> foo(1, 2, d=4, *(3,))  # works
1 2 3 4
>>> foo(b=2, *(1, 3), d=4)
Traceback (most recent call last):
  File "<pyshell#20>", line 1, in <module>
    foo(b=2, *(1, 3), d=4)
TypeError: foo() got multiple values for argument 'b'
``` @paper echo this is super weird!

somehow, when you use an iterable unpacking like this after a pos-or-kwarg, it ignores that b already has an argument

wait actually, it might not be so weird after all

because foo(1, 3, b=2, d=4) is definitely an error

so perhaps the behaviour is the same?

[nova iris]

OH WAIT I UNDERSTAND WHAT YOU MEAN NOW

yep, that makes sense

but in that case, weird stuff could arise like the unexpected behaviour above

@paper echo i got it!

First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots.

this is the reason why

no matter what order the args are in the parens

posargs have priority

[paper echo]

yeah that makes sense

the parameter list and arg list are handled separately

good info

[nova iris]

this part of the documentation has tons of stuff to dive into https://docs.python.org/3/reference/expressions.html#calls

[nova iris]

bruh the documentation explicitly states our problem 🤣

A consequence of this is that although the *expression syntax may appear after explicit keyword arguments, it is processed before the keyword arguments (and any **expression arguments – see below). So:

[paper echo]

yeah makes sense

it just gets "hoisted" to the front of the list, and the ability to put it in the back is purely for cosmetic reasons

still, TIL

[verbal escarp]

i find it a good practice to use * and / explicitely now that they are available

and avoid pos-and-kw-args

[static bluff]

Hey, yeah, what's the slash do in signatures? I only saw it for the first time a week ago

[verbal escarp]

pos-args-only / pos-and-kw-args * kw-args-only

[static bluff]

O.o That seems like overkill to me

But I suppose they put a lot of thought into it, so who am I to comment

[verbal escarp]

it's nicely symmetric

[charred wagon]

It hasn't been out for too long but so far I've only used it once

[grave jolt]

It's useful when you want to take arbitrary arguments

def render(template, **kwargs): ...
render("index.html", template="foo", something="bar")
"Error: two arguments provided for 'template'"

def render(template, /, **kwargs): ...
render("index.html", template="foo", something="bar")  # works

[charred wagon]

I think it lends itself well to functions with overloads, which is precisely what my 1 use case was

[verbal escarp]

i like to design my signatures with very few pos-only args and the rest kw-only

[grave jolt]

I also used it in Protocols for callables, so that they don't have to have the exact argument names, which makes them a direct upgrade from Callable[..., ...]

Foo = Callable[[Foo, Bar], Baz]

class Foo(Protocol):
    def __call__(self, /, foo: Foo, bar: Bar) -> Baz:
        ...

[grave jolt]

that does sound like a good idea for consistency

[static bluff]

And context. Something like open obviously takes a filename or path as its first parameter, but everything else being named specifically just makes life easier in the same way that typehints do

[verbal escarp]

exactly

[static bluff]

Anyway, I popped on here originally to discuss lexing

[grave jolt]

Yeah, the purpose of kwargs is that you don't do javascript-style or C-style f(true, false, false, undefined, null)

[static bluff]

I guess... I don't really know how to start 😐

A while back I built a way-over-engineered lexer. It was lots of fun and great experience

[grave jolt]

what do you need to lex?

[static bluff]

But I'm thinking about dusting that project off and I feel like I could use a point in the right direction

[verbal escarp]

whats the question? 😄

[grave jolt]

maybe you should open a help channel, not sure if this is the right place to talk about lexers

[static bluff]

For kicks, I want to build my own language, much like Python. I'd like to have at least the foundation of the interpreter up and running in three years, when I start year 4 and compiler design

[static bluff]

If I did, someone would tell me its far to advanced a topic; that the people who roll through help channels mostly deal with simple syntax issues and the like

And I'm more interested in best practice. You guys are the people to talk to about that

[static bluff]

Anyway. I guess I'm having trouble feeling out how much too much

[grave jolt]

Help channels are for all kinds of questions. This channel is more about meta-stuff about Python, like general best practices and language features.

[nova iris]

isn't it out in 3.8? forgot

[static bluff]

I actually spoke with the guy who wrote the built in tokenize.py module and he told me it was written in haste, a quick and nasty support module for python's fancy new PEG parser

And it shows in the code. Its a bit of a mess. So, I sat down and built a way better lexer that is apparently, totally overkill

[grave jolt]

I guess you can look at how pypy does tokenization (although it seems kinda cryptic to me)

https://github.com/mozillazg/pypy/tree/master/pypy/interpreter/pyparser

[static bluff]

I just feel a bit lost. I'm not sure what I need and what I don't. And Then there is actual implementation — some people on here believe that for something this low level a pure-functional approach is best

[verbal escarp]

i feel you. there are a couple of core modules that are just badly written/designed

[static bluff]

This lexer looks very similar to the tokenize.py lexer. After a lot of testing and refactoring, I discovered that this lexer works by repeatedly matching against a large, carefully sequenced regex pattern built from many subgroups. After detecting what kind of token is 'next', it applies five operations: check for termination and inject EOF token if needed; enforce indentation rules, detect parenthetical mismatch, create the token, and one other. Can't quite remember off the top of my head

[verbal escarp]

i don't know how to fix them though, there seems to be a gigantic backlog of PRs for cpython

[static bluff]

The salient point I deduced from this is that this huuuuuuuuge heuristic that most tokenizers use, if-this-token-and-some-state-is-this-then-do-that can be boiled down to callbacks attached to specific tokentypes — with flags on the lexer being set by certain token types also to assist in logic

[verbal escarp]

yeah, sounds reasonable

i'd have tried to make it work with nested FSMs, but that's just me ^^

[static bluff]

So thats what I did, I built a grammar which contained tokentypes by regex-pattern, with callbacks or other state mechanics embedded in tokens as needed. Not unlike a PEG parser

[static bluff]

That's what I would have done if I were building a one-off. But the idea was to build a lexer-generator that works with a lexer grammar. That way I could write different grammars for my language itself, for my language's regex engine, for the meta-grammar, and any other sub/utility language I needed

[verbal escarp]

so you wrote a kind of DSL for lexers

not a bad idea, i think i remember seeing something like that somewhere in python

[static bluff]

So yeah. Thats what I did. I ended up having a roaring argument with someone on this channel about storing state as attributes on the lexer instead of having all of that information existing as local values in a function. Something about scope leak and purity, and it haunts me to this day

Based on all this, I was hoping for some sage advice to get my started up again 😐 I feel lost

[verbal escarp]

heh

sorry to hear

well, just as a nudge - how about combining lexing with the AST?

[static bluff]

A single grammar that contains the protocols for both lexing and parsing? (for that specific DSL)

[verbal escarp]

as discussed the other day, it looks like analyzing and manipulating the AST is key to many interesting future applications, so maybe it's conceivable to have a language for AST-manipulations

couldn't get more meta than that 😉

[static bluff]

XD I actually had that thought, sorta, the other day

About exposing my language's interpreter much like Python exposes its import machinery. That gets tricky though because if the language itself is mutable, then it can't effectively debug itself

I guess you'd need an independent, immutable metalanguage

[verbal escarp]

kind of, yeah

yeah, that's probably the most appropriate name for the thing

a metalanguage

[static bluff]

I guess the main thing that is still haunting me

[verbal escarp]

well, since that discussion the other day, i've been pondering about a good approach to GP on the AST, so maybe that's a primer/reason d`être for a meta-language

[static bluff]

TOKENTYPE('SOME_TOKEN')    :    'some_regex_pattern'
  STATE_CHANGE('some_state', True)
  CALLBACK('some_callback', arg1=0, arg2='string')

STATE('some_state', False) # False is initial/default value
``````py
class MetaLexer(type):
  
  def __init__(lexer_class, tokentypes, states):
    
     lexer_class.TOKENTYPES = tokentypes
     lexer_class.STATE_DEFAULTS = states

class SpecificLexerClass(metaclass=MetaLexer):
  
  def __init__(self):

    self.states = copy.copy(self.STATE_DEFAULTS)

  def some_callback(self, arg1: int, arg2: str):

    if arg1 > 0:
      self.states.some_state = arg2

Forgive me, I've been away from Python a few weeks, taking a chance to get some last minute gaming in before school starts XD

The person I had the argument with took issue with: using inheritance in this way to attach the actual functions named in the grammar to the lexer; storing states on the lexer as attributes instead of being local within a singular create_token function; using a callback-centric approach to begin with

[verbal escarp]

looks almost like the FSM framework i wrote a while ago

[static bluff]

Oh?

[verbal escarp]

i didn't have lexing in mind for it, but it has some resemblance

[static bluff]

So nothing about this raises any real red flags for you

[verbal escarp]

not really a big fan of metaclasses 😉

but no, not really red flags

i'd use enums, if possible

[static bluff]

Oh?

[verbal escarp]

you don't know about https://docs.python.org/3/library/enum.html?

[static bluff]

Only vaguely

[verbal escarp]

i can very much recommend it

[static bluff]

Just that they are unique values, potentially bitwise enabled

[verbal escarp]

for instance you could have your TOKENTYPE as an enum etc

the states, too

would probably simplify a lot of code

it did for me

also IDEs reward you with autocompletion and static type checking

[static bluff]

Hmmmmmmm

Okay, thank you!

[verbal escarp]

having said that, i'm almost certain enums themselves use metaclasses.. so maybe i am a fan afterall 😉

[verbal escarp]

yeah, np

[static bluff]

In a similar vein

Is a hand built parser designed to handle input character by character faster?

I know that it's possible to completely bypass the lexing step. Parsing seems to be case-by-case type problem. Have you all any wisdom regarding a good ratio of speed and maximal readability?

[magic nova]

there's very few areas where I'd ever prioritize parsing performance over the readability of my lexer/grammar/parser/parser generator. In most languages, what you're doing after the parsing is going to dominate the time, unless it's like parsing big json blobs

[static bluff]

So how would you go about building a lexer, from an object oriented perspective

The only pythonic lexers for python source code I've found tackle it from a purely functional perspective. The generate_token[s] function is always a single, massive heuristic that checks the current position in the source code against a big regex pattern and then from that, checks which group(s) in the regex pattern are matched. Using that and some other state variables it does a bunch of logic and produces a token

I find this approach horribly ugly and quite hard to read. Its also next to impossible to extend short of rewriting the whole function

On the other end, there's PLY/SLY/Lark which is abstracted to the point of only supplying the regex patterns for tokens and the grammar for the AST

[summer forge]

What do you guys think about the tricks to use private access in Python?

Do you think it shouldn't be done?

[grave jolt]

what kind of tricks?

prefixing private stuff with _ is the prevailing pattern in Python, usually it's good enough

there's also __ which serves a different purpose (preventing collisions)

[summer forge]

I've seen some people using decorators to achieve private access

[raven ridge]

the very few times when I need more than just the underscore convention, I use Cython to make actually private privates. Pretty much only when the user could crash the interpreter or something by touching the privates. Which is a case that basically only comes up when you are already working with extension modules.

[grave jolt]

!e

def f():
    pass
f.__code__ = f.__code__.replace(co_consts=())
f()

[fallen slate]

@grave jolt :warning: Your eval job has completed with return code 139 (SIGSEGV).

[No output]

[raven ridge]

I, uh - what?

[surreal sun]

I heard it is also used for name mangling, but what is name mangling

[charred pilot]

usually it's for inheritance

[grave jolt]

!e

class A:
    def __very_private(self):
        return 42

print(A.__dict__)

[fallen slate]

@grave jolt :white_check_mark: Your eval job has completed with return code 0.

{'__module__': '__main__', '_A__very_private': <function A.__very_private at 0x7fb7e13588b0>, '__dict__': <attribute '__dict__' of 'A' objects>, '__weakref__': <attribute '__weakref__' of 'A' objects>, '__doc__': None}

[raven ridge]

it's a way to allow you to add a new private attribute to an existing class that has already been subclassed by other users, without any risk of clashing with a private attribute that the other user added to their subclass already.

[grave jolt]

Just a silly example of how to crash the interpreter by touching some "private stuff"

[raven ridge]

ah, I see.

[floral quiver]

Hey guys, random thought. Wouldn't it be cool if there could be expresions embeded in unformated strings? Like you know how you can do:

exampleStr = f"The tournament '{name}' is {'individual' if teamSize == 1 else 'in teams of '+teamSize}"

The problem is that you couldn't do that sort of things if the variable teamSize doesn't exist on the exampleStr's context. What I mean is that it would be could if you could do like:

exampleFormat = "The tournament '{name}' is { teamSize | 'individual' if teamSize == 1 else 'in teams of '+teamSize}"
exampleString = exampleFormat.format(name='Supreme Championship', teamSize=3)
# "The tournament 'Supreme Championship' is in teams of 3"
# But no, it actually throws error

[unkempt rock]

!e Why are wildcard imports only allowed at the module level? ```py
def f():
from random import *

[fallen slate]

@unkempt rock :x: Your eval job has completed with return code 1.

001 |   File "<string>", line 1
002 | SyntaxError: import * only allowed at module level

[unkempt rock]

I would have thought it is only syntactic sugar for from random import seed, randrange, randint, choice, ... and all the rest

[charred wagon]

IIRC something similar was brought up in a discussion here like a week ago. PEP 227 was mentioned.

Yeah it actually mentions that the PEP would eventually lead to such star imports becoming errors https://www.python.org/dev/peps/pep-0227/#import-used-in-function-scope

[raven ridge]

Because local variables of a function are known statically, and module imports occuring when the function is called are done dynamically

!e ```py
def func():
y = 1
# from random import *
print(func.code.co_varnames)

[fallen slate]

@raven ridge :white_check_mark: Your eval job has completed with return code 0.

('y',)

[raven ridge]

What would you expect that to print if the import were uncommented? Note that we've never called the function, so it has never executed the import, and if the random module has never been imported there's no way to know what names are in it.

Heck, there's not even any guarantee that it would have the same names in it every time it's imported, thanks to reload() or the fact that sys.modules is writable.

[charred wagon]

Do you know why it was never in the language spec to begin with?

[raven ridge]

Nah, that I couldn't say. Beyond that it's a good thing that it's not, since it would make static analysis harder, and prevent optimizations, and so on

[charred wagon]

It may have been in anticipation for something like pep 227

It's ancient knowledge at this point

[elder blade]

Dear God I thought this was a PEP for strict ES6-style let variables

[charred wagon]

I'm not familiar with those

Oh I am I just didn't know it was es6. Anyway what gave you that impression?

[elder blade]

The name 😅.

function test() {
    for (const i of [1, 2, 3]) {
        let my_var = i;
        break;
    }
    console.log(my_var);  // 'my_var' isn't defined
}

I feel like you can call that statically nested scopes

[pliant tusk]

!e ```py
load_addr = type(m:=lambda n,s:lambda v:s(v)or n)(
(M:=m.code).replace(
co_code=M.co_code.replace(b'\x87',b'\x88',1)
),{}
)(r:=iter(range(2**63-1)),r.setstate)

print(load_addr(id(1)))```

[fallen slate]

@pliant tusk :white_check_mark: Your eval job has completed with return code 0.

1

[pliant tusk]

@languid yoke ^ even smaller

[languid yoke]

That doesn't look the sort of thing that can continue to shrink much.

[verbal escarp]

you mean like f"The tournament '{name}' is {'individual' if teamSize == 1 else f'in teams of {teamSize+1}'}" ?

your example exampleStr = f"The tournament '{name}' is {'individual' if teamSize == 1 else 'in teams of '+teamSize}" doesn't parse

[verbal escarp]

oh, you meant like templating

see https://stackoverflow.com/questions/42497625/how-to-postpone-defer-the-evaluation-of-f-strings

well, since PEP 501 still is "deferred", you're stuck momentarily with constructs like ```python
def fstr(template):
return eval(f"f'{template}'")

[flat gazelle]

why not make a function like

def template(team_size):
    return f"The tournament '{name}' is {'individual' if team_size == 1 else f'in teams of {team_size+1}'}"

[verbal escarp]

too little confusing 😉

well, point is, you need an outer function to defer evaluation

[flat gazelle]

ye, it would be nice to have sth like the JS fun`hello ${world}`

but I think a function is adequate enough for enough usecases that isn't a massive issue

[verbal escarp]

i'm wondering if you could turn normal strings into f-strings on the fly

[flat gazelle]

I would really really avoid eval and similar constructs when dealing with string interpolation

[verbal escarp]

no, where's the machinery behind f-strings?

[elder blade]

Ummm, what's that?

[verbal escarp]

maybe it's possible to just call an internal function to evaluate a normal string as f-string

[flat gazelle]

I don't think so without just calling eval with extra steps

the machinery is part of the parser IIRC

[verbal escarp]

ahhh

they are part of the AST

[flat gazelle]

JoinedStr(values=[FormattedValue(value=Name(id='a', ctx=Load()), conversion=-1), Constant(value=' b '), FormattedValue(value=Constant(value=5), conversion=97)]))

[verbal escarp]

ast ast ast ast ast :p

yeah, i found tests for cpython on github, test_fstring.py

https://github.com/python/cpython/blob/8e832fb2a2cb54d7262148b6ec15563dffb48d63/Lib/test/test_fstring.py

hmm..

muahaha

i got it

pattern = (f"The tournament '{name}' is {'individual' if teamSize == 1 else f'in teams of {teamSize +1}'}" for _ in iter(int, 1))

to evaluate, next(pattern)

oh, oops. that walrus operator.. nvm ^^

[grave jolt]

you could just do a lambda, no?

or a normal function

[verbal escarp]

pattern = lambda: f"The tournament '{name}' is {'individual' if teamSize == 1 else f'in teams of {teamSize +1}'}"

yeah, you could

[grave jolt]

#esoteric-python 👀👀

[verbal escarp]

but it doesn't involve infinite generators :p

[languid yoke]

It exploits an optimisation to allow you to access any address in the Python interpreter, without needing ctypes.

[grave jolt]

pattern = iter(lambda: f"The tournament '{name}' is {'individual' if (foo := teamSize == 1) else f'in teams of {teamSize +1}'}",0)

[verbal escarp]

i wouldn't call that "esoteric" ;p

the problem is a real one, even got a PEP and all

[elder blade]

I am pretty excited about it, it's gonna be cool

[grave jolt]

what PEP?

am I under a rock again

[verbal escarp]

#internals-and-peps message

[grave jolt]

ah

[flat gazelle]

yeah, this is without a doubt a useful feature, and we should probably have it in some form

[verbal escarp]

although, i must say my lambda is elegant

[flat gazelle]

ye, lambdas for deferred evaluations aren't an API I would hate

[verbal escarp]

possibly the nicest application i've seen

[grave jolt]

Why not just make a defed function? is that too verbose?

[verbal escarp]

i'd say too verbose, yeah

in this case, there's not even a signature to annotate or anything

[unkempt rock]

        # Add a bound __format__ method to the 'y' instance, but not
        #  the 'x' instance.

comments have one space at the first line, interesting

[verbal escarp]

not really interesting. if you run black and sourcery in vscode like me and accidentally stumble into stdlib modules, it's usually accompanied with a flood of code quality suggestions

as a core dev once said "one should not view the stdlib as the paragon of good code"

[brave badger]

Python could definitely use some shedding (heh) of 30 years worth of baggage

[grave jolt]

I refuse to look at re again

[verbal escarp]

😄

sausages..

[verbal escarp]

to be fair, the python's constantly shedding

the rate old stuff is deprecated is quite high

[verbal escarp]

i'd say that's actually one of the things python is better than most other languages

[brave badger]

The feature creep is not as bad as that of C++, sure

Then again, the number of languages that hold the status of its [parent language], but done right is common enough that I can expect it happening to Python few years off

[verbal escarp]

everyone has 20/20 hindsight, but only few are willing and able to change things, especially if it means breaking backwards compatibility

i feel it's almost engrained in python that deprecation is part of the release cycle and viewed as a good thing

[paper echo]

I don't know any of these languages

Other than maybe Zig

Maybe kotlin is kinda like java done right

Otherwise idk, nothing is new and everything is bad and has baggage

[pliant tusk]

It leverages some bugs in cpython to be able to load python objects from any address