internals-and-peps

[static bluff]

Just by the fact that you aren't settings off any major alarm bells, I'm encouraged

If you're talking about the if not queue, its that any tokens within the queue implies that the yield-loop must continue, but by remove the 0th token from the queue in the same line, I can elegantly signal to the rest of the loop below that more tokens must be made (queue is empty)

    def tokenGenerator(self):

        while (token := self.queue.popleft() if self.queue else None) or self.continued:
            # By simultaneously popping the 0th item from the queue and using that token
            # as a boolean within the conditional above, the loop is signaled to execute
            # both the yielding of a token and— by virtue of the fact that the queue may
            # now be empty— generate more if possible.

            # Additionally, circumstances may occur where lexing has not finished but no
            # tokens are present within the queue because the previous iteration of the
            # loop detected a match but was signaled (for any number of reasons) not to
            # produce any tokens. In such circumstances, the lexer's 'continued' flag is
            # set to true (and then set to false again by default).

            if not self.queue:

                for typename, tokentype in self.tokentypes.items():
                    if literal := tokentype.match(self.buffer): break

                # If a literal does not exist, no match has been found and an error may
                # need to be thrown (below, outside the loop).
                if not literal: break

                self.resetScannerStates(**tokentype.switches);
                self.executeTokenCallbacks(literal, **tokentype.callbacks);

            if isinstance(token, TOKEN): yield token;

        if self.buffer: # source code remains to be lexed; lexing has failed
            raise SyntaxError('invalid syntax', self.context);

Its changed a bit, because I encountered a few issues. This isn't quiiiiiite up and running, but almost

[paper echo]

Yeah with comments it makes sense, but it's surprising and unusual control flow. Maybe the weirdness is a sign that the design is off somehow?

[static bluff]

Broooooooooooo

I'll take weird over the clusterfuck that is the tokenize.py module

Its effective, sure, and probably pretty fast- but it is NOT intuitive

[halcyon trail]

my eyes are bleeding from reading this 🙂

[static bluff]

Then perhaps you should go put on a bandaid 🙂

[halcyon trail]

unfortunately it doesn't work that way 🙂

[paper echo]

Also @static bluff I think maybe #software-architecture is a better place to discuss this kind of thing

[static bluff]

Every time I go somewhere else I get told that the subject matter is too advanced

[paper echo]

Wat

I guess the advanced people hang out here

[static bluff]

Maybe not on this in particular but as a general rule, this is the only place I ever get any meaningful discourse

[halcyon trail]

okay, I had clicked on the link and then I saw it was different from this code

I'm not sure what if you don't find a literal, you want to break, and then raise an error from outside the loop

why

*why

Also that will still throw a weird uninformative exception, it seems to me, if the for loop never assigns to literal

That's probably the kind of situation where you want to use an else clause on a for loop

                # If a literal does not exist, no match has been found and an error may
                # need to be thrown (below, outside the loop).
                if not literal: break

Unless there's code missing here, if not literal: break isn't an appropriate way to handle the possibility that literal doesn't exist

[static bluff]

You're totally spot on about the outside the loop! Thanks!

[halcyon trail]

this will probably throw an exception before it even breaks, but obviously not a very informative exception

                for typename, tokentype in self.tokentypes.items():
                    if literal := tokentype.match(self.buffer): break
                else:
                    # okay, here we deal with the fact that literal was never assigned

[static bluff]

As for just 'invalid syntax' as a message, well, the only other message that would be appropriate would be 'unrecognized literal'. 'invalid syntax' is actually what python itself issues when it finds itself in that situation

[halcyon trail]

Ah, I see, it will always assign literal I suppose, in the process of doing that if check

I haven't used walrus much

[static bluff]

❤️ ❤️ ❤️ I AM THE WALRUS

COO COO KOO-CHOO MOTHAFUCKAS

I'm off for dinner. Thanks for all the help my peeps! 😄

[static bluff]

Anyone working on anything fun today?

[lilac yew]

I wanted to have your(you all) opinion about gui frame works in python.Like which one is recommended , easy to use , versatile and lightweight. these keywords are just my random mentions , no need to emphasize on them. I would love to prefer your experiences. so far i have found KIVY , pyautogui , tkinter , pyqt5

[static bluff]

XD I mean, my pet project is developing a python framework that actually uses html embedded in a python driven web browser

Really gotta get back to that

[lilac yew]

from the current conversation , what reminded me that using web browesr based gui is also seems cool to me. as it seems light weight and no need to open separate window besides your browser.

[lilac yew]

i am going for pysimpleguiqt , it seems quite good.

[leaden geyser]

Right now, I've got a nice simple python GUI progam. I'm interested in 1) making this program persist 2) get easily installed/uninstalled/removed from startup and 3) to sit in system tray and raise toast notifications
Any advice on what to use for toasts/easy installation/persisting?

[lilac yew]

your requirement seems quite good. typical smooth and easy UI are like this. i will alos love to hear those advises.

[leaden geyser]

for that GUI I only used tkinter - (since it's part of standard library and because the system I was putting it on didn't let you use third party libraries) but for personal projects I've also been looking into other UI options and I also haven't managed to figure out what one's the best

so what you're also looking for is something that I'm also interested in but which I don't have much experience in

in my situation, I felt that tkinter is easy to setup but I feel that some parts of it are not too customisable and also I guess another thing I was thinking about was if I learned a cross platform GUI it might be useful because in other programming languages I'd be able to use it too so I wouldn't have to learn yet another GUI library

[lilac yew]

have you tried pysimpleguiqt ? or you wont use it as it has qt additional?

do u consider qt as third party ?

[leaden geyser]

I can use anything for personal projects, but when I was making that particular piece of code, pip's access to pypi.org was blocked by my workplace so I could only use tkinter

so the only thing I've tried so far is tkinter but I'm also thinking of picking up another GUI option and hearing about your experience with pysimpleguiqt

[lilac yew]

OK , let me try first. but from the reviews and it seems promising .

[leaden geyser]

there are so many options! 😱

[magic python]

this is something that i've often tried and that fails, but I don't understand why it isn't an option:

[*list(y.values) for _, y in {'a': [1,2], 'b' : [3,4]}.items()]

The error being : SyntaxError: iterable unpacking cannot be used in comprehension, why can't it be used? Just wondering if there's a logical reason here

[lilac yew]

for that qt? yeah , lots of good reviews. and try to read the documentation

[magic python]

Note - i'm aware [el for lst in {"a": [1, 2], "b": [3, 4]}.values() for el in lst] will work here, it's just I'm not sure why I can't unpack in a comprehension

[peak spoke]

list comps use list.append directly under the hood so extending wouldn't work there and there are (better?) alternatives for it

[magic python]

it might just be "because you can't", which is fine i guess.

[magic python]

better alternative in what way?

the other approach that I gave? I"m not really mad on that tbh, i just know it works lol

[peak spoke]

Yes, iterating through the iterable in an another for

[magic python]

wasn't aware of the append thing though, i guess it makes sense

something i bumped into yesterday and have never considered before: https://stackoverflow.com/questions/19302530/python-generator-send-function-purpose

[magic python]

How does the ordering work for sets?

set_data = sorted({0, 1, 17, 18, 35, 36, 39})
{x: y for x, y in zip(set_data, range(len(set_data)))}

gives (note positioning of the 18 key):

{0: 0, 1: 1, 17: 2, 18: 3, 35: 4, 36: 5, 39: 6}

whereas

set_data = {0, 1, 17, 18, 35, 36, 39}
{x: y for x, y in zip(set_data, range(len(set_data)))}

gives

{0: 0, 1: 1, 17: 2, 35: 3, 36: 4, 18: 5, 39: 6}

[peak spoke]

As far as the user should be concerned, it doesn't

[magic python]

it seems to be consistent tho

things like this

why have they be reordered there?

[peak spoke]

the implementation does store them in some way under the hood and exposes that for iteration, but it won't be consistent. I think it does something with the hashes and the order they were inserted in

[eager trail]

I think it's something about how the view is generated

[magic python]

but it won't be consistent
oh right, they seemed to be the same each time in this session

i guess i want to drop duplicates without changing the ordering , not sure what that is called

[peak spoke]

ipython also sorts them when printing I think

[magic python]

oh does it, i have compounded problems then 😅

[peak spoke]

>>> a = {1,2}
>>> a
{1, 2}
>>> a.remove(1)
>>> a.add(1)
>>> a
{2, 1}

You can use unique_everseen from more_itertools if you need it to be unique with order

[magic python]

hm i was going to try list(dict.fromkeys(<thing>))

[peak spoke]

Should also work (although relies on hashables) but I wouldn't exactly call it something that's clear

[magic python]

what's the issue with relying on hashtables?

[peak spoke]

well if you have anything non hashable it'll error

[magic python]

oh right, i'd never have that in this context as it's just ints, but fair

[magic python]

re "something that's clear", it's not rare to have something similar to the following:

{x: {a: b for a, b in y.items() if a in data[x].unique()} for x, y in vvl.items()}

I'm wondering whether this is 'clear', and if not, how would it typically be written?

maybe

for column in vvl:
    data_values = data[column].sort_values().unique()
    vvl_keys = vvl[column].keys()
    vvl[column] = {key: vvl[column][key] for key in data_values}

is "better" ? I'm aware that i write so much crap in notebooks that my intuition is probably off.

[visual shadow]

you might find this read interesting. https://stackoverflow.com/questions/41251729/unpacking-generalizations My personal opinion? there's zero logical reason for omitting this, and im not convinced.

[magic python]

When you say "omitting this" are you referring to unpacking in a list comp?

Ah reading now, thanks

Yeah I think it's odd that it's not a thing, it feels natural to try

[stone field]

it is slightly ambigious what the intent is

[mint hemlock]

Hello guys, I dont know where else to ask. I would like to use Visual Studio Code, but plain version is kinda tool-less. I wanna implement there https://github.com/viatsko/awesome-vscode (there are many useful things - readability mostly).My question is how do I implement that into my VS code installation? Just copy and paste things into the folder?

[stone field]

What you can do instead is [item for values in {...}.values() for item in values]

[magic python]

Unpack in a comprehension, unless I've missed your point

[stone field]

yes, I'm saying it is slightly ambiguous what the intent of the unpack is

it could be interpreted to mean you want the unpack to result in a tuple

or it could mean you want the unpack to result in a "flattened" list

[visual shadow]

i think its easier to use the intent of the code rather than the code itself here, since the code itself is flawed (y holds one list at a time, so y.values makes no sense). as for unpacking, there's not unpacking "to a tuple" without a tuple when it's inside a container. unpacking always just unpacks to whatever container holds it. otherwise it wouldnt be unpacking.

ie. it should essentially behave as this

!e

x = (10, 20)
y = (42, 100)
res = [*x, *y]
print(res)

[fallen slate]

@visual shadow :white_check_mark: Your eval job has completed with return code 0.

[10, 20, 42, 100]

[paper echo]

Ask in #editors-ides maybe?

[mint hemlock]

Mhm, thanks

[lilac yew]

not to discourage you, but i would like share my findings/experience. pls dont waste your time by selecting the best ide or modifying the settings. these are endless. rather gain such skill that, whichever ide you get, you can implement a code efficiently. Just think like hackers in the movie. they dont carry their tools, they create from the exisiting / scratch from the environment.

[charred pilot]

if you're going to be using a tool for long periods of time, why not customize it. especially if it would help you be more productive

[mint hemlock]

@lilac yew @charred pilot I agree with you both 😊 any idea how I can include the repository (fuctionalities) into the VS code please?

[lilac yew]

you are talking about git integration. actually , there is a popular git addon called gitlens. though i use the default setup in the vscode. if you look close in the left panel, there is a git icon. click on it , it will ask to connect with github. click further, and your browser will prmptup gihtub

[fallen slate]

Hey @pastel bluff!

It looks like you tried to attach file type(s) that we do not allow (.pdf). We currently allow the following file types: .gif, .jpg, .jpeg, .mov, .mp4, .mpg, .png, .mp3, .wav, .ogg, .webm, .webp, .flac, .m4a.

Feel free to ask in #community-meta if you think this is a mistake.

[lilac yew]

after that you will understand what to do. but it is better to see youtube tutorial.

[pastel bluff]

hello, im a newbie in coding, but im really inconsistent in coding, i skip many days, i cant sit for longer hours, i really need some tips to maintain my schedule, i am even using a calendar but its still no use

[lilac yew]

actually what i do , if it seems overwhelming - i just open github and explore python codes. even it is not similar to create your own code, but somehow you are using your brain to understand it. and what you are trying to do is not that easy to accomplish in a night. but, it is not a sin to complete the task of 100days in 150 days. as it is your self fixed goal. if you keep pressurize yourself, coding may seem boring to you after a time.

[polar owl]

I really like that mentality, I've done that in a few cases and it helps me better understand how to organize my codebases

[magic python]

is there anything that can generate typehints from some data?

[static bluff]

I'm wondering if I can ask a favor from one of you gurus

I'm really trying hard to bring my coding to within more 'normal' standards. And beyond that, I'm in seriously uncharted waters

I wonder if someone would be willing to sit down with me via video or voice chat and walk through my code with me to point danger areas or questionable practices, etc

I know its not acceptable to offer payment; but I'd be willing to make a donation (to a charity of some kind) in someone's name if that would be agreeable

[zenith yarrow]

#software-architecture has good pins that could be able to help with that

[static bluff]

😮

Thanks G

Great handle btw

[spark magnet]

@static bluff why not have a discussion here? why voice?

or, not this channel, but get a help channel

[static bluff]

I just don't want everyone here to have to watch what I hope is a fairly extensive back and forth

That'd be lovely rick! Thank you!

[spark magnet]

a help channel then

[paper echo]

Mypy does do some of its own inference, maybe there is a tool that uses the reveal_type machinery

https://github.com/google/pytype

https://github.com/dropbox/pyannotate

Never used the latter, used the former once and left it on the to-do pile

[peak pollen]

Instagram also have one, I've used it a couple times on some ancient projects I was planning to add types to

https://github.com/Instagram/MonkeyType

[magic python]

Argument "remap" to "func" has incompatible type "Dict[int, int]"; expected "Dict[float, float]"

I thought that using float as a type was more general than int, and that if I used float and passed an int that'd be ok 🤔

the distinction doesn't seem to make sense bc we can index with whatever

In [1]: d = {1 : 'this'}

In [2]: d[1.0]
Out[2]: 'this'

In [3]: d[1]
Out[3]: 'this'

[prime estuary]

Normally yes, but Dict's typevars are configured as "invariant", so it doesn't allow subclasses.

[magic python]

then how do typehint a dict that can have int or float? Do i need to do Dict[ int | float, str] or something?

seems kinda clunky, but fair enough

[prime estuary]

Or use Mapping, which is covariant.

See here for the explanation:
https://mypy.readthedocs.io/en/stable/generics.html#variance-of-generics

[magic python]

how'd that be used in this example?

[prime estuary]

Declare your function as taking a collections.MutableMapping[int, int]

Though, explicitly saying int or float might be more clear.

[magic python]

yeah this seems even worse than float | int

[prime estuary]

You might want to specify it anyway if it's more correct - then it's valid for users to pass in any mapping object, even if it's not a dict.

[magic python]

at what point do people just use a statically typed language i wonder, feels like the worst of both worlds with python sometimes rather than the best

[prime estuary]

Well the primary advantage is that you can abandon typing whenever it makes it problematic for you.

Just #type: ignore, cast() or use Any where needed.

[flat gazelle]

I mean, you would get that exact error in a statically typed language as well, most mutable data structures have to be invariant.

for example

a: dict[int, int] = {2: 5}
def fun():
    return range(10)[a[2]]
def mutate(b: dict[float, float]):
    b[2] = 3.14
fun()
mutate(a)
fun()
```is a type error and just about every statically typed language will prevent it(except Dart, there it is a linter error)

[eager trail]

Can you define class methods outside a class with the sole intention of importing them into classes as a sort of standard handler

[flat gazelle]

why not use subclassing?

[eager trail]

Okay so, we already have a sort of base class we're using, but I want these standalone class methods to be in a different file because I don't want anyone messing around in the baseclass file

In fact the only reason I want them as importable class methods is so they can do logging on the subclass logger

[flat gazelle]

you can have a second base class, or possibly a decorator

[finite sparrow]

thanks to multiple inheritance, its easy to make a base class simple for defining one or a few methods, and use it as a second third etc base class next to the main one

[eager trail]

The other thing I'm trying to avoid is an out of sight out of mind situation

That's why I want them to import it so they remember it exists

Basically, these are a bunch of handlers for api requests to aws where I made them all to basically give the returns in a standard format

It's a single responsibility principle thing

[finite sparrow]

!e
example of the multiple inheritance thing i was talking about

class MainBase:
    def __init__(self, a):
        self.a = a

class SomeMixin:
    def print_a(self):
        print(self.a)

class MyClass(MainBase, SomeMixin):
    def __init__(self, a):
        super().__init__("a =" +  a)

m = MyClass("hey")
m.print_a()

in this case, the person would import SomeMixin and add it to their own class like i did to MyClass

[eager trail]

I know it's less elegant than having multiple inheritance, but the quality of code drifted very badly from the original inheritance principles when the guy who did the original design got promoted, so this is me arm twisting everyone

[finite sparrow]

aw come on

[fallen slate]

@finite sparrow :white_check_mark: Your eval job has completed with return code 0.

a =hey

[finite sparrow]

there

[eager trail]

Haha nice

Problem is I'm the only person here who's even thinking about this right now, and I need to insulate as many shared things as possible from anything with frequent changes

I could just have all the methods outside a class with no cls argument and no logging

So im just wondering if there's a way to neatly keep both the cls argument in for logging without also making importing more obtuse

[finite sparrow]

im still not sure what the argument against using a mixin like this is.

[eager trail]

My interns

😭😭

[finite sparrow]

im not sure i understand? would they not understand what a mixin is?

[eager trail]

Mate, I had to explain why you can't put lists and dicts in sets yesterday

[finite sparrow]

see, the thing is, there is a way to define methods outside of classes and then put them in a class, but im very sure this would be WAY more confusing since it requires a good understanding of class vs instance scopes

[eager trail]

Oh? What's the name of that way? I'll give it a look, then decide if it's worth doing it like that or not...

[finite sparrow]

not really a name for it
heres the gist: classes have a __dict__ which stores all attributes. this includes methods, which are nothing but functions with the extra first argument.
that means you can write a function with a self argument, and assign it into a class

[flat gazelle]

!e

@classmethod
def clsmeth(cls):
    print(cls)
class A:
    mymeth = clsmeth
class B:
    mymeth = clsmeth

A().mymeth()
B().mymeth()
```this just works, but I doubt it is clear by any means, especially if you use an import statement instead of just `=`

[fallen slate]

@flat gazelle :white_check_mark: Your eval job has completed with return code 0.

001 | <class '__main__.A'>
002 | <class '__main__.B'>

[finite sparrow]

heh, you were first, i was typing out the same

[eager trail]

I'm trying to introduce higher level concepts like this gradually, at first before this I had them all try and follow a template when creating the handlers and then just copy paste the standard handlers in first while writing the rest of the logic, then I can slowly phsdd out the duplicated code and replace them with the imported methods

Hmm gotcha, I'll keep thinking about it

[finite sparrow]

why not instead just have them use inherited methods from a base class?

[eager trail]

Yup quite happy to have them do this, it's just that we have really bad automated uploading so if anything breaks literally our entire aws lambda library breaks, so I'm trying to insulate the core baseclass from them as they continue to make changes

The alternative is i do everything myself, but I'll go nuts

You don't know how bad it is here. I came in and asked the guy who got hired before me why they weren't following pep 8,i got a blank look and he asked me what pep8 was

[finite sparrow]

okay wait what? they are literally just subclassing a base class and using methods from that, how can this break the base class?

[eager trail]

Like one wrong indent somewhere in the base rulefile and when our automated compiler does its thing every single function will receive that broken class, because... No automated linting or code quality check. Its all eyeballs

So yes, eventually, automated linter. But first....

[flat gazelle]

I feel like you would have massive payoff from even pretty weak CI

[eager trail]

Yup

Wild wild west out here

[flat gazelle]

also, do you not even use editors that will yell at you about syntax errors?

[eager trail]

They're supposed to use a linter for the cloud formation templates before uploading

cfn-lint filepath

Still doesn't happen all the time

I do, but not everyone does
Basically, since I've come in I've basically taken over a lot of this stuff. Eventually, I'll get my way and enforce standard envs and all that good stuff

[finite sparrow]

but how would the interns be able to mess with the base class? would they not just subclass it and overwrite methods in their subclass if they needed to change something?

[eager trail]

But I'm just one dude

[eager trail]

Oh, because they're still creating the standard methods, we're basically refactoring a lot of code at once, and unless I'm the one doing the changes to the base class I can't guarantee some mistake won't sneak in when I'm not looking

I'm not dismissing just changing the baseclass BTW, I think it's not too far off that we're done with that bit of the refactoring

[flat gazelle]

ah, you want open closed for methods of a class

never ran into that before

[halcyon trail]

What's the down side of just having free functions?

[finite sparrow]

open closed?

[eager trail]

Can't publish to the logger attached to the instance, that's all

[flat gazelle]

one of the principles in solid, the only one I actually run into often. Essentially, you can extend functionality without modifying the source code, just by adding more code. One example is having one big list of endpoints vs decorators

[eager trail]

So it's like neither here nor there

[flat gazelle]

you can have a module level variable with a logger

[halcyon trail]

Why do you have loggers attached to class instances

That's rather unusual in itself

[eager trail]

Wait what

I shouldn't? Am I the dumb 😱😱😱

[flat gazelle]

there is no should or should not

[halcyon trail]

Well maybe you have a specific reason

[flat gazelle]

there are reasons for things generally

[halcyon trail]

But the "default"way to use loggers isn't that

Usually you just do logger = logging.getLogger(name)

[finite sparrow]

most common way is sthy like this

import logging
logger = logging.getLogger("my_project_name.my_module_name") 

just at the top of each file

[halcyon trail]

No

Use dunder name

But yes, most common is top level

[finite sparrow]

only if its the main file, for larger projects its common to explicitly name the loggers with a sublogger path

[eager trail]

Okay, then anytime I need to log within the class, I just do logger.info or whatever without the self?

[finite sparrow]

yes

[halcyon trail]

Why would you do that?

There's no reason to

[eager trail]

And I also call logger in the free functions, and they'll pick up the logger from the name space of the file they're imported to?

I AM THE DUMB

[halcyon trail]

The whole point of hierarchical logging is that it works out without usually needing to name loggers explicitly

Yeah

[eager trail]

Ahhhhhhhhhh

[halcyon trail]

The free functions just use the package level logger

[flat gazelle]

you could also look at loguru

[halcyon trail]

So they do logger.info(...) Etc

[eager trail]

Holy shit I got so paradigm locked to having instance loggers

Yeah this solves everything thanks lads

[halcyon trail]

That's very old Java tbh

There's rare cases where instance loggers makes sense, but its very very rare

[flat gazelle]

the logging module is old java, so I can see how it would feel like the obvious solution

[eager trail]

Bleh. Well, that actually removes another logger related headache I was having, so thanks fam 😘

[halcyon trail]

It's hierarchical logging

Which has been around a while but not since the start of Java

Pre hierarchical logging this is what you usually had

[finite sparrow]

@halcyon trail about the explicit naming: i was thinking of when you have a logger per folder, where a module is a entire folder instead of a single file. you're most likely right in most cases this isn't needed

[halcyon trail]

Maybe I'm not understanding you correctly, but in that case, the loggers will have names like foo.bar1, foo.bar2, etc (if you use dunder name)

You want to manually name them foo

[eager trail]

I was trying to do something similar, but yeah just name then differently

[halcyon trail]

Don't name them differently, unless you really have a good reason, which you probably don't since you're still learning about proper use of logger

[finite sparrow]

another reason to manually name loggers might be if multiprocessing is used in the main file, but yeah, generally just go with __name__ which will default to the proper module/file path (the dot seperated ones, what are those called correctly?)

[halcyon trail]

How does manually naming help with multiprocessing?

[finite sparrow]

because on windows __name__ changes based on what subprocess you're in

[eager trail]

Okay, without naming them differently, how do you modify the logger name if I wanted different names at the start of the print statement based on which file was sending to the logger

I'm not great on advanced logging

[finite sparrow]

generally for most things you would define a custom logging format

leave the names as they are, and use the format to add any additional info you want

[eager trail]

Aight fair

[halcyon trail]

You don't usually need custom names

The logging statement says what package is logging from

@finite sparrow that's pretty weird about windows. til

[finite sparrow]

yeah its because windows doesnt have anything like fork, so python has to re-run the entire file. this has the side effect of runnin any code thats at the base level multiple times too

!e ```py
from concurrent.futures import ProcessPoolExecutor
import logging

logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(name)

def b(a):
log.info(f"hey from main {a=}")

b(0)

if name == "main":
with ProcessPoolExecutor(max_workers=5) as executor:
executor.map(b, [1,2,3,4,5])

this on windows results in ```
INFO:main:hey from main a=0
INFO:mp_main:hey from main a=0
INFO:mp_main:hey from main a=1
INFO:mp_main:hey from main a=2
INFO:mp_main:hey from main a=3
INFO:mp_main:hey from main a=4
INFO:mp_main:hey from main a=5
INFO:mp_main:hey from main a=0
INFO:mp_main:hey from main a=0
INFO:mp_main:hey from main a=0
INFO:mp_main:hey from main a=0

as you can see, the 0 one is printed multiple times. on linux this wouldn't happen thanks to fork magic

[eager trail]

I caused a 4 day compile failure that blocked all uploads because I didn't realise git on Windows was blind to changes in folder capitalization, so when I changed some folder name capitalization I completely broke some stuff until I realised I had to force git to update the folder case

Meanwhile on Linux, not case blind

Sigh

[finite sparrow]

windows file system is case insensitive so thats really not gits fault

you cant have 2 folders named a and A in the same folder on windows either, same with file names

[grave jolt]

So if you have folders like heLLo and HeLlO, you won't be able to clone that on windows?

lol

[eager trail]

Ye it's Windows but I got so confused because I kept getting reference errors until I checked the file directory on the remote against my local

[finite sparrow]

not sure how git handles that

[halcyon trail]

Python reruns the entire file as well

Its calling form + exec

Fork + exec

Afaik at least

Maybe not actually

[eager trail]

No, I think if you start with two folders, you're fine. It's starting with one and changing the folder name that's the problem. It will behave as expected on Linux and change the folder name in your .Git but Windows will quietly ignore the folder case change

[finite sparrow]

jsut tested on my server, same code, python 3.8:

[halcyon trail]

Yeah I know Linux doesnt do that

I just don't think the name difference has anything to do with fork + exec

You can ask multiprocessing to do spawn instead of fork

On linux

See if it changes the name

[finite sparrow]

it does, python on windows changes the name so it can keep track of which process is parent and which isnt

[halcyon trail]

That's a choice it makes though, you could make the same choice regardless of fork vs spawn

[finite sparrow]

the same reason you cant run multiprocessing code outside of a __name__ == "__main__" block

[finite sparrow]

afaik its a limitation windows has over fork/spawn that forces python to use this ugly trick

[halcyon trail]

What's the limitation of windows compared to spawn specifically, I don't underestand

[finite sparrow]

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

this is the error you get on windows when using multiprocessing at the base level. im not entirely sure of the exact reasons either, but i think its down to memory management differences

like, fork does some stuff with sharing memory that windows just cant

[halcyon trail]

Yeah, when you do fork you're sharing memory, but not when you spawn, hence why I asked what the logging in Linux looks like when you use method spawn in multiprocessing

But anyhow I'd have a ways to go before I fully understand all this

[paper echo]

what's the equivalent of this technique for classmethods? https://gist.github.com/bnyeggen/1086393

[undone hare]

this man needs to chill out on indents

I'm not 100% sure of what I'm looking at

[halcyon trail]

did you try just doing pool.map(SomeClass.some_class_method, [SomeClass])

[paper echo]

it's pickling an instance method by pickling only the name of the method, then un-pickling by looking up the method name on the class

[halcyon trail]

well, it's pickling the instances too

[paper echo]

right

it also uses some internal attributes that i'm unfamiliar with

[halcyon trail]

Yeah. But, I would expect class methods to work the same way, just pass classes as the first argument instead of instances

[undone hare]

That's really weird

[halcyon trail]

if it ever gets confusing then it's easy enough to simply define a top level function that does whatever you want and do the map on that

[undone hare]

but all the im_ stuff are internal attribute, like all the co_ ones on code objects

[halcyon trail]

Oh, I had ignored that pickle method stuff at the beginning 🙂

What's the point of this though?

[paper echo]

@halcyon trail this was being used for ProcessPoolExecutor, not sure if that matters

[halcyon trail]

just define a top level function. I wouldn't touch this with a ten foot pole tbh.

[paper echo]

someone in #async-and-concurrency was writing a decorator to automatically run a classmethod in an executor

[halcyon trail]

ah, i see

[paper echo]

i think the whole idea is somewhat odd but

i do like that this functionality exists

[halcyon trail]

I still don't think you need any of this stuff but maybe I'm wrong

again, just define a top level function with suitable arguments, and pass that top level function to the executor

[paper echo]

well you can't pickle classmethods, not sure if using a top-level function lets you bpyass that

i agree this shouldn't be a decorator

[halcyon trail]

Yeah, but why not just have a top level function that gets the class, and the name of the class method

and just does the lookup

[paper echo]

they wanted a decorator

[halcyon trail]

your decorator could do that

def top_level_func(clss, class_method_name, ...):
    method = clss.__dict__[class_method_name]
    ...

I'd rather just do these things transparently in a function like this then to start messing around with "overloading" how pickling works

[paper echo]

oh, i see

how do you get the name of the bound class from a classmethod, in a decorator?

[halcyon trail]

I think you can get that via reflection

I mean in the code you linked, you have to get a bunch of stuff via reflection anyway

[paper echo]

right

i just wouldn't know how to do that in this case

[halcyon trail]

does the decorator run the class method immediately in an executor, or make it so that whenever it's called, it's executed in an executor?

I was able to extract the actual function and class from a decorator like so

You can do something like:

def dec(f):
    print(f.__func__)
    print(globals()[f.__func__.__qualname__.rsplit('.', 1)[0]])

class Foo:
    @dec
    @classmethod
    def bar(cls):
        return 0

This gets you the unbound classmethod, and the class itself

both of those things I'd expect are serializable for multiprocessing

hmm that won't actually work because Foo isn't yet defined

If you want to do this in a decorator I think you really will be forced to pass some strings instead and lookup the class in the function you pass to multiprocessing

[dusty oak]

always wondered what mixing is

I think it is just the same decorator but in its class version)

[paper echo]

a "mixin"?

[dusty oak]

oh yeah, mixin 😉

[native flame]

if you're familiar with interfaces in other languages, inheriting from a Mixin is like inheriting from a pre-implemented interface

[paper echo]

i answered in #software-architecture because it's not really on topic for #internals-and-peps

[fallen slate]

typing.Protocol

class typing.Protocol(Generic)```
Base class for protocol classes. Protocol classes are defined like this:

```py
class Proto(Protocol):
    def meth(self) -> int:
        ...
```  Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example...

[paper echo]

duck typing is when you don't care what the name of the data type is, you just need it to have certain methods or attributes, or otherwise implement certain interfaces

also known as "structural typing"

the "duck" name comes from the joke, "if it quacks like a duck, it's a duck"

[mild flax]

i.e. you don't care what an object is, you care how it behaves

[halcyon trail]

Protocols are like static duck typing

A class satisfies a protocol, statically, if the class defines all the members required by the protocol (meaning, correct names, and correct types/signatures)

What Go has is a lot like protocols in python. Having the option of protocols in python kind of makes sense since the language is dynamically duck typed to start, so it's basically a static mechanism which more closely matches how dynamic python works

The other static approach in python is via explicit inheritance, including ABC's, which is more like what you see in most statically typed languages

yes, but so is is abstract base class

Well, sorry, let me even go a step back

it's not really a set of rules

it's just a set of syntax/types

The rules may exist beyond that, but they're additional constraints that can't really be verified by the protocol

yeah, I mean the docs have such examples

class MyProtocol(Protocol):
    def a_member_func(self) -> int:
        ...

class Foo:
    def a_member_func(self) -> int:
        return 5

the key point here is that Foo is recognized (statically) as satisfying MyProtocol

because it has a member with the right name, and the right type signature

it's similar to a sub class, yes, you just don't have to declare it explicitly

basically implicit sub-classing

If we did the above exercise with abstract base classes, then you'd have to explicitly say somewhere that Foo inherits/satisfies MyAbstractBaseClass

With protocols you do not

For static type checking purposes

So now you can write:

def some_func(x: MyProtocol) -> int:
    return x.a_member_func()

and you can do some_func(Foo()) and this will (should?) all type check

If you modify Foo so that it no longer meets the protocol, or the protocol is expanded so that Foo no longer meets it, then it will no longer type check

mypy or whatever will complain

It's just a different approach, to accomplish something very similar to ABC's

hah yeah I noticed that

[paper echo]

abc/nominal version:

from abc import abstractmethod, ABCMeta
from typing import ClassVar


class AbstractCharacter(metaclass=ABCMeta):
    starting_hp: ClassVar[int]
    hp: int

    def __init__(self):
        self.hp = self.starting_hp

    @abstractmethod
    def receive_damage(self, amount: int) -> None:
        ...


class PlayerCharacter(AbstractCharacter):
    starting_hp = 100

    def receive_damage(self, amount):
        self.hp -= amount
        print('Ouch!')

protocol/structural version:

from typing import ClassVar, Protocol


class PlayerProtocol(Protocol):
    starting_hp: ClassVar[int]
    hp: int

    def receive_damage(self, amount: int) -> None:
        ...


class PlayerCharacter:
    starting_hp: ClassVar[int] = 100
    hp: int

    def __init__(self):
        self.hp = self.starting_hp

    def receive_damage(self, amount: int) -> None:
        self.hp -= amount
        print('Ouch!')

i think, anyway

not sure exactly what types mypy can infer vs what types are required, but i think the idea is clear

the main difference is that in the protocol version, there is no explicit inheritance; the protocol is purely a description

[fallen slate]

typing.ClassVar

typing.ClassVar```
Special type construct to mark class variables.

As introduced in [**PEP 526**](https://www.python.org/dev/peps/pep-0526), a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. Usage:

```py
class Starship:
    stats: ClassVar[dict[str, int]] = {} # class variable
    damage: int = 10                     # instance variable
```  [`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") accepts only types and cannot be further subscribed.

[`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") is not a class itself, and should not be used with [`isinstance()`](https://docs.python.org/3/library/functions.html#isinstance "isinstance") or [`issubclass()`](https://docs.python.org/3/library/functions.html#issubclass "issubclass"). [`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") does not change Python runtime behavior, but it can be used by third-party type checkers. For example, a type checker might flag the following code as an error:

[paper echo]

it's for a class attribute

whereas in the abstract-base-class version, something can only "implement" that interface if it inherits from the specific abstract base class

[halcyon trail]

in python specifically protocols are nice because they match the duck typing nature of the language. So you can add static verification without any "overhead" if you view it that way

Other than that, protocols have the advantage of totally decoupling things. They have the disadvantage of the fact that they are implicit, so things can satisfy protocols even if the owner of the type has not really thought things through.

[paper echo]

yes, like how PlayerCharacter has to specifically inherit from the AbstractCharacter in the first example. but it has nothing to inherit from in the latter example, as long as it implements the desired methods

[halcyon trail]

there's usually more rules that are implicit behind a protocol

and no way for say mypy to verify those implicit rules are being met

[paper echo]

also the abstract base class can implement common functionality, like the __init__ method

[halcyon trail]

People who prefer explicit ways of interface satisfaction would argue that it' simportant that people actually explicitly opt-in to satisfying these things

It's fair to note I think that protocols/structural sub-typing is far more rare in statically typed languages, compared to explicit sub-typing

You'd need to define some kind of special type that denotes an integer in a certain range

Not sure whether it's technically possible in python but almost certainly not really a great idea

there's languages with very powerful type systems that make such things more reasonable

Yeah, but what does it really get you

You'd have to really go all out and overload addition, multiplication, etc ikn specific ways to "propagate" this stuff

otherwise as soon as you multiple your IntRange(5, 10) by something or add it to something you'll be back to having an int

It would probably mostly be useful in the context of a dataclass that is designed to deserialize and verify some data

but I think in python it's more common to just use a regular int as the type and define a function that verifies the property that you want (including being in a certain range); things like attrs support this way of working for example

they're pretty similar in concept, I think it's just a question of the feature set

I'm more familiar with dataclasses vs attrs

dataclasses actually originated from attrs

my personal recommendation would actually be to never use dataclasses

just use attrs. Maybe if you really really really don't want a third party dependency but there's rarely a good reason for that in python.

I went with dataclasses and definitely regret it, though dataclasses is getting some new features (which have been in attrs forever) which will alleviate some (not all) pain points

pydantic is a big higher level maybe I think, it does offer some nice things out of the box which attrs doesn't

in particular pydantic comes with recursive to-from json, which can be extremely useful

yeah I know that bundle is popular with a lot of people, haven't used it so can't comment

But pydantic seems very nice generally. attrs is too.

I'd recommend against. If you want the "simpler" choice, I'd still just pick attrs.

[paper echo]

this is called a "refinement type" and generally it requires a very different type system altogether, usually with an SMT solver built into the type checker. it is a special case of "dependent types".

[halcyon trail]

Like, one place where I got really badly burned by dataclasses is the fact that dataclasses don't support keyword only init functions

the problem with this is that it means that all defaulted members have to follow all non-defaulted members

and this is a total disaster if you use inheritance in your dataclasses

attrs has had this feature for ages

[paper echo]

SMT solver: doesn't matter what it is, but you need it to figure out statically when x: int is or is not between 5 and 10
dependent types: when the type of a variable depends on values that are only known at runtime

https://www.python.org/dev/peps/pep-0647/ i believe you can use this to implement something like IntRange but i think there will be a lot of ways to leak around it

ctrl+f for my thoughts on it in this channel

from early in May

have fun typing.casting 🙂

[fallen slate]

typing.cast

typing.cast(typ, val)```
Cast a value to a type.

This returns the value unchanged. To the type checker this signals that the return value has the designated type, but at runtime we intentionally don’t check anything (we want this to be as fast as possible).

[halcyon trail]

I've only had to use typing.cast, incidentally, when I implemented my recursive from-json stuff for dataclasses

type checker just got confused

[paper echo]

no recursive types for json makes me sad

[halcyon trail]

i think you can, if you don't use mypy

I think pyright will handle it correctly

[paper echo]

ah ive heard that

[halcyon trail]

I had this discusion very recently with someone

Should probably look into switching, see if there's a downside

just lazy

and I haven't had too many issues with mypy outside of the json thing

[paper echo]

i do it

[peak spoke]

I mostly just use pycharm's builtin checker, whatever that is (which can handle recursive types). It definitely helps

[halcyon trail]

I'd say it's becoming pretty common practice, yes

A lot of people use mypy explicitly as part of their CI

(or one of the mypy alternatives like pyright)

[peak spoke]

I'm not that big of a fan of it being a part of the CI

[halcyon trail]

Can't really see any downside to it, tbqh

if you have an existing large codebase with many false positives, that you don't want to spend the time to clean up, sure

but for a new project I'd expect it to be pretty typical at this point

@unkempt rock one of the big reasons for things like dataclasses/attrs/pydantic, is that it lets you take a dynamic schema coming in (often json), and convert it at the "boundary" of your application, and do all the checking and verification of the schema at that point.

And then everywhere else, you just use the class instance that you got, and the static type checker will tell you if you make any mistake using it

10+ years ago in python land, people would pretty typically just pass around the actual dict, and even if they verified it in one place, it was easy to make mistakes downstream while accessing the dict

[peak spoke]

Without it being mandatory there you don't have to go out of your way to satisfy the checker. Any is an option but I'd say there's a bit more pressure to not use it in cases like that. If it's all local and done in reviews you can have hints that are incorrect from the PoV of a checker but make sense when an actual human reads them, or just completely ignore it in complex cases

[paper echo]

TypedDict also exists if you realllly can't use attrs/dataclasses for whatever reason, e.g. you're adding annotations to legacy code, or you're optimizing code in tight loops and don't want the overhead of creating short-lived class instances

[halcyon trail]

if you don't make it part of the CI, then in a decent sized team there will always be people who just ignore it completely, and then you open up their code in pycharm and now you have tons of errors

which presumably are "false positives" since the code worked

and now those false positives, you need to either clean up, or remember to ignore, and when you modify that file, you're less likely to get any benefit from new errors because there's already a gaggle of false positives

In an individual or very small team setting what you're saying is more reasonable but even in a moderate size team, it's going to massively reduce any benefit from type annotations

[peak spoke]

I do think that it should be checked outside of CI, but I guess whether that's worth it depends on the developers working on the project

[paper echo]

putting mypy in CI is a biig burden on a team that isn't used to using types

it's an organizational commitment

[halcyon trail]

Sure, if people aren't aware of it, then it can be difficult I suppose.

[paper echo]

or don't know how to use it effectively

or you're using libraries that do not play well with types

or you have highly polymorphic APIs

[halcyon trail]

there isn't really such a thing as a library that doesn't work well with types

[paper echo]

it's one thing to slap : int and : float in some places, it's another to start using overload and not abusing cast etc. etc.

[halcyon trail]

if a library isn't type annotated, then it will just not raise any errors for any types you get from the library, or pass into it

and you can still benefit from annotating your own stuff

Eh, like I said, python is gradually typed so it's very easy to "opt out" of more advanced use cases

If you really have relatively inexperienced programmers on your team (or maybe I should say "lack of breadth") who have never in their lives used a statically typed language, ok

[peak spoke]

You have got libraries that use annotations for their own non type checking purposes

[halcyon trail]

Not sure what that example has to do with things?

Usually such a library would play very nicely with types

pydantic is actually an example of a library that uses annotations for things other than type checking

[flat gazelle]

typing.Annotated is a very recent feature

and pydantic has issues once you start doing things like conint

well, mypy does

[halcyon trail]

what's conint?

[peak spoke]

If you use that alongside type hinting you'll have to explicitly set the checker ignore a lot of stuff. For example libs where you hint with a class and then it uses that as some sort of validation and then returns an unrelated type

[paper echo]

good point, but you do end up with lots and lots of "holes" where the types just fall through and it's almost worse than not having any annotations, because you get a false sense of security and can't easily determine where the "typing boundary" is

i suppose it's the same as adding tests to a legacy codebase, you just have to know

[halcyon trail]

I don't see how it's worse, having a false sense of security based on static typing in a gradually type dlanguage is simply a bad idea regardless

[flat gazelle]

class Model(BaseModel):
    x: conint(lt=1000)

[halcyon trail]

Unfamiliar with conint

@peak spoke can you give an example?

[flat gazelle]

it is int, but constrained, in this case must be less than 1000

it is a very useful pydantic feature

[peak spoke]

typing.Annotated is nice but yeah I haven't encountered a lib that supports it from the ones I use. I think there are some improvements in 3.10 for fetching annotations which aren't type hints

[flat gazelle]

but mypy will yell at you for using it since it is not a type hint

[paper echo]

@flat gazelle it knows about comparison operators > < etc? like how mypy knows about isinstance and is None

[flat gazelle]

pydantic does runtime validation, not static checking

[paper echo]

oh, this is runtime

sorry i misread "pydantic" as "pyright" 😆

yeah i do not like this at all

fake refinement types

[halcyon trail]

@flat gazelle I'm sure there are mypy compatible ways to do the same thing. For starters, given the python type system, not sure how useful it is to put the constraint into the actual type

[flat gazelle]

it isn't a type at all

[paper echo]

right, but it's abusing annotations for things that aren't types

[halcyon trail]

right

so just do it properly and then you're fine?

[flat gazelle]

annotations were originally not meant just for types

[peak spoke]

Yeah I wouldn't call it abuse

[flat gazelle]

though these days, that is a bit of an iffy statement

[halcyon trail]

What you're saying is that if you use a library in a way that's not compatible with static typing, then it's not compatible with static typing, It seems

you have the option to use it in a way that's compatible with static typing

[paper echo]

https://www.python.org/dev/peps/pep-3107/#rationale

This PEP introduces a syntax for adding arbitrary metadata annotations to Python functions [1].
interesting, i didn't realize this

[halcyon trail]

and it's not any harder to train people to use it in that way, then to use it in the other way

[flat gazelle]

there is a new thing which allows

class Model:
    a: Annotated[int, conint(ge=0, lt=1000)]

[paper echo]

def conint(n) -> Type[int]:
    return int

i assume this wouldn't make mypy happy?

[flat gazelle]

no, function calls are not valid in type hints IIRC

[paper echo]

i figured not

[halcyon trail]

The annotated thing isn't very different from something like

class Model:
    a: int = field(verifier=conint(ge=0, lt=1000))

which is already how attrs can do these things, I suspect pydantic offers something similar

[flat gazelle]

yeah, but pydantic does it with annotations

this is how pydantic is supposed to be used, you use annotations to describe the schema and pydantic can validate it from that. It could be done with just assignment as well in a different library

[flat gazelle]

also, how come mypy doesn't yell at this with field is not int

[halcyon trail]

dataclasses and attrs have special modules in mypy and other type checkers to handle all these things correctly

[flat gazelle]

ah

[grave jolt]

I think field just returns an Any

at least this is how it works in pyright

[halcyon trail]

Pretty sure it complains if the default is the wrong type

[paper echo]

side note: i attempted to figure out how to implement a kind of multivalidator in attrs, the internals are a lot more complicated than i remember

so many layers of indirection, very few comments

[halcyon trail]

I don't know what all the combinations that pydantic does/doesn't support, but pydantic does support using assignment

class Foo(BaseModel):
    x: int = Field(gt=1000)

This works exactly the way you'd want

[flat gazelle]

I am generally not fond of writing more code just to make tooling happy, but ye, it is not unsolvable I see

[halcyon trail]

I don't think it's about writing more code to make tooling happy here. if you want the tooling to know the type, then you need ot tell it the type.

the type is int

constrained int isn't part of the type really, both for more fundamental reasons, and even more basic it's not even checked by pydantic after init

Pythons type system isn't really design to support this

[paper echo]

this is mostly the same as the attrs version, although attrs doesn't have the convenient gt kwarg

[halcyon trail]

Yeah, that's what I'd expect. Though personally I'd prefer something more explicit.

[paper echo]

def greater_than(y):
    def validator(obj, attribute, value):
        if value <= y:
            ValueError(f'{attribute.name} must be greater than or equal to {y}.')
    return validator

@attr.s
class Foo:
    x: int = attr.ib(validator=greater_than(0))
    y: int = attr.ib(validator=greater_than(0))

[halcyon trail]

right

I actually prefer that personally, but it doesn't matter that much

as long as the type is annotated correctly, that's really the main thing

what I want to suggest to attrs is "class level" validators/converters

Meaning, a validator/converter that is called for every attribute

[paper echo]

yep, right now you have to use __attrs_post_init__ for custom validation

[halcyon trail]

this is still per-attribute validation

but say you want to validate that the value you receive is really an instance of the static type

pydantic does this automatically, in attrs I don't know any way to do this but to pass a validator for every single field

[paper echo]

yeah, attrs doesn't have any runtime type checking with annotations

however you can generate a marshmallow "schema" that will infer such

[halcyon trail]

Right, but if you had a "class level" validator it would be trivial

@attr.define(validator=type_validator)
class Foo:
    x: int
    y: int

So whenever something in Foo gets set, it also looks to see if there's a class level validator, if there is then it passes the Field being set and the value.

def type_validator(field: Field, value):
    if not isinstance(value, field.type):
        raise ValueError("...")

Very simple solution

[paper echo]

need to be careful with generic/subscripted types but yeah

[halcyon trail]

yeah, that's true, pydantic would still have to deal with that in some sense though. attrs could provide a reasonable instance checker that does ok with that sort of thing.

It just seems like a nice route to provide that kind of functionality easily, but also in a general, extensible way

you could do the same thing for converters. The reasoning there is that its very common that you don't want conversion logic "per field", but rather simply "per type"

So now imagine you have some kind of object that does conversions for different types, and lets you also register new types to do conversions

[paper echo]

i believe the pydantic docs state that collection and generic types will not be "deeply checked"

[halcyon trail]

You can pass that it in as "class level converter", instead of defining converters for every field

Right, that makes sense, attrs could provide something comparable

[paper echo]

it could definitely be nice

there is an argument to be made that validation doesn't belong on the attrs class at all

and that you should have an api boundary that does validation, and raw data is "deserialized" at the boundary

class Thing:
    def __init__(self, ...):
        # Just assign attributes, no checking of anything

    @classmethod
    def from_external_data(self, raw_data):
        # Perform validation

that's basically a performance optimization though

you'd be relying on your static knowledge of the code to enforce that you don't accidentally break any of the validation

yet another case where dependent types could be very helpful, if the developer ergonomics can be improved

[halcyon trail]

Yeah, probably in some cases the validation belongs separately but there's the convenience factor in many cases, I think, and in some cases I think it's reasoanble for it to belong there

I admit I don't have much real experience with dependent types, aside from the little that is possible in C++ with its non-type template parmaeters (which is still more than most languages can do)

waiting for it to percolate into a more mainstream language which I suspect will be a while

[potent pendant]

hello guys, do you think codewars.com is a good website to learn coding? I currently have no ideas for what to do next and i think that helps my algorithmic thinking 🙂

[paper echo]

@halcyon trail idris 2 is decently usable, the big problem right now is lack of library support for anything serious

they are working on a package manager right now

but dependent types in general pose some fundamental developer ergonomics problems

for example, the fixed-size list type (misleadingly called Vect) works mostly by isomorphism with the natural numbers, in that a chain of 3 value cons'ed together + Nil at the end is "the same thing as" the natural number 3

clever, but then in day-to-day programming you all of a sudden have to care about things like manually asserting that addition is commutative

take is a pretty straightforward function, which takes the first n elements of a sequence

so let's say you wanted to use take to write a function that takes all but the last elements of a sequence

butLast : Vect (S k) t -> Vect k t
butLast xs = take ((length xs) - 1) xs

this won't compile, and unless you really know what you're doing you probably won't be able to figure out why

S k is "the natural number after k"

[halcyon trail]

I guess it's always been confusing to me how dependent types inject runtime values back into the static type system

[paper echo]

there are known algorithms for it but it's all very abstract and mathy

curry-howard correspondence and stuff

https://en.wikipedia.org/wiki/Dependent_type

it's all kind of magic to me

https://donaldpinckney.com/idris/2019/06/25/idris-intro.html nice example here

[halcyon trail]

yeah, very magical to me as well

it's very practical and useful in the context of C++

they're not true dependent types, but you can inject compile time values into types, originally just integral types but this has been expanded

so for example you could do things like:

template <class T, size_t N1, size_t N2>
std::array<T, N1 + N2> concat(const std::array<T, N1>& a1, const std::array<T, N2>& a2) { ... }

[paper echo]

that makes sense, although imo it's more like macro expansion than anything

[halcyon trail]

why do you say that?

[paper echo]

wait, are a1 and a2 known at compile time?

[halcyon trail]

the values aren't, no

[paper echo]

but the sizes are

[halcyon trail]

the sizes do have to be known at compile time

yes

[paper echo]

so what does this do? it expands into an implementation of concat?

[halcyon trail]

well not I didn't implement it, that's just the signature 🙂

[bleak isle]

i have an advanced question :

[halcyon trail]

template <class T, size_t N1, size_t N2>
std::array<T, N1 + N2> concat(const std::array<T, N1>& a1, const std::array<T, N2>& a2) {
    std::array<T, N1+N2> result;
    for (std::size_t i = 0; i != N1; ++i) result[i] = a1[i];
    for (std::size_t i = 0; i != N2; ++i) result[N1 + i] = a2[i];
    return result;
}

Something like that

[paper echo]

does this get "inlined" basically?

[halcyon trail]

that's an implementation detail

but that has nothing to do with whether it's a macro, if that's where you are going

[paper echo]

what's the advantage of doing this over a function?

[halcyon trail]

this is a function?

it's a function template

or a generic function, as it would be called in other languages

[paper echo]

ohhhhh

huh

so a specialized version of this function will be generated depending on T, N1, and N2 at the usage site?

[halcyon trail]

in all C++ implementations, yes

[paper echo]

(if only semantically)

[halcyon trail]

semantically, yes

and in terms of actual code, also yes, in practice

but regardless of those issues, the static typing guarantees are there

auto x1 = std::array<int, 3>{1,2,3};
auto x2 = std::array<int, 3>{1,2,3};
auto result = std::array<int, 5>{};

result = concat(x1, x2); // compilation error

[paper echo]

makes sense

dependent types in some ways are overpowered for what most people need

the problem is when you want to "convert" an array of unknown size to be used with a function that expects an array of known size

how does c++ handle that? or does that just not work

[halcyon trail]

it just doesn't work

there's no such thing as a std::array of unknown size, std::array always has known size

in the context of C++ this makes a lot of sense, keep in mind that say std::array, one of the main selling points is that there's no heap allocation, it's all just created on the stack

you can't do that if the size isn't known

[paper echo]

i see

[halcyon trail]

A lot of the motivaton behind these types, in C++ specifically, is performance related. In the context of a performance sensitive language, they're super useful (and I'm looking forward to decent support for this in Rust)

I agree with you though that in the general case, I think this is past the sweet spot for type system power

I think the typical Java/C#/Kotlin/Swift, is probably roughly right (not that they're perfectly designed, but the amount of power vs complexity of the type system)

i.e. generics, but not too many more crazy features

[paper echo]

yeah so this is where idris gets funky - you can create a Vect (known size) from a List, as long as the compiler can figure out statically what the length of the list is

[halcyon trail]

right, but it seems in general you can't

and if you can you can probably just tell the compiler explicitly 🙂 So it's not that that exciting

[paper echo]

if you want to read an unknown-length Vect, you would return a dependent pair of the length as well as the vect itself

[halcyon trail]

hmm but the Vec itself always needs to encode its length I thought

well, okay, I guess obviously in this type system, this sort of thing is possible

it sounds very neat intellectually, the question is just what the ROI is like for making the type system/language so much more complex

[paper echo]

yeah, a dependent pair is a runtime value coupled with a type which uses that value

[halcyon trail]

gotcha

[paper echo]

so far in my amateur plunking around, the ROI is limited for something like a web server, but if you're designing an application with very well-defined state changes it could be very useful

[halcyon trail]

The problem is that in most real world applications, trying to track everything via the type system ends up being more trouble than it's worth

it becomes extremely complex quickly, and the errors that it catches are not the most interesting ones

[paper echo]

exactly

[halcyon trail]

If you were to say "very well defined state changes", my first thought would not be a super fancy type system, but rather a framework for creating finite state machines declaratively

which can be a very very clean way to handle this, takes advantage of types to an extent, and importantly it's very easy for humans to read, and debug, etc

Many languages have very nice libraries for this

[paper echo]

readVect : IO (len ** Vect len String)
readVect = do x <- getLine
              if (x == "")
                then pure (_ ** [])
                else do (_ ** xs) <- readVect pure (_ ** x :: xs)

interestingly idris can actually infer the length so you don't have to manually keep track of it in an accumulator

the ** means a dependent pair

[halcyon trail]

I think one of the consistent problems in these fancy type system languages is that most of the people using them in the real world, are pretty smart, they're well above the average

[paper echo]

that is 100% a consistent problem, as a "normie" attempting to use idris for "normie" things i see it a lot

i've been wanting to start writing up some blog posts that attempt to explain the advanced concepts i'm slowly learning in less-advanced language, with less-contrived examples. some of the examples in the docs are truly "big brain" material and not practical or useful to stupid people like me

[halcyon trail]

From an engineering point of view, designing a language for smart programmers is much easier. It's like designing a machine, where all the gears are cut with super low tolerance. Easy to design, but it's bad engineering.

My dad's a mechanical engineer who was in manufacturing for decades, and he'd always tell me that Ikea is peak engineering

low cost, garbage materials, assembled by idiots with terrible tools, and yet, still works pretty well. Those are your guiding principles for PL design right there 😛

[grave jolt]

I am extremely confused by IKEA's instructions.

They should test them on idiots with terrible tools like myself.

[halcyon trail]

Really? I think they're super clear. I wish most programming API docs were as good as Ikea instructions.

[bleak isle]

can i post an advaned question/ thx

i tried to convert code from c++ to python but i dont have the same output : can someone h2p?

[halcyon trail]

to be completely honest this is not advanced

did you try posting in a help channel?

[bleak isle]

xD

lmao

yeah i tried on post channels

sum1? for h2p?

[stone field]

I don't think this is true at all because even the most "advanced" type system can be broken down into simple axioms that 5 year olds can understand

vanity is not the same as intelligence

[flat gazelle]

proofs are hard, and dependent typing is full of them

well, any type system is, but for more usual type systems the proofs are really simple

[halcyon trail]

they can be broken down but in practice they are still hard to understand

I can make such a statement about anything

Quantum Field Theory can be broken down into simple axioms that 5 years olds can understand, if you keep breaking long enough

Yet, it's a hard topic

I don't think that advanced type systems are that hard, but they are hard enough, they require substantial effort, and I see day to day lots of intelligent people not even properly understanding much simpler type systems

and sometimes these people are smart, they just need to know many things, the programming language(s) is just one of them, sometimes even a relatively small part

[spark magnet]

Did you get it fixed?

[bleak isle]

nope 😢

[spark magnet]

range(4, 0, -1) --> 4, 3, 2, 1

[halcyon trail]

like I said before, this isn't advanced

and I asked if you had sought help in one of the help channels

to be clear, I'm not trying to be combative, even if you had responded and said something like "I tried in help, nobody replied, or somebody in help told me to go here", I probably would have responded

[spark magnet]

@bleak isle sorry, i shouldn't have started this up.

[bleak isle]

i did and more than once

@halcyon trail why can't you help then if is not advanced???

[spark magnet]

@bleak isle it's not right for this channel. A help channel would be best

[grave jolt]

Yes, you should see #❓｜how-to-get-help

[astral gazelle]

maybe this channel should be considered for a rename

[bleak isle]

i know how to use help channels

[grave jolt]

This channel isn't for "advanced" questions, it's for discussing Python in a meta way. Its implementation, future, new features etc.

[bleak isle]

ok i got your point

sorry

[grave jolt]

We have tried many names, this seems the best one so far. It's really unintuitive, but it has fewer issues than other names.

[bleak isle]

thanks it is clear now

[halcyon trail]

I have never heard before that this isn't for "advanced" questions before today

"advanced language concepts" is listed in the description above, so I think e.g. a question about how to use metaclasses properly would seem to be on topic

[paper echo]

c.f. my butLast example above

won't compile

butLast : Vect (S k) t -> Vect k t
butLast xs = take ((length xs) - 1) xs

one "right"(-ish) version

butLast : {k : _} -> Vect (S k) t -> Vect k t
butLast xs = take {m = 1} k (rewrite plusCommutative k 1 in xs)

mathematical "internals" leak everywhere

this version is nice of course, still linear-time so it's not any worse than take - but even recognizing why take ((length xs) - 1) doesn't work and deciding to switch to this version takes some knowledge and understanding

butLast : Vect (S k) t -> Vect k t
butLast (_ :: Nil) = []
butLast (x1 :: xs@(x2 :: xs')) = x1 :: (butLast xs)

[halcyon trail]

there's a pretty good pdf about the sieve of eratosthenes in haskell

and how when people proficient in Haskell provide the "look FP is cool" implementation of the sieve, it's actually not the sieve at all, and misses the point of the sieve, and has terrible runtime/memory characteristics

it's 10 pages long

[paper echo]

i'd love to read that

[halcyon trail]

https://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf

[paper echo]

thanks!

[gleaming rover]

I think the syntax makes a difference too

it’s super weird for most people

[paper echo]

For me the syntax is actually pretty nice, but I already know a bit of haskell and ocaml

[paper echo]

But I agree, I think being able to write a basic for a loop is valuable, even if you can't mutate anything without a monad or algebraic effect handler

[static bluff]

I'm so glad I switched to Pycharm

[slender oasis]

havent used pycharm, what made the change so glad ?, from vscode?

[static bluff]

From sublime

[unkempt rock]

makes lots of sense

lmao

[static bluff]

I mean, sublime isn't exactly a bad program

But its like comparing a bicycle to a jeep

[raven ridge]

This conversation would be a much better fit in #editors-ides

[blissful comet]

Would you write python that looks like this?

https://twitter.com/arundsharma/status/1407229297392115713
Context: https://github.com/adsharma/py2many/issues/322

Arun Sharma (@arundsharma)

Poll: What language is this?

Twitter · Jun 22, 2021, 6:49 AM

[eager trail]

@halcyon trail @finite sparrow re the discussion on logging yesterday, for whatever reason, in testing today, it seems that I have to specify a logger when instantiating an instance of a class to get it to run on aws lambda.

What you guys suggested about a top level logger worked on my local machine but didn't when I tried to run it as an aws lambda function, so I'm back to instance logging, but the top level package logger worked for importing free functions, so I have a solution that works for us, even if its a bit odd

[mossy spruce]

Can anyone tell me why this server has help and a fruit name channels, there almost 70% of these channels in this server. why??

[dusty oak]

bot automated help system.
those channels are sort of auto generated, and available in #691405807388196926
anyone get get them busy with their question, which will make them auto transfered to #696958401460043776 until the question is resolved

I guess anti flood system for help conversations

[mossy spruce]

k

[dusty oak]

It could be answered more shortly with #❓｜how-to-get-help

[mossy spruce]

k

ok

[vital tide]

I just made my own function that converts strings to ints without using the int function or anything similar

!e ```py
def str_to_int(string):
n = 0
for i, place in enumerate(string[::-1]):
n += "0123456789".index(place)* 10**i
return n

print((str_to_int("25")))

[fallen slate]

@vital tide :white_check_mark: Your eval job has completed with return code 0.

25

[vital tide]

!e ```py
def str_to_int(string):
return sum(("0123456789".index(place)* 10**i for i,place in enumerate(string[::-1])))

print((str_to_int("25")))

[fallen slate]

@vital tide :white_check_mark: Your eval job has completed with return code 0.

25

[vital tide]

!e ```py
def str_to_int(string):
n = 0
for i, place in enumerate(string[::-1]):
n += (ord(place)-48)* 10**i
return n

print((str_to_int("25")))

[fallen slate]

@vital tide :white_check_mark: Your eval job has completed with return code 0.

25

[vital tide]

Is this how it works (in theory) under the hood? I know python uses C for stuff like that, but is this how it is implemented there?

[lament sinew]

i assume it just uses atol or something similar under the hood, which is closer to the third one.

[peak spoke]

it is a bit more complicated as it does arbitrary bases (within limits) https://github.com/python/cpython/blob/145bf269df3530176f6ebeab1324890ef7070bf8/Objects/longobject.c#L2210-L2215

[fallen slate]

Objects/longobject.c lines 2210 to 2215

Viewing the input as a sequence <c0, c1, ..., c_n-1> of digits in base
convmultmax_base[base], the result is "simply"

   (((c0*B + c1)*B + c2)*B + c3)*B + ... ))) + c_n-1

where B = convmultmax_base[base].```

[halcyon trail]

@eager trail what do you mean by "it didn't work"

[eager trail]

Namespace error that occurred on aws but not on my local machine

I have no idea why

Same version of python and everything

[halcyon trail]

do you have the exact error?

[eager trail]

its the usual NameError: name 'logger' is not defined

just tested the way i implemented it on my home computer, definitely works. very odd

[eager trail]

that's with a top level logger in the file with the class that is throwing the NameError

[halcyon trail]

I suspect that something isn't executing the code you think it is

[eager trail]

yup... >.<

[bleak isle]

Hey folks, do you know websites where I can practice my python skills

To solve exercices

[visual shadow]

perhaps ask that in #python-discussion , this is not the right room for this.

[unborn basalt]

Hello, I am Mehmet from Turkey. I am a second year mechanical engineering student. I know python at intermediate level. I want to improve myself. If there are joint projects, I would love to participate.

[sand goblet]

If they make a JIT compiler for python like they’re thinking of doing, how much performance difference will that make? Will it be able to compete with Java, or will it be more like giving user defined functions the same speed as built-in functions?

I think I found something that answers it: https://www.theregister.com/2021/05/13/guido_van_rossum_cpython_3_11/
It seems like it’s saying it will be like having the speed of built-in functions, a 5x increase.

Would it even be possible to JIT compile to primitive C datatypes in a language like python, or is that one of those impossible things like finding an infinite loop in any program?

[halcyon trail]

i mean many things are possible

it's really more a matter of man hours

being dynamically typed puts python at a disadvantage

and the JVM already has an unimaginable number of man hours into optimization that python will not come close to for decades

[sand goblet]

So it’s probably possible, but it would be really really complicated?

[halcyon trail]

So I'm skeptical python will be giving performance comparable to Java anytime soon

Many things are possible. Definitely being dynamically typed puts you in a worse starting point than being statically typed, but if it's a JIT then ultimately everything depends on runtime characteristics, so you could reach the same performance in principle (I think)

I'm not sure if there is any JIT optimization that is fundamentally impossible for a dynamic language but possible for a static language. Obviously though with JITs, there's a big trade-off between the speedup it's giving you and how hard it's working, since the JIT is running during runtime, of course.

If you're dynamically typed, you know much less prior to running the code, so you need to work harder at runtime instead, and given that there's a limit to how hard you want the JIT to work at runtime, that might push back the performance you target

that said Julia is dynamically typed and does JIT'ing with llvm, seems to be pretty fast, and allegedly they are dealing with the slow jit startup issues

so, who knows

The thing though is that Julia is "piggy backing" on LLVM, which again, is an unimaginable number of man hours into optimization

LLVM and the JVM are probably the largest efforts in optimization ever put forth by humanity, so in this day and age trying to do your own thing in that regard (or having to because your design isn't compatible with those tools), is putting yourself at a relatively big disadvantage

(well, along with gcc, but that's targeting C/C++ more directly, whereas LLVM/JVM are easier to develop new languages for and take advantage of optimizations)

[flat gazelle]

Julia is always fully compiled into native code, unlike most JITed languages, just well, at runtime afaik

python is extremely unlikely to ever reach JVM levels of perf, simply because there is dynamism in python possible which java does not expose. The plan for python speedups rn is afaik not a JIT, but specializing code at runtime for specific types rather than always being generic

similar to what truffle does, just without all the JIT and compilation stuff

[halcyon trail]

neat stuff

[radiant fulcrum]

Luajit is a good example of the performance thats possible with dynamic languages, but equally lua as a language is alot simpler than python in areas not to mention the interpreter side being written in assembler, so its a trade between performance and amount of hours developers can dedicate to develop the runtime

[raven ridge]

My impression is that many CPython core devs can't write C. I imagine many more can't write even one flavor of assembly 😄

[halcyon trail]

I mean writing assembly well these days is a very rare skill

lots of people can read it, write something basic in it, or look up something they need to in order to write a function with inline assembly they can use in C

I don't think the interpreter being written in assembler is any kind of significant advantage; or at least I wouldn't claim it without a lot more information. Well written C or C++ is going to compile to assembly roughly as good as many (most?) experts are going to write

[radiant fulcrum]

In LuaJIT it's used for alot of hyper optimisations

Still amazes me that it was basically single handily written by one man on a mission

[halcyon trail]

In some cases you can hyper-optimize, sure, but writing the whole thing in assembler you can also miss opportunities for optimization that you'd get, particularly in C++

Most common approach in industry for ultra-fast stuff these days is mostly C++, with select things being written in assembly after benchmarking to see that the thing in question matters, and that the assembly is sub-optimal

[flat gazelle]

for computation, you also often see fortran in the scientific field as well

[static bluff]

o.O O.O

I am so freaking close to figuring this out

[paper echo]

https://github.com/Legrandin/pycryptodome/blob/master/lib/Crypto/Cipher/AES.pyi
https://github.com/Legrandin/pycryptodome/blob/master/lib/Crypto/Cipher/_mode_cbc.pyi
is it just me or are these type annotations kind of broken? AES.new should be using typing.overload, and CbcMode should inherit from Protocol, not object. am i making a mistake, or should i submit a patch?

[paper echo]

https://github.com/Legrandin/pycryptodome/blob/master/lib/Crypto/Cipher/_mode_cbc.py#L63
https://github.com/Legrandin/pycryptodome/blob/master/lib/Crypto/Cipher/_mode_cbc.pyi#L9
i see, it's probably not supposed to be a Protocol after all

[fallen slate]

lib/Crypto/Cipher/_mode_cbc.py line 63

class CbcMode(object):```
`lib/Crypto/Cipher/_mode_cbc.pyi` line 9
```pyi
class CbcMode(object):```

[paper echo]

https://github.com/python/cpython/blob/v3.9.4/Lib/_collections_abc.py#L1033-L1043
https://docs.python.org/3/library/typing.html#typing.ByteString

This type represents the types bytes, bytearray, and memoryview of byte sequences.
why don't they have ByteString.register(memoryview)?

[fallen slate]

Lib/_collections_abc.py lines 1033 to 1043

class ByteString(Sequence):

    """This unifies bytes and bytearray.

    XXX Should add all their methods.
    """

    __slots__ = ()

ByteString.register(bytes)
ByteString.register(bytearray)```

[raven ridge]

and memoryview of byte sequences
Not every memoryview is a ByteString, only a memoryview created from a ByteString is.

at least, according to that definition.

[paper echo]

ah

huh

but memoryview isn't generic

rather: how does a type checker know that memoryview-of-bytes is a valid ByteString when memoryview itself isn't generic?

i am looking at this https://github.com/Legrandin/pycryptodome/blob/master/lib/Crypto/Cipher/AES.pyi#L29 and wondering if it can't just be Buffer = typing.ByteString

[fallen slate]

lib/Crypto/Cipher/AES.pyi line 29

Buffer = Union[bytes, bytearray, memoryview]```

[raven ridge]

memoryviews aren't necessarily contiguous in memory - you'd need to check for that with something like mv.c_contiguous

so - I think typing just doesn't help very much with memoryview - it wraps the Python buffer interface, which is itself dynamic, and so just knowing that something is a memoryview doesn't tell you what operations it supports, or what shape it has, or...

[paper echo]

i guess what i'm wondering is, how does this pass typechecking

mv: ByteString = memoryview(b'asdf')

if they don't use ByteString.register(memoryview)?

[raven ridge]

I... dunno.

[paper echo]

spooky

[eager trail]

You're right, I goofed

[magic python]

I'm wondering about this SO post: https://stackoverflow.com/questions/37835179/how-can-i-specify-the-function-type-in-my-type-hints

Where the suggested typehint for a function is to use Callable, i'm wondering if this is generally preferable to creating something custom from Protocol ?

For example, if I have:

def f(x : list[int]) -> int: return sum(x)

then I might have some:

def function(sum_func : Callable[ ... , int], values : list[int]) -> int:
    return sum_func(values)

This could (i think?) also be written as:

class SumFunction(Protocol):
    def __call__(self, *, x: list[int]) -> int:
        ...

def function(sum_func : SumFunction, values : list[int]) -> int:
    return sum_func(values)

I'm not sure why one would use Callable over writing a custom class, or visa versa

(please ping me if you respond 🙂 )

[flat gazelle]

Callable is less verbose, that's probably about it.

you can often get a lot of different callable types, so having to make a class for each would be quite painful

see pretty much any java codebase using a lot of higher order functions

[magic python]

i've never seen java (anything other than python / some R actually), fair enough though.

What i'm currently unsure of is the "best" way to typehint things which have a consistent meaning, for example - we have some dictionary called xyz that always has the form dict[ str, dict[ float | int, str]], which makes me feel there should be some central "type" somewhere for it

[flat gazelle]

you can make an alias

Config = dict[ str, dict[ float | int, str]]

[magic python]

🤔 OK - so with this setup would I have some project_types.py and then use as

from project_types import Xyz


def f(xyz : Xyz ... )

[flat gazelle]

yeah

though I am not fond of central type modules

[magic python]

where project_types.py has:

Xyz = dict[ str, dict[float | int, str]]

ok cool, oh, why not?

i thought maybe NewType was what i was meant to use but couldn't seem to make good sense of it

i don't understand why a central type module would be an issue for something that was used across multiple modules though, seems that it could be documented in one place?

[flat gazelle]

NewType is more for types that have the same values, but are incompatible, for example Radians vs Degrees

I don't like central type modules because they spread a single type across multiple files, which means that in order to change it, you have to just across many files

but there are definitely cases for using them

[magic python]

hm fair, yeh hopefully this is one 😅 as its used in multiple places.

I was unsure whether one would just create a class for this, so instead of having the alias Xyz = dict[str, dict[float | int, str]] we'd have a class, will just use the alias for now though i think - thanks

[flat gazelle]

yeah, it is a fine solution

[paper echo]

IMO the main use case for a callable protocol vs Callable is when you have a complicated call signature that you can't encode in Callable

[halcyon trail]

what exactly do you mean by that?

like keyword only arguments and such?

[verbal escarp]

out of curiousity, is https://macropy3.readthedocs.io/en/latest/index.html well known and used around here?

[grave jolt]

keyword arguments or overloads

or maybe you want to explicitly name the arguments, as Callable[[int, str, int, bool], None] isn't very useful

[halcyon trail]

well, naming the arguments doesn't buy you that much because it's not enforced it seems, unless they're keyword only

from typing import Protocol


class Runner(Protocol):
    def run(self, foo):
        ...


class Foo:
    def run(self, bar):
        pass


def bar(r: Runner):
    r.run(foo=0)


bar(Foo())

this program type checks in mypy, for example

i guess you could argue this is a bug in mypy

wow, the type checking in mypy for ABC's is terrible

from abc import ABC, abstractmethod


class Runner(ABC):
    @abstractmethod
    def run(self, foo):
        ...


class Foo(Runner):
    def run(self):
        pass


def bar(r: Runner):
    r.run(foo=0)


bar(Foo())

this type checks 😦

i'm curious now to try this with pyright, but when I tried to pip install it, it depends on npm, and then gives me an error

ah you have to fully annotate all the the functions to get the error

but, still, this type checks:

class Runner(ABC):
    @abstractmethod
    def run(self, foo: int) -> int:
        ...


class Foo(Runner):
    def run(self, bar: int) -> int:
        pass

[spark magnet]

sometimes it seems like this channel should be renamed #metaclasses-and-mypy-puzzles 🙂

[magic python]

is an alias considered a constant? If i have:

Something = dict[str, float]

at the top of a module, should it be in uppercase?

[halcyon trail]

hah. well "advanced" python 🙂

An alias is probably most similar to a class, so I'd write it in camel case, not all upper case

[sacred tinsel]

I use camel yea, it makes the most sense to me

[true ridge]

It is a type, so I'd say kinda makes sense to name it with the type convention (same applies to creating a namedtuple for example).

[magic python]

fair, not too sure what the type convention is, List, or list?

[spice pecan]

Built-ins and certain types that act more like functions can start with a lowercase letter, but in general you'd start with a capital

[neon tartan]

It's still List because 3.10 is not out

But I think the convention will change once 3.10 is out

[native flame]

list works in 3.9

as in, with generics

[neon tartan]

I get a lot of errors when I do list[] thoough

[charred wagon]

Errors from what?

[neon tartan]

As in, my linter spits out a lot of red lines

[native flame]

just list[] probably isn't valid

does something like list[int] also error

[magic python]

how on earth is this not valid:

y = { ('a', 'b') :  {(1, 1): 2, (2, 3): 1} }
def h( k : Dict[ Tuple[str, str], Dict[ Tuple[Union[float, int], Union[float, int]], int] ] ):
    print(k)
    
h(y)

fails with:

main.py:15: error: Argument 1 to "h" has incompatible type "Dict[Tuple[str, str], Dict[Tuple[int, int], int]]"; expected "Dict[Tuple[str, str], Dict[Tuple[Union[float, int], Union[float, int]], int]]"
Found 1 error in 1 file (checked 1 source file)

but what is the point of Union?

i thought the whole point was that I could give int or float

[charred wagon]

Is this the same issue as one I discussed a few weeks ago with someone else? https://github.com/microsoft/pyright/issues/646

[magic python]

idk but mypy has wasted nearly a whole day for me if this is some other daft thing it's broken on

[charred wagon]

Never mind, this does not seem related

[neon tartan]

seems mysterious, I get the same error you do

[grave jolt]

That's because dicts are invariant.

I suppose

[charred wagon]

I am suspecting it's related to that

But haven't looked into it

[magic python]

how is a dict invariant? dicts can change can't they? Or not

[grave jolt]

invariant as in: not covariant and not contravariant

[charred wagon]

It works fine when it is just a tuple https://mypy-play.net/?mypy=latest&python=3.9&gist=29fce04f72df2c648b31f809928790a9

[magic python]

covariance is something i might between variables in a model, ive no idea what they mean here 😬

[flat gazelle]

essentially, k could add a float to the dict, which is invalid in a dict that only has int values

[magic python]

I have: https://mypy.readthedocs.io/en/stable/generics.html#variance-of-generic-types to read through re the covariant point

[charred wagon]

This is a decent explanation if you're unfamiliar https://blog.daftcode.pl/covariance-contravariance-and-invariance-the-ultimate-python-guide-8fabc0c24278

[flat gazelle]

you can use Mapping, which is not mutable

[magic python]

i want it to be float | int though

[halcyon trail]

Honestly you should technically be using Mapping much more than List

sorry, I mean, Sequence

You should be using Mapping and Sequence more than Dict/List

[magic python]

yea but that's not what i have, and i'm using a dict not a list or a sequence ( i think ? )

[halcyon trail]

is what I'm trying to say

(badly)

Unfortunately I haven't been following this, I admit for no real reason

[charred wagon]

I know. I'm demonstrating that it works when not in a dict, meaning that the dict is the problem

[halcyon trail]

should probably start

[flat gazelle]

consider a simpler example

N = int | float
a: dict[str, int] = {'a': 5}
def fun():
    return range(10)[a['a']]
def mutate(b: dict[str, N]):
    b['a'] = 3.14
fun()
mutate(a)
fun()

[magic python]

doesn't feel as natural i guess (that's my excuse - and idk how to use mapping)

[flat gazelle]

would you agree that this is a type error?

[halcyon trail]

You use them the exact same way

the only difference is that Sequence/Mapping refer to interfaces, rather than a specific type

and in particular they are non-mutable interfaces

[grave jolt]

TL;DR:

Tuples are covariant. It means that if A is a subtype of B, then tuple[A, ...] is a subtype of tuple[B, ...]

Functions are contravariant in their argument types. It means that if A is a subtype of B, then Callable[[B], int] is a subtype of Callable[[A], int].

Lists are invariant: If A is a subtype of B, list[A] and list[B] don't have any subtype relation. Why?

def f(things: list[Union[int, str]]):
    things.append("foo")

xs = [1, 2, 3, 4] # list[int]
f(xs) # oops, xs isn't really list[int] anymore!

Same for dicts:

def f(things: dict[str, Union[int, str]]):
    things["foo"] = "bar"

xs = {"fizz": 42} # dict[str, int]
f(xs)  # oops, xs isn't really dict[str, int] anymore!

[magic python]

https://mypy-play.net/?mypy=latest&python=3.9&gist=c78bb55eed566607c1821a2e0fab5b73 🤔

[halcyon trail]

on average when you have a function in python that you expect to pass a list, you're usually not mutating that list (I hope). If you annotate the argument as Sequence[whatever] instead of List[whatever], you'll still be able to pass your list just fine, but if you try to mutate in the function body you'll get an error from mypy, which is quite nice

[neon tartan]

Does mypy not adhere to this
https://www.python.org/dev/peps/pep-0484/#the-numeric-tower

[magic python]

I'm still getting the same error tho:

y = { ('a', 'b') :  {(1, 1): 2, (2, 3): 1} }
def g( k : Mapping[ Tuple[str, str], Mapping[ Tuple[Union[float, int], Union[float, int]], int] ] ):
    print(k)
    
g(y)

Found 1 error in 1 file (checked 1 source file)```

[flat gazelle]

it does, but only in cases where it is sound

that is odd, I would've thought mapping would work

[neon tartan]

👀 sound?

[magic python]

to me it still doesn't seem to be paying any attention to the Union

[flat gazelle]

sound meaning it will not cause errors

[halcyon trail]

what's the inferred type for y, is the first thing I'd wonder

[neon tartan]

Hmmmmmm

[halcyon trail]

the thing is that mypy doesn't infer unions, in most situations

it's more like to infer the common supertype, in many situations, rather than a Union

[neon tartan]

It shouldn't need to infer unions but it should at least accept them, no?

[magic python]

so is this just not an option or what? I'm really confused

[halcyon trail]

Yeah, I agree it shouldn't neat to

[magic python]

do i just not bother if i'm passing around data like this? Or make a custom class, or what

[halcyon trail]

if you eliminate the Union types in favor of straight int, then it works

[magic python]

so if i don't use the data that i'm using it works, that's not really a solution tho lol

[halcyon trail]

so I guess that mypy is not properly understanding some combination of covariance and the super-type nature of unions

Well, types this complex are relatively rare IME, in most cases you'd be defining classes before this

[grave jolt]

🤔 https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1675-L1676

[fallen slate]

Lib/typing.py lines 1675 to 1676

# NOTE: Mapping is only covariant in the value type.
Mapping = _alias(collections.abc.Mapping, 2)```

[halcyon trail]

hmm, that's interesting

[magic python]

I'm happy to make some kind of wrapper class rather than beat my head against mypy i just don't know what i'm meant to do to get it to pass

[halcyon trail]

actually, that seems correct

[grave jolt]

The key type of a mapping is invariant

[halcyon trail]

I just verified it's the same way in another language with covariance (Kotlin)

so this makes perfect sense

you need to be able to hash keys and compare them with ==, you can't do such things with a super/sub type in general

[flat gazelle]

oh wait, Mapping is invariant in key

[magic python]

does anyone know how to actually type this or not

bc i need to do this : ' )

[halcyon trail]

Well, you could annotate y manually

[grave jolt]

Just use float instead of float | int, and use float literals 🙂

[flat gazelle]

you are attempting something unsound as far as I can tell
https://github.com/python/typing/pull/273

[halcyon trail]

Beyond that though, I suspect you could probably use a dataclass here with some automated to/from dict conversions

[flat gazelle]

int keyed types are not necessarily keyed by float

[magic python]

so this would mean that all ints are converted to floats for this? Maybe that could work

[flat gazelle]

float is implicitly float | int

[magic python]

oh, then maybe it wouldn't

[halcyon trail]

to be a bit pedantic, key'ing a hash table by floats is actually a bad idea

[flat gazelle]

yeah, that too

[grave jolt]

comparing floats by equality generally is

[magic python]

How tho? I'm still not sure what anyone would practically do in the case of having to type something like this example

[halcyon trail]

Well, to give you a recommendation need a bit of a broader scope here

[magic python]

isn't the specific example pretty useful for recommendation purposes?

[halcyon trail]

recommendation of what? alternatives?

[magic python]

no just how i type this thing

[halcyon trail]

don't understand

[flat gazelle]

the issue is that program cannot be typed like this, because you could make a dict/mapping subclass which would have a type error when passed into it

[magic python]

i gave a specific example of something that fails, i'm asking how to make it pass

[halcyon trail]

yeah and I'm saying that probably the way to make it pass is to take a step back and change that part of the code at a slightly broader level

[magic python]

ok great

[flat gazelle]

https://mypy-play.net/?mypy=latest&python=3.9&gist=d7bfd3dadcedc41327ed7b0b62076482

[magic python]

i'll refactor everything for mypy lol

[flat gazelle]

this works

[grave jolt]

@magic python What data do you actually want to store?

[halcyon trail]

it's not about refactoring "for mypy"

[magic python]

it absolutely is if i have something that works but fails on typing

[flat gazelle]

essentially, the key type must always match exactly

[halcyon trail]

this type is easy to misuse, and hard to use correctly, regardless of mypy

it works today...

[flat gazelle]

def double(a: int) -> int:
    return a * 2
double('asdf')
```also works,  but fails on typing

[halcyon trail]

software engineering is programming integrated over time

[magic python]

it works today, and thats a design choice in a language like python - i don't enforce homogenous lists are used everywhere or whatever either, it's python not rust or whatever

[halcyon trail]

heterogeneous lists are also easy to misuse 🤷‍♂️

[flat gazelle]

you could just toss in an Any

if you are fine with the types allowing unsafe behaviour

[magic python]

sure - but they're a thing - and Any is annoying because i have a very specific structure

[flat gazelle]

cast() then

[halcyon trail]

You have a specific structure, but that specific structure is overly complex, and hard to work with

the same way that a function can "work" but be too complex and need to be refactored so it's easier to understand, etc, the same is true about types

[magic python]

it's not though, i know what it means and it works within the context

[halcyon trail]

if that's your feeling, why bother with mypy at all? I feel like this is turning into a static vs dynamic argument, which I want to avoid tbh

[magic python]

no you're turning it into that if anything, i'm just trying to use a typehint for a well defined piece of data in my code

[flat gazelle]

use cast() to just force the types to match

[halcyon trail]

you don't need to cast, just annotate the varaible explicitly should be enough

[flat gazelle]

or previously annotate the input type with union keys

[grave jolt]

quick proof:

If you have a m: Mapping[K1, V], you expect m.keys() to return an iterator of K1s. You also expect m[K1] to give a V.

Suppose that Mapping is covariant in key. Then Mapping[K1, V] is a subtype of Mapping[K1 | K2, V]. But if you have m: Mapping[K1 | K2, V], then m[K2] should give V, which would never be the case for Mapping[K1, V] (it will always fail). So Mapping isn't covariant in key.

Suppose that Mapping is contravariant in key. Then Mapping[K1 | K2, V] would be a subtype of Mapping[K1, V]. But that would not be right, since Mapping[K1 | K2, V].keys() would return an iterator of K1 | K2, whereas Mapping[K1, V].keys() must return an iterator of K1. So Mapping isn't contravariant in key.

Therefore, Mapping is invariant in the key.

</unnecessary academic interlude>

[flat gazelle]

I disagree with the covariance proof

[grave jolt]

But if you have m: Mapping[K1 | K2, V], then m[K2] should give V, which would never be the case for Mapping[K1, V] (it will always fail).
Well, yes, this is questionable.

[halcyon trail]

I wrote that you can just annotate y like a few pages back, you wrote "I'm not sure what anyone would practicall do in the case of having to type something like this"
And I answered you that they would refactor it to avoid having such a type

and also to avoid hashing floats

[flat gazelle]

the thing is that a mapping is a function

[grave jolt]

yeah, if you couldn't list the keys, it would be contravariant in the key type

[flat gazelle]

ye

[magic python]

unfortunately i don't think i can use this because I can't seem to do something like the following with it:

https://mypy-play.net/?mypy=latest&python=3.9&gist=24c735f0f6a41bde77533fbfc2a65d49

[grave jolt]

Can you explain what sort of data you need to store, and what you want to do with it?

[grave jolt]

🤔 Actually, why can't it be covariant? Guido's issue doesn't seem to provide some clear argument

Is it just because int aren't really a subtype of float, but typecheckers think it is, for convenience?

[flat gazelle]

int is a subtype of float more or less

it is defined in the type hint pep

[magic python]

hrm - I'm storing a dictionary where I have computed the paths between different levels, so if there's some data with columns a,b where a has values of 0,1 and b has values of 0,1 (these are not necessarily binary though) the dictionary for this might be:

{('a', 'b') : { (0, 1} : 2, (0, 0) : 1, ...}

where (0, 1) : 2 represents that there were two rows which had 0 within column a and 1 within column b

[halcyon trail]

it can't be covariant, but not for the reasons you wrote in your proof

[grave jolt]

And why do you want floats?

[flat gazelle]

class StrangeMap(Mapping[int, int]):
    def __getitem__(self, key: int) -> int:
        if 1 <= key < 1000:
            return key << 5
        raise KeyError
    def keys(self):
        return range(1, 1000)
```is probably the best example I have

[halcyon trail]

the union case isn't actually the problematic one

the problematic situation is simply with inheritance

you can't trust == to do the correct thing, basically

[flat gazelle]

https://mypy-play.net/?mypy=latest&python=3.9&gist=2e460221ae64677ceddd93bfd0e11261 that was just mypy not liking | for some reason

[magic python]

sometimes there are weighted values computed, meaning I might have (0, 1) : 2.4 or something, and sometimes the keys are floats not ints, resulting in there being (0, 1.0) : 2.4

[halcyon trail]

suppose that Mapping[Base, T] is supertype to Mapping[Derived, T]. Then you could have something of type Mapping[Base, T] that is really Mapping[Derived, T]. You have an object b of type Base, and now you want to do a lookup using b

[grave jolt]

isn't that the same as my proof?

[halcyon trail]

the problem is that == between two polymorphic types is messy

[grave jolt]

ah

[halcyon trail]

No, because "always failing" doesn't matter in the context of an argument about types

[flat gazelle]

the easy way would just be making __getitem__ take object instead of the key type

like java does

[halcyon trail]

failing lookup is something that a lookup is always allowed to do

[grave jolt]

yeah

[halcyon trail]

== between polymorphic types cannot both work the way you'd think it "should" and also satisfy LSP

fwiw, for this reason, Kotlin completely bans inheritance of dataclasses, unlike python

python dataclasses seem to completely ignore LSP

which is probably fine in the context of dataclasses

[grave jolt]

I think Python's stdlib often ignores LSP 😛

[halcyon trail]

but in the context of covariance/contravariance, it doesn't make sense to ignore LSP

[flat gazelle]

yeah, LSP isn't really a hard rule, but ignoring it does have its issues

[halcyon trail]

well, it kind of becomes a hard rule in the context of covariance/contravariance

[magic python]

@grave jolt did that make sense? idk what difference it makes

[grave jolt]

@magic python Perhaps you could make an explicit type alias (like Graph or something)?

[halcyon trail]

the problem with storing things in a hash table that are are computed weighted values

for example, IEEE addition isn't associative

[grave jolt]

and annotate the dicts manually

[halcyon trail]

so if you change the order in which you do operations, you can get a very slightly different float, and your lookup will fail

[magic python]

ok, idk how to do this though, isn't there a guide or something? Or do you mean putting :

Graph = Mapping[ Tuple[str, str] : Mapping[ ...

at the top?

[halcyon trail]

If these are computed values, do you actually do lookups based on these floating point values?

[neon tartan]

Sorry to resurrect something, but I think I got confused with this 3.9 and the new | in 3.10

3.9 is new (ish) after all

[magic python]

i'm going to do that, tho it's failing, so am just using Any now i give up typing in python

[grave jolt]

yeah, except s/:/,

[halcyon trail]

Or do you just iterate over them, and you made them keys in a dict because it seemed nice/correct?