#internals-and-peps

1 messages · Page 144 of 1

red solar
#

The code is pure Python, but it relies on cffi library and a C library that’s provided with the wheel, but the C library is ANSI C so can be built anywhere

#

So platform specific (cuz of the lib), but not abi specific, and compatible with python3.6 or higher

#

Unless I get rid of type hints, then it works with any version of python3 (and probably python2)

#

This is basically me not wanting to get rid of type hints but also not wanting to build wheels for every version of Python

verbal escarp
red solar
#

Uhhh… hmm

#

That sounds very complicated 🤔

verbal escarp
#

i think it sounds more complicated than it probably is

#

but i don't know for sure

red solar
#

It had better be simpler than it sounds 😂

#

Is there no other way to provide type hints in a way that’s compatible with Python versions before 3.6?

verbal escarp
#

i doubt it, it's a whole new syntax

red solar
elder blade
#

That's not in Python's grammar

#

That's how MyPy did it before type hints

#

Oh?

#

But the comment there can be anything yeah?

unkempt rock
#

Isn't that basically the -> ReturnClass part of a func def?

#

And before 3.6 that wasn't a thing?

#

Interesting

red solar
#

i want type hints but for my code to work with any python3 version so i can provide a python3 wheel rather than a separate wheel for every python3 version

#

hmm ok i'll try that - i assume that it's python2 compatible too?

#

i mean, seeing as it's a comment...?

#

special how?

elder blade
#

But that is then discarded

#

Can you see it from the AST?

red solar
#

when i say is it compatible, i'm just asking if the interpreter will accept the code

#

or if it'll break

#

not if tools can use it to check the types

#

ok nice 🙂 i'll try that 🙂

#

sry @verbal escarp this seems much simpler lol

#

damn 😦

#

how could a comment break the interpreter?

#

oh i was talking about the type comments

#

can you do type comments for variables too?

#

self._my_int = get_int() # type: int

#

but would it be interpreted as a type comment by interpreters that support it, or would it just be a regular comment?

#

ok neat 🙂

#

thanks

verbal escarp
main ginkgo
lusty scroll
#

you can have multiple tags in the whl filename

#

this should work in theory (numpy does this): yourpackage-1.0.0-py37-none-any.py38-none-any.py39-none-any.py310-none-any.whl

red solar
#

Wait what?!!!!!

red solar
red solar
lusty scroll
#

well fingers crossed

red solar
#

Before I test that tho I need to write my readme - been putting it off and I need it :/

verbal escarp
#

filename as API is just dumb

red solar
#

It’s amazing in that I don’t need to change anything 🙂

verbal escarp
#

that could be achieved in a different way, too..

#

if they had wanted to

#

instead we're now stuck with parsing tags in filenames

red solar
#

I mean, the only limitation this introduces is that package names can’t have a . In them

#

Which was already a thing

#

So i’m not too fussed

main lynx
#

there's not necessarily overlap between "this works" and "this is good"

halcyon trail
#

I was just thinking, at first I cheered when list[] became available

#

but now I wonder if it's a mistake, because people should really be using Sequence[] a lot more than list[]

#

and since list is just there, maybe it will cause people to be lazy and just use list[]

red solar
#

did many people use Sequence[] over List[]?

halcyon trail
#

i don't know if they do

#

I didn't at the start

#

but you should

#

most common places for type annotations is function arguments, most functions shouldn't mutate their arguments, ergo Sequence should be more common than list[]

red solar
#

what we actually need is a way to figure out if a type supports the buffer protocol without just putting memoryview(obj) into a try/except

#

so that i can then use it as a type hint

rich magnet
main lynx
#

i would only reach for Sequence in the most generic library code, most of the time i have a strong opinion on what type of sequence is best

#

or want to specify what is expected

#

e.g. in many cases where you could operate on either list or tuple you are not expecting a str or bytes

#

e.g. mypy approves this

from typing import Sequence

def foo(s: Sequence[int]) -> None:
    for i in s:
        print(i)

foo(bytes((1, 2, 3)))
foo([1, 2, 3])
foo((1, 2, 3))
halcyon trail
#

Why would you only reach for Sequence in generic library code?

main lynx
#

but did you really want bytes to qualify?

#

if you didn't care that it was bytes, then use Sequence

halcyon trail
#

the main point isn't really about str or bytes

#

(although that is surprising)

#

the main point is about mutation

#

which is a lot more common

#

If your function isn't supposed to mutate its argument, it's very valuable to have mypy tell you for free, when you accidentally mutate it

#

I've lost track of how many tricky bugs (to track down) have been caused by accidental mutation of arguments

#

Heck, unintended mutation is the main reason I tell everyone not to use defaultdict 😛

main lynx
#

unfortunate that there is a MutableSequence specifier but not a Sequence specifier

#

you can force immutable though with tuple[int, ...]

halcyon trail
#

I don't follow

#

what do you mean, a MutableSequence specifier but not a Sequence specifier?

main lynx
#

you can type hint MutableSequence to require that it's mutable, but not the opposite

spice pecan
#

...Sequence?

main lynx
#

tuple is in Sequence

spice pecan
#

The typecheker won't allow mutating a Sequence

main lynx
#

list is also in Sequence

halcyon trail
#

List is a sequence, yes

#

the point is not that it's immutable

main lynx
#

intresting, Sequence has to be immutable because tuple and bytes are in it

halcyon trail
#

the point is that the function cannot change it

#

no

#

interfaces are not immutable or mutable

spice pecan
#

If you typehint a parameter as a Sequence, you can only read from it, not mutate it (unless you violate the typehint, which begs the question "Why use it in the first place")

halcyon trail
#

interfaces are merely read-only or mutable

main lynx
#

mypy chokes on this

from typing import Sequence

def foo(s: Sequence[int]) -> None:
    for n, i in enumerate(s):
        s[n] = i + 1

foo(bytes((1, 2, 3)))
foo([1, 2, 3])
foo((1, 2, 3))
#

sequence.py:5: error: Unsupported target for indexed assignment ("Sequence[int]")

spice pecan
#

It doesn't choke

#

It follows the protocol correctly

halcyon trail
#

let me rephrase; interfaces can't really enforce the absence of something in classes that implement their protocol.

spice pecan
#

Sequence is a protocol for any sequence, there's no guarantee for mutability, therefore if you try to mutate it, you're breaking the protocol

main lynx
#

that does beg the question, why not just say tuple unless you also want bytes for some reason

halcyon trail
#

Something like ImmutableSequence can only be a convention and can only exist when your subtyping system is explicit and not implicit

#

does list implement tuple?

main lynx
#

oh ok, so as long as you don't mutate anything is allowed

spice pecan
#

You may want a list, as long as it's not mutated, or a deque, or literally any sequence, including custom classes

main lynx
#

very cool that mypy checks that for you

spice pecan
#

I don't see the point

main lynx
#

yeah it's not something i've ever wanted to reach for

spice pecan
#

Sequence is a protocol for something you can subscript to read from it. It may also be mutable, but you don't know that (and neither do you need that)

halcyon trail
#

it's literally the most common case though

#

the most common case in python is that your actual data structure is mutable, but you are passing it to a function, and you don't want that specific function to mutate it

main lynx
#

the most common case in what domain?

halcyon trail
#

in the python language generally?

spice pecan
#

Any operation on an immutable sequence is a valid operation on a mutable one, why would you want to limit yourself to one specific immutable structure

halcyon trail
#

list, dict, set are without a doubt the standard python data structures; they are built in, have first class syntax, etc, and they are mutable data structures. This part shouldn't be controversial.
That usually functions shouldn't mutate their arguments, because it's less clear, you may find more controversial, but I think it's a pretty common take these days

#

Even if you don't go out of your way to avoid it though, most functions just don't even need to mutate their arguments

main lynx
#

keeping track of mutating membership in a list is a pretty small win when all the members of the list are themselves mutable objects

#

and i don't expect Sequence is any help there

halcyon trail
#

It is not, but that's just fundamental to python

#

and yes, it's only a moderate size win, but it is still a win

#

Keep in mind that int, str, and floats are all immutable in python

#

and if you have a nested data structure you can use the read-only annotation in the nesting, and this all works out because of covariance

#

So in practical terms, it is more than a small win. It guards against some mutations all the time, and it guards against all mutations some of the time.

#

if you write most of your classes as dataclasses/attrs with Frozen=True then you will gain even more

#

But yes, to really go farther you need to have something that is properly transitive, like const in C++ (or mut in Rust)

main lynx
#

i agree that if you are dealing with sequences of int, str, float, tuple and/or bytes that annotating Sequence is safer

halcyon trail
#

It's more just that there's no real reason to annotate as List in an argument

#

okay, maybe not "no" reason, but it should be very rare

#

should be Sequence 90% of the time, MutableSequence 9%, and maybe List in the remaining 1%

#

even if Sequence only offers some protection against mutation it's more than 0

#

You are encouraged to annotate as List in the return type though

native flame
#

liberal in what you receive and conservative in what you return, heh

main lynx
#

seems mildly obnoxious to have mostly redundant typedefs for List, Sequence, MutableSequence

halcyon trail
#

what do you mean?

main lynx
# 
from typing import List, MutableSequence, Sequence

Widgets = List[int]
ImmutableWidgets = Sequence[int]
MutableWidgets = MutableSequence[int]

def read(s: ImmutableWidgets) -> Widgets:
    for i in s:
        print(i)

    return s

def write(s: MutableWidgets) -> Widgets:
    for n, i in enumerate(s):
        s[n] = i + 1

    return s

read(write([1, 2, 3]))
#

and this is not valid:

sequence.py:11: error: Incompatible return value type (got "Sequence[int]", expected "List[int]")
sequence.py:17: error: Incompatible return value type (got "MutableSequence[int]", expected "List[int]")
#

if you receive a Sequence you must return a Sequence

halcyon trail
#

well, yes, you don't know it's mutable, so how can you return a list from a sequence?

main lynx
#

good question

halcyon trail
#

well, you can't

main lynx
halcyon trail
#

but in practice people don't return the container that was passed in

#

liberal in what you receive and conservative in what you return, heh

#

it's python's view, the idea is that libraries should be as nice to people as possible

#

accept the broadest possible type, return the narrowest possible type

#

that makes life "easier" for the user

#

Also, do not use the name immutable in connection with Sequence... it's just incorrect

#

ReadOnlyWidgets

main lynx
#

that's fair

#

i still need to duplicate the typedef

#

unless there's some magic to steal the contents of a previous typedef

halcyon trail
#

I mean in most cases you just wouldn't bother with a typedef

#

unless it got a lot more funky than that

#

but yeah that's also one of the things that const/mut in other languages buys you

grave jolt
#

Sequence vs List isn't just about immutability

#

it also has a different variance

#

you can't pass List[str] where a List[str | int] is expected

lusty scroll
halcyon trail
#

read-only-ability! Please, forget the I-word when discussing sequence

#

an argument passed as a Sequence isn't immutable in any sense

#

(isn't necessarily immutable in any sense)

#

@lusty scroll yeah it's horrible, I never use std::map::operator[]

red solar
#

Stuff like keeping track of how often something appears

#

You can have an int key and increment it without checking if it exists 🙂

spice pecan
#

I think it's a side-effect of not having separate get and set operations for a square-bracket

#

You just return an lvalue that has to work correctly with both assignment and usage

red solar
#

A little bit of that, and a little bit of not wanting to throw an exception because the [] on other types doesn’t (cuz no bounds checking)

halcyon trail
#

it's terrible because people tend to use it in cases where they "expect" the key to already be there

#

and then when it's not, it just gets silently inserted, and your error instead of occuring immediately can happen much, much, much, much, later

#

there's better ways to insert, and better ways to access, and even to insert + access there are better ways 🙂

#

In python btw I always tell people to use dict.setdefault, which is an awesome little method that many people don't know about. It can replace like 95% of defaultdict usage IME.

grave jolt
lusty scroll
halcyon trail
#

well, I think the whole point is that its explicit at the point of use

#

and not implicit

#

so that would sort of defeat the purpose

lusty scroll
#

having a default= argument?

halcyon trail
#

I find people don't actually need the default carried around with them very often

#

Most times just using setdefault and get are all they need

lusty scroll
#

I'm not picking up on the distiction

#

you're saying setdefault should be done at the point of use--do you mean, it shouldn't be done when (or shortly after) the dict is created?

spice pecan
#

Since setdefault returns the inserted/looked up value, it's useful at the point of usage

#

You could do things like items.setdefault('key', []).append(...)

lusty scroll
#

oh, ok, I see what you're saying

#

I'm probably missing something, but if the point of creation and use are not in, say, the same function, seems like there could be consequences from setting a default and/or reading (implicitly creating new) entries.

halcyon trail
#

well, I'm really just saying that the most common uses of default dict are literally for things like above

#

they want to append to a list, if the list doesn't exist yet, created

#

or same thing but with nesting dicts, insert this value into a nested dict, if the dict doesn't exist, create it

#

they use defaultdict, and then later they get burned because they do x = "hello: " + my_dict["something"] and they were expecting "something" to be in the dict

lusty scroll
#

Yeah the expressiveness is good

halcyon trail
#

yeah. A lot of people don't know about setdefault I find.

lusty scroll
halcyon trail
#

no, if its a defaultdict

#

the point is that using a defaultdict because you need defaulting behavior at 1-2 locations is really bad because you get this defaulting, even defaulting + mutating behavior everywhere

lusty scroll
#

ohhh like it should be an error

halcyon trail
#

Yeah.

#

Just use a regular dict and explicitly opt into the behavior you need, where you need it, is better in like 95% of real world code I've seen.

lusty scroll
#

yesah for sure

#

I wasn't even thinking of that being on the table tbh

sacred tinsel
#

i personally think that setdefault is not very popular because the name just isn't very good

spice pecan
#

Yeah, the name doesn't really convey its purpose very well

#

Though I admit I can't come up with a better one

sacred tinsel
#

it reads like something that you'd use to set the dict's default value to something (like turn it into a defaultdict)

spice pecan
#

Yep, exactly

halcyon trail
#

In Kotlin this is called getOrPut

#

Also fun in Kotlin is the fact that instead of being a function that takes two arguments by value, the default is given by a lambda, which is only called if the value doesn't exist in the map

#

so you can have expensive calls in tehre

#

val x = myMap.getOrPut("key") { expensiveCallToGetValue() }

feral cedar
#

yeah, rust has this too

halcyon trail
#

or just val x = myMap.getOrPut("key") { 5 }

#

what's it called in rust?

#

Another fun thing about passing a lambda like that is that if you are using your dict as a cache for a recursive algorithm, you can put the recursive call directly in there 🙂 Not super practical day to day but it was pretty fun for advent of code type question

feral cedar
halcyon trail
#

ah, ok, but it has different named methods for each case

#

that's the nice thing about these insane lambdas you see in Kotlin and Swift; they literally don't cost any characters over a regular argument so anywhere a lambda makes sense you can just make it a lambda and not feel the need to have a different function for the non-lambda case

#

the entry thing is still nice though, especially since it's actually safe to use, you could write an equivalent in C++ but it would be quite easy to shoot yourself in the foot

feral cedar
#

that's usually how it is with rust lol

halcyon trail
#

pretty much

paper echo
verbal escarp
odd citrus
#

Hello,

I'm sharing with you two videos that you can download from here:
https://drive.google.com/drive/folders/1UfInMntlmlzZKxZ7QFGNNVv-m0B2YR5B?usp=sharing

1.- "Presentation.MOV" with me talking at the camera where the foreground looks good, but the background doesn't.
2.- "BackgroundVideo.MOV" was filmed in the same place but with different camera settings in order to have the background with more light and look better.

I'm planning to record a bunch of similar videos, but I would like to get the foreground from the first one, and the background from the second one.

I'm open about making some adjustments for improving the process and having it with a better chance of a successful merging, like for example, having a cleaner foreground, making sure the horizon is always at the level of the railing behind me (like it is on the attached videos), etc.

Is there a way to automatically process them in some way to achieve such a purpose?

red solar
odd citrus
#

but why?

red solar
#

oh crap i'm in the wrong channel

#

my bad

#

i'll take a look

odd citrus
#

Thanks

red solar
#

i'll be honest, i don't immediately see a way to use Python to help with this - the issue is that I also don't know how you would combine the two without python

#

you might be better off asking video editing professionals how they would do it first, just to get an idea of what it is you need to automate?

elder blade
red solar
#

you think that could work on a sample where the two video backgrounds have such differing contrasts?

#

As the name suggests, BS calculates the foreground mask performing a subtraction between the current frame and a background model
if it's just a subtraction i don't have high hopes

elder blade
#

Oh yeah the background file is useless. But there is at least one frame from the presentation video that can be used to subtract

heady mauve
#

Why did the PSF decide not to have private classes?

red solar
#

was this discussed recently on the mailing list? (if so i missed it)

heady mauve
#

not that I know of

red solar
#

oh, so you just mean in general

heady mauve
#

I have recently been looking into securing Python to allow ACE without it breaking out of containment, but so far the only solution that seems to be viable and actually secure is to sandbox an instance of the interpreter using the OS.

heady mauve
red solar
#

Classes implemented in C can have private stuff (unless you use ctypes to access it)

heady mauve
#

I do need a pure python implementation

red solar
#

But the only reasoning of read about for python is guido's consenting adults quote

heady mauve
#

<_<

red solar
#

the best you're going to do in pure python is using @property

#

bear in mind that even in compiled languages like C++, private is still only a compiler suggestion

#

you can always do some pointer trickery at runtime to access the private stuff

heady mauve
#

ok

verbal escarp
#

@heady mauve did you try other interpreters? you didn't like brython because your JS ban, but there are others like stackless etc?

heady mauve
#

I am trying to keep as close to CPython as I can because there will be multiple projects built on it.

#

And with the nature of one of them, it's a bad idea to start digging around with interpreters that don't match CPython execution style almost perfectly.

swift imp
heady mauve
#

so it would seem

swift imp
#

You can even bypass property

heady mauve
#

So then, if ctypes weren't available, would there be ways to secure objects?

red solar
#

if it's in python there will always be a weak spot

heady mauve
#

gah

red solar
#

do you need the class itself to be private, or just the instance?

halcyon trail
#

having private/public also makes a lot less sense in dynamic languages

red solar
#

maybe you could do something fancy with thread local variables

halcyon trail
#

because it's a bunch of extra cost on every single member access which is kinda silly

#

in fact, the design of private/public in static/compiled languages also takes this into account, it's specifically designed so that everything can be checked at compile time

#

this is why, for example, private access in C++/Java, etc isn't actually by instance, but by class

heady mauve
#

that is understandable

#

I appreciate the insight, thank you.

elder blade
#

You can just mount it as a Docker container

heady mauve
#

again, security is done from outside the interpreter

#

it uses nsjail to sandbox an interpreter instance

verbal escarp
# elder blade Isn't that what Snekbox is?

that's built on JS and they banned JS from their project for some reason.. I suggested translation by Brython, which would also get rid of those security issues, i think

#

btw, what do you think of first = some_list?[0] 😄

elder blade
verbal escarp
#

"The code is executed in a Python process that is launched through NsJail, which is responsible for sandboxing the Python process."

#

exactly

elder blade
#

JS = NsJail?

verbal escarp
#

ah, no. but it's linux only i think?

#

and they wanted it on windows, too

elder blade
#

Ah I see, it runs on Docker though so it can work on Windows

heady mauve
#

nsjail can't though

elder blade
#

But NsJail runs in Docker?

heady mauve
#

Sounds like linux-only to me

unknown.png
soft bloom
magic nova
#

Yeah, nsjail is Linux only. It uses the same underlying platform APIs as docker but it doesn't use docker.

wide shuttle
#

I always thought that all iterators were iterable, since the docs originally stated that they are required to implement an __iter__ method (in addition to __next__), which makes them follow the protocol. The PR Brett links/opened removes that requirement from the docs ("strongly encouraged").

main ginkgo
#

that makes very little sense to me tbh

feral cedar
#

why shouldn't iterators be iterable?

halcyon trail
#

i'm not sure if there is a good reason for it

#

but there isn't any strict reason requiring them to either

#

An iterator is something that has next, an iterable is something that can produce an iterator. Two orthogonal requirements 🤷‍♂️

#

the fact that every iterator can satisfy iterable by returning itself doesn't need to be part of this

quick snow
#

That means that you can't be guaranteed that you can iterate over part of an iterator, and then feed it in a for loop again...

halcyon trail
#

that's never guaranteed

#

an iterator is a thing that allows single pass iteration

wide shuttle
#

The docs used to say that iterators are required to implement an __iter__ method that returns the iterator itself, which would guarantee that for element in iterator would work. That's no longer the case, although I see the reason in it.

halcyon trail
#

you could probably make an argument that it is actually better that we discourage anything from ever implementing both dunder iter and dunder next

#

If we encourage people to implement iter to always return self, then they a) have to implement it, adn b) they have the opportunity to implement it in an insane way

quick snow
#

It's always possible to implement stuff "in an insane way", that is not a convincing argument to me

wide shuttle
#

The proposed version of the docs still say it's strongly encouraged:

The iterator objects themselves are required to support the following the method, which forms the :dfn:iterator protocol (although iterators are strongly encouraged to also implement :meth:__iter__ as well)

halcyon trail
#

instead, you could make for example the iter free function return the argument if it implements dunder next, or if that does not exist, return the result of dunder iter

quick snow
halcyon trail
#

It's always possible but if we think that there is only one sane way to do something

#

it's more logical to encode that behavior in code, rather than in an idiom

#

So you could simply encode that behavior in the free function, e.g. iter, rather than make it a convention

#

that is one of the reasons why we have the free-function vs dunder separation

#

you can have aspects of the behavior that are fixed, and aspects of the behavior that are customizable

quick snow
#

So you're in favor of iter(x) to return x if x doesn't have an __iter__ method and x has a __next__?

halcyon trail
#

soemthing like that

#

iter already checks for the presence of different protocols and does different things

#

many free functions do that:

Return an iterator object. The first argument is interpreted very differently depending on the presence of the second argument. Without a second argument, object must be a collection object which supports the iteration protocol (the iter() method), or it must support the sequence protocol (the getitem() method with integer arguments starting at 0).

#

so as you can see, right now it's already checking for dunder iter, or dunder getitem

#

instead of telling everyone "please implement your iterators to also have an iter dunder that returns self", why not just handle that automatically in iter? Makes far, far more sense.

wide shuttle
#

I'd consider the __getitem__ more of a legacy support than anything

quick snow
halcyon trail
#

I can't comment on the getitem bit.

wide shuttle
#

I wouldn't take that as an argument to make the behaviour more complex than it already is

halcyon trail
#

I don't really see an issue with it. This is a very standard strategy in software engineering that already has various names in various contexts

#

you separate interface from customization points because you want to enforce certain aspects of how something works, and limit what can be customized

#

a lot of python's free functions don't just forward the call to the dunder

wide shuttle
#

I'd argue that's more of a point for the interfaces you built for your implementation rather than that it's necessarily a good practice to design the core language around

#

Python is very flexible, and Is see that as a strong point instead of a weak point

halcyon trail
#

idk what you mean. but, again, if you think that there is no sane implementation of __iter__ other than return self for anything that supports __next__, what are the benefits to keeping it as a convention instead of just controlling it in iter?

#

And the thing is this: parts of python already work this way, that's the whole point of the link

#

For instance, if a pass an iterable to a for loop and it returns something that only defines next(), it will work

wide shuttle
#

That's something else, though

#

It means that if you call iter(iterable) and the object returned only implements __next__, it will work

#

iterable still has to implement __iter__ (or one of the other supported methods of iteration, such as the getitem iteration)

halcyon trail
#

okay, fair enough, I misunderstood (missed the extra level of "returning")

wide shuttle
#

The object that only implements __next__ would not work when passed to a for-loop itself (instead of the iterable)

halcyon trail
#

Anyhow, I don't really see any benefit to saying there's only one sane implementation of Iterable for every Iterator and making everyone do it

#

I mean, maybe, at this point, yes, given the reality of history and python's existing code, implementation, etc

wide shuttle
#

I agree, I don't mind the lessened requirements

halcyon trail
#

but on its own merits, no, there's no good reason

wide shuttle
#

I just don't think there's a good reason to add another magical piece of behaviour to iter (to make it act as if the object has an __iter__ method returning self)

halcyon trail
#

Right, the thing is that if you have these lessened requirements, it might be useful in some situations to ask for a way to iterate over something, not caring if you already have the iterator or an iterable

#

that function doesn't necessarily have to be iter, though to me it makes sense

#

I don't really understand why you call it "magical" tbh, when as I said before this is very vanilla software engineering, you seem to view it differently because it's more tightly part of the language implementation.
But that seems to imply that what these free functions "should" do is just call dunders. When in reality, the whole reason why we have separate free functions and dunders is to support this exact sort of thing.

wide shuttle
#

I don't really understand why you call it "magical" tbh, when as I said before this is very vanilla software engineering, you seem to view it differently because it's more tightly part of the language implementation.

It's magical in the sense that the function does something very different based on what you pass in, and it does so in a way you no longer control, even if you wanted to. The easiest way would just be to have iter always be controlled by the __iter__-method, instead of having fixed behaviour in addition to that. As it is, we already have support for alternative ways of implementation iteration (e.g., the getitem iteration).

#

I'd consider such behaviour to be more magical and less intuitive, which is why I don't agree that it would a be a good pattern

halcyon trail
#

That's not the easiest way though. The whole point is that these free functions exactly exist to limit user control and customization 🙂

#

almost every free function that has an equivalent dunder doesn't just call it

wide shuttle
#

What makes you say that these free functions exist to limit user control?

halcyon trail
#

why does next get to have a default argument, and return it when the iterator is exhausted, we could have that be customized too

#

they exist to either limit user control, or to provide automatic implementations of similar functionality, given different protocols

wide shuttle
#

I don't really get that point

halcyon trail
#

you don't understand why they limit user control?

#

by user in this case, I mean, the user that is creating the object that provides the dunder

wide shuttle
#

I'm still not sure what you mean, no. In one way, you could argue that they force a specific interface, as in that you have implement support using a specific interface, which you could argue as a limiting factor.

halcyon trail
# 
def foo(x):
    print("hello")
    x.__foo__
    print("bar")
#

the user does not have the ability to customize the foo dunder such that hello and bar are not printed. agreed?

wide shuttle
#

I won't argue that this is not the case, but I will argue that I don't see this as the main reason for using standalone functions

halcyon trail
#

I don't know about main, but it is one of the reasons

wide shuttle
#

Using functions, with dunders to allow you to implement support for them, gives you an uniform interface that you can still implement support for

halcyon trail
#

also any time that you are constraining things you are also usually providing something useful without the user implmeenting it, which means you are also helping avoid code duplication

wide shuttle
#

There's no guessing the method name that iterates over a sequence, like in some languages

#

And that principle of least astonishment is what I've always understood to be the main reason for this design

halcyon trail
#

for example next; next takes two arguments, but dunder next takes one. that's because next enforces the way that the default argument works, that part isn't open to customization (and people also don't have to implement it).

#

errr tons and tons and tons of languages achieve this

#

tons and tons

#

without using this free function/dunder split

wide shuttle
#

That's not really the point, though

#

Even if something is the main reason for a design, that does not mean that design is the only way to achieve that

halcyon trail
#

if that were the only actual reason, then the majority of free functions would just call the dunder and nothing else

#

but that is not what most free functions do (obviously, we don't count operators in this)

wide shuttle
#

No, I don't agree

halcyon trail
#

the main reason that next exists is to just to standardize the calling of __next__, even though next doesn't just call __next__, it actually does other things?

wide shuttle
#

This allows the language to be more flexible in how such functionality is implemented, e.g., many of the built-in types don't actually call the dunder, without sacrificing end-users, python developers, to still be able to implement support for it in a uniform way.

halcyon trail
#

yes, that's another reason.

#

separating API from customization in advance gives you a bunch of potential benefits, of which that is one, what I've been saying is another.

wide shuttle
#

I just don't see your main point, limiting user control, as having had a big influence on the design. I can't find something that supports that.

halcyon trail
#

At any rate, I cannot find any reason why it makes more sense to keep telling people "please implement __iter__ the only sane way" when we could just include the sane implementation in iter and call it a day... I don't see any particular problem with iter enforcing the behavior we already consider sane.

#

🤷‍♂️ . Sorry, it's mutual, I can't see any big deal about the alleged downsides of making iter more complex. Making one function have two extra lines of code, instead of asking everyone to keep implementing a certain function, the exact same way, indefinitely...

#

The behavior would be identical to what it always is now, except that every person implementing an iterator would save two lines of code, and you prevent the possibility of a bug in those two lines
unless you are saying there is a reason to implement __iter__ as anything other than return self, which you haven't said... If you don't think there is, I don't understand how you can say the behavior is more magical and less intuitive

wide shuttle
#

I can't think of a reason, but that doesn't mean that there isn't one. I've been looking for it now. I did find some pages stating the __iter__ of an iterator does not always have to return self, but no actual example of iterators that don't for a reason.

halcyon trail
#

at the very least, it's incredibly rare, and it would still be a net benefit to call __iter__ if it exists, and then if it doesn't look for __next__ (and if it exists, return the argument. Or, rather, probably do that after looking for the getitem protocol, for backwards compatibility)

#

this is giving identical behavior in every single situation where iter would have worked at all in the past, so again, I don't see how this could be called "unintuitive"

#

it would just give people the 99% correct, sane behavior by default in their iterators 🤷‍♂️

wide shuttle
#

You could post it as an idea to the ideas mailing list. I'd be intrigued to know how core developers react to that proposal.

halcyon trail
#

I imagine it would perhaps be legitimately tanked for other reasons e.g. performance

surreal sun
#

They’re probably gonna respond to that saying that you can just use collections.abc.Iterator

halcyon trail
#

there's a lot of practical considerations for something like this

wide shuttle
#

Yes, that would not surprise me, but I did not want to use it as an argument here for the more theoretical case

halcyon trail
#

also a fair response

#

Well, I would accept it as an argument, but yes, it's a rather boring argument in the kind of discussion we were having 😛

surreal sun
#

since if you have __next__ implemented collections.abc.Iterator will give you an __iter__ if you subclass it

wide shuttle
halcyon trail
#

@surreal sun yes, that's a good point as well

surreal sun
#

It would be nice to have though, I agree

halcyon trail
#

so probably the benefit is just too small, and the minor performance/complexity/overhead of changing deteriment overwhelms it

surreal sun
#

But the mailing-lists are quite infamous for very few ideas actually getting the Steering Council engaged in it

wide shuttle
#

If you allow me to develop your point, that does suffer from that it's still optional and you'd have to explicitly include it

halcyon trail
#

subclassing abc.iterator is probably something you should do anyway

#

Right, that is true.

halcyon trail
#

they have to filter

wide shuttle
#

Interesting ideas do typically get some discussion, even it's not necessarily by the steering council

halcyon trail
#

And honestly, taking a step back: who the hell am I to give them suggestions anyway? I'd hardly call myself an expert in python.

#

Let me propose things on the python mailing list is like letting high school students send papers to Nature.

wide shuttle
#

That doesn't mean that you can't make a solid observation

surreal sun
halcyon trail
#

Sure, it's just that if you're the steering committee and you get sooooooo much more stuff than you have time to look at, you have to bias heavily by who the source is

#

In an ideal world it wouldn't be like that, but that is life, and I don't take it personally

surreal sun
wide shuttle
#

Some of the ideas posted to that list are rather hilarious. "For me, it would be good if thing a that does x would actually do thing y. Could you make it so?"

halcyon trail
#

Yeah......

#

in python especially I can imagine, some complete beginners posting there

#

with ideas that are so bad that you can't explain to them the problem with it

#

"not even wrong"

wide shuttle
#

It doesn't always have to be the steering council, though. Some ideas get picked up by other contributors/core developers and even lead to rapid PRs.

halcyon trail
#

i tried briefly to get involved a bit with C++ standardization but in the end I just backed out 😛 it's a tough world I think.

#

That's pretty neat. If I did want to post something about it, where would I do it?

wide shuttle
#

Someone from here suggested something related to a mutex half a year or so, and a core developer picked up on it and produced a PR within what I remember to be something like 1 or 2 days.

halcyon trail
#

what if I just commented in the github thread?

wide shuttle
#

Would it be more of a separate suggestion to the current PR or more of a review comment on the current PR?

halcyon trail
#

or on the bug, actually

#

yeah no the github thread is bad

#

but maybe just a brief comment on the bug tracker

#

99% nothing comes of it but we'll see 🙂

tidal schooner
#

Can you help me. I would like to make this using Arduino. I need to move these syringes using motors but I can decide which motor to move though my phone, so there will be also an app ( so though the app in my phone I can control which syringe to move ).
Can anyone help me figuring out which product I must use to move those syringes ( instead of using my hands )? And what product I must have to connect those motors to my phone, so I can move them though an app I made using my phone ?( using IOS ) I would be really grateful

IMG_6228.jpg
raven ridge
red solar
#

What was the developer in training thing from the survey?

halcyon trail
#

I think I discovered something weird

#

defaultdict inherits from dict, and dict implements Mapping. So, defaultdict must inherit Mapping. But Mapping is a read-only view of an associative data class.

#

the problem is that with defaultdict, what looks like a "read-only" usage of operator[] is actually a potential mutation

#

since it inserts

#

am I missing something?

grave jolt
#

@halcyon trail is it documented anywhere that __getitem__ on a Mapping doesn't do any mutations?

#

hmm I guess defaultdict is a bit weird, it breaks this property of Mapping: "all valid keys are equal to an element of .keys()"

rich cradle
unkempt rock
#

.

halcyon trail
#

@grave jolt i mean the whole point of Mapping, when you also have a class called MutableMapping, is that it's a read-only API

#

nothing in Mapping is supposed to mutate the object

#

ABCs for read-only and mutable mappings.

#

getitem isn't special, it shouldn't mutate the object either

grave jolt
#

well, GET requests are not supposed to mutate anything. but it can mutate a cache

halcyon trail
#

err, not sure what GET requests have to do with anything

grave jolt
#

an analogy, perhaps...

halcyon trail
#

we don't need to go to GET requests

#

data structures can have caches

#

mostly famously the standard union find implementation

#

but if you do have a cache, and you claim that a method is non-mutating, but it does affect the cache, it is your responsibility to implement equality for that type so that the item will still compare equal to itself

#

this gets discussed a lot in languages that have built in support for const, like C++

#

"read only" is defined in terms of the observable state of the object, and the observable state of the object is defined in terms of ==

# 
from typing import Mapping, Dict

def foo(x: Mapping[str, str]):
    print(x["hello"])
    
d: Dict[str, str] = defaultdict(lambda: "world")
foo(d)
print(d)
grave jolt
#

hm yes, that will break

#

I agree now

halcyon trail
# 
world
defaultdict(<function <lambda> at 0x7f9d94c570d0>, {'hello': 'world'})
#

gross

grave jolt
#

well, the stdlib doesn't care much about LSP

halcyon trail
#

man I already hated defaultdict and now I hate it more

#

this is such a classic example of an LSP screw-up

#

and why you need some kind of automated implementation delegation without inheritance

#

defaultdict's operator[] breaks the contract of dict's operator[]

#

and therefore LSP

grave jolt
#

now let me play the devil's advocate for a moment

#

why is this bad?

halcyon trail
#

which is it bad to violate LSP?

grave jolt
#

yes

halcyon trail
#

because you get unexpected behavior when you pass the derived instance where a base instance is asked for

grave jolt
#

can you show an example of a function where this behaviour of dicts is expected?

halcyon trail
#

yes

#

the one I just showed you

#

I expected foo not to mutate my argument

#

it did

grave jolt
#

that's not a real one though

halcyon trail
#

it is real?

#

the whole reason you annotate Mapping instead of MutableMapping is so that.... mypy prevents mutations?

grave jolt
#

I mean, where would a real-world function expect it not to change (with some meaningful consequences)?

halcyon trail
#

what?

#

most of my real world functions don't mutate their arguments?

grave jolt
#

I think you'll have a hard time convincing Python programmers that defaultdict is bad

#

especially without a real-world example

halcyon trail
#

depends on how good of programmers they are 😛

#

I thought defaultdict was bad even before this

#

You shouldn't need an example beyond "this is allowing mutations in a function that mypy is promising me doesn't mutate"

#

I realize you want to play DA but "why is LSP important" and "why are unexpected mutations bad" are topics that have been hammered to death for decades at this point

grave jolt
#

I think the bigger problem is that these invariants aren't clearly defined (at least I can't seem to find where they are). It's not very clear to me that Mapping means "you can't mutate this". ```py
def do_something(my_mapping: Mapping[str, int]) -> 42:
if isinstance(my_mapping, dict):
my_mapping.clear()
return 42

halcyon trail
#

Right, so in this case you are doing an explicit downcast

#

Mapping doesn't mean you can't mutate the object for multiple reasons, one of which is downcasting (there are others)

#

Mapping means "you can't mutate the object through this interface"

#

that's the contract

grave jolt
#

where does it say that though? is it just what you understand under 'read-only'?

halcyon trail
#

what else could read only possibly mean?

grave jolt
#

I don't know - it's not really defined. It could just mean "does not support __setitem__".

halcyon trail
#

that's not what it means because it excluded other mutating methods too

#

"read-only" means exactly that, you can only read, a "read" is an operation that doesn't write, which means it doesn't mutate

#

like really, that's clearly the intent, no need to go into the weeds on this

flat gazelle
#

I wonder if it's possible to implement a correct equality for defaultdict

#

Given a pure factory

halcyon trail
#

If you want further proof, the fact that Mapping is covariant further proves it

#

and you can break mypy using this trick

grave jolt
#

I think you should file a bug to clarify the documentation

halcyon trail
#

actually, my bad, you can't use the covariance to break mypy, because of how the lambda is typed

#

the problem isn't the documentation, the problem is the type hierarchy, but there's no way to fix it

#

(realistically)

halcyon trail
grave jolt
#

I can't really think of an example where defaultdict causes unexpected behaviour because of this, maybe you can help

halcyon trail
#

I've had so many bugs caused by data structures getting mutated at times I didn't expect in the last year

#

I can't imagine anybody programming in python who hasn't

#

I will say the most common manifestation of this problem isn't that it makes a correct program incorrect

flat gazelle
#

The problem is that in, .keys(), .values() etc are incorrect. There is nothing inherintly wrong with having a dict that has every object as a key, but it is hard to implement correctly in python

halcyon trail
#

the most common manifestation is that your program isn't correct regardless, ok, but this issue makes the problem appear very very far away from the incorrect code

#

Let's say that we didn't use a defaultdict in that function

grave jolt
#

I think the overarching problem is that Python doesn't define formal properties, for example k in d <=> d[k] doesn't throw KeyError

#

with that, it would be really easy to see where stuff is breaking and where it's not

#

right now those properties are implicit

halcyon trail
#

there's two possibilities: 1) we are using operator[] because we want .setdefault. When we use .setdefault mypy complains though about Mapping, forcing us to rethink our design, and at least change Mapping to Dict. We now know this is a source of mutation. 2) more common, we were using operator[] there because we expected the key was present. with our ordinary dict, operator[] throws an exception, making the problem immediately apparent.

flat gazelle
#

Like, in the most common use cases of a Mapping, the mutation is not observable. If you access a key, you get a result and it shows up in the iterator. The problem is when you need a key to not be present , which is quite rare IME.

halcyon trail
#

I don't think it's rare at all to not need keys to be present... when you create dicts, they can have all kinds of invariants, properties, having a key in a dict that's not supposed to be there can definitely cause bugs

#

simply iterating through a Mapping creates an observable difference, or calling len.

#

these aren't "rare" pieces of API for a mapping

#

I iterate through my dicts all the time.

flat gazelle
#

If you iterate through a Mapping, all keys you get are in the mapping. This holds for the defaultdict

halcyon trail
#

@flat gazelle you're just playing with words now. It's observably different, full stop.

#

you could be looping over key-values, applying operations to the keys or values, and then looking them up in another dict, which now throws an exception because something isn't present because of the unexpected key-value

#

i could give a million other examples

grave jolt
#

I think I found an example. Would you expect these two to be different? Probably not ```py
def chainget1(seq: Sequence[Mapping[K, V]], key: K):
for mapping in seq:
if key in mapping:
return mapping[key]
raise KeyError(key)

def chainget2(seq: Sequence[Mapping[K, V]], key: K):
for mapping in seq:
with contextlib.suppress(KeyError):
return mapping[key]
raise KeyError(key)

halcyon trail
#

Sure

#

you can create tons of examples like that

#

or examples based on unexpected mutation

#

We can even write a python program that generates programs demonstrating why violating LSP is bad.... it may even already exist 😛

grave jolt
#

I guess Python was built on quite a bit of hacks

#

rhettinger has a talk somewhere where he says that you don't have to respect LSP

flat gazelle
#

Well, you could solve this by implementing contains as unconditionally true. But ye, it is somewhat inconvenient. The class is convenient enough that I don't think it matters too much. You would run into similar issues with a dict that stores normalised http headers for example

grave jolt
#

yep, if you take all these things into account the only useful Mappings are Dict, MappingProxy and immutables.Map

flat gazelle
#

Maybe a lesser protocol that is pretty much a function is the way to go

grave jolt
#

you mean, pass thing.__getitem__ to functions?

#

I think I did that somewhere

fallen slateBOT
#

lanim/threaded_cache.py line 82

return ThreadedCache(factory).__getitem__```
flat gazelle
#

No, a Mapping is a partial function defined by enumeration

#

With a different operator

#

If you remove the requirement that it must have an enumeration of all it's elements, you get left with a partial function

grave jolt
#

so what does mapping.keys() mean then?

#

some random elements that are guaranteed to be in the mapping?

flat gazelle
#

The domain of the function. But that's what I mean with a lesser protocol. Needing the function as a full blown set is a harsh limitation

halcyon trail
#

you can just say, ok, when we annotate dict, what we mean is exactly dict, dict isn't intended for polymorphic usage

#

this still kind of sucks, but whatever

#

However, as soon as they added ABC's it became much worse

#

because ABC's are obviously intended for nothing but polymorphic usage

#

and now every ABC that dict satisfies, defaultdict automatically satisfies

grave jolt
#

oops

halcyon trail
#

and in this case, you have an ABC with a much stricter contract (Mapping), which defaultdict blatantly doesn't satisfy.
It's just very bad, there's no other way to slice it.

grave jolt
#

so if the whole protocol of the LesserMapping is key -> value, why not just use a function instead of the mapping?

flat gazelle
#

Because .get

grave jolt
#

hmm

flat gazelle
#

And maybe clearer that a key error is an option

grave jolt
#

I guess

halcyon trail
#

unless you meant "in python specifically"

grave jolt
#

I meant in Python specifically

halcyon trail
#

Well, sure, I think that is fine?

#

Counter is also a Mapping

#

btw

#

People can also have their own implementations

grave jolt
#

hm yes, I missed the fact that you can have non-generic mappings

#

or rather, not generic in both parameters

#

like counter

halcyon trail
#

right

#

ninja'ed me

flat gazelle
#

When writing against a Mapping you have to assume it can randomly mutate regardless, so it mutating as a result of element access isn't that different.

halcyon trail
#

or you can even have collections that are generic but in different ways, e.g. Grouping[K, V] inherits from Mapping[K, Iterable[V]] for example

#

who is "you" ?

flat gazelle
#

The implementor of a function receiving the Mapping abc

halcyon trail
#

first of all, the most important part of the Mapping guarantee isn't for the implementer of the function

#

it's for the user of the function

#

Second, it can't change randomly, no

#

it can change under a few situations, either via aliasing, or via calling callbacks (transitively)

#

but yes, you don't write functions assuming that your Mapping is immutable for the whole function body

grave jolt
#

or threading

halcyon trail
#

threading is a fair point, my bad

grave jolt
#

or async, but I guess it falls under callbacks no I read it wrong

halcyon trail
#

but in any case, what's important here is not really the implementation, it's the contract

flat gazelle
#

That's a fair point, a function taking a Mapping should provide the same Mapping after it ends.

halcyon trail
#

Yes. The caller promises to provide something that implements all of Mapping, the callee promises that it doesn't need any functionality outside of Mapping. And therefore, promises it doesn't need to mutate.

#

You can of course be a dick and downcast and conditionally mutate but, you know, that's what that is

unkempt rock
#

It makes it much harder to create a new instance of that mapping though

halcyon trail
#

well, that's a separate issue

#

and it's not related to Mapping vs MutableMapping

unkempt rock
#

Yep yep

halcyon trail
#

it's also not really related to Dict, because if you want to hardcode that you're returning a Dict out of all the available mapping types, then you could just take a Mapping and return a Dict anyway (which is usually what happens)

#

You have to start using generics at that point with a bound.

#

though I'm actually not sure if the Mapping or MutableMapping interfaces mention construction at all

flat gazelle
#

Actually, there are definitely sane dicts that will change when accessed. Consider a dict representing some underlying external data which checks for the data on every access (maybe with some cache). It is a Mapping, but if you pass it to a function taking a mapping, you get a different mapping, yet the function didn't really mutate it

halcyon trail
#

yeah, we already discussed caching

#

things can be mutated in the literal sense, but not in the logical sense.

#

A Union-Find is a very easy example of this. Every time you do a lookup in a union find data structure (well, the common implementation), it uses path shortening to make future lookups faster

flat gazelle
#

If the underlying external data changes, the mapping now stores different data

halcyon trail
#

so the "literal tree structure" is getting mutated

#

but the logical state of the union-find (i.e. which items are in which sets) has not changed

#

mutation for a given data structure is always relative to ==

#

if you have a data structure using caching then you should be sure that your == doesn't depend on the state of the cache

flat gazelle
#

Ye

halcyon trail
#

typically

#

this comes up a lot in C++

#

well, not a lot, but it comes up

flat gazelle
#

But if I have a mapping representing a tweet, and someone likes the tweet and a function looks up the like count, thereby having the mapping update it's data, it does get changed over ==

halcyon trail
#

the kind of mutation you're talking about is called bitwise const in C++. C++ enforces bitwise const when writing member functions by default, but what you actually need is logical const, so then you have to use the mutable keyword if you have a cache, etc

flat gazelle
#

Despite it being done through a non-mutating operation like .get

halcyon trail
#

i mean, if it changes over == then it is a mutating operation, full stop

#

that is the definition of mutation

flat gazelle
#

Ye, and .get has to be a mutating operation sometimes

halcyon trail
#

that's a matter of taste but it's simply inconsistent with Mapping

flat gazelle
#

Since not all Mapping s are their own source of truth

halcyon trail
#

and all of the equivalents to Mapping I've ever seen in any language

#

Whether a Mapping is its own source of truth is totally irrelevant, I'm sorry

#

You could have a class that consists of a Mapping of countries to their capitals. when you call get/operator[] it goes to some online API, fetches some data, etc

#

and then of course it stores it internally

#

when does this Mapping change? Answer: never.

#

it's always exactly the same Mapping, the data that's changing is just a cache.

flat gazelle
#

What if a country changes its capital?

#

Then the mapping also has to change

halcyon trail
#

the mappikng has to change but it's not the act of calling operator[] that changes it

#

and this is also getting quite messy because we have a cache, yet it can change in real time, so as we have specified this data structure so far, it's not even guaranteed to be correct

#

so it's hard to even discuss

flat gazelle
#

Let's lose the cache then

halcyon trail
#

Okay

#

Capital of Estona changes at noon

#

so calls to operator[] before noon give one answer, after noon give another answer

#

but it's not the call to operator[] that changed it

#

whether you called operator[] at 11:59, or not, will not affect your operator[] calls at 11:50 and 12:10

#

This situation pretty much models a function that gets passed a Mapping, which is actually (underneath) a dict, that is also being touched in another thread.

#

the dict is getting mutated but the operator[] from the function receiving the Mapping isn't mutating anything

flat gazelle
#

I kind of see your point. There may be a defaultdict interpretation where it is simply a lazy proxy to the underlying function where it may work in it's current shape, but I can't really think of it rn

halcyon trail
#

If operator[] was the only way to access the information

#

Then you could argue it doesn't really mutate

#

The values are always there, they're just implied

#

And you could write your equality on that basis

#

Does that make sense? I think you would like that interpretation actually; it's very close to mapping as a function

#

But in reality we have other ways to observe, we have get and len and iteration, and equality that's not compatible with this notion, so for defaultdict it falls apart

surreal sun
#

Hmm, so for those that have heard of PEP 659 (adaptive bytecodes), is the way it works just an 'adaptive' count (similar to refcounting but with bytecode 'adaptive' counts that keep track of when to use adaptive opcodes)?

#

Looking at it right now and it seems like some very cool stuff

dusk comet
#

typo in typing.py (in ParamSpec class docstring)
should be P.__name__ == 'P' (according to the previous definition)

unknown.png
gleaming rover
unkempt rock
#

Alright 👍🏻

broken furnace
#

Generic collection types are a conceptual tool, right? I mean there's no difference between a list and a list[A] in terms of what the type of the objects are.

#

Or is there?

gleaming rover
#

how deep into type theory do you want to go

lusty scroll
#

not super deep

gleaming rover
#

oaky

lusty scroll
#

what should it be called

gleaming rover
#

long story short

#

list[int] is a type

#

list is a type constructor

#

I'll just be loose here

#

because it takes one type argument to create a "concrete type"

lusty scroll
#

ok

gleaming rover
#

so they're not really types in the same sense?

#

also, generic types are kind of...

#

for all A, list[A], right

#

you can think of them as "families" of types

#

but types in the same sense as list[int], for example

#

"concrete" types

#

on the other hand, list by itself doesn't make sense

#

because it needs a type to become "concrete"

#

so what does it mean when we annotate a function like this?

#

def f() -> list

#

in a statically typed language (except Java I think?) that'd be a type error

#

we could just take it to mean the "widest possible" list, which would be list[object]

lusty scroll
#

list[object]?

#

right

gleaming rover
#

if it were a parameter

#

we would need to go in the opposite direction

#

list[NoReturn]

#

i.e. the narrowest possible type

#

okay, that's not necessarily true I guess

#

you could still take it to mean list[object]

lusty scroll
# gleaming rover `list[NoReturn]`

so seeing -> list: and taking it to mean -> list[object] , are the two interchangable or is the code technically wrong if it just says list?

gleaming rover
#

because list is not a type

#

I don't know how mypy etc. handle that

#

but I am sure that in most statically typed languages that is a compiletime error

lusty scroll
#

ok, gotcha

dusk comet
#

list is equal to list[Any], not list[object]

lusty scroll
#

so Any means "we don't know the actual type" ?

gleaming rover
#

in all typecheckers?

dusk comet
#

Any means "we can do anything with this object"

dusk comet
gleaming rover
#

but I think that would make sense for a dynamically typed language

gleaming rover
#

including arbitrary casts and attribute accesses

#

you can't do much with object

#

yeah, list[Any] makes a lot more sense actually

dusk comet
gleaming rover
#

hm.

lusty scroll
#

but if Any is the generic argument?

gleaming rover
#

I don't think you can assign Any to NoReturn...?

#

right?

gleaming rover
dusk comet
lusty scroll
#

does it mean it's compatible with list[int], list[str] etc.

dusk comet
gleaming rover
gleaming rover
#

like l: list[Any] = [1, 2, 3]?

lusty scroll
gleaming rover
dusk comet
#

l: list[Any] = [1, 'abc', None] is valid
l: list[str | int] = [1, 'abc', 3, 4] is valid
l: list[str | int] = [1, 'abc', 3, None] is not valid (None isnt compatible with str|int)

lusty scroll
#

it has to do with parameters vs. return types, right

gleaming rover
#

well, assuming [1, 2, 3] is inferred to be list[int]

gleaming rover
#

so like.

dusk comet
gleaming rover
#

f wants a list[Any], which means you can't pass in a list[int], since int "is a subset of" Any (whatever you can do to int, you can do to Any, but not the other way round

gleaming rover
#

extract into a variable with a type annotation?

dusk comet
#

Any in args and Any in return type may not match
so this is correct code

lusty scroll
#

and it does so, I guess, because you could add a non-int to the list later

gleaming rover
#

okay I don't know how mypy coerces to Any

#

but statically, there is no way that is correct without Any coercion

gleaming rover
#

(covariant in parameter types)

#

on the other ahdn

#

okay this kinda breaks down around Any

#

never mind

#

like I said I'm not sure how typecheckers in Python (and in particular, mypy) do this, but I'm fairly sure in TS (at least), you need to explicitly cast to any

#

because Any basically means you can do anything

dusk comet
#

obj.__class__ is type(obj) - is it always true?

native flame
lusty scroll
#

is that even possible? haha

gleaming rover
#

or you could also say that type checking is just turned off around Any

#

NoReturn, similarly, is defined as the type that is a subtype of every other type and that has no members

dusk comet
lusty scroll
#

it's not a real "concrete" type

#

and NoReturn you can do absolutely nothing with

#

accurate?

gleaming rover
#

because no instances of it exist

fluid lake
#

Hey, I was wondering where I could find projects beyond my skill level to learn from / help with, and also maybe be able to ask questions to the more experienced people making it

halcyon trail
#

Just want to point out that this approach to Any is unique to python and basically exists to help facilitate transitioning from un-annotated code without having tons of false positives

#

in normal type systems, every type is a subtype of Any, but Any is not a subtype of every type

#

Any in python should really be called Untyped or something, I think

#

If you have an object of type Any in say Kotlin, you will not be able to pass it to a function that requires an int. Because Any isn't necessarily an int... so why expect this to work? You'll need to try to downcast and call the function conditionally.

#

object is the top type in Python, i.e. it is equivalent to Any in a number of other languages (kotlin, swift, scala)

grave jolt
flat gazelle
#

Dart and C# also have dynamic, which is pretty much Any

grave jolt
#

hm

#

and also other gradually typed languages

#

like tl

#

although I'm not sure

halcyon trail
#

dynamic is a better name for sure. And fair enough about TS and Dart and C#, I'm not too familiar with any of those, I should have qualified it.

#

The idea of a top type is much more common I should say, then the idea of something totally untyped, and using Any for the untyped thing instead of the top type is very rare

elder blade
#

I disagree, Any pretty nicely represents a type that allows any operation, getting any attribute, setting any attribute, and calling any method..

undone hare
#

Oh, so that's the difference with object

native flame
#

typehinting as object allows you to use things defined for all objects, typehinting as Any allows you to use things defined for any object

undone hare
#

Make sense

halcyon trail
#

I wouldn't go that far :-) but yeah, object is the top type. Any disables type checking effectively

#

Any is the default type used in many places if you don't have annotations

surreal sun
#

Anyone know the reason why the argument passed into strip, lstrip, and rstrip is interpreted as a set of characters?

peak spoke
#

how else would it be interpreted?

spice pecan
#

removeprefix and removesuffix are there to strip a substring instead of individual characters

surreal sun
red solar
#

Operation cannot be called outside a context manager.

#

is this the right wording for an error that gets raised if the function is called outside a with thingy?

boreal umbra
red solar
#

👌

boreal umbra
#

I'm sure other wordings are possible, but I knew exactly what you meant.

surreal sun
#

You could also be specific in case why it wouldn't work if you wanted to, I suppose

red solar
#

good enough, i already committed now

#

eh, it's part of the type (i.e. it's not for a specific operation, any operation on that type will raise it)

#

you can't use the type outside of that context

verbal escarp
#

oh godness, my logo is weird

#

need to work on that 😄

red solar
#

where's the logo?

grave jolt
#

seems weird

#

If you want to permit certain operations only inside with, just return a different object in __enter__

red solar
#

Currently I check that enter has been called and exit hasn’t

red solar
grave jolt
#

or if you don't want to create garbage: ```py
def enter(self):
return MyContextManager(weakref.ref(self))

red solar
#

Ohhhhhh

#

I’m so dumb

grave jolt
#

this will also play well with linters/typecheckers - they won't let someone call your_operation on MyObject

#

well, I wouldn't've come up with this if I haven't seen this pattern 😉

#

I think I saw it in asyncpg first

red solar
#

On the downside now I need to think of another name for the extra class i’ll need :/

grave jolt
#

What does the main class do?

red solar
#

Gives you atomic operations over a provided buffer

#

AtomicIntView

grave jolt
#

AtomicIntViewContextManager would be the easy but lazy way...

#

also, you should still think about what happens if someone calls MyObject().__enter__() and just doesn't ever close it

red solar
#

If you manually call stuff prefixed with _, even if it’s a special method, I figure that’s on them

grave jolt
#

dunder methods aren't private

#

I use them as partial methods all the time

red solar
#

(I still implement __del__ and release)

red solar
grave jolt
#

yeah, calling __enter__ is usually a footgun

red solar
#

If they attempt to modify or destroy the buffer before .release() gets called (either directly or from del or exit), it’ll raise an exception on CPython, so that’s nice (it won’t on PyPy :/)

grave jolt
#

You can do a bit of metaprogramming and do this: ```py
@has_context_manager
class MyObject:
@only_inside_with
def your_operation(self, a, b, c):
...

red solar
#

If __enter__ returns AtomicIntViewContextManager, would it be frowned upon to have that types __str__ method show the name as just AtomicIntView? Just so it’s nicer if people print it

grave jolt
#

that'd be confusing

red solar
#

Would anyone ever actually refer to that type by name?

halcyon trail
#

what fix suggests is an improvement but you should be ware that in python with isn't a scope

#

so the context object will be available after the context ends

#

so in practice you still probably want to have some flag and check it raise an error, if you want to be careful

red solar
#

Well that’s kinda shit…

#

They go to the trouble of making a whole context with almost lifetimes, and then it’s not a scope :/

#

Idk i’ll see tomorrow

grave jolt
#

oh right, I forgot about that

halcyon trail
#

I would probably just have one class tbh

#

and make the init a no-op

#

have everything happen in enter

#

then the object is basically useless without enter anyway

#

Hmm I guess that can't quite work

grave jolt
#

@halcyon trail if you do it the weakref way, the context object will actually be unusable after with in a scenario like this (in CPython): ```py
with MyObject(...) as ctx:
...

but, of course, people can do ```py
my = MyObject(...)
with my as ctx:
    ...
candid pelican
#

hello!!! when should i implement a abstractclass or a Protocol which case can i use each one?

grave jolt
#

oops, sorry

#

sent early

#

I hate my keyboard

#

Protocol is when you want to define a duck-typed interface people don't have to explicitly specify.

#

For example, typing.Iterable is a protocol (although now it's deprecated and you should use collections.abc.Iterable)

surreal sun
#

Would this be a good example?

from typing import Protocol


class HasFooFunction(Protocol):
  def foo(self):
    ...

class HelloWorld:
  def foo(self):
    ...

hello_world: HasFooFunction = HelloWorld() # this would work in the type checker
#

I haven't worked with protocols too much

grave jolt
#

The whole point of protocols is that you don't need to inherit from it

surreal sun
#

ah

grave jolt
#

or even know that it exists

surreal sun
#

Protocols are invariant by default, right?

grave jolt
#

hm?

surreal sun
#

wait no, wrong terminology

grave jolt
#

so if you want to type an external interface, use Protocol

#

if it's just an abstraction you implement in your code, i'd use Protocol

candid pelican
#

ic

surreal sun
#

But with Protocols, you can use "covariant" and "contravariants" right?

like:

from typing import Protocol, TypeVar

T_co = TypeVar("T_co", covariant=True)
class Foo(Protocol[T_co]):
  def bar(self):
    pass

class Test:
  def bar(self):
    pass

test: Foo = Test() # This would pass a type checker, since Test is Foo
foo: Test = Foo() # This would not pass a type checker, since Foo is not Test, by covariance rules

would this be correct? ^

candid pelican
#

thank you so much

grave jolt
gleaming rover
halcyon trail
halcyon trail
gleaming rover
grave jolt
silk pawn
#

why are floats bigger than ints in python?

from sys import getsizeof
import time

print(getsizeof(time.time()), getsizeof(int(time.time())), getsizeof(float(time.time())))

returns 24 32 24

prime estuary
#

What's the size of say 1000 on your interpreter?

silk pawn
#

lemme check

peak spoke
#

ints have some inherent overhead from the way they work while floats just wrap doubles

prime estuary
fallen slateBOT
#

Include/cpython/floatobject.h lines 5 to 8

typedef struct {
    PyObject_HEAD
    double ob_fval;
} PyFloatObject;```
prime estuary
#

2 pointers in PyObject_Head = 16 + 8 for the double = 24.

#

Ints are resizable to avoid overflow, so they contain at least 1 uint32_t and a Py_ssize_t size (64-bit).

silk pawn
#

where would i find the raw definition for the pylong object or py_ssize_t or whatever

prime estuary
peak spoke
#

Does the int store a number when it's 0? I recall the size being different on 0/1

fallen slateBOT
#

Include/cpython/longintrepr.h lines 85 to 88

struct _longobject {
    PyObject_VAR_HEAD
    digit ob_digit[1];
};```
silk pawn
#

ah sorry, i meant where would i find the raw definition for an integer

prime estuary
#

In Python 3 we only have PyLongObject, all ints are those.

silk pawn
#

oh

prime estuary
#

In Py2 there were both Int and Long types, the name was kept to avoid changing API names unnecessarily.

#

Yep, so time.time() produces a 2-digit integer it seems.

silk pawn
fallen slateBOT
#

Include/cpython/longintrepr.h line 53

typedef unsigned short digit;```
prime estuary
#

If it's PYLONG_BITS_IN_DIGIT == 15 yes, if it's 30 it's uint32_t (which is a type that is required to be 32 bits).

silk pawn
#

oh oops

#

and how many bytes does PyObject_VAR_HEAD take up

prime estuary
#

It's PyObject_Head + a Py_ssize_t for the len(obj) value - that's required to be basically the size of a pointer.

silk pawn
#

i'm not sure how to run that snippet

prime estuary
#

The long bit count depends on if the interpreter is 32/64-bit, this way doing multiplication on a digit can be done without overflow.

silk pawn
#

sorry, some of this is going over my head, i only have a rudimentary knowledge of C

prime estuary
fallen slateBOT
#

Include/object.h lines 105 to 118

typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
} PyObject;

/* Cast argument to PyObject* type. */
#define _PyObject_CAST(op) ((PyObject*)(op))
#define _PyObject_CAST_CONST(op) ((const PyObject*)(op))

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;```
silk pawn
#

ah ok that makes sense

prime estuary
#

There's the definitions of those, _PyObject_HEAD_EXTRA can be ignored, it adds some extra stuff for refcount debugging in a special mode.

silk pawn
#

cool cool, thanks for the help! this makes a lot more sense now

prime estuary
#

PyVarObject is used for immutable container types like bytes, tuple, int, so you can allocate them with the data array attached to the end.

elder blade
fallen slateBOT
#

@elder blade :white_check_mark: Your eval job has completed with return code 0.

001 | 24
002 | 28
003 | 28
004 | 28
elder blade
#

For 0... Apparently

#

Maybe because it can skip a pointer (NULL)?

prime estuary
#

It's allocating 0 digits, as a special case the size is then set to zero.

#

PyLong is sneaky, and sets the number of digits to negative for negative numbers.

fallen slateBOT
#

Objects/longobject.c lines 124 to 125

/* Fast operations for single digit integers (including zero)
 * assume that there is always at least one digit present. */```
prime estuary
#

Given there's only 2 0s in existence (0 + False), it probably isn't important to care about 8 bytes.

pallid moon
#

What do you think about an object that compares equal to any other

#

It can be used for this:

#

for k, v in os.environ.items() & {('LANGUAGE', Any), ('USER', Any)}:

#

So one can work with the required keys and values right away

elder blade
#

You can trivially create one yourself! ```python
@lambda cls: cls()
class Any:
def eq(self, other):
return True

#

I don't know how you want __ne__ (not equal) behaviour to be

unkempt rock
#

I'd say always False

#

Because nothing is equal to anything

#

It is only one thing

pallid moon
unkempt rock
#

and it is consistent with the equality

elder blade
pallid moon
#

I'll take me a few minutes understand the decorator, but noice

#

Oh, it doesn't work

#

Because Python returns Any as the key value, not the original key value 😛

#

Also, I had to add a __hash__ returning True to make it work at all

#

Strangely, it doesn't work at all here:

# 
@lambda cls: cls()
class Any:
    def __eq__(self, other):
        return True

    def __hash__(self):
        return True

mapping = {
    'LANGUAGE': 'en_US',
    'USER': 'adelfino',
}
for k, v in mapping.items() & {('LANGUAGE', Any), ('USER', Any)}:
    print(k, v)
elder blade
#

The thing, you're trying to AND an iterator?

bright galleon
#

would anyone mind if i asked a build question in here ?

pallid moon
#

How do I make the code show like Python code?

elder blade
elder blade
bright galleon
#

gotcha

pallid moon
#

@elder blade sorry, I mean, highlighting so I can run it and show you

elder blade
#

It looks like you have an accidental /, other than that it should probably work?

pallid moon
#

!e

fallen slateBOT
#
Command Help

!eval [code]
Can also use: e

*Run Python code and get the results.

This command supports multiple lines of code, including code wrapped inside a formatted code block. Code can be re-evaluated by editing the original message within 10 seconds and clicking the reaction that subsequently appears.

We've done our best to make this sandboxed, but do let us know if you manage to find an issue with it!*

pallid moon
#

!e

@lambda cls: cls()
class Any:
    def __eq__(self, other):
        return True

    def __hash__(self):
        return True

mapping = {
    'LANGUAGE': 'en_US',
    'USER': 'adelfino',
}
for k, v in mapping.items() & {('LANGUAGE', Any), ('USER', Any)}:
    print(k, v)
fallen slateBOT
#

@pallid moon :warning: Your eval job has completed with return code 0.

[No output]
pallid moon
#

But:

#

!e

import os

@lambda cls: cls()
class Any:
    def __eq__(self, other):
        return True

    def __hash__(self):
        return True


for k, v in os.environ.items() & {('LANG', Any), ('LC_CTYPE', Any)}:
    print(k, v)
fallen slateBOT
#

@pallid moon :white_check_mark: Your eval job has completed with return code 0.

001 | LC_CTYPE <__main__.Any object at 0x7fc21304e650>
002 | LANG <__main__.Any object at 0x7fc21304e650>
pallid moon
#

!e import os; print(os.environ)

fallen slateBOT
#

@pallid moon :white_check_mark: Your eval job has completed with return code 0.

environ({'LANG': 'en_US.UTF-8', 'OMP_NUM_THREADS': '5', 'OPENBLAS_NUM_THREADS': '5', 'MKL_NUM_THREADS': '5', 'VECLIB_MAXIMUM_THREADS': '5', 'NUMEXPR_NUM_THREADS': '5', 'PYTHONPATH': '/snekbox/user_base/lib/python3.10/site-packages', 'PYTHONIOENCODING': 'utf-8:strict', 'LC_CTYPE': 'C.UTF-8'})
pallid moon
#

I'd say the second example is clear to me, Python returns the object in the set, but the first example, I don't understand why it doesn't work like the second

flat pond
#

Can someone explain to me why this code is returning two classes and how do I get the class I want? I'm guessing not the pycache one. 😦

        for key, obj in inspect.getmembers(module, lambda member: inspect.isclass(member)):
            if issubclass(obj, BaseModel):
                print(obj)

Returns:

<class 'src/models/post.py.Post'>
<class 'src/models/__pycache__/post.cpython-39.pyc.Post'>
flat pond
#

Never mind. I'm using os.walk and I need to exclude the pycache directories. Duh!

tough merlin
quick snow
#

I can't remember: Can I get the code object that was used to create a class after the fact ( = without using inspect)?

quick snow
quick snow
red solar
#

is it ok for mixins to depend on multiple attributes?

#

(it seems ok, but also seems like a bit iffy, like if you change the class that uses the mixin, and then the mixin breaks, and that might be annoying to debug?)

verbal escarp
#

if i ever need something like a mixin, i usually go all-out with proxies or delegation/composition and skip the whole subclassing

#

although i appreciate having an actually working solution in python for multi-subclassing

lusty scroll
lusty gust
# 
class A:
    def __init__(self):
        self.foo = "hello"
        self.__setattr__("bar", "hello")

a = A("my")

print(a.foo)
print(a.bar)
#

foo have a otocomple but bar dont have otocomplete

#

how can add bar otocomple list

bright galleon
peak spoke
#

Not really suited for this channel, but no ide can provide code analysis like that for dynamic code

red solar
verbal escarp
#

ehh.. that looks overly convoluted, but that may be just me

red solar
#

it is, i've been trying to simplify it over the past few days

verbal escarp
#

ah, you want to use OOP as wrapper there

red solar
#

um... ig?

verbal escarp
#

why do you need multiple classes?

#

or why do you think you need multiple classes?

red solar
#

AtomicBytes is the base, AtomicInt is AtomicBytes but with arithmetic operations as well and the type hints are int instead of bytes

#

and then AtomicUint is just unsigned

#

i don't see how these would be a single class

maiden nacelle
bright galleon
#

How do we build Python PGO without installing it ?

#

see last comment there

#

per : Unable to find package 'pythonx86'
Cannot locate python.exe on PATH or as PYTHON variable

red solar
#

This isn't exactly a solution, but do you need the latest version of python for a "xeon machine we are testing against nuclear and solar radiation"? Can you not just build an older version that doesn't require bootstrapping?

#

@grave jolt any chance you're on? i have more questions about your __enter__ solution from yesterday :/ as in i struggled to make it work

grave jolt
red solar
#

yay 🙂 so you told me i shouldn't return self from __enter__ and instead i should have a new type with the operations i wanna make available

#

so that linters can help the user not call those operations outside of the context manager

feral cedar
red solar
#

lol. i don't know that!

red solar
# grave jolt what... is your quest?

anyway, i really want __enter__ to return a type derived from self so that it can be used anywhere an object of type(self) can be used without breaking type hints

#

is this a bad ideal to have?

# 
def print_atomic_val(a: AtomicIntViewCM):
    print(a.load())

b = get_some_buffer()
atom = AtomicIntView(buffer=b)
with atom as a:
    print_atomic_val(a)
#

like the type hinting seems a little off there

#

unless AtomicIntViewCM derives from AtomicIntView, then you can just have AtomicIntView in the type hint

#

(but also don't think it's possible to do that nicely in Python, which leaves me stuck)

grave jolt
#

@red solar ```py
with atomic_int_view(buffer=b) as buf:
assert isinstance(buf, AtomicIntView)

#

why the 😒 ? 😦

red solar
#

so how do i solve this? have them both derived from some baser class?

#

wait does isinstance take inheritance into account?

grave jolt
#

actually, what was your original issue?

#

I was thinking of this: ```py
@contextmanager
def atomic_int_view(arg1, arg2, arg3):
view = AtomicIntView(arg1, arg2, arg3)
try:
yield view
finally:
view.close()

red solar
#

your issue was that __enter__ shouldn't return self, i should have a new type so that linters could help (i wanted to prevent operations outside of with...)
my issue is that I don't want type hints working with the type returned by __enter__ to need to list some special type

red solar
grave jolt
#

grrrr

#

it's from contextlib

#

!d contextlib

fallen slateBOT
red solar
#

so do you have AtomicIntViewManager.__enter__() -> AtomicIntView? because i can live with type names like that

grave jolt
#

no I mean, just have people use this function

#

and not create AtomicIntView directly

red solar
#

oh

#

hmmm

#

why yield?

#

is that important?

grave jolt
#

maybe even make the class private, like AtomicIntView

#

@red solar contextmanager works roughly like this:

class _GeneratorCtxManager:
    def __init__(self, generator):
        self.generator = generator

    def __enter__(self):
        return next(self.generator)

    def __exit__(self, exc_type, exc_value, exc_tb):
        if exc_type is None:
            next(self.generator)
        else:
            self.generator.throw(exc_type, exc_value, exc_tb)

def contextmanager(fn):
    def new_fn(*args, **kwargs):
        return _GeneratorCtxManager(fn(*args, **kwargs))
    return new_fn
#

When it enters the context (i.e. the with block begins), it fetches an item from the generator. When it exits the context, it returns control back to the generator to let it clean up

red solar
#

ohh, ok that answers one question, but raises another; in __exit__ am i supposed to be checking exc_type and dealing with it?

grave jolt
#

depends