type-hinting

[mortal fractal]

src/pyffstream/encode.py:193:8 Missing attribute annotation [4]: Attribute selectorof classFileStreamValshas typestr but no type is specified.

even though it knows the type...

Eric has really angered pyright's CI it seems

[soft matrix]

Yes

You're welcome for that maybe

[oblique urchin]

I was just going to complain to him about some NoReturn problems

[trim tangle]

https://pypi.org/project/flake8-pep585/ 🎉

[blazing nest]

I am in great pain, I have a TypedDict where either of two fields exists. It seems like I need to create two TypedDict's and then Union them

😩

[trim tangle]

@blazing nest you won't be able to narrow though, would you?

unless you're using pyright and mark them as @final

[blazing nest]

I can't no, so I can't inherit and narrow them.

[trim tangle]

no I mean, like ```py
if "foo" in x:
...
else:
...

[grave fjord]

Do you have an example of what it is?

[blazing nest]

Type of TypedDict field "type" cannot be redefined
  Type in parent class is "Literal[2, 3, 4]" and type in child class is "Literal[2]"

[trim tangle]

You can do this if they have something in common:

class A(TypedDict):
    x: int
    y: str

class B(A):
    z: bool

class C(A):
    w: list[int]

[grave fjord]

Might be better to use an attr.frozen ?

[blazing nest]

Yup that's what I am doing

[trim tangle]

I meant type narrowing when you're actually using the dict
<#type-hinting message>

[blazing nest]

If the class looks like this: ```python
class ABC(TypedDict):
a: NotRequired[str]
b: NotRequired[str]

Then (to the type checker) there's 3 cases:

- `a` and `b` are both present
- only `a` is present
- only `b` is present
- none of them are present

[grave fjord]

Show your actual code 💢

[trim tangle]

I think we're talking about different things

What type checker are you using?

[blazing nest]

With ```python
class _AABC(TypedDict):
a: str

class _BABC(TypedDict):
b: str

ABC = Union[_AABC, _BABC]

We now have:
- only `a` is present
- only `b` is present

[blazing nest]

Oh?

[trim tangle]

Suppose now that you use ABC in a function. What will you do with it?

def f(abc: ABC):
    # ???

[blazing nest]

def f(abc: ABC):
    a = abc.get('a')
    if not a:
        b = abc['b']
        ...
        return ...

    ...
    return ...

[trim tangle]

the type checker will yell at you

[blazing nest]

Yup, but I am looking at getting it not to do that 😅

[trim tangle]

In Pyright you can do this:

def f(abc: ABC):
    if "a" in abc:
        # abc: _AABC
    else:
        # abc: _BABC

but only if _AABC and _BABC are @final. (explained in https://github.com/microsoft/pyright/issues/1899)

mypy doesn't support it. Moreover, it prohibits @final on a TypedDict 😄

[blazing nest]

Ah yeah I realized as I put my code above to test in the editor it didn't narrow the type, but yes using unions like this is the only way to accomplish it.

Which brings me great pain at the moment 😔

[trim tangle]

With your particular code, a could also be a ""

but I don't think type checkers will understand this complex relation at a distance anyway

[blazing nest]

Yeah I changed it to is None but it still didn't catch on

...yeah I wish I didn't have to do this: https://paste.pythondiscord.com/julimecuba.properties

[trim tangle]

@blazing nest Why are you using typeddicts anyway?

[blazing nest]

I am typing Discord's payloads

[trim tangle]

Why not have proper classes and use something like pydantic or dataclass_factory?

[blazing nest]

What should the typehints be for the dictionaries that their __init__ methods take?

That's what this is, and because I have a few places where I operate on the dictionaries directly

[trim tangle]

Mapping[str, Any]

you're not getting any benefit from adding a precise type there, you're still going to pass the data discord gives you directly

If you're doing something directly with the data after the object is created, you're coupling your library to the details of the Discord API.

[blazing nest]

Right but I am building upon that. First I have a few subpackages that operate directly on top of the API, then I build more packages upon this that abstract away more and more of the API.

https://github.com/wumpyproject/wumpy/blob/main/library/wumpy-rest/wumpy/rest/endpoints/channel.py#L5-L9

[rough sluice]

library/wumpy-rest/wumpy/rest/endpoints/channel.py lines 5 to 9

from discord_typings import (
    AllowedMentionsData, AttachmentData, ChannelData, EmbedData,
    FollowedChannelData, InviteData, MessageData, PermissionOverwriteData,
    ThreadMemberData, UserData
)```

[trim tangle]

I guess that's fair

[blazing nest]

On top of this, my models will sit which wrap the data more memory-efficiently and in a more abstract way but if someone simply wants to use the requester and get the benefits of typed dictionaries then feel free y'know 😅

[trim tangle]

well, it's not very useful given how TypedDicts don't support narrowing, generics and some other stuff

You can have "data transfer objects" which are dataclasses or pydantic models, that you deserialize from Discord's data

[blazing nest]

Indeed, I would've loved that when writing the actual types.. but where would the user run into this? I take the hurt up-front inside discord_typings.

[trim tangle]

If you receive a union of typeddicts from the API, you can't narrow it

no matter what you write in discord_typings

[blazing nest]

I can

[trim tangle]

yes, in Pyright

But many people use PyCharm, or have mypy in their CI

[soft matrix]

I'm pretty sure more people are using pyright in CI

Also it's only really for the dev right

Like you can use pyright if you want

[trim tangle]

Are you saying that more people are using pyright than mypy in the CI?

[soft matrix]

No

Definitely not

Just more are now

[trim tangle]

ah

maybe

Does PyCharm have a plugin for pyright?

[soft matrix]

No 😔

I'd probably still use it otherwise

[blazing nest]

I know a fair share of people using MyPy in the CI, who aren't my target audiences and simply won't get the benefits of discord_typings until Mypy is able to understand the narrowing

sadly 😔

[trim tangle]

But if you use proper classes, those people will be able to benefit from it

[blazing nest]

Yeah so they can use the subpackage I have with models, which abstracts away from the inner payloads and is built on top of it.

Just saw #reddit and https://www.reddit.com/r/Python/comments/sa8ofv/pep_646_variadic_generics_has_been_accepted_for/

I kind of forgot that PEP 646 required grammar changes, does that mean that it's not possible to support in typing_extensions O_o?

[soft matrix]

Not with the syntax

But it adds Unpack which is the same thing to a type checker

[blazing nest]

Oh phew

[soft matrix]

I'm sure Jelle could tell you more cause they are actually adding it

[oblique urchin]

https://github.com/python/typing/pull/963

it already required the most horrific hacks in typing-extensions history 🙂

[soft matrix]

Is this why you think runtime typing checks should be removed ;)?

[oblique urchin]

yes

[blazing nest]

btw, the constraints mentioned, are those present even if you use typing_extensions's versions of the typing classes?

[oblique urchin]

for ParamSpec we already had to do horrible hacks because typing.py validates too hard that you really give it a typevar

[oblique urchin]

there is no typing_extensions.Generic

or typing_extensions.Callable

and typing.Callable really wants its first argument to be a list

which is why typing_extensions.ParamSpec is a list and a TypeVar at the same time

[blazing nest]

Ah 😅

[oblique urchin]

and there's still some things PEP 612 allows that we couldn't get to work

[blazing nest]

Yeah I am reading the diff, and uhhh wow

[oblique urchin]

the PEP 612 or the 646 one? 😄

[blazing nest]

The one linked above

[blazing nest]

Haa this ever been fortified into a PEP or are there too many people against it?

[oblique urchin]

Not really, it's just something that comes up when implementing new features

I guess it might be worth writing up the general idea somewhere

[blazing nest]

I am not sure I would consider it a breaking change, because of how it's an error in the first place.. it's like complaining that adding new syntax breaks their reliance of SyntaxError lmao

[oblique urchin]

I think Guido is also on board now that we should avoid runtime checks in typing.py

[oblique urchin]

Sorry are you talking about a specific change?

[pastel egret]

Perhaps TypeVar, ParamSpec and TypeVarTuple should just take **kwargs?

[blazing nest]

I was reflecting on whether removing the type checks would pose as a breaking change (and as such, be rejected until Python 4.0)

[oblique urchin]

oh definitely not

[blazing nest]

I can imagine it would be quite nice to remove it all and subsequently focus more on how the API of the objects are, making it easier for introspection to access different data.

[soft matrix]

I feel like you shouldn't be relying on them anyways and a type checker should tell you what's up

[oblique urchin]

yes, it might be worth removing the existing checks. So far I've mostly just pushed for being permissive in new features and against adding any new strict checks

[oblique urchin]

Exactly, it's the job of static type checkers to tell you when you're using typing stuff wrong

[soft matrix]

Just ignore my pr to cpython

[oblique urchin]

I like the one about subclassing NewType, is there another?

[soft matrix]

No

I guess it's a different thing tbf

[oblique urchin]

In that case we throw an error anyway, so it's nice to make it comprehensible

without your diff it says something like "got 2 arguments, 1 expected" right?

[soft matrix]

Yeah

It was pretty weird

[blazing nest]

Do we have an official reasoning for why the checks were added in the first place or did everybody just implement types with the checks (since there's an expection that's tested and there's boundaries that someone has to enforce)?

[oblique urchin]

I guess it was just a decision taken when typing.py was first implemented

I wasn't around at the time so I don't know if there was any discussion about it

[soft matrix]

Guido is meant to have an alt here or something

I'm sure you could ask them :)

[oblique urchin]

this is the oldest version of typing.py I could track down. I guess at the time types were still always actually types ```+def _type_check(arg, msg):

• """Check that the argument is a type, and return it.
• As a special case, accept None and return type(None) instead.
• The msg argument is a human-readable error message, e.g.

•    "Union[arg, ...]: arg should be a type."
  

• We append the repr() of the actual value (truncated to 100 chars).
• """
• if arg is None:

•    return type(None)
  

• if not isinstance(arg, type):

•    raise TypeError(msg + " Got %.100r." % (arg,))
  

• return arg

[soft matrix]

Huh

[mortal fractal]

looks like i get a regression in latest pyright

# pyright: strict
def sec_to_str(seconds: float | int) -> str:
    m, s = divmod(abs(seconds), 60)
    h, m = divmod(m, 60)
    return f"{'-' if seconds<0 else ''}{int(h):02d}:{int(m):02d}:{int(s):02d}"

oh god the type of divmod is so opaque; I mean I know why, but still

this no longer type checks in the version of pyright just published, but it type checked before

I'll make an issue

# pyright: strict
def sec_to_str(seconds: float) -> str:
    a = divmod(abs(seconds), 60)
    return str(a)

for some reaosn it can't find the type of a

[oblique urchin]

he apparently added bidirectional inference for lambdas, I wonder if that got a bad interaction

[mortal fractal]

if I change to 60. it works fine, I will have to actually read this typeshed to figure out if pyright is wrong or not

I'll file a minimal issue for now https://github.com/microsoft/pyright/issues/2917

https://tenor.com/view/whack-a-mole-cute-cat-funny-gif-14926253

insider pic of eric fixing pyright bugs

note the abs() is required

[oblique urchin]

https://github.com/microsoft/pyright/commit/24b6cf0c85578e73c0751d39ce82acbb0a099104 hmmm

[mortal fractal]

could be related 🤔

my typing-sig message about annotated ordering went well; I was expecting pushback heh

[oblique urchin]

now to convince Eric to change Pyright's behavior on Annotated[ClassVar] 🙂

as I recall this isn't actually specified in any PEP

655 is the first that explicitly deals with the interaction

[mortal fractal]

is it already okay in mypy?

[oblique urchin]

I don't think so

[mortal fractal]

@soft matrix you got reopened 😮

\o/ hurrah my CI is passing again

[soft matrix]

I saw but I don't know why

[soft matrix]

oh so i did actually break ci

nice

[proven fog]

def with_commit(func) -> Callable[[Arg, KwArg], None]:
    def inner(*args, **kwargs):
        func(*args, **kwargs)
        commit()

    return inner 

Can Callable used in this way for a nested function?

[trim tangle]

You might want to look at ParamSpec

!pep 612

[rough sluice]

**PEP 612 - Parameter Specification Variables**

Link

Status

Accepted

Python-Version

3.10

Created

18-Dec-2019

Type

Standards Track

[trim tangle]

I don't think mypy supports it fully, but pyright does

[proven fog]

ok thank you

this is exactly what i need LMAO

tysm

[mortal fractal]

@oblique urchin since the other ordering of Annotated+ClassVar/Final is disallowed at runtime by typing.py it doesn't make sense to try to convince Eric to change it for now I think; if it's not in the PEP the best we could do is propose it and put it in typing_extensions first and then ask

I think perhaps the runtime behavior for no Annotated on the outside may be justified by "The first argument to Annotated must be a valid type" in PEP 593 (Annotated)

[oblique urchin]

oh I didn't realize that. I'd be supportive of changing that in typing.py

[mortal fractal]

you can comment on the ML again and maybe Eric will respond, or we can open something on typing github or some such (or new thread, or?)

the runtime error is e.g. TypeError: typing.Final[int] is not valid as type argument

[oblique urchin]

I feel like I have reached my budget of new typing-sig threads for a while 🙂

[mortal fractal]

what do you think the best approach is if I wanted to? (typing-sig thread?)

[oblique urchin]

I think just bugs.python.org may be enough

[mortal fractal]

okay I will open in a little bit and link in case you want to comment on it

[mortal fractal]

okay my code now type checks pyright strict, mypy strict, and pyre non-strict except for that bug I reported

[oblique urchin]

time for pytype

[mortal fractal]

I don't think that's possible; pytype is too far behind on support relative to my project

I'm currently Python 3.9+, but I use | union notation with __future__ which isn't supported by pytype I believe

they support it in typeshed only, or something weird like that (only pyi files)

I'm not interested in attempting pyre strict, because they require annotations in a bunch of extra spots I don't care for

I'm actually mostly compatible with mypyc too, which requires some extra annotations

[mortal fractal]

hmm, pyright should be using bidirectional inference here maybe?

# pyright: strict
def get_set(setlist: list[set[str]]) -> set[str]:
    return set().union(*setlist)  # Return type, "set[Unknown | str]", is partially unknown

I can do set[str]() obviously to fix it

is that what mypy is doing? mypy resolves set[str]

I don't know if this is late binding behavior or bidirectional inference on mypy's part

[mortal fractal]

noooooo mypy why are you doing me like this src/pyffstream/cli.py:186: error: Incompatible return value type (got "Tuple[Set[str], ...]", expected "Tuple[Iterable[str], Iterable[str], Iterable[str]]") [return-value]

[blazing nest]

Does Tuple[Set[str], ...] guarantee that there's at least XYZ elements? It could technically have 0 elements

[oblique urchin]

the error is right. But presumably the code is actually right, so maybe if mypy was smarter it would recognize the tuple has three elements 🙂

[mortal fractal]

yes, although pyright couldn't figure it out either so whatever

it was statically guaranteed to have 3 elements, but they they couldn't follow the 3 element tuple through a comprehension

which...makes perfect sense, but still sadness

[oblique urchin]

I think I made pyanalyze able to do that 🙂 but I had to limit the length of the iterables it handled that way for perf reasons

[mortal fractal]

I think bidirectional inference should be fine here as long as you whitelist it to stuff constructed on the same line? or maybe you only want it for empty literals (..and set()?)

[mortal fractal]

oh hey, Guido is on board for the Annotated change

[sick lynx]

any direct way to get the class name that defined a certain method?

better yet, get all classes in an inheritance tree that defined a certain method

[trim tangle]

Why do you want that?

[tranquil turtle]

look at cls.__subclasses__()

[sick lynx]

because some people in other team will have to subclass a certain ABC

and up until now they used to implement a method that from now on, is implemented in the ABC

how to i write code here? to look like it's formatted

basically i'm going to raise en exception if 'mymethod' is in self. _ _ class _ _ . _ _ dict_ _ . keys()

[lethal niche]

I believe metaclass should help you out.

[sick lynx]

true but i have 10 guys in my team about 8 of them will complain if i use metaclasses

i'll just check the methods defined in the classe's dunder dict property

[hearty shell]

Hey guys, is there any timeline for mypy development? There seems to be a roadmap tab in mypy's website but it leads to a 404 on github

http://mypy-lang.org/

[oblique urchin]

nobody is working on mypy full time. the project is dependent on what volunteer contributors choose to spend their time on

(and there's already an open issue about fixing the 404 roadmap)

[hearty shell]

Yeah I figured, I just wondered because of the roadmap tab

[fierce ridge]

which is bonkers to me. where is the corporate sponsorship??

[oblique urchin]

dropbox had a full-time mypy team for a while, not sure what happened there

[fierce ridge]

maybe they pulled them all to work on pyston again

does mypy get supported through the PSF?

[oblique urchin]

no

not to my knowledge at leastt

there have been GSOC projects a couple of times, that has worked pretty well and yielded a few new core devs

[trim tangle]

pyright moment 👀 🪟

[mortal fractal]

@oblique urchin can I submit the change for my bpo ticket myself? a concern I have is removing the final/classvar parts of the check remove the only use of some of the function arguments in the function (should I not worry about that?) hopefully it doesn't break any tests too 👀

[oblique urchin]

yes please! you may also have to adjust some tests

[mortal fractal]

it would be my first cpython submission but I already signed the CLA a while ago so that should pass immediately

[void panther]

Was wondering about that myself a bit, there's 3 checkers I know of that were implemented by companies instead of extending mypy

[oblique urchin]

pytype predates or nearly predates mypy for what it's worth

[mortal fractal]

yeah, I wish dropbox had someone working part time in official capacity on mypy to help deal with triage and PR backlog

but oh well

[mortal fractal]

mypy -> dropbox
pytype -> google
pyre -> facebook
pyright -> microsoft

are the ones I know of offhand, I think

[void panther]

I guess it's because of not exactly fitting their internal requirements so doing their own thing instead of mypy plugins or some internal fork

[trim tangle]

sobolevn recently joined the mypy team, and iirc he is working full-time (?) on open source stuff. not just mypy tho

[oblique urchin]

correct

[mortal fractal]

do I assign pyanalyze to quora 😉

[oblique urchin]

yes

[mortal fractal]

I haven't tried pyanalyze, but when I looked at it I think it's focused more on lints that intersect with typing knowledge than strictly type checking (?)

[oblique urchin]

it's a full-ish type checker. it started out as a more of a linter but I have added support for nearly all of the type system

definitely not as polished as many of the others though

[trim tangle]

I feel like Python linters are really missing out on type information.

sobolevn mentioned that he was working on some sort of solution for this in some talk, but I am not sure there is any progress

Stuff like "don't += to a string repeatedly" or "don't match on a non-@final dataclass" could be really useful

[oblique urchin]

open an issue on pyanalyze and I'll see what I can do 🙂

[trim tangle]

eh, it's not much of an issue

more of a general observation

I meant that there are existing linteres like flake8 that don't know anything about types, unfortunately

[fierce ridge]

my cynical view is that they are all looking to compete with msoft on vs code and to make some kind of play into the enterprise tooling market like what jetbrains does

[mortal fractal]

@oblique urchin looking at pyanalyze right now

looks like it's missing some assignment expressions

[void panther]

and open source cries with everything so fragmented

[oblique urchin]

walrus operator? that's possible, we don't use those internally yet so the support isn't well tested

[mortal fractal]

yeah, there's a bunch of false positives for undefined_name where it was assigned with a walrus

.....I didn't even think I had this many walrus in my code

[oblique urchin]

interesting, do you have a code sample? not sure from the code how that could happen

[mortal fractal]

yes, I am working on making one right now

[oblique urchin]

thanks!

[void panther]

just did a search an out of 15 walruses in the project I have open only one is not an if checking the result of a get against a default

[mortal fractal]

def func(myvar: str, strset: set[str]):
    if (encoder_type := myvar) and myvar in strset:
        print(encoder_type)

encoder_type undefined name

obviously this walrus use makes no sense I was just distilling the error

seems to be related to using the walrus with other operators like the and, I guess

[oblique urchin]

yeah that's probably it. thanks, I'll fix it

[mortal fractal]

hmm, it doesn't like a toy versioning class I made

[soft matrix]

PyAnalyse actually importing your stuff is one of its most powerful features

[mortal fractal]

@oblique urchin it doesn't like that NotImplemented can't be assigned to bool, but I think this use is typical pattern: https://bpa.st/7LBA but I guess subject to debate (?)

[oblique urchin]

yeah I should probably special case that

[mortal fractal]

pyanalyze is real upset with configparser usage I'll try to figure out an example in a bit

[leaden oak]

fwiw, black is under the psf github organisation but this doesn't really grant us anything special, we get some admin help and could probably access legal services (from the PSF) more easily, but other than that I don't know what we get.

[void panther]

a cool prefix! /s

[leaden oak]

fair enough 😄

[mortal fractal]

@oblique urchin yeah, so the walrus thing, NotImplemented, a ton of anger at configparser, some weird anger after an assert thing I haven't looked at that's in a real ugly part of my code anyway, and one typing bug that pyanalyze found

I had set something to Iterable when I meant Collection because I was checking truthiness

it was used correctly in the code, but typed incorrectly

[oblique urchin]

thanks for checking! is this an open source project?

[mortal fractal]

yes, do you want to run on it directly?

[oblique urchin]

yes, would like to take a look

[mortal fractal]

https://github.com/GBeauregard/pyffstream it passes pyright strict and mypy strict and almost pyre not strict except for the bug I reported

of course that doesn't mean it's free of bugs (even typing ones)

it's usually pretty good for looking at typing things because it's not an enormous project, but it's just large enough to run into weird edge cases for things and provide a playground for me to mess around with stuff

and it's mostly stdlib python (except for platformdirs/rich)

[soft matrix]

Self type just got accepted @oblique urchin 🎉

[oblique urchin]

hooray!

[mortal fractal]

anyway you've found my (not so) secret playground for python stuff now 😛

[soft matrix]

Should I open a pr with the stuff to cpython for it with a copy of the stuff from typing_extensions and the extra check for Tuple[Self, Self] or will you do it?

[oblique urchin]

please do it, I can review

[mortal fractal]

do you see what I mean by configparser anger? 😛

[oblique urchin]

yeah I think it crashes on some overloaded typeshed methods. weird that I didn't run into that before. I'll fix it

[mortal fractal]

cool, glad it's helpful

[fierce ridge]

i see, is there not enough funding flowing through the PSF to provide full-time funding for projects like mypy?

[oblique urchin]

I'm not aware of any efforts to get PSF funding for mypy at all, but I don't think the PSF has a lot of money for this sort of thing

I know PyPA has been able to get funding for a few projects

[fierce ridge]

i remember working at a large insurance company, they would buy everyone a $1000 steelcase chair and a $250 humanscale keyboard tray, but wouldn't donate a cent to open source software that our team depended on

right now i'm at a nonprofit so it's kind of excusable in my current org

[leaden oak]

the PSF last year was able to scrape enough money to pay for the CPython developer in residence position but other than that not really

all of the PyPA money is rewards and grants from third-parties, and the money is usually focused on a specific project / goal

[fierce ridge]

wild. doesn't ruby get a bunch of corporate funding?

[leaden oak]

why pay for something when you can get it for free and then ask for free support when it doesn't work :)

[terse sky]

well, tragedy of the commons. not really your company's fault per se.

[leaden oak]

I think black at this point has more core maintainers than mypy which surprises me (although we have a bunch of inactives and I'd guess mypy does too)

[terse sky]

i was in this exact conversation in a C++ discord re clang, gcc, etc

Yeah, to me that is one of the overall biggest downers of python + type annotations + mypy/whatever

compared to a proper statically typed language

[oblique urchin]

I believe it's 10 for both

[terse sky]

not so much any principled objection towards optional/gradual typing

it's just type annotations/mypy/etc are always going to feel like second class citizens, relatively speaking

[soft matrix]

Well maybe one day with enough gradual syntax changes and stuff we can get it to be better

[terse sky]

in a statically typed language where types not working is a huge deal breaker instead of something you slap with # ignore

it's different

I'm sure it will get better, but I think it will always be different. But it depends what you want like anything.

[trim tangle]

$1000... chair? is that a thing?

[terse sky]

it's not just a thing. It's the only thing.

[buoyant swift]

I've got a standing desk for a really low price

[trim tangle]

wow you're donating 100% of your profits to mypy. good job

[trim tangle]

It's about/slightly below the average monthly pay of a junior programmer here, so that sounds pretty wild to me

[terse sky]

In the US, your doctor will charge you 1K for smilling and saying hello.
So, spending 1K+ on your desk, chair, keyboard, etc, to make sure you have good ergonomics, is just a no brainer for a well earning developer

if you have back, neck, or wrist issues, even medium/moderate ones, it's going to effectively cost you much, much, much more than that, between medical bills, damage to your career, etc

also in case it wasn't clear, my previous comment was a joke. I've worked with chairs of wildly varying quality and price. Pre covid I had a $100 office chair from amazon.

It was totally fine for sitting at a couple hours at a time, a few times a week. Even when I worked from home one day a week it was still ok. But once I went 5 days a week, I started having issues

and pretty shortly I bought the same keyboard, and same chair, that I have in the office, which are silly expensive... but it's your health

[soft matrix]

@trim tangle is the pyright playground down?

[mortal fractal]

0:14:16 load avg: 1.24 running: test_socket (8 min 38 sec)

😡

iirc we skip over test_socket in our PGO in gentoo because it has a tendency to hang

probably same thing here

[trim tangle]

oops... let's see

ah

I forgot to pay for my VDS

should be up @soft matrix

[soft matrix]

Thank you!

[trim tangle]

almost

it's deployed in a hacky way

now it should fully work 👍

[mortal fractal]

@oblique urchin we can't make the change you suggested, I think. It's used in a lot of places and would cause things like Optional[Final[int]] to no longer error. I can make tests pass by removing the type check from annotated itself and just processing strings into forward ref, but that's probably not ideal either. not sure if I should make a special function to type check for annotated, and I don't see any helpers that encapsulate "valid type annotations" more generally heh

[oblique urchin]

well maybe we should make Optional[Final] not error

could also just add a parameter to _type_check that makes it more permissive, and use that in Annotated

[mortal fractal]

yes, I was thinking that

(the second option)

[fierce ridge]

not for the home office of course. but have you seen the prices on brand name office chairs?

im sure there are bulk discounts from dealers for large companies that need 100 for an office or whatever

[jovial sundial]

Ok I have a type hinting question. I have a base class where I define at init a dict that has class A instances for values. I then have a derived class that uses that dict, but instead of A it uses a subclass of A, B. How can I make type checkers understand that the dict value in the derived class has the methods of B?

Other than just type hint the variable again the line before

[trim tangle]

class HasDict(Protocol):
    @classmethod
    @property
    def the_dict(cls) -> Mapping[str, Self]:
        ...

if your type checker doesn't support Self from typing_extensions yet, ```py
Selb = TypeVar("Selb", bound="Some")

class Some:
@classmethod
@property
def the_dict(cls: type[Selb]) -> Mapping[str, Selb]:
...

[soft matrix]

typing.Self should be tomorrow

[oblique urchin]

you have a lot of trust in the CPython core devs

[mortal fractal]

the is_class that's there already effectively does this for Final and ClassVar so we can just make use of this

[soft matrix]

I was looking at that but I thought it was just for forward references

[mortal fractal]

@oblique urchin okay I added a trivial patch https://bugs.python.org/issue46491

[blazing nest]

Huh.. do people mean Final[Optional[int]] or do people mean like "an integer, and if so final, otherwise None and can be overriden with an integer"?

[mortal fractal]

For Optional[Final[int]]? this is a runtime error atm 🙂

hurrah

[terse sky]

Yeah, I don't understand how you can have Final on a type annotation on anywhere other than the outermost level

Final isn't really a type constructor, so passing the result of Final somewhere that a type is needed is weird

[mortal fractal]

thx for the review, I addressed it I think; let me know if there's anything else on GitHub. Not sure what Guido means by adding him back since I didn't see a status change but we'll get there later anyway

[oblique urchin]

maybe he unsubscribed

I realized your change would allow List[Annotated[Final[int], ...]]

that's probably harmless though

[mortal fractal]

@oblique urchin reviewing the code even more, this type of nonsense is already possible with the existing code and is not introduced by my change

from typing import Annotated, ClassVar, Final, TypeVar, Generic

T = TypeVar("T")

class MyClass(Generic[T]): ...

a: MyClass["Final[int]"] = 3

this example reverse engineered from the existing code

(this throws no runtime error)

[oblique urchin]

yes, it probably also works with list[] because types.GenericAlias doesn't do typechecking

it's fine

[mortal fractal]

interestingly this is doing type checking on the forward reference part, it's just under the is_class special casing (I didn't bother figuring out why when engineering this example)

[mortal fractal]

anyway, just let know via GH review when ready again. I see you were working on pyanalyze stuff so I'll check that out again later 🙂

[mortal fractal]

backporting (in cpython) is mostly automated by the bots with handling by the stable maintainers, right? (is action required of me?)

[oblique urchin]

yes, it should be automated unless there's a merge conflict

[mortal fractal]

@oblique urchin I'm going to open another bpo issue tonight or tomorrow for dataclasses to see if:

1. eric is interested in (me?) implementing the dataclasses support for Annotated on the outside, for really the same reason. This will require back and forth with eric (smith), because dataclasses uses a bespoke implementation of type hint introspection that the comments say is for performance reasons, so I would need his feedback on what is the appropriate way to modify it. Currently it begins by doing a regex inspection to find the outermost annotation, so I don't know what he will want to do. I'm capable of fixing the regex, but it's kind of ugly (and I don't know if it would negate whatever performance concerns there are?)
2. Annotated[InitVar] is currently a runtime error, but completely by accident: typing.py requires special type forms be callable() (usually with a __call__ that throws a runtime error) in order to pass as typeforms, but dataclasses' implementation of InitVar doesn't implement __call__; adding it will allow Annotated[InitVar] at runtime

I will also canvas the type checkers to make issues for changing their side of things re: annotated on the outside

[mortal fractal]

https://github.com/python/mypy/issues/12061

[tranquil turtle]

!mypy

import typing as t

T = t.TypeVar('T')

def f(a: T, b: T) -> tuple[T, T]:
  return a, b

f(1, 2)
f('', '')
f([], ()) # error

it would be great if there is an bot-command for type checking small piece of code

[trim tangle]

We had this idea. But we decided that it's quite complicated with the setup we have

If interactions (slash commands) support codeblocks I could hack something together

[tranquil turtle]

imo slash commands are inconvenient
it is more easier to just write command by hand

[mortal fractal]

@oblique urchin one more chore:
we need to verify the runtime behavior of typing_extensions for backports

and I think that's everything

[mortal fractal]

it's getting late so I'll work on this more tomorrow

[trim tangle]

Autocomplete is pretty nice. And it has some non-functional advantages such as not exposing your entire private message history to a bot, and being computationally cheaper on both Discord's hardware and bot hardware

[tranquil turtle]

is it a known bug of mypy?

it finds syntax errors in comments

# type: int - or it is recognised as old-style type-annotation?

class A: ...
class B(A): ...

class X:
    def f(self, obj: A) -> None: ...

class Y(X):
    def f(self, obj: B) -> None: ...
#   ^^^
# override - Argument 1 of "f" is incompatible with supertype "X"; supertype defines the argument type as "A"
# This violates the Liskov substitution principle
# See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

how to declare X.f so that type-checker doesn't complain about violating "Liskov substitution principle" in subclasses?

in my case, violating LSP is ok

[hearty shell]

Looks like it is recognising it as old-style annotations

And if violating LSP is ok just do what the docs suggest and # type: ignore[override], mypy certainly doesn't think so

[hearty shell]

I mean

[hearty shell]

You could I think do this

from typing import TypeVar, Generic


T = TypeVar('T')

class A: ...
class B(A): ...

class G(Generic[T]):
    def f(self, obj: T) -> None: ...

X: type[G[A]] = G[A]

Y: type[G[B]] = G[B]

X().f(A())
X().f(B())

Y().f(A())
Y().f(B())

main.py:19: error: Argument 1 to "f" of "G" has incompatible type "A"; expected "B"
Found 1 error in 1 file (checked 1 source file)

[terse sky]

The thing is that this isn't an LSP violation

Or actually, maybe it is, maybe I'm getting mixed up with which way the inheritance goes

Yeah I guess I am.

If A inherited from Bb it ought to be fine

[hearty shell]

I think it would violate it because

class A:
    def foo(self) -> ...:

class B(A):
    def bar(self) -> ...:

class X:
    def f(self, obj: A) -> None:
        obj.foo()

class Y(X):
    def f(self, obj: B) -> None:
        obj.bar()

def baz(obj: X):
    obj.f(A())

X().f(A())  # Ok
X().f(B())  # Ok, because I can use A, I should also be to use it's subclass B

baz(X())  # Ok
baz(Y())  # Error, but should be ok since Y is a subclass of X

I think the assumption is that if you are specifying that Y.f will take a more specific A, it is because it could rely on a method that B implements but A doesn't

I am not sure though x)

[terse sky]

In your example B still inherits from A

@hearty shell

That was my original statement which I admitted was wrong

Y.f should be able to take something more general and return something more specific

Compared to X.f

Without violating LSP

[hearty shell]

Yeah sorry I missread, you are right x)

[terse sky]

Yeah it all gets super confusing quickly :-)

[blazing nest]

I have a lot of optional dependencies, which I am okay with if they fail at runtime. So I need to supress the ImportError but I don't want the "X is possibly unbound" error from Pyright, and if I try to set up some fallbacks then Pyright will somehow disregard the original import

Any tips? I need imports that don't fail at runtime (when importing; it'll fail later in the code), but will always exist during development.

[magic talon]

How to you do a type hint to the class you're using? Like:```py
@dataclass
class Person:
name: str
age: int
friends: List[Person]

[worn nexus]

from __future__ import annotations
or
List["Person"]

[magic talon]

wot

What is the first one and you can use strings in type hinting?

[worn nexus]

First one allows to use testing feature, which will be implemented in 3.11 afaik. And yes, you can use string of type instead.

[magic talon]

hm...

First one can be buggy then?

[void panther]

no, it just transforms all annotations to strings

If you're not inspecting them at runtime it has no impact

[magic talon]

ah okay

[oblique urchin]

there's already a PR open that adds __call__ to InitVar fyi. But yes, not great if we have to further complicate the regex stuff dataclasses does

[hearty shell]

cough Pydantic cough

[tranquil turtle]

__future__ isnt for testing, it is for backporting features from future versions

[worn nexus]

Yea, you are right. But for me using feature from unreleased version is more like testing.

[brazen horizon]

How do I type hint the options for a parameter? I've done it before but completely forgot

def gas_price(unit: str = "wei") -> Union[int, decimal.Decimal]:

The allowed options for unit are "wei" and "ether"

[soft matrix]

!d typing.Literal

[rough sluice]

typing.Literal

typing.Literal```
A type that can be used to indicate to type checkers that the corresponding variable or function parameter has a value equivalent to the provided literal (or one of several literals). For example:

```py
def validate_simple(data: Any) -> Literal[True]:  # always returns True
    ...

MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
    ...

open_helper('/some/path', 'r')  # Passes type check
open_helper('/other/path', 'typo')  # Error in type checker
```...

[brazen horizon]

Ahh right, thanks!

[brazen horizon]

Do I have to do something else?

from typing import Literal

def gas_price(unit: str = Literal["wei", "ether"]):
    ...

gas_price(unit="definitely not either")

Doesn't trigger a PyCharm warning

[soft matrix]

Pycharm bad moment?

[oblique urchin]

unit: Literal["wei", "ether"] = "wei"

[soft matrix]

Oh wait

[oblique urchin]

but yeah PyCharm should complain about that

[soft matrix]

Didn't read that properly

[brazen horizon]

Oh whoops, now it definitely complains, thanks!

[terse sky]

If this is new code I would consider an enum instead, btw

[hearty shell]

Hey guys, is anyone aware of this issue (if it is one)?

python:

def sorted_guild(guild: discord.Guild) -> list[GuildChannel]:
    a = lambda c: c.position
    return sorted(guild.channels, key=a)

mypy 0.931:

+ Success: no issues found in 2 source files

python:

def sorted_guild(guild: discord.Guild) -> list[GuildChannel]:
    return sorted(guild.channels, key=lambda c: c.position)

mypy 0.931:

- bot.py:82: error: Returning Any from function declared to return "Union[SupportsDunderLT, SupportsDunderGT]"  [no-any-return]

I was just asking on #help-burrito and etrotta found a github issue but it seems that it should have been resolved 5 year ago

https://github.com/python/mypy/issues/1160

[rustic gull]

# https://mypy-play.net/?mypy=latest&python=3.10&gist=6862df753f5265d95b01d49e0c6c3eff
from typing import Optional

class ListNode:
    def __init__(self, val=0, next=None):
        self.val = val
        self.next = next
        
class Solution:
    def reorderList(self, head: Optional[ListNode]) -> None:
        if head is None: return

        # split at the middle node
        fast = slow = head
        while fast and fast.next:
            fast = fast.next.next
            slow = slow.next
        if slow:
            slow.next, slow = None, slow.next

# + py -m mypy main.py
# main.py:1456: error: Incompatible types in assignment (expression has type
# "None", variable has type "ListNode")
#                 slow.next, slow = None, slow.next
#                                         ^

I really don't understand what's wrong or why. I tried saying both fast and slow were Optional[ListNode] but it didn't help.

[oblique urchin]

I'd start by annotating the arguments to ListNode.init

you probably hit an edge case in mypy's inference

[terse sky]

Or make ListNode a dataclass

And annotate the fields

Much less repetition than currently

[rustic gull]

right, I missed annotating ListNode. it still gives the same error for the last line
https://mypy-play.net/?mypy=latest&python=3.10&gist=2059715a7b975b51c4cd3c7583a725ef

[hasty hull]

https://mypy-play.net/?mypy=latest&python=3.10&gist=79bcb5ce99db0980ea76da6c9ac25319

mypy was erroring because fast is assigned to be ListNode when it is defined and mypy does not like assigning a variable with a wider type like Optional[ListNode] to ListNode

the fix is to explicitly annotate the variable to the type you want to be at definition time

mypy can't infer that you want it to be optional

I also had to add an additional check that slow is truthy to satisfy mypy

[mortal fractal]

I'm gonna work on the dataclasses part of the change next since I think I handled all the tooling stuff for the primary change now. Unfortunately the dataclasses.py type introspection code is a bit finicky so I'll need to spend a bit of time digesting before I can give a substantive response to Eric. While I would like to just say dataclasses should go ahead and use the typing.py introspection tools, I feel like changing dataclasses to always import typing might rub people the wrong way that are already antagonistic to typing

It might be worth benchmarking anyway

since, on the other hand, dataclasses was designed to work with type hints so it's not like it's particularly weird it would need something from typing.py

[oblique urchin]

an intermediate option could be to ast.parse string annotations instead of just using regexes

[mortal fractal]

I'm comfortable doing the regex if we accept the caveat that it would have to be an Annotated that's not renamed (..?). If we're not doing that I think we're better off importing typing.

It's not too bad, it's something like this:

^\s*(?:(?:\w+\s*\.)?\s*Annotated\[)?(?:\s*(\w+)\s*\.)?\s*(\w+)

which isn't any more or less opaque than it already was, probably

probably needs an \s* in there again, because of weird syntax

I am slightly worried that people who use Annotated might be likely rename it for line length reasons

I'm not sure how nested classes that Eric mentioned could break things I will need to play with this

oh, ForwardRef doesn't have a public resolver? :\

oh well, get_type_hints evaluates anyway

@oblique urchin ahahhah of course I finda mistake immediately after I submit it

needs yet another \s*

^\s*(?:(?:\w+\s*\.)?\s*Annotated\s*\[)?(?:\s*(\w+)\s*\.)?\s*(\w+)

who knew python allowed so many spaces everywhere? 😛

[fierce ridge]

is there any academic theoretical CS work that studies the python type system?

[tranquil turtle]

C++ templates are turing-complete
is any part of python typing system turing-complete?

[oblique urchin]

mypy was Jukka's PhD thesis. Not aware of anything more recent

[tranquil turtle]

is checking code by a type-checker a turing-complete?

[fierce ridge]

interesting

[fierce ridge]

in dependent type systems, it is

i dont think it is in other "well-founded" type systems, there are theorems about this

(eg Hindley Milner)

[trim tangle]

typescript and (IIRC) rust are also turing complete

[fierce ridge]

the type systems? 👀

[trim tangle]

Yeah

[fierce ridge]

if typescript is, then python surely is

[trim tangle]

press X to doubt

TypeScript's type system is much more powerful

You can't even add two numbers in Python types!

[fierce ridge]

you can in typescript?

[trim tangle]

Yes

[fierce ridge]

it supports refinement types?

[trim tangle]

No

[fierce ridge]

well that's turing complete cause there's literally an expression evaluator embedded!

like in dependent types

[trim tangle]

There aren't really refinement types, but you can express peano naturals and recursive functions in types. I guess it is also somewhat dependently typed if you also consider the strong structural typing elements like literal types and object types.

[tranquil turtle]

python @main valve looks like c++ template specialization

[oblique urchin]

it's not as powerful

[trim tangle]

Template specialization is more like generics

Overloads are, well... C++ overloads :)

[terse sky]

In C++ overloading I think is strictly more powerful than specialization; specialization can be expressed in terms of overloading on a single argument

but python overloading is obviously something else

but they also do different things, overloading can obviously only be used for functions while specialization is actually mostly used for classes

(specialization for functions is limited and usually discouraged)

[acoustic thicket]

how do you do it in ts

[trim tangle]

I'll show you when I get home

[mortal fractal]

I don't have a ton of experience with introspection, but are there issues with get_type_hints and stringized annotations that would need to be special cased e.g. for the purposes of dataclasses? From looking at the code it wasn't immediately obvious what the problem area is since get type hints looked to evaluate forward references cc jelle?

or the concern Eric mentioned with nested classes

[oblique urchin]

did you see https://bugs.python.org/issue41370?

[mortal fractal]

oh, briefly noticed it when looking at PRs yesterday and forgot to look into it. I will look.

@oblique urchin oh, so get_type_hints will fail with e.g. Annotated['ClassVar[int]', 3] ?

with a run time error (!?)

[oblique urchin]

haven't tried it but I don't think so. it doesn't work properly for built-in generics

but probably it will just leave it as a string

[mortal fractal]

no it fails at runtime this is a bug with my patch approach I think

or a bug somewhere else, I haven't had a chance to look into it

this fails at runtime on cpython main https://bpa.st/OYDA

I have a job interview in 1.5 hours so I don't have time to figure this out now (I have to get ready and stuff still), but we definitely need to figure this out

[oblique urchin]

definitely prioritize the job interview 🙂 good luck!

[mortal fractal]

Thanks

[trim tangle]

https://www.typescriptlang.org/play?#code/C4TwDgpgBAWhBOB7KBeKBtAugKFJKAygK4DGAPAHIB8qGFO2u40AKogKoB2AhvCJTRTYoIqBSgQAHsAicAJgGdCpMgEtOAMwRiw8VQFsIVYaID8UAAYASAN5suvfhV0GjAXygBqKAEYLJkQAuKAAiAAYQxiZ8AEE5OTIYgBooACFBAKgYiWlZRWVydS14LJdDY1EocziEmLKIFOJydIrRYNSovFYAC3gIaDQmsiGhuCQqCejoADFVADcBguGVEZWVMcQJrc7mKAB9PoUiABtgWhqyFl7+lNmF4y79w5PgACUIXVp7Hj4yA4gjqdjEA

[acoustic thicket]

damn

[trim tangle]

https://github.com/susisu/typefuck

[acoustic thicket]

Lmao

[trim tangle]

fuck: int = 42

[mortal fractal]

https://bugs.python.org/issue46539

[hallow flint]

if python's type system is turing complete, i am not aware of it. i wouldn't be totally shocked if it is, PEP 612 and 646 are pretty powerful.

[oblique urchin]

recursive types (as supported by pyright) probably help

[terse sky]

I mean it seems like most type systems with generics end up turing complete

Even Java's type system, for example, is turing complete: https://arxiv.org/pdf/1605.05274.pdf

and it's not a terribly powerful type system

[mortal fractal]

I wish type checkers supported a bail-out for recursive types so we could use them

right now iirc mypy just throws an error, so you can't use both pyright and mypy

I wish it just converted the recursion internally to Any or something instead

luckily this isn't that common. I've only wanted to do it once, when doing a particular kind of graph representation. Even then there are other representation options, but still

[hasty hull]

Just use codegen ;) https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/generator/templates/types.py.jinja#L68-L98

[dim flume]

What the hell, I need to see this

[mortal fractal]

Oh, I think I thought of a way the bug I fixed today could be triggered before my prior PR.

Annotated["Annotated[... is valid I think

But it's not marked as a special form so I guess it doesn't matter

[grave fjord]

An if typechecking.mypy: would be nice

And if typechecking.mypy >= (0, 930):

[soft matrix]

I don't think it would ever be accepted though cause all the type checkers should be standardised

[oblique urchin]

pretty sure if MYPY: works for that

mypy also has an --always-true= option that makes it assume arbitrary constants are always true

[grave fjord]

Right but they're not, eg asgiref doesn't want to add ParamSpec yet

[hearty shell]

Hey, should mypy be warning me about this?
error: Argument 1 of "__eq__" is incompatible with supertype "object"; supertype defines the argument type as "object" [override]
Saying it violates substitution principle because my class derives from object

sounds a little silly to me x)

I mean, it does violate it

but...

[trim tangle]

show the method

[hearty shell]

@frozen
@total_ordering
class ChannelState:
    position: int
    channel: GuildChannel
    category: discord.CategoryChannel | None

    def __eq__(self, other: ChannelState) -> bool:
        return (self.position, self.category) == (other.position, other.category)

[trim tangle]

This is how __eq__ should work:

    def __eq__(self, other: object) -> bool:
        if not isinstance(other, ChannelState):
            return NotImplemented
        return (self.position, self.category) == (other.position, other.category)

it is right to complain

That is, __eq__ should work with any object.

[hearty shell]

Interesting, I wasn't aware, is this a convention?

[soft matrix]

yes

[trim tangle]

yes

https://docs.python.org/3/reference/datamodel.html#object.__eq__

[soft matrix]

comparison operations like eq ne lt and le etc shouldnt raise an error

and that current implementation would probably raise an error if you didnt pass a ChannelState

[trim tangle]

numpy 🥴

[soft matrix]

does numpy?

[hearty shell]

I see, well I guess I will have a read, thanks for the info!

[soft matrix]

lol

[upbeat wadi]

for things like this, is it recommended to use object instead of Any?

[trim tangle]

yes

they are indeed different

[soft matrix]

its always recommended to use object i thought since Any just turns off type checking

[upbeat wadi]

thanks

[trim tangle]

well, sometimes you do have to use Any

but if you can, use object

[mortal fractal]

In type introspection when comparing types is identity comparison more pattern than equality? They're classes

[oblique urchin]

yes

[mortal fractal]

Thanks

[acoustic thicket]

Numpy arrays violate lsp, dont they?
Because of its eq not returning a bool

[trim tangle]

lsp, shmlsp

[acoustic thicket]

Shmlsp?

[trim tangle]

https://en.wikipedia.org/wiki/Shm-reduplication

[acoustic thicket]

oh lul

[mortal fractal]

https://bugs.python.org/issue46552

[dim fox]

I am trying to create a typing alias with defaultdict but getting an error:```
10
11 # Type Aliases
---> 12 multi_defaultdict = defaultdict[str, Union[int, str, None]]
13
14 class Helper:

TypeError: 'type' object is not subscriptable

a dict might look like:```py
{"review_type": None, "project_name": "fellowship rejuvination", "project_number": 5501}

[oblique urchin]

what Python version are you on? you may have to use typing.DefaultDict

[mortal fractal]

What is the purpose of GenericAlias in typing.py

[oblique urchin]

it powers List[int], right?

[mortal fractal]

It's literally = to it

As in somewhere it's GenericAlias = list[int], or something

I'm on my phone or I'd link the code sample

[oblique urchin]

you mean types.py not typing.py?

[mortal fractal]

typing.py imports the one from types.py, but yeah

apparently mobile GitHub is better than I thought it would be: https://github.com/python/cpython/blob/20f53136679e260466a765de5befa3b9db396c9e/Lib/types.py#L300

[rough sluice]

Lib/types.py line 300

GenericAlias = type(list[int])```

[oblique urchin]

right, that's just so the name is public

[mortal fractal]

I guess I just expected to find something that wasn't just type(explicit implementation), but I don't know what

[oblique urchin]

and you can do isinstance(..., types.GenericAlias)

that's sometimes what's done for C types. Otherwise you'd have to write a C module that explicitly exposes the type

[mortal fractal]

I see

[mortal fractal]

https://github.com/microsoft/pyright/blob/c8273c33e8cca34560d59a53590fee99a511d063/packages/pyright-internal/src/tests/samples/arrowCallable3.py

lol

[oblique urchin]

that's what we need PEP 677 for

[trim tangle]

Do you use type annotations with pytest fixtures? It seems to me that it's a lot of work with little benefit. Nobody will ensure that you have the right type, and if something is the wrong type, you will find the error very quickly (just run the tests)

[fluid phoenix]

I use type annotations everywhere out of habit so that I never forget them when I should have them

[trim tangle]

is there any way to annotate this faithfully? ```py
class About:
def init(self, target, epsilon):
self._target = target
self._epsilon = abs(epsilon)

def __eq__(self, other):
    try:
        return abs(self._target - other) <= self._epsilon
    except TypeError:
        return NotImplemented

I should be able to do `About(3, 0.01)` and `About(datetime.now(timezone.utc), timedelta(seconds=2))`

Something like this in haskell pseudocode ```hs
class Abs a where
abs :: a -> a

class Diff a d where
(<->) :: a -> a -> d

approx :: (Diff a d, Abs d, Ord d) => d -> a -> a -> Bool
approx eps a1 a2 = abs (a1 <-> a2) <= abs eps

(i know that this breaks transitivity and probably international law, but it's kinda useful when you're comparing deeply nested structures in tests, and you suddenly realize that you have a float somewhere deep inside...)

[dim flume]

For a second I thought I saw a JS arrow function

[trim tangle]

don't give me esoteric ideas

[dim flume]

return abs(self._target - other) <= self._epsilon

Esoteric Python arrow functions

Even if it's a reversed arrow

This snippet right here

This is what made me think of arrow functions

I must be sleepy too

[trim tangle]

!e

class function:
    def __init__(self, body):
        self._body = body

    def __le__(self, params):
        return eval(f"lambda {params}: ({self._body})")

fibo_step = function("(b, a + b)") <= "a, b"

print(fibo_step(3, 5))

[rough sluice]

@trim tangle :white_check_mark: Your eval job has completed with return code 0.

(5, 8)

[trim tangle]

@dim flume done ^

[dim flume]

good now turn it into a library just like that goto package

[blazing nest]

I don't understand, wouldn't a Union be enough? Do you need this to be duck-typed?

[trim tangle]

I don't want to put overloads for all the possible combinations

so yeah, I want it to be duck-typed

[dim flume]

Thinking about it, can Python's type hints be abused to execute code

That makes me think of Whitespace, code inside your code

[blazing nest]

You'd have to make something like ```python

Target and T are all TypeVars

class LeftHandUtilityIGuess(Generic[Other]):
def sub(self, other: Other) -> typing.SupportsAbs:
...

class About(typing.Generic[Target, T]):
def init(self, target: Target[T], epsilon: typing.SupportsAbs):
self._target = target
self._epsilon = abs(epsilon)

def __eq__(self, other: T) -> bool:
    try:
        return abs(self._target - other) <= self._epsilon
    except TypeError:
        return NotImplemented

This... is ugly

Not even sure if it works

Let me see

[brisk hedge]

Is the typeerror from the __sub or the __le ?

[blazing nest]

I tried it out in the Pyright playground and this is the closest I got: ```python
import typing

T = typing.TypeVar('T')
Target = typing.TypeVar('Target', bound='LeftHandUtilityIGuess[T]')
Other = typing.TypeVar('Other')

class LeftHandUtilityIGuess(typing.Generic[Other]):
def sub(self, other: Other) -> typing.SupportsAbs:
...

class About(typing.Generic[Target, Other]):
def init(self, target: Target[Other], epsilon: typing.SupportsAbs):
self._target = target
self._epsilon = abs(epsilon)

def __eq__(self, other: Other) -> bool:
    try:
        return abs(self._target - other) <= self._epsilon
    except TypeError:
        return NotImplemented

The current errors say TypeVar bound type cannot be generic and TypeVar "Type[Target@About]" is not subscriptable.

[brisk hedge]

You definitely need to typeguard though since eq takes in other: object

Be it through isinstance or a custom typeguard

A set of protocols such as

class GenericAbs(Protocol[T]):
    def __abs__(self) -> T:...
class GenericSub(Protocol[T, U]):
    def __sub__(self, other: T) -> GenericAbs[U]: ...

could also work
then

def abs_diff(a: GenericSub[T, U], b: T) -> U:
    return abs(a - b)

type checks (I believe)

And fwiw there's no point making a protocol for <= since object has that by default (I think)

[brisk hedge]

Then you just need to typeguard from object to T inside eq

I think runtime_checkable protocols handle that

[blazing nest]

!e They don't since objects don't support greater than.

object() <= object()

[rough sluice]

@blazing nest :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 1, in <module>
003 | TypeError: '<=' not supported between instances of 'object' and 'object'

[brisk hedge]

Fancy, another generic to keep track of

[blazing nest]

yeah lmao

[brisk hedge]

I'm still not certain you can isinstance a generic protocol

either way that's the place to #type: ignore because eq taking in object is kind of hard to overcome otherwise

[brisk hedge]

But yeah the try: except TypeError: is literally runtime protocol checking

[warm holly]

i came here looking for this

[hasty hull]

I use type annotations so that I can typecheck the tests and ensure that the public API I'm providing doesn't cause any unintended type errors when actually using it

[trim tangle]

I guess that does make sense

[mortal fractal]

\o/ hurrah my third typing cpython patch merged

that ClassVar[Final] thing I discovered at the same time was unexpected (this is a valid runtime annotation)

[inland mist]

I have
class SomeCtxManager(AbstractManager):
and use it as

with SomeCtxManager as manager:
  manager.doStuff()

unfortunately, mypy treats manager as AbstractManager and does not recognize methods implemented by SomeCtxManager but not present in AbstractManager

is there any way to explicitely type manager in with ... as ... syntax?

[oblique urchin]

That's a problem with the type annotation on the __enter__ method

Probably you wrote -> "AbstractManager"

[terse sky]

That's just a funny interaction because neither Final nor ClassVar are really types

[oblique urchin]

Instead you should use a TypeVar bound to AbstractManager, or in the near future, PEP 673's Self type

[soft matrix]

You could use it now if you didn't care about mypy or pytype support (afaik)

[inland mist]

That's right, that's what I wrote. Thanks, I will look into how to use TypeVar here

[oblique urchin]

It's a bit awkward syntactically. PEP 673 has some examples of how to do it

(along with the better way that will be possible soon)

[inland mist]

I probably don't need that abstract class now so will be deleting it for now, but it's nice to know anyway

Looks cools, looking forward to 3.11

[oblique urchin]

no need to look forward to 3.11, it's already in typing_extensions 🙂

the only reason I didn't recommend it to you is that you mentioned mypy and mypy doesn't support it yet

[soft matrix]

I'm gonna work on it after my last mock exam (which is Friday in 2 weeks)

[mortal fractal]

hmmm, not sure what to do with ForwardRef here, I guess I can try to just extract the text inside

[mortal fractal]

🤔 now to figure out what code formatter dataclasses is using

looks like yapf good enough for my purposes to match my snippet to existing style

\o/ hurrah dataclasses updated. Now to come up with all these unit tests 😰

[lyric hull]

i have an intenum that i've delcared via funtional api:

InOrOut = enum.IntEnum("InOrOut", {"out": 0, "in": 1})

this isn't getting along very with with vscode, as variables annotated as InOrOut show as having type Any on mouse hover. has anyone run into this and are there any tricks i can do to make it work better at development-time? (not sure if this is pylance or mypy, i think pylance; mypy has no complaints at lint-time but i don't suppose it would if it also considers InOrOut to be Any?)

[terse sky]

the thing is that this is a very dynamic way to declare a type

why not do it in the more usual way

[lyric hull]

in is a reserved word

[terse sky]

class InOrOut(enum.IntEnum)...

well, then use another word 🙂

enumerators are often encouraged to be all caps in any case

so you could just use IN and OUT

most likely the development experience is always going to be worse if you declare it that way. e.g. auto completion and such.

and of course (at least traditional) auto completion can never really work with a keyword

[lyric hull]

disappointing that if it's a top-level declaration in a module that it isn't just evaluated

[terse sky]

not really

it's a big ask

[lyric hull]

that doesn't not make it disappointing

i'd consider making the member names all caps though

[terse sky]

yeah, I mean it's defintely the way forward

those forms for things like enums, dataclasses, named tuples, etc, I consider for programmatic use only

I actually never use them really

[lyric hull]

even type annotating it with InOrOut: enum.IntEnum = enum.IntEnum("InOrOut", {"out": 0, "in": 1}) it's coming up as Any

[terse sky]

that's odd, but it still wouldn't really be the type that you want

Also, InOrOut here is the class

not a value

[soft matrix]

I didn't think you'd even be able to access that member without getting a syntax error

[terse sky]

so you're asking about the type of a type

oh sorry

you mentioned you were using InOrOut as an annotation

[soft matrix]

The other by far better option is to just use in_ as the variable name as suggested by pep 8

[terse sky]

nevermind that part

[lyric hull]

yeah i use InOrOut["in"] when doing equality checks, etc.

[terse sky]

where does pep8 suggest that?

[lyric hull]

i guess don't need it to solve everything but it would be nice to get away from Any. this seems more like a pylance (or whatever it's using behind the scenes) issue

[soft matrix]

single_trailing_underscore_: used by convention to avoid conflicts with Python keyword, e.g.

tkinter.Toplevel(master, class_='ClassName')

[terse sky]

it's not really a pylance issue though. That's what I was getting at before when I said it's not disappointing.

It's not a minor QoI issue, it's a very fundamental property of the type system.

most type systems can't handle this kind of thing

[lyric hull]

i'm not asking for dynamic evaluation but if it's module-level assignment and tagged with a type annotation, it should just be using that

[terse sky]

well, the type annotation you used above, is not correct, because your annotating the type of a type

but that's not the type of the type

it's another type which happens to be a base of your type, which is different

other than that annotation, yes, you'd exactly need to evaluate the function enum.IntEnum to know anything about the type that was returned

[lyric hull]

hot take: in this case, it should actually just do that (evaluate it). but if they don't want to, fine

[terse sky]

python type checkers aren't going to actually run any functions generally, afaik

it's just a really really complicated undertaking

[lyric hull]

then special case IntEnum v0v

[terse sky]

there's no reason to do that

dataclass is special cased

by mypy

[lyric hull]

????? this, right here, is a reason to do that

[terse sky]

but only the @dataclass class foo:... form

if you use that form, it works fine

There's no reason to special case the dynamic/programmatic forms of these things

if you want static type checking, use the static forms

[lyric hull]

????? this, right here, is a reason to do that

[terse sky]

somebody wanting to use "in" as a field of their enum isn't remotely close to a good enough reason to motivate such an enormous amount of work

usually (like, 99% of the time) people are using the programmatic forms because it is in fact dynamic/programmatically generated, so in those cases, you just fundamentally can't benefit from the static type system anyway

that's why nobody will special case those (i'm willing to predict)

[lyric hull]

that's not the same thing as "no reason" lol. other languages do far more complicated things all the time. i believe you're right that it's not pragmatic in python though

[terse sky]

statically typed languages aren't even going to allow you to define enums or classes in such a dynamic way

[lyric hull]

as far as i'm concerned it's not dynamic if it's at the top of a module with a constant set of arguments

[terse sky]

sorry "no good reason" 🙂

🤷‍♂️

as far as you're concerned; that's great, but this isnt' going to happen, and this wouldn't work in almost any mainstream language

[soft matrix]

this is all completely avoidable anyway

[terse sky]

anyway, just use in_ and out_ as Gobot suggested

and use the static form

[soft matrix]

i really dont see why you are dying on this hill

[lyric hull]

i think mypy does support some kind of functional api for this (at least, there are issues related to it), but it's hard to tell because aiui if mypy is deciding it's Any it's not going to speak up about it

[terse sky]

could have been done with this and getting perfect help from the IDE (including auto completion) in much less time than this conversation took 🙂

you can ask mypy to reveal the type

[lyric hull]

believe it or not python's reserved words are not the entire universe of computing

tired of hand-rolling ser/de haha

[terse sky]

idk what that means. But you have a really simple solution.

[soft matrix]

is this something you dont (really) have control over the names for?

[lyric hull]

if you have e.g. incoming json with a value "in", it's a lot easier to deserialize to an intenum member with a member "in" than "in_", etc

[terse sky]

it' snot a lot easier, it's a minor inconvenience.

[lyric hull]

the python experience ™️

[terse sky]

If you add _ to every member then you know to just remove/add it when serializing/deserializing

okay, probably random snide comments about python are off topic here, so unless you have any more questions about type-hinting...

[lyric hull]

i'm just saying, we can and should aspire for python to be better than what python currently is and provides-- that's why type hinting is here in the first place

[terse sky]

the things you think are big problems aren't big problems. This is very easily solvable, right now, in python, with minimal effect, like, a few extra characters of code.

and it would take a lot of work to solve those problems in the way you want

There's just better places to spend the programmer time and the language complexity

but if you really believe in this, then obviously you can always submit a PR to mypy.

[lyric hull]

genuinely, do you use languages besides python? there's lots of ways this could or couldn't be a problem. it is what it is, fine, but you don't seem able to admit it's something that isn't good

not interested in a slapfight or anything. again, it's fine, i was curious and it is what it is, so that can be the end of it if you want

[terse sky]

My main language is C++

so I've seen how much effort has gone into constexpr

which could kind of do the things that you want

and it's an enormous amount of effort. It has some nice benefits but in C++ you get extra justification from the performance angle.

So I'm very well acquainted with this issue.

and even then, actually, constexpr won't really be able to do something like this

this is really creating a type in the most general sense, the best that constexpr can do is compute a value at compile time which can parametrize a type.

Maybe D can sort of, maybe do something like this, but D is pretty famous for advanced compile time metaprogramming and introspection.
Rust would be using macros for something like this, not regular function calls. etc.

I can't claim much knowledge of advanced research languages like Idris etc but I know more than enough languages to know this is a hard problem, and while you can solve it like anything, it's a lot of effort, a lot of complexity... just a lot.

[soft matrix]

(this is definitely something serde can do)

[terse sky]

Right, with macros, not functions

[lyric hull]

i get what you're saying and fully believe you but you are vastly overestimating what i'm asking for here

[terse sky]

No, I don't think you're understanding what a huge can of worms it is to add a language feature like that. You can't just add it for one special case, you have to think really careful about what's going to be allowed, what not, what delineates them, etc

[lyric hull]

here's one trivial solution: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/verbatim

another is to just look at the fact that IntEnum is an interesting special case and to evaluate the type if it has constant arugments-- like it does in this case

[terse sky]

but it's not worth doing as a special case. I've already explained exactly why.

I'm not too familiar with C#, but the first example on your page is runtime reflection

so if you screw up for example you'll get an error at runtime

at least, that's as best as I can tell

so that really has nothing to do with this

[lyric hull]

that doesn't sound right-- let me check the link to make sure it worked
you mean the @for example? how can you tell it's reflection based

[terse sky]

oh sorry, maybe I misunderstood

you're saying that you could use @ and then just use in as the identifier?

so define the fields as @in and @out

[lyric hull]

sure

[terse sky]

you could, but it's still going to be a crappy development experiences

[lyric hull]

that'd be neat and a solution

[terse sky]

auto completion won't work

it's obscure

I can tell you that for example Kotlin has this

and it uses it almost exclusively in generated code

that's the main purpose of it

like a quick google in the C# ecosystem suggests that the views on it are similar

For codegen, legacy code, etc

[lyric hull]

i see typ, type_, klass, class_ in everyday python so it's not absolutely crazy

[terse sky]

I don't think that most C# devs would solve the analogous problem with @ (though I'm not sure)

[lyric hull]

c# doesnt have this problem though, lol, thats my point

[terse sky]

it's pretty crazy. It's much simpler and easier to have a convention. And the variable name very rarely matters. Reflection is one of the few cases where it does.

C# does have the problem... you'd have to use @ to "solve" it

but most people still wouldn't

is my guess

[lyric hull]

no, if i solved it with @, its solved

[hearty shell]

Are you really just getting Any? Mypy seems to do a better job, at least it will give you a Union of the possible types

But I also would not use the functional api for my types

[mortal fractal]

oh I came up with a much less invasive way to do the dataclasses patch; luckily nobody has reviewed it yet

[terse sky]

and I'm telling you, most C# developers would probably object to @, is definitely my guess.

[lyric hull]

def need to check what mypy thinks of it. like i said i suspect mypy actually does handle this slightly better but i'm not sure. as far as i can tell vscode is using pylance which really just seems to provide it as Any

[terse sky]

(would object to @ being used for that)

[hearty shell]

https://mypy-play.net/?mypy=latest&python=3.10&flags=strict&gist=e0f8ba60d0be5583d422b74f99d3a0cd

[terse sky]

in frameworks that use extension reflectively, e.g. for serialization, it's basically a given that you have the opportunity to give a string to be used instead of the field name

e.g. in pydantic, you can give an "alias" for a field, and then ask to serialize/deserialize using aliases instead of field names, where defined.

[lyric hull]

this owns bones, thanks

[terse sky]

Very simple, works great.

[lyric hull]

yeah i think python's ser/de story is pretty disappointing overall, which is why i'm so tired of shoving in hooks/shims everywhere, but idk

[terse sky]

i haven't had an issue.

I rolled some of my own stuff based on dataclasses for compatibility with some existing stuff, it was pretty easy overall

if you are doing things from scratch and want it to be easy pydantic realistically does whatever you need

but if you think that having to do int_ instead of in is a huge problem, then yes, I believe that you'd find things disappointing about everywhere

[lyric hull]

the system i've been building tries to be a couple things:

1. recursive
2. polymorphic
3. declarative
4. supports different context (fields have different names depending on which place they're going)
   i am not sure pydantic can do these?

[terse sky]

pydantic does 1. Not sure about 2. Not sure what you even mean by 3.

not sure what you mean by 4.

In any case, python's reflection and annotation story is quite good, I don't think the language is the issue here.

[lyric hull]

haha alright bud

[golden terrace]

How do I stop Python from closing the command line after running?

[hearty shell]

Off topic, #❓｜how-to-get-help

[dim fox]

Google colab so Python 3.7.12

How do I use typing.DefaultDict?

[hearty shell]

The same way you defaultdict, you can call it and pass the factory as the first argument, and you can annotate is just like you would with a dict

DefaultDict[str, int]

[terse sky]

Just be careful with DefaultDict

It implements Mapping but that's a lie

[hearty shell]

What do you mean?

[acoustic thicket]

iirc it breaks lsp or something because __getitem__ can potentially mutate

[hearty shell]

Got curious, and it lead me to this opinion, really interesting imo

https://softwareengineering.stackexchange.com/a/332221

[oblique urchin]

we had a long discussion about this here a little while back

[terse sky]

@hearty shell DefaultDict is a dict, maybe

But it's definitely not a mapping

But dict is a Mapping so you're basically painted into a corner

Easiest thought experiment is if you pass x, into a function foo which takes a Mapping, and x is only bound locally (not accessible from a global or aliased)
Then assuming your program type checks, foo(x) won't mutate x

That's a huge part of the point of Mapping

Default dict breaks that

[hearty shell]

Humm, but doesnt dict have the same problem? if you mutate a dict inside the local scope, it affects the global

[terse sky]

Not sure what you mean exactly, but no

Mapping isn't a promise that the type is immutable

It just says you can't mutate it through this reference

It's a read only interface

That's why if x is aliasing a global which has type dict, then sure, foo(x) can still mutate x

What's happening with DefaultDict is different

foo can mutate x directly

You just do x[5]

This will type check since it's part of the Mapping API

But still potentially mutate

[hearty shell]

Yeah, your right, no, because what his argument was that one should not expect KeyError to be raised if a key is not present with a dict, because __missing__ is part of the interface of dict, however if you expect a Mapping, __getitem__ should not mutate

Right?

[hearty shell]

I think I get what you mean now

Well lucky me, ctrl+f is a thing, pretty sure if I had any remaining doubts the long thread will answer them for me x)

[terse sky]

Right

[rustic gull]

Does Iterable[str, int, bool, enum.Enum] mean each item can be one of those? Or should it be Iterable[Union[str, int, bool, enum.Enum]]?

[oblique urchin]

it doesn't mean anything, it's an error. Iterable[Union[...]] does mean what you say

[rustic gull]

Thanks

[mortal fractal]

it looks like typing.py isn't the only one bit by the annoyance of forward reference classvar inside annotated annotations 🙂

[stray summit]

anyone aware of a way to tell mypy a specific folder is a specific module thats importable (i want to map the src folder to mypackage)

[trim tangle]

stub file?

hmm I'm not sure

[stray summit]

i don't follow, how would a stub file help? (bascially whats in src/* would end up in SITE/mypkg/* if installed

[trim tangle]

ah

[opaque shard]

Does anyone have any clue as to why this raises mypy errors?
I have a protocol

class ResponseModel(Protocol):
    def update(self, other: ResponseModel) -> None:
        ...
```and a couple classes that look something like
```py
class QueryResponse(BaseModel):  # these are pydantic models
    # bunch of attributes and methods omitted
    def update(self, other: QueryResponse) -> None:
        ...  # implementation here
```and a function (method of another class `WikiCog`) that takes a `ResponseModel`-like class and returns an instance of it, defined as follows:
```py
ResponseModelT = TypeVar("ResponseModelT", bound=ResponseModel)

async def request_api(
    self,
    params: dict[str, str],
    response_model: Type[ResponseModelT],
) -> ResponseModelT:
    ...  # implementation here
```if I then call `await self.request_api(_params, QueryResponse)`, mypy raises `Value of type variable "ResponseModelT" of "API_request" of "WikiCog" cannot be "ContentResponse"` (error displays on the `self` of the method call, specifically).
FWIW, if I make the protocol runtime-checkable, `isinstance(QueryResponse, ResponseModel)` returns True. Furthermore, `self.request_api(_params, set)` etc. also raise the same error.

[oblique urchin]

the protocol accepts any ResponseModel, your implementation only accepts QueryResponse

you probably want to use a bound typevar as the annotation instead

runtime_checkable doesn't catch this because it just looks at whether methods exist, not at the annotations

[opaque shard]

ah wait yeah that actually makes sense

thanks a lot ^^

yup, that fixed it, awesome

[solid light]

If I have a function that will return either tuple or list[tuple], depending on the value of one of the parameters, what's the correct way to typehint this? My currently solution is using typing.overload but I'd like to have an in-line solution if possible.

Minimal version of code:py def foo(a: str, *args, fetchall: bool = False): if fetchall: return [(1,2,'a')] # when `fetchall` is `True`, function returns `list[tuple]` else: return (1, 2, 'a') # when `fetchall` is `False`, function returns `tuple` My current solution: ```py
from typing import Literal, overload

@overload
def foo(a: str, *args, fetchall: Literal[True]) -> list[tuple]:
...

@overload
def foo(a: str, *args, fetchall: Literal[False]) -> tuple:
...

def foo(a: str, *args, fetchall: bool = False):
# has actual function implementation```

[trim tangle]

have two functions

[solid light]

Hmm

[trim tangle]

why do you need one function?

[solid light]

I mean it seems weird to have two separate functions

The code inside is the exact same apart from the return line

[trim tangle]

Can you show the code perhaps?

[solid light]

def execute_sql(self, query, *args, fetchone=False, fetchall=False) -> Union[tuple, list[tuple]]:
    if not query.upper().startswith("SELECT"):
        raise ValueError("Execute sql can only be used with 'SELECT' & transaction statements")

    if not fetchone and not fetchall:
        fetchone = True


    cur: psycopg2._ext.cursor  # noqa
    with self.db_conn.cursor() as cur:
        try:
            cur.execute(query, *args)
        except psycopg2.DatabaseError as e:
            cur.execute("ROLLBACK;")
            self.db_conn.commit()
            raise e
        else:
            if fetchone:
                return cur.fetchone()
            elif fetchall:
                return cur.fetchall()```this is my actual code

I realise the redundancy of having fetchone and fetchall parameters, hence my example above only had fetchall

[trim tangle]

@contextmanager
def _with_readonly_cursor(self) -> Iterator[psycopg2._ext]:
    cur: psycopg2._ext.cursor  # noqa
    with self.db_conn.cursor() as cur:
        try:
            yield cur
        except psycopg2.DatabaseError as e:
            cur.execute("ROLLBACK;")
            self.db_conn.commit()
            raise e

def fetch_all(self, query: str) -> list[tuple]:
    if not query.upper().startswith("SELECT"):
        raise ValueError("Execute sql can only be used with 'SELECT' & transaction statements")
    with self._with_readonly_cursor() as cur:
        cur.execute(query, *args)
        return cur.fetchall()

def fetch_one(self, query: str) -> tuple:
    if not query.upper().startswith("SELECT"):
        raise ValueError("Execute sql can only be used with 'SELECT' & transaction statements")
    with self._with_readonly_cursor() as cur:
        cur.execute(query, *args)
        return cur.fetchone()

there's a little bit of duplication, which you could remove if you really wanted to

[solid light]

Hmm

[trim tangle]

Why can the query only contain a SELECT?

[solid light]

The thing is, I have 100+ calls to this function, and I really don't fancy refactoring everything

[trim tangle]

if you just want to type hint it, overload is the only way

[solid light]

Rip

Guess that's the best option for me then

def execute_ddl(self, query: str, *args, auto_commit: bool = True) -> Optional[tuple]:
    if query.upper().startswith("SELECT"):
        raise ValueError("Use execute_sql() for 'SELECT' statements")

    cur: psycopg2._ext.cursor  # noqa
    with self.db_conn.cursor() as cur:
        cur.execute(query, *args)
        if auto_commit:
            self.db_conn.commit()

        if "RETURNING" in query.upper():
            data = cur.fetchone()
            return data```I suppose the same applies to this? In fact I don't think I could even use `overload` for this

Basicallypy if "RETURNING" in query.upper(): # the return type is tuple else: # the return type is None since doesn't return

[trim tangle]

yep

[oblique urchin]

that seems really sketchy anyway

[trim tangle]

I think you're working on some very strange piece of code 😄

[oblique urchin]

select * from table where x = "returning" would trigger that

[solid light]

Well then it's a good thing I never do that statement :p

You're right that it's jank though

Actually, there's cases where I might so that sucks

And it'd raise an error too because it'd try to cur.fetchone() when there's nothing to fetch

[oblique urchin]

well at least I learned something, https://www.postgresql.org/docs/9.5/dml-returning.html

[solid light]

Heh

I guess I need to rethink this

[oblique urchin]

yes, it probably makes more sense to have a separate method for statements that use RETURNING

[trim tangle]

You could refactor these flagged methods like this:```py
def execute_sql(self, query, *args, fetchone=False, fetchall=False) -> Union[tuple, list[tuple]]:
if not fetchone and not fetchall:
fetchone = True
if fetchone:
return self.fetch_one(query, *args)
else:
return self.fetch_all(query, *args)

unless it's deemed as unnecessary work

[solid light]

Eh, apparently pycharm doesn't even understand typing.overload which sucks

[solid light]

Yeah, I should probably do that tbf. Thanks

[solid light]

Ah, it does, but I have to explitly put fetchall=False in the call. It doesn't realise it defaults to False

Does mypy have support for *args recognition? I'm messing around with the playground and can't seem to get it to work

[trim tangle]

show the code?

*args should work

like *args: object or *args: str

[trim tangle]

for *args, you specify the type of each arg

not the whole tuple

wait.

I don't get what you're trying to do

[solid light]

Honestly neither do I at this point lol

[trim tangle]

If you're passing *args, you need to do fetchall=False as a kwarg

that's how Python works

[solid light]

Yeah

You're right

Okay, that works

But it breaks when I don't pass fetchall

It doesn't realise that it defaults to False

I tried adding =False to the overload functions and that gave a different error

[trim tangle]

that's because your call doesn't match any of the overloads

if you have a default, you could add a third overload

(I guess)

or Literal[False] = False

[solid light]

Hmm

[trim tangle]

what error are you getting with that?

[solid light]

I think that's all I need actually

I was passing = False to both of them

But I only need for that

This works perfectly, thank you```py
@overload
def execute_sql(self, query: str, *args, fetchone: bool = False, fetchall: Literal[True]) -> list[tuple]:
...

@overload
def execute_sql(self, query: str, *args, fetchone: bool = False, fetchall: Literal[False] = False) -> tuple:
...

def execute_sql(self, query: str, *args, fetchone: bool = False, fetchall: bool = False):```

Now I just need to sort out execute_ddl

[trim tangle]

for some definition of perfectly 👀

[solid light]

lol

try:
    return cur.fetchone()  # tuple[Any, ...]
except ValueError:  # or whatever the error is
    return None```I'm guessing there's no way for me to typehint this, so I'd have to use two functions?

[hearty shell]

Union[tuple[Any, ...], None]?

[solid light]

I want the distinction

[hearty shell]

What do you mean?

[solid light]

See this ig

[solid light]

.

Except that if statement needs to be changed to a try/except

And there's no way to know in advance whether a try/except is going to fail or not (well, I do, but I can't really convey that to the program -- it'll fail when there ISN'T a RETURNING statement, but I can't just check for that string since it could be elsewhere in the statement)

"INSERT INTO table VALUES ('foo', 'bar', 3) RETURNING name"  # a RETURNING statement
"INSERT INTO table VALUES ('returning', 'bar', 3)"  # not a RETURNING statement but has the string in statement```

I guess I can regex to see has string and it's not in parentheses but that's just horrible

And that won't even work anyway, because of UPDATE statements

[hearty shell]

You could tell the function that it is a returning function by using NewType but I think this is going overboard

[solid light]

I think for this I just need to accept that tuple | None is the best I'm getting

[hearty shell]

I mean, the problem is that when you will want to index or do whatever with that tuple mypy will complain unless you check for it being None, otherwise you can cast it

I did once use NewType for defining some graph queries as being of different types

but I am not sure that is ideal at all xD

[solid light]

The thing is before I even figure out the type hinting stuff, I'd need a way of distinguishing whether a statement is RETURNING or not

The only thing I can think of is a try/except

And there's no way for mypy to know the outcome of that in advance

So what I'm asking is therefore just impossible

[oblique urchin]

It's possible in some cases. With our typechecker (pyanalyze) you could write a plugin that looks at the query string being passed (if it's a string literal) and adjusts the return type based on that. A mypy plugin might be able to do that too.

[trim tangle]

but is it worth it?

[oblique urchin]

yeah probably not

[solid light]

Heh

I guess it's nice to know it's at least theoretically possible though

[oblique urchin]

At work we use this sort of approach for inferring what column types a query will return, that's pretty useful

[solid light]

Yeah

[hearty shell]

You could make a function that returns different types based on a query and a return

[solid light]

I mean I'd need to do more than just analyze the string

[oblique urchin]

yes, this requires knowledge of the schema too

and you need to parse the SQL

[terse sky]

the thing is that just throwing a raw string that you need to parse to figure out things like this is one of the less ideal ways to do a DSL, IMHO

[solid light]

I'd have to actually execute the statement in the string, and then try/except a cur.fetch_one()

[terse sky]

that's why it's more common to see other approaches

[trim tangle]

what even is the point in all these methods?

[terse sky]

e.g. languages that support infix functions will try to let you write code that looks like the queries

but it's still code, not a string

[oblique urchin]

yes, we actually do this for a higher-level API that maps to SQL queries

[solid light]

There's a distinction between statements that read, and statements that write

Because this is for my coursework and that's what the exam board says we need to do 🤷

[hearty shell]

Oh, for that I would 100% over complicate and do what I said xD

[solid light]

Lmao

I'm honestly happy to just leave it as-is

[trim tangle]

W T F

I thought you were working on some ancient legacy codebase made by ancient humans with boolean flags

[solid light]

I got execute_sql() working exactly how I wanted it, I can live without execute_ddl(). If I really want it I can just do data: tuple = database.execute_ddl("a returning statement") and then PyCharm knows it's a tuple

[solid light]

Yeah no, it's just British exam boards being dumb asf

[terse sky]

is this a university course

[solid light]

Nope

Pre-uni

[terse sky]

knowing something is a tuple isn't very exciting though

[solid light]

True, but it means it knows it's not None

[terse sky]

it's better than knowing zero but not by much

[solid light]

So I won't get a warning when iterating over it p.e.

[terse sky]

right, you can iterate over it, but you have no idea what you can do in the loop body

[solid light]

I suppose

I mean tbf I'm planning on eventually having each table as a namedtuple or something so then I can do data: Account = database.execute_ddl("...") or whatever

That way I'll know what I can do inside the loop

But that's a thing to do once I've got everything else working

[terse sky]

usually you wouldn't use a loop tbf since the types inside a tuple are heterogeneous, typically

at that point I would pass the type to the function

because at least you can get a runtime error if they're not compatible

data = database.execute_ddl(Account, "....")

although, I personally wouldn't do the whole query in a string literal that way, but from what I understood that's how you're required to do it

[solid light]

I have actually got validation functions for making sure it is what I want it to be

Although it is manual

[terse sky]

yeah I mean you'd have to call a function afterward

[solid light]

Yeah

[terse sky]

here, you don't have any extra repetition because you either annotate the variable, or you pass the type to the function

but if the type is passed to the function that makes it a) mandatory, b) you can do a validation check immediately to make sure the type and query are consistent

[solid light]

Yeah

I mean that's definitely something to consider

Just not something I plan on doing immediately

I'll add a note about it though

[terse sky]

that's what my deserialization functions tend to look like, I pass the name of the dataclass I want to deserialize to, to the function

[solid light]

I gtg now, but thanks everyone for the help 👍

[trim tangle]

collections.Counter[str] (or typing.Counter[str] if you're not on 3.9 yet)

yes

well, it's Counter

from collections import Counter

c: Counter[str] = Counter()

[void panther]

Counter[str]() is also an option, though I don't think I ever saw that style compared to annotations

[trim tangle]

It sounds foreign at first, but other languages have Vec<u64>::new() and it's fine 🤷

I think it's something like new Counter<string>() in C#

[soft matrix]

i like it but only if youre on 3.9+

[gray zenith]

how do you type hint something in a list?
e.g

def test_list_items(the_list):
  #code here

I want to type hint what the things in the list need to be, and all the items in the list are the same object to be clear.

[trim tangle]

What version of Python do you have?

[soft matrix]

from __future__ import annotations

def test_list_items(the_list: list[int]):
```this is a list of int

you can put any type in list[...]

[gray zenith]

oh thanks, just what i needed

👍

[soft matrix]

the future annotations is only necessary if you have <3.9

[trim tangle]

Note that most of the time you don't want to annotate it as list. Most of the time you just need it to be a sequence, or an iterable.

For that, use collections.abc.Sequence or `collections.abc.Iterable

[gray zenith]

ok thanks :)

[trim tangle]

you can't

You can only do that with tuples

[hearty shell]

What do you mean by two-item iteravle, do you mean an iterable of tuples?

[trim tangle]

why do you want that? @rustic gull