type-hinting

[trim tangle]

It's mostly useful when two pieces of code interact with each other. Like, your editor can immediately tell you that you made a mistake when using some library. For example, passing a value that could be a str or None to an argument that must always be a str.

It also enables editors to have stuff like "rename method" and "jump to method definition"

[sick parcel]

Hello, got a question about Optional .

Let say I have a class attribute:

class SomeClass:
  attrib: Optional[Set[str]] = None

  def check_attrib(self) -> bool:
    # perform null check
    if not self.attrib:
      return True
    if 'somevalue' in self.attrib:  # <type hinting warning
      ... # rest of the codes

and I have a method that use that attributes and will perform null checking.
Because the attribute can be optional and init as None , how do I prevent the warning in the subsequents codes after the null checking is completed?

is there other method other than just #type: ignore?

[chrome hinge]

Your snippet actually works fine in pylance.

[sick parcel]

does it require additionally setting? I am using pylance.

in vs code

[chrome hinge]

(Strictly speaking though, it's correct for it to not work, because what if between these two lines, something sets self.attrib = None?)

For it to be correct you could do attrib = self.attrib, check attrib for truthiness, and then you can be sure it's valid.

[chrome hinge]

Hmm, that's weird. Might be a pylance version difference.

[sick parcel]

ah thanks.

i have something like this.

maybe I should just use set() instead of Optional and assign as None?

[chrome hinge]

❯ bat pyright_test_attrib.py
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: pyright_test_attrib.py
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ class SomeClass:
   2   │     attrib: set[str] | None = None
   3   │ 
   4   │     def check_attrib(self) -> bool:
   5   │         # perform null check
   6   │         if not self.attrib:
   7   │             return True
   8   │         if "somevalue" in self.attrib:  # <type hinting warning
   9   │             ...  # rest of the codes
  10   │         raise
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

MiscPython on  master [!?] is 📦 v0.1.0 via C v11.4.0-gcc via 🐍 v3.11.4 (venv) 
❯ pyright --version; pyright pyright_test_attrib.py
pyright 1.1.332
0 errors, 0 warnings, 0 informations

(i changed it slightly but it should be equivalent)

[chrome hinge]

That wouldn't be valid - None isn't a set.

[chrome hinge]

Huh, interesting, you can actually see in this screenshot that it correctly infers it to not be None due to that if-statement.

[sick parcel]

ah right.

[chrome hinge]

wait a minute.

[sick parcel]

it is not complaining about None

[chrome hinge]

the warning in your screenshot actually says "pylint", not pylance

[sick parcel]

😮

[chrome hinge]

so maybe it's pylint being dumb while pylance is okay with this? I don't use pylint in vscode so I'm not sure.

[sick parcel]

silly question. how do I switch to pylance. I installed it a while back but maybe I didn't config the json setting probably ? I only recently turn on type-check to basic, so I wasn't aware of that before :D:D

[sick parcel]

think i figure it out

the problem is that the set[str] | None can only be checked when it is a parameters.

but cannot be checked, or maybe is typed incorrectly, when it is a classvar

[chrome hinge]

hmm, i don't see why not

[quartz obsidian]

This is my first day using mypy and I cannot for the life of me figure out why this,

# pick removal function based on whether backup is a file or directory
removal = shutil.rmtree if os.path.isdir(backup_path) else os.remove
# remove backup
removal(backup_path)

passed through mypy gives me,

error: "object" not callable  [operator]

Could somebody give me some tips?

[trim tangle]

sounds like a mypy bug 🙂

Probably related to rmtree being complicated in typeshed: https://github.com/python/typeshed/blob/main/stdlib/shutil.pyi#L94-L132

pyright doesn't complain: pyright playground

[hallow flint]

yes, it’s a mypy inference choice that is commonly suboptimal in ternary expressions (join vs union)

if you give removal an explicit type hint with Callable, it might be able to figure it out

[solemn sapphire]

whoops, discord didn't ping me at all for this message, weird.

The thing is mypy infers the type as tuple[int, ...] and not tuple[int, int]

and tuple[int, ...] is not a subtype of tuple[int, int], so I get type errors when I assume that is the case

(I believe they are invariant?)

[trim tangle]

tuple[A, A] is a subtype of tuple[A, ...], but not the other way around

[solemn sapphire]

right

[undone saffron]

Little ugly, but if you need that...

def get_seq() -> Sequence[int]:
    return (0, 0)


b0, b1, *_ = get_seq()
b = b0, b1
reveal_type(b)  # Type of "b" is "tuple[int, int]"

Seems like there might be a better way to handle this with more info, but if working in isolation with just this part of it and not holistically with thinking about the entire application's types...

[solemn sapphire]

I had a bunch of 2-tuples that I needed to add up.
I initially did:

data = [(1, 2), (2, 3), (3, 4)]
res = reduce(lambda acc, component: map(add, acc, component), data)

But a map[Any] is incompatible with tuple[int, int] (or so mypy says) and eagerly converting the map into a tuple also doesn't work, because tuple[int, ...] is incompatible with tuple[int, int].

changing it to

a = map(lambda component: reduce(add, component), zip(*data))

also didn't work.

I could perhaps do

sx, sy = 0, 0
for x, y in data:
    sx += x
    sy += y
res = (sx, sy)

and perserve the types, but that's not fun :p

[high wigeon]

Seeing these subtle differences between mypy, pyright, pylance etc, I have a question. Is it generally (fairly) well defined (by the PEPs, I assume) what type checkers for Python should, and especially should not, do? That is, if I write code that type checks on one type checker, then is it a valid statement that absent bugs and unimplemented features, it should also type check on another? Or would a type checker be "free" to do more advanced deduction and accept types that it can show are correct, perhaps using much more heavy machinery on the code, even if that usage has not been contemplated in the PEPs?

[chrome hinge]

You could make a function that adds two tuples which preserving the types:

T = TypeVar("T", bound=tuple[int, ...])
def add_tups(a: T, b: T) -> T:
    return tuple(x + y for x, y in zip(a, b))

It works in pyright: https://pyright-playground.decorator-factory.su/?gzip=H4sIAOwkNWUC_1VPy2rDMBC86yuGnCSqGpr0ZEi-QvQSjFGjdTE4klhJwe7Xd1Oali7sA2Zmd3bidEXd8hw_MF9z4gq3ZXrzbOFLIa5jpBux-gGnFi81paUo5XB8cPXO7SzeU4vhWFte6DzHatF13WBUoAk-hFGAon0PJ0ypBs8nuF5Bgqk2jviW6hVP2DAlxmplmCM-56y9qIxRKvjq5fBZv1jsjYXeWxzu_WDxagbFVAT-tdkxhXYh_TBgcV9ghHYjv4zyOWmRGLX2-HMuOcgWAf4RV_MF4BI5_S8BAAA%3D

ah, or rather like this: https://pyright-playground.decorator-factory.su/?gzip=H4sIAColNWUC_0WP0WrDMAxF3_0Vlz7ZzAus3VOg-wqzlxCCVyvDkNpGdkazr586llVgyVj3yLoz5yvaVmL6RLyWzA1uK_Tu2cLXStymRF_E6q85r-nScl6qUg7nXasP7mDxkdcUzm0tCw0xNYuu60ajAs3wIUzSqNr3cKKUbPD8BtcrSDC1lRN-UX3DEzbMmXGzcokJ37FoL5QxSgXffL_E2obHR3LGUbYZ9IvF0Vjoo8XpXk8Wr2ZULBb8MolP0vcBRl6qAP9uOqawXkjve1rssgcoiPkBQIjIvS8BAAA%3D

[solemn sapphire]

formulating it like that works, but the function seems to have an error

[chrome hinge]

it does, yeah. Not much can be done about it I think, for it to not be an error would require statically inferable iterator lengths. But that error can be type:ignored, and the rest of the code is clean at least.

[solemn sapphire]

TypeVarTuple could perhaps solve that, https://pyright-playground.decorator-factory.su/?gzip=H4sIAJ4nNWUC_21Qy2oDMQy8-ytETjZVC016CrT33tNeFhPcXbmYbGyjlUu2X1-b7fYBMRiLkWY0Y8_pDDLnEN8hnHNigZeYXX9COMyZXh0fSh4JgemD3Hiso6R8I6VM7CTxSnPDoL5LX2IvKY2TUs9RJnj8p6U3DdwYpQbyjXaUkift9iCt3S37uzZkLcLbVdzA7dM1fK-gHiYpHJe-vtzM4KvPC84QInyGrF2VNc2Ak7p2DJN0i1aIgvVaWz13-h5haxD0FmHX3h3Cg7Hqz1fopmAq0kL-xL5jGkpPes2GsI79EivFfAH7kDiJfQEAAA%3D%3D

mypy seems to crash, persumably because the features are incomplete

[oblique urchin]

The specification in the PEPs isn't very precise, and different type checkers can work very differently

There is an effort underway to work towards a better specification (PEP 729)

Personally though, I think we should aim for consistency mostly in places where interoperability matters, such as in the interfaces to libraries. If you write a library, the types you use should be interpreted the same by different type checkers

But for type checking within a function, that concern isn't as strong. And it's important for different type checkers to be able to innovate

[undone saffron]

IMO the potential for type checkers to innovate is directly stifled by the goal of interop, and it really needs to be the place of specification to state what should be done, and then for type checkers to just do it.

right now, if you write a library intended to be used by others, you can only use features that mypy and pyright both agree on how they work in the public interface

so why isn't that just specification instead of implicit?

of course, if the specifications only match the current supported set, it then becomes stifling on improvement as well.

google's pytype doesn't support latest python versions, but has better inference that checks everything you are doing is consistent in totality rather than only with first use

it's a really messy situation

[high wigeon]

So I wonder, coming from a theoretical computer science background, and with the caveat that while I'm a heavy user of mypy and familiar with type systems such as those of Haskell and ML, I haven't looked almost at all yet into how these are implemented by Python type checkers.

I know that any kind of formal semantics for the entirety of Python would be a huge undertaking, but would it be more feasible to define such a thing for typing, at some level of abstraction? I don't mean necessarily as anything normative, but for example as a way to reason about typing and to try out different ideas. It would presumably allow for experiments, as well as very specific statements about what different type checkers do and don't do, or what they do differently.

And don't get me wrong, I'm not saying this is what should be done and especially not that anyone else than me should work on anything like that. Things like this just fascinate me, and might be an interesting way for me to dive into Python typing internals :)

[undone saffron]

that's the route I'd prefer for resolving this, but there's not many who share that opinion

Because python's type system doesn't have HKTs, it would be entirely possible to have everything be inferred strictly by interaction with primitives.

(pytype does some level of this already)

there's some prior discussion on this if you want to see what's been tread on this
https://discuss.python.org/t/a-more-useful-and-less-divisive-future-for-typing/34225/
https://discuss.python.org/t/typing-stability-evolution/34424
https://discuss.python.org/t/proposed-new-typing-governance-process/34244

[undone saffron]

sorry, i started on the pytype tangent and got distracted and lost my train of thought for a moment,

one of the reasons I think the type system shouldnt be defined with any regard to type checkers, and that type checkers should just be responsible for working within the spec, and that we should make the spec put more work on the tools, not the users:

def unique_from_iterables(*iterables: Iterable[T]) -> set[T]:
    return set().union(*iterables)  # pytype has no issue with this, mypy and pyright do

def should_be_fine():
    l = []
    l.append(1)
    l.append("this")
    print(*l, sep=", ")
    # pytype is fine with this, the type was never explicitly constrained by an annotation
    # and all uses of it are consistent with inferred possible types

def shouldnt_be_fine():
    l: list[int] = []
    l.append("fail")

Right now, the most popular type checkers force users to do more work for the type checker rather than only doing the work when it is their intent to further limit the type.

[eager vessel]

Honestly I personally don't think that's a problem, when you create an empty container it's type is unknown and I wouldn't want my type checker to do any guess work here

[undone saffron]

it isn't unkown, it's directly infered from how you use it rather than assuming your use must be wrong by default, pytype checks that all uses of it are consistent with the information it has

[eager vessel]

I just wouldn't want a type checker to do any guess work here as I said 🤔

[undone saffron]

it isn't guess work tho

l = []
l.append(1)
(s.encode() for s in l)

pytype still fails this for example

[solemn sapphire]

It isn't doing any guess work, its just inferring, set() is basically some set[T] and then when you do set().union(Iterable[U]) it infers the type of T to be U

[undone saffron]

it's literally using the information you provide not just in annotations, but if you provide an annotation, it will factor that in

This is incredibly powerful, because you can start by writing code, use reveal_type if you need to check your dev assumptions, or to copy a complex type's signature in to "lock it in" based on how you actually use it.

(or leave it annotation free if you only care that people use it consistent to your use)

[eager vessel]

Mypy is actually fine with

from typing import Iterable, TypeVar

T = TypeVar("T")


def unique_from_iterables(*iterables: Iterable[T]) -> set[T]:
    return set().union(*iterables)

In here it constraints l to be a list[int] because that's the first element appended to it I think 🤔

def should_be_fine():
    l = []
    l.append(1)
    l.append("this")
    print(*l, sep=", ")

[undone saffron]

pytype does not

[eager vessel]

You can always create an issue in mypy if you think it's a bug or it should work differently

Personally I prefer current behavior

[undone saffron]

you should read the full conversation this is related to, as it isn't sufficient to handle this at mypy

There's problems with type checkers diverging in behavior, and it's my opinion that tools should do more of the heavy lifting (That's what tools are for). You can keep annotating as you are and have no change on your part if this were the "standard behavior."

[eager vessel]

As I said I prefer current mypy behavior personally

[solemn sapphire]

and with --new-type-inference mypy is soo much nicer than pyright

[eager vessel]

I mostly use --strict so that wasn't a big change for me

[undone saffron]

doesn't really matter for (public) library authors. you're going to have to support both mypy as the most common CI tool and pyright as the most common baked into language server if you intend to support typing

[solemn sapphire]

The pylsp plugin's readme stated --strict was too strict to be useful

[eager vessel]

I don't know what pylsp uses under the hood

[undone saffron]

that's a ... rather opinionated stance

strict is arguably the only useful way, and still doesn't provide enough

[eager vessel]

And I use mypy as a command-line tool and in CI 🤔

[undone saffron]

(arguably, it depends on your goals how strict is useful)

[solemn sapphire]

https://github.com/python-lsp/pylsp-mypy

Its a plugin that allows using just mypy for typechecking

[eager vessel]

Eh, strict isn't that bad

[undone saffron]

I don't start new python projects without a pyproject.toml that sets pyright to strict anymore, mypy as well if it's intended to be used as a library (for the reasons of consistent behavior this started with)

strict can be a pretty negative experience if you're dealing with a lot of untyped libraries or untyped data from external sources, though arguably you should be validating that data meets expectations before using it anyhow. There are some nice libraries that are lightweight (Small, easily reviewable, and fast) and type-friendly (msgspec being my personal favorite here) that can help with that.

[eager vessel]

Honestly I didn't see many untyped libraries myself, but that could happen

[undone saffron]

ecosystem pressure (for both better and worse) has led to a lot of libraries adding type information.

[eager vessel]

For better, imo

(Looking at you, Django)

[undone saffron]

if the specification and tooling were smarter, arguably they shouldn't have needed to and the tooling could have just seen if the use was consistent :)

there's been a lot of additional work for various library authors because type checking inference isn't good in popular type checkers.

[eager vessel]

I would much prefer having annotations in my code to the absence of them

Including code of the libraries I use

[undone saffron]

and nothing about inference being better would preclude that

library authors would even be Encouraged to at least copy the infered type (such as with reveal_type) at api boundaries.

[eager vessel]

So where would inference would help much? 🤔

[undone saffron]

because they could limit the typing to where it matters for their users with less work, and ensure the types remain consistent at api boundaries (via placing an annotation there) while remaining more flexible in internals without needing type ignores or seperate stubs

[eager vessel]

I don't know, but this somehow feels worse than using strict typing

[undone saffron]

why? [in that hypothetical] If the api boundaries have an annotation there, it's checked that the library is behaving consistently at that boundary, and the users are too

[high wigeon]

I was actually pointed at this channel after asking where I could ask deep questions about Python type checkers after trying to figure out the answer myself. The motivating question, surely one that is easily resolved by looking at the source code, is if any Python type checker defines the rules in a declarative fashion or if they all just use procedural code.

[undone saffron]

the internals can be flexible, but still must be consistent with that

[high wigeon]

I think I prefer it to constrain it. It's much less likely that I want a heterogeneous collection, and I'm fine annotating it in that case if in exchange the cases where I didn't intend it get caught.

[eager vessel]

Honestly this feels like a programmer's mistake:

l = []
l.append(42)
l.append("")

This doesn't:

l = []
el: int | str = get_something()
l.append(el)

[undone saffron]

and yet the former can still be checked properly for consistency

there's nothing preventing the latter from being enforced by things that arent the type checker, such as a lint rule for no-implicit heteregenous collection if you have that as a project preference, but type checkers enforcing it when it isn't neccessary creates more work.

[undone saffron]

could look at pyre, which is written in ocaml and should be closer to this (though I think there's still a lot of procedural checks)

pytype uses a graph to determine consistency here which might be interesting to you as well

if you're looking for a fully formal type grammar, I haven't seen one for a type checker

[eager vessel]

I think it was introduced here
https://github.com/python/mypy/issues/1055

[undone saffron]

that's not what im talking about

at least not fully

[eager vessel]

Honestly I would rather go with strict typing than inference 🤔

[undone saffron]

@high wigeon There's also this https://gist.github.com/mniip/0bfb52f35da3ad6c96878082e6af37e6 which was written by someone who did a little bit of feedback into the intersections WIP PEP which is a bit mroe formal, but assumes interpreting python according to the results of a more generic math paper.

[undone saffron]

the two aren't incompatible in pytype, you shouldnt have to choose at the type checker level.

if you place a stricter annotation, your annotation is the rule

[eager vessel]

Doesn't it already kind of works this way in mypy?

[undone saffron]

no

there's a lot of inference in mypy that is based on the first scope mypy sees you use it rather than checking that all uses are consistent at the point in code flow which they exist.

[high wigeon]

Why would the only role of a type checker be consistency? I think it should be as helpful as possible, which often means requiring explicit annotations in cases that are likely to be bugs.

[high wigeon]

Interesting, thanks!

In principle, I'd be interested in a logic that can express propositions about types, and propositions like "In Python [type checker X], there is no way to type this code [or other code with the same effect] so that given input X we get type Y". Perhaps especially for function types (like zip, which I think is now possible, and asyncio.gather, which I think is not). But I realize this is a high dream.

[undone saffron]

checking for consistent use with known types. That is, don't assume a lack of annotation should be an error, and use all available information. Being stricter just happens by adding more strict annotations where more is wanted then

Type checker shouldn't not use available known info

[undone saffron]

And checking for consistency with use based type knowledge still catches "likely bugs" because it wouldn't be consistent to not handle how the types differ when that matters

Anyhow, the direct reason it shouldn't check more than just what is typed for consistency with those types @high wigeon is that doing more because it may be a bug (and not strictly a type error) creates divergence, making only what is agreed upon by all type checkers in their strictest configuration appropriate for library authors

There should be a seperate tool for that that doesn't infect the ecosystem and only checks the projects that agree with those tools opinions (by them using it). We tend to call those linters :)

[high wigeon]

Interesting take. This reminds me of a discussion when someone had written a "clever merge tool" that understands variable renames (it was not interactive). My view of that was that variable renames likely happen for a reason, so it's better to require a human to take some time to look at it to ensure it makes sense. Or spell checkers. Spell checkers that accept all correctly spelled incredibly rare words (or syntactically sensible sentences) are pretty useless because they don't catch common mistakes. Asking someone to add stricter types doesn't seem like a solution; that would just mean that to be confident, you'd need to add types everywhere.

That's why I believe the default should try to be the most helpful choice and not, for example, imagine a concocted custom type that supports all the operations (which likely is completely possible in Python!).

[undone saffron]

But it wouldn't be a random concocted type that allows everything...

We can correctly, from static information, infer so much more than mypy or pyright do. I'm not saying to just allow anything if it isn't typed, I'm saying better inference would still provide value without having diverging behavior caused by opinionated things not defined in the type system.

[reef yacht]

We could also have domain names map to types. Not strictly inference, but similar. https://kodare.net/2023/02/02/names-can-be-so-much-more.html

This would be more useful imo. Less magic, more enforcement of rules.

Also easier to build good tooling and less noisy code.

[high wigeon]

Random thought: I wonder how much overlap there is between protocols and type stubs. Maybe I haven't used type stubs enough to appreciate them, but protocols feel so much neater, and I'd want to even type modules with them. Maybe modules could have a protocol in addition to all that would allow neater implementation hiding.

I know that a module can be compatible with a protocol. I suspect, though, that protocols cannot have everything that a module can have, like type definitions. Still, I'm intrigued by how much overlap there seems to be between these two things.

[trim tangle]

The purpose of a type stub is different. It is needed to provide an external source of type information to an otherwise "untyped" module

For example, a C extension. Or a library that doesn't want to have inline type annotation (the standard library, or requests for example)

they're probably more similar to a class definition

[tranquil turtle]

im curious, why standard library doesnt want inline annotations?

i can come up with several reasons that cause unnecessary overhead:

• almost everything would require importing typing
• some functions have extremely complicated signature
• some modules would require importing other ones for typing purposes

[high wigeon]

Yes, I understand that. I must be weird, but I always found writing type stubs for untyped 3rd party libraries and their separate type stub path mechanic rather clumsy. Also, I'd actually like to be able to define a similar stub for my own modules within a project instead of just using whatever types there are. This would be wonderful to abstract away some confusing details of types only used internally, to hide internal names, ... . Feels like I'd almost prefer defining a protocol and casting a module as it, because you can do those things in it (but probably there are some things you cannot), also because that would be handled by what feels to me like the standard Python machinery instead of a very separate type stub thing.

So, what I want to say: I understand that they are meant for different purposes, but they seem surprisingly similar to me to be that separate.

[oblique urchin]

a couple other ones: slower release cycle to get typing improvements relative to typeshed; difficulty to make type checkers use the standard library code (though that's probably solvable)

and just not that much advantage to adding types now

[high wigeon]

Not entirely unlike modules in ML where you can actually define a module interface and implementations separately. Of course, Python is not ML and doesn't want to be :)

[trim tangle]

Not sure about mypy, but pyright should understand something like ```py
class MyProto(Protocol):
...

import foo as _foo

foo: MyProto = _foo

[high wigeon]

I believe (based on a mypy bug that I read) that it does understand that.

I haven't tested in practice how much can be done with those. I suspect this approach will fail at least when you start having type definitions in those modules, because probably protocols don't support all that.

[rustic gull]

is there a way to fix this?

[undone saffron]

remove the uneccessay if GUI: and just do them both at once.

[rustic gull]

thanks 🫶
i think i prefer the imports to be together at the top (in the actual file there is some code between the two if's)

i already added #type: ignore but was more just making sure that this was intended behaviour

[soft matrix]

you can assign it as None in the else case and then check it against none

[fluid jay]

this looks a little bit ridiculous, am I following mypy correctly (i assume so since mypy shows no errors)
https://paste.pythondiscord.com/OUMQ

[trim tangle]

You probably want something like TypedDict

if you actually want to validate the JSON data, use pydantic, cattrs or similar libraries

[fluid jay]

i just needed the typehints

typeddict it is

thank you

[rare scarab]

If you need it separated, do this. ```py
if GUI:
from tkinter import filedialog as fg
else:
fd = None

if fd is not None:
fd.askopenfilename()

[shell minnow]

I'm working on type hinting in pytables, which makes use of generated classes. I'm having problems getting mypy to identify the generated classes as classes and feel I need help.

Here is the code I'm currently working on (type hints are my additions and may not be correct:)

T = TypeVar('T', type[IntAtom], type[UIntAtom], type[FloatAtom])

def _create_numeric_class(baseclass: T, itemsize: int) -> T:
    """Create a numeric atom class with the given `baseclass` and an
    `itemsize`."""

    prefix = '%s%d' % (baseclass.prefix(), itemsize * 8)
    type_ = prefix.lower()
    classdict = {'itemsize': itemsize, 'type': type_,
                 '__doc__': "Defines an atom of type ``%s``." % type_}

    def __init__(self, shape: Shape=(), dflt: Any=baseclass._defvalue):
        Atom.__init__(self, self.type, shape, dflt)
    classdict['__init__'] = __init__
    return type('%sAtom' % prefix, (baseclass,), classdict)


Int8Atom = _create_numeric_class(IntAtom, 1)
Int16Atom = _create_numeric_class(IntAtom, 2)
...

So what I would want is for Int8Atom and Int16Atom to be identified by mypy as classes. Is this possible at all with mypy and normal python type hinting?

[grave fjord]

You'll need to do it in an if TYPE_CHECKING block with a class statement

[shell minnow]

Ok, so you're saying the type-hinting system can't understand function generated classes? So I essentially need to expand each of the generated classes manually for the type-checker to be happy?

[sour kestrel]

Python 3.8.17 (default, Sep 14 2023, 13:58:34) 
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from typing_extensions import get_type_hints, Annotated
>>> from typing import Optional
>>> def f_no_default(p: Annotated[Optional[str], "abc"]) -> None: ...
... 
>>> def f_with_default(p: Annotated[Optional[str], "abc"] = None) -> None: ...
... 
>>> get_type_hints(f_no_default, include_extras=True)
{'p': typing_extensions.Annotated[typing.Union[str, NoneType], 'abc'], 'return': <class 'NoneType'>}
>>> get_type_hints(f_with_default, include_extras=True)
{'p': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'abc'], NoneType], 'return': <class 'NoneType'>

In the above example, is it expected that the default None value would cause the type to be wrapped in the extra Union with NoneType?

Oh

Changed in version 3.11: Previously, Optional[t] was added for function and method annotations if a default value equal to None was set. Now the annotation is returned unchanged.

[sour kestrel]

is the only way to be able to handle these types agnostic to py ver to undo this myself?

def _unwrap_implicit_optional_hints(defaults: dict[str, Any], hints: dict[str, Any]) -> dict[str, Any]:
    """Unwrap implicit optional hints.

    On python <3.11, if a function param annotation has a `None` default, it is unconditionally wrapped in an
    `Optional` type. This function reverses that process.

    Args:
        defaults: Mapping of names to default values.
        hints: Mapping of names to types.

    Returns:
        Mapping of names to types.
    """
    for name, default in defaults.items():
        if default is not None:
            continue

        hint = hints[name]
        origin = get_origin(hint)
        args = get_args(hint)
        if origin is Union and len(args) == 2 and args[1] is type(None):
            hints[name] = args[0]
    return hints

any gotchas here?

[sour kestrel]

ok yeh, there are gotchas

if annotated isn't involved the annotation will come out flattened, but if annotated, then not.

so have ended up with this

def _unwrap_implicit_optional_hints(defaults: dict[str, Any], hints: dict[str, Any]) -> dict[str, Any]:
    """Unwrap implicit optional hints.

    On python <3.11, if a function param annotation has a `None` default, it is unconditionally wrapped in an
    `Optional` type. This function reverses that process.

    Args:
        defaults: Mapping of names to default values.
        hints: Mapping of names to types.

    Returns:
        Mapping of names to types.
    """

    def _is_optional(origin: Any, args: Any) -> bool:
        return origin is Union and len(args) == 2 and args[1] is type(None)

    for name, default in defaults.items():
        if default is not None:
            continue

        hint = hints[name]
        origin = get_origin(hint)
        args = get_args(hint)

        if _is_optional(origin, args):
            unwrapped_inner, _, _ = unwrap_annotation(args[0])
            if not _is_optional(get_origin(unwrapped_inner), get_args(unwrapped_inner)):
                continue

            hints[name] = args[0]
    return hints

🫣

[sour kestrel]

it passes these tests:

def test_get_fn_type_hints_with_none_default() -> None:
    def fn(
        a: Annotated[Optional[str], ...] = None,
        b: Optional[str] = None,
        c: Union[str, None] = None,
        d: Union[str, int, None] = None,
        e: Optional[Union[str, int]] = None,
    ) -> None:
        ...

    hints = get_fn_type_hints(fn)
    assert hints == {
        "a": Annotated[Union[str, NoneType], ...],
        "b": Union[str, NoneType],
        "c": Union[str, NoneType],
        "d": Union[str, int, NoneType],
        "e": Union[str, int, NoneType],
        "return": NoneType,
    }

[tight forge]

let's not spam

[hollow prism]

am I not supposed to put anything else other than type annotations in a TypedDict subclass? even though this is fully functional

from typing import TypedDict

class TestDict(TypedDict):
    a: int
    b: str
    c: float

    @classmethod
    def test_init(cls, a, b, c) -> "TestDict":
        return cls(a=a, b=b, c=c)


test = TestDict.test_init(1, "2", 3.0)
print(test)

[void panther]

calling the classmethod from instances wouldn't work, as any TypedDict is a direct subclass of dict and instantiating the class will give you a dict

[hollow prism]

ah okay

[cunning plover]

Incompatible types in assignment (expression has type "ArrayField[Union[Sequence[Union[str, int]], Combinable], List[str]]", variable has type "List[str]")  [assignment]mypy

i am totally confused how to type ArrayField of Django ORM

"ArrayField" expects 2 type arguments, but 1 given [type-arg]mypy error on ArrayField[models.CharField]

hmm..

values = ArrayField(
        models.CharField(max_length=key_max_length, db_index=True)
    )

original value is like this

    pk_values: ArrayField = ArrayField(
        models.CharField(max_length=key_max_length, db_index=True)
    )

works good enough if not specifying its insides

[reef yacht]

pk_values: ArrayField[int]?

[hardy linden]

I'm running into an issue where it seems like when a subclass narrows the type of elements of a data structure inherited from the parent, methods inherited by the subclass that refer to that data structure don't see that the type of its elements has been narrowed.

Here's a mockup of my structure.

class Foo:
    bars: list[Bar]
    @property
    def true_bars(self):
        return [bar for bar in self.bars if bar.some_boolean is True]        

class ChildOfFoo(Foo):
    # Important: ChildOfFoo expects only ChildOfBar in its bars
    bars: list[ChildOfBar]  # Overrides (narrows) type hint in Foo
    # true_bars is inherited from Foo

class Bar:
    some_boolean: bool

class ChildOfBar(Bar):
    some_boolean: bool  # Inherited from Bar
    other_boolean: bool

Okay, so with all that in mind, here's the issue I'm having.

Because the type of bars in ChildOfFoo is more specific than it is in Foo (each element should be a ChildOfBar, not just a Bar), then if I understand everything correctly, that means that if I do my_cof = ChildOfFoo(), the result of my_cof.true_bars should be a list[ChildOfBar], not just a list[Bar].

But that doesn't actually happen. Instead, mousing over my_cof.true_bars (in VS Code, via Pylance) indicates that its type is list[Bar], and if I write my_cof.true_bars[0].other_boolean, I'm getting a type-checking error from Pylance that Cannot access member "other_boolean" for type "Bar".

So my questions are:

• Am I simply wrong about the type system here, and I'm wrong to think that my_cof.true_bars should resolve to a list[ChildOfBar]?
• If not, is this something that Pylance is doing incorrectly, or something they've declined to implement?
• Either way, is there a way I can write this better so that the types are understood correctly (by me and by Python)? Obviously, I would ideally prefer not to have to duplicate the code of true_bars in each subclass of Foo.

[trim tangle]

Maybe you need a generic?

class Bar:
    some_boolean: bool

class ChildOfBar(Bar):
    other_boolean: bool

B = TypeVar("B", bound=Bar)

class Foo(Generic[B]):
    bars: list[B]

    @property
    def true_bars(self) -> list[B]:
        return [bar for bar in self.bars if bar.some_boolean is True]        

class ChildOfFoo(Foo[ChildOfBar]):
    ...

If every subclass of Foo is supposed to have its own specialized subclass of Bar, that's probably the way to go

[hardy linden]

... Yep, I think you're absolutely right

[trim tangle]

Actually this should not type-check

or wait, maybe it should

yeah it should

[hardy linden]

I haven't done anything with generics before and the examples I'd read before gave me the idea that the use-case for generics was just "writing a custom LinkedList or Queue class or something that's just a data structure"

but now that you say that, I can see how this applies

[trim tangle]

I have a WIP series of articles on generics: https://decorator-factory.github.io/typing-tips/tutorials/generics/

not sure how useful it's going to be given you already know about them, but just maybe

[hardy linden]

It'll be my first time at it, so I'll probably need a few different perspectives on it - cheers, I'll look through that

[trim tangle]

it is mildly abandoned, mostly because I'm lazy

so if you have any suggestions or questions, I'd be happy to hear 🙂

[hardy linden]

I'll take a stab at this and let you know if I run into any snags. Do you mind pings? nvm, it's in your profile

[young hawk]

did 3.12 get the ability to do union extends or are we still praying for that feature?

[oblique urchin]

I don't think so, but I don't know what you mean by "union extends"

in general, to get new features into Python, writing PEPs tends to work better than praying

[young hawk]

class MyModel(SqlModel, SqlModelMixn):

and I need to be able to accept a generic parameter that subclasses both of those basically

[oblique urchin]

oh you mean intersections?

[young hawk]

yeah I guess it would be wouldn't it

[oblique urchin]

That's not currently in the language but there's some work in that direction (https://github.com/CarliJoy/intersection_examples)

[hardy linden]

Augh. I just realized that I should check, and the docs are telling me that "Syntactic support for generics is new in Python 3.12."

The machine that I have to run this stuff on is (drumroll) 3.7. But Google's saying that Generics were introduced in 3.5. What parts, then, are unavailable to me? It... seems like you couldn't have Generics at all without the square-bracketing syntax, so I'm not sure how they could have been introduced in 3.5 without the syntax.

(Yes, I have successfully badgered the folks upstairs into getting a up-to-date version approved, but it'll take a while for that to happen.)

[young hawk]

yeah right now I end up having to do a typevar similar to this ```python
MappingModel = TypeVar('MappingModel', bound=SQLModel | MappingMixin)

And then doing a dance similar to this to make mypy happy:```python
        model = self.model if issubclass(self.model, SQLModel) else None
        mixin = self.model if issubclass(self.model, MappingMixin) else None

        if not model or not mixin:
            # making mypy happy is hard
            return None

Would love a better workaround if you got any ideas though 🙂

[oblique urchin]

Generics work on any version by subclassing typing.Generic (class X(Generic[T]):. What's new in 3.12 is dedicated syntax (class X[T]:).

[trim tangle]

yeah

[oblique urchin]

oops, fixed

[hardy linden]

Ah, okay! That's good news. Thank you!

[oblique urchin]

an alternative approach is to use a Protocol with all the attributes you need as the bound

[young hawk]

hrm, I'll give that a try

[hardy linden]

Just wanted to shout out that this is definitely the right answer - not only is it working, but I'm already able to strip out a bunch of kludgy overloads and #type: ignores as a result of it. Thank you!

The only followup question I have is - my real use-case has a couple nested levels of this; the types get narrower as I go down the chain, where a GrandchildOfFoo requires a GrandchildOfBar. I've drafted it up, and I'm not getting any typing errors, but just to double-check, this is not just syntactically valid, but how I should actually be doing this, right?

AnyBar = TypeVar("AnyBar", bound = "Bar")
# etc., etc.,
class Foo(abc.ABC, Generic["AnyBar"]): ...
class ChildOfFoo(Foo[ChildOfBar], Generic["AnyChildOfBar"]): ...
class GrandchildOfFoo(ChildOfFoo[GrandchildOfBar]): ...

[young hawk]

yup I was able to make it work with the protocol thanks! 🙂

[young hawk]

Ack, spoke too soon, when I go to actually use it I get

error: Type argument "ProfileUpload" of "AssetFamily" must be a subtype of "_UploadModel"  [type-var]
error: Type argument "ProfileImage" of "AssetFamily" must be a subtype of "_MappingModel"  [type-var]
``` doesn't seem to be doing a protocol check there

tried using the protocols directly in that scenario but now I get a type mismatch for the type of the protocol assignments

[young hawk]

aha, turns out that was a error in my protocol 😄 thanks again for the help

[young hawk]

hrm, is there any ability to type hint an expected inner class definition?

[teal skiff]

Hi y'all,

I'm trying to migrate a codebase that uses mypy to pyright. I haven't worked in this project for about 4 years so a lot of python I'm not aware of has happened since then.

I had a function argument annotated with argparse._SubParsersAction (ignore that I'm using a private type) which was giving me partially unknown type errors in pyright. So I added the generic like so argparse._SubParsersAction[argparse.ArgumentParser]. This made pyright happy, but then it failed at runtime with type '_SubParsersAction' is not subscriptable . I tried from __future__ import annotations which made the runtime error go away. Is this the standard way to deal with it?

I'm targeting python 3.10 and I have 3.11 installed in my laptop.

[rustic gull]

I am not sure if the question you are asking is about accesing private variables. If so, then you can easily access those private variables through following code syntax

Objectname._classname__privateVarName()

[teal skiff]

no my question is not at all related to using private variables, but more about type stubs being incompatible with runtime behavior.

but I think I answered it myself. To deal with these problems you either wrap the annotation in quotes or use from __future__ import annotations

[teal skiff]

So here is my next question. I 'm using pypdf 's PdfWriter and the append function (https://pypdf.readthedocs.io/en/stable/modules/PdfWriter.html#pypdf.PdfWriter.append) has an argument annotated with IO (without the generic). Pyright gives me a reportUnknownMemberType when using that method. I want to keep the strictness of forbidding unknown in my own code. Is there a way to keep the strict behavior in my code but treat the external dependencies as an explicit Any?. Or alternatively, is there a way to "instantiate" the method to a particular disjunct (I don't know what's the right terminology here, narrowing?)? I'm calling append with a path so I don't care about the IO

[grave fjord]

You might want to let them know and contribute a fix?

[grave fjord]

Also that feels like a pyright bug to care about external code missing generics

[teal skiff]

Sure, I think this could be fixed both in pypdf and pyright, but I'm sure people routinely have to deal with similar problems so I was wondering if there are any good known workarounds.

[trim tangle]

Pyright seems to have so many bugs, it might be easier to report the ones it doesn't have

||\lh||

[teal skiff]

I don't consider this a bug, but an enhancement

[high wigeon]

Uhh. Reading https://github.com/CarliJoy/intersection_examples/issues/5 really made me want to think about formalizing the type system somehow. Would be kind of neat to be able to just have a solver answer questions like "would this addition allow you to break the Liskov substitution principle". (Though it's likely to be messy, because I think the current type interpretations are apparently not sound.)

Not sure though if the logic would be one that any solver can tackle.

[eager vessel]

Why currently this isn't type checked correctly by mypy?

from result import Result, Err


def read() -> Result[str, ValueError | TypeError]:
    return Err(ValueError("Oops!"))


def main() -> None:
    result = read()
    match result:
        case Err(ValueError()):
            print(result)
            return
        case Err(TypeError()):
            print(result)
            return
    reveal_type(result)
    print(result.ok_value)

t.py:17:17: note: Revealed type is "Union[result.result.Ok[builtins.str], result.result.Err[Union[builtins.ValueError, builtins.TypeError]]]"
t.py:18:11: error: Item "Err[ValueError | TypeError]" of "Ok[str] | Err[ValueError | TypeError]" has no attribute "ok_value"  [union-attr]

result is https://github.com/rustedpy/result

@oblique urchin Sorry to ping you, bu maybe you have some insight into this too? Black appears to have it's won little result implementation too

[oblique urchin]

I suspect the type narrowing just can't handle this case. The type of Result[str, ValueError | TypeError] minus Err[ValueError] can't be expressed by mypy, so it can't do this narrowing

[eager vessel]

It can narrow it in this case: 🤔

from result import Result, Err


def read() -> Result[str, ValueError | TypeError]:
    return Err(ValueError("Oops!"))


def main() -> None:
    result = read()
    if isinstance(result, Err):
        match result.err_value:
            case ValueError():
                return
            case TypeError():
                return
    reveal_type(result)  # note: Revealed type is "result.result.Ok[builtins.str]"
    print(result.ok_value)

[oblique urchin]

yes, that's different

[eager vessel]

I suppose there's already an issue for a similar case on github?

[tranquil turtle]

you can try something like ```py
match result:
case Err(ValueError()):
print(result)
return
case Err(TypeError()):
print(result)
return
case Err(_):
typing.unreachable()
# here result cannot be Err, because all Errs are handled in match-case
print(result.ok_value) # should be ok

[eager vessel]

Thanks, I'll try tomorrow, but I don't think it would work since even in the case scope result is typed as Err[all errors]

[oblique urchin]

no, I think that should work. the case Err(_) should narrow the type to OK

(I assume Result is a union of OK and Err but I don't know this library)

[glacial pollen]

Is there any howto on how to use .pyi files to get typehinting in VSCode for a project I'm developing (i.e not for a pip installed module, but one under development)? I have tried just dropping a .pyi alongside the module in the same directory, and using pylance's default typings directory, each with no apparent success.

I'm currently trying to get the on-hover hints to work (which seems to be handled by pyright), but the aim would be to get mypy using the stubs, too.

[eager vessel]

It is a union

[trim tangle]

hey we have some big news

Pyright just launched an official playground https://pyright-play.net/? so my thing is kinda deprecated

[trim tangle]

( https://pyright-playground.decorator-factory.su now redirects to that)

[acoustic thicket]

end of an era

[trim tangle]

eaha rinky aoop

[proud brook]

Why is this a thing?

I guess for when you're not working on a project

[oblique urchin]

very useful for investigating bugs/features in type checkers

[sour kestrel]

Why is this an error:

from typing import Callable, TypeVar

ExceptionT = TypeVar("ExceptionT", bound=Exception)

class MyException(Exception): ...

def receives_exception(exc: MyException) -> None: ...

c: Callable[[ExceptionT], None] = receives_exception

error: Incompatible types in assignment (expression has type "Callable[[MyException], None]", variable has type "Callable[[ExceptionT], None]")  [assignment]
Found 1 error in 1 file (checked 1 source file)

but this OK:

from typing import Callable, Mapping, TypeVar, Type

ExceptionT = TypeVar("ExceptionT", bound=Exception)
ExceptionCallable = Callable[[ExceptionT], None]

class MyException(Exception): ...

def receives_exception(exc: MyException) -> None: ...

c: ExceptionCallable = receives_exception

Success: no issues found in 1 source file

[oblique urchin]

the latter is implicitly ExceptionCallable[Any]

[sour kestrel]

ahh understood, thanks!

[eager vessel]

Hm, this just wouldn't work as I want

If you add new error into errors union it wouldn't notify developer that that case isn't handled

[low escarp]

So not sure if this is a vscode issue, or something, but I feel like my code is correct...
I have the following:

class Other:
    def __init__(self, func, generic):
        self.func = func
        self.generic = generic

    def __call__(self, *args, **kwargs):
        print(f'Calling func with generic params: {self.generic}')
        return self.func(*args, **kwargs)

class Decorator:
    _generic = None
    def __init__(self, func):
        self.func = func

    def __new__(cls, func):
        return Other(func, generic=cls._generic)

    def __class_getitem__(cls: "Decorator", key):
        cls._generic = key
        return cls

@Decorator[int, float]
def func(x):
    print(x)

func(3)

vscode thinks that the @Decorator[int, float] is invalid:

It looks like this error https://github.com/microsoft/pylance-release/issues/4656
But I don't have this issue when I follow the steps in this ticket...

Code runs as expected:

Calling func with generic params: (<class 'int'>, <class'float'>)
3

[trim tangle]

I think type checkers don't really like __class_getitem__...

why do you need the generic params while decorating?

[low escarp]

ah damn. Weird that it works with the example in that issue but not in my case... 😦
And I'm kinda emulating some c++ code

there are basically some functions which are templated in c++, and so to make it nicely I figured I could do something like this:

@hooks.cTkMetaData.ReadGlobalFromFile[nms_structs.cGcWaterGlobals]
def func():
    # do something...
    pass

because the function in c++ has a name like cTkMetaData::ReadGlobalFromFile<cGcWaterGlobals>

[cunning plover]

is there a way to make generic MyList[T] as a type?
which will be checked by mypy

pretty much i wish to keep that my type has... Generic inside? 🤔 hmm

T = TypeVar("T")
class ChunkedList(list, Generic[T]):
    pass
    @classmethod
    def trycast(cls, value: List[T]) -> "ChunkedList[T]":
        return cast(ChunkedList[T], value)

look to be working

[tranquil turtle]

why not simply subclass list[T] ?

[cunning plover]

how? 😅

[tranquil turtle]

class X(list[T]):...

also, you probably should use typing.Self in return type of your method

and your code is not safe
ChunkedList[int].trycast([]).trycast([]) typechecks, but doesn't work at runtime

[cunning plover]

your version is not working for some reason. Mypy no longer checks it if it is nested value in Callable

plus unable to initialize some empty list without errors (Missing positional argument "value" in call to "trycast" of "ChunkedList" [call-arg]myp)

T = TypeVar("T")
from collections import UserList
class ChunkedList(UserList, Generic[T]):
    pass
    @classmethod
    def trycast(cls, value: List[T]) -> "ChunkedList[T]":
        return cast(ChunkedList[T], value)

this version is a good one

ensures i still have typed List methods

and works with mypy

(i get expected correct error when trying to pass List[variable] into ChunkedList[my_thing] variable)

ModelObj = TypeVar("ModelObj", bound=Model) # , 
class ChunkedQuerySet(Generic[T]): # if i add QuerySet inhertied it is no longer typed checked
    pass
    @classmethod
    def trycast(cls, value: QuerySet[ModelObj]) -> "ChunkedQuerySet[ModelObj]":
        return cast(ChunkedQuerySet[ModelObj], value)

Having trouble to do the same for QuerySet now

hmm

idea.

all functions that have ChunkedQuerySet, have ChunkedList. no exceptions

i will just use regular alias for queryset

ModelObj = TypeVar("ModelObj", bound=Model)
ChunkedQuerySet = QuerySet[ModelObj]

i am feeling confident i will be able to set everywhere correctly alias (Since mypy will point me where i am wrong if i did not use ChunkedList)

[cunning plover]

is there a way to type function as

can return Dict/List or can be not returning?

Optional is not working for that.
Mypy asks to have return statement

Union with NoReturn is not working too :/

Missing return statement  [return]mypy

error for function that is not having return

used workaround, and just disabled this type of error for this module 😅

[rare scarab]

Never is used to not return

or NoReturn

[cunning plover]

not working when NoReturn is optional or union part

[rare scarab]

used an overload?

@overload
def run(*, daemon: Literal[False] = False) -> None:
  pass

@overload
def run(*, daemon: Literal[True]) -> NoReturn:
  pass

[cunning plover]

legit idea 🤔 but too hard to propagate through all the code.
I typed original functions as

SerializableActionReturn = Optional[Union[Dict[str, Any], List[Any]]]

basic_action_func = Callable[[FloweyViewMixin, Serializator], SerializableActionReturn]

like this for example.

problem that it is not exactly original function, i decorate the original function with my own decorator and pass around its type as basic_action_func

ergh... i'll prefer just leaving disabled error

def basic_action(
    _func: Optional[basic_action_func] = None,
    /,
    *,
    # other params
) -> Callable[[basic_action_func], BasicActionBinder]:
    if _func is not None:
        action = BasicActionBinder(
            _func=_func,
        )
        return cast(Callable[[basic_action_func], BasicActionBinder], action)

    def decorator_repeat(func: basic_action_func) -> BasicActionBinder:
        return BasicActionBinder(
            _func=func,
        )

    return decorator_repeat

no way i will be able to configure overrides for all code like that (it is decorator for basic_action_func)

[cunning plover]

hmmm

    @decorated_stuff()
    def query_stuff(
        self, serializer: Serializer
    ) -> ReturnSerializableOrRaiseException:
        pass

do u like very thick hint to users that i expect them to raise Exception or return Serializable?

even if it is having bad kind of grammar naming, but it leaves no room to misunderstand that Exception needs to be Raised, not returned? 😅

i just feel like using any alternative wording/verb, will not make it understandable that it needs to be actually raised

may be even RaiseExceptionOrReturnSerializable ? 🤔

to exclude possibility of missunderstanding that Return can be related to exception in any way

Surely no one will missunderstand such type name as RaiseExceptionOrReturnSerializable 😅

Dirty cheating to lack of python typing syntax for raises Exception 😊

RaisesExceptionOrReturnsSerializable as a bit better grammar may be

[rare scarab]

There's so many discussions on discuss.python.org about why not to have checked exceptions.

[cunning plover]

this is library that explicitely awaits Exception for properly showing error related results in some return

So... i think it is good idea to document expectations a bit more thoroughly

[cunning plover]

it would be fine... if it is within my own code. I would be feeling okay enough i guess to leave documentation about Exception raising, but i document type this situation for external dev users. So feeling like some additional type documenting effort is justified

[fierce ridge]

i wouldn't mind having them as an opt-in feature, like in nim

if you don't want to check effects, you can ignore it. if you want to check effects, the system exists

[edgy gorge]

would everyone say that type racer helps your speed and accuracy in coding?

[ancient river]

not at all.

[fierce ridge]

this channel is about static type annotations in python code, not about typing on the keyboard

[edgy gorge]

Thanks

[sick parcel]

hello. I have a question about typealias on the Callable type, for example:

FuncSignature = Callable[[int, int], float] # (int, int) => float

Is there anyway I can do some type alias on the [int, int] of the signature's input, since some of my parameters is getting more complex and my signature become very lengthy.

like

FuncInputType: TypeAlias = [int, int]
FuncSignature = Callable[FuncInputType, float] # something properly done?

[chrome hinge]

Hmm, you could do

T = TypeVar("T")
IntIntFun = Callable[[int, int], T]

[sick parcel]

thanks but I want to typehint the [int, int] input part.

because some of my functions is getting more complex

something that can simplify the [int, int] in the signature but is also a valid typehinting syntax

[tranquil turtle]

you can use what Reptile provided

...
FloatCallable = IntIntFun[float] # Callable[[int, int], float]
StrCallable = IntIntFun[str] # Callable[[int, int], str]

[reef skiff]

Probably f.a.q, however, is it possible, to pass a type as a generic to the method call in Python, to annotate, for example, a concrete return type for the particular call?
Like in C#, I would be able to utilize generic as such: obj.Method<int>() to annotate return type of int.
So it would be something like:

def method[T]() -> T:
  ...

[trim tangle]

Nope, there's no way to change the behaviourof the function like that
You can pass a type as an argument though

[reef skiff]

Since, I'm doing a simple config class with CRUD operations, but since it's being used everywhere, it kinda ruins typehinting and checking, as I have to do -> Any basically in every call

[reef skiff]

Oh, sad

[trim tangle]

Maybe you could show your code?

[reef skiff]

Yep, sec

Say we have something like this:

class ProviderABC(abc.ABC):
    def get(self, key: Sequence[Any], default: Any = ...) -> Any:
        raise NotImplementedError

    def set(self, key: Sequence[Any], value: Any) -> None:
        raise NotImplementedError

    def require(self, key: Sequence[Any]) -> Any:
        raise NotImplementedError

    def exists(self, key: Sequence[Any]) -> bool:
        raise NotImplementedError

    def delete(self, key: Sequence[Any], required: bool = False):
        raise NotImplementedError

I have mypy on my project, and I was thinking: either disable return typechecking somehow, when I call provider methods, or add proper typehinting somehow, so that's why asked original question.

But if there's no generalized way to tell typechecker which return I will expect instead of Any on particular call, I will probably endup changing mypy config to ignore those calls.

[tranquil turtle]

you can make your class generic:

class MyClass[T]:
   def get(self, key: ...) -> T: ...

instance = MyClass[int]()
instance: MyClass[int] = MyClass() # or like that

instance.get() # -> int

[reef skiff]

That will make original provider more like a generic repository, right?
It is probably good for overall design, however, my goal was not to overcomplicate things and make a one interface to interact with different storages.

I understand that this is an antipattern and things, but for my current needs I don't think that engineering a whole DAL would be worth it

I will endup throwing # type ignore ig :D

okay, I might rethink things, maybe repository is not that hard to implement in my project, after all. Thanks for giving insight

[tranquil turtle]

you also can do something like this: ```py
def get[T](self, key: ..., cls: type[T]) -> T: ...

x.get(key, int) # -> int
x.get(key, X) # -> X
x.get(key, Something) # -> Something

but it is not a really good idea

[reef skiff]

Uh, oh, I see. Could you explain why is it bad? Except to the fact it practically does nothing whilst requires you to add one more argument to the signature?

Also, could you tell, which version of Python does support that? PyCharm warns me that 3.11 does not support that syntax.

[chrome hinge]

That's 3.12+.

[reef skiff]

Oh, thanks!

[tranquil turtle]

it is also bad because it is not actually typesafe

[brisk hedge]

Wasn't obj.method[int]() part of the 3.12 typing pep?

Or was it separated 🤔

[soft matrix]

its

!pep 718

[rough sluice]

**PEP 718 - Subscriptable functions**

Status

Draft

Python-Version

3.13

Created

23-Jun-2023

Type

Standards Track

[soft matrix]

not accepted yet

i really need to add more usecases for this at some point

but

https://cdn.discordapp.com/emojis/947108233183629333.webp?size=96&quality=lossless

[reef skiff]

Oh, cool, gonna dig that, thanks

[soft matrix]

well the thing is you cant actually use it at runtime to do anything

its purely runtime only for now

[trim tangle]

if you want it to work at runtime, you can always accept a type as an argument

[soft matrix]

thats boring though

custom descriptors are far cooler

[trim tangle]

yeah, why use an already working thing when we can make the language even more complex

though to be fair, TypeForm or whatever would be useful for passing type hints around

[soft matrix]

consistency™️

[granite stratus]

I have an insane idea that I'm not sure is possible. Mypy type hints are turing complete: https://arxiv.org/abs/2208.14755, and support recursive type constraints, etc (IIRC), but is it possible to do constraint programming with them? I saw this article https://ocamlpro.com/blog/2017_04_01_ezsudoku/ and was wondering if anyone knows if it is possible to do the same with python type hints.

[high wigeon]

Hey, I'm puzzling about a mypy error; could anybody help me? I have:

_T = TypeVar("_T")

class StartedToken:
    pass

class ProducerContext(Generic[_T]):
    pass

async def coro_to_aiter(
    coroutine: Callable[[ProducerContext[_T]], Coroutine[Any, Any, None]]
) -> AsyncIterator[_T | StartedToken]:
    yield None  # type: ignore[misc]  (just removed the implementation)

# tests
async def test_coro_to_aiter() -> None:
    async def coro(ctx: ProducerContext[int]) -> None:
        pass

    # Option 1: No error
    # c = coro_to_aiter(coro)
    # it = aiter(c)

    # Option 2: Error
    it = aiter(coro_to_aiter(coro))

and I'm getting this mypy error from the last line:

Argument 1 to "coro_to_aiter" has incompatible type "Callable[[ProducerContext[int]], Coroutine[Any, Any, None]]"; expected "Callable[[ProducerContext[int | StartedToken]], Coroutine[Any, Any, None]]" [arg-type]

Interestingly, splitting the line in two (see Option 1) does not give an error.

Also, are the type parameters for Coroutine actually documented somewhere? I search for them almost every time I use them...

[chrome hinge]

the docs just say:

The variance and order of type variables correspond to those of Generator

[soft matrix]

i thought i made them slightly better doced in 3.12

[high wigeon]

That could be :)

[chrome hinge]

Perhaps you can just avoid using Coroutine, but instead use Awaitable? That one only has one parameter.

[soft matrix]

https://docs.python.org/3/library/typing.html?highlight=coroutine#typing.Coroutine

[high wigeon]

No, I need to wrap it in a Task.

[high wigeon]

Ok, that's actually better than it used to be, I think :)

But then that's only for the deprecated (according to the docs) typing.Coroutine; https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine isn't very helpful.

[soft matrix]

it doesnt doc them currently

[oblique urchin]

those two should be exactly the same

but yes we should document the type params in the collections.abc docs too

[lunar dune]

I want to add a "Generators and coroutines" section to the top of the typing docs, similar to how we already have for tuples and Callables

Haven't got round to it yet

(We should also link to that from the collections.abc docs, definitely)

[primal raptor]

https://github.com/python/mypy/issues/15238#issuecomment-1546814785 What are the practical ramifications of this?

Context is the mypy PEP 695 issue, about not being able to adopt pep 695 syntax for a while. I'm just curious what that means in practise, like does it mean mypy won't support 695 for years to come?

[oblique urchin]

It just needs some work to implement it

When that happens depends on whether someone does the necessary work

Oh, and for that comment specifically: Alex is referring to the fact that typeshed stubs have to be syntactically compatible with all supported Python versions

So even if mypy supported PEP 695, we won't be able to use the syntax in typeshed (and in most libraries) until Python 3.11 goes out of support

However, users writing applications would be able to use it as soon as their applications are 3.12-only and mypy adds support

[primal raptor]

I'm unfamiliar what typeshed is? Oh I see, that's good

[oblique urchin]

typeshed is the repository that collects types for the standard library and some third-party packages

[primal raptor]

So that's basically talking about adopting the feature for the standard library / third party packages?

[oblique urchin]

yes

[primal raptor]

Alright, thank you Mr. core developer 😄

[high wigeon]

This seems weird too. Is _ somehow special as a variable name (except in pattern matching)? I always thought it's just a convention.

import asyncio
from typing import Any

async def foo(task1: asyncio.Task[None], task2: asyncio.Task[Any | int]) -> None:
    done, _ = await asyncio.wait([task1, task2], return_when=asyncio.FIRST_COMPLETED)  # leads to error below
    # done, _pending = await asyncio.wait([task1, task2], return_when=asyncio.FIRST_COMPLETED)  # does not lead to error below

    if task1 in done:  # mypy: Cannot determine type of "done" [has-type]
        pass

So whether there's an error depends on if I name the variable _ or something else.

[proud brook]

It has nothing special about it other than being the last evaluated object on REPL

PyCharm uses it to name unused variables

[high wigeon]

Hm, I guess it must be special in mypy too, because otherwise things would break a lot when a type gets assigned to _ and then later another type.

[proud brook]

I don't know whether PyCharm or mypy know to not care about type checking _

It's conventionally used for throwaway values

So it can be reassigned to anything

[high wigeon]

They do, I'm sure I would have run into it a lot if it didn't. (Maybe they just treat it as an Any? Or a write-only Any?)

[proud brook]

PyCharm doesn't force type checking on names which weren't type hinted

Reassignments to different types of objects may give names Any

[high wigeon]

I think mypy does, at least for typed functions, even without --strict (it takes the type from the first assignment, which I personally find useful behavior)

[proud brook]

You might want to use a different value for return_when as the default is ALL_COMPLETED

[high wigeon]

I know, but it didn't affect type checking :)

Maybe makes sense to still have it in the example.

[proud brook]

I believe this error doesn't occur on PyCharm

As long as asyncio.wait is type hinted or the type of the return value is inferred by PyCharm

[high wigeon]

This is a heavily minimized example. Otherwise I don't think Task[Any | int] would make a whole lot sense as a type.

[proud brook]

I didn't know there's a convention for generically type hinting tasks

[high wigeon]

Yes, it's the type of the result.

[proud brook]

Makes sense

[high wigeon]

It does, really :)

[proud brook]

What happens when you use asyncio.as_completed instead of asyncio.wait(return_when=FIRST_COMPLETED)

Or are you only waiting once

Though I guess you could just not iterate over the lazy iterator more than once

[high wigeon]

In my original function I wait for an asyncio.Queue.get() and for the task that enqueues stuff. The only reason to wait for the task is error handling; I want to know if that task terminates instead of being stuck waiting for the queue. So the normal case is always only one of them becoming done.

[proud brook]

Don't you have to wait for the get task to know the get task is terminating 🤔

The put task should only wait when the queue is full

And in any case I feel like they would both terminate unless a middleman had gotten the enqueued resource before the get task could

[high wigeon]

So I have one task that is kind of semi-eternal, alive for the whole duration of a stream (which may or may not be infinite). It takes its data from some queues and outputs it on another. That task should generally not exit, especially not in the context of this function, but because of programming errors it might happen, and I'd rather propagate those errors.

And the other task is just a wrapper for queue.get(). That task routinely completes.

So I can wait for the first of {task, getter} to complete. Then either

1. The task has died, which means it raised an exception, and we want to reraise it by calling task.result()
2. The task has enqueued something, in which case the getter returns that item.

All of this is kind of unrelated to typing, though, even if it's the original context in which I encountered this mypy behavior :)

[grave fjord]

You want a race function?

[high wigeon]

What do you mean?

[shadow olive]

A day ago I was being told about TypeVar and Paramspec which I don’t really get still

Was suggested to replace my Any with a TypeVar

[grave fjord]

#async-and-concurrency message

[grave fjord]

Show your code?

[grave fjord]

You probably want to ask in #async-and-concurrency though

[shadow olive]

Here. I didn't write it but have been using it to run blocking code in a non-blocking way:

async def run_blocking(blocking_func: typing.Callable, *args, **kwargs) -> typing.Any:
    """Runs a blocking function in a non-blocking way"""
    func = functools.partial(blocking_func, *args, **kwargs) # `run_in_executor` doesn't support kwargs, `functools.partial` does
    return await client.loop.run_in_executor(None, func)```

to me the typing.Any makes sense because an arbitrary function could return any type

[grave fjord]

There's asyncio.to_thread

[shadow olive]

How do you pass it args of a function and get the result?

[shadow olive]

had no idea about this

[grave fjord]

It takes *args **kwargs

[high wigeon]

Still not sure if that's what I want. :) My code works as I need it, to the best of my knowledge. This is the full thing: https://gist.github.com/sliedes/973e1e31169f34bf797c6e57100a67d4

[shadow olive]

What does the second argument, "/" mean? https://github.com/python/cpython/blob/main/Lib/asyncio/threads.py

[grave fjord]

It makes the parameters before it positional only

https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters

[shadow olive]

ty

for discord bots when I look up how to run blocking code, most examples I get have loop.run_in_exeutor() so I wonder why ppl aren't using asyncio.to_thread()

but I'll give it a shot later

[grave fjord]

Probably too old

Should be get_running_loop() rather than client.loop that's a python 3.5ism

[shadow olive]

interesting

[trim tangle]

idk why to_thread exists tbh, it's a really thin wrapper around run_in_executor

[grave fjord]

I think asyncio.to_thread is 3.9 but you can copy paste it and merge the type hint from typeshed

[trim tangle]

Also it doesn't let you specify the executor. Which is probably wrong most of the time lol

[acoustic thicket]

to answer the original question, this is how youre supposed to annotate functions with callbacks

P = typing.ParamSpec("P")
R = typing.TypeVar("R")
async def run_blocking(blocking_func: typing.Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R:
  ...

[grave fjord]

anyio.to_thread.run_sync ftw

[shadow olive]

In the examples I'm seeing loop = asyncio.get_running_loop() is used right before loop.run_in_executor()

[grave fjord]

Oh well that's ok

[trim tangle]

Yeah, alternatively consider this py T = TypeVar("T") async def run_blocking(blocking_func: Callable[[], T]) -> T: ... and just make the caller use a lambda or partial

[grave fjord]

It's using client.loop.run_in_executor() that's old fashioned

[shadow olive]

there's a new python syntax that lets you define P and R in the same line as the async right?

anyways I don't understand TypeVars or Paramspecs yet

[grave fjord]

Yeah in 3.12 I don't think mypy supports it yet

[shadow olive]

word

[acoustic thicket]

3.12 lets you define the TypeVar in the same line

not sure about the paramspec

[brisk hedge]

Paramspec defined similarly using **P, and typevartuple using *Ts

[rare scarab]

Because discord formatting: **P *Ts

[grave fjord]

Can't say I like **P

[gray stirrup]

anyone using a dependency injection framework that uses type hints to automatically resolve dependencies ?

[rare scarab]

fastapi

tiangolo needs to externalize his dependency resolver.

[grave fjord]

expresso

Uses the di package called di

[gray stirrup]

ah, I've seen that di package, I think it's being built by one of the pydantic maintainers

[gray stirrup]

last I had checked FastAPI's dependency injection doesn't quite do what I'm looking for, but I'll have a look thanks

[eager vessel]

True

I think you can simply use any other DI library, would probably be somewhat faster to run too

and use fastapi DI only for framework-specific operations, like getting a user using a auth header

[rare scarab]

yeah, fastapi's DI is heavily reliant on the Request object

[eager vessel]

I think I tried dependency-injector some time ago, as it's the most popular DI library, but I'm not sure how it works with async and generators

A friend told me it doesn't work that well with async

[gray stirrup]

yeah, that's the plan Doctor

just trying to find a framework that will probably get buy-in from coworkers. Was planning on using dependency-injector

[eager vessel]

I built my own library for DI, pretty much satisfies my needs without overcomplating stuff (like dependency-injector does)

[gray stirrup]

I did build one as well... but that will be a tough sell on my job 😄

[eager vessel]

@contextlib.asynccontextmanager
async def get_session() -> AsyncIterator[AsyncSession]:
    async with async_session_factory() as session:
        yield session

container = Container()
container.register(Factory(get_session))

async with container.context() as ctx:
    await ctx.resolve(AsyncSession)

I think it should be running in prod somewhere, though, mostly internal services

https://gitlab.com/ThirVondukr/fastapi-cookiecutter/-/blob/61c513b8e9fdc205dea0af12e2607f7138955d10/src/app/adapters/api/books/_router.py#L21-32

[rough sluice]

src/app/adapters/api/books/_router.py lines 21 to 32

@router.post(
    "",
    responses={
        status.HTTP_201_CREATED: {"model": BookSchema},
    },
    status_code=status.HTTP_201_CREATED,
)
@inject
async def books_create(
    schema: BookCreateSchema,
    command: Annotated[BookCreateCommand, Inject],
) -> BookSchema:```

[eager vessel]

What did you come up with?

[gray stirrup]

Mine was very inspired by .NET implementation, basically you'd register all dependencies + lifecycles, when requesting something it would use constructor type hints to resolve the entire dependency tree:

class Config:
    
    @staticmethod
    def from_env():
        return Config()


class Database:
    def __init__(self, config: Config):
        self.config = config

class Service:
    def __init__(self, database: Database):
        self.database = database

container = (
    ContainerBuilder()
    .add_singleton(Config, provider=Config.from_env)
    .add_scoped(Database)
    .add_transient(Service)
    .build()
)

container[Service]

though it's pretty fragile, unions/typevars and etc would probably break the code 🙂

[eager vessel]

So like MS DI?

Looks neat

Would it work with contextmanagers or async?

[gray stirrup]

hm, so the way I've implemented right now it just expects a function that it can call, it doesn't do anything fancy to check if the func is async or a contextmanager 😦

[eager vessel]

I think with typevars you'd have to override __class_getitem__ and return a new type (if you're storing all dependencies in something like a dict) and do this:

add_scoped(Repository[A], provider=RepositoryA)
add_scoped(Repository[B], provider=RepositoryB)

I think there might be a different solution, but I didn't look into it that much

I'm surprised that fastapi's di is considerably slower too, though it's only 0.5ms and is nothing to worry about:

Name                      iterations sum        mean       median     p95        p99 
FastAPI - /depends        10000      7336.670ms 0.734ms    0.709ms    0.863ms    0.956ms         
FastAPI - /aioinject      10000      2656.135ms 0.266ms    0.252ms    0.338ms    0.411ms  
FastAPI - /by-hand        10000      2477.166ms 0.248ms    0.234ms    0.327ms    0.350ms

[gray stirrup]

oh, your aioinject lib seems to be very efficient, almost no overhead 😮

[eager vessel]

The dependency tree looks kind of like this btw:

async with create_session() as session:
    use_case = UseCase(
        service_a=ServiceA(repository=RepositoryA(session=session)),
        service_b=ServiceB(repository=RepositoryB(session=session)),
    )
    return await use_case.execute()

At least that's what the /by-hand endpoint does

[eager vessel]

It's nothing special, though

[gray stirrup]

When I was implementing my container I got sidetracked by performance :). I ended up doing a whole thing with source generators to minimize overhead as much as possible.

$ python3.11 benchmark/run_benchmark.py 
Direct 563 ns
MetaDI Container 792 ns
Dependency Injector Container (Cython) 1183 ns
Rodi Container 1983 ns

In case you're interested: https://github.com/GabrielCappelli/meta-di

[eager vessel]

I think dependency-injector is faster than my library

iirc

[gray stirrup]

yeah those timings are in ns, so I'd say the difference is too little to make any real impact on apps perf 🤔

[eager vessel]

At least it's not fastapi di

@gray stirrup As I said it's really stupid 😅
https://gitlab.com/ThirVondukr/aioinject/-/blob/main/aioinject/context.py?ref_type=heads#L67

[gray stirrup]

ahh... that's pretty nice, I was not aware of contextlib.enter_async_context

[eager vessel]

In order for "custom" parameters to work in frameworks that use annotations like fastapi or strawberry-graphql you have so scrub annotations from the function 😅

[gray stirrup]

ah yeah, there's always some nasty things on these implementation details 😛

[rare scarab]

Consider using Annotated for non-types in your annotations

Fastapi has support for it

[eager vessel]

Well, it won't resolve dependencies automatically
Though you can store it in a variable and use it as a type

I think main downsides of fastapi's DI is that it would require you to annotate everything with Depends() so it permeates into your application core/main logic, which doesn't have anything to do with fastapi

And obviously you can only use it inside of fastapi endpoints

[rare scarab]

I mean, in fastapi you can do this. py def foo(bar: Annotated[MyObj, Depends(get_my_obj)]): ...

[eager vessel]

You can, also it doesn't guarantee you'll get MyObj

[rare scarab]

Yes, sadly, you can't typecheck that

[eager vessel]

If it worked with types it would be somewhat type-checkable in most cases

[rare scarab]

But you can extract the Annotated so you don't make mistakes each time you use it.

[eager vessel]

Yeah, I mentioned that

[rare scarab]

yup.

[eager vessel]

Main issue is really that you have to import it in your app core

And annotate everything

[gray stirrup]

Yeah I see this issue on many DI libs as well, some ask you to decorate all of your classes/methods with their own decorator, I'd like to keep my domain code free from any framework specific code

[eager vessel]

Don't use decorator in your domain code

E.g. in my library it should only really be used on entrypoints to your application

Like CLI, API, etc

[rare scarab]

then make a bridge function

[gray stirrup]

haha, yeah I know :). Btw, your cookie cutter looks pretty organized. Were you able to use a DDD-like approach at work as well ?

[eager vessel]

Can't do that if you have to decorate all classes 😅

[rare scarab]

proxy class

do you even java?

[eager vessel]

Do you even see sharp?

[rare scarab]

No, that's why I wear glasses

[eager vessel]

I think there's like 2 main approaches to DDD?
One that favors storing logic in models and one that favors only storing data in them?

I was using the second one, storing relevant logic in services/commands/queries

[gray stirrup]

yeah, tbh, I've never worked on a project where we kept it at models... neither did anyone I know, but most DDD/OO purists would bash you for that 😛

I've a hard time getting people to do basic separation of concerns as it is (e.g, no business logic in controllers)

[eager vessel]

I do write some small logic in models sometimes, but mostly it's some access methods/properties, in case I need anything complex I would probably move that into a function or service method

[tranquil turtle]

is there a way to annotate function that takes sequence of types and returns tuple of instances of these types?
i can do that as huge overload for 1-arg, 2-arg, ... cases, but is there a more elegant solution?

def f(*clss: *type[T]) -> tuple[*T]: ... # no idea :)

f(X, Y) -> tuple[X, Y]
f(X, Y, Z) -> tuple[X, Y, Z]

[oblique urchin]

No, this would require something like the Map operator on TypeVarTuple

hypothetically, def f[*Ts](*clss: *Map[type, Ts]) -> tuple[*Ts]: ...

[tranquil turtle]

that is sad
are there any plans for type-level Map?

[oblique urchin]

there is no PEP, and I don't know of anyone who is currently writing such a PEP

it was said to be a potential followup for PEP 646, but I haven't heard any progress around this recently from the PEP 646 authors

[tranquil turtle]

if there is no pep and nobody is writing it, i should do it myself 🤔

[oblique urchin]

true!

[trim tangle]

that would be useful for something like asyncio.gather

type AnyCoro[T] = Coroutine[T, Any, Any]

async def gather(*coros: *Map[AnyCoro, T]) -> tuple[*Ts]:
     ...

[eager vessel]

Currently only via overload , I think even TS does that, sadly

[oblique urchin]

though that's still wrong because it actually returns a list 😛

[eager vessel]

And TS probably has one of the most complex typing systems?

[trim tangle]

yeah i know...

we should make a gather2 which returns a tuple

[oblique urchin]

TS has some features that Python doesn't have, but Python also has some that TS lacks

[eager vessel]

Where's gather1?

[trim tangle]

you want Python to be consistent?

[oblique urchin]

Overall Python's gradual type system may well be more complex. TS is essentially all TypedDicts

[trim tangle]

Conditionals and mapped types are pretty complex indeed

[fierce ridge]

tuple(await a for a in asyncio.as_completed(args))

would something like that work?

[trim tangle]

i uhh

[fierce ridge]

actually wait, asyncio.wait already returns a tuple

[trim tangle]

return tuple(await asyncio.gather(args))

[fierce ridge]

def gather_tuple(*args):
    return asyncio.wait(args)[0]

doesn't gather use wait internally anyway?

[oblique urchin]

wait only takes a single future, not a tuple, right?

[fierce ridge]

you can pass a sequence of futures/tasks/awaitables

i use that all the time for dealing with queues and events

[oblique urchin]

oh so it does, I was confused with some other function

[fierce ridge]

while True:
    queue_get_task = asyncio.create_task(proucer_queue.get())
    shutdown_task = asyncio.create_task(shutdown_event.wait())
    done, _ = asyncio.wait((queue_get_task, event_set_task), return_when=asyncio.FIRST_COMPLETED)
    if queue_get_task in done:
        item = await queue_get_task
        __do_something(item)
    if event_set_task in done:
        break

that's what i do for consumers in a consumer-producer type of setup

[grave fjord]

We should probably add a race function to cpython

[oblique urchin]

I guess I was thinking of asyncio.wait_for. Those two functions aren't the greatest in terms of naming

[fierce ridge]

i'd love a helper function for this. trio simply works around this in the particular case of queues/channels by having better mechanisms to communicate with both ends of the channel and shut things down cleanly. but there's still no helper for the very general use case of "wait for a collection of n things and take action when k <= n of them are finished"

wait is of course usable as the underlying mechanism, but it's a little ugly as above. and gather is too high level i think.

that, or i'm doing this wrong

[grave fjord]

You can use messenger exceptions with TaskGroup instead

[fierce ridge]

what's a messenger exception? something like StopIteration?

what's the idea with that? you raise SystemShutdown, and the task group will just close everything all at once when handling it?

[grave fjord]

You make a class Result(Exception) in your function

Then try except* it

[fierce ridge]

how does that work with the taskgroup? i haven't really used them at all (and i'm not even sure how i'd do that with a trio nursery / anyio taskgroup either)

[grave fjord]

And because it's unique to your call of the function it's safe

[grave fjord]

#async-and-concurrency message

[fierce ridge]

oh i forgot i was in #type-hinting , thanks

ahh i see

yeah perfect, that makes sense

so the exception is caught outside of the taskgroup, meaning that the task group shuts itself down (and therefore also all its child tasks) as the stack is unwound? or something like that

[grave fjord]

It's using anyio so you could just assign a nonlocal and call .cancel

[grave fjord]

The stack doesn't get unwound here because we're waiting in __aexit__

Well I guess it gets unwound by one frame

But the cancellation happens before the unwind not as

[fierce ridge]

ah right

the one thing that doesn't really solve is deadlocking between queue.get and shutting down, but if you're using the exception you don't need the shutdown event

however you might still need race for other things

great pattern though, i'll definitely try using it

[undone saffron]

Checking here before I figure out how I want to word reporting this to each.

from __future__ import annotations

from collections.abc import Iterable
from typing import Protocol, TypeVar

T = TypeVar("T")


class Reduction(Protocol[T]):

    def __call__(self, *__args: T) -> T:
        ...


def element_wise_apply(func: Reduction[T], seq: Iterable[Iterable[T]]) -> Iterable[T]:
    return (*(func(*args) for args in zip(*seq, strict=True)),)


data = (1,2,3), (4,5,6), (-1, 7, 2), (4, 5, 4)


print(element_wise_apply(min, data))  # prints the expected output of (-1, 2, 2)
# Both mypy and pyright think min is incompatible with Reduction, despite being trvially shown as
# callable compatibly with the protocol

Not sure if this is both mishandling that protocols only need to check that something is usable in the way shown or something else. pyright also turns that zip into tuple[Any, ...] which is an annoying thing that it doesn't understand zip even for fully homogenous data, but Any is compatible with everything, so it doesn't error for that just for min at the end.

python3.11 locally, was confirmed in the respective playground for each on latest

[oblique urchin]

Arguably min() isn't compatible with that protocol, as the typeshed stubs require at least two positional args: https://github.com/python/typeshed/blob/3c872ca8fd875f2dc5fe5f5d771e35c58390cd0e/stdlib/builtins.pyi#L1531

[rough sluice]

stdlib/builtins.pyi line 1531

def min(```

[undone saffron]

Great, more places where the use of the type shed rather than the standard lib itself needing to be typed causes problems when it's just wrong.

hmm

[oblique urchin]

Not sure why you're blaming typeshed here. min() is implemented in C, you need some way to tell type checkers what its type is

[undone saffron]

class Reduction(Protocol[T]):

    def __call__(self, args: Iterable[T]) -> T:
        ...

(And updating use) is also seen as incompatible.

and there's an overload that should match

Because if the .pyi files had to live in cpython and cpython had to pass type checking, things like this wouldnt happen. I'm not saying it wouldn't still need to be in a stub

[oblique urchin]

I don't see why. Is your argument that type checking the CPython test suite against the stubs would catch all possible errors?

[undone saffron]

All possible? No, not with the state of python typing being way too loose, but if you don't see how python's standard lib being dogfooded would ensure more things had more correct typing, I don't think we will ever see eye to eye on ideals

[oblique urchin]

Possibly the stubs for min() could be written in a way that fixes your error, but they might well break other use cases. The trouble is that min() supports both calls with *args and with a single iterable, and we need both cases to be handled by type checkers

[undone saffron]

.

even updating it to that case doesn't let either mypy or pyright work

[oblique urchin]

It would surely help somewhat but I doubt it's significant for widely used parts of the stdlib. Typeshed's stubs are tested (explicitly or implicitly) against a ton of typed code, I don't think the stdlib itself would make that much of a difference for widely used functions like min(). It would help more in the obscure corners of the stdlib, things like xml.dom

[oblique urchin]

I would investigate further by copying the typeshed stubs for min() into your code to see if you can figure out where the type checkers are going wrong. If you can find a solution that works for your code, we can consider making that change in typeshed (if it doesn't regress other use cases).

[undone saffron]

Pyre handles it correctly https://pyre-check.org/play?input=from __future__ import annotations from collections.abc import Iterable from typing import Protocol%2C TypeVar T %3D TypeVar("T") class Reduction(Protocol[T])%3A def __call__(self%2C *__args%3A T) -> T%3A ... def element_wise_apply(func%3A Reduction[T]%2C seq%3A Iterable[Iterable[T]]) -> Iterable[T]%3A return (*(func(*args) for args in zip(*seq%2C strict%3DTrue))%2C) data %3D (1%2C2%2C3)%2C (4%2C5%2C6)%2C (-1%2C 7%2C 2)%2C (4%2C 5%2C 4) reveal_type(element_wise_apply(min%2C data))

[oblique urchin]

Looking at the error messages, it's possible pyright is overzealously inferring literal types.

[undone saffron]

mypy isn't much better here

which now means I need to now dig into the internals of each to determine what's going on 🙃

[oblique urchin]

yes, no argument that this is a bad case and potentially a bug in mypy and pyright

mypy also isn't helping here by displaying the type as merely "overloaded function"

[undone saffron]

And as for why on blaming the typeshed: The two type checkers that actually have CI in the typeshed are tripping up on basic function composition with builtins. So even if it isn't a problem with the stubs, it's frustrating how little seems to actually be supported for anything more than the most basic, even in situations where it's intended to work, and there's supposedly tests

[soft matrix]

i thought this was all waiting till type Map comes along

which was meant to be at some point this year

[undone saffron]

this doesn't need map to work.

[soft matrix]

oh the issue is min not zip

[undone saffron]

Followup on the above,

mypy and pyright are both deciding that because they don't know the length of the iterable, it must be an error. pyre isn't. considering that the length is unknowable from the type checker's perspective, this seems incorrect, but it's been closed in pyright as not planned. (Better knowledge of zip could enable this to work, as would not assuming this to be an error since it isn't a type error)

Neither provides a useful error message, which paired with my frustration with something that by all accounts hsould work, contributed to me missing that both allow the modified single argument form if the protocol is made positional only. I'd have been more likely to notice that if the typeshed was using positional only instead of double underscores for names as I practically never think about type checkers special casing that as private by convention is not the same as private, but the error message was also

Argument 1 to "element_wise_apply" has incompatible type overloaded function; expected "Reduction[int]" [arg-type]

which didn't even show an incompatible type, just referring to it as "overloaded function"

[mellow talon]

Hello All,
Happy past halloween.

If you haven't got scared from the treat or trick yesterday night, I hope the following will not scare too.
I'm trying to clean a couple of classes I wrote from the type hinting error reported by PyLance.

This class declare some parameters, one of them being an object (another class of mine), that during the initialisation create another parameter of another class of mine.

As you can see, Pylance says it cannot access to the access member.

Is there any Pythonic way to solve this without recurring to the # type: ignore?

Below a reconstructed piece of code (the first class is in another file):

class WebProperty:
      def __init__(self, raw, account: Account):
        self.account = account
        self.raw = raw
        self.url = raw["siteUrl"]
        self.permission = raw["permissionLevel"]
        self.query = Query(self)

  ...

class something_else:
  ...
  def __init__(self, webproperty: Type["account.WebProperty"]):
    self.webproperty = webproperty
    ...

  def execute(self):
        url = self.webproperty.url
        try:
            self._wait()
            response = (
                self.webproperty.account.service.searchanalytics()

        ....

[hallow flint]

you don’t need the Type[], just use account.WebProperty

[mellow talon]

Thanks.
So when I read around re the usage of Type ... what went wrong?
When is that has to be used?

[hallow flint]

if you are passing the class itself, instead of an instance of the class

[lost sierra]

Sir, are you CPython dev?

[scarlet nacelle]

Ye

[lost sierra]

Oh my God, this cant be real. He's one of the creators of what I spend all day doing.
Forgive me if i am disturbing you.
Yours sincerely, sir.

[silver dagger]

What is meaning of braces in coding eg((),[],{})

[viscid spire]

this channel is for typehinting specifically

[lyric palm]

why does the vs code type checker allow testA but not testB?

def testA[F: Callable[..., Awaitable[None]]](f: F):
  pass
async def func(*args, **kwargs):
  pass
testA(func)

def testB[F: Callable[..., Awaitable[None]]]() -> F:
  async def func(*args, **kwargs):
    pass

  # Expression of type "(*args: Unknown, **kwargs: Unknown) -> Coroutine[Any, Any, None]" cannot be assigned to return type "F@testB"
  return func 

is it simply because func in testB is a closure?

[trim tangle]

In the second example you're promising to return a value that's bound to some arbitrary type variable

Like if you set F to some CustomCallable (for example via a variable annotation) the function should return CustomCallable, which is not possible

Generally, using a type variable only once in a function signature is wrong

[lyric palm]

in my actual code, the function header is closer to:

def decorator[F: Callable[..., Awaitable[None]]](func: F) -> F

which has the same issue as testB

[trim tangle]

Well, suppose that func is an instance of some CustomCallable class. Your function will return a function instead

But the signature promises thay you will return a value of the same type

[lyric palm]

what do you mean by F is an instance of another class? is F not a typevar?

[trim tangle]

Sorry, I meant func

You probably want a ParamSpec here

[lyric palm]

hm

[lyric palm]

if func was CustomCallable, wouldnt the typechecker be upset at the callsite? provided it's not a subclass of Callable?

[trim tangle]

For example, int is definitely a callable. The signature of decorator lets you do this:

the_type: type[int] = random.choice([int, bool])
decorated: type[int] = decorator(the_type)

[trim tangle]

Well, a CustomCallable is probably a subtype of Callable

i.e. it defines a __call__

[lyric palm]

i think i see the issue there yea

[trim tangle]

ParamSpec is probably the way to go here. It lets you transfer the parameters from one callable type to another

[lyric palm]

im not really sure what this snippet is doing, wouldnt the typechecker disallow the first line? since there's no guarantee it would be type[int]?

[lyric palm]

ill check that out, thanks

[trim tangle]

bool is a subclass of int

because of... historical reasons

[lyric palm]

oh ok that makes sense haha

[trim tangle]

(before py2.1 or something there was no bool, 0 and 1 were used; so to keep backwards compatibility as much as possible, bool was made to mimick int, including mathematical operations)

[soft matrix]

i mean id just use cast(F, func) personally

because Callable is erasing too much stuff imo

[lyric palm]

def decorator[**P](func: Callable[P, Any]) -> Callable[P, Any]:
  @functools.wraps(func)
  async def wrapped(*args: P.args, **kwargs: P.kwargs) -> None:
    # ...
    return await func(*args, **kwargs)

  return wrapped

with some other help from #1035199133436354600, this snippet works perfectly! thank you so much!

[trim tangle]

You probably want a typevar for the return type though

[high wigeon]

Can someone help me understand the type of cls.__subclasses__()?

What does def [T] () -> tests.bug.Some[T`9663] mean—specifically the [T]? Is that something that is expressible in Python syntax? That looks like a Callable to me, but not sure.

from typing import Any, Generic, Protocol, TypeVar, reveal_type

T = TypeVar("T")


class SomeProto(Protocol[T]):
    f: T


class Some(SomeProto[T], Generic[T]):
    pass


reveal_type(Some.__subclasses__())  # Revealed type is "builtins.list[def [T] () -> tests.bug.Some[T`1]]"

reveal_type(Some.__subclasses__()[0])  # Revealed type is "def [T] () -> tests.bug.Some[T`9663]"

for a in Some.__subclasses__():
    reveal_type(a)  # Revealed type is "def [T] () -> tests.bug.Some[T`9693]"


x: type[Some[Any]]
for x in Some.__subclasses__():  # Error: Can only assign concrete classes to a variable of type "type[Some[Any]]"
    reveal_type(x)

And why does that error arise in the for x in Some.__subclasses__() line only if Some derives from SomeProto?

And what does type[C] actually mean for a class C? I think it includes all of its subclasses? (I'm confused about how, if at all, __init__ gets handled with this, because a subclass could obviously require very different parameters.)

[grave fjord]

You might want Callable[[YourArgs, Here], C] instead of type[C]

[high wigeon]

Yes, that would make sense if I wanted to instantiate them, but for now I'm just trying to understand the type of __subclasses__()

But that admittedly is a way to handle __init__() :)

[high wigeon]

Is there a type for "any subclass of C, even if abstract"? type[C] seems to only mean concrete classes...

[grave fjord]

You mean Protocol?

[high wigeon]

Not only protocol. So, suppose I'd want to write a function find_all_subclasses(cls: type[T]) -> list[type[T]]. This cannot, according to mypy, be called for values of cls that are abstract classes.

[oblique urchin]

mypy's decision to disallow abstract classes there is questionable

there is a mypy issue with a lot of discussion

a reasonable option is to turn off that mypy error code

[thick pelican]

Agree

[acoustic thicket]

huh, why does mypy not allow it

[high wigeon]

Hey, does anyone know if there's any document or discussion that collects the ways Python's type system is unsound (i.e. where a fully typed, Any-less, castless program that correctly passes --strict type checking can still lead to a type error at runtime)?

One clear case is that (apparently) for a class C, cls: type[C] implies that you can instantiate the class using the same constructor arguments as you could with class C, which is clearly incorrect because a subclass can have wildly different constructor arguments.

[high wigeon]

So the reason is apparently that "usually" when you pass around a type[T], the reason is that you want to instantiate it. I'd admit that this is a common case. If you allowed passing abstract types, then you'd not catch this at static typing time.

Yet this is silly for the above reason, in that you cannot anyway know how a subclass of T can be instantiated. (You can, instead, enforce it to have a factory class method, though.)

[undone saffron]

One possible solution to this one has been discussed before and will probably come up with the typing council eventually.

That being, any dunder that exists with the intent of being changed (so, not __name__ being something other than a str) on type or object should not be considered an LSP violation to modify in subclasses. With that exception explicitly stated, type checkers not currently checking __init__ (etc) for compatability could.

This being because the fact that object and type implement these is an implementation detail not an intent of it being the only valid constructor

There's also the fact that just because a type checker allows something doesn't mean it's consistent with what has been specified in the type system (and the same for the inverse of that) mypy and pyright both make opinionated decisions to error on some things which are not specified, and allow some things they arguably should not.

[high wigeon]

Yes, this is obvious (and reasonable). You have to be pragmatic; if a type system is not good enough to allow you to type correctly common things, it's probably better to be lax in checking.

But I would be interested in a compilation of known cases of unsoundness in the type system, just for understanding :)

[undone saffron]

The point I was somewhat getting at is that I don't think that type[C] is unsound :)

[high wigeon]

Allowing calling type[T] with T:s constructor parameters feels like one of those cases which, while unsound, the type system doesn't really currently have much better ways to express a common pattern, so I think allowing calling it is ok.

[undone saffron]

nothing about the specification of type would have it be, that's a consequence of type checker's current behavior not covering a definition we could fine a level of soundness in.

[high wigeon]

Ah, I haven't really looked at how it has been specified (given it's Python, probably not that precisely). Makes sense. :)

[undone saffron]

FWIW, I don't think you'll find a "complete" list of unsound things. the type system is both very underspecified and also turing complete(proof of that: https://arxiv.org/abs/2208.14755) (even if no type checker can do anything with that)

[high wigeon]

Interesting, I didn't know it was strong enough for that, thinking that only HKT would make it. But yes, if type correctness is undecidable, that obviously opens a can of worms.

Huh, you can do this?

from typing import TypeVar, Generic, Any
Z = TypeVar("Z", contravariant = True)
class N(Generic[Z]): ...
X = TypeVar ("X")
class C(Generic[X], N[N["C[C[X]]"]]): ...
_: N[C[Any]] = C[Any]() # infinite subtyping

[undone saffron]

I don't think expansive inheritance (what pushes it into turing complete and therefore undecidability) is necessarily a good thing for python to have, and arguably isn't intended just not forbidden, but that's the current state.

I also don't think HKTs would necessarily improve people's ability to type even more complex ideas, at least not compared to other options,

Theoretically, it could be possible to gain decidability back as a quality by disallowing expansive inheritence.

[high wigeon]

I do run quite often to the lack of HKTs in my practical programming with Python, but I'm probably an extreme case.

[undone saffron]

I find that most of the time when a problem looks like a good case for an HKT (not all the time, but...) there are other means with functional composition that just work better anyhow.

[high wigeon]

That may be true.

[grave fjord]

I think I've found a mypy bug: https://mypy-play.net/?mypy=latest&python=3.11&gist=46efbd045419321e55b61d58cea56cfa this passes on 1.5.1 but fails on 1.6.1

I've minimized the issue a bit https://mypy-play.net/?mypy=1.6.1&python=3.11&gist=e282d121f479fcb77ec89daeb2a07b60

pyright is happy https://pyright-play.net/?code=GYJw9gtgBALgngBwJYDsDmUkQWEMoBUAUEQCoD6ASgKYwBuAhgDZQC8Upi1AagyABQAiCjXrNBASiIAFNlGl8GEAMoJqAYyHTJJIgBNqwKCACuKfkShWowM%2BoBcUAMLMmDAEZNqAbW8BdABoOKlpGJkDLawI%2BNABnRzB3ACsNGADIq3cGdQBrahQ9R1iYEDlBBli4FHUkMEF0iSgAWgA%2BYNEw%2BwzjWhMQFBs7fikSAyNTFHIcPGZhrusoBArY3TGoWJN3CCQYCwXbascXJjdPH2kgkVDmCIWAenSF6JA4x2kAOhjYx6iCHIB3L5vd4Ar4NZptK5iJjzawgXr9QbVfjPOJBAh-QEvWIjIgbLY7PZwsw-KwTKa4GDMUlQLK5fJ6VjlSrVWr1IgSIA

I opened an issue https://github.com/python/mypy/issues/16405

[eager vessel]

@grave fjord This doesn't seem to work either?

from typing import *

T_Retval = TypeVar("T_Retval")
P = ParamSpec("P")


def run(
    val: T_Retval
) -> T_Retval:
    return val

def submit(
    func: Callable[P, T_Retval],
    /,
    *args: P.args,
    **kwargs: P.kwargs,
) -> T_Retval:
    return func(*args, **kwargs)


submit(
    run,
    val="Value",
)

main.py:23: error: Argument 1 to "submit" has incompatible type "Callable[[T_Retval], T_Retval]"; expected "Callable[[str], T_Retval]" [arg-type]

Issue seems to be related to generic in the run function

Without it it type checks fine

I had something similar in my code, but didn't encounter this bug:

def partial_full(
    func: Callable[P, T],
    *args: P.args,
    **kwargs: P.kwargs,
) -> Callable[..., T]:
    return functools.partial(func, *args, **kwargs)  # noqa: TID251

[hallow flint]

the main place this comes up is that str is a Sequence[str]

[undone saffron]

which is a footgun anyhow currently. Disallowing str to qualify as a Sequence[str] is probably a good thing, but not cleanly expressible without either type negation or explicitly stating that T should never implicitly result in Container[T] pytype makes this decision currently and when you want to allow str as Sequence[str] you just do str | Sequence[str]

I can't think of anywhere else it comes up with any validity.

[tranquil turtle]

there are recursive type aliases (for arbitrary JSON objects, fpr example)
does it count?

[high wigeon]

Hey, this seems weird to me, as if ParamSpec here discards the keywords and uses values only. Any hints, am I missing something?

from typing import Callable, ParamSpec, TypeVar

_ParamsT = ParamSpec("_ParamsT")
_T = TypeVar("_T")

def smoke_testable(*args: _ParamsT.args, **kwargs: _ParamsT.kwargs) -> Callable[[Callable[_ParamsT, _T]], type[_T]]:
    def decorator(cls: Callable[_ParamsT, _T]) -> type[_T]:
        assert isinstance(cls, type), type(cls)
        cls._smoke_testable_kwargs = kwargs
        return cls
    return decorator

@smoke_testable(name="bob", size=512, flt=0.5)
class SomeClass:
    def __init__(self, size: int, name: str, flt: float) -> None:
        ...

I get this mypy error:

Argument 1 has incompatible type "type[SomeClass]"; expected "Callable[[str, int, float], SomeClass]" [arg-type]

[bright tangle]

Is there a way to create new no-op type-hints for a tool I'm writing that parses Python?
E.G.

foo: mylib.Flag[int] = 3 # mypy etc. treats foo as `foo: int`

[tranquil turtle]

Flag: typing.TypeAlias = typing.Annotated
foo: Flag[int] = 42

this should work

if not - make Annotated generic

[bright tangle]

Ok, seems like
Flag = Annotated[T, "Flag"] should work fine

[glass stirrup]

How does one effective type hint that an object returned by a method is a subclass of a given class, but without making it then become a linting error when the user goes to try and access their specific instance

I was thinking of a generic which subclasses from the Type rather than returning -> MyBaseType

Reason for this is we dynamically load some things with importlib, so we can't hard code the type, but also I don't want to cause linting errors just because the methods say it returns the base class

[undone saffron]

You can do this by making certain things generic, but without a clearer picture of what you want to happen and how it's interconnected, it's hard to recommend what should be generic for ergonomic use.

An example of this would be an SQL library which wasn't attempting to be an ORM having a row type that's generic over a typevar tuple. The library doesn't know the resulting type, but the use informs the type checker

rows: list[Row[int, str]] = some_library.execute("""SELECT uid, name from ...""")

On the other side, if the user knows what should be available via importlib, and you have classes that do things, making the user instantiated class generic can work

user_instance: LibraryClass[DynamicImport] = LibraryClass()

It'll be up to the user to ensure that the generic is accurate at that point. There's also explicit dependency injection as an option, but this sounds like it's for existing code that you probably don't want to change the behavior of for existing users.

[glass stirrup]

Atm the function signatures are:

def train_model(
    model_path: str,
    *,
    loader_config: ModelLoaderSettings = None,
    no_import_prefix: bool = False,
) -> BaseClassifierModel:

But the actual return type will be the class in the model file, we don't need to worry about it necessarily being confusing to the user because they should already know the type being returned

But atm, the linter complains if you try access anything outside of the base class, which makes sense, but effectively yeah, I was thinking generic might be the way to go

just so the linter is happy and it doesn't lead the user down the wrong path thinking they've done something wrong when using it

[undone saffron]

There's also

T = TypeVar("T", bound=BaseClassifierModel)

def train_model(
    model_type: type[T],
    *,
    loader_config: ModelLoaderSettings = None,
    no_import_prefix: bool = False,
) -> T:

but it's unclear if this is a better option here or if this would be unacceptably breaking. Given the dynamic imports via importlib, I'm imagining there's a reason users aren't passing a type in directly.

[glass stirrup]

yeah Unfortunately this system is effectively for managing a lot of AI models during the development cycle

where a lot of people are using different frameworks and tools

which means the startup time is huge if everything is imported directly

hence the dynamic import aspect of this

and the model loaded by services is configured by a config option

[undone saffron]

There's also typing the return type as Any and explicitly documenting users should annotate their use of it with the expected model, but this requires people understand that this is throwing away base type information and if they don't annotate on the recieving end they won't have any checks at all.

[bright tangle]

I don't see the difference between the caller importing the type they want to use vs your function importing the type they want to use, other than being harder to type-hint

import foo
your_fn(foo)
# vs
your_fn("foo") # imports foo

[undone saffron]

there's a lot you can do with importlib and multiple large imports to delay importing till needed or even (in some cases, if the imports are threadsafe) import in background without delaying startup

[glass stirrup]

The TL;DR is atm, the code base is a a spaghetti mess, which means currently 😅 running anything also imports and loads all the services that will type and import the model based on a config.

The old codebase only ever assumes there is 1 model, which causes some big issues because developing multiple variants across the model to compare against.

The primary idea is that people developing the model only focus on the system within the model, everything else is automatically handled

to also prevent people accidentally overriding stuff in S3 -_- (Yes this is currently an issue in the old code)

This means the prod code and services only load the module they need, while if the user wants to test their model or play with it, they can do so as if it was the prod system loading it

(send help )

[undone saffron]

Almost sounds like you've hit the threshold for a critical rewrite prior to the new models making it worse while at the same time (basing on the s3 comment and the lack of forethought for multiple models) having some less than ideal collaboration requirements.

[glass stirrup]

🙃 It is basically a full rewrite, but my suggestion of creating a new blank repo was shot down. Even though I will be effectively stripping out all the old code anyway.

Baby steps though, once this model loader is setup, it lets the others work on their models, while I can remove the old legacy stuff

[undone saffron]

but this way you do it incrementally and suffer create a history of why things were done this way that is barely beneficial if there exists a snapshot of the prior version. 🙃 .

My condolences. I've been there before with legacy code.

[glass stirrup]

Want to know the sad thing?

This was only written like 2 years ago

[solid light]

Is there a way to type a list as containing keys of a TypedDict? Currently have an issue when looping over the list saying that it expected a valid key

[soft matrix]

nope unfortunately not

[solid light]

😔

[frigid jolt]

list[Literal[...]] doesn't work either?

[rare scarab]

Make attrs_to_keep a tuple and annotate with Final

[urban imp]

is there a way for me to typehint *args, **kwargs in a decorator that's like, "these are the same type as the things the wrapped function would take in"?

e.g.:

def wrapper(func: Callable)  # ... i think i just figured out where i can do it

[acoustic thicket]

!d typing.ParamSpec

[rough sluice]

typing.ParamSpec

class typing.ParamSpec(name, *, bound=None, covariant=False, contravariant=False)```
Parameter specification variable. A specialized version of [type variables](https://docs.python.org/3/library/typing.html#typevar).

In [type parameter lists](https://docs.python.org/3/reference/compound_stmts.html#type-params), parameter specifications can be declared with two asterisks (`**`):

```py
type IntFunc[**P] = Callable[P, int]
```  For compatibility with Python 3.11 and earlier, `ParamSpec` objects can also be created as follows:

```py
P = ParamSpec('P')
```  Parameter specification variables exist primarily for the benefit of static type checkers. They are used to forward the parameter types of one callable to another callable – a pattern commonly found in higher order functions and decorators. They are only valid when used in `Concatenate`, or as the first argument to `Callable`, or as parameters for user-defined Generics. See [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic) for more information on generic types.

[rare scarab]

P = ParamSpec("P")
R = TypeVar("R")

def decorator(func: Callable[P, R]) -> Callable[P, R]:
  @functools.wraps(func)
  def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
    return func(*args, **kwargs)

  return wrapper

[urban imp]

oh i like using R for the return type, i've been using T

[rare scarab]

I also like using K, V for dicts/mappings

[urban imp]

one more, this is making my eyes boggle:

Argument 1 to "matches" of "Matcher" has incompatible type "set[int]"; expected "Sequence[Any]"

... is set[int] not compatible with Sequence[Any]?

[fathom river]

No they are not, Sequence implements index (along with other methods), which set does not

[urban imp]

oh! Thank you

[urban imp]

two last ones i think. I have a function that can either take in keyword arguments, a dictionary, or alternating key/value pairs to make a mapping. That is, all of these produce the same result:

foo(a=1, b=2)
foo({"a": 1, "b": 2})
foo("a", 1, "b", 2)

i've got typing working for the first two, but the last one is giving me this error:

error: Argument 1 to "foo" has incompatible type "str"; expected <nothing>  [arg-type]
error: Argument 2 to "foo" has incompatible type "int"; expected <nothing>  [arg-type]

here's my typedef:

K = TypeVar("K", bound=Hashable)
V = TypeVar("V")

...

    # Keyword argument form
    @overload
    def foo(**kv_args: V) -> None:
        ...

    # Key to value dict form
    @overload
    def foo(kv_args: Mapping[K, V]) -> None:
        ...

    # Alternating key/value form
    @overload
    def foo(*kv_args: K | V) -> None:
        ...

    def foo(*kv_args: Any, **kv_kwargs: Any) -> None:

I'm not sure i understand what the error is telling me, but is there a way for me to say "alternating K then V" in typehint form?

[soft matrix]

look at the signature for dict.update in typeshed it looks similar

apart from the key/value form but you should probably be using something like that anyway\

[urban imp]

you're saying i should get rid of the alternating key/value form?

[soft matrix]

no just do it in a simliar way to dict.update

[urban imp]

i only found these two bits, am i looking in the right spot? https://github.com/python/typeshed/blob/17f8a823765ddaed9740f55ac55ef04bb5a54a5a/stubs/WebOb/webob/multidict.pyi#L48C1-L51C49

    @overload  # type:ignore[override]
    def update(self, __m: Collection[tuple[_KT, _VT]], **kwargs: _VT) -> None: ...
    @overload
    def update(self, **kwargs: _VT) -> None: ...

[soft matrix]

nope, youre not in builtins

[urban imp]

ah, i've never looked at typeshed, i'll look in there

[soft matrix]

https://github.com/python/typeshed/blob/17f8a823765ddaed9740f55ac55ef04bb5a54a5a/stdlib/typing.pyi#L692-L697

[rough sluice]

stdlib/typing.pyi lines 692 to 697

@overload
def update(self, __m: SupportsKeysAndGetItem[_KT, _VT], **kwargs: _VT) -> None: ...
@overload
def update(self, __m: Iterable[tuple[_KT, _VT]], **kwargs: _VT) -> None: ...
@overload
def update(self, **kwargs: _VT) -> None: ...```

[urban imp]

what is that __m bit? It's very hard to search for

(also thank you for all this info and guidance)

[soft matrix]

its a positional only argument

[urban imp]

oh cool, that's nifty

[regal summit]

class _Game(Protocol):
    players: list[_Player]
    @property
    def turn(self) -> _Player:
        ...

class View:
    async def multiplayer_interaction_check(self, interaction: discord.Interaction[Butters], game: _Game) -> bool:
        code



# in some other files

class Player(_Player):
    ...

class Game:
    players: list['Player']
    @property
    def turn(self) -> 'Player':
        ...

class SomeView(View):
    game: Game
    async def interaction_check(self, interaction: discord.Interaction[Butters]) -> bool:
        return await self.multiplayer_interaction_check(interaction, self.game)

Why is this complaining? Player is a subclass of _Player which is hinted in _Game protocol

It does agree with then turn property, which in my game is also a Player and not a _Player

[tranquil ledge]

The annotated type for what a list can contain is invariant in Python, iirc, meaning the exact class is needed, not a subclass.

You could try changing the list[…] in your protocol to Sequence[…]? If memory serves, that doesn’t have the same restriction.

[echo dew]

Is it possible to typehint the conn part of py async with pool.acquire() as conn: in the same line? Right now I do ```py
async with pool.acquire() as conn:
conn: Connection

[tranquil turtle]

just type pool.acquire() properly and type of conn will be calculated

[tranquil ledge]

If the library you’re using didn’t type the acquired connection well, unsightly is what you have to work with, unfortunately.

[urban imp]

OK, i'm stumped. I have this:

class ContainsTheItem:
    ...
    def __init__(self, item: T) -> None:
        self.item = item

    def resolve(self) -> Matcher[Sequence[T]]:
        return has_item(self.item)

I use it here:

class TestContainsTheItem:
    ...
    def test_the_test(self) -> None:
        """Matches lists containing the item"""
        cti_matcher = ContainsTheItem(1).resolve()

and mypy says:

tests/test_resolutions.py:203: error: Need type annotation for "cti_matcher"  [var-annotated]

Why can't it figure out that cti_matcher is gonna be a Matcher[Sequence[T]]? Is it because i'm using T wrong somehow?

[viscid spire]

it's because typeshed is written in an older version of Python, which doesn't have the / syntax for pos-only (forced)

def f(unimportant_name: str, /, *, flag: bool = False) -> None: ...

[urban imp]

i think i'm narrowing it down to maybe i can't do T across multiple places, am i on the right track?

for instance:

T = TypeVar("T")

class Foo:
    def __init__(self, bar: T) -> None:
        self.bar = bar
    
    def baz(self) -> T:
        return self.bar

the T for __init__ isn't remembered as the return value for baz?

[viscid spire]

definitely isn't

you need to make the class generic for that T to be the same across methods

[undone canopy]

Hey there!
I wanted help with one question about the default implementation of a setter in a Protocol:
https://github.com/python/typing/discussions/1498
In particular, I do not see currently any way of creating a default implementation for a property with a setter, as it would use a private variable, which is not allowed.

[high wigeon]

I sometimes use cast(), but the downside of that is that it will hide type errors if the library one day is properly annotated. Hmm, I guess I'd want a narrowing-only cast. Other than that, I didn't realize you can put the conn: Connection after the place where it gets defined; I've always put it on the line above the with.

[high wigeon]

Hey, either of these is a valid way to say that C has a field x:

class C:
    x: int

class C:
    def __init__(self) -> None:
        self.x = 5

Does anyone know if there is a mypy flag to forbid the second? I prefer to declare all my fields explicitly, and detect typos in __init__.

Apparently not. https://github.com/python/mypy/issues/10552

[urban imp]

can you tell me more about this? Do i just do class Foo[T]: ... to make that happen?

[viscid spire]

Yes, then you use it in a typehint like Foo[int]

And also don't create the TypeVar variable

just use the type parameter in that case

[oblique urchin]

note that syntax is only in 3.12

[urban imp]

yeah i need to support down to 3.8 with this, thanks for that note

is this something i can get access to with from __future__ import annotations?

[oblique urchin]

then use T = TypeVar('T') and class Foo(Generic[T]):

[urban imp]

OK, i'll give it a shot

thank you!

[urban imp]

this fixed it, thank you Jelle and @viscid spire!

actually, is this the same problem to my other one, where the alternating key/value option isn't passing type checks?

I have:

K = TypeVar("K", bound=Hashable)
V = TypeVar("V")

class ContainsTheEntry:
    ...

    # Keyword argument form
    @overload
    def __init__(self, **kv_args: V) -> None:
        ...

    # Key to value dict or list of tuples form
    @overload
    def __init__(self, kv_args: Mapping[K, V] | list[tuple[K, V]]) -> None:
        ...

    # Alternating key/value form
    @overload
    def __init__(self, *kv_args: K | V) -> None:
        ...

    def __init__(self, *kv_args: Any, **kv_kwargs: Any) -> None:
        ...

and i use it like:

cte_alternating = ContainsTheEntry("key3", False)

and mypy says:

tests/test_resolutions.py:149: error: Argument 1 to "ContainsTheEntry" has incompatible type "str"; expected <nothing>  [arg-type]
tests/test_resolutions.py:149: error: Argument 2 to "ContainsTheEntry" has incompatible type "bool"; expected <nothing>  [arg-type]

[oblique urchin]

your class isn't generic

[urban imp]

do i need to make ContainsTheEntry inherit from Generic[K,V], or is that only when i need it to remember between methods?

[oblique urchin]

use class ContainsTheEntry(Generic[K, V]):

[urban imp]

well... i get the same errors, but now all the other invocations are also complaining:

cte_single = ContainsTheEntry(key="value")
cte_multiple = ContainsTheEntry(key1="value1", key2="value2")
cte_dict = ContainsTheEntry({"key2": 12345})
cte_alternating = ContainsTheEntry("key3", False)

tests/test_resolutions.py:146: error: Need type annotation for "cte_single"  [var-annotated]
tests/test_resolutions.py:147: error: Need type annotation for "cte_multiple"  [var-annotated]
tests/test_resolutions.py:149: error: Need type annotation for "cte_alternating"  [var-annotated]
tests/test_resolutions.py:149: error: Argument 1 to "ContainsTheEntry" has incompatible type "str"; expected <nothing>  [arg-type]
tests/test_resolutions.py:149: error: Argument 2 to "ContainsTheEntry" has incompatible type "bool"; expected <nothing>  [arg-type]

only cte_dict is OK

in general, if i accept a typevar, do i need to be using a generic class? Even if i don't return that value later?

[bleak imp]

Is this the correct way to type hint this decorator? py def add_target_strings(*strings: str) -> Callable[[Callable[[str], str]], Callable[[str], str]]: def decorator(function: Callable[[str], str]) -> Callable[[str], str]: def wrapper(data: str) -> str: return function(data) wrapper.target_strings = strings return wrapper return decorator
It's meant to be used like this, where all the functions have the same str -> str signature py @add_target_strings("abc", "def") def func1(a: str) -> str: return f"ran func1 with arg {a=}"
and while it does pass mypy (aside from the [attr-defined]) I feel like there should be a better way.

[oblique urchin]

that looks right. maybe make a type alias for Callable[[str], str]

[bleak imp]

Ah, that would help a bunch

[trim tangle]

I wish type inference was standardized a little bit...

Typing out decorators is horrible

[regal summit]

        if self.unsorted_symbols is None:
            return
        self.unsorted_symbols
        self.cards.sort(key=lambda c: self.unsorted_symbols.index(rank_to_symbol[c.rank]))```

is this a bug?

or somehow some weird intended behaviour

        if self.unsorted_symbols is None:
            return
        unsorted_symbols = self.unsorted_symbols
        self.cards.sort(key=lambda c: unsorted_symbols.index(rank_to_symbol[c.rank]))``` it finds this ok, guessing that is a bug?

[soft matrix]

nope probably not

you probably assign the self.unsorted_symbols = None somewhere

[regal summit]

no thats not my point

[soft matrix]

and pyright cant track how things like key functions are used

[regal summit]

2 lines earlier im literally checking if its not None

[soft matrix]

so it does the safe thing and says it could have changed after it is called to None again

[regal summit]

but why would this be allowed since its pointing to the exact same object

[soft matrix]

its not, if you change self.unsorted_symbols after that first assignment the type of unsorted_symbols will not change

[regal summit]

what would be stopping me from doing the same with unsorted_symbols (without self)

[soft matrix]

nothing but it would un-narrow the type

[mighty linden]

**No topics found for this channel.**

Suggest more topics here!

[solid light]

@staticmethod
    def map_col(col: MovieRatingsColumn):
        attrs_to_keep: Final[tuple[str, ...]] = ("movie_title", "user_rating")
        return {
            k: col[k]
            for k in attrs_to_keep
        }```like this? It doesn't seem to work...

[tranquil turtle]

maybe omit args to Final to let typechecker infer it

[rocky agate]

Typescript has such a thing as Omit<T, fields>, which takes a type and a list of field names that need to be removed, and returns another type, like the old one, only without the specified fields. Is there something similar in python typing? Or do you need to build your own? It is necessary to make a decision at the type level in order to write hints and make static checks.

[tranquil turtle]

no, it doesn't exist
what is your usecase?

[rocky agate]

class T(Protocol):
  a: int
  b: int

I wanna get the type with reduced set of fields. Like to narrow down the requirements from the structure that the function should receive.

[tranquil turtle]

make another protocol with only a?

[rocky agate]

If T changes, the changes won't be reflected.

[tranquil turtle]

why would you need to narrow the requirements?

[rocky agate]

Ofc I can create T0 with only a as a parent of T, but that feels not right.

[tranquil turtle]

i think just using protocol T is fine

[rocky agate]

So I have a protocol with 5 fields, but the function uses only 3 of them. I wanna reflect that. That's not a big deal, but after Typescript this feels pretty intuitive to me 🙂

[tranquil turtle]

there is no such thing as you described (yet, it might be done like T & ~T_with_b in the future)
the simplest way is to use T, but if is not ok for you - make another protocol (ir make parent as you described)

[solid light]

But... it is a string 🤔

[trim tangle]

no it's a str

[rare scarab]

how is Review defined?

[solid light]

TypedDict

class Review(TypedDict):
    overall: float  # Whole-number rating for the product 1.0 - 5.0
    verified: bool
    reviewTime: str  # 06 3, 2015 -> mm d(d), yyyy
    reviewerID: str
    asin: str  # Product ID
    reviewerName: str
    reviewText: NotRequired[str]
    summary: NotRequired[str]
    unixReviewTime: int  # 1433289600
    style: NotRequired[RatingStyle]  # Product Metadata
    vote: NotRequired[str]  # String of how many times the review has been marked 'helpful'
    image: NotRequired[list[str]]  # Images that reviewer attaches to their review```

Where RatingStyle is another typed dict

[rare scarab]

Try reloading the python language server

oh, that's pycharm

it works in pyright

if you like typing, you probably don't want to use pycharm

So the issue is that required_fields is inferred to tuple[str, str, str, str, str] in pycharm. pyright, and I assume also mypy, will resolve it to literals.

oh great, pycharm doesn't implement reveal_type

so mypy doesn't resolve to literals, but TypedDict.get also isn't overloaded with each key, so that doesn't break it

[solid light]

😔

What should I use then?

I want to avoid VSC, which is the only other IDE I really know how to use

[rare scarab]

neovim with mypy or pyright LSP?

[tranquil turtle]

i would recommend anything with mypy/pyright plugin
sublime text in my experience is pretty friendly, you install extensions you want and they just work

[rare scarab]

tried jetbrains fleet?

[solid light]

Could I just install the Mypy plugin on pycharm?

[rare scarab]

you could try

But it only runs on one file at a time.

[solid light]

I guess I need to disable whatever pycharm uses by default first

[rare scarab]

even when you tell it to check the whole project.

it's very slow

[solid light]

😔

Guess I'll just live without typing

[tranquil turtle]

living without types has its benefits
you are no longer fighting typechecker, and if you wanna add types you can jump to console and use mypy-pyright from there

[rare scarab]

You could redesign your script via ```py
if review.get("reviewText") is None or review.get("summary") is None:
raise ValueError("Incomplete review")
return {
"overall": review["overall"],
"asin": review["asin"],
"reviewText": review["reviewText"],
"summary": review["summary"],
"unixReviewTime": review["unixReviewTime"]
}

You could also genericize it.

[mild root]

NonDecresingList = List
NumberOfDuplicates = int

def remove_duplicates(nums: NonDecresingList[int]) -> NumberOfDuplicates:
    ...

is this abuse of type hinting or a valid way to use it?

[trim tangle]

there's definitely nothing illegal here, but I'd just put that in a docstring or comment

[hazy condor]

o

[mild root]

I mean, I know it's allowed by syntax, just trying to gather if it is prohibited by any sort of standard or anything like that

[trim tangle]

typing is the wild west

[mild root]

I see

I mean I do prefer reading and maintaining the type hints rather than the docstrings

[rare scarab]

You may want to consider using Annotated[int, Doc("Number of Duplicates")] from pep 727

!pep 727

[rough sluice]

**PEP 727 - Documentation in Annotated Metadata**

Status

Draft

Python-Version

3.13

Created

28-Aug-2023

Type

Standards Track

[rare scarab]

^ That could really benefit from an Annotated shorthand syntax, like int @ Doc("doc here")

no language changes need to be done for that, just add __matmul__ to type

[mild root]

Looks easy to implement, how do changes get through ? Can people just open a PR ?

[rare scarab]

We'd need a pep

!pep 1

[rough sluice]

**PEP 1 - PEP Purpose and Guidelines**

Status

Active

Created

13-Jun-2000

Type

Process

[rare scarab]

Pep because it defines a standard

[mild root]

Ah, is literally PEP 1 explaining the thing

So anyone can propose a PEP ?

I'm gonna read it

[oblique urchin]

PEPs need a sponsor (generally a core dev)

[rare scarab]

It's a good idea to open a thread about it in the ideas group on https://discuss.python.org

[frigid jolt]

Could be a little off topic with the channel but, can core devs sponsor themselves?

[mild root]

I wouldn't mind contributing to python, I've been looking around in the interpreter code from time to time.

[mild root]

oh just noticed this is 3.13 and draft status

still looks quite verbose tho

am I documenting, or am I obfuscating the actual types

I wouldn't say nothing was achieved, because it did get me to pay attention to the constraints of the problem

but still looks like a lot of work when I could just paste it in the docstring

no this could be cool I think, if I now hover over the "word" and get its description like you do when you hover over a method

[mild root]

Word = str @ (
    Doc("Character sequence consisting of non-space characters only.")
    + Assert(lambda x: 1 <= len(x) <= 20)
    + Assert(lambda x: " " not in x)
)

ListOfWords = List[Word] @ (
    Doc("List of words with lenght between 1 and 300.")
    + Assert(lambda x: 1 <= len(x) <= 300)
)

MaxWidthInt = int @ Assert(lambda x: 1 <= x <= 300)
Sentence = str @ Doc("A string with max width of ´MaxWidthInt´")

[soft matrix]

Core devs don't need sponsors for peps (so sort of ig)

[drifting pike]

so I have this class, which in order to get working with starlette middlware I need c2App to be a keyword arg. Though if I do not provide an instance of the app it fails, so I just set it to =None.

How would I got about type hinting it?

from starlette.middleware.base import BaseHTTPMiddleware

class FilesystemLoggingWare(BaseHTTPMiddleware):
    # c2App : base.asgiApp
    def __init__(self, app, c2App=None):
        super().__init__(app)

Or does python not support type hinting keyword arguments with default values?

I guess typcically the type of the default value is enough to infer, but in this case by default case is None since the keyword always needs to be provided.

nvm, found this... https://www.programming-books.io/essential/python/type-hints-for-keyword-arguments-a076af2fcb824840af68cb87c3fa0c6f

[viscid spire]

that is only for style

[viscid spire]

you need a union of the type you expect, and the type of the default value (if they mismatch)

if your version is new enough use T | None where T is the expected passed type. Else use Union[T, None] from typing, or Optional[T] from typing

[drifting pike]

I see

thank you

[drifting pike]

@viscid spire looking more into how type hinting works, would Required[T] work in this case?

def __init__(self, app, c2App: Required[base.C2App] = None):

[viscid spire]

that is for TypedDict

[drifting pike]

ah

[viscid spire]

also think logically about that: a default value + Required?

if it's Required, then why would you ever use the default value?

are you looking for a way to force the kwarg to be used?

you can force a parameter to be a kwarg by putting it after *args or *

!e

def f(*, x: int): ...
f(x=3) # fine
f(3) # error at runtime

[rough sluice]

@viscid spire :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 3, in <module>
003 |     f(3) # error at runtime
004 |     ^^^^
005 | TypeError: f() takes 0 positional arguments but 1 was given

[drifting pike]

https://www.starlette.io/middleware/#basehttpmiddleware
because that's the way the documentation tells you to do it.

I think it's because I'm overloading a function, so otherwise my argument would invalidate the signature of the class I'm inheriting from.

if I remove the default value the code crashes

though there is never an insntance where the class can be constructed without the arg being provided

[viscid spire]

in what way?

[drifting pike]

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/hypercorn/asyncio/task_group.py", line 23, in _handle
    await app(scope, receive, send, sync_spawn, call_soon)
  File "/usr/lib/python3/dist-packages/hypercorn/app_wrappers.py", line 33, in __call__
    await self.app(scope, receive, send)
  File "/usr/lib/python3/dist-packages/starlette/applications.py", line 115, in __call__
    self.middleware_stack = self.build_middleware_stack()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/starlette/applications.py", line 102, in build_middleware_stack
    app = cls(app=app, **options)
          ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: FilesystemLoggingWare.__init__() got an unexpected keyword argument 'c2App'

whereas otherwise the server works and logs my requests for static files properly.

[viscid spire]

🤔

what's the differenc between app and c2App?

[drifting pike]

c2App is just a basic wrapper I have surrounding asgi apps, it mainly contains SSL config details. app is the actual asgi application.

[viscid spire]

think I'm getting lost in the terminology

this doesn't seem like a typehinting problem

[drifting pike]

it's an atypical case. typically this logic would make sense, but here it doesn't for other semantic reasons.

[viscid spire]

the issue is that **options contains "c2App", but FilesystemLoggingWare doesn't take this kwarg

[drifting pike]

I think that's specifically the issue.

but I can't work around it within my code.

[viscid spire]

you must

[drifting pike]

ig I could override __new__ or something, I think I might just go with T | None since that's easier.

[viscid spire]

I don't see how a typehint will fix your runtime error

[drifting pike]

the error only occurrs if I don’t set the default value to None

[pliant vapor]

from typing import Protocol
from operator import itemgetter
from collections.abc import Callable


class SupportsGetItem[T, V](Protocol):
    def __getitem__(self, key: T) -> V: ...


def multi_itemgetter[T, V](*keys: T) -> Callable[[SupportsGetItem[T, V]], tuple[V, ...]]:
    ig = itemgetter(*keys)
    return lambda obj: (ig(obj),) if len(keys) == 1 else ig(obj)

anyone know why this isn't valid type hinting? pyright says

Argument of type "SupportsGetItem[T@multi_itemgetter, V@multi_itemgetter]" cannot be assigned to parameter "obj" of type "SupportsGetItem[Any, Any]" in function "__call__"
  "SupportsGetItem[T@multi_itemgetter, V@multi_itemgetter]" is incompatible with protocol "SupportsGetItem[Any, Any]"
    "__contains__" is not present  (reportGeneralTypeIssues)

[soft matrix]

well the protocol that is for SupportsGetItem in _typeshed which is the one itemgetter uses for call requires contains to be implemented

might be a typeshed bug

[pliant vapor]

weird

[grave fjord]

I'm trying to type annotate the twisted trial TestCase class but I'm getting an issue with mixins defining asynchronous tearDown methods: https://mypy-play.net/?mypy=latest&python=3.11&gist=721064188d7ccae3d8360fc748e1c781&flags=strict

from typing import TypeVar, Generic

from unittest import TestCase

T = TypeVar("T")
class Deferred(Generic[T]):
    def __init__(self, v: T):
        self._v = v

class TxTestCase(TestCase):
    def tearDown(self) -> Deferred[None] | None:  # type: ignore[override]
        return super().tearDown()
        
        
class FooMixin:
    def tearDown(self) -> Deferred[None]:
        return Deferred(None)


class TestFoo(FooMixin, TxTestCase):
    pass

the problem is when I define a mixin with a tearDown and then mix it into a TestCase subclass the ignore[override] doesn't seem to be taken into account

ah issue is here https://github.com/python/mypy/issues/12372

[bright tangle]

Is there a way of using Annotation[] such that Foo[type] and Foo[extra_data][type] are both valid? (I'm not super attached to the specific syntax, FWIW, but I would like the extra data to come before the type)

(currently I've got Foo = Annotated[T, "__myproject_Foo"])

(I'm also not super attached to it being Annotation specifically, having a custom class with no effect on type-checker behaviour is also fine)

[rare scarab]

That seems fine.

though you can't do Foo[extra_data][type]

[tranquil turtle]

i think you can, if extra_data is a typevar

[rare scarab]

so typealiases are auto-curried

[tranquil turtle]

i want to do something like this: py x: array[int, 5] array is a generic class, in this case it is specialized with int
it also has length 5

typechecker will not accept this in any way, because 5 is not a valid type

is there a way to trick typeckecker into thinking that array[int, 5] is just array[int]?

i was thinking about something like this: py class _array(Generic[T]): ... # actual class array = Annotated[_array[T], T2] # public type alias to use but this doesnt work, because only first thing in Annotated is subscriptable, everything else if left as is

one possibility which i dont like: x: array[int] = array_info(len=5) where def array_info(len: int)->Any:... (and then it will be inspected and correct descriptor will be put into that place)

[tranquil turtle]

i forgot to mention that I've implemented this behaviour at runtime by defining __class_getitem__, so the only thing left is to make typechecker understand it

[undone saffron]

data model is pretty explicit about this that the purpose is for this to be for type info, the length of a container isn't something the type system considers as type info. (see: https://docs.python.org/3/reference/datamodel.html#object.__class_getitem__)

Use of annotated here would need to be as

a: array[Annotated[int, 5]]  # poor form, see note below
a: array[Annotated[int, ArrayLen(5)]]  # plays well with others and introspection

However, doing this is heavily discouraged without wrapping your use of things in Annotated in a container that identifies it as yours. Several libraries that do validation (msgspec, pydantic, etc) all do this, and have to have their own classes for this because you are supposed to ignore everything in Annotated which isn't yours.

without explicit support for refinement types in the type system, there isn't a way for this to be typed and be type info. You can hack stuff together that looks like type information, but there's no guarantee of any specific tool understanding it. We have a subset of refinement types with NewType + TypeGuards, as well as Literals, but we don't have anything as powerful as many other languages for this.

[unique wigeon]

ImmutableMultiDict([('urunListesi[0][urun][aciklama]', ''), ('urunListesi[0][urun][ad]', "Alüminyum Zil Butonu 2'li"), ('urunListesi[0][urun][barkod]', '2800001547386'), ('urunListesi[0][urun][birim]', 'adet'), ('urunListesi[0][urun][stokKodu]', 'ZB.ALÜ02'), ('urunListesi[0][urun][id]', '4'), ('urunListesi[0][urun][marka]', 'Yok'), ('urunListesi[0][urun][markaId]', '8'), ('urunListesi[0][urun][alisFiyati]', '0.0'), ('urunListesi[0][urun][satisFiyati]', '60.0'), ('urunListesi[0][miktar]', '1.0'), ('indirimTutari', '0.0'), ('satisNotu', ''), ('odemeYontemi', '')])
Hello,
how can i parse this object?

I tried everything I found on the internet and AI

(I send it from the Flutter mobile application to an API created with Flask via the post method.)

[trim tangle]

is this about type hints?

[elfin nexus]

if a function does not have a return type hint, then is it assumed that the "default" return type is None or Any?

[void panther]

Any, or a typing error

[trim tangle]

Depends on the type checker 🙂

Pyright infers the return type (you can hover over the function to see it), while mypy assumes it's Any iirc

[unique wigeon]

what about

[trim tangle]

this is a channel about type annotations (also known as 'type hints'). if you have a question about something else, check out a different channel or see #❓｜how-to-get-help

[unique wigeon]

thank you so much

[undone canopy]

Does anyone know a good tutorial that explains what is __class_getitem__ and what can it be used for?

[lunar dune]

I sank some time into revamping the data-model docs on __class_getitem__ a while back (it was one of my first CPython contributions!), so hopefully they're useful 🙂 https://docs.python.org/3/reference/datamodel.html#emulating-generic-types

[undone canopy]

From that I understand that I should not implement that method as Mypy won't understand it. However, some projects such as NumPy implement it, and apparently the Pytorch typing project is able to do crazy things, such as optional generic parameters, which I do not think are available for poor mortals that inherit from Generic.

[lunar dune]

From that I understand that I should not implement that method as Mypy won't understand it.

Yes, that's the general recommendation. It's possible you could make mypy understand your custom use of __class_getitem__ through clever use of stub files, if TYPE_CHECKING blocks, or a custom mypy plugin. But the first two would mean you'd basically be lying to the type checker about what you're doing (which is fine, in some cases, but you should be aware that that's what you're doing 😉 ). The third option would mean you'd be utilising nonstandard extensions to the type system that might be fragile, and would probably be only understood by mypy

The standard library uses __class_getitem__ so that classes in other parts of the standard library can be generic at runtime without having to depend on the typing module (for various reasons, it's generally good for there not to be complex dependencies between modules within the standard library). At "typing time", type checkers only look at typeshed's stubs, so they think that all these classes inherit from typing.Generic, even though that's not actually true. It's possible that numpy does something similar with their stubs

I don't know what PyTorch is doing, so can't comment there, I'm afraid

[undone canopy]

Ok, I see. Thank you!

[lunar dune]

no worries!

[high wigeon]

It's fine because the type system is also a bit of a lie and doesn't quite correspond to the runtime types :P

[undone canopy]

Another (unrelated) question. Now that PEP 646 is finally implemented in Mypy, does anyone know which syntax will be used to type shapes in NumPy?

[fierce ridge]

there's a numpy plugin for mypy, so anything magical they might be doing is possibly implemented there

[tranquil turtle]

i know that i am lying to the typechecker, but i really want pure "annotation-only" style without class-level values
i think i will introduce another lie: i will shadow Annotated and do x: Annotated[array[int], 5] (or x: makearray[array[int], 5]) which will create _array[int, 5] at runtime

[rain warren]

Is there a nice one-liner to get a list[T] from a list[Optional[T]], enforcing that no elements in the list are None?

[viscid spire]

from typing import Optional, cast
def f[T](xs: list[Optional[T]]) -> list[T]:
    if None in xs:
        raise ValueError
    return cast(list[T], xs)

def gimme_an_int_list(xs: list[int]) -> None:
    pass

xs: list[Optional[int]] = [1, 2, None, 4, None, -1, -2]
gimme_an_int_list(f(xs))

that's what I came up with just now (works with pyright)

although I don't think you should use Optional[T] for anything except parameters with a default value of None

because the word "optional" doesn't convey that so well imo

[rain warren]

That seems to work! Not exactly a one-liner through 😅

[viscid spire]

also should add a better error msg

[viscid spire]

instead use T | None / Union[T, None] for semantics

[rain warren]

It's a list I'm incrementally constructing in the program. Initially, it's [None] * length, but then more string elements are set. When it's done, there are only strings in it.

[viscid spire]

I mean, if you can be certain there are no Nones left you can just cast the variable without a function

cast is for when you wanna say "I know better than the typechecker"

[pastel egret]

Why not instead do [''] * length?

[rain warren]

What I have now is

assert all(isinstance(i, str) for i in my_list)
return cast(list[str], my_list)

Maybe that's as good as it gets 🤷‍♂️