type-hinting

[trim tangle]

😩

maybe I should've just avoided variadic generics here

ok another question:

@dataclass(frozen=True)
class Row:
    frac: Fraction = Fraction(0)
    min: int | None = None
    max: int | None = None


_ToRow = int | Fraction | tuple[int, int] | tuple[Fraction, int, int]


def _to_row(raw: _ToRow) -> Row:
    match raw:
        case int(n):
            return Row(min=n, max=n)
        case Fraction(f):
            return Row(f)
        case (minh, maxh):
            return Row(min=minh, max=maxh)
        case (frac, minh, maxh):
            return Row(frac, minh, maxh)
        case invalid:
            assert_never(invalid)

pylance is telling me that invalid has the type Fraction. How so?

I guess I can do ```py
case Fraction():
return Row(raw)

oh wait, maybe Fraction just doesn't define a class pattern like that

[oblique urchin]

I think a few builtin classes were special-cased

[trim tangle]

💀

[soft matrix]

In [3]: fractions.Fraction.__match_args__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 fractions.Fraction.__match_args__

AttributeError: type object 'Fraction' has no attribute '__match_args__'
```yeah looks like youre out of luck

[trim tangle]

ok then it's not a typing issue

more of a

decorator-factory/skill#1

[mighty linden]

GitHub

[skill] #1 skill issue

[soft matrix]

what would f be here?

[trim tangle]

the same as raw, I guess?..

[soft matrix]

cause you can always just use

case Fraction():
    return Row(...)

[trim tangle]

like with int

[rocky sigil]

error: Incompatible types in assignment (expression has type "List[TBaseGrain]", variable has type "List[TBaseGrain]")

pulling my hair out here

it's literally the exact same

it's the same TypeVar

what am I doing wrong

[trim tangle]

Show the code

Sometimes that happens if you import something incorrectly

[tranquil turtle]

this happens when you bind one typevar in two different places, so they are not the same

or you have some meaningless function signatures

I give up, maybe im wrong, i cant reproduce it in one file.

[slender timber]

Why can ClassVar not hold TypeVars?

[soft matrix]

Cause it's technically ambiguous as to what they would refer to at runtime but I think there was talk of changing that in the last typing meeting

I think it was more of an issue with early typing but nowadays it's less of an issue

[slender timber]

Also, how can I derive from a protocol but keep it unimplemented?

[rocky sigil]

this sounds right to me, how do I fix it though?

do I have to redeclare the typevar separately in each individual file?

[soft matrix]

Keep what unimplemented?

[brisk hedge]

add Protocol to the bases

[slender timber]

class IFixedSizeEvent(Protocol[T]):
    ID_RANGE: ClassVar[tuple[int, int]]
    CODEC: c.FormatField[T, T]    # can't make this ClassVar as well


class PODEventBase(EventBase[T], abc.ABC):
    """Base class for events whose size is predetermined (POD types)."""

    def __init__(self, id: int, data: bytearray):
        if id not in range(*self.ID_RANGE):
            raise EventIDOutOfRange(id, *self.ID_RANGE)

        if len(data) != self.CODEC.sizeof():
            raise InvalidEventChunkSize(self.CODEC.sizeof(), len(data))

        super().__init__(id, data)

class ByteEventBase(PODEventBase[T], abc.ABC):
    """Base class of events used for storing 1 byte data."""

    ID_RANGE = (BYTE, WORD)

class I8Event(ByteEventBase[int]):
    CODEC = c.Int8sl

class U8Event(ByteEventBase[int]):
    CODEC = c.Int8ul

I would like to keep impl in base class

but child class should implement the protocol

[slender timber]

how?

[oblique urchin]

class ChildProtocol(BaseProtocol, Protocol):

[slender timber]

how would i make use of that for PODEventBase

[rocky sigil]

ok I have a MWE for this problem I'm having

file_1.py:

from typing import TypeVar, List


class Bird:
    def __init__(self, species: str) -> None:
        self.species = species


TBird = TypeVar('TBird', bound=Bird)


def get_first_two_birds(list_of_birds: List[TBird]) -> List[TBird]:
    return list_of_birds[0:1]

file_2.py:

from typing import List

from file_1 import Bird, TBird, get_first_two_birds


def make_two_birds() -> List[TBird]:
    bird1 = Bird(species="owl")
    bird2 = Bird(species="ostrich")
    bird3 = Bird(species="pelican")

    birds_list = [bird1, bird2, bird3]

    two_birds = get_first_two_birds(birds_list)

    return two_birds

if I run mypy file_2.py I get: file_2.py:17: error: Incompatible return value type (got "List[Bird]", expected "List[TBird]")

[trim tangle]

def make_two_birds() -> List[TBird]:
This signature doesn't really work, the purpose of a type var is to link two types together

[rocky sigil]

actually its not exactly the same problem by the looks of things

[rocky sigil]

I thought it was to make lists of types work?

[trim tangle]

hm?

did you mean list[Bird]?

[rocky sigil]

if I don't use TypeVars, I run into big problems with lists of subtypes

e.g List[Owl] is incompatible with List[Bird] even if Owl is a subtype of Bird

[trim tangle]

I have a small tutorial on generics, variance and all that stuff:
https://decorator-factory.github.io/typing-tips/tutorials/generics/
have you checked it out?

[rocky sigil]

also I ran into that exact same filtration problem

[trim tangle]

Imagine if List[Owl] was compatible with List[Bird].

def add_penguin(birds: List[Bird]) -> None:
    birds.append(Penguin("Quack"))

owls: List[Owl] = ...
add_pengin(owls)
# oops, now there's a penguin in the list of owls!

[rocky sigil]

so would it have to accept List[Bird] | List[Owl]?

[trim tangle]

Most of the time you don't really need a list, you're just reading the items, and it will work with a tuple, and maybe even a set. For that purpose, use collections.abc.Sequence or collections.abc.Iterable depending on what you're doing

[rocky sigil]

in your linked example is exactly the use of a TypeVar that I am experiencing

[trim tangle]

what example?

can you show your real code maybe?

[rocky sigil]

afraid not, I don't have copyright permissions yet 😦

and I'm struggling to recreate an MWE

evens = filter_list(lambda x: x % 2 == 0, [1, 2, 3, 4, 5, 6])
# evens: Iterable[int]

numeric = filter_list(str.isdigit, ["abc", "123", "", "abc3.14"])
# numeric: Iterable[str]

this is the exact issue I'm walking into

which is why I'm using a TypeVar

I want to say "this function will accept a list of Bird or a list of any Bird subclass, and will return a list of the same type"

[trim tangle]

you're working on something confidential, and you don't have any colleagues who can help with your concrete code?

[rocky sigil]

so if it gets a list of Owl, it'll return a list of Owl

[trim tangle]

Will it explode if you pass in a tuple?

[rocky sigil]

no-one that knows Python 🙂

[trim tangle]

from collections.abc import Iterable
from typing import TypeVar, Generic

B = TypeVar("B", bound=Bird)

def filter_birds(birds: Iterable[B]) -> list[B]:
    return [bird for bird in birds if bird.can_fly()]

actually, a list would work

[rocky sigil]

ok that's basically what I ended up with

but still running into this one error

Incompatible return value type (got "List[TBird]", expected "List[TBird]")

i'll try again to make a MWE

[trim tangle]

This works for me @rocky sigil:
https://pyright-playground.decorator-factory.su/?gzip=H4sIAOuoOWMC_52SwUrEMBCG73mKoSA0sIp4LNRDYQUv6mHRQykl2062wTQpk9Slb28SW1E8CDuHmSHzz5efEEl2BL9MypxAjZMlD4dlwldBjHVaOAeVor5gEKJHCZ0wrdRL7lBLDtf3cLRWF0BCOYQn6x_HSeOIxmO_J7LflDcMmIjKY-L_AdM4BqGfycAg3JBn3aC6dzQZhyu4g7KE2w3_fNaXgQ804wZ5QXOalbkM9CC0CyRWQbm9YJ5V2S6oZ9OXiclY5EmlPVJ7DCcuT7kArZyvqybh1_7rhpVeRx1IS5AaZVJ1oGRqbjaPvAkW7Fm74KKOj8J38LM0adj6Qfh23QnKX46igDPCDxS6DT8D8z8rnH0C_YoRNjcCAAA%3D

[rocky sigil]

yeah it works for me too, I'm struggling to recreate it

[trim tangle]

what version of mypy are you running?

[rocky sigil]

0.981

it's gotta be import-related

because the TypeVar is defined in one file, and imported in another file

all the stuff in the first file using the TypeVar works fine

but when I use the TypeVar in the second file, it breaks

[trim tangle]

importing typevars is generally not the best idea

I think

just make a new one

[rocky sigil]

should I re-define it?

[trim tangle]

yes

[rocky sigil]

like, import the class and make a new typevar?

[trim tangle]

yes

[rocky sigil]

ok one sec

[trim tangle]

typevars are a bit of a hack to be honest

[rocky sigil]

same error i'm afraid

[slender timber]

How to extract the type of a generic type param?

It was T.__args__[0] ig?

[rocky sigil]

kinda feels like I should've written this in a statically-typed language lol

this is my first attempt at inheritance etc with base classes

and I wanted to type hint for safety

[trim tangle]

did you by any chance name the type var the same as the class?

[rocky sigil]

no

it's TBird rather than Bird

got it!!!

@trim tangle

ok it's in two files:

file_1.py:

from typing import List, TypeVar


class Bird:
    def __init__(self, species: str) -> None:
        self.species = species


TBird = TypeVar('TBird', bound=Bird)


def filter_birds_to_penguins(birds_list: List[TBird]) -> List[TBird]:
    penguins_only = [a_bird for a_bird in birds_list if a_bird.species == "penguin"]
    return penguins_only

file_2.py:

from typing import List

from file_1 import TBird, filter_birds_to_penguins, Bird


class BirdsCollection:
    def __init__(self, birds_list: List[TBird]):
        self._birds = birds_list

    @property
    def birds(self) -> List[TBird]:
        return self._birds


def filter_to_penguins(birds_list: List[TBird]) -> List[TBird]:
    return filter_birds_to_penguins(birds_list)


bird1 = Bird("penguin")
bird2 = Bird("ostrich")
bird3 = Bird("owl")

birds = [bird1, bird2, bird3]

birds_collection = BirdsCollection(birds_list=birds)

just_penguins = filter_birds_to_penguins(birds_collection.birds)

it's related to the @property

file_2.py:27: error: Argument 1 to "filter_birds_to_penguins" has incompatible type "List[TBird]"; expected "List[TBird]"

(this is very similar structurally to my actual code)

@trim tangle it even works in just one file

so it's not import-related

https://pyright-playground.decorator-factory.su/?gzip=H4sIAOauOWMC_51Ty2rDMBC8-yuWXOJAGmhyC7iU9lp6Cr0YIxxnnaookpDkFv99V7L8Kg0p1cXSeHZ2NGvXRl3AtZrLM_CLVsbBC7duDYdW41tpkiSpRGktPHFz2idA64Q1MMYld4ylFkW9Bqux4mj3YJ1Zwd0DvCqJHdsvT9pEDmQ9m6QPXpWQ2CxdBmC5hqNq5CnzhxXRfMeaC4eGHQmyzCmmUZ4bLm3aIYJM74P1PGgUwcbk3Lnpq5iSoqXGeRkUoVYG4pZLGCWB1xEf_WewiDKLIogadI2Rc-1ZbvZZCYGV40pejfDaNX6k2AVAzkd-EhiP2iiNxrVDg8AI6r9nMbE-kZ7H_e-go_AfpuYH7E_3dCkvkA7prgK-HXBFXxev3iO-G_EvQVjSJ5MHtS7RbffYFfE1q4ZJxPLJbCamsrAl0Y_GusE1ldy40Ci_6RUMfmIpGP1jmN5sGKu-AU9G7zKWAwAA

Expression of type "List[TBird@__init__]" cannot be assigned to return type "List[TBird@birds]" TypeVar "_T@list" is invariant Type "TBird@__init__" cannot be assigned to type "TBird@birds"

[trim tangle]

You need to inherit BirdsCollection from Generic[TBird]

There are some examples in the link

[rocky sigil]

could this also be solved by refactoring BirdsCollection as a proper container type?

like subclassing MutableSequence

your suggestion did fix the error

but not the error in my actual code

the error i'm having is on a collection of collections (i.e a BirdsCollectionCollection)

I've made sure that BirdsCollection is a Generic[TBird]

and that BirdsCollectionCollection is a Generic[TBirdsCollection]

ok I think this is a mypy bug

@trim tangle I've been able to re-create the actual conditions now of my bug:

https://pyright-playground.decorator-factory.su/?gzip=H4sIABy1OWMC_42UQW6DMBBF95xilE1AohwgElXVLrqpuoq6iSKLwBBZcmzLdqRy-9rGAQMhDSswf97M_za0SlzAdJLyM9CLFMrAF9Umh30n8adSOXwiR0XrJElqVmkN71Q1uwTs1WALhFBODSGpRtbmoCXWFPUOtFEZvLzCt-DYq93lREXQQHlTW_TeUe1K6Jpu_cI2h5O48qZ0D9lkAv0hGMPaUMHTMOHB1xyz1eFOro4wa2_nTU4LhvmI19lhRn3iFW9SCYnKdEMDr_B0bzaCjkyF5qp4jL75jTwsrEfvJilE66uBrEUTSf5NqR6ki8DuYKbZRcVjjDPgo0Tj-nvhxgM8yDnGrLWrGCPPbOKgWx4f6_BwHIStUHMPQMNU86FG_LJFgb8GeZPOajxDZ3PT09o1sy1lBhV51nNgh6oebwSRyM9XGnamGGDuNEZNlvK1r-9u_1sVEZx1LuHKE3284damOiKBtmF9_L2UsAmYTb8_wdCE_QfFyJxjAAUAAA%3D%3D

[trim tangle]

@rocky sigil I think you want what's called "Higher Kinded Types". and those are not possible in Python's typing system

If you're coming from Haskell it's a bit of a bummer

If it's not that, maybe you could give an example of what you want to do in a different language?

[rocky sigil]

I think that's the idea

[rocky sigil]

I'm not really coming from any other language (Python is the only one I know in any significant detail)

I think I was just trying to think of a structure that would solve this problem

I've solved it in another way - by flattening some of my inheritances

it means more duplicated code, but solves the typing ambiguities

[trim tangle]

well it's kinda hard to help if we don't know the problem 🙂

[rocky sigil]

I think the way I've tried to do inheritance is a bit dumb (as I said, it's my first time doing this)

[trim tangle]

I would suggest asking some colleagues, even if they don't know Python (but know some statically typed language)

[rocky sigil]

yeah, I get that - I appreciate all your support so far though, you've been very helpful

I'm a postgrad student writing a data analysis package for some scientific data I've collected

so it's basically down to me to write the package to analyse it

none of my peers are co-workers and neither are my supervisors

they use pre-existing software tools, but none of those exist for the type of data I'm collecting, so I'm kinda on my own here

[trim tangle]

well, it's not the end of the world

sometimes you'll have to do a bit of Any business

[rocky sigil]

aye, it's actually solved now but I had to tear down my inheritance idea and remove a base class (and put its methods inside the old subclasses to make them independent)

to be honest I think I was trying to force inheritance that makes things harder rather than easier

[sturdy willow]

Huh, so I'm using a dict as a registry to store class types for later retrieval and initialization. Making such classes a dataclass breaks PyCharm's inline documentation.

@dataclass
class Position:
    x: int = 0
    y: int = 0

registry = {}
registry['Position'] = Position

position: Type[Position] = registry.get('Position')
p1 = position(x=10, y=10)  # <-- no documentation

I know this isn't specifically a type hinting issue, but wanted to see if anyone else had come across this issue.

[fierce ridge]

pycharm's typechecker is kind of notoriously buggy and incomplete

[sturdy willow]

Bah, good to know

Do you recommend an alternative, or do I just deal with it? lol

[oblique urchin]

pyright/pylance (which works in vscode) is great

[sturdy willow]

Gotcha - it's been a while since I've worked with vscode, so I'll try that out

[weak elm]

Does anyone know that Rust based Python linter made by someone? I heard about it somewhere and apparently it is 90% faster than Black

[green gale]

black is not a linter

this claims to be faster than flake8, but i haven't tried it https://github.com/charliermarsh/ruff

[buoyant swift]

doesn't say blazingly fast, can't be true

[weak elm]

Sry meant to say formatter perhaps?

[weak elm]

Ah I think this was it. Thank you!

[brisk heart]

No plugins ain't ideal tbh

[slender timber]

Black is a formatter, like rustfmt
Ruff is a linter, like pylint (shit) or flake8

[brisk heart]

wouldn't --fix make it also a formatter?

[rare scarab]

you should prefer autopep8 if you want to correct flake8 issues.

Though the difference is flake8 isn't limited to format issues.

[trim tangle]

yeah it's a bit different

it can also operate on the level of the AST

and more high-level concepts

!pypi flake8-pep585

[rough sluice]

flake8-pep585 v0.1.5.1

flake8 plugin to enforce new-style type hints (PEP 585)

[trim tangle]

🔌

[brisk heart]

base flake8 is useless if I use black but it's really worth it for all the minor stuff that plugins do

The most helpful one has been some that reported any found print calls, saved me way too many times

[blazing cobalt]

I just wrote this up: https://dogweather.dev/2022/10/03/i-discovered-that-python-now-can-do-true-match-exhaustiveness-checking/

If I'm wrong, please oh please let me know 🙂

[fierce ridge]

i think your presentation is a little confusing here. it sounds like you're conflating python itself with pyright

discovering new features is always exciting and fun though

and good on you for writing it up, i and i'm sure many people are always subject to the "i'll write it up later" fallacy

[blazing cobalt]

Thanks! Yeah, I edited the title to make the Pyright aspect clearer. I go into it more in the article.

Yep, I'm guilty as well of ignoring a backlog of blog ideas. 🙂

[slender timber]

Is there any library that uses Annotated for binary data structures?

They will look like dataclasses at the end

[fierce ridge]

like struct but with class/attribute syntax? i'm not aware of one but it sounds like a good idea

[trim tangle]

Something like this but with classes?
https://pypi.org/project/borsh-python/

hm well, do you really need Annotated?

oh hm, you do need some additional information like endianness

[fierce ridge]

imo yeah because if you do something like pydantic conint mypy chokes on it without a plugin

[trim tangle]

conint?

[fierce ridge]

s / pyright / pydantic

[trim tangle]

you could use dataclass_transforms

and specify some descriptors

[slender timber]

No that's not what I meant. I mean something like the examples in pep526. To get type annotations for binary types

[oblique urchin]

what do you mean by binary types?

things like int64?

[slender timber]

Yes

[trim tangle]

how would that help

[slender timber]

Like for example:

class Binary:
    int1: uint32

[trim tangle]

Maybe you meant something like this?

@binary
class City:
    x: int = uint64()
    y: int = uint64()
    name: bytes = pascal_blob(max_length=128)

[slender timber]

Type hints primarily

[slender timber]

Yea

[trim tangle]

I'm actually not sure if that's possible with dataclass_transforms, but definitely look into it

[soft matrix]

id personally just use NewType here

[trim tangle]

NewType also works, yeah

Although if you have a lot of parameters it's going to be a pain

[soft matrix]

oh very fair

[oblique urchin]

somewhat relatedly, mypy people are working on adding support for fixed-size integers to mypyc

[trim tangle]

integers can be:

• signed/unsigned
• big/little endian
• contain different amounts of bytes
• variable length

[slender timber]

I am using construct for the structs. It uses strings for the field names so there's no type hints in it

!pypi construct

[rough sluice]

construct v2.10.68

A powerful declarative symmetric parser/builder for binary data

[slender timber]

!d typing.NewType

[rough sluice]

typing.NewType

class typing.NewType(name, tp)```
A helper class to indicate a distinct type to a typechecker, see [NewType](https://docs.python.org/3/library/typing.html#distinct). At runtime it returns an object that returns its argument when called. Usage:

```py
UserId = NewType('UserId', int)
first_user = UserId(1)
```   New in version 3.5.2.

Changed in version 3.10: `NewType` is now a class rather than a function.

[slender timber]

No, this will not be able to encode / decode

I would still need to use descriptors or something

[trim tangle]

well, Annotated doesn't encode/decode on its own either

furthermore, if you have an annotated thing like uint64, there's no way to prohibit Annotated[int, uint64]

but then, you can just check it at class definition time, which hopefully happens first thing when you run the program

since you'll have to add some decorator to the class anyway

[slender timber]

No but using annotations, encoding / decoding can be implemented without having to use anything more like descriptors

[oblique urchin]

annotations don't do anything by themselves

[slender timber]

Exactly, but a metaclass can parse the annotated fields to internally create a struct.Struct or BytesIO if its a stream, plus some magic?

[brisk hedge]

With added difficulty to type checking.

[slender timber]

Well all integer types are just int in python

[void ibex]

I have a pydantic model: ```py
class UserUpdate(pydantic.BaseModel):
is_mod: bool
is_banned: bool

I'd like to have another model, which has the exact same fields, but each field in from parent model is Optional (so that it can be used with a partial PATCH, rather than a full PUT/POST.

I currently have this that _works_ with FastAPI the way I want.
```py
class UserPatch(UserUpdate):
    __annotations__ = {k: t.Optional[v] for k, v in UserUpdate.__annotations__.items()}

However, my IDE's lang server doesn't pick up the overwrite of the annotations, so whenever UserPatch is used, none of the attrs appear as optional.

Is there a way to do this, but in a way that preserves typing?

[heady flicker]

No. LSPs are intentionally stupid like that. They can't see through all the dynamic code you're writing.

So doing what you're doing non-manually is impossible and likely will never become possible.

[void ibex]

Ah yea, that does make sense. Thanks for your help.

[trim tangle]

Yeah, type checkers have a limited set of patterns they understand

Although, for example, in TypeScript you could express something like that

[brisk heart]

maybe it could become possible once subscripted generics become possible (like the T1[int] thingy, idk what it was called)

[trim tangle]

higher-kinded types?

[oblique urchin]

that would be higher-kinded types

[trim tangle]

well, not in the general case

[brisk heart]

ye

that

[trim tangle]

although hmmm

yeah, you could do ```py
class UserUpdate(pydantic.BaseModel, Generic[K]):
is_mod: K[bool]
is_banned: K[bool]

and then provide `K` as one of ```py
K = Lambda[[T], T]
K = Optional
X = Lambda[[T], ErrorValue | T]

[trim tangle]

Does pydantic understand generic models? I guess you could do ```py
Missing = TypeVar("Missing")
class PartialUserUpdate(pydantic.BaseModel, Generic[Missing]):
is_mod: bool | Missing
is_banned: bool | Missing

NullableUserUpdate = PartialUserUpdate[None]
FullUserUpdate = PartialUserUpdate[NoReturn]

[brisk heart]

is Lambda something official?

[trim tangle]

that's pseudocode for type-level functions 🙂

[brisk heart]

ah makes sense

[trim tangle]

Or ```py
Missing = TypeVarTuple("Missing")
class PartialUserUpdate(pydantic.BaseModel, Generic[Unpack[Missing]]):
is_mod: Union[bool, Unpack[Missing]]
is_banned: Union[bool, Unpack[Missing]]

FullUserUpdate = PartialUserUpdate[()]
NullableUserUpdate = PartialUserUpdate[None]

[brisk heart]

huh, is NoReturn completely disregarded like that? I'd expect it to get turned into None 🤔

[trim tangle]

NoReturn is the empty type, which has no members

so NoReturn | T is the same as T

[brisk heart]

oh that's interesting, never though of it like that

[oblique urchin]

yes, you can think of it as an empty union

[trim tangle]

There's a new Never thing which is just a synonym for NoReturn

fixed naming

I'm actually surprised Union works with Unpack, that seems super useful

I wonder if I can make something cursed with this

[void ibex]

Doesn't seem so :(

[trim tangle]

(before 3.11 you have to do PartialUserUpdate[Unpack[tuple[()]]] 🥴 )

[trim tangle]

what does it tell you?

[oblique urchin]

by "work" do you mean pyright supports it

[trim tangle]

lmao

haven't checked with mypy

[soft matrix]

Mypy doesn't support 646

[oblique urchin]

mypy has some initial support

[trim tangle]

well

"TypeVarTuple" is not supported by mypy yet

[void ibex]

Doesn't even get into user code, fastapi just returns a 422 when trying to put the below into that type ```
{
"is_mod": True
}

[oblique urchin]

--experimental-features I believe

[soft matrix]

Oh no

[void ibex]

using PartialUserUpdate[None] as the model it tried to push it in

[soft matrix]

Does that mean I need to implement TypeVarTuple defaults?

Pain

[oblique urchin]

you wanted them 🙂

[trim tangle]

What if you provide both fields but one is null?

oh wait @void ibex, I think in pydantic Optional does not make the field not-required?..

you'll have to configure it or add a default

[void ibex]

Yea

Optional makes it not required

it works if passed as null

[trim tangle]

Oh

hmmm

well that's cursed and weird

If you define it by hand as Optional, does it work?

[trim tangle]

Oh actually, you're breaking LSP here ™️

a function which accepts UserUpdate will be in great pain

tbh I'd just make a second class and not over-engineer

it is possible, i suppose, that the two diverge in the future

(e.g. a partial update always requires some field)

[void ibex]

Yea, that was the hope in doing this I'd just need to update in one place, since the two models will always support the same fields

[trim tangle]

well, now they support the same fields

[void ibex]

but it's only 2 for now, so likely not an issue in just duplicating

[trim tangle]

yep, wrong abstraction is much worse than duplication

and I'm not sure this is more of a duplication thing than a coincidence thing

[rose root]

Whats the conventional directory to place your type definitions? typings/ types/?

[soft matrix]

I put them in /types

But honestly I don't think it matters

[rose root]

Yea, I'm mostly leaning towards types right now because I see it's placed in a lot of projects source directory

whereas typings is outside of the source directory

which makes it harder if I, for say want to access a typeddict I guess?

Well, either way I don't think it would be too tough

[trim tangle]

What would you put in there?

[rose root]

like typeddicts, protocols some stuff like that

[trim tangle]

Do you have a special classes folder and a functions folder?

[rose root]

Nothing yet, I'm still in a designing phase

I'm just trying to get my directory sorted out

[trim tangle]

well, it will change over time 🙂

[trim tangle]

My point is @rose root: put the types in the module they belong. For example, if you have a module with some business logic like this:

# types.py
class EmployeeRepository(Protocol):
    def find_by_ids(self, ids: Iterable[str], /) -> list[Employee]:
        ...

# business/logic/hierarchy.py
def make_employee_hierarchy(repo: types.EmployeeRepository, root_id: str) -> Iterator[list[Employee]]:
    layer = [root_id]
    while layer:
        yield layer
        layer = repo.find_by_ids(set().union(*(e.subordinate_ids for e in layer)))
``` The repository is the way `make_employee_hierarchy` defines its interface, its expectations. So if it changes, `EmployeeRepository` may change.
So instead, do this:
```py
# business/logic/hierarchy.py

class EmployeeRepository(Protocol):
    def find_by_ids(self, ids: Iterable[str], /) -> list[Employee]:
        ...

def make_employee_hierarchy(repo: Employee, root_id: str) -> Iterator[list[Employee]]:
    layer = [root_id]
    while layer:
        yield layer
        layer = repo.find_by_ids(set().union(*(e.subordinate_ids for e in layer)))

This will make dependencies more clear, make the code more cohesive (things that are related, things that change together are placed together), and some smaller advantages like reducing merge conflicts (because everyone will touch the types module)

basically, the same reason you don't put all your classes into classes/ or all your functions into functions/

[rose root]

Oh yea, that makes a lot of sense. I could avoid all that confusion

Also I was recently made aware of generic typed dicts, does this mean I can make a typed dicts value generic now?

[trim tangle]

Well, it's more for this scenario, I believe:

class ErrorResponse(TypedDict):
    ok: Literal[False]
    error: str

class OkResponse(TypedDict, Generic[T]):
    ok: Literal[True]
    detail: T

ApiResponse = ErrorResponse | OkResponse[T]
``` where you want to reduce duplication

[rose root]

Makes sense, but when testing out via something like ```py
class A(TypedDict, Generic[_T]):
b: _T

c = Aint
reveal_type(c.b) # Unknown

[trim tangle]

Did you mean c["b"]?

idk why it doesn't show an error, but if you do c.b standalone it does

[rose root]

Ohhh lmao, I thought it was like NamedTuple

[trim tangle]

NamedTuples and dataclasses can already be generic

TypedDict is just a dict at runtime

[rose root]

Right, I thought I saw NamedTuple get a generic addition though?

On some GH issue

[soft matrix]

yes it is

its recent

[trim tangle]

ah icic

I don't use namedtuples a lot tbh

[rose root]

Same kek

But this is pretty neat addition

I can finally do that thing I wanted a few months ago

[soft matrix]

you have technically been able to have generic "named tuples" for a while

[rare scarab]

NT is basically interchangeable with dataclasses since typing.NamedTuple was added.

class Foo(NamedTuple):
  x: int
  y: str

print(Foo(3, "x"))

[soft matrix]

the heterogeneous iterable unpacking isnt replicate-able with dataclasses

[rare scarab]

We need a proper unpack dunder method.

Or maybe just a typed iterator

we also need dict unpacking like javascript's const {x, y} = z

closest thing we have is part of the match statement

[brisk heart]

yes please

[trim tangle]

🥴 ```py
match point:
case Point(x=x, y=y):
dist = (x2 + y2)
print(dist)

actually, if you don't care about the type, you can do this

match point:
    case object(x=x, y=y):
        dist = (x**2 + y**2)

which looks... cursed

also type checkers don't like this

[rare scarab]

does match support dict?

[trim tangle]

yes

[rare scarab]

I guess the proper question is "do dicts support match"

[trim tangle]

in fact, pyright correctly understands that x and y are ints and dist is an int:
https://pyright-playground.decorator-factory.su/?gzip=H4sIALeYPGMC_2VPMQ7DIAzceYVHSDN1jBSpU-f-IHLBUVMlBYEVwe8LpBKKeoN9Oh9nPHu7gUFGvWIIFGDZnPXcJCFujdcKD7t8eBCQEQfIvNJ0UGFohlm66jms6vBuyPoFrj0u0BgI7PNNmmUcYw9pTKqNCzzthOvEyZGUxaHUaW6WwDCCjF13hQuk3NQ5fzoHlkPziXdcA4m_DfWDfU39LXI-K7IKXz70fEsvAQAA
but for some reason it doesn't think it's exhaustive 🤔

[rare scarab]

I am curious why they chose case _: instead of else:

[trim tangle]

I don't care either way 🤷‍♂️

I suppose it keeps the grammar a tiny bit smaller

[rare scarab]

and it's not strictly typing related

[oblique urchin]

this was hotly debated when match was added

[rare scarab]

I hate how I can't connect to that decorator-factory.su domain from work.

[trim tangle]

also with else: you can accidentally get confused with indentation and end up with nonsense

[trim tangle]

because of .su? or what

[rare scarab]

probably

[trim tangle]

well that was probably a stupid choice xD

[rare scarab]

how can we have .su domains? soviet union doesn't exist anymore

[oblique urchin]

I think else was mostly rejected because it would be redundant with case _, and also because the indent level would be ambiguous

[trim tangle]

...however, type checkers can help you catch these mistakes. type checkers, yes, type hinting stuff

types -- and hinting

let's move to #internals-and-peps probably

[tranquil turtle]

I thing it is better to put your type-related code into submodule .typing of you package
There is numpy.typing, etc...

[real quail]

Is there some way that I can say that a function requires a version of a type where a specific property is not None? Something like this:

class A:
  propA: Optional[int]

def thing(arg: A): # Where A.propA must not be None
  pass

[trim tangle]

You could have a protocol

class AwithA(Protocol):
    propA: int

[real quail]

Okay, haven't heard of Protocols before but it does seem like what I'd need. Is there some way that I could have it still retain all the type information from the main class? I'd like to be able to do something like this:

class A:
  propA: Optional[int]
  propB: str

objects = listOfAs()
filteredObjects = [obj for obj in objects if obj.propA]
func(filteredObjects) # And have func know that propA is not None, but still have access to all the other properties of A

[trim tangle]

I don't think so

[real quail]

Okay, thanks. I'll just have to try and work around that then.

[dim trail]

I'm a bit puzzled by type narrowing - specifically it seems to not apply to global variables inside a class which is defined later.

def maybe() -> str | None:
    return "something"

maybe_str = maybe()

if maybe_str is None:
    raise Exception("Its None")

reveal_type(maybe_str)

class A:
    def __init__(self) -> None:
        self.thing: str = maybe_str

My expectation would be that maybe_str inside the init is narrowed to str by the exception before - but that doesn't seem to be the case in either MyPy or Pyright.

main.py:9: note: Revealed type is "builtins.str"
main.py:13: error: Incompatible types in assignment (expression has type "Optional[str]", variable has type "str")

Would anyone know a good reason why this shouldn't work? Or should I open an issue with mypy/pyright?

[wise rose]

hello everyone noob here. I"m trying to use # type: ignore but mypy is saying it is unused?

error: Unused "type: ignore" comment

here is how i'm using it

super().__init__(**kwargs)  # type: ignore [call-arg]

[dim trail]

that just means you don't need it there - the line doesn't raise a type error which could be ignored

[dim trail]

surprisingly, i can work around this by reassigning the narrowed version

def maybe() -> str | None:
    return "something"

maybe_str = maybe()

if maybe_str is None:
    raise Exception("Its None")
    
maybe_str_narrowed = maybe_str

class A:
    def __init__(self) -> None:
        self.thing: str = maybe_str_narrowed

here, maybe_str_narrowed is str even inside the class init

[trim tangle]

# type: ignore # type: ignore 🤪

interesting

I think that's how variables from the outer scope are generally treated

although that is weird

[dim trail]

its incredibly wierd - its like it forgets the narrowing ever happened unless i assign to a new name

which is actually not changing anything

[trim tangle]

Suppose that you have this

def maybe() -> str | None:
    return "something"

maybe_str = maybe()

def banana():
    global maybe_str
    maybe_str = None
    
if maybe_str is None:
    raise Exception("Its None")
    
maybe_str_narrowed = maybe_str

class A:
    def __init__(self) -> None:
        self.thing: str = maybe_str_narrowed

I think type checkers don't yet analyse when a variable is reassigned with global or nonlocal

[dim trail]

so you're saying when creating the class the typechecker doesn't know whether i called banana() elsewhere?

[trim tangle]

yep

[dim trail]

hmm, wack

[trim tangle]

Well, it could be called from another module which is not even in the codebase

[dim trail]

i see, still, i really don't like the ugly reassigning workaround

[trim tangle]

see microsoft/pyright#2225

[mighty linden]

GitHub

[pyright] #2225 value is incorrectly marked as 'Never' in a generator

[trim tangle]

oh wait, it doesn't have a reply from Eric

but there's some stuff in the linked issue

[dim trail]

aye, interesting

would you have any idea how to make the narrowing of a global variable read a bit better than with the new assignment?

[trim tangle]

maybe make another functon?

[dim trail]

yeah thats seems to be the only way

huh

[trim tangle]

like ```py
def definitely_str() -> str:
s = maybe_str()
if s is None:
ctypes.string_at(0)
assert False
return s

[dim trail]

i've clearly done to much Rust, my expectations of type checking is way too high

[trim tangle]

ikr right

[dim trail]

it'll have to do

(importing from a path is ugly with static typing anyways, so whatever)

[tranquil turtle]

Where can i read about type level Map?
Is it a proposed change or just imaginary thing?

[slender timber]

Should I guard imports by an if typing.TYPE_CHECKING: condition if its only used for type hinting purposes throughout that particular module?

[trim tangle]

I don't think so

it's a horror to read and maintain

[slender timber]

how so?

[tranquil turtle]

Pep 563 is accepted, but not enabled by default. Why?
Why this pep is not deferred?

[trim tangle]

Why?
this is a breaking change

[slender timber]

In 3.10+ its by default ig

[trim tangle]

no, not even 3.11

[slender timber]

and on 3.7+ there's from __future__ import annotations

[slender timber]

no? i remember to have read that somewhere, was it cancelled?

[tranquil turtle]

In this case you will not be able to get real annotations, because there is no such module in namespace.
Everything else seems good.

[slender timber]

I don't care about real annotations, I am not accessing them programmatically anywhere

[tranquil turtle]

(iirc) it was default in early 3.10. Then it was disabled in 3.10 and enabled in 3.11. Now it is deferred again and is not default in 3.11

[trim tangle]

1. Now imports are split into two parts:

from .foo import Bar
#...

if typing.TYPE_CHECKING:
    from .foo import Baz
``` which might be confusing if you're searching for where that damn `Bar` is used
2. If you decide to use something at runtime, it now requires changing imports, which is just unnecessary diffing (+extra merge conflicts)
3. You have to use `from __future__ import annotations` or use stringly annotations. Many people already default to that, but maybe you want the raw annotations
4. As far as the editor is concerned, it is a real import:
```py
if typing.TYPE_CHECKING:
    from .foo import Bar

def function():
    if rarely_taken_path() and path not in tests().because("yolo"):
        baz = Bar()  # pyright: LGMT!
        # blows up at runtime

5. If you need Bar in a type alias, you will have to use a string (yuck) or runtime-import it

[slender timber]

Ohh right

Bad idea

[tranquil turtle]

Id prefer to import everything i need without guard except there is a import cycle or huge slowdown because of massive import

[slender timber]

I should use it only to solve circular import issues ig

[trim tangle]

yeah TYPE_CHECKING is a bit of a hack

if TYPE_CHECKING:
    from curses import _CursesWindow as Win
else:
    Win = object

[crimson gale]

how can I solve this:

from attrs import define, field
from functools import partial

my_frozen: ??? = partial(define, frozen=True, slots=True)

attrs has type definition for define defined in pyi file and ideally I would want to reuse that

infact, frozen is defined by in attrs' own .pyi file just frozen=define

Is there someway I can reuse that definition?

[slender timber]

How can I get TypeVar's type in __init__?

Also how to get type annotations for __orig_class__?

[soft matrix]

there's nowhere currently to read about it (that I know of) in python

I'm sure there's something on TS's docs about it though

[soft matrix]

you can't

[slender timber]

ok that i figured out

saw your issue on cpython's github

[soft matrix]

yes it's annoying

I hope to one day if PEP 695 is accepted write a PEP allowing some better runtime typechecking stuff like that possible

[soft matrix]

what exactly do you mean by this?

[slender timber]

If I inherit from a generic type which has a typevar repeated do i need to repeat while i use it?

class StdEnum(ct.Adapter[int, int, ET, ET]):
    def _encode(self, obj: ET, context: Any, path: Any):
        return obj.value

    def _decode(self, obj: int, context: Any, path: Any) -> ET:
        return self.__orig_class__.__args__[2](obj)  # type: ignore

this is my class. Now due to some fuckups in the typings of this library, I cannot redefine __init__ without getting type errors

[soft matrix]

technically no but i think pep 695 encourages redefining generic parameters

oh wait no i misinterpreted what you were saying

[slender timber]

Now I can construct a instance like this:

StdEnum(something)   # no type errors, how?
StdEnum[some_enum_type](something)  # no type errors

[soft matrix]

enable strict mode in your typechecker?

[slender timber]

i have

pyright gets really confused with this library

!pypi construct-typing

[rough sluice]

construct-typing v0.5.2

Extension for the python package 'construct' that adds typing features

[slender timber]

yes this

its like I can do stuff like c.Adapter[...] and no pyright errors, but it will throw runtime errors

[soft matrix]

what is reveal_type(StdEnum[some_enum_type](something))

[slender timber]

need to check

[soft matrix]

use a conditional base depending on if TYPE_CHECKING

ie ct.Adapter[int, int, ET, ET] if TYPE_CHECKING else ct.Adapter

[slender timber]

hmm nice idea

where is reveal_type in which package?

[soft matrix]

in the eyes of a type checker its a built in

[slender timber]

huh

[soft matrix]

although there is a runtime implementation in typing and typing_extensions

[slender timber]

there is no typing.reveal_type in typing

[soft matrix]

its in 3.11 or maybe 12

[slender timber]

Ok so

import construct as c
import construct_typed as ct
from typing import TypeVar, Any
from typing_extensions import reveal_type

ET = TypeVar("ET", bound=ct.EnumBase)

class MyEnum(ct.EnumBase):
    A = 1
    B = 2

class StdEnum(ct.Adapter[int, int, ET, ET]):
    def _encode(self, obj: ET, context: Any, path: Any):
        return obj.value

    def _decode(self, obj: int, context: Any, path: Any) -> ET:
        return self.__orig_class__.__args__[2](obj)  # type: ignore

print(reveal_type(StdEnum[MyEnum](c.Int32ul)))

prints

Runtime type is 'StdEnum'
<StdEnum <FormatField>>

[trim tangle]

I don't think it's possible 🤔

if typing.TYPE_CHECKING:
    from attrs import frozen as my_frozen
else:
    my_frozen = define(frozen=True, slots=True)

[soft matrix]

i meant what does your typechecker report?

the runtime stuff doesnt matter

[slender timber]

Also i get this information?

[soft matrix]

that looks correct

[slender timber]

but it doesn't happen in my library module

[soft matrix]

whats reveal_type(StdEnum(c.Int32ul))?

[slender timber]

it happens only when i replicate in a diifferent file

[soft matrix]

have you got a py.typed file?

[slender timber]

yes

[soft matrix]

what does it look like inside of your library?

[slender timber]

ok wait hang on

reveal_type is like some magic

if it remove it, the blue line goes away

is it a good idea tho, to use generic types at runtime like this tho (in general)?

[soft matrix]

not if they cause your code to crash

[slender timber]

why will it crash my code

[slender timber]

and yea this, pyright doesn't detect __orig_class__ attribute. is there like a protocol or something for this to get type hints for it? currently i need to use # type: ignore everytime i use it

[soft matrix]

i thought you said it gave you runtime errors?

[slender timber]

nah that was due to inheriting the wrong class. turned out construct_typed had its own Adapter. hence i inherit from ct.Adapter

[soft matrix]

https://github.com/Gobot1234/steam.py/blob/main/steam/trade.py#L268 this this what i do, its not perfect

[rough sluice]

steam/trade.py line 268

__orig_class__: InventoryGenericAlias```

[soft matrix]

but "it works"

[slender timber]

hmm

why isn't this in stdlib yet

[soft matrix]

why isnt what in the stdlib?

hopefully in 3.12 ill be able to get rid of all of this garbage with TypeVar defaults

but __orig_class__ should probably be in the stubs for typeshed sometimes

but idk how it would be implemented

[brittle socket]

If a parameter can be dict or collections.Counter, is typing.Mapping the right way to type it?

[oblique urchin]

Counter is a subclass or dict, so dict covers both

however, it's generally better to use abstract classes like Mapping in type hints

especially because dict is invariant in both of its type parameters

[brittle socket]

I see

Thank you!

Mapping would also cover a list of 2-tuples I suppose?

[oblique urchin]

no, only classes that inherit from Mapping

e.g. types.MappingProxyType

[brittle socket]

def foo(Mapper):
  return Mapper(...)

Mapper = dict if pred else collections.Counter

What should be Mapper's type be here?

!e

print(type(dict))

[rough sluice]

@brittle socket :white_check_mark: Your 3.11 eval job has completed with return code 0.

<class 'type'>

[brittle socket]

def foo(Mapper: type))? 🤔

[pastel egret]

Type[MutableMapping]

type[T] is a class which is a subclass of the specified one.

[brittle socket]

https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping

Huh, I didn't know this was a thing. TIL

Why Mutable though? To signal that the mapped values can be mutable?

And where do you get Type from? typing?

[pastel egret]

Yes, since both classes are mutable. You could do Mapping if you don't intend to modify them.

Yes, though as of 3.10 regular type can be subscripted - Mypy has had some trouble with that though so you may still need the typing version.

[brittle socket]

Right. I'm on 3.10 but also in Jupyter Notebook, and it doesn't like subscripting dict, list...etc. as typehints, probably won't for type either

Thank you!

[pastel egret]

An alternative type hint which would also work is typing.Callable[[], MutableMapping] - treating the types as functions which return instances.

[brittle socket]

Oh, that's interesting. Types are callables, duh

But I think I prefer Type[x] semantically speaking

[pastel egret]

It might not work in this case, since what __init__ does isn't usually defined by ABCs - defaultdict for example takes 2 args, not 1 like dict.

[brittle socket]

Oh. Darn it, you're right

Is [] in Callable[[], ...] shorthand for Any?

Or does it just mean no args

[soft matrix]

No args

Autocorrect sucks yes

[brittle socket]

Thanks

[cedar sundial]

Is there any way to copy a function signature to another function via typing without listing all of the parameters manually + docstring?

I've tried this which I found on the github/typing:

from package import foo

F = TypeVar('F', bound=Callable[..., Any])
class copy_signature(Generic[F]):
    def __init__(self, target: F) -> None: ...
    def __call__(self, wrapped: Callable[..., Any]) -> F: ...

class Bar:
    @copy_signature(foo)
    def baz(self, *args, **kwargs): ...

This almost works but the intellisense doesnt show the the first argument and when calling the function I get a type error:
TypeError: 'NoneType' object is not callable

[tranquil turtle]

you must provide implementation for copy_signature methods

[paper salmon]

i believe ParamSpec is what you'd use for this, it was introduced as a builtin in 3.10
https://docs.python.org/3/library/typing.html#typing.ParamSpec

[fierce ridge]

is there any way to get mypy to recognize the ... as valid in a pydantic model?

class AccelerationTriple(BaseModel):
    x: Acceleration | None = ...
    y: Acceleration | None = ...
    z: Acceleration | None = ...

i have enabled the pydantic.mypy plugin but apparently it's missing coverage for a lot of pydantic features like this one

error: Incompatible types in assignment (expression has type "ellipsis", variable has type "Optional[Acceleration]"

[soft matrix]

why cant you set the value to None?

[fierce ridge]

i didnt want to allow default values. i am realizing that have refactored this enough since i first wrote it that my original concerns about default values are no longer relevant, so i gave up and just set it to None

still, this is a valid use case

i wish i knew anything about writing mypy plugins...

[pastel egret]

You could just do this:

NoDefault: Any = ...

class Whatever(BaseModel):
    attr: list = NoDefault

[fierce ridge]

how does assigning an Any to a list[Any] work?

wouldn't mypy complain?

[oblique urchin]

Any shuts up any errors

[oblique urchin]

fyi we now have https://peps.python.org/topic/typing/ listing all typing-related PEPs

[fierce ridge]

i didn't think it would work in "the other direction"

[oblique urchin]

it's compatible with everything both ways

[low escarp]

Hey all, quick question...
I have written a pytest plugin for work which mocks some methods and properties for a class.
So for example I have a class Something and I added a method receive. When I use my fixture this method is monkeypatch-ed in with pytest so that I can call receive on the instantiated class and it does some stuff (not really important what).
My question is this; is there some way of making my linter happy? It (rightfully) complains that the class Something doesn't have the receive method as it only is added by monkeypatching, not before.
I wasn't exactly sure what channel to put this in sorry (it's not technically unit testing because I want to know more about how to make the linter happy, but happy to ask in a more appropriate channel)

[twin lantern]

is there any difference with unioning with none and optional

[brisk hedge]

no

besides the runtime representation of the annotation

[trim tangle]

interestingly, Union[None, int] is exactly the same as Optional[int] at runtime, but int | None is different

[tranquil turtle]

Because Union and Optional are implemented in python in typing module, but int|None is implemented in C

Fun fact: there are two GenericAlias'es: in typing module and builtin type (types.GenericAlias iirc).

[trim tangle]

cursed moment

[brisk heart]

Not fun, completely messed up my reflection code

Especially not fun because they keep changing version to version

[trim tangle]

everything in typing seems to change from version to version

idk why they can't make it simple... like a uniform tree structure

it's just type stuff, after all

[brisk heart]

I like constant changes if it means improvement

Just please give me a changelog 😭

I suppose I could do a git blame every time I do any reflection but I really don't wanna

[soft matrix]

There's no breaking changes if you don't rely on internals

And there are ways to avoid even things like name changes

[fierce ridge]

i imagine it's because it's all still kind of work in progress

[chrome lynx]

It feels like I'm going crazy! I have this workflow with mypy which I have been using for quite a long time, but I am starting from scratch now and I can't get rid of these mypy errors:

(.venv) $ mypy library 
library/package/submodule.py:1: error: Cannot find implementation or library stub for module named "library.module"
library/package/submodule.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports

I have

/library
├── module.py
└── package
    ├── __init__.py
    └── submodule.py

In submodule.py:

from library.module import func

In module.py:

def func(x: int) -> int:
    return x

[trim tangle]

I think you're supposed to have an __init__.py in library

otherwise it's a "namespace package" which you probably don't want

[chrome lynx]

OMG you're right

I accidentally deleted it when I cleaned up everything

My hero 👍

[haughty heron]

Hey, I just wanted to ask, is this valid at all?

import typing

K = typing.TypeVar("K", bound=typing.Callable)


def do_something() -> str:
    return "something"


def handle(f: K) -> K:
    return f()


if __name__ == '__main__':
    output = handle(f=do_something)

handle accepts Callable object, and I want to annotate that it returns the output of this callable

[trim tangle]

def handle(f: K) -> K:
    return f
``` this would be valid

f() doesn't have the type of "K", it's something else.

[haughty heron]

That's what I thought

[trim tangle]

What you want is probably

T = TypeVar("T")

def handle(f: Callable[[], T]) -> T:
    return f()

[haughty heron]

Nice, thank you!

That's exactly what I want 😄

[trim tangle]

(also, since version 3.9 you need to import Callable from collections.abc instead of typing)

because some typing symbols are deprecated now

!pep 585

[rough sluice]

**PEP 585 - Type Hinting Generics In Standard Collections**

Link

Status

Accepted

Python-Version

3.9

Created

03-Mar-2019

Type

Standards Track

[rare scarab]

I wouldn't say "need"

[haughty heron]

In the current project I work on python3.8 still, but it's good to know, thanks

[trim tangle]

well, it's not very urgent

but they will be removed eventually, so I don't see the harm tbh

[rare scarab]

typing.XXX probably isn't deprecated for removal.

eventually meaning undecided date

[trim tangle]

Importing those from typing is deprecated. Due to PEP 563 and the intention to minimize the runtime impact of typing, this deprecation will not generate DeprecationWarnings. Instead, type checkers may warn about such deprecated usage when the target version of the checked program is signalled to be Python 3.9 or newer. It’s recommended to allow for those warnings to be silenced on a project-wide basis.

The deprecated functionality may eventually be removed from the typing module. Removal will occur no sooner than Python 3.9’s end of life, scheduled for October 2025.

I guess, yeah

it's honestly a bit of a pain

especially given that the only way to discover this deprecation is to read the PEP which... most people probably don't do in their free time

[oblique urchin]

not if I can help it

[rare scarab]

I'll switch over when pyright decides it needs to warn me

[trim tangle]

and actually removing those will break a lot of otherwise perfectly fine code that worked since python 3.6 or whatever

[oblique urchin]

(for context I'm one of the maintainers of the typing module)

[rare scarab]

I thought this was talking about removing typing.List instead of collections.abc.List

[trim tangle]

yeah

[oblique urchin]

I agree, actually removing them will break a lot of perfectly fine code for little gain

[trim tangle]

wait

[rare scarab]

then how would it affect 2.7 if the typings module was added in 3.6?

[trim tangle]

right yeah it was a brain fart

[oblique urchin]

it's about typing.List in favor of list, typing.Iterator in favor of collections.abc.Iterator, etc.

[trim tangle]

yeah

[rare scarab]

yeah.

[oblique urchin]

there is a typing backport for 2.7 (no longer maintained)

[rare scarab]

assuming it's imported behind if False:

[trim tangle]

replaced 2.7 with 3.5 which is also quite ancient nowadays

[oblique urchin]

typing was in the stdlib in 3.5 🙂

you need 3.4

[trim tangle]

well I meant, maybe there's some library which was made in 3.5 times and still works on 3.12

but it uses typing.List or whatever

[oblique urchin]

true

[trim tangle]

oh and let's combine it with from __future__ import annotations becoming the default eventually ™️

[oblique urchin]

that's unlikely to happen

[rare scarab]

I've seen the opposite where a library would work perfectly fine on 3.7, but it uses collections.abc.Mapping as a generic.

[trim tangle]

that is also very true

[trim tangle]

so what's the state of that? i haven't been following

will we have a __future__ flag that's not actually corresponding to something in the future? 🙂

I guess it's more of a compiler directive

[oblique urchin]

we'll likely end up with a version of PEP 649

!pep 649

[rough sluice]

**PEP 649 - Deferred Evaluation Of Annotations Using Descriptors**

Link

Status

Draft

Created

11-Jan-2021

Type

Standards Track

[trim tangle]

ahh ic

yeah that sounds much better

the string thing was mildly cursed IMO

[rare scarab]

Combine that with deferred imports

[trim tangle]

because you need eval and such to get the previous behaviour

are there any problems with PEP 649?

ah, circular imports

well, don't have those!

[rare scarab]

You don't have to worry about circular imports if you put everything in one file.

[trim tangle]

[trim tangle]

not if you import the module from itself

[restive warren]

Not sure if this question fits the channel, but I'm having trouble with mypy's stubtest on custom typesheds. When running, it is unable to find any sources, causing everything to appear as missing. I've specified the --custom-typeshed-dir argument and it didn't complain so I assume I did it correctly. I can also reproduce the same result if I remove ignore_missing_stub from typeshed's test suite, which I believe to be incorrectly suppressing the issue

If I trace the error within stubtest's code, it gets to find_module_cache.find_modules_recursive, which always returns an empty list. I'm unable to jump into the source code of find_modules_recursive (or the init of that class for that matter), or get any breakpoints to trip. I assume there is some funky dynamic manipulation to get the cache stuff to work that returns a cached result instead of executing the function

Not sure if this is a mypy bug, typeshed bug, or a me bug

[oblique urchin]

not sure about the full answer (pinging @hallow flint if he's around), but your debugger issues are likely because mypy is compiled by mypyc

debugging may be easier if you install the pure-python version of mypy (pip install --no-binary mypy)

[restive warren]

One other thing to note is that stubtest runs properly if I install the typeshed package for this module with pip, at which point it produces different (correct) errors

[restive warren]

That's a good lead, I'll try it now

Thanks

[oblique urchin]

hm are you using --custom-typeshed-dir and testing third-party (non-stdlib packages)?

not 100% sure but I think that flag may only affect the stdlib now

[restive warren]

Yes

[restive warren]

Hmm, interesting, I'm still following the directions in typeshed's readme and the test script which does the same thing

The requirements pins mypy to 0.982, so I assume it should be fine?

Btw, debugging is now working, thank you

[restive warren]

When searching the paths, modulefinder specifically looks for <package>-stubs, which will match the name of the package installed from pypi, but not the name of the local package

That part hasn't changed in 4 years, so I assume it's fine, but I'm wondering if I was supposed to install the local typeshed typings as a package before running this? Didn't see anything in the instructions indicating so

[restive warren]

Turns out this might be a me problem after all. Running through the test script (which uses a different command from the one listed in the instructions), and manually removing the ignore seems to produce the correct errors. I thought I had tried that, so maybe I just didn't notice it working when trying it

The script sets MYPYPATH which seems to get stubtest to behave as desired

[median tangle]

this is so annoying, is there a typed version of functools.cache somewhere that actually preserves the argument types?

since this only seems to preserve the return type

[oblique urchin]

we can probably fix this in typeshed if we resurrect a version of https://github.com/python/typeshed/pull/7771

[soft matrix]

not yet

[median tangle]

that would be nice

guess I'm stuck with this for now ```py
def _typed_cache(func: Callable[P, R]) -> Callable[P, R]:
return cast(Callable[P, R], cache(func))

[soft matrix]

still confuses me that param spec didnt actually address this rather large issue

[oblique urchin]

seems like the difficulty is in supporting it on methods

[soft matrix]

yeah

ideally id like to see the day that FunctionType is generic in the same way as Callable

but it "just works" inside of classes

[oblique urchin]

not sure that would help here since methods are still FunctionTypes

[soft matrix]

yeah its a similar-ish issue though

thoughts on having Unpack[dict[str, Any]] -> **kwargs in a callable without the need for a typed dict?

[oblique urchin]

pyanalyze supports that. I haven't pushed for putting it in PEP 692 because I don't see much point

[soft matrix]

very nice

[slender timber]

Is there anything like an "optional" type parameter to a generic class?

[brisk hedge]

not yet

https://peps.python.org/pep-0696/

[slender timber]

😦

[soft matrix]

soon™️

hopefully

[slender timber]

It says this pattern is common in existing libraries, how?

Its not even implemented yet

[brazen jolt]

will this make it to typing_extensions any time soon?

[soft matrix]

im saying it would be useful because currently these typevars go as unknowns

[soft matrix]

its in typing extensions as of yesterday (4.4.0)

[brazen jolt]

👀

nice

[slender timber]

Ooooo greatt 💥

[brisk hedge]

though not in type checkers I believe (?)

[slender timber]

I can use this for passing type param of TypedDict used in ctor with Unpack

[soft matrix]

i have a 99% functional mypy fork with it

[slender timber]

Yea ryt 😦

[soft matrix]

https://github.com/Gobot1234/mypy/tree/TypeVar-defaults

[brisk hedge]

what's missing?

[soft matrix]

support for respecialising type aliases, typevar tuple defaults and tests

[slender timber]

How does mypy test itself with newer features btw

Because the older versions will definitely error when it is given something new

[soft matrix]

yeah and it sees the errors go away with the newer features

[next apex]

Hi

Someone could help me

I create telegram bot using telebot (Telegram API), and i need to call url each time(url needed to be refresh after calling), but foretunately it doesn't change random photo

bot = telebot.TeleBot(config.TOKEN)

@bot.message_handler(commands=['start'])
def start(message):
    #keybord
    markup = types.ReplyKeyboardMarkup(resize_keyboard=True)
    button1 = types.KeyboardButton("♬ download music")
    button2 = types.KeyboardButton("🖼 get image")
    markup.add(button1, button2)

    bot.send_message(message.chat.id, "Choose your request:", parse_mode='html', reply_markup=markup)

    
@bot.message_handler(content_types = ['text'])
def excecution(message):
    if message.chat.type == 'private':
        if message.text == 'Hi' or 'Hello':
            bot.send_message(message.chat.id, "Hi! {0.first_name}, let's begin.\nSend me command /start".format(message.from_user))
        if message.text == "♬ download music":
            bot.send_message(message.chat.id,"I can't do this now")
        elif message.text == "🖼 get image":
            photo_image = 'https://picsum.photos/200/300'
            bot.send_photo(message.chat.id,(photo_image), parse_mode='html', timeout=3)



    #bot.send_message(message.chat.id, "I don't understand.\nPlease, use buttons.".format(message.from_user))

#RUN
bot.polling(none_stop=True)

[trim tangle]

Is this related to type hints in any way?

If not, please see #❓｜how-to-get-help and claim a help channel

[restive warren]

I have a system which looks like:

@dataclass
class BaseModel:
    prop_1: int


Model = typing.TypeVar("Model", bound=BaseModel)


class Base(typing.Generic[Model]):
    state: dict[int, Model]

    def __init__(self):
        self.state = {}

    def func(self) -> None:
        self.state[1] = BaseModel(prop_1=100)

The idea being subclasses of Base can define their own func, or leave it to the default. funcs of subclasses which use the subclass' Model will be type hinted correctly, but how could I type hint the base class?

With the above, mypy will complain about the assignment on the last line:

Incompatible types in assignment (expression has type "BaseModel", target has type "Model")

Or perhaps this needs to be redesigned in some other way

(the behavior itself is correct to the desired spec, just the typing issues)

[soft matrix]

youre going to run into variance issues here if you want this to be generic and read write

does this actually need to be generic especially if it just adds BaseModel to the state independent of whether its generic over BaseModel?

[restive warren]

So, the idea here is that if a subclass is defining it's own model, it also has to redefine func

[soft matrix]

ah

[restive warren]

This does make it kind of fragile, and if you miss defining func it'll cause errors

But I'm not sure how to best design this

[soft matrix]

could make it an abstractmethod

[restive warren]

That's true, though if a subclass uses the same model, they don't need to redefine it

I guess that's the lesser of two evils

[soft matrix]

they can still call super().func() if they want the same behaviour

but i think if you have an abc/protocol version of this then you can subclass the abc and implement func where Model is constrained to BaseModel

@typing.runtime_checkable
class ProtoBase(typing.Protocol[Model]):
    state: dict[int, Model]

    def __init__(self):
        self.state = {}

    @abc.abstractmethod
    def func(self) -> None:
        raise NotImplementedError()

class Base(ProtoBase[BaseModel]):
    def func(self) -> None:
        self.state[1] = BaseModel(prop_1=100)

something like that

[restive warren]

I see, thank you

I'll have to look into protocols, I'm not familiar

[slender timber]

is there a way to use typevar with a typeddict

[soft matrix]

Yes

Use typing_extensions.TypedDict and generic

[slender timber]

That doesn't seem to work

It says TypedDict is a type alias

This is my code

TD = TypeVar("TD", bound=TypedDict)

class A(Generic[TD]):
    def __init__(self, **kw: Unpack[TD]):
        ...

[soft matrix]

oh wait you meant as a bound

you just need to py class A: def __init__(self, **kw: Unpack[TypedDictSubclass]): ...

[rustic gull]

Hi all, I've got the following boiled-down scenario:

from typing import TypeVar, Generic

A = TypeVar("A")
B = TypeVar("B")

class Base(Generic[A, B]):
    valA: A
    valB: B

class intAType(Base[int, B], Generic[B]): ...
class strBType(Base[A, str], Generic[A]): ...

class Both(strBType, intAType): ...

x = Both()
reveal_type(x.valA) # Type of "x.valA" is "Unknown"
reveal_type(x.valB) # Type of "x.valB" is "str"

Base is generic in A and B.
intAType is a Base, where A is int.
strBType is a Base, where B is str.
Both is both intAType and strBType.

I was hoping that pyright would be able to decipher that Both.valA is int and Both.valB is str, but it's only able to tell the type of Both.valB. If I swap the inheritance order on Both to be intAType, strBType, then only valA is inferred correctly.

Does anyone know a way that I can achieve this generic type inference? Is it even possible?

[trim tangle]

class Both(strBType[int], intAType[str]): ...

although that sounds like a really strange construct

could you show some context maybe?

[tranquil turtle]

I once coded something similar, but with no specialisation in Base subclasses. In this case (if you put typevars instead of int/str) it works, iirc

[slender timber]

No the entire point I make A a generic class, is that otherwise I need to make a "fake" constructor with an annotation for the TypedDictSubClass

And idk if the implementation of typeddict + unpack is flaky yet, but pyright doesn't detect the type of kw like this as well.

class A:
    def __init__(self, **kw: Any):
        self._kw = kw

class B(A):
    def __init__(self, **kw: Unpack[TypedDictSubClass]):
        super().__init__(**kw)

b = B()
reveal_type(b._kw)   # Any!!!!!

@soft matrix

[soft matrix]

It's not flaky this won't ever be supported

It would need to be generic for this to work

But you're better just not having a base class here

[slender timber]

That's not possible unfortunately

[slender timber]

You mean **kw: Unpack[TypedDictVar]?

[soft matrix]

No

Well maybe

[slender timber]

@runtime_checkable
class ModelCollection(Iterable[MT_co], Protocol[MT_co]):
    @overload
    def __getitem__(self, i: int | str) -> MT_co:
        ...

    @overload
    def __getitem__(self, i: slice) -> Sequence[MT_co]:
        ...

def getslice(func: Callable[[ModelCollection[MT_co], str | int | slice], MT_co]):
    """Wraps a :meth:`ModelCollection.__getitem__` to return a sequence if required."""

    @functools.wraps(func)
    def wrapper(self: ModelCollection[MT_co], i: str | int | slice):
        if isinstance(i, slice):
            return [
                model
                for model in self
                if getattr(model, "__index__")() in range(i.start, i.stop)
            ]
        return func(self, i)

    return wrapper

class ChannelRack(MultiEventModel, ModelCollection[Channel]):
    @getslice
    def __getitem__(self, i: str | int | slice):
        ...

The line at getslice raises the error:

Argument of type "(self: Self@ChannelRack, i: str | int | slice) -> Channel" cannot be assigned to parameter "func" of type "(ModelCollection[MT_co@getslice], str | int | slice) -> MT_co@getslice" in function "getslice"
  Type "(self: Self@ChannelRack, i: str | int | slice) -> Channel" cannot be assigned to type "(ModelCollection[MT_co@getslice], str | int | slice) -> MT_co@getslice"
    Parameter 1: type "ModelCollection[MT_co@getslice]" cannot be assigned to type "Self@ChannelRack"
      "ModelCollection[Channel]" is incompatible with "ChannelRack"

Decorators are tough to type hint 😦 and if I just use Any for them, they "erase" the type annotations of the functions or classes they are used on

ok nvm

fixed it by putting some extra overloads and Anys

[maiden mesa]

What would be the equivalent of this in Python?
(Typescript):

type URLType = "https://xyz.com" | "https://xyz1.com" | "https://xyz2.com"

const showUrl = (url: URLType) => {
    return url
}

basically, the union type like this defined separately.

I know I can do this:

def (url: "https://xyz.com" | "https://xyz1.com" | "https://xyz2.com"):
    print(url)

but i feel like for longer union types or more complex types like this, defining everything the function param is not very much readable and not reusable

[green gale]

you can assign types to a name the same way as a variable

also

!d typing.Literal

[rough sluice]

typing.Literal

typing.Literal```
A type that can be used to indicate to type checkers that the corresponding variable or function parameter has a value equivalent to the provided literal (or one of several literals). For example:

```py
def validate_simple(data: Any) -> Literal[True]:  # always returns True
    ...

MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
    ...

open_helper('/some/path', 'r')  # Passes type check
open_helper('/other/path', 'typo')  # Error in type checker
```...

[maiden mesa]

Oh, damn didn't know. Gotcha, thanks. And yea I should use Literal here

[pastel egret]

Optionally you can also annotate that variable as a typing.TypeAlias, to make it explicit it's a type hint and not a regular variable - then checkers can comfortably warn about errors there.

[trim tangle]

consider making an Enum instead of literals IMO

Especially if your value is like... a full-blown url

[winter elm]

is there a known way to make a type for a Sequence of up to n things ? 6 strs in my case

My current solution is this monstruosity:

up_to_six_strs_type = tuple[str] | tuple[str, str] | ... | tuple[str, str, str, str, str, str]

[brisk hedge]

No, I'm afraid not

[winter elm]

the alternative of sending a tuple of n optional things sucks, consumers of that type have to filter the optional items

[brisk hedge]

you can overload getitem for literal ints 0-5 if you're desperate, or making a whole load of tuples, but there's no native integer generics in the type system

[winter elm]

not yet at least! Ok I see, thanks

[winter elm]

you mean overload getitem for a custom tuple / seq class ? hah didn't yet think of that, thought I could keep it inside the typing hint system. interesting, yeah!

[brisk hedge]

It wouldn't allow for n-length unpacking or using non-literals as indices but it might work in a pinch

[trim tangle]

Why do you want this though?

Some context would help

[winter elm]

I have a task where we encode parts of many kinds into a part code. each kind can have up to 6 attributes used for the encoding

so those up to 6 values are sent around between functions. I wanted to add type hinting to these encoding functions

[brazen jolt]

Could you maybe provide some example for this?

[winter elm]

it's a cursed part number encoding task
we have a datamodel for metallic parts of many kinds, like pipes, cylinders, square profile bars and so on
those each have their significant geometrical properties (round things have diameter, inner and outer if empty - square things have side length, rect profile things have 2 side legths. plus inner if empty), length, metallurgical properties like steel grade
up to 6 are significant for the part number encoding

[brazen jolt]

So, these are represented by some classes?

[winter elm]

yes, sure

[brazen jolt]

Probably just make a class like Model, that all of these inherit and type-hint the inputs as models

[winter elm]

I already have a PartNumber class, with the values of the up to 6 attributes

[brazen jolt]

What do you mean up to 6 attributes

[winter elm]

but I wanted to see if I could do it with a more basic type

[brazen jolt]

Are some of these attributes not required?

[winter elm]

yes, again, the part number encoding function uses up to 6 attributes, depending on the part type

[brisk hedge]

can "something with 5 attributes" mean many different things?

[winter elm]

sure, dfiferent part types can all have 5 attributes

a round and square steel bar are different part types, but they'll have the same number of significant attributes for the part encoding task

[brazen jolt]

So you're trying to typehint the fact that these 6 variables, if present, will have certain specific types?

[winter elm]

oh no

[brisk hedge]

up to 6 from a mixed bag?

[winter elm]

I want to typehint that there are from 1 to 6 of them

yeah

[brisk hedge]

yeah honestly you'd be better off with a runtime assertion

maybe with an optional TypedDict for the input data depending on how you get it 😄

[brazen jolt]

How are these attributes stored? They're different model classes and these are literal attributes? Or do you have like a list of length up to 6 for them?

[winter elm]

sure, I wanted to try using the typing system

[brazen jolt]

Yeah a typed dict sounds like something you could utilize

[winter elm]

yeah there's a database with different model classes etc

[winter elm]

each encodable part type has a part specific encoder function that takes the significant attribute values, does attribute encoding, and returns this list of up to 6 attribute values that I'd like a type hint for

[brazen jolt]

class MaterialAttributes(TypedDict, total=False):
    roughness: int
    ...

with something like this, you could then store the attributes in a simple dictionary, like attributes = {"roughness": 5, "rigidity": 10, ...} and mark that attribute as MaterialAttributes

the total=False means that not all of the listed attributes need to be present, but if they are present, they will be of specified type

what you could then do is something like: ```py
def from_part_number(part_no: str) -> MaterialAttributes:
...

[brisk hedge]

I say the current state is as far as you can go with your current part system
and that's fine tbh

[brazen jolt]

I'm still not entirely sure that this is what you wanted though, so correct me if you wanted something different

[winter elm]

ah wait so if I get this right the list of attributes of this MaterialAttributes thing is the union of the list of attributes of all different part types ? yeah that could be hundreds, not up to 6

misunderstood your earlier question

[winter elm]

yeah I think so too

[brazen jolt]

it's not necessarily a union of attributes, it's just showing that the returned type will be a dictionary, which will have certain types for keys with specific names

but yeah for hundreds of possible attributes, it might just be easier to avoid this completely

[brisk hedge]

sometimes you just shrug it off, use the less precise type hint and prioritize other parts of your code 😄

[brazen jolt]

if you just wanted to mark that there will be up to 6 values returned, you could do that with a union of differently sized tuples, but it's probably not worth it

UpTo6MaterialsType: TypeAlias = tuple[str] | tuple[str, str] | tuple[str, str, str] | ...

def get_material_types(...) -> UpTo6MaterialsType:
    ...

[winter elm]

yeah, I do that now

felt dirty, hence my question

[brazen jolt]

depending on how your function works, it might make sense to split it somehow

[winter elm]

see here

[brazen jolt]

or if you don't really care that it's always at most 6, you can just do tuple[str, ...] to mark that it's just a tuple containing any amount of strings

[winter elm]

yeah that's possible actually

[brazen jolt]

we usually don't really care about the length of a sequence in these cases when it comes to typing, so this is probably a better solution and it's much more convenient to type out than having a union like that

[median tangle]

Is there some pre-made protocol class like SupportsEq (i.e. object that defines __eq__) or do I need to make one myself?

[brisk hedge]

object

[median tangle]

oh, didn't know objects support it by default, interesting, I'd like something that's generic though, i.e. SupportsEq[MyType] showing that it supports comparisons against MyType

[brisk hedge]

Yeah that would have to be custom

though I'm not 100% sure it's fine under lsp

[median tangle]

hm? why would it not be?

[brisk hedge]

Never mind, works fine

[pastel egret]

Generally you shouldn't need this, since equality comparisons between unrelated objects should fall back to always returning False. The ordering methods aren't present on object though, so it makes sense there.
The _typeshed module does have protocols, but using that outside stubs is a bit awkward:
https://github.com/python/typeshed/blob/master/stdlib/_typeshed/__init__.pyi#L53-L70

[trim tangle]

Sigh. Converting from { "foo": Hello[int], "bar": Hello[str] } to World[{ foo: int, bar: str }] and such not being possible is a pain

I am spoiled by TypeScript!

Sometimes you can sorta do it with chained calls and TypeVarTuple but it's kinda scuffed

[brisk hedge]

Of the sort t[u[a]] -> u[t[a]]?

[trim tangle]

I think it's like the Map idea

But for dicts

Although Map would already be good

The use case is basically for when you want to parse a nested structure, so you define a parser declaratively using a dict mapping keys to other parsers

Or stuff like SQLAlchemy where you define tables with a dict

[median tangle]

hm, am I missing something? Why does pyright report an issue here?

T = TypeVar("T")
T_contra = TypeVar("T_contra", contravariant=True)
U_contra = TypeVar("U_contra", contravariant=True)


class SupportsEq(Protocol[T_contra, U_contra]):
    def __eq__(self: T_contra, __o: U_contra) -> bool:
        ...


def foo(x: T, a: SupportsEq[object, T]) -> None:
    ...


foo("hi", "hi")  # Argument of type Literal["hi"] cannot be assigned to parameter "a" of type SupportsEq[object, T@foo]

[trim tangle]

Anyone knows when mypy will get PEP646 support?

[oblique urchin]

there's work in progress on it, there's some limited support

no definite timeline though

[rustic gull]

sorry, didn't get to this yesterday - yeah it'll have to be that. It makes sense, of course the generic type parameters for two classes have to be completed unrelated to one another, even if they have the same name

The context needs quite a bit of an explanation - I have a base class and decorators that let me define properties/methods as discord embed attributes. Currently only the colour of the embed and fields that should appear:

class User(EmbedFillableMixin):
    @embedField(showLast=True)
    @property
    def id(self): return str(self._id)

    @embedColour
    def embedColour(self): return Colour.blue()

EmbedFillableMixin adds a _embedAttributes: List[_BaseEmbedAttribute] class attribute onto the class:

class _BaseEmbedAttribute(Generic[TUnderlyingMethod, TMethodOutput]):

class _PropertyEmbedAttribute(_BaseEmbedAttribute[property, TMethodOutput], Generic[TMethodOutput]): ...

class _EmbedColourAttribute(_BaseEmbedAttribute[TUnderlyingMethod, Colour], Generic[TUnderlyingMethod]): ...

class _PropertyEmbedColourAttribute(_EmbedColourAttribute, _PropertyEmbedAttribute): ...

Here _PropertyEmbedColourAttribute is going to have to be _PropertyEmbedColourAttribute(_EmbedColourAttribute[property], _PropertyEmbedAttribute[Colour]):
I'm not sure why I'm spamming the channel with all this when I understand why it has to be the case, but here we are

[slender timber]

Are overloads not inherited?

[brisk heart]

Not when the function is overwritten

[slender timber]

hmm, why is that so

[brisk heart]

I mean, not even the signature is kept if it's overwritten

So why would overloads

not saying it shouldn't, just that it wouldn't at all be consistent

[fierce ridge]

i really wish it was kept by default

[brisk heart]

feel like that makes adding new parameters hard

how would you determine when def f(*args, **kwargs) is saying to keep the signature and when it actually does want to use *args and **kwargs

best I can think of is adding a decorator that says the function should copy the signature of its parent

[fierce ridge]

yes, or even just overriding a function that modifies behavior but doesn't change any parameters

it's incredibly wasteful and makes it's really hard to "correctly" override methods with complicated type signatures

that or maybe i'm just doing something wrong

[brisk hedge]

Sometimes I want to mimic typescript's function foo( { /*...*/ }: FooProps) and use def foo(**kwargs: Unpack[FooProps]) 😄

but then I remember TypedDict intersections (or structural type intersections in general) aren't really a clean thing either

[shell robin]

Hi, I am trying to annotate a callable that could have different parameters types.

I've tried something like this:

from typing import Callable, Type

def my_func_1 (var_a: str, exception: Exception)
def my_func_2 (var_a: str, exception: SomeCustomException)
def my_func_3 (var_a: str, exception: AnotherCustomException)


def my_func_4(fn: Callable[[str, Type[Exception]]) # this can be my_func_1, my_func_2 or my_func_3

Any resource or guide on how this can be properly implemented?

[brisk heart]

it's Callable[[arg1, arg2, ...], return] so fn: Callable[[str, Exception], object] if you don't care about the return type

ah sorry kind of a late reply

[shell robin]

Thank you for your reply 🙂 . If I use Callable[[str, Exception], int] mypy still complains since my_func_2 and my_func_3 have a different second argument type than Exception

[brisk heart]

ah I see, that seems to be an issue with covariance, you want it to be contravariant

Although I genuinely have no idea how to make anything other than a typevar be covariant/contravariant

[shell robin]

could you elaborate a little bit on typevar?

[soft matrix]

It's the other way round isn't it?

[brisk heart]

ah damn

I mess this up every time

wait no, wouldn't this get close to working?

ExceptionT = TypeVar("ExceptionT", bound=Exception, contravariant=True)
def func(fn: Callable[[str, ExceptionT], int]) -> object: ...

though I have no idea how to make this not use a typevar

[oblique urchin]

contravariant=True on a TypeVar is meaningless unless it's used as a class parameter

[brisk heart]

makes sense

[oblique urchin]

but with just the bound this might do approximately what you want

[brisk heart]

yeah but then mypy would complain it's used only once right?

[trim tangle]

Can you show an example of how you'd use that?

Suppose that you have this setup:

class Animal: ...
class Cat(Animal): ...
class Dog(Animal): ...

def for_each_animal(animal: Callable[[Animal], None]) -> None:
    ...

Do you think for_each_animal should accept Callable[[Dog], None]? Should it accept Callable[[Cat], None], Callable[[Animal], None], Callable[[object], None]?

[brisk heart]

pretty sure I've done similar things when making error handlers and such but I forgot how I managed to make it work 🤔

[trim tangle]

Essentially, the problem is: how does my_func_4 know what exception fn expects?

[dire bobcat]

Does the return of __init__ have to be typed?

[trim tangle]

Yeah ideally you should do __init__(...) -> None

[brisk heart]

my gosh you're right it just hit me, damn

[trim tangle]

otherwise mypy by default will treat it as "untyped" and just not bother checking it

[dire bobcat]

Okay, it just seems unnecessary

[shell robin]

This is a more complete snippet of my code

class SomeCustomException(Exception):
    pass


class AnotherCustomException(SomeCustomException):
    pass

from typing import Callable, Type

def my_func_1 (var_a: str, exception: Exception)
def my_func_2 (var_a: str, exception: SomeCustomException)
def my_func_3 (var_a: str, exception: AnotherCustomException)


def my_func_4(fn: Callable[[str, Type[Exception]]) # this can be my_func_1, my_func_2 or my_func_3

my_func_4(fn=my_func_1)
my_func_4(fn=my_func_2)
my_func_4(fn=my_func_3)

[trim tangle]

But how does my_func_4 know what kind of exception to pass to var_a?

[shell robin]

my_func_4 is called multiple times on an web app startup

[trim tangle]

what you show is basically:

def f1(e: ValueError) -> None: ...
def f2(e: RuntimeError) -> None: ...
def f3(e: IndexError) -> None: ...

def my_func_4(fn: "f1 or f2 or f3") -> None: ...
``` I don't see how this makes sense tbh -- how does `my_func_4` know what exception to use on a given function? I think your example is incomplete

[shell robin]

something like this

from my_namespace import my_func_1, my_func_2, my_func_3

def my_func_4(fn: Callable[[str, Type[Exception]]) # this can be my_func_1, my_func_2 or my_func_3

def init_app()
    my_func_4(fn=my_func_1)
    my_func_4(fn=my_func_2)
    my_func_4(fn=my_func_3)

[trim tangle]

Can you show the actual code in my_func_4?

[trim tangle]

You could set up mypy to check "untyped" definitions like this one.

The reason it is this way: Python's typing system is gradual. Meaning, if you have 2 million lines of Python, you can't just add types to everything in one sitting. You would add types gradually (hence -- "gradual typing"). It would be a real pain if mypy checked those 2 million lines of Python written in 2022 B.C. without any considerations for type checkers, it would just be millions of errors. So by default mypy only checks functions with type annotations: ```py
def f(x: int) -> str: # checked
...

def untyped(whatever, **have_fun): # not checked
...
so if mypy sees something likepy
def init(self):
self.foo = 1
``` it thinks it's an untyped function. You can remove this behaviour with the --check-untyped-defs flag

[shell robin]

This is the actual code, app is a FastAPI instance

def my_func_4(self, fn: Callable[[str, Type[Exception]]) # this can be my_func_1, my_func_2 or my_func_3
    sig = signature(fn)
    self.app.add_exception_handler(sig.parameters["exception"].annotation, fn)

[trim tangle]

Oh, so you're inspecting the annotation. That's an important bit

Why do you need my_func_4 though? Just do ```py
self.app.add_exception_handler(SomeCustomException, my_func_2)

[shell robin]

I want to separate the exception handlers adding from the startup process, so I have multiple functions each doing something specific for building the app

[trim tangle]

hm?

I don't quite get you

it sounds like we're missing some important bits of context here, maybe you could explain more about what you're doing, with some code?

[shell robin]

sure, I will give more context, just a moment

[shell robin]

This is a closer example of what I am trying to do, there is an AppBuilder class which instantiates a FastAPI app and configure it. The app is configured using multiple methods that proxy FastAPI methods for configure mutiple settings of the FastAPi app, the functions my_func_1, my_func_2, my_func_3 are being imported as I stated before. Hopefully this is clearer.

class AppBuilder:

    def __init__(self, config) -> None:
    self.settings = config
    self.app: Union[FastAPI, None] = None

    def _include_routers(self) -> None:
        self.app.include_router(api_router_v1)
        self.app.include_router(api_router_v2)
        self.app.include_router(api_router_internals)

    def _include_middlewares(self) -> None:
        self.app.add_middleware(CORSMiddleware)
        self.app.add_middleware(CustomWebSecMiddleware)
        self.app.add_middleware(AnotherMiddleware)
        
    def _setup_logging(self) -> None:
        pass

    def _include_exception_handlers(self) -> None:
        self._register_exception_handler(handler=my_func_1) # mypy is complaining here
        self._register_exception_handler(handler=my_func_2) # mypy is complaining here
        self._register_exception_handler(handler=my_func_3) # mypy is complaining here

    def _register_exception_handler(
        self, handler: Callable[[Request, Exception], Coroutine[Any, Any, JSONResponse]]
    ) -> None:
        sig = signature(handler)
        self.app.add_exception_handler(sig.parameters["exception"].annotation, handler)

    def build(self) -> FastAPI:
        self.app = FastAPI()

        self._include_exception_handlers()
        self._include_middlewares()
        self._include_routers()
        self._setup_logging()

        return self.app

[trim tangle]

so why can't you do self.app.add_exception_handler(...) in _include_exception_handlers?

Inspecting the signature is generally pretty brittle and should be used as a last resort.

1. If you attach a decorator to my_func_N, it will likely break
2. If you use from __future__ import annotations where my_func_N is defined, it will likely break
3. If my_func_N accepts a Union or something more complicated (even Any), it will definitely break

4. You can't use a general exception handler for a more specific exception. For example, you might want to handle CatException and DogException with a handler of AnimalException, but use a separate handler for OwlException.

[shell robin]

I am inspecting the handler functions to get the exception type so thought it was cleaner just have a different method and not inspecting each handler inside _include_exception_handlers. But based on your point I think it would be simpler and clearer just use _include_exception_handlers passing the actual exception type and the corresponding handler

It is still confusing to me - regardless of my problem - if there is a way to annotate callables with different arguments types, as I was trying initially

[oblique urchin]

unions work for that, but it's hard to come up with examples where this produces useful, type-safe code

[trim tangle]

To answer your question, the reason why mypy is complaining is because functions are "contravariant" in their parameter types (https://decorator-factory.github.io/typing-tips/tutorials/generics/variance/)
So, for example, a fn: Callable[[Exception], None] will be accepted where Callable[[ValueError], None] is expected.fn can accept any Exception, so it can definitely accept a ValueError.

However, a fn: Callable[[ValueError], None] will not be accepted where thing: Callable[[Exception], None] is expected. fn only knows how to work with ValueErrors, but the code accepting the thing might do thing(Exception()) or thing(IndexError()).

generally, you can't "know" what particular type the passed in callable expects

I think the general pattern is to pass some key of what the function can handle together with the function. e.g.:

class EventKey(Generic[T]): ...

E_RESIZE = EventKey[tuple[int, int]]

class Widget:
    ...

    def add_event_listener(self, key: EventKey[T], handler: Callable[[T], None]) -> None:
        ...

with exceptions you can just use the type ```py
E = TypeVar("E", bound=BaseException)

def on_exception(exc_type: type[E], handle: Callable[[E], Response]) -> None:
...

[shell robin]

with bound=BaseException the checker will infer any subclass of BaseException, is this correct?

if that is correct what is the unbounded (no bound argument) behaviour for TypeVar ?

[trim tangle]

It will prohibit doing stuff like on_exception(int, print)

so that you can rely on the fact that E is a subtype of Exception in the function

I think you could say that the unbounded behaviour is equivalent to bound=object but @oblique urchin might correct me

[oblique urchin]

I think that's right

since every type is a subtype of object

[shell robin]

I see, now it is clearer for me the usage of TypeVar . I think gotta need to read the typing hints section in python docs

[trim tangle]

Yeah there's quite a lot of material... it's a bit all over the place tbh.

• docs.python.org
• documentation for mypy: https://mypy.readthedocs.io/en/stable/
• the actual reference is scattered across a lot of PEPs
• my attempt at some useful articles @ https://decorator-factory.github.io/typing-tips/, includes a section on type vars

[soft matrix]

dont forget https://typing.readthedocs.io

[trim tangle]

oh yes...

[soft matrix]

;)

[trim tangle]

It's not all over the place, it uses the new "distributed documentation" architectural pattern 🤪

[shell robin]

lol

seriously guys thanks all of you for your patience, I learnt a lot from you and sorry for my poor english

[trim tangle]

that's what we're here for 🙂

I think your english is fine. Most people here are not native English speakers

[shell robin]

it is motivating for me and google translate to read that, anyway thanks!

[safe lance]

hi... so I know I can specify what kind of thing a function returns, with an arrow thingy, like: def foo() -> str:

how can I say that foo takes as arguments, a string called s1, an integer called n1 and another string called s2?

do you use a colon, like: s1: str?

[acoustic thicket]

yes

[safe lance]

so then the whole signature would be: def foo(s1: str, n1: int, S2: str) -> str: ?

[acoustic thicket]

yes

[safe lance]

good enough; thanks a lot 🙂

there is one more thing... say I wanted to take a more complex object as a parameter... howbout a request, assuming I have no imports yet in that file, how would I take a request?

def func_that_takes_a_request(r: request) -> dunnowhatIdreturn:

this isn't enough I guess, because python hasn't seen the definition of request

[trim tangle]

what kind of request?

[safe lance]

@trim tanglehi...

[trim tangle]

You can use any class in a type annotation, so it's something like ```py
from starlette.requests import Request

def do_something(request: Request) -> None:
...

[safe lance]

howbbout the kind of request used in a django view function?

[trim tangle]

According to the docs:
https://docs.djangoproject.com/en/4.1/topics/http/views/
these functions accept a django.http.HttpRequest object:
https://docs.djangoproject.com/en/4.1/ref/request-response/#django.http.HttpRequest

so you'd do

from django.http import HttpRequest
``` and use `HttpRequest`

[safe lance]

so... def funcName(request: HttpRequest):

[trim tangle]

yes

[safe lance]

ok, in those examples, the name of the class is used as the type name

[trim tangle]

indeed, just like with int and str

[safe lance]

but int and str are names that are already present in any python3... if not, you import the class

[slender timber]

I have an enum:

class AEnum(enum.IntEnum):
    A = (1, a_type)
    B = (2, another_type)

How do I make a TypedDict from this enum such that when I have a dict of AEnum keys and object values, the type of object will be inferred as the one given in AEnum?

[trim tangle]

I don't think you can, why do you want that?

[slender timber]

As i said I have a dict of AEnum keys and objects are of type as defined in the enum

[brisk heart]

Also enum.IntEnum is the wrong base class here, you wouldn't be able to use an int

[trim tangle]

and why do you need a TypedDict?

[slender timber]

I have to cast everytime making it a pain

[slender timber]

Its for a demonstration purposes I have my own metaclass and its subclass which AEnum derives from

[brisk heart]

I honestly think you should first do a typed dict instead of an enum

[trim tangle]

If you know the key, then just use a_type directly. If you don't, well, then you don't know what it is

[slender timber]

Wdym

[trim tangle]

Can you show an example of how you'd use your setup?

[slender timber]

How can I annotate an integer value with any arbitrary type?

[slender timber]

Hangon

[brisk heart]

Pretty sure Foo = TypedDict("Foo", {1: str}) works but idk

[slender timber]

https://github.com/demberto/PyFLP/blob/f1f9621251f994762a9dcf2788ecb9539be5aae3/pyflp/_events.py#L71-L99

Here is my enum metaclass and its subclass

[brisk heart]

Actually wait I think it has to be Mapping[str, T] so idk

[slender timber]

https://github.com/demberto/PyFLP/blob/f1f9621251f994762a9dcf2788ecb9539be5aae3/pyflp/channel.py#L307
Here is how a EventEnum looks like

Here is how I get the type from an enum member
https://github.com/demberto/PyFLP/blob/f1f9621251f994762a9dcf2788ecb9539be5aae3/pyflp/__init__.py#L137

[rough sluice]

pyflp/__init__.py line 137

event_type = getattr(enum_type(id), "type")```

[slender timber]

Where will I store the name of the enum member?

Is there a better ™️ enum library for python

[fierce ridge]

i would love the ability to "derive" a typeddict from a class and its attribute annotations

[void panther]

Is there some way to type this with TypeVarTuple/Unpack to preserve the type information and order for the returned instances?

def f(*types):
    return tuple(klass() for klass in types)

I'm not sure how I'd stick in the type in there to get a different return type but one that's based on the type var

[soft matrix]

Not currently

You'd need a Map type which was in a draft of 646

[trim tangle]

yeah

[soft matrix]

It's supposed to be coming some time later this year

[trim tangle]

without Map PEP 646 is kinda... weak, idk

like yeah, you can have a tuple of things

but you can't do anything useful from it besides accepting that tuple and returning it

[soft matrix]

If it has bound and variance it'd be a lot better imo

[blazing cobalt]

Building on Pyright's Exhaustiveness checking to make ADTs and a Result monad:

• ADT: https://github.com/dogweather/python-exhaustiveness-adts-monads/blob/master/src/adt_1_order_status.py
• Result type: https://github.com/dogweather/python-exhaustiveness-adts-monads/blob/master/src/adt_2_result.py
• Result monad: https://github.com/dogweather/python-exhaustiveness-adts-monads/blob/master/src/adt_3_result_monad.py

Seems pretty Pythonic to me still. The types are actually mostly dynamic.
tldr; By using a Result, Pyright can flag when you forget to check for errors.

[trim tangle]

Why not make Ok and Err generic?

[slender timber]

How does marking a type alias as TypeAlias help?

Might as well do

str_or_int = str | int

[trim tangle]

it helps if you want to use a string, for example

# oh damn, we're on Python 3.6

str_or_int: TypeAlias = "str | int"

[slender timber]

Python 3.6 is getting dumped universally anyways 😁

But yea, it can be used for a type declared later than the alias

[trim tangle]

!pep 613

[rough sluice]

**PEP 613 - Explicit Type Aliases**

Link

Status

Accepted

Python-Version

3.10

Created

21-Jan-2020

Type

Standards Track

[blazing nest]

How would I type a weakref.ref?

[trim tangle]

weakref.ref[int]

same as weakref.ReferenceType[int]

well, weakreffing an integer is silly but you get the point

[blazing nest]

Not even weakref:able lmao

Thanks

[undone lantern]

Is there a way I can overload using a specific argument type -> return type?

For example:

class AnimalType(Enum):
    DOG = 1
    CAT = 2
    HORSE = 3

@overload
def get_animal_by_id(animal_id: int, animal_type: AnimalType.HORSE) -> Horse:
    ...

@overload
def get_animal_by_id(animal_id: int, animal_type: AnimalType.CAT) -> Cat:
    ...

def get_animal_by_id(animal_id: int, animal_type: AnimalType) -> AnimalObject:
    ...


cat = get_animal_by_id(1, AnimalType.CAT)  # Not recognized as just Cat

[oblique urchin]

yes, use Literal[AnimalType.HORSE]

[undone lantern]

I tried that as well, still did not work.

[soft matrix]

what does your updated code look like?

[undone lantern]

@overload
def get_animal_by_id(animal_id: int, animal_type: Literal[AnimalType.HORSE]) -> Horse:
    ...

@overload
def get_animal_by_id(animal_id: int, animal_type: Literal[AnimalType.CAT]) -> Cat:
    ...

This?

[soft matrix]

and what about it doesnt work?

do you get any errors?

[oblique urchin]

works for me https://mypy-play.net/?mypy=latest&python=3.10&gist=f7a2ddc7519adf0d5c9f0735345e2513

[undone lantern]

It still sees it as Horse | Cat and shows Horse methods

[oblique urchin]

is that pycharm?

[soft matrix]

pycharms typechecker is notoriously incomplete

[oblique urchin]

pycharm isn't very good at this

[undone lantern]

Yeah

Hmm

Is there a way to integrate a better checker into it or is there an alternative?

[trim tangle]

well, afaik the only real alternative autocompletion thing is pylance

[oblique urchin]

vscode comes with pyright which is generally better. There may also be plugins for using mypy with pycharm, not sure

[trim tangle]

well, mypy doesn't do suggestions

[soft matrix]

there is a mypy plugin but yeah its not hooked up to the suggestion engine

[slender timber]

class A(enum.IntEnum):
    A = (1, a_type)

How can I make this enum a TypedDict (or something else)

I have a ton of this enums I would really like to make them infer a_type when A.A is used as a key or in a function argument

[soft matrix]

enums are special cased in every typechecker i know of so doing this the way you seem to want isn't possible, so here are a couple of questions i think you should ask yourself

1. why is this an IntEnum, tuples are not integers
2. enum members are ordered in every version the module is present (definitely 3.6+), so why does this need the enum number. If the numbering isnt sequential you could add a .type property which just gets the type you want
3. why do you want special behaviour when a member is used as a key or function argument, i think that will be rather surprising to a lot of people

[slender timber]

1. IntEnum was just for demonstration, its actually a custom enum class I made https://github.com/demberto/PyFLP/blob/e30238bcf6abbccc0d6e43f7cd608547263d0853/pyflp/_events.py#L85-L99
2. Enum members are fragmented into various enums, not sequential.
3. These enums are for event IDs and the classes which need them use a dictionary or list of events to lookup these events by their ID. If found, I need to typing.cast them most of the times.

The enums have a type because it used by the event parser to choose the correct type for the event. The enum class and the metaclass above it confuse the type checker a lot already 😅

@soft matrix

[soft matrix]

i think your best option is a type property

and a lot overloads on that property

oh wait that might not be possible

you might have to stick with a very broad type in that case

[slender timber]

I am doing that rn unfortunately