type-hinting

[trim tangle]

Why do you need the information that it's exactly 5?

what's wrong with list[list[pygame.Surface]]?

[past summit]

I guess nothing, i don't usually type hint so i was a bit unsure about how much information i should include.

Thank you!

[fierce ridge]

this answer might be a little unsatisfying, but in general it's very difficult to define statically sized arrays in type signatures

in python, the only data structure that supports it is the tuple, and only as a special case

it's unsatisfying because in principle you can have statically sized arrays if you promise not to add or remove elements... but statically guaranteeing that you have not added or removed elements is difficult

[trim tangle]

well, you can provide an interface that doesn't include appending and removing 🙂

[fierce ridge]

also true, but it's not like you have a compiler enforcing that

[sick notch]

Should you type hint when you declare the attributes in the init function as well?

class Eplet:
    def __init__(
        self,
        name: str,
    ):
        self.name = name

should I do self.name: str = name?

[fierce ridge]

you can annotate both the attribute name and the __init__ parameter name

class Eplet:
    name: str

    def __init__(self, name: str) -> None:
        self.name = name

i think mypy might do some type inference here, but i usually just write them both

or i use attrs

[sick notch]

Why not self.name: str = name?

[sick notch]

What is attrs?

[fierce ridge]

i don't think mypy recognizes it

[fierce ridge]

https://attrs.org

it's like dataclasses, but it came first and has some different features

[sick notch]

Fair enough thx!

[slender timber]

I have a class, having all attributes returning int | None. How do I make some typeguard for its instances such that type checker thinks all attrs are always int

without casting at tons of places

[hallow flint]

mypy should recognise it just fine

[hallow flint]

if you have invariants that you cannot teach the type checker about, one pattern you could use is:

def checked_attr(self):
    assert self.attr is not None
    return self.attr

and then used checked_attr everywhere you have manually confimed your invariant

if the attributes are only briefly None and aren't exposed, e.g. maybe you have some minorly complicated init procedure, you could also just lie and make the types int instead of int | None

[fierce ridge]

mypy will recognize calls to type guard functions on instance attributes inside instance methods, right?

i don't think there's any way to apply a type guard to multiple attributes at the same time by just passing the instance to a function

[stable moth]

i have this dynamic thing and i'm struggling to figure out how to make it work with type hinting. basically i have a class like Foo. it has a bunch of methods that are generators. I want to create from it a new class Bar, that has the same names as the generator methods on Foo, but where it exhausts the generator in a while loop and returns the return value of the generator.

I already know how a solution to this would look if I did it by dynamic means; some combination of __getattr__ and looking at the __dict__ of Foo. but I can't figure out a way of doing it such that a static type checker would okay it.

so, to clarify: I want this:

from typing import Generator


class Foo:
    def func_a(self, x: int) -> Generator[int, None, int]:
        yield 1
        yield 2
        yield 3
        return x

    def func_b(self) -> Generator[str, None, str]:
        yield "aaaaa"
        yield "bbbbb"

        return "ccccc"

    def func_c(self, x: int) -> Generator[int, None, None]:
        yield x


class Bar(Foo):
    def func_a(self, x: int) -> int:
        gen = super().func_a(x)
        try:
            while True:
                next(gen)
        except StopIteration as e:
            return e.value

    def func_b(self) -> str:
        gen = super().func_b()
        try:
            while True:
                next(gen)
        except StopIteration as e:
            return e.value

    def func_c(self, x: int) -> None:
        gen = super().func_c(x)
        try:
            while True:
                next(gen)
        except StopIteration as e:
            return e.value

but without having to literally write out every single method on Bar like I just did. I want a typechecker friendly way of factoring out all the duplication in Bar. does this make sense? (also it doesn't have to be via inheritance, I just used that for this example)

[slender timber]

This exactly☝️

[slender timber]

How can this be done outside tye class itself and for multiple attrs at once?

Can I do something like

 class_having_optional_attrs.__getattr__: Callable[[Self, str], int]

So that whenever an attribute of of that class instance is accessed type checker thinks it as int?

pytest.mark.__getattr__ does this for example, to support plugins

But its inside the class itself

[soft matrix]

why do you need to do it outside the class?

[trim tangle]

Can you show a more realistic example maybe? Why do you need inheritance here?

oh, I think I see what this is

Why not have one function that does this? ```py
def run(gen):
try:
while True:
next(gen)
except StopIteration as e:
return e.value

That's how async frameworks do it. Imagine if every class with an async method foo also had a sync method sync_foo

[hallow flint]

that's an unusual set of requirements. maybe a good pattern for you is:

class Original:
    a: int | None
    b: str | None
    c: bytes

class OriginalChecked:
    a: int
    b: str
    c: bytes

original: Original
# but i'm sure none of the fields are None, so cast to the checked duck type
cast(OriginalChecked, original).a

[hallow flint]

at the very least seems like you could significantly reduce your boiler plate with something like:

class Bar:
    func_a = make_gen(Foo.func_a)
    func_b = make_gen(Foo.func_b)

but not sure you have other options without refactoring (other than maybe writing a mypy plugin)

[dull lance]

Is the (true) int | None type annotation only used internally? If so then maybe you can write a custom descriptor that accesses an internal attribute but exposes it as int, and make an instance of that descriptor for each attribute you want to expose as int

But as far as I know, you cannot automatically change the exposed type for every attribute that is originally of a given type

Perhaps you can have a dedicated get_as_<type> function that lets external code pass in an attribute name (which you can control with Literal) and annotate the function as returning the exposed type. Internally the function would just call getattr and perhaps assert at runtime that the type is correct. However you will have to remember to update the list of allowed literals whenever you rename an attribute.

[slender timber]

I am using my lib as a dependency where this behaviour is needed. In reality, None values can be returned. But I check the struct size of underlying data to make sure no fields are None, but Mypy will definitely not understand that.

[slender timber]

I can give this a go.

[slender timber]

The class attributes are custom descriptors themselves returning None if underlying struct is smaller than the offset of a certain field

I was confused whether such access should return None or raise Exception. Now I feel I should have gone with the latter, so that the type hints could be better

[stable moth]

I do have a function that does exactly that. but that's precisely my problem. mypy doesn't like it

it thinks the type of e.value is Any

also it doesn't need inheritance, I just wrote it that way for the example

from typing import Generator, TypeVar

T = TypeVar("T")
U = TypeVar("U")

def exhaust_generator(gen: Generator[T, None, U]) -> U:
    while True:
        try:
            next(gen)
        except StopIteration as e:
            return e.value

$ mypy --strict exhaust.py 
exhaust.py:11: error: Returning Any from function declared to return "U"
Found 1 error in 1 file (checked 1 source file)

[stable moth]

yeah but I can't get it to typecheck, that's my actual problem, see above

[trim tangle]

Well, you'll just have to put a # type: ignore here

exceptions aren't part of the interface of a function, so as far as mypy is concerned, next(gen) could raise StopIteration with any value

[stable moth]

damn that sucks

thanks tho

[trim tangle]

well, mypy wasn't designed for mathematically proving programs, it's just a programmer aid 🙂

[stable moth]

also, is there any way around writing out all the methods twice, even if the second time they're just oneliners that convert the original methods

don't want to accidentally forget to write one

[trim tangle]

Why do you need the second set of methods?

Just use exhaust_generator at call site, no?

[stable moth]

exactly as you guessed, to have a sync and an async interface

exhaust generator is an implementation detail

[trim tangle]

can you show a more realistic example maybe? I don't really get it

[stable moth]

it's kinda hard to give an example but bear with me

have you heard of "sans io"

[trim tangle]

yeah

[stable moth]

https://sans-io.readthedocs.io/ i'm trying to do this on a project

although it isn't exactly a "network protocol", it's a gigantic project to talk to industrial machinery

so i want the implementation to be "pure" generator-based coroutines that i can test, that just yield and receive bytes. and then i want some kind of thingy that can take all those generators and attach them to different io systems

one of them would be sync, and we would recover our existing sync API. and another would be async

[trim tangle]

I guess there isn't really a good way other than doing a little bit of boilerplate code 🤷‍♂️
I guess you could make a decorator like ```py
class Foo:
def init(self, ...):
self._async_foo = AsyncFoo(...)

...
fizz = to_sync(AsyncFoo.fizz)
buzz = to_sync(AsyncFoo.buzz)
bar = to_sync(AsyncFoo.bar)

[stable moth]

yeah that's more or less what i'm doing. i guess i'll just have to use #type:ignore to make it type check

i guess i could decorate each method with like @sansio or something, and then have a test that checks that every such decorated methods has an equivalent sync and async implementation

thanks :)

although it would be: FooImpl which is neither sync nor async, because it doesnt do any io at all. and then FooSync and FooAsync are derived from it

[trim tangle]

You could have something like ```py
class Foo:
...
@sansio
def bar(self, baz: int) -> list[str]:
...

foo = Foo(...)
foo.bar.s(baz=42)
await foo.bar.a(baz=42)

I guess Python's type system is just bad at dealing with "proxies"

[stable moth]

yeah I considered something like that, but I want it to be a backward-compatible change. i.e. we already have a sync api that looks like machine_foo.do_thing(...)

[trim tangle]

you could make bar callable

[stable moth]

so i dont want to make people add an extra .s lookup every time to use the old api

[trim tangle]

like, you could do foo.bar(baz=42) and await foo.bar.a(baz=42)

[stable moth]

oh like, just glue the async version as an attribute of the sync version?

[trim tangle]

yeah, kinda

sansio could return an object which is both callable and has an a attribute

although you'd have to do some descriptor magic

[inner hollow]

is it possible to make a parameter of list with a type attached to it - a list of ints for example?

def someFunc(param0=list) But then I'm not sure how to ensure the list is a list of ints

[oblique urchin]

list[int]

[inner hollow]

ty now I feel stupid cuz I tried list(int) XD

[brisk heart]

sans unrdtail 😳😳

[trim tangle]

when the impostor is sus

[rare scarab]

specifically, def someFunc(param0: list[str])

[sudden ether]

I'm trying to understand the practical use cases for variadic typing... apparently numpy arrays is an example. If I wanted to type a function as taking numpy arrays of the same dtype, and with shapes (X, 1) and (X, 2) can I express that? Maybe you could already do that particular example because the ranks are concrete?

[brisk hedge]

Yes, in that case a normal type variable over X works

An example of where the variadics become more apparent is when taking function arguments
So for an array arr of shape S and datatype T, you should be able to index it like arr[i0, i1, ..., ik] to get an instance of T back, but how do you properly type the argument list of __getitem__?

[tender lynx]

I'm trying to use typehints more liberally in my code and that's prompted some questions.

Is doing something like this (hinting that a string will be returned, but if the question is invalid, returning None) considered a poor practice?

def get_new_question(prompt: str = "Please enter a new question prompt: ") -> str:
    """
    Prompts the user for a new question, validating that it meets the question
    criteria as defined in the validate_question method. Returns nothing if the
    question does not meet the criteria.
    """
    question = input(prompt)
    if validate_question(question):
        return question

[trim tangle]

You could return None, but then you should mark that in the return type as str | None

Why would it accept the name of a function instead of a function? Could you provide an example maybe?

[trim tangle]

unrelated to types but I'd remove the docstring

[tender lynx]

Why’s that? Just because the function is fairly trivial?

[paper salmon]

typing.Callable would likely be appropriate for that

e.g. Callable[[], Any] to signify no parameters (or at least not required parameters) and any return type

[trim tangle]

yeah Callable[[], None]

[trim tangle]

No, because the docstring just duplicates the implementation

it doesn't add extra information

[trim tangle]

just to nitpick, you should actually be using collections.abc.Callable because of PEP 585 and such...

Well, you don't really have to

||we'll probably all die in a nuclear blast before typing.Callable is removed||

||especially me 😰 ||

[paper salmon]

i dont think ive ever used collections.abc in my code before, despite having seen it a dozen times

[trim tangle]

TL;DR since 3.9, various classes like collection.abc.Sequence and re.Pattern became subscriptable, rendering their counterparts from typing redundant. That is what PEP 585 was about

[tranquil turtle]

Agree
Implementation is clear, there is no unobvious (tricky or hard to read) code. And function and args names are good enough. So, there is no need for such long docstring.
If you really want docstring, you can shorten it to one line: "Read question, return it if it is valid, else return None"

[trim tangle]

Well, haiku docstrings run the risk of being highly ambiguous, only adding confusion 😄

[tranquil turtle]

I read that all non-special types in typing will be moved from typing to other modules

[trim tangle]

wdym non-special?

well, Callable is strange because it is not a "collection"

[tranquil turtle]

List, Dict, Iterable, Callable, etc...

[trim tangle]

Hashable isn't a collection either yet it's in collections.abc

[tranquil turtle]

I mean types that are not handled by typechecker in other way

Special types: NoReturn, Any, TypeAlias, ...

[trim tangle]

ah, special forms or something like that IIRC

[tranquil turtle]

Not exactly. Almost everything in typing is _SpecialForm, iirc

[trim tangle]

everyone is special

[tranquil turtle]

Crazy idea: add builtin var __typing__ that is reference to typing module
So, you dont need to import typing anymore

T = __typing__.TypeVar('T')
G = __typing__.TypeVar('G')

def apply(x: T, f: __typing__.Callable[[T], G]) -> G:
    return f(x)

[trim tangle]

ah yes, __typing__.Callable[[__typing__.AsyncIterator[int] | __typing__.Awaitable[int]], __typing__.NoReturn]
beautiful

[tranquil turtle]

perfect

[fierce ridge]

from collections.abc import Callable
from typing import ParamSpec, overload

_P = ParamSpec("_P")

class LazyString: ...

@overload
def lazystr(func: Callable[_P, str], *args: _P.args, **kwargs: _P.kwargs) -> LazyString:
    ...

@overload
def lazystr(func: str, *args: object, **kwargs: object) -> str:
    ...

def lazystr(func: Callable[_P, str] | str, *args: _P.args, **kwargs: _P.kwargs) -> LazyString | str:
    ...

what am i missing here?

lazystr.py:58: error: Overloaded function implementation does not accept all possible arguments of signature 2  [misc]

here's my stackoverflow post if anyone wants a couple of internet points for answering: https://stackoverflow.com/q/74283050/2954547

[tranquil turtle]

Add builtin t = typing
Then you can use it, it doesnt need a lot characters.
I've seen import typing as t a lot of times.

It is backward compatible change, nothing will break (except for some code, that relies on that t does not exist)

[tranquil turtle]

I think you should add | object to args and kwargs

[fierce ridge]

ugh, let me try

nope, that broke it more

lazystr.py:58: error: Name "_P.args" is not defined  [name-defined]
lazystr.py:58: error: Name "_P.kwargs" is not defined  [name-defined]
lazystr.py:100: error: Argument 2 to "LazyString" has incompatible type "*Tuple[Union[Any, object], ...]"; expected "_P.args"  [arg-type]
lazystr.py:100: error: Argument 3 to "LazyString" has incompatible type "**Dict[str, Union[Any, object]]"; expected "_P.kwargs"  [arg-type]

[tranquil turtle]

Im not sure if P.args | object is even correct

[fierce ridge]

it's not valid apparently

i also tried putting *args: _P.args, **kwargs: _P.kwargs in the 2nd overload, but that gave the same error i think

[tranquil turtle]

Maybe add | Unpack[tuple[object, ...]]?

Or something like that

[fierce ridge]

yep, same error with _P

[tranquil turtle]

We need mypy or typing experts there

[fierce ridge]

of course my SO post gets -1 with no comment

i'm convinced that people just downvote questions that they don't know answers to

this is what i get for the hours i've spent answering questions in detail 🙄

[tranquil turtle]

You can write Any for args and kwargs as workaround

[fierce ridge]

let me try that

i guess it should be OK since the @overloads provide more detail, right?

[tranquil turtle]

Yes, implementation annotations are ignored

[fierce ridge]

ah yep that fixed it

(why exactly does mypy require annotations on the implementation?)

[tranquil turtle]

Except at function definition time. Mypy checks that implementation is correct, and then it throw away implementation annotations

[fierce ridge]

right, but why is that check needed?

it adds a lot of duplication in big overloads

[tranquil turtle]

I dont think it is required by pep. It is nice feature to catch some errors

[tranquil turtle]

You can put Any, if you don't want to put annotations

But then your implementation body will not be typechecked, because all args are Any

[soft matrix]

I don't think that's true

[fierce ridge]

shouldn't be able to infer the parameter types from the preceding overloads?

it can match up parameters by name and Union the annotations

[soft matrix]

I'm pretty sure that's just if it's unannotated

[fierce ridge]

you're right about it not being in the pep: https://peps.python.org/pep-0484/#function-method-overloading

[tranquil turtle]

I guess it will emit "function have no annotations" error

[fierce ridge]

what i am wondering: is there a technical reason why the implementation must also be annotated? why can't it infer that all from the overloads?

[tranquil turtle]

Maybe it can, try it

[fierce ridge]

i know that it can't 😛 it will give some error saying that the implementation does not match the overloads

but i was really surprised when i first saw that

seemed like an unnecessary thing to require

[tranquil turtle]

Mypy is very dumb sometimes

[tender lynx]

I’m still a total rookie when it comes to documentation. I’m trying to force myself into writing doc strings for everything because I know there are some powerful tools for automatically documenting libraries based on doc strings. I agree the documentation doesn’t add much for anybody looking at this code, but I’m trying to think of the person who is not wanting to look inside the function to see what it does.

[trim tangle]

What you're describing is called a "reference documentation". It mirrors the structure of the code and it explains each function/class/etc.. IMO there's little point in making it into a fancy HTML document because if you're at that level, you might as well just look at the code.

And certainly not at the expense of people reading the code.

For many tasks, from the function name, parameter names and other context it should be easy to infer what the function does. If there's some elaborate missing context or explanation, put it in the docstring.

[trim tangle]

Very often I switch to reading the actual code. It's much easier to navigate the code in your editor.

What belongs into non-code documentation is better described here
https://diataxis.fr/

[fierce ridge]

IMO there's little point in making it into a fancy HTML document because if you're at that level, you might as well just look at the code.
i strongly do not agree that source code is a substitute for good reference docs

good html reference docs are heavily cross-linked/anchored and are optimized for a balance of rapidly finding very specific pieces of information, and "scanning" to get a sense of overall structure. plus they are searchable and can be cross-linked from the non-reference docs if needed.

it's why i don't love the python "reference"-type docs in a lot of cases. not enough cross-linking, not enough structure. and it's infuriating to use pydantic and fastapi because they have no reference docs at all.

reading the source code is necessary with those libraries, and it is much more difficult than if it were part of the docs site, even using github code search. it would be much less necessary if they had reference docs.

attrs and click meanwhile have excellent reference docs, and it pays off

[trim tangle]

For overall structure, there are options on editors that show the outline of a module or package. For cross-linking — yes, good point, if that's done. But I'd argue it's an issue with editors: it would be good to be able to jump to stuff within the editor.

[fierce ridge]

imagine using pandas or scikit-learn or matplotlib without reference docs ☠️

or anything nontrivial in pytest for that matter

[trim tangle]

Yeah, but perhaps it's more of an issue with the complexity in the library itself

[fierce ridge]

at bare minimum, there's no reason to force people to be stuck with plain-text docstrings instead of formatted html

sure, but complexity is a part of life, and certainly a part of programming

[trim tangle]

I guess in theory, reference docs are good. But in practice, I often see just a fancy rendering of plain docstrings with questionable CSS (I'm looking at you aiohttp)

[fierce ridge]

yes, even that is useful. more useful than reading the source code

[trim tangle]

Well, in those cases I usually have to jump to the implementation anyway

[fierce ridge]

it's searchable, indexed by search engines, cross-linked, easy to scan, and has all the relevant information right in front of me without tons of code in the way

[trim tangle]

Yeah indexing is good

[fierce ridge]

i've never had to read the aiohttp source code! but if i didn't have this reference doc page, i 100% would have needed to at some point

even medicore reference docs are a huge time and effort saver for end users

[trim tangle]

I just find the html completely unreadable, especially the parameter lists

[fierce ridge]

well yeah the css is broken

[trim tangle]

Oh and how auto documentation tools render type annotations

[fierce ridge]

i still think it's better than the alternative of not existing at all

[trim tangle]

Idk, maybe it's just personal preference

Maybe your editor renders docstrings in the wrong colour 🙂

[fierce ridge]

indeed it is. but the recent trend of library developers imposing that preference on users is frustrating. it's not a preference that many people share, and frankly doesn't make sense to me. it fits into one or two specific styles and workflows

starlette, pydantic, all of tiangolo's packages (fastapi, sqlmodel, typer), ... basically any package using mkdoc

it makes me not want to use them, and i'd avoid them for that reason, if they weren't otherwise high quality. at least starlette is hard to avoid, the others are replaceable to some extent and with effort.

anyway this is now salt rock ranting and not #type-hinting

yes, sphinx is horrible at rendering type annotations

i don't know how epydoc fares (does it even recognize them?)

pdoc seems okay good, but not so much better than sphinx to warrant losing all the power of sphinx plugins like :math: (really useful for data science libraries) and intersphinx

[trim tangle]

I think Rust's solution to autodoccing is good. The generator is shipped with cargo and works automagically with your code. You type one command to build it, no extra build steps required. It's very hard to mess up

Anyway yeah, offtopic

[tender lynx]

@trim tangle @fierce ridge I greatly appreciate the discussion even though it’s gone off topic for the channel. I think I’m gonna stick with writing docstrings (especially since this is just a random homework assignment)

[sonic fern]

Hello!
Is there a way to type-hint something as to only accept immutable (floats, strs, tuples...) types?

Currently I'm doing it at runtime by using a decorator that tries to hash() all arguments and raises and exception if that fails. But that is kind of an awful hack and I'd rather not do it at runtime for obvious reasons. Any help?

tl;dr tfw no typing.Hashable ;_;

[oblique urchin]

um there is a typing.Hashable

[little hare]

!d typing.Hashable

[rough sluice]

typing.Hashable

class typing.Hashable```
An alias to [`collections.abc.Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable "collections.abc.Hashable").

[little hare]

huh, its been there since 3.5?

[oblique urchin]

part of the original typing

[sonic fern]

o_0
Nice, could swear it wasn't a thing
I think Fluent Python mention that not being a thing - and that's the 2022 version. Maybe the author overlooked that?

[little hare]

oh btw jelle, is typeshed-client still maintained?

[oblique urchin]

yes

[oblique urchin]

it's been kind of problematic in the typing context historically, but it exists

[little hare]

i've been considering using it for a thingy because I haven't figured out how to write a language server client

[oblique urchin]

e.g. we haven't been able to say so far that dict keys must be Hashable

[little hare]

ideally i'd use a language server since those automatically use typeshed or source code but i guess I can write both my self

[rare scarab]

note that Hashable also includes tuple

and frozenset

[oblique urchin]

sounds like a good use case for typeshed-client

[little hare]

yes..., Hashable is anything with a __hash__ method defined @rare scarab

[little hare]

yeah, probably will use it

[sonic fern]

frozenset is fine because everything there needs to be hashable anyway
But will a tuple with a list inside be Hashable?

[rare scarab]

If you want a actual type, you can do ```py
Primitive: TypeAlias = str | bytes | int | float | bool | None

[rare scarab]

Only tuple[Hashable, ...] is Hashable

[oblique urchin]

yes, a non-hashable tuple will still be hashable to a type checker

it's of course not hashable at runtime, but the type system doesn't know that

[sonic fern]

Oh, then I'll need to find another solution. I'd really like to prevent mutability as much as I can

[rare scarab]

use the TypeAlias solution I posted

[sonic fern]

I don't want only primitives, though.

A hashable class should be fine.

As long as everything inside is hashable as well.

[rare scarab]

types are hashable

[little hare]

that feels like an oversight on typechecker implementation of tuple, honestly

[oblique urchin]

this doesn't sound achievable with the static type system

[little hare]

since tuple[Hashable] errors properly

[little hare]

*in python

since a class can just define __hash__ and then lie and return whatever

[sonic fern]

I might want to do AST trickery to achieve that then, requiring typehints and checking if they are "properly" hashable.

[oblique urchin]

for what it's worth, you might get close in my pyanalyze typechecker with CustomCheck: https://pyanalyze.readthedocs.io/en/latest/typesystem.html?highlight=CustomCheck#customcheck

[slender timber]

It is, but the ways to extend or hook autodoc methods is atleast documented (kinda) properly.

Sphinx has like only one full time dev judging from its changelog

[trim tangle]

immutable != hashable 🙂

For example, classes are mutable and hashable.

[sudden ether]

Right, but I'm still unclear as to how to express it: https://numpy.org/devdocs/reference/typing.html doesn't seem to have anything about shapes, just dtypes

Is there a concrete example anywhere? e.g. just typing a function that takes two numpy arrays of the same dtype, and with shapes (X, 1) and (X, 2) - how can I type hint that?

[tranquil turtle]

Functions are also mutable and hashable

[trim tangle]

yep

[brisk hedge]

https://github.com/numpy/numpy/issues/16544#issuecomment-1058675773

[sudden ether]

Right - but the variadic generics are not actually necessary for the simple case I outline? So in principle it should be possible now, it's just that numpy.typing doesn't support it?

[brisk hedge]

are you sure? since the documentation claims ndarray is generic over shape and dtype

[sudden ether]

No, I'm not sure...

But I can't see an obvious example: https://numpy.org/devdocs/reference/typing.html

[fierce ridge]

as far as i remember, the typing design for shapes isn't actually finalized, so they recommend using NDArray for now, which is generic only in dtype

i raised an issue about it in the issue tracker a while ago and that's what they recommended

https://github.com/numpy/numpy/issues/19716

As for the parameters: I'd strongly recommend keeping the shape-type as Any for now, as it is very much a placeholder slot until we can properly get shape typing going once PEP 646 is live. Until that time use anything else at your own risk (or just use the more compact npt.NDArray).

[oblique urchin]

PEP 646 is "live", except that mypy doesn't support it

[fierce ridge]

well i'm not sure if numpy has finalized their design yet either

it seems like they maybe have? but i don't see a spec for it

[rare scarab]

So which pep will extend 646 adding "typevar comprehension"?

e.g. ```py
Ts = TypeVarTuple("Ts")

def foo(*args: Ts) -> tuple[[Awaitable[T] for T in Ts]]: ...

[oblique urchin]

no PEP for that exists yet

[brittle plover]

I have multiple TypedDicts, that might have (key, value) pairs, where the value is given as a NoneType. See for example the definition of DataStore below:

from typing import TypedDict, Mapping, TypeVar
from dataclasses import dataclass


@dataclass
class Group:
    name: str

@dataclass
class Subject:
    name: str

class DataStore(TypedDict, total=False):
    groups: dict[str, Group] | None
    subjects: dict[str, Subject] | None

For the following examples, the last one requires some special handling when getting the value:

example_complete: DataStore = {
    "groups": {
        "A": Group("A"),
        "B": Group("B"),
    },
    "subjects": {
        "001": Subject("001"),
        "002": Subject("002"),
    },
}

example_incomplete: DataStore = {
    "groups": {
        "A": Group("A"),
        "B": Group("B"),
    },
    # missing subject - ok
}

example_incomplete2: DataStore = {
    "groups": {}, # empty container - ok
    "subjects": None, # Allowed, but want to get this as an empty dict
}

This is ok, as expected:

>>> example_incomplete.get('subjects', {})
{}
>>> example_incomplete2.get('groups', {})
{}

This returns None and needs some boilerplate, as we get None:

>>> example_incomplete2.get('subjects', {})
None

With a simple or, we can default to an empty dict:

>>> example_incomplete2.get('subjects') or {}
{}

Creating a helper function, I start out with the following naive typing:

K = TypeVar("K")
V = TypeVar("V")
def get_from_map_or_default(d: dict[K, V | None], key: K, default: V) -> V:
    return d.get(key) or default


a: dict[str, Subject] = get_from_map_or_default(example_incomplete2, "subjects", {})
# error: Argument 1 to "get_from_map_or_default" has incompatible type "DataStore"; expected "Dict[str, Optional[Dict[str, Subject]]]"

I'm wondering, if it is even possible to find an abstracted typing for the function above without too much of a hassle, as the DataStore.get method is showing up with the following type hint:

get: Overload[(k: Literal['groups']) -> (dict[str, Group] | None), (k: Literal['groups'], default: dict[str, Group] | None) -> (dict[str, Group] | None), (k: Literal['groups'], default: __TDefault@DataStore) -> (dict[str, Group] | __TDefault@DataStore | None), (k: Literal['subjects']) -> (dict[str, Subject] | None), (k: Literal['subjects'], default: dict[str, Subject] | None) -> (dict[str, Subject] | None), (k: Literal['subjects'], default: __TDefault@DataStore) -> (dict[str, Subject] | __TDefault@DataStore | None), (k: str) -> (Any | None), (k: str, default: __TDefault@DataStore) -> (Any | __TDefault@DataStore)]

[merry current]

In file1.py I have```py
from future import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
from file2 import Foo

data: dict[str, ...] = {}
data["foo_list"]: list[Foo] = []and in `file2.py` I havepy
from file1 import data

class Foo:
# data is used inside some methods```

The import just for the type hint is a potential cyclic import as its using names from a partially loaded module

How to resolve this design?

Condition that they do need to be separate files

[rare scarab]

Use a TypedDict

from typing import TypedDict

class Data(TypedDict):
  foo_list: list[Foo]

data: Data = {"foo_list": []}

[merry current]

Excellent! Thanks

[sick notch]

I made a function that returns str | None.
According to an optional argument, I return a string or I don't return anything basically.

so this looks like

if something:
  return a_string

and mypy complains:
19: error: Missing return statement

Should I do:

else:
  return None

?

this seems unnecessary. Am I wrong? Is it good practice to add this bit?

I found this SO answer (https://stackoverflow.com/questions/60542075/missing-return-statement-in-simple-conditional-structure) that talks about adding an overloaded signature, what does that mean? (actually i googled it and understood it but not really relevent here)

[trim tangle]

Can you show the code and the full output?

[sick notch]

def test(a: bool = False) -> str | None:
    If a:
        Return « something »

The full mypy output is error: Missing return statement

@trim tangle

[tranquil turtle]

if you have some x: str | None, do return x

[sick notch]

I have x: str

It’s never None

[acoustic thicket]

pyright accepts it

bit surprised mypy doesnt

[rare scarab]

!zen

[rough sluice]

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

[rare scarab]

explicit is better than implicit

[sick notch]

Fair enoug

So I should just do else return None

[rare scarab]

You can disable that by setting the config warn_no_return = False

https://mypy.readthedocs.io/en/stable/config_file.html#confval-warn_no_return

[soft yarrow]

hi! in a project i want to opportunistically use orjson instead of the builtin json module. i'm therefore only interested in calling the functions common to both. if i do the try: ... except ImportError: ... trick to assign to the json_module variable, mypy sees right through me. what's the most elegant way of telling mypy it should always assume json's api?

[wise rose]

hello everyone noob here, 🙂
how come my py saying this function is missing an extra annotation?

    def __has_no_errors(path) -> bool:
        try:
            assert path.suffix in [".mp4", ".mkv"], "not supported file type"
            assert path.exists(), "file does not exist"
        except AssertionError as err:
            logging.critical(err.__str__())
            raise
        else:
            return True

[soft yarrow]

path in the arguments needs an annotation

[wise rose]

riiight

ty ty ty

[tranquil turtle]

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    import json
else:
    try:
        import orjson as json
    except ImportError:
        import json

# Now you can use `json` variable. Typechecker thinks that it is json-module.
# If you use something that is specific to json module only, you will get no type-check errors, but you may get runtime errors (if you actually imported orjson).
# if you use something from orjson module, that doent exist in json, you will get type-check error and you may get runtime error (if you actually imported json).

[soft yarrow]

ah yes. thanks 🙂

[tranquil turtle]

also, if you dont want import json at all, you can:

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    import json
else:
    import orjson

[soft yarrow]

but basically type checking opportunistic imports seems to be tricky, especially when you actually need to consider api differences

thanks 🙂

[hallow flint]

on mypy master, modules are allowed to implement protocols. so you could define a protocol with the exact interface you want, and mypy would check everything

[rare scarab]

Yeah, that's useful if you have 2 or more implementations of a module, like a numpy or pandas replacement

We need a way to make a protocol from a variable. like Numpy = Protocol(np)

[cosmic cave]

I would like to enforce that all instance attributes are declared at top-level in the class, or at the very least, declared within __init__

Here is a concrete example of the mistake I want to avoid:

class Zombie:
    shambling: bool
    __init__(self):
        self.shambling = True
    def take_hit(self):
        self.shabling = False

A typo in take_hit means I am not stopping the zombie's shambling as intended. I am instead declaring a new instance attribute.

Is there a mypy configuration which requires all instance fields to be declared within the class at top level or within __init__?
Is there a way to turn the mistake above into a mypy diagnostic?

[rare scarab]

if you use slots, you would have to define all attrs in init

class Zombie:
  __slots__ = ("shambling",)
  def __init__(self):
    self.shambling = True

slots also gives you some performance and memory footprint benefits if you have a ton of instances.

[cosmic cave]

Thanks. At a glance, the syntax seems less ergonomic. Does the use of slots impose other restrictions? How do they interact with inheritance?

[rare scarab]

slots means it won't use a dict as the backing object.

I think it makes it more resemble a struct

dataclasses can do this for you

[cosmic cave]

ok thanks. it seems odd to me that I need to switch the runtime semantics to achieve this typechecking behavior

[rare scarab]

ok, technically you won't get a type error, but you will get a runtime error when using slots

[cosmic cave]

My goal is to get a typechecking error, since that will be raised earlier in the feedback loop

[rare scarab]

Actually, slots works when using strict

https://mypy-play.net/?mypy=latest&python=3.11&flags=strict&gist=b6aa8d04a59e6bf12c4265064616304d

[cosmic cave]

Do language services offer to keep the slots tuple up-to-date, e.g. if I add health: int to the zombie, will the editor offer to add health to the tuple? The ergonomics feel bad here, which is why I am naively suspecting that real python devs don't do this

I'm surprised mypy doesn't have a --require-attribute-declarations-in-init flag

funky, wrapping the slots declaration in if TYPE_CHECKING avoids the change in runtime semantics

[rare scarab]

Some things make sense to initialize elsewhere, like in __enter__

[cosmic cave]

Exactly, having them spread across multiple methods makes the class less readable. What I ideally want is to mandate that each attribute is declared outside of any method:

class Foo:
  health: int # this is necessary

[rare scarab]

maybe a custom flake8 plugin will do the trick

[cosmic cave]

or a mypy plugin? I imagine that the plugin should:

• get mypy's inferred list of attributes
• check that an explicit declaration exists for each

[cosmic cave]

I'm trying pyright, and it is raising diagnostics of the form:

Method "on_key_release" overrides class "Window" in an incompatible manner
  Parameter 2 name mismatch: base parameter is named "symbol", override parameter is named "key"

Coming from other languages, the parameter names do not need to match as long as the types match, because the parameters are positional.

Is this diagnostic raised because calls might do on_key_release(symbol=foobar), so the parameter's name is effectively part of the API surface, and changing the name in an override might break calls?

[oblique urchin]

yes. you can make your params positional-only to avoid this

[cosmic cave]

How do I make them positional-only?

[rare scarab]

names need to match if there isn't a / or they come after /

[oblique urchin]

!pep 570

[rough sluice]

**PEP 570 - Python Positional-Only Parameters**

Link

Status

Final

Python-Version

3.8

Created

20-Jan-2018

Type

Standards Track

[rare scarab]

def foo(x, /):
  pass

x is positional only

If you need to support python 3.7, you can prefix the variable with __

def foo(__x):
  pass

[oblique urchin]

to be clear this doesn't actually make it pos-only, it's just a type checker convention

[rare scarab]

and if you try to provide it as a keyword, your type checker will warn you about private access.

[cosmic cave]

I see. I'm subclassing from a third-party library, and they haven't made args positional-only. Makes sense, it's extra syntactic weight
So I'll have to keep the names identical

[wise rose]

hello everyone noob here 🙂
having trouble figuring out the type hint syntax please help 😦

@dataclass
class Card:
    summary: str | None = None
    owner: str | None = None
    state: str | None = "todo"
    id: int | None = field(default=None, compare=False)

    @classmethod
    def from_dict(cls, d: dict[str, Union[Optional[str], Optional[int]]]) -> Card:
        return Card(**d)

mypy keeps complaining at this saying:

    def from_dict(cls, d: dict[str, Union[str, int, None]]]) -> Card

like this?

[trim tangle]

As far as mypy is concerned, it's valid to call Card.from_dict({"foo": "bar", "id": "hello"})

in what context do you use this method?

also, why are all the fields optional?

[wise rose]

it is a to do app where a Card obj is a todo item
does it matter if the fields are optional?

[trim tangle]

I mean, can you show an example or some context of where you'd use Card.from_dict?

Why do you need this method?

[wise rose]

def test_from_dict() -> None:
    c1 = Card("something", "brian", "todo", 123)
    c2_dict = {
        "summary": "something",
        "owner": "brian",
        "state": "todo",
        "id": 123,
    }

    c2 = Card.from_dict(c2_dict)
    assert c1 == c2

[trim tangle]

But why does this method exist?

[wise rose]

maybe i can use json from somewhere else and pass it in there idk

what do you mean?

[trim tangle]

Well, you normally don't create functions at random and not use them

[wise rose]

what about mypy though

[trim tangle]

Do you actually need to load a card from a JSON?

[wise rose]

not yet but i might

[trim tangle]

If you want to convert some kind of untrusted data into dataclasses, you'll have to perform some validation. After all, that JSON can reasonably contain something like {"summary": 42, "owner": [1,[[]]], "state": 666}.

You can write it by hand if you don't have a lot of these, or you can use an existing solution like pydantic or dataclass_factory

[wise rose]

yes but with the existing code though how do i make it so mypy stops complaining?

[trim tangle]

Use # type: ignore or d: dict[str, Any]

[regal summit]

how do you typehint

def __iter__(self) -> ???:
  return iter(self.something)```

[rare scarab]

typing.Iterator

[regal summit]

typing.Iterator[int] assuming self.something is a list of ints

oh ok

[rare scarab]

I think there's also collections.abc.Iterator

[tranquil turtle]

It is the same as typing.Iterator, but typing.Iterator is deprecated (or will be deprecated)
If something exist in collections.abs, it is preferred over typing.

[fierce ridge]

is it just me or is it really hard to write "slightly" signature-changing decorators that are properly type-annotated?

[rare scarab]

blame pycharm

https://youtrack.jetbrains.com/issue/PY-56935/Functions-modified-with-ParamSpec-incorrectly-report-missing-arguments

[fierce ridge]

it's hard even without pycharm's buggy type checker 😬

[oblique urchin]

yes. ParamSpec lets you write decorators that keep the sig unchanged or add a param at the front, but not much else

[rare scarab]

or cut off a parameter (partial)?

[fierce ridge]

yup exactly, i would love to be able to inject or remove a parameter, especially a kwarg

[oblique urchin]

oh yes, only from the front though

[rare scarab]

Callable[[Callable[[T, *P], R]], Callable[P, R]]

[oblique urchin]

no *. I think it's Callable[[Callable[Concatenate[T, P], R]], Callable[P, R]]

[rare scarab]

so we can't unpack a paramspec into an argument list?

[oblique urchin]

no

Unpack is only for TypeVarTuple

(and for TypedDict if PEP 692 is accepted)

[fierce ridge]

this is my use case:

from collections import Callable
from pathlib import Path
from typing import Concatenate, ParamSpec, TypeVar

import jinja2
from pydantic import BaseSettings

queries_dir: Path = ...

class Settings(BaseSettings): ...

def get_settings() -> Settings: ...

_P = ParamSpec('_P')
_R = TypeVar('_R')

def with_app_settings(
    func: Callable[Concatenate[_P, Settings], _R]
) -> Callable[Concatenate[_P, Settings | None], _R]:
    @wraps(
        func,
        # This is a signature-changing decorator, so exclude __annotations__, which is
        # part of the default set.
        assigned=('__module__', '__name__', '__qualname__', '__doc__')
    )
    def _wrapper(*args: _P.args, settings: Settings | None = None, **kwargs: _P.kwargs) -> _R:
        if settings is None:
            settings = get_settings()
        return func(*args, **kwargs, settings=settings)
    return _wrapper

@with_app_settings
def make_jinja_env(settings: Settings) -> jinja2.Environment:
    env = jinja2.Environment(loader=jinja2.FileSystemLoader(queries_dir))
    env.globals["settings"] = settings
    return env

however the position of settings in _wrapper doesn't line up with the position in the Callable annotation

and i don't think it's possible to make it line up, because **kwargs has to come last syntactically

but i'm starting to realize that this is a bigger problem than i thought

i want it at the end and not at the beginning so that i can also write functions like this:

def foo(x, y, z, settings=None): ...

but then i don't have any ability to pass a default argument without annotating it None anyway

i suppose my only other option is to put assert settings is not None at the top of every decorated function

[tranquil turtle]

None: Settings

[merry current]

How would I define a function's return type hint as being the same type as the passed parameter, whose type is not known beforehand?

Is TypeVar to be used here?

T = TypeVar("T") 

def foo(param: T) -> T:```like this?

[acoustic thicket]

yeah, that's correct

[trim tangle]

Yep, that's exactly what type variables are for

minor plug from myself: I have a short tutorial series on this stuff https://decorator-factory.github.io/typing-tips/tutorials/generics/

[tranquil turtle]

There is also tutorial from mypy

https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#cheat-sheet-py3

[real finch]

Hey, I'm using from __future__ import annotations in order to get the type1 | type2 syntax in Python < 3.10. However, it seems that we still cannot do myvar = type1 | type2 and that it only works in function definitions. Is that correct?

[oblique urchin]

that's right. the __future__ only affects code that is syntactically in a type annotation

myvar = type1 | type2 is syntactically an assignment, not an annotation

[real finch]

lol right the name of the import. Is there a way to get it on assignments too?

[oblique urchin]

no

well I guess you can say myvar: TypeAlias = "type1 | type2" in quotes

I believe type checkers will accept that

[real finch]

seems that TypeAlias is only available in Python 3.10

Anyway, that's a bummer that assignments aren't supported the same as annotations. It seems to me that both go together and should both have been implemented in __future__.

[oblique urchin]

you can get it from typing_extensions

[real finch]

I see

[merry current]

Thanks! I will check it out

[brittle socket]

How can I type an iterable parameter to signal it as an iterable of hashable objects?

[oblique urchin]

Iterable[Hashable]

[brittle socket]

Oh lol. Nice. Thank you

[soft matrix]

although Hashable isnt generally that well checked

[brittle socket]

The common use case is a list of strings, it shouldn't pose a challenge

Hashable is a generic right? I can subscript it with aTypeVar?

[void panther]

It isn't

[brittle socket]

Aw

[soft matrix]

what would it be generic over?

[brittle socket]

This is what I currently have

T = TypeVar('T')
def f(pred: UnaryPred[T]) -> Callable[[Iterable[T]], Iterator[T]]:

I can get it to this:

def f(pred: UnaryPred[T]) -> Callable[[Iterable[Hashable]], Iterator[Hashable]]:

But then I lose the semantics enforcing the input and output iterated elements to be the same type

[oblique urchin]

make your TypeVar bound to Hashable

[brittle socket]

Oh interesting. I'll look up the syntax and semantics of that

Hmm...iiuc, rather than H = TypeVar('H', bound=Hashable) I would want H = TypeVar('H', Hashable) to enforce it no?

Oh wait, I want any type that is a subtype of Hashable i.e. implements __hash__, so I want the bound version

[oblique urchin]

you want bound= yes. Your second version isn't legal

[brittle socket]

Right. The 2nd version only works with built-in types?

[oblique urchin]

no, it just doesn't

[brittle socket]

from the doc:

S = TypeVar('S', bound=str)  # Can be any subtype of str
A = TypeVar('A', str, bytes)  # Must be exactly str or bytes

[oblique urchin]

you need at least two constraints

[brittle socket]

Oh, right, makes sense

H = TypeVar('H', bound=Hashable)
def unique_if_(pred: UnaryPred[H]) -> Callable[[Iterable[H]], Iterator[H]]:
  ...

Awesome

[acoustic thicket]

why does the TypeVar("T", A, B) exist at all
wouldnt bound=A | B work in all cases you want that while being inheritance friendly

[oblique urchin]

they aren't the same

TypeVar("T", A, B) mostly exists because of AnyStr, which usually you don't want to be able to bind to bytes | str

[acoustic thicket]

because the former wont accept subtypes of A and B, right?

when would you want to do that and reject subtypes

[acoustic thicket]

i don't understand 😅

[oblique urchin]

def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b

you want to reject concat("x", b"b")

[acoustic thicket]

hmm i see now

[atomic lodge]

Would you recommend type hinting things like logger = logging.getLogger()? It's obvious what it is even without a type hint but type hinting it requires me to specifically import from logging import Logger. I tried to be quite strict in the project with type hinting otherwise

[pastel egret]

There shouldn't be a need to, since that can be trivially inferred to the correct type by the checker. Perhaps annotate as : Final. Actually you could do logger: Final[logging.Logger] = logging.getLogger(), but it's not really useful.

[empty hull]

Can we annotate keyword arguments for a callable?

Example:

def func(arg: int, *,  kwarg: bool = False) -> str:
    ...

def another_func(f: <Type of Func>) -> str:
    ...

Here, I know if it was only arg, it'll go something like Callable[[int], str] but we have a kwarg as well now. How do we annotate that?

[trim tangle]

You can use a callable protocol

class Foo(Protocol):
    def __call__(self, arg: int, *, kwarg: bool = ...) -> str: ...

[empty hull]

Oooh! Thanks for the answer!!

[slender timber]

how to return a lambda which does nothing in a type safe way?

this is my function

def __getattr__(self, name: str):
    if not self.silent:
        return getattr(self, name)
    fake_func = lambda *_, **__: None
    return fake_func

[trim tangle]

What about kwargs?

[slender timber]

yea kwargs too forgot them

[trim tangle]

Callable[..., Any]

[slender timber]

doesn't that not work in strict mode pyright?

[trim tangle]

Also, won't getattr just call the method recursively?

Did you mean object.__getattr__ or __getattribute__

[slender timber]

oh right

i should use __getattribute__

[trim tangle]

Well, you are using very type-unsafe things after all 😄

[slender timber]

i just want to avoid checking self.silent in every instance method

[trim tangle]

Can you show more code maybe?

[slender timber]

long strings already make black do weird things

[slender timber]

from rich.console import Console

class _Output:
    def __init__(self, silent: bool):
        self.stdout = Console()
        self.stderr = Console(stderr=True)
        self.silent = silent

    def __getattribute__(self, attr: str) -> Callable[..., Any]:
        if self.silent:
            return lambda *_, **__: ...
        return getattr(self, attr)

    def success(self, what: str):
        self.stdout.print(f"[green]✔ {what}[/green]")

    def error(self, what: str, exc: Exception):
        self.stderr.print(
            f"[bold red]❌ Failed to {what}[/bold red] due to: \n\n"
            f"[red]{str(exc)}[/red]"
        )

    def warning(self, what: str, exc: Exception):
        self.stderr.print(
            f"[bold yellow]⚠ Failed to {what}[/bold yellow] due to: \n\n"
            f"[yellow]{str(exc)}[/yellow]"
        )

[trim tangle]

__getattribute__ will be called in the self.silent expression

[slender timber]

I can use self.__dict__["silent"] instead?

[trim tangle]

No, define a __getattr__ method that uses super().__getattr__ or object.__getattr__

[slender timber]

and get rid of __getattribute__?

[slender timber]

says type of both is unknown

it is only letting me use super().__getattribute__ or object.__getattribute__()

[trim tangle]

You can use polymorphism here

from rich.console import Console

class _Output:
    def __init__(self):
        self.stdout = Console()
        self.stderr = Console(stderr=True)

    def success(self, what: str):
        self.stdout.print(f"[green]✔ {what}[/green]")

    def error(self, what: str, exc: Exception):
        self.stderr.print(
            f"[bold red]❌ Failed to {what}[/bold red] due to: \n\n"
            f"[red]{str(exc)}[/red]"
        )

    def warning(self, what: str, exc: Exception):
        self.stderr.print(
            f"[bold yellow]⚠ Failed to {what}[/bold yellow] due to: \n\n"
            f"[yellow]{str(exc)}[/yellow]"
        )

class _SilentOutput:
    def success(self, what: str):
        pass

    def error(self, what: str, exc: Exception):
        pass

    def warning(self, what: str, exc: Exception):
        pass

And have a factory function creating an output object.
If that seems boilerplatey, you can accept the two consoles as parameters, and pass in something like ```py
class SilentConsole:
def print(self, s: str, /): pass

with a factory function as well. This actually would be type-safe, and you won't repeat any code

[oblique urchin]

yeah object has no __getattr__

[trim tangle]

Oh huh

[slender timber]

this successfully made the actual implementation smaller than this stub code 😅

[trim tangle]

Wdym

[slender timber]

the function i made this class for is like 40 lines without docstrings

[trim tangle]

What function?

[slender timber]

the function which uses this _Output class

[trim tangle]

Ah

Can you show it?

[slender timber]

def run(
    flp: pathlib.Path = typer.Argument(..., dir_okay=False, resolve_path=True),
    license: bool = typer.Option(True, "--unlock/--lock", "-u/-U"),
    output: pathlib.Path
    | None = typer.Option(None, "--output", "-o", exists=False, dir_okay=False),
    licensee: str
    | None = typer.Option(
        None, "--licensee", "-l", help="An ASCII string", callback=_validate_ascii
    ),
    silent: bool = typer.Option(False, help="Don't show output or warnings"),
):
    """Modifies an FLP ('unlocks') such that a trial version of FL Studio can open it.

    The default operation is --unlock; it doesn't need to be specified. Use the
    --lock option if you want to do the opposite. An error code of 1 is returned
    if this fails due to any reason and any further operations are cancelled.

    If --output is specified, a new FLP is created at the path specified by it.
    If it isn't, the FLP is overwritten. An error is shown if FLP is unwritable
    with error code 2.

    If --licensee is specified, the registered username encoded inside the FLP
    is modified. If this operation fails, an error is displayed but the operation
    doesn't fail and an error code 3 is returned.
    """
    if output is None and not os.access(flp, os.W_OK):
        raise typer.Exit(2)

    project = pyflp.parse(flp)
    console = _Output(silent)

    try:
        project.licensed = license
    except Exception as exc:
        action = "unlock" if license else "lock"
        console.error(f"{action} {str(flp)}", exc)
        raise typer.Exit(1)

    exitcode = 0
    if licensee is not None:
        try:
            project.licensee = licensee
        except Exception as exc:
            console.warning("set licensee", exc)
            exitcode = 3

    savepath = output or flp
    try:
        pyflp.save(project, savepath)
    except Exception as exc:
        ...
    else:
        console.success(f"Saved to {str(savepath.resolve())}")

    raise typer.Exit(exitcode)

tbh this rerouting of output via a different class always makes the call site arguments ambiguous

if there was a way to monkeypatch Console itself I would do it

[trim tangle]

Create either a console or a SilentConsole of your own

[slender timber]

could just as well have used some global variable to avoid all this, avoid making a class to

but i just don't like using global silent

[rustic gull]

dunder methods such as __str__ are not type-hinted usually, yes?

[trim tangle]

I think they should be

Otherwise mypy will not check them by default

[rustic gull]

gotcha

ty

[tranquil turtle]

__getattribute__ will be called in the self.__dict__ expression 😄

[torpid yarrow]

Is there any way to typehint a literal infinite number? Something like Literal[float("inf")], which of course doesn't work

[soft matrix]

not currently

this might change

see https://github.com/python/typing/issues/1160

[torpid yarrow]

Thanks. Funny coincidence that the issue was stale for months but was just bumped today

[fierce ridge]

is there any work in progress to expose the various convenience types from typeshed in typing? like SupportsWrite used here https://github.com/python/typeshed/blob/94dc53ee90992aef31a4bc6873a2666339f624c4/stdlib/logging/__init__.pyi#L3

[rough sluice]

stdlib/logging/__init__.pyi line 3

from _typeshed import Self, StrPath, SupportsWrite```

[fierce ridge]

https://github.com/python/typeshed/blob/main/stdlib/_typeshed/__init__.pyi#L202-L223
all of this would be super useful in a lot of real-world code, it's a shame that people have to copy and paste it

if not, does this seem like something that would be an acceptable PR to cpython?

[oblique urchin]

https://github.com/python/typing_extensions/issues/6

[fierce ridge]

nice!

[slender timber]

Why do certain libs not have type hints at all?

Its too big a problem to make them work with type hinted code

[oblique urchin]

nobody has done the work yet. some are also difficult to type correctly

[slender timber]

kivy for instance is genuinely unaware about type hints entirely ig, not a single issue on their gh or pr either mentioning type hints

It being a C extension should have atleast something. Even imports aren't detected correctly

[soft matrix]

Be the change you wish to see in the world

:p

[slender timber]

I am not type hinting what seems like 60-70 modules just from their docs 😛

Plus idk if they will even accept (care about) it

[oblique urchin]

typeshed will accept it

[upper pivot]

I'm using flake8 and rather than run it against current file.. I'm wondering if anyone knows how i could run it against git status -s

I want to run it against files i've edited/added

[spiral fjord]

@upper pivot what platform are you on?

git diff --name-only --staged | xargs flake8

[upper pivot]

I'm using windows, WSL2

That doesn't show all files I believe

shows only added file

git diff --name-only HEAD shows both added and modified

[blazing nest]

Then why should they 😔🥴🤷‍♂️

[blazing nest]

True that should make figuring out the types a bit easier, because that means that the API isn't as dynamic. That said, it being a C extensions means that typing needs to be placed in .pyi files which is a bit more of a burden than keeping it in the file

[rustic lagoon]

i'm considering reporting a bug with mypy and pyright, and i'm wondering whether you'd classify it as a bug as well

https://gist.github.com/ehllie/931ff5647c2f25fce1b9a14d0ca9af01

[rustic lagoon]

The issue is that the type checkers can't monomorphise the generic functions returned from higher order generic functions

[velvet spindle]

Hello everyone !
I gonna use Python with my students but I want to get them at good programming practices and for that I would like to use Mypy to force them to strongly type their codes.
We are using Visual studio code for everything.
I did got Mypy to work with the type-hinting into VS code, but it's only visually, I didn't found how to prevent VS Code from running the code if there's hinting.

Does anyone knows how to do this ?

Thanks a lot !

[tranquil turtle]

it is possible to put # type: ignore on every line
also it is possible to annotate everything as Any
in such cases typechecker will not emit any errors and code will be able to run

i dont think it is good idea to prevent code from running, because it will encourage students to bypass the restrictions

also, it violates idea of type-hinting in python. Type-annotations should only be hints, not strong restrictions that are breaking your code

Anyway students always can run from console. To prevent that you should write your own interpreter with builtin type-checking

If students want, they can use some runtime type-checker. For example, beartype or typeguard

[trim tangle]

I still don't really get beartype

[tranquil turtle]

it is not very strict type-checker, because it checks only one element from any collection
but it is fast, so you can use it to catch some stupid errors without very big overhead

[velvet spindle]

If my students wants to cheat it's their bad, but the idea is to help them not to constraint.
We chose Python because a lot of our students already knows it, but we all agreed that for the fundamental course we will do strong typing and we do need the type-checking to at least prevent VS code to run the code.

It is only for educational purpose, of course there's ways to bypass but that's not my question

beartype/typeguard will through errors if there's type-checking errors ?

It isn't possible at all with Mypy ?

[oblique urchin]

if you're using vscode, it will be easier to use pylance/pyright which is better integrated with vscode

[tranquil turtle]

beartype/typeguard will through errors if there's type-checking errors ?
yes, if you annotated function with @beartype decorator
runtime type-checkers dont perform variable type checking, they are only checking functions argument and return value types

[oblique urchin]

and it's very comparable in quality with mypy for typechecking Python

(pyright, not beartype)

[velvet spindle]

Ok thanks, I'm totally willing to switch for Pylance/Pyright
Will it prevent code from running if there's type-checking errors ?

[oblique urchin]

no, that's not generally how the Python type system works

[tranquil turtle]

you should write your own plugin that only allows run code if there is no type errors

[trim tangle]

Well... What's the point of checking if it only checks one random element? If all elements are off it's probably going to die anywyay

Also leads to inconsistent results

Gives you false confidence, kinda

[velvet spindle]

VS code plugin ?
Do you know a plugin that would do it ?
I never coded a plugin for VS code, but I would be willing to dig into it if you think that it will be able to respond to my need

[trim tangle]

If you really want static typing, maybe use a statically typed language?

Python is, fundamentally, dynamically typed

[tranquil turtle]

die earlier is better than die later
so if you pass some random unrelated collection to function, probably it will die after several frames
but if you @beartype'd function, it will die immediately and you will not be confused by long traceback

[trim tangle]

Why do you want to prevent the code from running?

[velvet spindle]

Yeah we debated this a lot with the other teachers, I'm not alone for the choice I would prefer to use another language but that's not a possibility for me

[trim tangle]

So why do you want static typing?

[tranquil turtle]

you can give custom sitecustomize.py to your students
in this file you can patch some builtins, so that every constructed function become wrapped in type-checking wrapper

[velvet spindle]

It's actually a thing that my students asked for, they understand why for the basic we want to insists on strongly typed, but not getting errors at run get them forget a lot of the errors that Mypy throw to them

[oblique urchin]

if I were you I'd just tell the students "your code must pass pyright strict mode" and subtract points from their grade if it doesn't

[trim tangle]

Yeah, let the students choose their own workflow

[velvet spindle]

That's what I plan to, but they asked from me if there's a way for Mypy/VS Code to point the errors more significantly by preventing to run

And here I am trying to find a way to do it

[tranquil turtle]

there is a lot of good code without type annotations
for example, python standard library

[trim tangle]

They're already bright red innit

[oblique urchin]

if they ignore the red squiggly lines, that's on them

[trim tangle]

You could make a script that first runs the code through mypy

[velvet spindle]

Sure that's on them, but if there's a way to help them and they asked for it I try to find it xD

[trim tangle]

If someone wants that, they can make a script or a precommit hook, something like that

[velvet spindle]

I'll take a look thanks !

[trim tangle]

Well... It's not always good 🙂

[tranquil turtle]

you can shadow python command-line command with your wrapper that first runs mypy/etc on file, and if there is no errors, runs actual python interpreter

[brisk hedge]

sitecustomize.py is not necessarily the best plan, since it requires setup and also infects non-course python code.

[tranquil turtle]

agree
but it isn't true that if the code doesn't have type annotations, then the code is bad

[velvet spindle]

That's something I though about, but I'm not entirely sure on how to do it concretely

[brisk hedge]

I would just encourage the use of an editor with type checker support (like VSC) and provide a shell script (or something) that acts as a "stricter" python

[tranquil turtle]

(replying to this)

[oblique urchin]

the sitecustomize idea sounds like a very risky can of worms, would not recommend it

[brisk hedge]

students can then run myscript file.py instead of python file.py

[velvet spindle]

A shell script would means that they wouldn't be able to execute directly into VS Code if I get it right ?

[brisk hedge]

You would need to make a custom build configuration for that to work, yes

something like a premade workspace file

[tranquil turtle]

yes, but i believe in vscode you can set custom path to python interpreter and you can set it to your script

[brisk hedge]

workspaces are portable and require no setup by the students 😄

just open them and write code

it depends on the style of tasks of course

[velvet spindle]

So if I get it right :
Step one : write a script that first runs a lighter and check if there's errors, if not then execute with Python
Step two : setup a VS Code workspace that use that script instead of the python interpreter

?

[brisk hedge]

if you have "template files" that students fill in and complete with their own code, workspaces would synergize quite nicely

[brisk hedge]

That would be basically it

Simple enough

[velvet spindle]

Ok, I gonna try to find how to do this I have a more clear idea of it thanks

I never messed with an editor so far but it shouldn't be too difficult I hope xD

Thanks everyone !

[lavish crow]

collections.abc is triping me out

[acoustic thicket]

why is that

[rustic gull]

I'm creating a few models which look like this:

@model_dataclass
class SavedTracks(Model):
    """Abstracts saved tracks returned by the API."""

    next: t.Optional[str]
    items: tuple[SavedItem]
    total: int

There are a few models which should be considered paged items since they have a next field so I defined a protocol:

@t.runtime_checkable
class PagedItem(t.Protocol):
    """Models that have a next page are considered paged."""

    __slots__ = ()

    next: t.Optional[str]

How do I properly type-hint this function which returns an instance of the same model (which follows the paged item protocol)

PagedItem_T = t.TypeVar("PagedItem_T", bound=PagedItem)

# This doesn't assert that model should be a model's instance
def next(self, model: PagedItem_T) -> t.Optional[PagedItem_T]: 
    return type(model)(self._get(model.next))

[trim tangle]

I don't think I follow

could you give an example maybe?

[rare scarab]

use Self?

[trim tangle]

what object does the method live on?

[rare scarab]

It seems like using Optional is unneeded. doing type(x)(...) will always return a result.

unless you do type(None)()

[trim tangle]

type(x)(...) is usually a bad idea because __init__ tends to be incompatible between subclasses

[rustic gull]

So the context is that I'm wrapping the Spotify API's json responses

class Model:
    ...

    def __init__(self, data: ModelableJSON) -> None:
        """Load the data for all the fields from the JSON data."""
        for field in dataclasses.fields(self):
            self._load_field_value(field, data)

model_dataclass = dataclasses.dataclass(
    slots=True,
    eq=True,
    init=False,
    frozen=True,
)

@t.runtime_checkable
class Linkable(t.Protocol):
    """
    Models that have a name and a url are considered linkable.

    Linkable models inherit this protocol since it provides a string
    representation.
    """

    __slots__ = ()

    name: str
    url: str

    def __str__(self) -> str:
        """Links to the object in Rich's syntax."""
        return f"[link={self.url}]{self.name}[/]"


@model_dataclass
class Album(Model, Linkable):
    """Abstracts albums returned by the API."""

    name: str
    url: str


@model_dataclass
class Artist(Model, Linkable):
    """Abstracts artists returned by the API."""

    name: str
    url: str
    id: str

...

you can call Artist({"name": "foo", "url": "url}, "id": "uri"}) to get an instance, for example.
All these models share the same __init__ since they load json the same way.

The client itself has a next method which should call the endpoint specified by the next field of a model, and create a new instance of the model with the newly returned json

(the actual code)

# client.py
class Client:
    ...
    def next(self, model: ...) -> t.Optional[...]:
        response_data = self._get(model.next)
        if response_data:
            return type(model)(response_data)

I was wondering how I'd club models which have next fields together

Should I just use simple inheritance instead?

class LinkableModel(Model):
    __slots__ = ()

    name: str
    url: str

    def __str__(self) -> str:
        """Links to the object in Rich's syntax."""
        return f"[link={self.url}]{self.name}[/]"


class PagedModel(Model):
    __slots__ = ()

    next: t.Optional[str]


@model_dataclass
class Album(LinkableModel):
    """Abstracts albums returned by the API."""

    name: str
    url: str


@model_dataclass
class SavedItem(Model):
    """Abstracts saved items returned by the API."""

    track: Track


@model_dataclass
class SavedTracks(PagedModel):
    """Abstracts saved tracks returned by the API."""

    next: t.Optional[str]
    items: tuple[SavedItem]
    total: int
``` this seems to be the way to go

[slender timber]

@velvet spindle there's a library that forces type hints validation at runtime

!pip strong-typing

[rough sluice]

strong-typing v0.3.1

Classes to create strongly typed structures in Python

[slender timber]

I don't think this one is it, but it has a similar name

[rare scarab]

There's also pydantic.

[oblique urchin]

this sounds like a primitive version of pydantic

[slender timber]

how can one apply type hint validation for normal functions using pydantic?

[rare scarab]

There's a decorator you can use

[slender timber]

i thought its only for dataclass-esque classes / models

[rare scarab]

https://pydantic-docs.helpmanual.io/usage/validation_decorator/

[slender timber]

nice

[velvet spindle]

Oooh that's interesting, does that mean that we have to import it everytime ?

[rare scarab]

it only applies to data structures

[velvet spindle]

Oh sad

[blazing nest]

You should use (PagedItem[T]) -> T

Make the protocol generic

[blazing nest]

Oh wait do you want to do (Type[PagedItem_T]) -> PagedItem_T?

Hmm, your PagedModel is typed incorrectly with attributes as opposed to a method

[rustic gull]

It means that you need to declare types of variables, parameters, and return values of a function upfront.

[lethal delta]

I'm not sure if I'm doing something wrong, but is there a way to mix specifying that something is a type with a protocol?
I know typeguards exist, but they don't seem to solve this issue:

class DataclassType(Protocol):
  # Using this alone matches both dataclass types and instances
  __dataclass_fields__: Dict

[tranquil turtle]

x: type[DataclassType]
You can assign dataclass types to x, but not dataclass instances

Maybe you can annotate field as ClassVar[dict]

Or add __init__(*a,**k) to protocol definition (I'm not sure if it helps)

[fierce ridge]

i literally just needed this in my app today:

import dataclasses
from typing import ClassVar, Protocol, TypeGuard
import pydantic

class DataclassLike(Protocol):
    __dataclass_fields__: ClassVar[dict[str, dataclasses.Field]]

def _guard_dataclass(obj: object) -> TypeGuard[DataclassLike]:
    return dataclasses.is_dataclass(obj)

_Model = TypeVar('_Model', DataclassLike, pydantic.BaseModel)

and type[_Model] seems to work fine:

@dataclasses.dataclass
class Widget1:
    size: float = 1.0

class Widget2(pydantic.BaseModel):
    size: float = 1.0

w1t: type[_Model] = Widget1
w2t: type[_Model] = Widget2

w1: _Model = w1t()
w2: _Model = w2t()

actually... wait that doesn't quite work

let me tinker with this more

anyway DataclassLike and type[DataclassLike] does work

[oblique urchin]

TypeVars with constraints are rarely the right answer, maybe you want a bound instead

[fierce ridge]

still didn't quite work with a bound

import dataclasses
from typing import Any, ClassVar, Protocol, TypeGuard, TypeVar
import pydantic


class DataclassLike(Protocol):
    __dataclass_fields__: ClassVar[dict[str, dataclasses.Field[Any]]]


def _guard_dataclass(obj: object) -> TypeGuard[DataclassLike]:
    return dataclasses.is_dataclass(obj)


_Model = TypeVar('_Model', DataclassLike, pydantic.BaseModel)


@dataclasses.dataclass
class Widget1:
    size: float = 1.0


class Widget2(pydantic.BaseModel):
    size: float = 1.0


def factory(t: type[_Model]) -> _Model:
    obj = t()
    if _guard_dataclass(obj):
        return obj
    else:
        return obj


w1 = factory(Widget1)
w2 = factory(Widget2)

modeltype.py:29: error: Incompatible return value type (got "DataclassLike", expected "BaseModel")
modeltype.py:34: error: Value of type variable "_Model" of "factory" cannot be "Widget1"
Found 2 errors in 1 file (checked 1 source file)

@oblique urchin

[oblique urchin]

that's not a bound?

[fierce ridge]

oh, lol

let me see if that's just the issue

i thought i did bound= but that might have been in the mypy-play window 😆

[oblique urchin]

I don't think this will work with bound= either though

[fierce ridge]

right, it doesn't, although frankly i'm not sure why

modeltype.py:27: error: Incompatible return value type (got "Union[DataclassLike, BaseModel]", expected "_Model")
modeltype.py:35: error: Value of type variable "_Model" of "factory" cannot be "Widget1"
Found 2 errors in 1 file (checked 1 source file)

[oblique urchin]

are you on 0.990? the second error might be fixed by 0.990

[fierce ridge]

i tried:

def factory(t: type[_Model]) -> _Model:
    return t()

and hoped that maybe this would work:

def factory(t: type[_Model]) -> _Model:
    obj = t()
    if _guard_dataclass(obj):
        return obj
    else:
        return obj

[oblique urchin]

(which was released this week)

[fierce ridge]

this env still has 0.982, let me try to upgrade

with

def factory(t: type[_Model]) -> _Model:
    return t()

i get this under 0.990:

modeltype.py:27: error: Incompatible return value type (got "Union[DataclassLike, BaseModel]", expected "_Model")  [return-value]
Found 1 error in 1 file (checked 1 source file)

[pure sonnet]

guys whats the type for Protocol Copyable?

like does this already exist or should i go with the one i have

class Copyable(Protocol[T]):
  def __copy__(self) -> T:
    ...

ping on reply

[rare scarab]

copy.copy just uses Any

[fierce ridge]

i don't see SupportsCopy in typeshed either

[pure sonnet]

okay so i roll with this

[fierce ridge]

seems reasonable

[soft matrix]

id personally just have it return Self

unless you try

from typing_extensions import TypeVar

CopyT = TypeVar("CopyT", bound=Copyable)
DefaultSelf = TypeVar("DefaultSelf", default=CopyT)

class Copyable(Protocol[CopyT]):
  def __copy__(self: CopyT) -> DefaultSelf:
    ...```

or do this for the very weird case where copy doesnt return Self

[tranquil turtle]

What is default= arg in TypeVar constructor?

!d typing.TypeVar

[rough sluice]

typing.TypeVar

class typing.TypeVar```
Type variable.

Usage:

```py
T = TypeVar('T')  # Can be anything
S = TypeVar('S', bound=str)  # Can be any subtype of str
A = TypeVar('A', str, bytes)  # Must be exactly str or bytes
```  Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function definitions. See [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic "typing.Generic") for more information on generic types. Generic functions work as follows:

[oblique urchin]

!pep 696

[rough sluice]

**PEP 696 - Type defaults for TypeVarLikes**

Link

Status

Draft

Python-Version

3.12

Created

14-Jul-2022

Type

Standards Track

[oblique urchin]

I think pyright already supports it

[tranquil turtle]

Nice, TIL

[oblique urchin]

@soft matrix is the author to be clear 🙂

[chrome hinge]

Here's a weird question... how are the docs for e.g. discord.py generated, specifically the descriptions of each field on each class? Can you generate these from typehints and comments next to them somehow?

[acoustic thicket]

dpy just seems to have them as part of the class docstring https://github.com/Rapptz/discord.py/blob/master/discord/client.py#L216-L219

[rough sluice]

discord/client.py lines 216 to 219

Attributes
-----------
ws
    The websocket gateway the client is currently connected to. Could be `​`​None`​`​.```

[acoustic thicket]

most of them seem to be properties tho

[chrome hinge]

welp, so much for hoping python has some sane way of doing it like rust's autogenerated docs

[chrome hinge]

oh, right, property docstrings - that's a partial answer at least

[slender timber]

Take a look at sphinx-autodoc

Fairly autogenerated, if you want to let it use its own formatting.

mkdocs has mkdocstrings but its not as customizable, but you can then use all sorts of markdown goodies in docstrings if you use it.

[soft matrix]

if you need md you can also use myst

[tranquil turtle]

mypy 0.990 has some weird bug:
Im running two different commands on two versions (0.982 and 0.990):

path/to> mypy file.py
> mypy path/to/file.py

file.py has obvious errors (such as undefined variable and x:int=[]). file.py is module inside package.

On 0.982: im getting report about errors in both cases. It is ok
On 0.990: im getting report about errors only in path/to> mypy file.py case. If i run > mypy path/to/file.py, it prints Success: no issues found in 1 source file. Not ok, i should get errors in second case too.

Im not passing any CLI flags, and i have no config files for mypy.
I on Windows 10, CPython 3.11.0.
I tried pip install mypy==0.990 --force-reinstall, that didnt help.
I have no venv's.
Deleting all .mypy-cache's didnt help.

Im not reporting it on github, because im not sure that it is not my fault.
I'll try to reproduce this in the simplest case, but I'm not sure if I can do it. Could you try this?

[oblique urchin]

sounds like https://github.com/python/mypy/issues/14042. it's fixed on main and we'll do a patch release for it soon

[tranquil turtle]

It's good that you know about it. Could you please remind me the command to install mypy from github?
EDIT: found it: pip install -U git+https://github.com/python/mypy.git

It is fixed on mypy 1.0.0+dev.f78d1fdc154d507353b34e7ea2037ef68de4e6fc (compiled: no). Thanks!

[tranquil turtle]

pyright is crazy

element of this large tuple has this type:

im explicitly annotating it as tuple[tuple[float, float, float], ...]

[soft matrix]

It's going to engage bidirectional inference

Your annotation is "ignored" because Pyright has a better annotation and so meets the two types to the better one

If you want it to be that you need to use cast

[trim tangle]

well, the inferred type is correct innit? 🙂

[tranquil turtle]

yes, it is correct, it is even better
but i didnt expect pyright to make such huge union or literal types

[tranquil turtle]

did mypy become stricter about missing return in empty functions?
before (on 0.982) mypy was happy about this code:

def f() -> int:
    pass

now (on 1.0.0.something) it is complaining: Missing return statement

[trim tangle]

yes

[lethal delta]

I'm not familiar with pydantic and am having a little trouble making sense of your subsequent messages. If you get something working, I'd be eager to hear about it. If you turn it into a module, I'll be eager to link it in docs of projects I'm working on.

Also this seems like a glaring omission in builtin typing that needs a PEP, but I don't have enough familiarity with the process to write one

[lethal delta]

If you have the time, please explain more about this or link anything you think explains it well.

[tranquil turtle]

mypy also become stricter about implicit Optional 😭

[rare scarab]

It's already been more strict than pyright

[oblique urchin]

https://github.com/python/typing/discussions/1080

[lethal delta]

Ah, I see, I misunderstood when I first read your posts. I agree. That form is very confusing.

Thank you for responding.

[tranquil turtle]

why typing.Sequence is not a protocol?

[pastel egret]

Because being a sequence or mapping isn't just about having all the methods - there's also specific behaviours you expect the class to have. For instance that x in sequence means sequence.index(x) won't fail, and iterating gives each item (not a key).

[oblique urchin]

I think the main reason is there's just too many methods

Builtin protocols tend to have very few (3 or less) methods

Sequence has 8 or so

[tranquil turtle]

two methods here are useless, i think

three actually

so, there is only __getitem__ and two non-dunders index and count

no, they are not useless
they are redefined without @abstractmethod decorator, so they behave differently

[acoustic thicket]

lmao

[hallow flint]

--allow-empty-bodies if you want. you can also raise to shut mypy up

[pastel egret]

Or use ...

[lethal delta]

This is probably the best option imo since it's recommended in the PEP

[tranquil turtle]

that doesnt work in latest mypy

it is very annoying

[tranquil turtle]

allow_empty_bodies = True helps, thank you

[slender timber]

i just ran into this

why is mypy getting dumber

[rare scarab]

You can only omit the return in protocols, overloads, and abstract methods

[old dagger]

from inspect import signature
from collections.abc import Callable
from typing import Typed Dict

proxy = Typed Dict("proxy", {'http': str, 'https': str, 'ftp': str})

def func(proxy: proxy, user_agent: str) -> tuple:
    return proxy, user_agent

def parser(func: Callable, annotate: bool=True) -> None:
    params = signature(func).parameters

    for param in params:
        print(param, params [param].annotation)


parser(func)

parser.py:6: error: No overload variant of "dict" matches argument types "Dict[st
parser.py:6: note: Possible overload variants
... (lots of stuff)

Why does mypy complain

[rare scarab]

Did you mean to put a space in TypedDict?

The preferred way to make a typeddict is with a class. ```py
class Proxy(TypedDict):
http: str
https: str
ftp: str

[old dagger]

Its not a syntax error dw. Just discord copy pasting error.

[trim tangle]

can you show the full error?

mypy playground shows no issues:
https://mypy-play.net/?mypy=latest&python=3.10&gist=8b6a19f4462264c1af63d7afd9014b75

[old dagger]

Erm. Its from my pc and the text extract i use didnt do a good job i got an image though

[trim tangle]

What is dict(proxy, user_agent) supposed to do?

[rare scarab]

You should fix your generics

[old dagger]

Oh shi. It was supposed to be just a tuple.

[old dagger]

Wdym

[rare scarab]

tuple

[old dagger]

oh

[rare scarab]

and Callable

could just be Callable[..., Any]

[old dagger]

Is ... valid?

[soft matrix]

Yes

[old dagger]

Couldnt it be Callable[Any, Any]

[soft matrix]

It means var args and kwargs

And they have no names

[rare scarab]

no, callable expects either ... or a list of types

[old dagger]

Oh, because ive seen ... in empty functions too

[rare scarab]

e.g. Callable[[Any], Any] is def x(_: Any) -> Any

[soft matrix]

It'd be easy to misread as Callable[[Any]

[old dagger]

Yeah

I see.

aight thanks

[blazing nest]

I am returning a callback which has two overloads, I need a protocol to correctly typehint this right? I am looking for an intersection type, yeah?

[trim tangle]

Yes, you need a Protocol with a __call__

although returning a callback with two overloads sounds very strange

can you show the code maybe?

[blazing nest]

It is a bit complicated, but I have a decorator which can either be applied to a specific function which then turns into a decorator or a function can be passed to it: ```python
@check()
def verify(value: str): ...

@verify('abc')
def func(): ...

```python
def doublecheck(): ...

@check(doublecheck)
def func(): ...

[trim tangle]

I think I don't quite get it

[blazing nest]

I may be overcomplicating it because @check needs to be overloaded anyways

[trim tangle]

why not have two functions?

they seem to do completely different things

[blazing nest]

@check() accomplishes the same thing in both cases (verify something before running func) but has two APIs

[trim tangle]

Can you show an example maybe? I don't understand what it does

[blazing nest]

@check()
def verify(value: str) -> Callable[..., bool]:
    def predicate(arg: str, *args, **kwargs):
        print('verifying...')
        return arg == value
    return predicate

@verify('abc')
def func(arg: str) -> None:
    print(arg)

func('abc')
# verifying...
# abc

func('xyz')
# verifying...
# ValueError: Check failed

[trim tangle]

Why not have a single interface and do ```py
@check(verify("abc"))

also makes it much easier to unit-test verify

it also makes it clear that the decorator is just a call do check, so e.g. it doesn't perform any side effects or modify the arguments to the function

[blazing nest]

Hmm, thanks. I'll consider it; you do bring up a good point, as the usage is very similar and makes all code involved easier

[slender timber]

I have an abstract property, shouldn't that count?

[unborn sandal]

I have a function that, depending on how the developer chooses to call it, will return different data formats; a pandas dataframe, a pyarrow table, a datatable frame, or a list of dictionaries. How would I type-hint the return? would myfunc(format: str) -> list, pandas.core.frame.DataFrame, pyarrow.lib.Table, datatable.Frame: be valid syntax?

[soft matrix]

!d types-union

[rough sluice]

types-union

Union Type

A union object holds the value of the | (bitwise or) operation on multiple type objects. These types are intended primarily for type annotations. The union type expression enables cleaner type hinting syntax compared to typing.Union.

X | Y | ... Defines a union object which holds types X, Y, and so forth. X | Y means either X or Y. It is equivalent to typing.Union[X, Y]. For example, the following function expects an argument of type int or float:

def square(number: int | float) -> int | float:
    return number ** 2

[oblique urchin]

No, you need a union as @soft matrix mentions. However, returning a union is often inconvenient for callers

Because they have to call isinstance() or similar to narrow the type afterwards

[soft matrix]

generally this sounds like it should be a different function for each format

[oblique urchin]

You can potentially use overloads

[soft matrix]

what does format actually look like? could you instead use an enum or literal strings?

[unborn sandal]

yeah, I suppose this is the right way to do it.. have .to_pandas() , to_arrow, etc functions for each format

this is my project that I'm trying to get setup with mypy

https://github.com/dmidlo/histdata.com-tools

here's the link in the readme for how one would interact with the API

https://github.com/dmidlo/histdata.com-tools#api-other-scripts-modules-jupyter-support

the reason for the complexity on that is that callers can request multiple datasets through a single call to the api. the api returns data in either pandas, arrow, or a datatable frame, or, if the user requests multiple instruments or timeframes, they get returned a list of those instruments and timeframes in the requested format

I'm working in a different branch right now, but this is the method I'm working on

https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/api.py#L106

[rough sluice]

src/histdatacom/api.py line 106

def merge_jays(self, records_current):```

[unborn sandal]

and this is the helper method used by merge_jays to sort and output in the correct format

https://github.com/dmidlo/histdata.com-tools/blob/main/src/histdatacom/api.py#L143

[rough sluice]

src/histdatacom/api.py line 143

def merge_records(self, tp_set_dict):```

[unborn sandal]

I guess I'll try Union | and see if that does what I'm expecting and hoping for

thanks!

*note: this is a project I started to learn python, requests, and concurrency, now I'm trying to learn testing, so I figured getting mypy set up would be a good first step to that end... there's probably a lot of coding horror in here 😛

[unborn sandal]

yeah, totally. Here it's a trade-off. I'm believe I'm giving a good deal of convenience on the front end in exchange, and functionally the caller is deterministically requesting the desired return value... the method doesn't isn't dynamically self-determining its return types. Union it is! thanks everyone!

[trim tangle]

Well, you might as well return Any

Because of all the checks a user will need to do

[young stone]

How assert that the collection is of type ListCollection? (variable) collection: ListCollection | DictCollection

[fierce ridge]

you can use a typeguard

def is_list_collection(val: object) -> TypeGuard[ListCollection]:
    return isinstance(val, list) and all(isinstance(elem, str) for elem in val)

for collection in collections:
    assert is_list_collection(collection)
    ...

actually because of list type variance maybe you should be using type(elem) is str

[young stone]

thanks

[slender timber]

Is using type hints programmatically a good idea if it saves code?

Like using get_args or __orig_bases__ for example?

[soft matrix]

yes i think so

[pure sonnet]

ok so here is what i have:

class Vec(List[T], Generic[T]):
  ...

  def iter(self: Vec[T]) -> Iterator[T]:
    ...


class Iterator(ABC, Generic[T]):
  @abstractmethod
  def next(self) -> Option[T]:
    """Advances the iterator and returns the next value.

    Returns [`NONE`] when iteration is finished. Individual iterator
    implementations may choose to resume iteration, and so calling `next()`
    again may or may not eventually start returning [`Some(Item)`] again at some point.

    Examples:
        >>> a = Vec[int][1, 2, 3]
        >>> it = a.iter()

        >>> # A call to next() returns the next value
        >>> assert some(1) == it.next()
        >>> assert some(2) == it.next()
        >>> assert some('3') == it.next()

        >>> # and then None once it's over
        >>> assert NONE == it.next()

        >>> # More calls may or may not return `None`. Here, they always will.
        >>> assert NONE == it.next()
        >>> assert NONE == it.next()
    """
    raise NotImplementedError

the problem is in the docstring test

a warning is not being raised when i do

assert some('3') == it.next()

and it would be raised, if we hinted that it is of type Iterator[int] generic towards int, and not Any

when i do

>>> it: Iterator[T] = a.iter()

>>> # A call to next() returns the next value
>>> assert some(1) == it.next()

the last line gives a warning, as it should,

but the type of it is not automatically inferred, why?

@ me if you reply

[soft matrix]

@pure sonnet because you are using pycharm

its typechecker is generally very buggy and incomplete

[pure sonnet]

vscode doesnt even check the types in docs

vsc

[soft matrix]

wait sorry what?

you want typechecking inside the docstring?

[pure sonnet]

i have it in pycharm

i write code normally in docstrings, and its checked normally in pycharm,
vsc doesnt even check the code in docstrings

[soft matrix]

youll need to file a feature request probably for that

[pure sonnet]

nonetheless

i agree with the pycharms docs being buggy sometimes

but look i found an anomaly

outside docstrings

why is it wrapped inside type? idk

it is working outside docstrings

ok this is a ughh IDE issue

ok i fixed it

WAIT

i found the error i had

bruh i had Vec[int][1, 2, 3]

should have been
Vec[int]([1, 2, 3])

strict generics in python 😈

[trim tangle]

nope that won't work

if you have a subclass S of str, it's perfectly fine to have [S()] : list[str]

[fierce ridge]

is that covariant then? not invariant?

[oblique urchin]

it is invariant, but invariance is only about the static type

it's still permissible for a list[str] to contain subclasses of str

e.g. x: list[str] = []; x.append(StrSubclass()) is legal

[trim tangle]

oh hey, I actually have a thing for that 👀
https://decorator-factory.github.io/typing-tips/faq/cat-in-animals/

[fierce ridge]

A list being invariant only means that you can't put a list of cats into a variable which stores a list of animals (or vice versa).
aha, this clarifies things immensely. thank you!

i always forget why mutable collections need to be invariant

this is a great explanation

[trim tangle]

mutability kinda complicated everything

[paper salmon]

and i thought i knew invariance before... but i guess it still works out

[tranquil turtle]

from typing import Self
class X:
 @classmethod
 def f(cls) -> Self:
  ...

why i am getting Variable "typing.Self" is not valid as a type error?

im on 3.11
mypy 1.0.0+dev...

it also happened on older mypy

[oblique urchin]

mypy doesn't support Self

[tranquil turtle]

😭

[oblique urchin]

but the next release probably will

[tranquil turtle]

🥳

[blazing nest]

Variance only talks about the relationship between two parametrized generics yeah? Ie. is list[X] assignable to list[Y]

[hallow flint]

yes

(although people still can get a little tripped up when bidirectional type inference magically makes things work, i.e. things that naively look like list[Y] can sometimes be inferred as list[X])

[tranquil turtle]

['a'] can be inferred as list[Literal['a']], but you can use it as list[str]

How it is implemented in type checkers? How they differentiate between actual lists of literals (that are explicitly annotated as list of literals) and inferred lists of literals (that can be used as list of strings)?

[tranquil turtle]

Note this behaviour of pyright

[brisk hedge]

tuples of literal values are aggressively narrowed

[haughty heron]

Hi, how to (and should I?) improve type hints in this example? Line with def wrapper(...) looks like potential case to improve

import functools
from typing import TypeVar, Callable, Any
 
T = TypeVar("T", bound=Callable[..., Any])
 
 
def example(message: str) -> Callable:
    def decorator(f: T) -> Callable[[T], T]:
        @functools.wraps(f)
        def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
            print(message)
            value = f(self, *args, **kwargs)
            print(message)
            return value
 
        return wrapper
 
    return decorator

[empty hull]

I have a TypedDict type called Document and now I want to use it against a generic class, DocWrapper. The generic defines a getitem that gives another instance of DocWrapper but now wrapping the subdocument. How do I accomplish this? And how do I accomplish type hints for the various keys in getitem?

T = TypeVar('T')

SubDoc1Type = TypedDict(...)
SubDoc2Type = TypedDict(...)
Document = TypedDict('Document', { 'subdoc1': SubDoc1Type, 'subdoc2': SubDoc2Type })

class DocWrapper(Generic[T]):
    def __getitem__(self, key: ???) -> Wrapped[self[key]]:
        ...


doc = Document(...)

doc['subdoc1']  # Typehint works and Pycharm tells me it has the key I wanted

wrapped = DocWrapper[Document](doc)

wrapped['subdoc1']  # Typehint doesn't work and IDE doesn't autocomplete keys >.<
# Expected type to be DocWrapper[SubDoc1Type]

wrapped['subdoc2']  # Same as above, expected type to be DocWrapper[SubDoc2Type]

[soft matrix]

creating a wrapper type doesnt really work in the current type system

[empty hull]

F

[hallow flint]

most of the time from context. e.g. if you have f_takes_list_str(["a", "a"]) it won't be inferred as a list of literals because it will use the context from the argument type of f_takes_list_str. similar thing applies if there's an explicit variable annotation. mypy calls this "type context", pyright calls this "bidirectional type inference"

if you have a specific example in mind i can look into how it works the way it does

[midnight niche]

So I have some code which stringifies dictionary values in various ways if they are of a set of types coming from an api (this is discord.py, but that's largely irrelevant for this) and seeing as there are a punch of different possibilities, I've implemented it as a decorator. like so:

class _Stringify:
    act_on = (
        discord.Member,
        discord.Role,
        discord.TextChannel,
    )
    types = typing.Union[
        discord.Member,
        discord.Role,
        discord.TextChannel,
    ]

    def __init__(self, method: typing.Callable[[types], str]):
        self._method = method

    def __call__(self, value):
        if isinstance(value, self.act_on):
            return self._method(value)
        return value

    def for_dict(self, dict_in: dict) -> dict:
        """ Maps over all the values of a dictionary """
        return {
            key: self(value) for key, value in dict_in.items()
        }

I'm not happy with having to define my list of types twice, but trying to unpack like types = typing.Union[*act_on] raises a SyntaxError - is there any way around this?

[oblique urchin]

typing.Union[tuple(act_on)] works at runtime but most type checkers won't like it

however you can do it the other way around, act_on = typing.get_args(types)

[midnight niche]

ah, I wasn't aware of typing.get_args

Thank you

[vivid ore]

Hi, I have a problem
I’m using mypy to typecheck my code, and I have this code:

class dummy(type):
    def __prepare__(metacls, name: str, bases: tuple[type, ...], **kwds: Any) -> Mapping[str, object]:
        return _dummy_dict()

According to mypy, this is wrong

dummy.py:24: error: Signature of "__prepare__" incompatible with supertype "type"
dummy.py:24: note:      Superclass:
dummy.py:24: note:          def __prepare__(metacls, str, Tuple[type, ...], **kwds: Any) -> Mapping[str, object]
dummy.py:24: note:      Subclass:
dummy.py:24: note:          def __prepare__(metacls, name: str, bases: Tuple[type, ...], **kwds: Any) -> Mapping[str, object]
Found 1 error in 1 file (checked 3 source files)

Does anybody understand why? It looks like the signatures match to me…

In fact, Pyright has no problems with it

[oblique urchin]

the problem may be that __prepare__ is marked as a classmethod in typeshed: https://github.com/python/typeshed/blob/main/stdlib/builtins.pyi#L179

[rough sluice]

stdlib/builtins.pyi line 179

@classmethod```

[oblique urchin]

but your subclass isn't a classmethod

mypy tends to give somewhat unhelpful error messages in this situation

[vivid ore]

thank you

[dry orchid]

I'm returning 3 DFs and 3 dicts from a function
how should I add type.
Currently I'm using

def f(
  ) -> Tuple[DataFrame, DataFrame, DataFrame, dict, dict, dict]:

I want to make it shorter

[vivid ore]

no, that doesn’t seem to be it

it now reports

Python/dummy.py:25: error: Signature of "__prepare__" incompatible with supertype "type"
Python/dummy.py:25: note:      Superclass:
Python/dummy.py:25: note:          def __prepare__(metacls, str, Tuple[type, ...], **kwds: Any) -> Mapping[str, object]
Python/dummy.py:25: note:      Subclass:
Python/dummy.py:25: note:          @classmethod
Python/dummy.py:25: note:          def __prepare__(cls, name: str, bases: Tuple[type, ...], **kwds: Any) -> Mapping[str, object]

[soft matrix]

you changed the name of the first parameter

[trim tangle]

If you're returning a 6-element tuple, you probably want a NamedTuple or a dataclass

[oblique urchin]

and a more precise type than dict (what's in those dicts?)

[soft matrix]

what are the rules about an unannotated TypeAlias```py
class Foo: ...
Bar = Foo

is this kind of behaviour deprecated?

[brisk hedge]

if the expression is a valid type

(is type[T])

[hallow flint]

it's a type alias. and not deprecated, at least not by pep 613

[soft matrix]

ok thanks

[dry orchid]

seems good, will try

[dry orchid]

[str, dict | str | None]

the internal dict is mongo o/p, but I'm not sure for the str(not my code)

[slender timber]

is there anyway to use isinstance for generic types without later getting "Type is unknown..." errors?

also is there a way to make type param optional?

[tranquil turtle]

No, but there is pep about it

[tranquil turtle]

Can you provide example? Maybe it is solvable with TypeGuard's

[rare scarab]

If you make your own class, there's a __instancecheck__ and __subclasscheck__ hook

https://docs.python.org/3/reference/datamodel.html#customizing-instance-and-subclass-checks

[slender timber]

I am not always super sure to write extra code just for the purpose of satisfying a type checker 😅

[vivid ore]

Is there an analogue to TypeScript’s unknown, that works like Any, but is assumed to have no methods and attributes?

[void panther]

object?

[hasty jungle]

hello, I have a question about PyCharm IDEA and typing

I have this construct in my repo:

class Foo:
  def __call__(self): print("Hello")

class Bar(Foo): pass

and later I do: Bar()()

PyCharm is yelling at me that Bar instance is not callable. however, I've checked in a console and it is actually perfectly valid, as far as I test it. is there a type hint I could use to make it explicit? or is it just about idea config (in which case I aopologize for the wrong channel usage)

[acoustic thicket]

yeah just a pycharm issue, mypy/pyright should accept it

[hasty jungle]

okay thx!

[terse swallow]

Can I use the overload and class method decorator together?
Thanks

[soft matrix]

yes

[quaint mesa]

class A:
    def __init__(self, param_1, param_2):
        """Initializator description"""

class B(A):
    def func_A(self, param_1, param_2, param_3):
        super().__init__(param_1, param_2)
        self.param_3 = param_3

how can I get same type hint for func_A without pasting it?

[trim tangle]

no, not really

[quaint mesa]

sadge

[fierce ridge]

i really really wish you could

this has been my #1 complaint about python typing from day 1: the inability to tell mypy "please inherit these parameter list and its types"

i suppose it's an issue in other statically typed languages as well, but at least you have better ide support for it

unless pycharm and/or vs code can do this now?

[quaint mesa]

is it ok to paste lets say discord.py constructor description

[quaint mesa]

I'd assume so since its open source but still

[trim tangle]

sure

[fierce ridge]

how 👀

does pycharm have this

[quaint mesa]

I stopped using vsc because it fails to load any type hints if your project is too big

[fierce ridge]

tbh i still want a "please inherit *args and **kwargs" thing in typing

[trim tangle]

You type bar in a child class and it suggests autocompletion. Press Enter and it autocompletes with all the parameters

although it's kinda super janky because if the parent class is in a different module and is using names from other modules, it's kinda all over the place

tbh, if you have a lot of inheritance and long signatures that are actually a pain to even copy, there might be something with your code 🙂

[fierce ridge]

i mostly run into this when extending classes from libraries

like pandas where every method has 20 parameters with several overloads

and now i have to copy literally all of that and keep it in sync

even if my IDE does that for me, it's still a huge pain

[trim tangle]

yeah that's definitely a problem in itself 😄

or maybe it's not, idk

but 20 parameters does seem excessive

[fierce ridge]

the point is that it's sometimes a legitimate need, especially in frameworks that support a lot of options because they're "frameworks" and that just tends to happen in frameworks

i appreciate that it's often a code smell, but if you have a magic *args: typing.Inherit you can call that a code smell without making it extremely punishing in the cases where it's a legitimate thing to do

[trim tangle]

I think it would be reasonable for *args (with no annotation) to mean just that

[fierce ridge]

i can imagine there being objections to this, because you can end up with an Any that way by mistake

whereas def foo(self, *args: Inherit, **kwargs: Inherit, use_magic: bool = True) -> Inherit) is still "explicit" in some sense

[rare scarab]

You should be able to unpack named tuple and typeddict onto args and kwargs

from typing import TypedDict, NamedTuple

class Args(NamedTuple):
  arg1: str
  arg2: int

class Kwargs(TypedDict):
  kwarg1: str
  kwarg2: str

class A:
  def __init__(self, *args: *Args, **kwargs: **Kwargs): ...

It's more useful when used with just kwargs, since I don't think you can have any overlap.

Actually, can you use NamedTuple there, or is it just a regular tuple that's allowed?

[oblique urchin]

pyright doesn't allow it

PEP 646 isn't very precise at https://peps.python.org/pep-0646/#args-as-a-type-variable-tuple

but it only talks about Tuple specifically, not about namedtuples

[rare scarab]

Now that I think about it, namedtuple wouldn't be very useful with *args

[oblique urchin]

yeah at runtime your args would still be a vanilla tuple, not a namedtuple

[rare scarab]

unless you wrap it with args = Args(*args)

but still, what's the point?

[oblique urchin]

huh pyre doesn't even support Unpack for args, and that's supposed to be the reference implementation for PEP 646 https://pyre-check.org/play?input=from typing import TypedDict%2C NamedTuple from typing_extensions import Unpack class Args(NamedTuple)%3A arg1%3A str arg2%3A int class Kwargs(TypedDict)%3A kwarg1%3A str kwarg2%3A str class A%3A def __init__(self%2C *args%3A Unpack[tuple[int%2C str]]%2C **kwargs%3A Unpack[Kwargs])%3A ...

[rare scarab]

I guess most people would prefer to use kwargs for repeated, common parameters

[copper geode]

hello, is there a way to type hint something as a function?

example:

def b():
  pass
def c():
  pass
def foo(func):
  func()

I want to type hint the fact that func is a function

[soft matrix]

Use

!d collections.abc.Callable

[rough sluice]

collections.abc.Callable

class collections.abc.Callable```
ABC for classes that provide the `__call__()` method.

[soft matrix]

Maybe the docs for typing.Callable are more appropriate here

!d typing.Callable

[rough sluice]

typing.Callable

typing.Callable```
Callable type; `Callable[[int], str]` is a function of (int) -> str.

The subscription syntax must always be used with exactly two values: the argument list and the return type. The argument list must be a list of types or an ellipsis; the return type must be a single type.

There is no syntax to indicate optional or keyword arguments; such function types are rarely used as callback types. `Callable[..., ReturnType]` (literal ellipsis) can be used to type hint a callable taking any number of arguments and returning `ReturnType`. A plain [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "typing.Callable") is equivalent to `Callable[..., Any]`, and in turn to [`collections.abc.Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable "collections.abc.Callable").

[copper geode]

oh that's cool, thanks

[hallow flint]

see also https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html#functions

[rare scarab]

If you want something more advanced, you can use a Protocol with a __call__ function

from typing import Protocol

class MyFunc(Protocol):
  def __call__(self, a: str, b: str, /, c: int, d: int, *, e: bool, f: str) -> Foo: ...

def run(f: MyFunc):
  f(...)

[slender timber]

How can I get tuple like narrowing from an enum?

class MyEnum(MyCustomEnum):
    Member = (1, Type1)

# suppose I have a function
Some_dict.get(MyEnum.Member)  # should automatically infer return type as Type1 object(s)

I have asked this a few times, asking again just in case there's some experimental new feature or smth which allows for this? Typing evolves pretty fast

I have to use typing.cast everywhere to get this to not complain and I hate how code unnecessarily gets bigger for just the type checker.
Ofcourse I can use # type: ignore but that might accidentally silence some other errors

[slender timber]

class Channel: ...
class Automation(Channel): ...

CT_co = TypeVar("CT_co", bound=Channel, covariant=True)

def get_model(suffix: str, type: type[MT], ...) -> MT: ...
def load_channel(preset: str, type: type[CT_co] = Channel): return get_model(..., type)
def load_automation(preset: str): return load_channel(preset, Automation)

In load_automation, I want inferred type to be just Automation, not Automation | Channel

[tranquil turtle]

False positive means that tool finds error, but there is no errors actually
False negative - tool finds no errors, but there are errors
True negative/positive - tool works as expected.

False negatives/positives are bugs. True negatives/positives are not bugs.

Am i right?

[acoustic thicket]

yes

[copper geode]

How can I type hint this:

def visit_unary_expr(self, expr: Unary) -> object:
    right = self._evaluate(expr.right)

    match expr.operator.type:
        case TokenTypes.BANG:
            return not right
        case TokenTypes.MINUS:
            right: SupportsFloat
            return -float(right)

What I'm trying to say with this is: if you get to the case TokenTypes.MINUS then assume that right supports float.

right now if I do this PyCharm tells me Expected type 'SupportsFloat', got 'object' instead in the line right = self._evaluate(expr.right)

[trim tangle]

what do you want to do if right doesn't support float?

[copper geode]

I did add another check before that to make sure that it does support it and crash otherwise. I still couldn't properly type hint that tho.

[trim tangle]

how did you add the check?

[copper geode]

def _check_number_operands(operator: LoxToken, *operands: object) -> None:
    for operand in operands:
        if not isinstance(operand, float):
            raise RuntimeException(operator, f"Expected numeric operand. Got '{operand}'.")

Although now that I think about it it's kinda pointless to convert to float if I already checked that it was a float :P

[vivid ore]

I have a problem with mypy. mypy does not use narrowed types inside function definitions.
I have the following code:

from typing import Callable


def foo(a: str | int) -> list[str]:
    x: list[str] = ["abc", "def"]
    if isinstance(a, int):
        x.insert(a, "ghi")
    elif isinstance(a, str):
        x.insert(0, a)
    return x


def bar(a: str | int) -> Callable[[list[str]], list[str]]:
    if isinstance(a, int):
        def modify(x: list[str]) -> list[str]:
            x.insert(a, "ghi")
            return x
    elif isinstance(a, str):
        def modify(x: list[str]) -> list[str]:
            x.insert(0, a)
            return x
    return modify

foo is correctly identified as well-typed.
I believe that bar should also be well-typed, but mypy gives this error:

16: error: Argument 1 to "insert" of "list" has incompatible type "Union[str, int]"; expected "SupportsIndex"
20: error: Argument 2 to "insert" of "list" has incompatible type "Union[str, int]"; expected "str"

[hallow flint]

the reason mypy complains is because generally speaking this kind of thing is not necessarily safe... closures are late binding and a could be reassigned. that said, a is not reassigned (and is immutable), so your code is safe and mypy could have heuristics for this. see https://github.com/python/mypy/issues/2608

[vivid ore]

Thank you

[hallow flint]

https://mypy.readthedocs.io/en/stable/common_issues.html#narrowing-and-inner-functions is the bad case that mypy is worried about

[vivid ore]

Thank you

Do they bind on call or do they bind on return?

[grizzled rapids]

Hey, I asked this in a help channel but it might be more suited here: I'm having trouble getting a library from Typeshed to behave in Pycharm, and I know I'm doing something wrong but I don't know what. I'm trying to get types for pywin32 and installed types-pywin32 to my project, but... now what? Do I need to import it manually and use types from it that way? I'm a bit lost as to how to actually make all this behave

[tranquil turtle]

You shouldn't do anything. It should work after installation as is

[analog fable]

Is there a way to typehint a possible reverse relationship in a django one-to-one?

class A(models.Model):
    thing = models.CharField(max_length=8)
    b: Maybe[B]


class B(models.Model):
    a = models.OneToOneField('A')

I have something where A.b is either an instance of B or it doesn't exist

[hallow flint]

on call

!e```
def f(x):
z = x
def g():
print("!!!!", z[0])
return g
val = [1]
g = f(val)
g()
val[0] = 2
g()
val[0] = 3
g()

[rough sluice]

@hallow flint :white_check_mark: Your 3.11 eval job has completed with return code 0.

001 | !!!! 1
002 | !!!! 2
003 | !!!! 3

[vivid ore]

thank you

I’m getting a very weird error with mypy

Incompatible return value type (got "Union[Callable[[D], T], Callable[..., Callable[[D], T]]]", expected "Union[Callable[[D], T], Callable[..., Callable[[D], T]]]")

…aren’t they the same type?

hm it works if I move the function that I call to be a locally defined function inside the calling function

[lusty drum]

Hi. I'm working on a Django project that has a middleware that extends the HttpRequest object to contain a un attribute subdomain . On my view when I access request.subdomain I get a mypy error alerting that the HttpRequest types does not have the subdomain member.

How can I instruct mypy of the extension made to HttpRequest? How would you handle this issue if you where trying to make the most out of the types?

[tranquil turtle]

You can point mypy to patched typeshed

[acoustic thicket]

what's the code

[vivid ore]

https://gist.github.com/schuelermine/44ba433e81d2a0b4de7d4ed4a3f070b8

error on line 62

[quasi spindle]

How do you check which version mypy is?
This common way appears to not work:

import numpy as np

np.__version__
Out[14]: '1.23.3'

import mypy 

mypy.__version__
Traceback (most recent call last):
AttributeError: module 'mypy' has no attribute '__version__'

[fierce ridge]

you can do it from the command line mypy --version. the __version__ convention is not a requirement, it's just something that library authors offer as a convenience

[quasi spindle]

Thanks, How do I get there from import mypy?

[rare scarab]

you could use the importlib.metadata.version("mypy") function

That will give you the currently installed version of the mypy package

!d importlib.metadata

[rough sluice]

importlib.metadata

New in version 3.8.

Changed in version 3.10: importlib.metadata is no longer provisional.

Source code: Lib/importlib/metadata/__init__.py

importlib_metadata is a library that provides access to the metadata of an installed Distribution Package, such as its entry points or its top-level names (Import Packages, modules, if any). Built in part on Python’s import system, this library intends to replace similar functionality in the entry point API and metadata API of pkg_resources. Along with importlib.resources, this package can eliminate the need to use the older and less efficient pkg_resources package.

[quasi spindle]

Nice that works great @rare scarab , thanks!

[soft matrix]

The version number is in mypy.version iirc

[fierce ridge]

there's no guarantee that it even exists. like i said, it's just something that library authors like to provide for your convenience. however in this case gobot said that they library authors provided it at mypy.version so try that.

[analog fable]

still wondering about this if anyone knows

[haughty heron]

Hi! Is there a way to annotate type of ExternalPackage.variable in File 1? (which does not exist in ExternalPackage)

# External package file
class ExternalPackage: pass

# File 1
def some_hook():
    ExternalPackage.variable = 123

def some_function():
    value = ExternalPackage.variable  # IDE complaining that "variable" does not exist

[tranquil turtle]

def get_package_variable(x: ExternalPackage) -> <your_type>:
    return x.variable # type: ignore

def set_package_variable(x: ExternalPackage, value: <your_type>) -> None:
    x.variable = value # type: ignore


def some_hook():
    set_package_variable(ExternalPackage(), 123) # ok

def some_function():
    value = get_package_variable(ExternalPackage) # ok

[haughty heron]

Nice, thank you!

[fierce ridge]

what does django return if it doesn't exist? it's been too long since i used django, i don't remember anymore 😆

[analog fable]

it's an error

it's not assigned to anything, raises AttributeError? I think?

[fierce ridge]

in that case there's no annotation currently available in the Python type system that indicates "this attribute might not exist"

[analog fable]

Ah ok, thanks. It would have to do some fancy stuff with narrowing after checking with hasattr too.

[fierce ridge]

is there a sensible way to type-annotate a stack frame object? https://docs.python.org/3/reference/datamodel.html#frame-objects

[oblique urchin]

types.FrameType

[fierce ridge]

thanks!

[rain warren]

I have a class that stores a path to a file of either format A or format B. Is it possible to somehow use the type system to ensure I don't conflate these two?
My first idea was to use two NewType wrapping Path and store a Union[newtype_A, newtype_B] in my class. However, these types do not actually exist at runtime, so when I lead the attribute, I can't ask which type it is

[oblique urchin]

make the class generic over two possible types for formats A and B

[fierce ridge]

class A(TypedDict):
    ...

class B(TypedDict):
    ...

_SystemObject = TypeVar("_SystemObject", bound=A | B)

class RemoteObject(Generic[_SystemObject]):
    ...

is this what you're suggesting?