type-hinting

[echo knot]

Because Self will be only loaded for type checkers. Not at Python runtime

[sand merlin]

OK

[chrome hinge]

i'd just use from __future__ import annotations

[echo knot]

Yes! That's also valid

[sand merlin]

I Use C-Python API If User Writes function It is hard to convert it into static.

Language

[rare scarab]

Not as ideal if you're using a framework which depends on runtime types, like pydantic or fastapi. Still possible, but may require more work.

[echo knot]

When they do, I do this kind of atrocity lol:

if TYPE_CHECKING:
    from typing_extensions import override
else:
    override = lambda x: x

In this case because override decorator is always read at runtime by the Python interpreter

[oblique urchin]

I'd encourage you to simply depend on typing_extensions at runtime. It's a very small and widely deployed package with no further dependencies.

[rare scarab]

in that case, just have a hard requires on typing-extensions;python_version<3.12

[echo knot]

Most of the times I do. But in my company there are very specific cases in which there has to be 0 extra dependencies

[rare scarab]

Then tell them to update python

or don't use the typing features that have runtime side-effects

[sand merlin]

Adding typing-extensions as an Optional Dependency. Like MoviePy have https://github.com/Zulko/moviepy/blob/master/setup.py

[rare scarab]

I don't see typing-extensions there

[sand merlin]

E.G How It Add the Optional Dependencies.

[rare scarab]

You would use a marker so it's only installed on python versions it's needed on

[sand merlin]

"You would use a marker" what? what is it?

[rare scarab]

typing_extensions; python_version<3.12

you can add a marker for python version, platform, or cpu architecture

https://pip.pypa.io/en/stable/reference/requirement-specifiers/

[sand merlin]

Learned Something New.

[echo knot]

Question about TypedDict. I understand that using NotRequired would mean that a key MIGHT not be present in the dictionary.

However, from the point of view of type checking, this is only checked when instantiated a dictionary. When getting an item from the typed dict (like dct["key"], there are not any differences whether "key" is Required or NotRequired. At least according to mypy.

Wouldn't it make sense that a warning is raised when accessing a not required key using a literal instead of with .get?

[oblique urchin]

I don't remember if this was discussed before. It might be too noisy to always turn on, but probably makes sense as a strictness option

[echo knot]

Well, and on top of that, for this case, usng get with a literal str that matches a required key should be str, not str OR None

I just think this feature has much more potential that the one given. Maybe it is because it is still in development

I am not sure if that feature is even possible as it seems to be the returned type hint of the get method. This one will produce no extra noise and it would make NotRequired more useful

[oblique urchin]

That should be possible to change in mypy. The way the .get() method works on TypedDicts is hardcoded in the typechecker.

[viscid spire]

I would prefer to use indexing syntax when it is known that the key exists, so that we will get an understandable runtime error when we get bad data, instead of getting something like "NoneType doesn't implement that method". It is common that something with an unknown type, such as dict from json in an API is simply casted to a TypedDict, but you don't know if the site may change their API at some point.

[echo knot]

I agree. I almost never use get because of this same reason

[echo knot]

Do you know if typing.ReadOnly is implemented in mypy? I cannot find any issue related to it and I get some inconsistencies when I play with this one in mypy playground

[oblique urchin]

it is not

[echo knot]

Thanks!

[true totem]

Hi there! Any ideas how to annotate this code?

NO_VALUE = object()

def foo(x: float | None = NO_VALUE) -> float:  # type: ignore
    if x is NO_VALUE:
        x = 1.0
    if x is None:
        x = 2.0
    return x + 1.0

[grave fjord]

It's good to use an enum member as your sentinel then you can use Literal[ThatEnum.member]

[trim tangle]

that behaviour would confuse me a fair bit... why is the None case different from the not-specified case?

[true totem]

Some functions (timeout for example) accepts numbers and None. But I want to use some default from self when there was not passed any value

[true totem]

Interesting point, maybe this will work, thank you

[sand merlin]

Why not Making own class to anotate.

class No_Value(): ...
NO_VALUE = No_Value()

def foo(x: float | None | No_Value = NO_VALUE) -> float:
    if x is No_Value:
        x = 1.0
    if x is None:
        x = 2.0
    return x + 1.0

[trim tangle]

you'd have to do if isinstance(x, No_Value): though

[sand merlin]

Now OK

But Why Not.

[trim tangle]

Because there could be another instance of No_Value that is not NO_VALUE

the new version will not work either, you're comparing an instance to a class

[sand merlin]

Now it Will Act like None. Only 1 Instance. of it will get created.

class No_Value:
    _instance = None
    
    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __repr__(self):
        return 'No_Value'

>>> NO_VALUE = No_Value()
>>> x= No_Value()
>>> NO_VALUE is x
True

[trim tangle]

Perhaps. But a type checker will not recognize this 🙂

[sand merlin]

it shood

[trim tangle]

no, it doesn't automatically understand arbitrary code

[tranquil turtle]

NO_VALUE: Literal[NoValue] = NoValue()

[trim tangle]

pyright-play link

[tranquil turtle]

class _NoValue(Enum):
    x: ...

NoValue: TypeAlias = Literal[_NoValue.x]
NO_VALUE: NoValue = _NoValue.x

if x is NO_VALUE: ... # ok, typechecker understands it
if isinstance(x, _NoValue): ... # also ok
``` does this work?

[sand merlin]

I Did not Understand Your Code.

Can You Explain it.

[tranquil turtle]

i also barely understand it, and im not sure it works

[sand merlin]

From Where Did You copy.

Dose this will work without enclosing Self in quotes "Self".

from typing import TYPE_CHECMING

if TYPE_CHECKING:
    from typing_extensions import  Self
else:
  class Self: ...

[tranquil turtle]

i wrote it -_-

[sand merlin]

Then What is below thing.
i also barely understand it, and im not sure it works

[tranquil turtle]

this is the truth

[tranquil ledge]

It will (for basic annotations, at least), but some linters may complain about Self not being the actual typing Self class wherever it's used.

[sand merlin]

...

[tranquil ledge]

Oh, interesting. I didn't know type-checkers special-case enums for this purpose.

[sand merlin]

Explain ME Barely thing You Know

[tranquil ledge]

Here's a version with more comments. Hopefully it helps.

import enum
from typing import Literal, TypeAlias

# Create a enum with a single member, i.e. a singleton.
_NoValue = enum.Enum("_NoValue", "x")

# Create an alias for the singleton type.
NoValue: TypeAlias = Literal[_NoValue.x]

# Create an alias for the singleton value.
NO_VALUE = _NoValue.x


# Add the singleton type alias as a possible input to account for the default value.
def foo(x: float | None | NoValue = NO_VALUE) -> float:
    if x is NO_VALUE:  # Check if it's the singleton.
        x = 1.0
    reveal_type(x)  # As expected, type of "x" is "float | None"
    if x is None:
        x = 2.0
    return x + 1.0

# This all should type-check.

[rare scarab]

You should prefer the class syntax

[tranquil ledge]

I like how the functional syntax completely disregards the relevance of the enum value (since it is irrelevant in this case beyond being a singleton), but fair enough.

[rare scarab]

from enum import Enum, auto
from typing import Literal, TypeAlias

class _NoValue(Enum):
  x = auto()

NoValue: TypeAlias = Literal[_NoValue.x]
NO_VALUE = _NoValue.x

def foo(x: float | None | NoValue = NO_VALUE) -> float: ...

[tranquil ledge]

Yeah, fair. The typing docs use the class syntax as well, but with 0 instead of auto() for the enum value.
https://typing.readthedocs.io/en/latest/spec/concepts.html#support-for-singleton-types-in-unions

[hasty phoenix]

How can I avoid the incompatible type in the list comprehension?

class A:
    """A class"""
    pass

def fn(v: A|list[A]|None) -> A|list[A]:
    if isinstance(v, list):
        # How can I avoid the incompatible type in this list comp?
        return [fn(k) for k in v]
    # ..do something that produces A
    if v is None:
        return A()  # Some default value
    return A()

[rare scarab]

What's the type of fn?

oh, fn is the function itself.

[rare scarab]

Use an overload. ```py
from typing import overload

class A:
"""A class"""
pass

@overload
def fn(v: list[A]) -> list[A]: ...
@overload
def fn(v: A | None) -> A: ...

def fn(v: A|list[A]|None) -> A|list[A]:
if isinstance(v, list):
# How can I avoid the incompatible type in this list comp?
return [fn(k) for k in v]
# ..do something that produces A
if v is None:
return A() # Some default value
return A()

[trim tangle]

Yeah, the core issue is that fn always returns either an A or a list

You could use an overload, or you could just extract a function that handles a single A

Usually it's better to avoid overloads if they're not required

[hasty phoenix]

Thanks. If given the two alternatives, I think I'd prefer overloads. Having extract function is an altering of the python code for the sake of type hinting. Coding for satisfying the type hinter is a razors edge imho.

[acoustic thicket]

imo its better design to have fn handle a single A regardless of the typechecker

[rare scarab]

single responsibility principle

[hasty phoenix]

I can agree that its a good principle, but there's plenty of examples in py with duck typing. I'm working on typing a package that use several other pypi packages that doesn't have typing. Not all authors are into it, and I've learned first hand that many py packages are very hard to type annotate due to extensive polymorphism. I suppose the discussion if that is a good principle or not can quickly become a fiery one.

[tranquil ledge]

One option might be returning a list no matter what, assuming you're allowed to change the code's functionality at all. If not, never mind.

def fn(v: A|list[A]|None) -> list[A]:
    if isinstance(v, list):
        return [fn(k) for k in v]
    # ..do something that produces A
    if v is None:
        return [A()]  # Some default value
    return [A()]

[crisp steppe]

If I have a pytest fixture returning a MagicMock with a spec of a type I manage, what type would we say my fixture returns? MagicMock? Or the type I specc'ed? Or is there a MockProtocol[MyType] that applies more appropriately?

[trim tangle]

is there a MockProtocol[MyType] that applies more appropriately?
No, that's not possible right now
I think it depends on the particular circumstances, but I'd probably declare it as returning MagicMock just because it has some extra methods IIRC?

like assert_called

[crisp steppe]

aye it does, but then if I want to check method results it's a bit faffy

but yeah MagicMock it is.

[trim tangle]

I mean, I'd avoid mocks if possible 🙂

[crisp steppe]

sure. DI is sweet but we aren't using DI for our google pubsub client. ergo... mock

oof. using with patch(...) as mockedObject: yield mockedObject means the type needs to be Generator[MagicMock, None, None] or Iterator[MagicMock].

[trim tangle]

yep

I'd go for Iterator if you don't need send or throw

[crisp steppe]

i dont. and i prefer Iterator since I find [, None, None] ugly as sin 😛

[cinder bone]

Does anyone have any good resources to learn more "advanced" typing stuff? Things like maybe co/contra/invariance, Generics, etc.

[trim tangle]

I have this page on generics and variance specifically: https://decorator-factory.github.io/typing-tips/tutorials/generics/ though it's mildly stillborn

there's also https://typing.readthedocs.io/

Actually I should pin this because I always forget it exists

https://typing.python.org/

[cinder bone]

Ooh thanks

[viscid spire]

you should do from __future__ import annotations if it is supported in your version

that makes all annotations strings at runtime

[sand merlin]

I don't think it is Good Idea.

[viscid spire]

it will be the default at some point soon

[sand merlin]

How?

[viscid spire]

wdym "how"

!pep 649

[rough sluice]

**PEP 649 - Deferred Evaluation Of Annotations Using Descriptors**

Status

Accepted

Python-Version

3.13

Created

11-Jan-2021

Type

Standards Track

[sand merlin]

@viscid spire Dose not the #!py from __future__ import annotations messes with the Code, dose pydantic will function.

[viscid spire]

what

[sand merlin]

@viscid spire dose pydantic will work. after the ``from future import annotations`

[viscid spire]

no I don't think newer versions might handle it properly

[sand merlin]

ok

I thing it will be good if.

from typing import TYPE_CHECMING

if TYPE_CHECKING:
    from typing_extensions import  Self
else:
    from __future__ import annotations

[viscid spire]

what

[sand merlin]

Or

try:
    from typing_extensions import  Self
except :
    from __future__ import annotations

[viscid spire]

you cannot import __future__ except on the first line

[sand merlin]

OK.

Dose PYPY set TYPE_CHECMING to true or False. I know that cpython interpreter keeps it false.

[trim tangle]

TYPE_CHECKING is always false at runtime

[sand merlin]

https://tenor.com/view/smiling-smiling-cat-cat-cute-kitty-gif-12097880735853048722

[tranquil ledge]

Just want to point out that future annotations and PEP 649’s model for annotations aren’t the same thing, though the latter will support stringified annotations and thus maintain backwards compatibility with that future import.

[viscid spire]

they will have the same effect in the simple case is what I was thinking

[tranquil ledge]

Yeah, fair enough.

[hasty phoenix]

It seems mypy and pylance is unable to infer the type relationship:

TD = TypedDict('TD', {'a': int, 'b': str})
def fn(a: TD|dict[int, TD]):
    for p in a:
        if isinstance(p, int):
            # Mypy doesn't seem to infer that p must be dict[int, TD] from the int hint
            v = a[p]
            reveal_type(v)  # "Unknown | TD", expected TD

[trim tangle]

yeah, it's a tricky thing to reason about
if you strongly feel that this is common enough to be worth the extra complexity in the type checker, you can file an issue for mypy or pyright

[hasty phoenix]

I'm conflicted about it. TypedDict have its quirks. This is one of them. Another is that keys must be Literal. Perhaps the best solution (long term) is to migrate away from them and create distinct dict-like objects where type ambiguity is less...

[trim tangle]

i mean, this is not really a "quirk", I don't know any other language that would do this kind of inference (except for very advanced systems like agda)

It might be easy for a human to make these general judgements, but I don't see how something like this would be accounted for in a type checker without just hard-coding this rule

But yeah, I would avoid TypedDict unless you really need a dict for some reason.

a dataclass would work better in 99% of cases

[hasty phoenix]

I don't know how deep type searching is done when dealing with type unions. The isinstance() statement does only match one of the keys

Yeah, I'm type annotating an old python package which use dicts for all and everything

It's hard to not touch functional behavior

[trim tangle]

The issue is that a type checker would need to combine these facts:

• p is an int in this branch
• p came from iterating over a's keys
• keys of a TypedDict are always strings
• a is either a TD or a dict[int, TD]
  and then make the conclusion that a (a variable not directly linked to p) can only be a dict[int, TD]

[hasty phoenix]

Yes, I think the breaking link here is that it doesn't follow the type of the dict key back to the selection of the union of dicts

I tested it with def fn(a: dict[int, T]|dict[str, S]) and it doesn't work either. The key is never use to infer the type. Even v=a[0] doesn't resolve if its T or S.

Well, then I learned something new today 🥳

[trim tangle]

yeah current type checkers don't do this type of "backtracking" at all

[trim tangle]

Oh, there's a fifth fact I missed. An object cannot be a str and an int at the same time

though... is that the case for all of Python? Or just a CPython quirk?

If it is a quirk, then the fifth fact is "keys of a TypedDict are str literals, so they cannot be an int"

[hasty phoenix]

And thus any TypedDict types would be excluded as type candidates

[trim tangle]

yes

multiple inheritance does make things more complicated sometimes

Pyright says this: ```py
def f(xs: int) -> None:
if isinstance(xs, str):
reveal_type(xs) # <subclass of int and str>

But mypy doesn't say anything, because reveal_type doesn't do anything inside an allegedly unreachable path

[hasty phoenix]

heh, curious

[trim tangle]

so mypy does know that int and str are incompatible, huh

[hasty phoenix]

While working on type hinting older py code, I'm torn between the added code impact of type annotation vs. the added convenience and niftiness of having annotations to help development and use. It is nearly impossible to not affect code in one way or another. -- which is another way of saying that either type hinting is incomplete as a descriptive system or they imply a coding standard/methodology which not all code abide to. And I think its a mix of both.

[trim tangle]

well, you have to ask why you're annotating the code

[hasty phoenix]

jep

[trim tangle]

it will kind of "warp" the code, annotating an existing codebase is not easy because it hasn't been written with a typechecker in mind (same with testing)

[hasty phoenix]

indeed. I've spent hours

[trim tangle]

But your program will work fine, even long term, if you don't have type annotations. (whereas with testing, if you don't have tests you are kind of doomed in the long term)

[hasty phoenix]

But I've gained much better understanding of the code, found a couple of bugs and corner cases (which is a huge motivation for type annotation) and it's a part of the path moving forward.

I suppose the start point in my specific case was that I was tired of not having proper type navigation in the IDE

[lunar dune]

I think that's because of a more general bug where mypy fails to consider the possibility of multiple inheritance when doing reachability analysis. Not mypy doing something clever there 🙂

[trim tangle]

class Foo: pass
class Bar: pass

def f(x: Foo) -> None:
    if isinstance(x, Bar):
        reveal_type(x)  # Revealed type is "__main__.<subclass of "Foo" and "Bar">"

that __main__. prefix is wack, but it does work

[lunar dune]

Oh yeah, it does do those ad-hoc intersections with isinstance(). Hmm.

Maybe it has some special-casing for builtin types somewhere?

[trim tangle]

probably

[hasty phoenix]

That is a bug isn't it? Foo and Bar are unrelated, so the isinstance() test should be definitive.

[trim tangle]

multiple inheritance

class Baz(Foo, Bar):
    pass

!e
for str and int it does cause a runtime error

class Strumber(str, int):
    pass

[rough sluice]

@trim tangle :x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 1, in <module>
003 |     class Strumber(str, int):
004 | TypeError: multiple bases have instance lay-out conflict

[trim tangle]

interestingly it also errors on pypy and graalpython

[lunar dune]

Yeah, to be able to do that analysis statically in a general way you need to know exactly which __slots__ each class has. If two classes both define __slots__, the __slots__ of the two classes are unequal, and at least one of the two classes had a nonempty __slots__, then you can't have multiple inheritance from the two classes

[trim tangle]

I am glad that I don't maintain any type checkers

[lunar dune]

But we don't generally include __slots__ definitions in typeshed, so mypy can't be doing the analysis in a general way using __slots__ to determine that multiple inheritance is impossible between str and int

Which leads me to believe that mypy is probably special-casing builtin types somewhere if it knows that multiple inheritance is impossible in this case

[hasty phoenix]

Aha, so an object can have multiple supers, the type hint does not? E.g. def f(x: Foo) may well be passed a which is a type inherited from Foo and Bar? So the type hint of f(x) does not need to declare that its dependence on Bar?

[trim tangle]

An object that is both a Foo and a Bar can be used whenever a Foo is expected

[hasty phoenix]

Yes, but you test for Barjust out of the blue

[oblique urchin]

I don't think it's special casing. It knows that if you have a subclass of both str and int, you'd have some incompatible methods: https://mypy-play.net/?mypy=latest&python=3.10&gist=5aace42c72ec0db57100c981a62686aa

And clearly nobody would ever do that

so the subclass can't exist

[trim tangle]

oh lol

[trim tangle]

well, it is a perfectly valid check

x: Foo doesn't prevent x from also being a Bar

[hasty phoenix]

It is, but it's undeclared (imho).

[trim tangle]

well, we don't have intersection types, so you can't declare it 🙂

even then, Foo | (Foo & Bar) would be kinda strange since it's exactly the same as Foo

[hasty phoenix]

right. I see. There is no way to declare this

[trim tangle]

I don't really enjoy that there's a type that mypy/pyright can infer and show to the user, but there's no way for me to spell out that type

but oh well

[lunar dune]

Oh right, yeah 🙂

[lunar dune]

Of course not, nobody would ever do an unsound thing with python

[oblique urchin]

At least some of the methods mypy complains about seem compatible though. For example, it complains about __add__, but I could write my StrInt class so that StrInt + str gives you a str and StrInt + int gives you an int, and I don't think it'd be incompatible

[lunar dune]

surely StrInt + int, StrInt + str, int + StrInt, str + StrInt, and StrInt + StrInt should all produce a StrInt

[oblique urchin]

https://github.com/python/typeshed/blob/2e240c9b0892f86cd115c208ca1963c735214c57/stdlib/builtins.pyi#L573

[rough sluice]

stdlib/builtins.pyi line 573

def __add__(self, value: str, /) -> str: ...  # type: ignore[misc]```

[lunar dune]

Yeah but StrInt.__radd__ gets called instead of str.__add__ because the subclass's methods always take priority in binary operations, right?

[oblique urchin]

my point is that the only promise I have to fulfill is that adding a str to a StrInt gives a str

though I haven't thought too hard about how reverse ops play into that

[lunar dune]

Yeah I wasn't disagreeing with you, just quibbling with minor implementation details of the proposed StrInt class 😄

[hasty phoenix]

Interesting. mypy even fails on cast assignment. Here I was lead to believe that cast existed for this very purpose, but I'm evidently wrong 😇

def fn(a: int):
    # Error: Incompatible types in assignment (expression has type "str", variable has type "int")
    a = cast(str, a)

[trim tangle]

That's unrelated to cast, b = cast(str, a) would work

[tranquil turtle]

this fails not on cast
it fails because of a = some_str where a: int

[hasty phoenix]

Is there a way to redefine a var, like a in this case, and have it change type from this point on?

[tranquil turtle]

there is some option that allows reassignments with different type, but i cant remember how and when it works

  --allow-redefinition      Allow unconditional variable redefinition with a new type (inverse: --disallow-
                            redefinition)

[hasty phoenix]

Oh, only as a global option

My usecase is working with dict objects. I try to use TypedDict where possible. Sometimes a dict is mutated to something else that doesn't fit and I need to cast it to another TypedDict type. I probably need to dob = cast(T, a) thing to do it, but hoped to avoid it due to the code change.

Another thing is that some authors tend to reuse vars within a function for different things, which gives mypy a fit sometimes 😄

[terse sky]

Are you forced by existing APIs to use dict?

I would be trying very hard to use dataclasses instead of types dict where reasonable; can also convert to/from dict fairly easily

I've always viewed TypedDict as mostly for legacy code and a few specific use cases rather than someone I'd be eager to put down anywhere

Ah I see you are probably working with legacy code

[sand merlin]

How do i type hint python generator.

from typing_extensions import Generator
def _() -> Generator["What to put here", "What to put here", "What to put here"]:
    yield 1

[paper salmon]

Generator needs the yield, send, and return types

though for a generator as simple as your example where send/return isn't used, you might consider using the Iterator type instead

i.e. ```py
from typing import Iterator

def foo() -> Iterator[int]:
yield 1
yield 2```

if you're still interested in Generator, the docs has an example of it too
https://docs.python.org/3/library/typing.html#typing.Generator py def echo_round() -> Generator[int, float, str]: sent = yield 0 while sent >= 0: sent = yield round(sent) return 'Done'

[sand merlin]

What hapens on line 4.

[paper salmon]

it raises StopIteration with .value set to that value

oh wait were you referring to the second yield statement?

[sand merlin]

yes

[paper salmon]

!e here's an example of using the generator: ```py
def echo_round():
sent = yield 0
while sent >= 0:
sent = yield round(sent)
return 'Done'

gen = echo_round()
value = next(gen) # equivalent to gen.send(None)
print("Generator yielded:", value)

print("Sending 2.6")
value = gen.send(2.6)
print("Generator yielded:", value)

print("Sending 3.14")
value = gen.send(3.14)
print("Generator yielded:", value)

print("Sending -1")
try:
value = gen.send(-1)
except StopIteration as e:
print("Generator returned:", e.value)```

[rough sluice]

@paper salmon :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | Generator yielded: 0
002 | Sending 2.6
003 | Generator yielded: 3
004 | Sending 3.14
005 | Generator yielded: 3
006 | Sending -1
007 | Generator returned: Done

[paper salmon]

fyi this is getting a bit off-topic for type-hinting and is better suited for #python-discussion or #1035199133436354600

[sand merlin]

Thanks I dident know that python generator can sent and return value.

[sour kestrel]

Is there a way to have the following work for union types?

from typing import TypeVar, Union, cast
    
T = TypeVar("T")
    
def do_something(annotation: type[T]) -> T:
    data = ... # network data coerced into T or raises
    return cast(T, data)

reveal_type(do_something(Union[str, int]))

E main:9: note: Revealed type is "Never" (diff)
E main:9: error: Argument 1 to "do_something" has incompatible type "object"; expected "type[Never]" [arg-type] (diff)

reveal_type(do_something(list[Union[str, int]))

main:9: note: Revealed type is "builtins.list[Union[builtins.str, builtins.int]]"

What does the Never refer to in the context of those error messages?

[sand merlin]

@sour kestrel Can You provide more context over your problem.

[sour kestrel]

its basically the same as pydantics parse_obj_as, receive a type and some data, coerce the data into an instance of the type and return it, or raise an error if the data cannot be coerced

the internals can handle discriminating unions, but the typing cannot seem to handle it

[sand merlin]

from typing import TypeVar, Union, cast

T = TypeVar("T")

def do_something(annotation: type[T]) -> T:
if annotation == str:
data = "some string"
elif annotation == int:
data = 42
else:
raise ValueError(f"Unsupported type: {annotation}")
return cast(T, data)

reveal_type(do_something(str)) # Revealed type is 'builtins.str'
reveal_type(do_something(int)) # Revealed type is 'builtins.int'

[sour kestrel]

Yes, it works for simple types, and even generic collections - specifically the issue is when do_something() receives Union[int, str] (or any union for that matter)

So I want reveal_type(do_something(Union[str, int])) to return Revealed type is "Union[builtins.int, builtins.str]"

[sand merlin]

It is Due to Limitation of Python's type system.
typing Module is for static analysis and type checking, and it dose not support full runtime type introspection.

reveal_type function is from MyPy which is static type checker when you use reveal_type, it is for static type checking and it reveals the type that mypy infers based on the provided code.
E.G

reveal_type(do_something(Union[str, int]))

Mypy sees the type as Never, meaning do something(Union[str, int]) can't be reached. That's because Union[str, int] isn't an actual type you can create, but more of a hint about what types could be used.

You need to override the function.

for difrent types.

[trim tangle]

If Union[str, int] is not assignable to a type, mypy should give an error and not silently treat it as Never

[sand merlin]

yep, it throwed error.

[pastel egret]

I guess the logic is that because there is no possible type that can be a union, it produces Never.

[trim tangle]

There's this
python/mypy#9773

[mighty linden]

GitHub

[mypy] #9773 TypeForm[T]: Spelling for regular types (int, str) & special forms (Union[int, str], Literal['foo'], etc)

[sour kestrel]

Thanks that's helpful

[trim tangle]

You can use an ungodly hack like this

from typing import TypeVar, Union, cast, TYPE_CHECKING, Generic

T = TypeVar("T")

if TYPE_CHECKING:
    class Spec(Generic[T]):
        pass
else:
    class Spec:
        def __class_getitem__(cls, item):
            return item
    
def do_something(annotation: type[Spec[T]]) -> T:
    data = ... # network data coerced into T or raises
    return cast(T, data)

reveal_type(do_something(Spec[Union[str, int]]))  # str | int

[sour kestrel]

Certainly an interesting approach @trim tangle - but maybe not the best UX for users to have to wrap their annotation like that.. plenty of leads for me in that GH issue though so thanks for that.

[trim tangle]

Yeah, it's an abomination

[sour kestrel]

This is a similar take from discussion in that issue: https://github.com/python/mypy/issues/9773#issuecomment-1058366102

[grave quartz]

I have a generic type class MyClass(Generic[_X, _Y]). There are a number of @classmethod functions that have a signature like def foo(cls: Type[_MyClass], x: _X, y: _Y) -> _MyClass (i.e. it returns an instance of subclass, and accepts _X and _Y).

I want to support _Y being effectively an "optional" type, in that if it is specified as None then all of the methods on the class lose the y parameter:

class SubWithoutY(MyClass[str, None]): pass  # ideally None wouldn't be needed, but I don't think that's possible
class SubWithY(MyClass[str, int]): pass

SubWithoutY.foo("foo")  # foo is type hinted with no Y parameter
SubWithY.foo("foo", 1)  # type hinted with the Y parameter here

The only solution I can think of is to do something like this, but I'd prefer to keep usage to a single class if possible:

class _MyClassImpl(Generic[_X, _Y]):
    @classmethod
    def foo(cls: Type[_MyClass], x: _X, y: _Y = None) -> _MyClass:
        # actual implementation

class MyClassX(_MyClassImpl, Generic[_X]):
    if TYPE_CHECKING:
        @classmethod
        def foo(cls: Type[_MyClass], x: _X) -> _MyClass: ...

class MyClassXY(_MyClassImpl, Generic[_X, _Y]):
    if TYPE_CHECKING:
        @classmethod
        def foo(cls: Type[_MyClass], x: _X, y: _Y) -> _MyClass: ...

And then use MyClassX / MyClassXY downstream. Any ways to avoid splitting downstream usage?

If mypy was able to pick up __class_getitem__ or __getitem__ on metaclasses, then it'd be solved but to my understanding it only can use Generic for that?

[tranquil ledge]

Hmm, I would’ve expected something like this to work. Wonder why it doesn’t.

[oblique urchin]

The type parameter having a default doesn't mean the method parameter also has a default

[tranquil ledge]

Wouldn’t _Y being Never imply that anything with it as an annotation doesn’t need a default value, since it’s not meant to be used? That’s how I understood Never as a parameter annotation at least.

[oblique urchin]

Never as a parameter annotation means no values are legal

so it's impossible to call that method

[tranquil ledge]

Hmm. Interesting. I swear I’ve seen this work before, where Never effectively disallowed some parameters from being passed in, but so long as that condition was met, method calls would pass type-checking fine. Maybe I’m missing something.

[oblique urchin]

presumably those were optional parameters

https://pyright-play.net/?pythonVersion=3.13&strict=true&code=GYJw9gtgBALgngBwJYDsDmUkQWEMoByApgG5EgBQFAJkcFMABQBUANFAB4Bchp5UAXigA6Ue2bMA1gHcAhiDQBnHqhgBKLhSjaoCWYsVUmHAQGY1FJnDMWgA

[tranquil ledge]

My instinct is still that if a parameter is annotated as Never, then there should be no expectation by a type-checker that a value must be passed in nor that the parameter must have a default value. I’ll do some reading to try to rectify that mental block, I suppose. Cheers.

[oblique urchin]

if a function has two required parameters then callers must pass two arguments, regardless of what the annotations are

[tranquil ledge]

Ohhh. I see it now, gotcha.

[rose mist]

Greetings,

Hope all are well. How can I set --allow-redefinition to True in mypy config?
I tried making a mypy.ini, and putting it there but it still flags no-redef.

[mypy]
allow_redefinition = True

[wraith linden]

Do you have an example of code that gets flagged?

[rose mist]

class Template(ABC):
    __init__(self):
        a: int = 2

class Sub(Template):
    __init__(self):
        a = 3.2

Issue is not the example, issue is that mypy still flags it even though it is explicitly told not to.

I find it really troublesome to deal with, and it doesn't really help, so would like to suppress it globally.

Another example :

[wraith linden]

I think --allow-redefinition is intended more for this kind of situtuation: ```py
response = input('enter a number: ')
response = int(response)

[rose mist]

I don't imagine so, on their documentation it says for class redef, function redef, method redef, and object redef.

if backend is None:
            # If no backend is provided, use the AerSimualtor
            backend: BackendSampler = BackendSampler(AerSimulator())

For instance this.

It doesn't let me redefine backend if it is None.

You see how hard it makes things? I'd lose the ability to have optional args.

Name "backend" already defined on line 892Mypyno-redef

I want it to look at the latest definition when checking stuff.

[tranquil turtle]

do you need : BackendSampler part?

[rose mist]

Yes, it helps the checker for the rest of the parts.

Again, I don't want advice on changing my code so it doesn't get flagged, I want to just have it not flag.

[wraith linden]

Is the context something like the following? ```py
def foo(x: int | None):
if x is None:
x = 123
...

^ In this code, the type checker will infer after the if block that the type of x is now narrowed to just int. You wouldn't re-annotate the type of x.

If you could change the type of a variable inside an if, then the type checker wouldn't necessarily be able to statically determine what the type of the variable is after that point.

[rose mist]

Can you join vc for a second, I can show you why I need to reannotate.

[low elm]

does anyone know the right way to use mypy with sqlalchemy 2.0 All of the info I'm finding online is about v1.4

[fierce ridge]

nothing special required, just make sure you're using mypy installed in your project environment and not another mypy

sqlalchemy 2.0 includes its own type stubs

@low elm for the mypy plugin with more functionality, see https://docs.sqlalchemy.org/en/20/orm/extensions/mypy.html

[low elm]

thanks, I think the problem is with vscode. It's giving me this error: Cannot find implementation or library stub for module named "sqlalchemy" Mypy(import-not-found)

and I'm using 2.0.29 of sqlalchemy

[fierce ridge]

It sounds like VS Code isn't using the right Python / Mypy executable

Make sure it's set up to use the one in your project venv

(Hopefully you are using a venv)

[low elm]

I am and it's using the right venv. I use pyenv and pipenv. I'll read up on the vscode extension, it's doing something weird because mypy is working as expected when I run it from the command line. Thanks for helping!

[eternal surge]

from typing import Any, TypedDict


class ExampleDict(TypedDict):
    x: str
    y: list[int]


def example_filter(arbitrary_dict: dict[Any, Any]) -> ExampleDict:
    match arbitrary_dict:
        case ExampleDict():
            return arbitrary_dict
        case _:
            raise ValueError()

In this example, the case ExampleDict(): line is raising a type warning on pyright: Pattern will never be matched for subject type "dict[Any, Any]"
Why exactly will dict[Any, Any] never be matched as ExampleDict?

[rare scarab]

ExampleDict is effectively dict() at runtime.

Is it just a typing error, or does it not work at runtime?

[eternal surge]

If I try to run the function it gives the following error for the case ExampleDir() line

  File "C:\Users\Dextop\AppData\Local\Programs\Python\Python312\Lib\typing.py", line 2901, in __subclasscheck__
    raise TypeError('TypedDict does not support instance and class checks')
TypeError: TypedDict does not support instance and class checks```

[rare scarab]

You can't use typeddict in match

Maybe you sould use a TypeGuard instead

def is_example(arb_dict: dict[Any, Any]) -> TypeGuard[ExampleDict]:
  match arb_dict:
    case {"x": str(), "y": list() as y} if all(isinstance(item, str) for item in y):
      return True
    case _:
      return False

the if part is optional since it will make the function O(n) instead of O(1)

Simplified: ```py
def is_example(arb_dict: dict[Any, Any]) -> TypeGuard[ExampleDict]:
return "x" in arb_dict and "y" in arb_dict

[eternal surge]

I see! Thank you for the solution, though I wish I could use TypedDict in the match/case, which would save me the trouble of defining ExampleDict twice, essentially

[rare scarab]

then you can do ```py
if is_example(my_dict):
reveal_type(my_dict) # ExampleDict

Consider using a dataclass if you want that.

[eternal surge]

I don't really control the dictionaries I get (hence dict[Any, Any]), so unfortunately I can't use dataclasses here

[rare scarab]

btw, this is effectively what you need. ```py
def example_filter(arbitrary_dict: Any) -> ExampleDict:
match arbitrary_dict:
case {"x": _, "y": _}:
return arbitrary_dict
case _:
raise ValueError()

[eternal surge]

Yep, this works!

[elfin crane]

i wanna check if something is iterable and well

i used collections.abc.Iterable

but when checking against a range object for example, it outputs false

[tranquil ledge]

!e

from collections.abc import Iterable
test = range(10)
return isinstance(thing, Iterable)

Well, ignoring my fail with the bot, this should work, since range objects implement the Iterator protocol.

[elfin crane]

tried

i apparently did the mistake of doing

isinstance(range, Iterable) and thats why it outputted false

[muted iron]

It fails, because you're using return outside of a function body.

[buoyant swift]

the range class itself isn't iterable. instances of range are. though you can check issubclass

[rare scarab]

The way to test if something is an iterable is to call iter() on it.

[viscid spire]

🤔

[rare scarab]

def is_iterable(obj: Any) -> TypeGuard[Iterable]:
  try:
    iter(obj)
  except TypeError:
    return False
  else:
    return True

[viscid spire]

I guess that covers cases that Iterable doesn't really work well for, such as only implementing __getitem__ that takes ints

[fierce ridge]

!e ```python
from collections.abc import Iterable
print( isinstance(range(5), Iterable) )

[rough sluice]

@fierce ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

True

[fierce ridge]

https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=19c40ad4628bec5b377e53acea7a6d02

!e ```python
from collections.abc import Iterable
print( issubclass(range, Iterable) )

[rough sluice]

@fierce ridge :white_check_mark: Your 3.12 eval job has completed with return code 0.

True

[fierce ridge]

@elfin crane issubclass is what you want

[viscid spire]

I think the mistake was in not making a range object, not in the function being used

[cinder bone]

Is there a good way to type Proxy objects? Right now I'm doing something like

def foo(self) -> ProxyForSelf | Self: ...

[acoustic thicket]

from abc import ABC
class Base(ABC):
    x: int

class SubA(Base):
    @property
    def x(self) -> int:
        return 7
    @x.setter
    def x(self, _):
        pass

# "x" overrides symbol of same name in class "Base"
# "property" is incompatible with "int"

is this a pyright bug?

mypy strict doesn't complain

[elfin crane]

nvm found it

[grave fjord]

It's weird to define a concrete attribute on an ABC, and I think you should have to define a deleter to make it count as a valid override, but still adding a deleter fails in the same way

[acoustic thicket]

it fails even if its not an abc

[oblique urchin]

Pyright isn't strictly wrong here though the error is somewhat pedantic. The ABC promises that if you access .x on the class (e.g., Base.x), you get an int or an AttributeError. But if you do SubA.x, you get a property object instead.

[acoustic thicket]

that's true

how do you work around it

[tranquil turtle]

there is another difference: base_obj.x is deletable, while subA_obj.x is not

[acoustic thicket]

it fails even with the deleter added

[tranquil turtle]

you can lie to typechecker by using TYPE_CHECKING

[jade viper]

Hey guys, how do I annotate an image loaded as an np.ndarray? Is it np.ndarray[int, int]?

[eternal surge]

What's the difference between list[Unknown] and list[Any]?

[rare scarab]

list[Unknown] (supposedly) will give you an error if you try to use any items.

though generally list[Unknown] is a fake type added by pyright to indicate a generic type wasn't given.

[eternal surge]

For example here:

import json

def test():
    x = json.loads('hello world')

    if not isinstance(x, list):
        return False

    try:
        for a, b, c in x:
            print(a, b, c)
    except ValueError:
        return False

Pyright complains about a, b, c in x as the type of x is list[Unknown]. How could I avoid this error?

[rare scarab]

because x is Any and not something like str | list[str], it can't possibly know the generic.

if you always know the input of json.loads() is a list of lists, you may want to just use cast and let it fail naturally when it isn't.

x: list[tuple[Any, Any, Any]] = json.loads('...')

technically, it won't be a list of tuples, but you don't use it in a way that would matter.

[eternal surge]

I hope that it is, but I can't know for sure because that's external data and I have no control over the types
But it seems like doing the cast works well enough to make Pyright stop complaining! Thanks ❤️

[rare scarab]

In that case, might I suggest something like pydantic?

!pip pydantic

[rough sluice]

pydantic v2.6.4

Data validation using Python type hints

Released on <t:1710249636:D>.

[eternal surge]

I think I tinkered with it for a bit but I couldn't get it to work with recursive data structures

[rare scarab]

Can you share the model you used?

[eternal surge]

It's lost for now, but I might try again and see if I fail a second time
The data structure I'm serializing is like a filesystem hierarchy with directories and files. I recall that I couldn't get pydantic to understand how to read a directory due to its recursive nature

[rare scarab]

it's fully supported.
https://docs.pydantic.dev/latest/concepts/postponed_annotations/#self-referencing-or-recursive-models

[remote thicket]

I am putting "1" as an address and it is passing validation, but if I put a string it fails. Location and ce_name are working, but not ip_interface. In ip_interface If I input a single digit it passes, but any string fails. I am using pydantic 2.6.3

from pydantic import BaseModel, Field, validator, field_validator
from pydantic.networks import IPv4Address
from typing_extensions import Annotated, Literal

ENABLED_DISABLED = Literal["disabled", "enabled"]


class GlobalSchema(BaseModel):
    ce_name: Annotated[str, Field(description="SBC Name")]
    location: Annotated[ENABLED_DISABLED, Field(description="Location", default="disabled")]
    ip_interface: Annotated[IPv4Address, Field(description="Testing IP Interface")]

[tranquil turtle]

https://github.com/python/typeshed/blob/main/stdlib/ipaddress.pyi#L150

[rough sluice]

stdlib/ipaddress.pyi line 150

class IPv4Address(_BaseV4, _BaseAddress): ...```

[remote thicket]

So what options do I have to have this validate correctly?

[rare scarab]

What are you passing in to ip_interface?

1 is valid. "1" is not valid. "1.0.0.1" is valid.

[tranquil turtle]

how is int valid? IPv4Address is a class, not an alias to some typeform

[remote thicket]

figured it out apparently 1 is a valid IP. 1.2.34534 is invalid and it validating correctly. I thought single digits was an invalid IP.

[rare scarab]

These are all valid and point to the same ip ```
http://1.1.1.1
http://0x1010101
http://16843009
http://0100200401
http://0b0001000000010000000100000001

oh. discord "fixed" them

the binary one doesn't seem to be working.

these ips are valid at the tcp/ip level

[fierce ridge]

did we ever settle on a protocol for immutable overridable typeddict fields? or is that still WIP?

aha, that looks like PEP 705 https://peps.python.org/pep-0705/

[fierce ridge]

very nice, accepted and implemented in typing-extensions

aw, mypy doesn't support it yet

stubs/geo_interface.pyi:30: error: Variable "typing_extensions.ReadOnly" is not valid as a type  [valid-type]

pyright appears to support it though, very nice

[jagged helm]

What is the diff b/w typing.List and list

both says "Built-in mutable sequence."

[rustic gull]

Hello, how do i define types if function return for example 2 lists

[pastel egret]

From a type checker's perspective, there is none. When typing was originally added, the Python devs didn't want to make core code changes to support it, just it case it didn't work out and needed to be removed. So List etc was added to allow you to do List[int]. But as of Python 3.9 the subscripting ability was added to the regular class, so they're no longer necessary.

[rustic gull]

    def get_givers_and_receivers(self) -> List[List[str, int]], List[List[str, int]]:
        receivers = []
        givers = []
        balances = self.compute_balance_net()

        for k, v in balances.items():
            if v > 0:
                givers.append([k, v])
            else:
                receivers.append([k, v])
        return givers, receivers

[pastel egret]

A function can only return a single value, what you're most likely meaning is that it returns a tuple of lists.

[jagged helm]

got it
thanks

[rustic gull]

Thats true thanks

[eternal surge]

Why does pyright report the value of b as int | None? The only way b would be None is if a was of type int as well, but this has been ruled out in the condition

def example(x: bool) -> tuple[str, int] | tuple[int, None]:
    if x:
        return 'hello', 2
    else:
        return 0, None

def main():
    a, b = example(True)

    if isinstance(a, int):
        return

    reveal_type(b)  # Type of "b" is "int | None"

[trim tangle]

It's kinda unfortunate, but it just doesn't track types that deeply

as in, after a and b have "left" that tuple, it doesn't correlate them

[eternal surge]

I see... That is pretty unfortunate

[trim tangle]

Actually, even if you do ```py
def example(x: bool) -> tuple[str, int] | tuple[int, None]:
if x:
return 'hello', 2
else:
return 0, None

def main():
foo = example(True)

if isinstance(foo[0], int):
    return

reveal_type(foo)

[eternal surge]

So it basically treates tuple[str, int] | tuple[int, None] the same as tuple[str | int, int | None]?

[trim tangle]

Not quite
If a function expects tuple[int, int] | tuple[str, str], you can't give it a tuple[str | int, str | int]

but it's difficult or impossible to distinguish between the two cases if that's what you mean

You could start a discussion in https://github.com/microsoft/pyright/discussions if you think it's a useful inference rule

[eternal surge]

I guess this would solve it, but it just feels redundant

def main():
    a, b = example(True)

    if isinstance(a, int):
        return
    assert isinstance(b, int)

    reveal_type(b)  # Type of "b" is "int"```

[trim tangle]

yep

I could give some suggestions if you have a more realistic code example

[chrome hinge]

perhaps you could do

match foo:
    case str(a), int(b):
        ...
    case int(a), None:
        ...
    case _:
        assert_never(foo)

[trim tangle]

quite often returning a union can be a sign that the code can be simplified

# before
def fetch_message(self, decode: bool = False) -> str | bytes:
# after
def fetch_message(self) -> bytes:
def fetch_utf8_message(self) -> str:
# or
def fetch_message(self) -> bytes:  # decode it yourself!

[eternal surge]

I have a method like this: (DisconnectedUser is type alias for LoggedOutUser | OfflineGuest)

async def _handle_request(self, user: ConnectedUser, request: sprotocol.ServerMessage) -> tuple[ConnectedUser, sprotocol.ServerMessage] | tuple[DisconnectedUser, None]:

And I handle it in a loop:

while True:
    logger.debug(f"{user}: Waiting for requests...")
    request = await requests.get()
    logger.info(f"{user}: Got request: {request}")
    user, response = await self._handle_request(user, request)
    if isinstance(user, OfflineGuest) or isinstance(user, LoggedOutUser):
        break
    logger.info(f"{user}: Sending response: {response}")
    await responses.put(response)

I just noticed that the responses.put(response) raises a similar issue where it asserts response could be None despite the check beforehand

[chrome hinge]

@overload time :p

[trim tangle]

Oh yeah, somehow this works

from typing import Literal


def example(x: bool) -> tuple[Literal[True], int] | tuple[Literal[False], str]:
    if x:
        return True, 42
    else:
        return False, "nope"

def main():
    foo = example(True)

    match foo:
        case True, x:
            reveal_type(x)

        case False, s:
            reveal_type(s)

[trim tangle]

What's ConnectedUser and DisconnectedUser?

[eternal surge]

type ConnectedUser = LoggedInUser | OnlineGuest
type DisconnectedUser = LoggedOutUser | OfflineGuest

[chrome hinge]

it'd also work to do -> OkResponse | ErrorResponse, where OkResponse has user:ConnectedUser and message: sprotocol.ServerMessage fields, and ErrorResponse just has user:DisconnectedUser

[trim tangle]

In that case I'd make classes like ```py
@dataclass(frozen=True)
class UserConnected:
user: ConnectedUser
message: sprotocol.ServerMessage

@dataclass(frozen=True)
class UserDisconnected:
user: DisconnectedUser
``` (or NamedTuples if you like that better)

while True:
    match await self._handle_request(user, request):
        case UserConnected(user, message):
            ...
        case UserDisconnected(user):
            ...

[eternal surge]

I don't really see the point as this tuple pair is not really used anywhere outside of these two methods
Using match/case and defining dataclasses for the responses seems a bit excessive, but it is a good suggestion

[trim tangle]

if it's just a one-off thing you can do -> tuple[ConnectedUser, ServerMessage] | DisconnectedUser

I don't know why match-case works here but an if doesn't... that's strange. Maybe file a bug

[eternal surge]

Does it work the same with pypiy? Just curious, I don't have it installed

[trim tangle]

You mean pypy?

[eternal surge]

Yes

[trim tangle]

It shouldn't matter, pyright doesn't care whether you use cpython or pypy or graalpy or rustpython

(pyright itself isn't even written in Python)

either works fine at runtime after all

[eternal surge]

I see, I've never filed a bug report before so I'll have to figure it out

[trim tangle]

I think this would be better as a question/discussion (https://github.com/microsoft/pyright/discussions)

[eternal surge]

I started a discussion here in case anyone's interested: https://github.com/microsoft/pyright/discussions/7627

[hazy lake]

data: str = input(prompt)  # 3, 6; 5, 10; 9, 14
coords: List[Tuple[int, int]] = [
    tuple(map(int, _.strip().split(','))) 
    for _ in data.split(';') if len(_.split(',')) == 2
]

Expected type 'list[tuple[int, int]]', got 'list[tuple[int, ...]]' instead

anyone?

[oblique urchin]

that is not something I'd expect type checkers to understand. It requires some deep special casing of several methods

[hazy lake]

ah

i mean this is just a small program for an assignment

[hazy lake]

should i just ignore it?

[oblique urchin]

you can either ignore it or make a workaround

[hazy lake]

also if you dont mind, is the way i have written the code correct?

like using conventions and how the "community" does it?

[hazy lake]

thanks

[oblique urchin]

the simple workarounds are like # type: ignore or cast. To write it in a type-safe way that type checkers would accept I'd probably turn the tuple(map(int, _.strip().split(','))) part into a helper function that contains assert len(...) == 2

[oblique urchin]

Have you tested it? I'm pretty sure it would fail on your example

[hazy lake]

it worked

i did check

@oblique urchin

[oblique urchin]

ah, I forgot that this works:

!e print(int("3 "))

[rough sluice]

@oblique urchin :white_check_mark: Your 3.12 eval job has completed with return code 0.

3

[oblique urchin]

Usually though _ is used only for values that are ignored

[hazy lake]

understood

thanks a lot ^_^

[rustic gull]

Hello, if i am returning list of lists that will have structure str,str,int i should have type like this ```python
List[List[str, str, int]]
or
List[List[str,int]]

[oblique urchin]

neither, list only accepts a single type parameter and the type system doesn't have a way to express lists with specific types at specific positions

you can write list[list[str | int]] (or List[List[Union[str, int]]] in older versions)

but that doesn't express the fact that the str is at the beginning

usually you'd use tuples instead of lists for fixed-size sequences with specific values at specific places

[jade viper]

Why is my linter complaining? DenoisingMethod is an enum.Enum

[trim tangle]

!e

from enum import Enum

class Color(Enum):
    red = 1
    green = 2
    blue = 3

print(Color.red)
print(type(Color.red))
print()
print(Color.red.value)
print(type(Color.red.value))
print()

[rough sluice]

@trim tangle :white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | Color.red
002 | <enum 'Color'>
003 | 
004 | 1
005 | <class 'int'>

[jade viper]

So I should remove .value from the hint?

Or the default value rather

[trim tangle]

yep

Also, consider accepting an Iterable[DenoisingMethod] or Sequence[DenoisingMethod] if your function will work with tuples/generators/etc. (or if it just doesn't mutate the list)

[jade viper]

Yup, already changed to Sequence

[trim tangle]

The suggestion from Pylance here is kinda misleading though

[jade viper]

Would that work with match statements?

[trim tangle]

!e

from enum import Enum

class Color(Enum):
    red = 1
    green = 2
    blue = 3

foo = Color.green
match foo:
    case Color.red: print("red!")
    case Color.green: print("green!")
    case Color.blue: print("blue!")

[rough sluice]

@trim tangle :white_check_mark: Your 3.12 eval job has completed with return code 0.

green!

[trim tangle]

It won't work if you mix 1 and Color.red for example though

[jade viper]

Makes sense

Thanks so much!

So enum.Enum has an __eq__?

[trim tangle]

Yep

[jade viper]

Good to know lol

Again, thank you sm! 🙂

[lethal delta]

Is there a prepacked library for arithmetic ops as a Protocol?

I didn't see anything in the doc.

TL;DR: thinking of something like this:

class Lerpable(Protocol):

    def __add__(self, other) -> Self:
        ...

    def __sub__(self, other) -> Self:
        ...

    def __mul__(self, other) -> Self:
        ...

[grave fjord]

typeshed

[grave fjord]

https://github.com/hauntsaninja/useful_types maybe?

https://github.com/hauntsaninja/useful_types/blob/main/useful_types/__init__.py#L75

[rough sluice]

useful_types/__init__.py line 75

class SupportsAdd(Protocol[_T_contra, _T_co]):```

[lethal delta]

Yep

ty

[lethal delta]

I'd have to vendor it for one project, but it's a good lead and might help another. Ty!

[rare scarab]

Isn't useful_types a dependency of pydantic now?

[leaden oak]

Is there any way for the __eq__ method generated by dataclasses to be treated as __eq__(self, other: Any) instead of __eq__(self, other: Self)?

I'd assume the answer is no (probably up to the type checker), but I'm curious.

[soft matrix]

not if its generated

[leaden oak]

thought so, coolio

[soft matrix]

though having eq as other: Self is wrong and you should open an issue with your typechecker

[leaden oak]

I just needed something that was a shorthand for same type of the class

It's probably more subtle than that, I'm aware :)

[soft matrix]

even then its still wrong the one you want it to be treated as is correct

DataclassInstance.eq will accept any object

[undone saffron]

^, it's part of the data model, and implementing eq (by any class) should utilize NotImplemented for handling types they don't implement equality for, type checkers synthesizing things incorrectly is uh... not helpful.

[leaden oak]

if someone wants to verify that it still occurs on latest mypy and open an issue, feel free to

I'm not qualified to champion a type checker issue

[trim tangle]

I think people sometimes don't realize the full extent of what Self means

class Foo:
    def bar(self) -> Foo:
        ...
# is not the same as
class Foo:
    def bar(self) -> Self:
        ...

actually, is it ever valid to accept Self as a parameter?

if you do that, it automatically breaks LSP

[leaden oak]

and this is why I refuse to file issues or engage in typing discussions

I absolutely do not have the inclination to learn type theory or semantics to that level :)

[trim tangle]

Semantics is just "what this means"

[leaden oak]

any now I can't even reproduce the issue on the mypy playground, sigh

[trim tangle]

from typing import Self

class Foo:
    def quack(self, thing: Self) -> None:
        pass

class Bar(Foo):
    pass

foo = Foo()
bar = Bar()

foo.quack(foo)
bar.quack(foo)  # error is reported, as expected

def do_something(other_foo: Foo) -> None:
    other_foo.quack(foo)
do_something(bar)  # no error

actually maybe I should make a discussion on the typing page

[leaden oak]

I bet I'm going to feel like an idiot, but this is the relevant PR discussion: https://github.com/pypa/pip/pull/12571#discussion_r1555923403

[trim tangle]

my brother in christ you are literally a contributor to mypyc

[undone saffron]

I think it's more that people often are using over-using inheritence to build up a class, and never considering intermediary classes they've built up as "valid", so they skip over thinking about aspects of LSP impact.

[wraith linden]

Hey, I was just answering a question in the help forum, and I was wondering whether python would ever support something like the following (and whether it actually makese sense): ```py
class A: ...
class B(A): ...
class C(A): ...

def foo[S: A, T: (list[S], set[S])](arg: T) -> T:
...

I understand that you can achieve the desired effect using overloads.

[oblique urchin]

That's higher-kinded types. It's brought up occasionally but it's unlikely to be added in the immediate future

[cinder bone]

Good question

Also does anyone know how to type a proxy

[soft matrix]

you cant really

there are typing issues open about it

[cinder bone]

:(

Guess I'll stick with unioning it with the proxy type

[outer lantern]

@wraith linden Forgot about the question I asked when I got back from work yesterday. The overloading solution was great. Thanks for the help 😎

[echo knot]

Is there a way of using TypeVar defaults (PEP696) in previous Python versions? Either using typing_extensions or by any other trick

[pastel egret]

Yep, typing_extensions provides versions of TypeVar, TypeVarTuple and ParamSpec that support a default= parameter.

[echo knot]

I realized that default keyword is not "active" when type hinting something inside the generic class (see image). Is this expected?

Maybe I am just doing the wrong interpretation

from __future__ import annotations

from collections.abc import Mapping
from typing import Generic

from typing_extensions import TypeVar, reveal_type

T = TypeVar("T", default=Mapping[str, str])


class MyClass(Generic[T]):
    value: T
    value2: MyClass


a = MyClass()
reveal_type(a.value)
reveal_type(a.value2)

[trim tangle]

seems like a bug

is that pylance?

[echo knot]

Actually, I get an error in line 13 saying that I am missing the type parameters for MyClass (only when I use Mypy targetting Python version 3.9)

[echo knot]

Let me check because I work with both pylance and mypy and at some point I get confused

It is Mypy

It seems there are a few bugs:

• Default type parameters are not being used properly when the TypeVar variable is used within the generic class
• In the same scenario, false positive (missing type parameters error) is raised when Mypy is using the flag "python-version" with any version + strict flag is enabled

[oblique urchin]

Make sure you are using the latest version of mypy, and even then support is not complete

[echo knot]

It seems the latest version fixes it. I thought I had the latest one, but after some research it seems VSCose is using a wrapper around an older version

[rain bear]

what does LSP means

[oblique urchin]

https://en.wikipedia.org/wiki/Liskov_substitution_principle

[rain bear]

thank you

[tranquil turtle]

there is another meaning: Language Server Protocol
Language server is a thing that runs behind your text editor to provide you autocomplete suggestions, information about variables, code actions, etc...

[lethal delta]

Evil rewriting tricks via a decorator?

[lunar dune]

but that's not the meaning that's generally applicable in the #type-hinting channel 😉

[tranquil turtle]

it is closely related to typing: usually it is a thing that tells you about type errors 😉

[finite moon]

I've been fighting the typechecker (pyright) AND pylint to get something working, with only partial success.

I want to have a method that can work both as an instance method and as a classmethod.

Here is a simple example of what I'm trying to achieve (not the actual use case I have):

class A:
    def __init__(self, v: int):
        self.v = v

    def _obj_add(self, o: int) -> int:
        return self.v + o

    @hybridmethod(_obj_add)
    def add(cls, a: int, b: int) -> int:
        return a + b

assert A.add(3, 4) == 7
assert A(3).add(4) == 7

hybridmethod is a descriptor. I couldn't get the above to work. I'm sure my type hints are 100% correct.
With a few changes, I managed to get pyright to understand the above correctly (but not pylint), but then I have to do it this way:

    @classmethod
    def _cls_add(cls, a: int, b: int) -> int:
        return a + b

    @hybridmethod(_obj_add, _cls_add)
    def add(cls) -> None:
        pass

Pyright absolutely needs my hybridmethod to inherit from classmethod, otherwise nothing works. I'd rather have it inherit from nothing and use add(self), since hybridmethod doesn't use anything from that inheritance; or inherit from staticmethod, because then I can have add(). But no, it doesn't work at all, it's like pyright hardcoded the behaviour for these descriptors, instead of actually understanding descriptors.

Now, for the descriptor that actually worked in pyright but not pylint (this is the closest I got to a solution):

_SelfRT = TypeVar("_SelfRT")
_SelfP = ParamSpec("_SelfP")
_ClsRT = TypeVar("_ClsRT")
_ClsP = ParamSpec("_ClsP")
_T = TypeVar("_T")


class HybridMethodDescriptor(classmethod, Generic[_T, _SelfP, _SelfRT, _ClsP, _ClsRT]):
    def __init__(
        self,
        self_fn: Callable[Concatenate[_T, _SelfP], _SelfRT],
        cls_fn: Callable[Concatenate[type[_T], _ClsP], _ClsRT],
    ) -> None:
        self.cls_fn = cls_fn
        self.self_fn = self_fn

    def __call__(self, _empty_method: Callable[[_T], None]) -> Self:
        return self

    @overload
    def __get__(
        self, obj: _T, objtype: type[_T] | None
    ) -> Callable[_SelfP, _SelfRT]: ...

    @overload
    def __get__(self, obj: None, objtype: type[_T]) -> Callable[_ClsP, _ClsRT]: ...

    def __get__(
        self, obj: _T | None, objtype: type[_T] | None = None
    ) -> Callable[_SelfP, _SelfRT] | Callable[_ClsP, _ClsRT]:
        get = self.cls_fn.__get__ if obj is None else self.self_fn.__get__
        return get(obj, objtype)


def hybridmethod(
    self_fn: Callable[Concatenate[_T, _SelfP], _SelfRT],
    cls_fn: Callable[Concatenate[type[_T], _ClsP], _ClsRT],
) -> Callable[
    [Callable[[_T], None]],
    HybridMethodDescriptor[_T, _SelfP, _SelfRT, _ClsP, _ClsRT],
]:
    return HybridMethodDescriptor(self_fn, cls_fn)

As weird as it sounds, I do need to typehint the return of hybridmethod as a callable that takes in a Callable and returns a HybridMethodDescriptor, otherwise pyright again cannot understand it.

Edit: btw, the error I get in pylint is Too many positional arguments for classmethod call PylintE1121:too-many-function-args. That's because for pylint, the decorated method is ONLY a classmethod, and it can't understand the descriptor at all. Another annoying thing is that vscode colorizes the method as a property, not a function, but that's the least of my problems.

Any ideas on how to get it working on both pyright and pylint? (sorry for the long message!)

[undone saffron]

pylint isn't a type checker and seems to not understand descriptors based on the issue you're having. It probably is just checking "is this a classmethod" with special cased knowledge for how classmethods work.

Funnily enough, that's also why the color is the same as class variables within vscode.

I also can't replicate needing to type this the way you have here...

from collections.abc import Callable
from typing import Any, Concatenate, Self, overload


class HybridMethodDescriptor[T, **IP, IR, **KP, KR]:
    def __init__(
        self,
        instance_method: Callable[Concatenate[T, IP], IR],
        klass_method: Callable[Concatenate[type[T], KP], KR],
    ):
        self.im = instance_method
        self.km = klass_method

    def __call__(self, _anyc: object) -> Self:
        return self

    @overload
    def __get__(self, obj: T, objtype: type[T] | None) -> Callable[IP, IR]:
        ...

    @overload
    def __get__(self, obj: None, objtype: type[T] | None) -> Callable[KP, KR]:
        ...

    def __get__(self, obj: Any, objtype: Any):
        if obj is None:
            return self.km.__get__(obj, objtype)
        return self.im.__get__(obj, objtype)


class A:
    example = 1

    def __init__(self, val: int):
        self.val = val

    @classmethod
    def _c_add(cls, a: int, b: int) -> int:
        return a + b
    def _i_add(self, other: int) -> int:
        return self.val + other

    @HybridMethodDescriptor(_i_add, _c_add)
    def add(*args: object): ...

A.add(1, 2)
A(1).add(2)

This passes strict type checking under pyright.

example is there to point out the coloring for it being identical to that of add in vscode

[finite moon]

Thanks @undone saffron
We are still on python3.10, so I don't have support for python 3.12 typing features :/

[undone saffron]

should be able to just pull the paramspecs and type vars out for equivalence

but I'm not seeing a need to subclass from classmethod or to wrap it in a function from pyright, so pylint might not complain after that (not sure if that's actually why it's complaining, I don't use pylint, but a large number of tools have special cased knowledge for classmethod, staticmethod, etc)

[finite moon]

That worked I think!

Yeah, it did, thanks a lot!

Aaah!! Actually my code was working, it simply doesn't work in my very particular hybrid method. When I put your descriptor in my class (instead of this simple A), it doesn't work anymore with pyright. However, the add(*args, **kwargs) idea fixes pylint, so that's great!

I simplified my use case in the image below. I have self: TT and I return SomeClass[TT], then it stops working

I think I'll just have the (*args, **kwargs) which fixes pylint, and keep the intermediate function, then it works for my case

[rustic gull]

Can anyone explain Optional Type?

[grave fjord]

It's a legacy alias for T | None

[rustic gull]

what extension are you using for the typing hints?

-> None

[finite moon]

I'm using Pylance, which uses Pyright underneath.

[rustic gull]

hmm have the same extension

[soft matrix]

It's in the settings I think it's called inlay hints or something

[finite moon]

These are my settings:

    "python.languageServer": "Pylance",
    "python.analysis.inlayHints.variableTypes": false,
    "python.analysis.inlayHints.functionReturnTypes": true,

[rustic gull]

thanks yeah that did the trick

[forest rampart]

Hello !

I'm writing an ORM for fun, and you declare models like in Django/SQLAlchemy, with an optional inner class:

`class Users(Table):
id = UUID(primary_key=True)
username = String(collation="case_insensitive")

class Meta:
    constraints = [Unique(name="username_uniq", fields=["username"])]`

How can I define a type for class properties in the inner class Meta ?

I would like mypy to complain if the user set an unknown attribute, and set wrong values.

I tried to type Table like this:

`class Meta:
constraints: list[Unique | Check | PrimaryKey] | None

class Table:
Meta: Type[Meta]`

[trim tangle]

to me an inner class looks kinda weird...

I'd use inheritance kwargs, like ```py
class User(
Table,
constraints=[Unique(...), ...],
):

Or just a class attribute like `_constraints`

[trim tangle]

You can do that, and have the Meta insider users inherit from your_library.Meta

[forest rampart]

The constraints= argument to the class definition is "nice" on paper I it's not very elegant in the code because you read it before reading the columns definitions. I find it unintuitive

[forest rampart]

Without the inheritance it cannot work ?

[trim tangle]

You can make it a protocol, but then the protocol would have to implement all options, you can't make a protocol with possibly missing attributes

Why do you want it to be an inner class though?

Why not ```py
class User(Table):
id = UUID(primary_key=True)
username = String(collation="case_insensitive")

_settings = Settings(
    constraints = [Unique(name="username_uniq", fields=[username])]
)

then you can just type a _settings attribute in Table

[forest rampart]

It's just a "visual choice", indeed your proposal is nice too and allow for referencing attributes directly

Ah no its doesn't work, I can't reference username 🙂

[trim tangle]

Yeah if you use it as a class variable, you have access to the class scope

though you could use fields=["username"]

[forest rampart]

Indeed it's working fine, hum I like it 🙂

Thanks !

[hearty shell]

Hey guys, does anyone know if there is any ongoing PEP for this kind of thing?

class Resources(enum.Enum):
    GASOLINE = 1
    COAL = 2
    OIL = 3
    IRON = 4
    LEAD = 5

T = TypeVar('T', 
  Literal[Resources.GASOLINE], 
  Literal[Resources.COAL],
  Literal[Resources.OIL],
  Literal[Resources.IRON],
  Literal[Resources.LEAD],
)

class KLineData(TypedDict, Generic[T]):
    resource: T
    timestamp: int
    open: float
    close: float
    high: float
    low: float
    volume: NotRequired[float]
    turnover: NotRequired[float]

This works, but you need to manually convert the enum into literals, as far as I know you can't do that programmatically right?

[undone saffron]

Code sample in pyright playground

import enum
from typing import Generic, TypeVar, TypedDict, NotRequired

class Resources(enum.Enum):
    GASOLINE = 1
    COAL = 2
    OIL = 3
    IRON = 4
    LEAD = 5

T = TypeVar('T', bound=Resources)

class KLineData(TypedDict, Generic[T]):
    resource: T
    timestamp: int
    open: float
    close: float
    high: float
    low: float
    volume: NotRequired[float]
    turnover: NotRequired[float]

shouldn't need the literals

[oblique urchin]

It's not strictly equivalent because your solution allows e.g. KLineData[Resources] or KlineData[Literal[Resources.GASOLINE, Resources.COAL]]]. Probably not a big deal in practice though.

[hearty shell]

In this case that is the property I would be looking for, KLineData[Literal[Resources.GASOLINE]] should be its own type

So that I can have list[KLineData[Literal[Resources.GASOLINE]]], where the list only contains KLineData of a specific resource

[undone saffron]

code above works for that.

it just also allows you to specify

list[KLineData[Resources]], if you specify that a list is for a specific enum member like that, it's going to work how you expect.

[hearty shell]

Ah wait, I think I see what you mean

[undone saffron]

hmm

[hearty shell]

Yeah nvm I get what you mean, the above is just an inference problem

Your code does what I want actually, I just need to type it accordingly, otherwise it won't infer Literal (which in my case is fine actually)

Thank you

[undone saffron]

glad that works for you, I'm not sure if pyright is doing the right thing for inference there, but enums are a bit special in their behavior

the way enums are designed

type(Resource) == type(Resources.GASOLINE) == type(Resources.GASOLINE.GASOLINE)

I'm not sure that type checkers should ever be inferring the enum type when given a member, even if a bound is as wide as allowing the whole enum

[hearty shell]

Its been a while, but I think in most cases where you introduce TypeVars it usually does't resolve to a literal

K = TypeVar('K')
V = TypeVar('V', bound=int)

def foo(arg: K) -> K:
    return arg

class Bar(Generic[V]):
    def __init__(self, arg: V): ...

a: Literal[3] = 3
b: Literal[3] = 3
reveal_type(foo(a)) # Type of "foo(a)" is "int"
reveal_type(Bar(b)) # Type of "Bar(b)" is "Bar[int]"

So I don't think this is specific to enums, I think pyright has some heuristics to resolve to Literal under specific circumstances

[tranquil turtle]

wasn't this deprecated a while ago? i definitely remember discussions about E / E.X / E.X.X / ... stuff

[hearty shell]

Ah right I remember now, pyright only resolve a TypeVar to Literal if it is asked to do so

[sand merlin]

How Do I Typehit the Function lambda like function, with multiple OverLoads.

[swift forum]

I'm not any good with lambda !

[tranquil turtle]

you need callable Protocol

[swift forum]

protocol is important i agree

[sand merlin]

i fixed Miss spell.

[finite moon]

do you have an example of what you need to typehint?

[sand merlin]

self.c_setStart: This_Is_Function_with_multyple_OverLoads  = lib.Clip_setStart

[finite moon]

from typing import Protocol, overload

class MyFunctionWithOverloads(Protocol):
    @overload
    def __call__(self, a: int) -> int: ...
    @overload
    def __call__(self, b: float) -> float: ...

# ........
self.c_setStart: MyFunctionWithOverloads  = lib.Clip_setStart

[sand merlin]

Thanks I Found Other solution.

self.c_setStart: Callable[[], None] | Callable[[int], float] | ...  = lib.Clip_setStart

[finite moon]

That's different though, so it depends on what you want.

from typing import Callable, Protocol, overload

class MustSupportBoth(Protocol):
    @overload
    def __call__(self, a: int) -> int: ...
    @overload
    def __call__(self, a: int, b: int) -> int: ...


MustSupportOneOrTheOther = Callable[[int], int] | Callable[[int, int], int]

def fa(a: int) -> int:
    return a

def fb(a: int, b: int) -> int:
    return a + b

def fab(a: int, b: int = 0) -> int:
    return a + b

fp1: MustSupportBoth = fa  # ERROR: fa doesn't support the second overload
fp2: MustSupportBoth = fb  # ERROR: fb doesn't support the first overload
fp3: MustSupportBoth = fab # OK: fab supports both overloads


fu1: MustSupportOneOrTheOther = fa  # OK: supports Callable[[int], int]
fu2: MustSupportOneOrTheOther = fb  # OK: supports Callable[[int, int], int]
fu3: MustSupportOneOrTheOther = fab # OK: supports both

https://pyright-play.net/?pythonVersion=3.12&strict=true&enableExperimentalFeatures=true&code=GYJw9gtgBALgngBwJYDsDmUkQWEMoDCAhgDYlEBGJApgDRQAK4MYAxmCfWAG7UglgiAEwCwAKHGtyAZ2lQAsgFdpMAMqKEOPACEwMABYAKJnrYcAlAC5xUW1AACPPgOE27Q6sCgB9b61IkvobS1CTA9ESWmCgw5lAAtAB80TBRAHQZbraOvPyComJ2UB5evv5kQSFhEVGoMPQUtTFxSSnpmRKdSirqmrgwAPIo1AMgACr6IwZ8UAC8hAGUNADay3UAuvQbUAA%2BC2RL1Kt1WzGbKevi4iVQwESGkSktyXXWhXYg1DCKIChQRFcxDdgBQHk16lBGk8Ei8Ym8ip9vr9-lAANSQwHAyhglINcFzKAABmebSyUERPz%2BRDRGM6wAQAEYot01BotDBdAYCXdbABiKAAUQASkKBkKojyhGBqNIUAByfDSNn9WCTKAhdgoIRQJx5VxiekAJmZylZfR0en03IofMFIrFEptUpl8sVyrwquotyQIBUOtyLgK9IAzCaeu6OZbuZQoPyBgBpCUxpXmmByChR3WB6SA8TARRMhSm3rsoYjcaTAbTEDR20JqIp9lyYgHKhHNZnU4wS4GxTGovh1Nl0YTKaTGvzEF1xPqiPNxZt44xLvnDZ5xShgdm0vDEeV6vRm1xmeN-rpy1AA

[sand merlin]

👀

[viscid spire]

What the naming conventions

[sand merlin]

Which

[trim tangle]

In most cases having a union of callables is not correct, because you can't distinguish which case you've got

[trim tangle]

No, an overload is an intersection of callable types, not a union.
If you have this:

@overload
def f(x: int) -> int: ...
@overload
def f(x: str) -> bool: ...
``` Either of these would be correct:
```py
first: Callable[[str], int] = f
second: Callable[[int], int] = f
bonus: Callable[[str | int], int] = f

Because f is a (int -> int) and it's a (str -> bool)
But if you have an f: Callable[[str], int] | Callable[[int], bool], it means that f is a (int -> int) or it's a (str -> bool). And there's no way to find out which case it is, so you can't do anything with f really

[sand merlin]

Oh

[trim tangle]

We don't have intersection types in Python, so a Protocol with overloads will work

I'd personally just accept two different functions instead of requiring users to create an overloaded function

If they already have an overloaded function, they can just provide the same function as both arguments

[undone saffron]

There's an important detail about overloads that makes them distinct from a theoretical intersection

type checkers use some heuristics to pick an overload, and one of those is the order of the overloads

it's somewhat neccessary because of overlapping/partially overlapping input types not being forbidden

and forbidding those can't be practically achieved (at least without harming existing code) without type negation

the classic case of this is

(overload) (str) -> Never
(overload) (Iterable[str]) -> Something

[trim tangle]

Oh yeah that's true

[rare scarab]

^ that doesn't actually make str not a valid argument type. it just makes the linter treat it the same as if it raises an error

I tested it last week after my own false assumption about it.

[undone saffron]

Yep. Now this is good enough for a lot of people to help indicate that you don't intend to handle plain strings in an API, but it doesn't actually help with things like CI runs meant to catch misuse as a result

[trim tangle]

sometimes it's nice to use a string when an iterable or sequence is expected. Like random.choice("abcdef")

[rare scarab]

as long as it's understood that it will treat it as an iterable instead of a single str

[trim tangle]

yeah, I'd say it's often a programming error

[reef vortex]

guys how can i make a python helper type that exludes None or t.Optional from its typehint?

[rare scarab]

Make an overload probably

not possible in the current type system

[trim tangle]

Can you show an example maybe?

[rare scarab]

Typescript defines ```ts
type NonNullable<T> = T & {}

{} means any type besides null or undefined

[reef vortex]

Like

int | str | None

To

int | str

Sorry for my weird questions
I have a habit of recreating typescript types in python

[rare scarab]

python doesn't have type transforms, so I don't see the usecase

[trim tangle]

def unnonify[T](x: T | None) -> T:
    if x is None:
        raise TypeError("You lost the game")
    return x

So sometimes it's possible. Depends on where you need it

[rare scarab]

TypeGuard is also usable, but at that point might as well use if x is not None:

[fierce ridge]

i do somewhat question the practical value of "anything other than None"

[cinder bone]

Is there a way to type this?

ParseableT: TypeAlias = T | Iterable[T} | types.GeneratorType[T, None, None]
def add_to_submobject(arg: ParseableT[Mobject]) -> None:
    arg_flattened = flatten(arg)
    # do stuff with arg_flattened

[fierce ridge]

1. what does flatten do?
2. do you need Generator? usually Iterable suffices even when it's actually a generator

that is, if you're only yieldng and not sending or returning, then Generator[T, None, None] is functionally equivalent to Iterable[T]

[cinder bone]

1. flatten the arguments

UNFLATTENED_ITER = T | Iterable["UNFLATTENED_ITER"]
def flatten(arg: UNFLATTENED_ITER):
    flattened = []
    for ele in arg:
        if isinstance(ele, Iterable);
              flattened.extend(flatten(ele))
        flattened.append(ele)
    return flattened

2. Never knew that, good to know. Do type-checkers treat it the same way?

[fierce ridge]

flatten just returns a list[T], no?

and what is T? a TypeVar?

@cinder bone you have a bug in that code actually. you're passing a Parseable into flatten(), but a Parseable might be a T which is not an iterable (as far as I know)

I think these are the correct types for your intent: https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=d83299e62d47091b52b3b118f412ff4d but it reveals the bug in your code

from collections.abc import Iterable, Sequence
from typing import TypeAlias, TypeVar


T = TypeVar('T')
NestedIterable: TypeAlias = T | Iterable['NestedIterable[T]']

def flatten(arg: NestedIterable[T]) -> Sequence[T]:
    flattened: list[T] = []
    for ele in arg:
        if isinstance(ele, Iterable):
            flattened.extend(flatten(ele))
        else:
            flattened.append(ele)
    return flattened


Parseable: TypeAlias = str | Iterable[str]

def do_something(parseable: Parseable) -> None:
    tokens = flatten(parseable)
    print(tokens)

[cinder bone]

It was supposed to be pseudocode mwe, my real intent is a bit more complicated. But this answers my question, so thanks!

[reef vortex]

guys what would be its python?

export type Input<TSchema extends BaseSchema | BaseSchemaAsync> = NonNullable<
  TSchema['_types']
>['input'];

this is my current code

TInput = tx.TypeVar("TInput", default=t.Any)
TOutput = tx.TypeVar("TOutput", default=TInput)

@dataclass
class TypedSchemaResult[TOutput]:
    output: TOutput
    issues: t.Any
    typed: bool = True

@dataclass
class UntypedSchemaResult:
    output: t.Any
    issues: t.Any
    typed: bool = False

type SchemaResult[TOutput] = TypedSchemaResult[TOutput] | UntypedSchemaResult

@dataclass
class Types[S, R]:
    input: S
    output: R

@dataclass
class BaseSchema(t.Generic[TInput, TOutput]):
    type: str
    expects: str
    _parse: Callable[[t.Any, t.Any], SchemaResult[TOutput]]
    _types: Types[TInput, TOutput]
    async_: bool = False

@dataclass
class BaseSchemaAsync(t.Generic[TInput, TOutput]):
    type: str
    expects: str
    _parse: Callable[[t.Any, t.Any], Coroutine[t.Any, t.Any, SchemaResult[TOutput]]]
    _types: Types[TInput, TOutput]
    async_: bool = True

Tschema = tx.TypeVar("Tschema", bound=BaseSchema | BaseSchemaAsync)

[rare scarab]

Python doesn't have type transformation

type Input = Types[TInput, TOutput]

[frigid rover]

#type-hinting

[trim tangle]

exactly, type hinting

[lunar dune]

Yup

[trim tangle]

can confirm that the types are being hinted to at this very moment

[viscid spire]

Type has been dropping hints the whole night, but I'm oblivious to their advances

[trim tangle]

I think this is about time that you get disappointed in Python's type system capabilities 🙂

It seems like you're trying to do something very strange

For example, what's up with _types: Types[TInput, TOutput] ?

[viscid spire]

Mixing of 3.12 and pre-3.12 generic classes 😵‍💫

[trim tangle]

pep 695 syntax doesn't support defaults on 3.12

[viscid spire]

ok? Then make everything the old style

consistency

[trim tangle]

perhaps

I'm generally not too much of a fan of the new syntax. It does save a few characters, but it is an extra thing to teach to (typing) beginners. And if you want to use defaults, you need to use TypeVar anyway

It also made it difficult to add more features to type variables. (defaults is a convenient example). Either you force people to go back to the TypeVar(...) syntax or you have to add more syntax to Python the language

[viscid spire]

Well, we will get defaults with the new syntax in 3.13 hopefully, although I didn't see it in the feature list yet

Having the same syntax for defaults as function arguments is intuitive

[soft matrix]

it will be in 3.13

though in all truth i have no idea where the actual implementation is up to

https://github.com/python/cpython/pull/116129

[echo knot]

Do you have any ideas about how to subclass a class to just extend or be more accurate about some of its type hints?

Ideally I would just like to use some overloads and that's it. However, I need to define the "base/default" case. The only solution I see is to copy/paste the signature of the superclass, which makes it harder to maintain and might conflict with some versioning stuff. Is there any other possible way?

[glacial bison]

Hey all, does anyone know if it's possible to constrain a TypeVar , where the only acceptable values are contained within a TypeVarTuple? Here's a small example

TList = TypeVarTuple("TList")
T = TypeVar("T")


class X(Generic[Unpack[TList]]):
    def get_something(self, type: type[T]) -> T: ...


# In this case, TList = (int, float)
# I want T to only be int or float, e.g. T = TypeVar("T", int, float)
x: X[int, float] = X()
x.get_something(int)  # ok
x.get_something(float)  # ok
x.get_something(str)  # error?

[reef vortex]

Ye i guess , btw one question like is there any way I can access properties of generics?

[viscid spire]

such as?

[reef vortex]

Like I have a generic named T which is bound to a class called Pet | Stray , so i wana type a variable with type of name attribute of Pet or Stray like T.name

[viscid spire]

don't think you can do that in Python

[reef vortex]

Hmm , pretty disappointing

[grave fjord]

You'll need to call super().__getitem__(...)

[brisk hedge]

this is an unsafe operation, it means that instances of B can't be compatible with A anymore

[echo knot]

I am aware of that, but I only want it for type hinting purposes (more specifically, for the IDE to autocomplete). Otherwise at runtime I would be breaking the Liskov substitution principle

But I already managed to do it 🙂

[trim tangle]

Have you considered making a stub?

Or since it's not that critical, just making a PR 🙂

or is that not a library?

[echo knot]

Yes and no. I asked about it a few weeks ago here and came to the conclusion that what stubs do are not enough. I needed to generate importable code, and that's something I cannot do with pyi files

I did it with the purpose of using a tool to automate type hint generation for pandas dataframes. Only for the purpose of IDE autocompletion

[low escarp]

Is there some way to type hint that a function should have some attribute? I know it's kind of a weird thing to do and from readin online it seems like a Protocol would be the right way to do this, but I am not sure exactly how.
Example (contrived) code:

import inspect
from typing import Callable, Any

def decorator(func):
    func.value = 3
    return func

class Thing:
    @decorator
    def function(self):
        pass

    def function2(self):
        pass

    def do_thing(self, func: Callable[..., Any]):
        print(func.value)

    def decorated(self):
        print([x for x in inspect.getmembers(self) if hasattr(x[1], "value")])

a = Thing()
a.decorated()
a.do_thing(a.function)

I would like some way to indicate in the do_thing function that the func argument has the value attribute (or any number of other attributes)
(Also yes I know I could use a predicate on inspect.getmembers, this was just less code to write for this example)

[fierce ridge]

i think you could use a protocol here:

from typing import Any, Protocol, TypeVar

T = TypeVar("T")

class FuncWithValue(Protocol[T]):
    value: T
    def __call__(self, *args: Any, **kwargs: Any) -> Any: ...

[low escarp]

ok, thanks! I'll give this a go later! 😄

[cunning plover]

is it possible to assert type of an object

instead of using from typing import cast

so the code next after that will see the object as specific type, without usage of cast() function?

https://mypy.readthedocs.io/en/stable/type_narrowing.html
hmm, here it is

assert isinstance(service, V1Service)

looks to be doing the trick

[tranquil turtle]

typechecker will assume that assert succeeds even if it might not be true at runtime

[rose mist]

Greetings,

Hope all are well. May I ask how I can write the N dimensional generalization of Collection?

T = TypeVar("T", covariant=True)

@runtime_checkable
class Collection(Protocol[T]):
  def __len__(self) -> int:
    ...
  def __iter__(self) -> Iterator[T]:
    ...
  @overload
  def __getitem__(self, idx: int) -> T:
    ...
  @overload
  def __getitem__(self, idx: slice) -> Self:
    ...
  def __add__(self, other: Self) -> Self:
    ...
  def __mul__(self, other: int) -> Self:
    ...

Issue is that it doesn't apply if the type is N dimensional, i.e., list[list[int]].

It only suffices for when you have a 1D sequence of objects.

[fierce ridge]

it works fine as-is:

from collections.abc import Iterator
from typing import Protocol, Self, TypeVar, overload, runtime_checkable


T_co = TypeVar("T_co", covariant=True)


@runtime_checkable
class Collection(Protocol[T_co]):
  def __len__(self) -> int: ...

  def __iter__(self) -> Iterator[T_co]: ...

  @overload
  def __getitem__(self, idx: int) -> T_co: ...
  @overload
  def __getitem__(self, idx: slice) -> Self: ...

  def __add__(self, other: Self) -> Self: ...

  def __mul__(self, other: int) -> Self: ...


coll: Collection[Collection[int]]
reveal_type(coll[3])

main.py:25: note: Revealed type is "__main__.Collection[builtins.int]"
Success: no issues found in 1 source file

https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=d98b2520e6a9bc2d5e4106a6a80484f7

@rose mist ☝️

note that there's already a Collection type in collections.abc, along with some others like Sequence that you might want to consider inheriting from as mixins, instead of defining it all yourself

[rose mist]

No no, you wouldn't want to explicitly define the dimension.

It should interpret it like NDArray.

[rose mist]

There is no suitable type which would cover both NDArray and list.

I spent a week on that.

[fierce ridge]

well numpy NDArray is internally a flat 1D array anyway, it's not a list of lists, and the illusion thereof is purely due to numpy indexing conventions that are designed to resemble the C array-of-arrays style

NDArray is a completely different case and IMO irrelevant

[rose mist]

Well, I'd like sth like that, it's merely for type hinting.

[fierce ridge]

can you give a practical example of how you want to use this? maybe I'm missing something here

[rose mist]

Sure

One moment...

class Data:
    def __init__(self,
                 data: Collection[Number]) -> None:

So here, if you pass for instance

data = Data([[1, 2, 3], [4, 5, 6]])

It'll raise an error.

[fierce ridge]

so you want to accept arbitrarily-deeply-nested collections as input?

[rose mist]

It should pass as long as this is some container where the the dtype is Number. It may be flat, or nested, etc.

[rose mist]

Exactly.

[fierce ridge]

ok, let me see here. you should be able to do it with a recursive TypeAlias

[rose mist]

It should work for any type of container, i.e., list or NDArray.

[rose mist]

Could I see an example please?

[rose mist]

Ohh by the way, it should be a Protocol like my original code, otherwise mypy apparently raises an error.

[fierce ridge]

what error? you can't put @runtime_checkable on a subclass of collections.abc.Collection

[rose mist]

I recall getting that error, and why I had to make a Protocol.

[rose mist]

It's not collections.abc.Collection.

[fierce ridge]

it looks like it follows that protocol, no?

[rose mist]

from __future__ import annotations

__all__ = ['Collection']

from typing import (Iterator, overload, Protocol, TypeVar,
                    Self, runtime_checkable)


T = TypeVar("T", covariant=True)

@runtime_checkable
class Collection(Protocol[T]):
  def __len__(self) -> int:
    ...
  def __iter__(self) -> Iterator[T]:
    ...
  @overload
  def __getitem__(self, idx: int) -> T:
    ...
  @overload
  def __getitem__(self, idx: slice) -> Self:
    ...
  def __add__(self, other: Self) -> Self:
    ...
  def __mul__(self, other: int) -> Self:
    ...

This is locally made.

[fierce ridge]

i'm suggesting that it can and should inherit from collections.abc.Collection

it doesn't need to, but it's nice to use those mixins imo

but if you want a protocol that makes sense too

it's kind of unfortunate that there isn't a collections.abc.CollectionProtocol

which maybe is the obstacle here to inheriting. not relevant anyway to your problem

[rose mist]

So, could I see how I could implement sth like this for an arbitrarily deep Collection of dtype number? FYI I also have a local Number type alias, which is basically all floats, integers, and complex values supported by both CPython and Numpy. Stupid, I know, it's just for my current testing, I'll fix it later after I fix this collection thing.

[fierce ridge]

https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=bfad7dac71d7cb219557627b2c7881ed

[rose mist]

If I recall correctly, it's sth mypy forced me to have, so yeah.

[fierce ridge]

from collections.abc import Iterator
from typing import Protocol, Self, TypeAlias, TypeVar, overload, runtime_checkable


T_co = TypeVar("T_co", covariant=True)


@runtime_checkable
class Collection(Protocol[T_co]):
  def __len__(self) -> int: ...

  def __iter__(self) -> Iterator[T_co]: ...

  @overload
  def __getitem__(self, idx: int) -> T_co: ...
  @overload
  def __getitem__(self, idx: slice) -> Self: ...

  def __add__(self, other: Self) -> Self: ...

  def __mul__(self, other: int) -> Self: ...


NestedCollection: TypeAlias = Collection[T_co] | Collection[NestedCollection[T_co]]

data1: NestedCollection[int] = [1,2,3]
reveal_type(data1[0])

data2: NestedCollection[int] = [[1,2,3]]
reveal_type(data2[0])

data3: NestedCollection[int] = [[[1,2,3]]]
reveal_type(data3[0])

NestedCollection: TypeAlias = Collection[T_co] | Collection[NestedCollection[T_co]]

there might be a way to specify this directly on the Protocol, which can also be self-referential

[rose mist]

Ohhh

It says NestedCollection not defined.

[fierce ridge]

You might need to quote it

[rose mist]

Yeah sorry, took me a bit.

[fierce ridge]

NestedCollection: TypeAlias = Collection[T_co] | Collection['NestedCollection[T_co]']

[rose mist]

Oh. Quick question, what does quoting do?

Apologies if it's a stupid question.

[rose mist]

Thank you so so much for this by the way!

I've been looking for a better way of playing around with what would pass mypy.

[fierce ridge]

the python interpreter just sees a string, so python itself doesn't fall over trying to evaluate it. and apparently pylance is playing along with the eager evaluation behavior, the quoting basically defers evaluation

[junior cloud]

Hi
Is func1 here's type return type hint correct? My IDE doesn't complain, but something feels... off...

def func1(a: int, b: int) -> (int, int):
    return a + b, a - b


def func2(a: int, b: int) -> tuple[int, int]:
    return a + b, a - b

[oblique urchin]

no, that's not a legal annotation in the type system

[fierce ridge]

it's definitely been proposed before though

[junior cloud]

I agree. I couldn't find something stating that is legit, so expected PyCharm to complain though it didnt'

[tranquil ledge]

iirc, it's "legal" in PyCharm's eyes, but I haven't used it for a few years.

Yep.

[fierce ridge]

pycharm's type checker is... not the best

i'd always encourage cross-checking with mypy and/or pyright

[rose mist]

@fierce ridge mypy raises this error.

[tranquil ledge]

I actually ad-hoc learned it with parentheses because of PyCharm. Only upon migrating to VSCode + Pyright did I realize it wasn't legal.

[fierce ridge]

the first line of the error explains it, kind of. just do isinstance(..., Collection) instead, since a NestedCollection is just a Collection anyway

[rose mist]

Thank you so much, for now it seems to have passed the mypy check.

[pastel egret]

PyCharm's support for type hints predates the typing module, originally it just detected them in docstrings. I think that's where the tuple etc syntax came from.

[stray summit]

anyone aware if there is a simple way to make numerous functions mirror overloads of a wrapped function

i have multiple helpers that have to wrap fdopen style functions and its most pain-full to map the overloads for all of them to get correct type inference

[tranquil turtle]

if they have exactly the same signature, then you can do this: ```py
if TYPE_CHECKING:
class _Signature(Protocol):
def call(...):...

func1: _Signature
func2: _Signature
...

else:
def func1(...): # actual implementattion
def func2(...): ...

[stray summit]

thats not quire clear - i need to inherit overloads from functions like open/fdopen, if i have to completely clone the overloads them to begin with im not exactly at a good place

[tranquil turtle]

if TYPE_CHECKING:
    my_fdopen = fdopen
else:
    def my_fdopen(...):...

maybe this?

this will work iff your function has exactly the same signature as fdopen

if you wanna add one more overload to it - you are out of luck, i think

[fierce ridge]

I don't think it's possible without using a shared protocol for both

not even sure the protocol would work

[stray summit]

for context im trying to figure how to get the execnet "execmodel backends" fully type annotated in a sensible manner
(those are or managing the differences between working with native threads + files descriptors and gevent/eventlet based ones

(execnet is the library pytest-xdist uses for multi process and multi host testing)

[tranquil ledge]

You can use a helper function that copies & “forwards” the annotations via ParamSpec.

e.g. something I wrote recently to get the full signature of ast.parse (including overloads) onto a function that wraps it:

import ast
import functools
from collections.abc import Callable
from typing import ParamSpec, TypeVar

_T = TypeVar("_T")
_P = ParamSpec("_P")


def copy_annotations(original_func: Callable[_P, _T]) -> Callable[[Callable[_P, _T]], Callable[_P, _T]]:
    # Overrides annotations, thus lying, but it works for the final annotations that the *user* sees on the decorated func.
    @functools.wraps(original_func)
    def inner(new_func: Callable[_P, _T]) -> Callable[_P, _T]:
        return new_func

    return inner

...

# The return annotation and some of the parameter annotations are too narrow or wide, but they should be "overriden" by this decorator.
@copy_annotations(ast.parse)  # type: ignore
def parse(
    source: str | ReadableBuffer,
    filename: str = "<unknown>",
    mode: str = "exec",
    *,
    type_comments: bool = False,
    feature_version: tuple[int, int] | None = None,
) -> ast.Module:
    return transform_ast(
        ast.parse(
            transform_source(source),
            filename,
            mode,
            type_comments=type_comments,
            feature_version=feature_version,
        )
    )

EDIT: Fixed example from below.

[tranquil turtle]

@functools.wraps(original_function) here is incorrect, you shouldn't add it at all

[tranquil ledge]

Why, if you don’t mind elaborating? I sorta threw it on there without thinking in retrospect.

[tranquil turtle]

@wraps(f)
def g(...): ...

this is used to make g look like f. Internally it changes some attributes of g to amke it more similar to f. This is used to make wrapping more seamless, so that you cant tell the difference between f and g. This implies that f and g should have similar/same signature.

In your case it is not true, inner will not survive for long, it will return parse as is, without wrapping it, so there is no function that replaces/wraps ast.parse, and there is no point in making inner more similar to ast.parse

[tranquil ledge]

Hmm, I kinda see your point. wraps doesn’t actually do anything here that’s helpful.

(Beyond overriding __annotations__, at least)

[tranquil turtle]

i think (not entirely sure), @copy_annotations(...) will yheows an error in cases when original_func and new_func (ast.parse and parse in this case) have different signatures
it kinda defeats the purpose of the decorator: it is supposed to lie to typechecker, not to bother it

it can be fixed by replacing some (not all!) occurrences of Callable_P,_T] with typing.Any (and maybe slapping a couple of #type:ignore inside of the decorator)

[tranquil ledge]

I think I see what you mean. I could probably replace inner’s parameter annotation with Any to facilitate the lie.

[tranquil turtle]

the things you can do is to return functools.wraps(original_func)(new_func) (instead of just new_func) from inner
it will copy some attributes from ast.parse to parse

[tranquil ledge]

Ahh, I see now. Something more like this, then?

I guess in a case where one just wants annotations forwarded on, manually assigning __annotations__ will have fewer unintended side effects than using wraps/update_wrapper. This is close enough for my purposes at least.

[tranquil turtle]

yeah, this looks fine

iirc, you can pass attributes you want to copy to update_wrapper

!d functools.update_wrapper

[rough sluice]

functools.update\_wrapper

functools.update_wrapper(wrapper, wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)```
Update a *wrapper* function to look like the *wrapped* function. The optional arguments are tuples to specify which attributes of the original function are assigned directly to the matching attributes on the wrapper function and which attributes of the wrapper function are updated with the corresponding attributes from the original function. The default values for these arguments are the module level constants `WRAPPER_ASSIGNMENTS` (which assigns to the wrapper function’s `__module__`, `__name__`, `__qualname__`, `__annotations__` and `__doc__`, the documentation string) and `WRAPPER_UPDATES` (which updates the wrapper function’s `__dict__`, i.e. the instance dictionary).

[tranquil ledge]

Neat. That's customizable enough to work, then, so long as one is okay with importing functools. Cheers.

[celest iron]

from typing import TypeVar

JsonType = int | str | float | bool | None | list["JsonType"] | dict[str, "JsonType"]
JsonTypeT = TypeVar("JsonTypeT", bound=JsonType)

def normalize_quotes(current: JsonTypeT) -> JsonTypeT:
    match current:
        case {**d}:
            return {
                key.replace('"', ""): normalize_quotes(value)
                for key, value in d.items()
            }
        case [*l]:
            return [normalize_quotes(value) for value in l]
        case item:
            return item

Are these sort of generics compatible with the match statement. I'm using pyright tho I'd be curious if mypy is alright with this i guess. I'm just trying to ensure the same input is returned for the same output.

I'm guessing noo?

[rare scarab]

You can't use parameterized generics with isinstance or match

[celest iron]

gotchya figured as much

[weak anchor]

does mypy have any way to declare generic type aliases?

I want an ID<T> type aliased to str

such that eg, an ID<User> can't be used where an ID<Message> is expected

[oblique urchin]

sounds like you're looking for NewType

[weak anchor]

i didn't think NewType supported generics?

if it does I can't seem to find how

[oblique urchin]

Ah, I guess you want to connect the type to some other type. Yes, then NewType isn't enough

[fierce ridge]

i think TypeAlias "just works"

from collections.abc import Sequence
from typing import TypeAlias, TypeVar

T_co = TypeVar("T_co", covariant=True)

MySequence: TypeAlias = Sequence[T_co]

s: MySequence[int] = [1,2,3]

[oblique urchin]

I guess they want a phantom type parameter, like type Id[T] = str. I don't think that works with mypy.

[fierce ridge]

oh, i see

[echo knot]

What do you think about these two cases?

from typing import Any, TypedDict


class Example(TypedDict):
    value: int

def func() -> dict[str, Any]:
    return Example({"value": 2})  # ERROR: Incompatible return value type (got "Example", expected dict[str, Any])

First, returning a TypedDict when a Dict[str, object] is espected. Shouldn't this one be legal?

What do you think about these two cases?

from typing import Any, TypedDict


class Example(TypedDict):
    value: int


def func2(value: dict[str, object]) -> None:
    print(value)


func2(Example({"value": 2}))  # ERROR: Argument 1 to "func2" has incompatible type "Example"; expected dict[str, object]

And secondly, sending a TypedDict when a Dict[str, object] is required

[trim tangle]

the_dict = func()
the_dict["value"] = "not an integer"  # legal
the_dict.pop("value")  # also legal

In this case it doesn't matter, but in general someone else could have a reference to that typeddict

the second case is similar ```py
def func2(vlaue: dict[str, object]) -> None:
value["value"] = "not an integer"
value.pop("value")

If you're only reading, you can use Mapping

[echo knot]

Ah right, that's true

[echo knot]

Yeah, I use Mapping whenever I can. It was just that I was creating a type-hint based tool and I have to deal with this issue

[trim tangle]

my condolences

[echo knot]

It would be nice having a "MappingTypedDict" 😮‍💨
Because I guess that, even if I use ReadOnly for all attributes of the TypedDict, I could still mutate it by adding a new key

[oblique urchin]

!pep 728

[rough sluice]

**PEP 728 - TypedDict with Typed Extra Items**

Status

Draft

Python-Version

3.13

Created

12-Sep-2023

Type

Standards Track

[echo knot]

It is a corner case that is messing up a whole feature lol

[echo knot]

I read this a few months ago but I do not remember much of it. Gonna read it again. Thanks!

[reef vortex]

guys why its not giving error like Int pipes can't be used in string schema?

name = String([MinLength(5), Minvalue(1)])

class String(Schema):
    def __init__(self, pipes: list[Pipe[str]]):
        super().__init__(pipes)

    def validate(self, data: t.Any) -> str:
        for pipe in self.pipes:
            data = pipe(data)
        return data


class MinLength[T: str | list[t.Any] | dict[t.Any, t.Any]](Pipe[T]):
    def __init__(self, min_length: int):
        self.min_length = min_length

    def __call__(self, data: t.Any) -> T:
        if len(data) < self.min_length:
            raise ValueError(f"Data length is less than {self.min_length}")
        return data


class Minvalue[T: int](Pipe[T]):
    def __init__(self, min_value: int):
        self.min_value = min_value

    def __call__(self, data: t.Any) -> T:
        if data < self.min_value:
            raise ValueError(f"Data value is less than {self.min_value}")
        return data

[trim tangle]

What is Pipe?

[reef vortex]

It's a abc.ABC which defines a structure that it's inheriting classes should have a validate method

[trim tangle]

can you show the code for Pipe?

also, what type checker are you using?

[reef vortex]

Pyright

Pylance extension

[trim tangle]

Does foo: str = 42 give you an error?

[reef vortex]

Yes it gives

It's working fine i guess

My code haves issue

[trim tangle]

The issue might be that you're not parametrizing the class

[reef vortex]

Wdym?

[trim tangle]

What is the type of Minvalue(42)?

if you do x = Minvalue(42) and hover over x, you'll probably get something like x: Minvalue[Unknown].

[reef vortex]

x: Minvalue[Unknown]

[trim tangle]

Actually the generic there is wrong. If you have a Minvalue[bool](-7), it is hinted as returning a bool, but it will return -7.

you probably meant something like ```py
class Minvalue(Pipe[int]):
def init(self, min_value: int):
self.min_value = min_value

def __call__(self, data: t.Any) -> int:
    if data < self.min_value:
        raise ValueError(f"Data value is less than {self.min_value}")
    return data

[reef vortex]

ye , for this case
i need a generic

class MinLength[T: str | list[t.Any] | dict[t.Any, t.Any]](Pipe[T]):
    def __init__(self, min_length: int):
        self.min_length = min_length

    def __call__(self, data: t.Any) -> T:
        if len(data) < self.min_length:
            raise ValueError(f"Data length is less than {self.min_length}")
        return data

do i need to add default value to generic to make it work?

[trim tangle]

Again, this is wrong. If you have this:

pipe = MinLength[str](1)
``` and you give it a list, it will tell you that it returns a string

[reef vortex]

So basically I need to give it a default value ?

[trim tangle]

MinLength[str] will still be a MinLength[str]

[reef vortex]

I am confused

[trim tangle]

pipe = MinLength[str](1)
mystery = pipe([1, 2, 3, 4])
```  the type of `mystery` here is inferred as `str` (because of the `__call__` signature). But in reality it will be a list

In TypeScript it would be something like ```ts
class MinLength<T extends string | any[] | Record<any, any>> extends Pipe<T> {
constructor(public minLength: number) {}
public validate(data: any): T {
if (data.length < this.minLength) throw new Error
return data
}
}

If you have a `pipe: MinLength<string>`, `pipe.validate([1, 2, 3, 4])` promises that it returns a string, but in reality it returns an array

[reef vortex]

oh, so how do i fix that and ensuring it return string or list based on schema?

[trim tangle]

I don't know, that's up to you 🙂
One option would be to change the signature of __call__ to (self, data: T) -> T

[terse sky]

i see that there's typing.TextIO
is there not something for more specifically just input, and just output?

[oblique urchin]

no. our general recommendation is to use tailored protocols instead of the IO types

[terse sky]

so just like write a protocol myself?

[oblique urchin]

yes, or use one from useful_types

[terse sky]

is that a library that provides useful types? 😛

[trim tangle]

$ pip install useless-types
Requirement already satisfied: typing

maybe i should make a package with that name, just for the memes

[terse sky]

that's... too good

i actually find a lot of the types in typing quite good

like the collection types

[trim tangle]

you mean the ones you're supposed to import from collections.abc now? 👀

[terse sky]

are you?

what's the difference

[trim tangle]

If you mean Callable, Iterable etc., importing them from typing was deprecated in PEP 585

!pep 585

[rough sluice]

**PEP 585 - Type Hinting Generics In Standard Collections**

Status

Final

Python-Version

3.9

Created

03-Mar-2019

Type

Standards Track

[oblique urchin]

my personal goal as a core developer is to make sure we don't go through with that deprecation for a very long time

[trim tangle]

lol

That's understandable

maybe the right thing would be to soft-deprecate them?

[oblique urchin]

I think we effectively did that

[terse sky]

I was thinmking more of Mapping, Sequence

[trim tangle]

those too

[terse sky]

I get the sentiment but from an end user perspective this is just pointless busy work

[trim tangle]

yep

[terse sky]

honestly this makes me just want to create a typing prelude

[trim tangle]

"🤓 these terms you're using are deprecated"

(instead of doing actual work on your code)

[terse sky]

and then start all of our files with

from typing_prelude import *

[trim tangle]

What would it add?

[tranquil turtle]

from __future__ import annotations
import typing as t
import collections.abc as t_abc

[brisk hedge]

It would consolidate all the types currently commonly used in typing

so typing, typing_extensions, collections.abc, ...

[tranquil turtle]

and types

[terse sky]

only the really bog standard annotations

the abstract container/mutable, probably callable and iterable

maybe something for file handles

basically: types that correspond to things that you create in the core language

if I'm using a python timezone and that comes from a library, fine, I can import the timezone type at the same time as I'm doing other stuff

Especially for the abstract container annotations its actually kind of harmful if they're not easy to access

having people passing in list, dict, and set when they typically need Sequence, Mapping, and Set is a bad thing

[viscid spire]

Set?

maybe meant AbstractSet or something?

[terse sky]

https://docs.python.org/3/library/collections.abc.html#collections.abc.Set

oh, and Optional, Tuple, Union

[viscid spire]

Why do you need Optional and Tuple

[terse sky]

honestly, probably not a popular take but I feel like adding list, dict, and set as type annotations was maybe kind of a bad thing

[viscid spire]

I get Set for frozenset

[terse sky]

Set isn't there for frozenset

Set is there for the same reason that Sequence and Mapping are there

Oh, I guess you meant "why do you need Optional and Union", rather than Tuple

[viscid spire]

Variance?

[terse sky]

thats true, Union is no longer needed.

I still like Optional[T] better than T | None

[terse sky]

to prevent accidental mutation

[slender timber]

me too

I would like nullable syntax in python

T?

[terse sky]

if you pass a list to a function, one of the most useful guarantees to have is that the function doesn't mutate the list, and to have that both communicated to the user, and enforced inside the function

most languages nowadays try to have a way to do that

in python, you would do that by annotating with Sequence

if you annotate with List or list then all bets are off, the function can mutate to their hearts content

[slender timber]

code mutating lists isn't a pattern i like anyways

it reminds me of C

[terse sky]

it's useful soemtimes but it should certainly be the rarer case

but that's why it's good to annotate with Sequence, then if the function does lst.append(5) mypy flags it as an error

[oblique urchin]

I have been thinking of adding a check to pyanalyze that disallows mutating lists that are "owned" by another function, but haven't had time to finish implementing it.

[viscid spire]

oh I see, Set vs MutableSet

[terse sky]

idk how you would determinate that. but for me, I feel like the tools are already there.

[slender timber]

i try to use those collections.abc stuff in my code as much as i csn

[terse sky]

yeah, the naming is very confusing 😦

List is list, but Set isn't set, rather Set is to set as Sequence is to List

[viscid spire]

isn't there builtin frozenset tho?

[terse sky]

terrible

[slender timber]

this sounds so much c++ to me

[terse sky]

frozenset is not the same thing

[slender timber]

we are introducing ownership concepts in python now?

[viscid spire]

sadge

[terse sky]

FrozenSet is an immutable set.

Set is a read-only API to a set

the set doesn't have to be immutable

[viscid spire]

No not FrozenSet

[terse sky]

in most cases, what you actually want is the read-only API: you don't care if the container is actually immutable

[viscid spire]

frozenset built-in

But yeah I see what you mean

[slender timber]

couldn't agree with this more

[viscid spire]

I think it's a terrible choice because it doesn't actually mean "optional"

[viscid spire]

No I meant Tuple

[terse sky]

the thing is that function arguments are possibly a majority of all type annotations... and the most convenient type annotations (list, dict, set) are almost never what you want

[slender timber]

we end up worrying too much about the shape rather than the contract, the interface

[viscid spire]

Union you can spread into ig

[terse sky]

why wouldn't I want Tuple?

I use it pretty oten

[viscid spire]

because there is tuple

[terse sky]

oh, I didn't realize that

ah well

[viscid spire]

it adds nothing

[terse sky]

so no Union, no Tuple. So its just the read-only/mutable containers (6 types), Iterable, callable, maybe some kind of protocols for files

[oblique urchin]

Self?

[terse sky]

I think that's basically it

oh, is that a thing now?

lol I can't keep up

can I now do -> Self on static and class methods?

[viscid spire]

yes, but why would you have a static method with that return type

[oblique urchin]

I think we disallow it on static methods

[terse sky]

@viscid spire it's a really really common pattern

[oblique urchin]

https://typing.readthedocs.io/en/latest/spec/generics.html#valid-locations-for-self

[viscid spire]

classmethod should be used for the pattern you are thinking

Alternate constructor

[terse sky]

in a lot of cases it makes no difference

but if Self just works with classmethod, then I would use that

[viscid spire]

classmethod works with inheritance better

and yeah allows nice typehint

[terse sky]

I'm usually avoiding using inheritance that way; when I have used inheritance with classmethods that create the instance, I didn't find it a particularly enjoyable experience anyway

you have to use kwargs to make it work properly, dont you

[viscid spire]

What?

To follow LSP, you need to be able to call a child exactly the same as the parent

This means extra parameters need a default value IG?

And if parameter names differ, the parent parameter need to be positional I presume

[terse sky]

i don't really see how LSP applies, or at least, necessarily applies, when it applies to obtaining the instance to begin with

LSP wants instances of the subtype to be substitutable for instances of the base type; that says nothing about the process of obtaining an instance

it's very common in inheritance hierarchies for derived classes constructors to have more arguments than base classes

[regal summit]

from abc import ABC, ABCMeta

class StatMeta(ABCMeta):
    def __getitem__(cls, name: str) -> type[int]:
        return int 

class Stat(ABC, metaclass=StatMeta):
    ...

class PlayerStat(Stat):
    ...

def foo(p: PlayerStat['name']) -> None:
    pass

print(foo.__annotations__)
# {'p': <class 'int'>, 'return': None}
``` I have a structure like this, Im trying to have the type checker understand that inside of `foo`, `p` will be an integer. The code itself works perfectly, Im just having trouble making the type checker understand and I cant seem to figure out why its not liking it, anyone got an idea?

[rare scarab]

You can replace StatMeta with __class_getitem__

[regal summit]

oh thats a thing???

[rare scarab]

It's how generics are implemented

[regal summit]

lol I always thought it was some messy stuff like my example

thanks

[rare scarab]

It was originally. It's why this protocol was added

[regal summit]

from abc import ABC


class Stat(ABC):
    def __class_getitem__(cls, name: str) -> type[int]:
        return int


class PlayerStat(Stat):
    ...


def foo(p: PlayerStat['nameddd']) -> None:
    pass


print(foo.__annotations__)
# {'p': <class 'int'>, 'return': None}``` ah still getting this from ruff, pylance seems to be happy though

[rare scarab]

!d object.class_getitem

[rough sluice]

object.\_\_class\_getitem\_\_

classmethod object.__class_getitem__(cls, key)```
Return an object representing the specialization of a generic class by type arguments found in *key*.

When defined on a class, `__class_getitem__()` is automatically a class method. As such, there is no need for it to be decorated with [`@classmethod`](https://docs.python.org/3/library/functions.html#classmethod) when it is defined.

[rare scarab]

This isn't for making custom typing logic

[paper salmon]

seems a bit odd, you're trying to make the runtime annotation specifically show as an int, but for pyright to see it as an instance of PlayerStat?

[regal summit]

no well my context is VERY specific, im not calling foo directly, this is for a wrapper im writing for a scripting language inside of a game

its really weird but I have a bunch of Stat subclasses which need a name

[regal summit]

do you know of a way to make it work for my use case

[rare scarab]

Conditional declarations with if TYPE_CHECKING:

[paper salmon]

and the wrapper requires the runtime annotation to return int?

[rare scarab]

!d typing.NewType may also be useful

[rough sluice]

typing.NewType

class typing.NewType(name, tp)```
Helper class to create low-overhead [distinct types](https://docs.python.org/3/library/typing.html#distinct).

A `NewType` is considered a distinct type by a typechecker. At runtime, however, calling a `NewType` returns its argument unchanged.

Usage:

```py
UserId = NewType('UserId', int)  # Declare the NewType "UserId"
first_user = UserId(1)  # "UserId" returns the argument unchanged at runtime

[regal summit]

no it will be my own custom class but I did not think that was relevant

[rare scarab]

Why not normal generics?

Do you require str type mapping?

[regal summit]

do you mean PlayerStat? I do need this, this I am initiating at runtime

hmm maybe generics will work

Ill have the function object foo and need to abstract what all the arguments are annotated as, as well as the name given

[rare scarab]

What about using aliases?

Or typed attributes?

Overloads may also be useful

[regal summit]

oh generics wont work of course since im passing in a string instance not the actual type

[rare scarab]

Overloads it is

[paper salmon]

or is PlayerStat supposed to be like metadata for that parameter?

[regal summit]

hold on I can maybe give a quick example of what the wrapper does might make it more clear

[paper salmon]

Im trying to have the type checker understand that inside of foo, p will be an integer.
Annotated would be a way to add runtime metadata to an annotation, for example: ```py
from typing import Annotated, get_type_hints

def foo(p: Annotated[int, PlayerStat("health", maximum=1000)]): ...

get_type_hints(foo) # {'p': <class 'int'>}
get_type_hints(foo, include_extras=True) # {'p': typing.Annotated[int, PlayerStat(name='health', maximum=1000)]}```

[regal summit]

from pyhtsl import (
    PlayerStat,
)

x = PlayerStat(name='x')
y = PlayerStat(name='y')

x += 10
y -= x * 2``` Id have python code like this, during runtime it would write to some file with this content:```
stat x += "10"
stat temp1 = "%stat.player/x%"
stat temp1 *= "2"
stat y -= "%stat.player/temp1%"```

[regal summit]

Im trying to have the declaration of foo be very simple, that is what you will actually write yourself, it doesnt 'really' have to make sense since Im not actually calling the function

Id like to have the syntax of a function like foo py def foo(p: PlayerStat['nameddd']) -> None: pass and do whatever is needed in the back to have it behave how I want it to

[viscid spire]

from pyhtsl import (
    PlayerStat,
)

skull

[rare scarab]

A descriptor could be useful as well

[regal summit]

haha i removed a bunch of imports to make the example

[rare scarab]

The type changes depending on if you get it from the class or an object

[regal summit]

im not familiar with this let me look into it

[rare scarab]

!d object.get

[rough sluice]

object.\_\_get\_\_

object.__get__(self, instance, owner=None)```
Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). The optional *owner* argument is the owner class, while *instance* is the instance that the attribute was accessed through, or `None` when the attribute is accessed through the *owner*.

This method should return the computed attribute value or raise an [`AttributeError`](https://docs.python.org/3/library/exceptions.html#AttributeError) exception.

[**PEP 252**](https://peps.python.org/pep-0252/) specifies that [`__get__()`](https://docs.python.org/3/reference/datamodel.html#object.__get__) is callable with one or two arguments. Python’s own built-in descriptors support this specification; however, it is likely that some third-party tools have descriptors that require both arguments. Python’s own [`__getattribute__()`](https://docs.python.org/3/reference/datamodel.html#object.__getattribute__) implementation always passes in both arguments whether they are required or not.

[regal summit]

Then Id still have to have it be an attribute no?

[rare scarab]

Yes