type-hinting

[median tangle]

my guess is still on some weird issue with that invariance there, but I'm not sure what it is exactly

[dull lance]

oh

I see the problem

[median tangle]

why are these contravariant?

class SupportsDunderLT(Protocol[_T_contra]):
    def __lt__(self, other: _T_contra, /) -> bool: ...

class SupportsDunderGT(Protocol[_T_contra]):
    def __gt__(self, other: _T_contra, /) -> bool: ...

class SupportsDunderLE(Protocol[_T_contra]):
    def __le__(self, other: _T_contra, /) -> bool: ...

[dull lance]

SupportsRichComparison uses a union while SupportsAllComparisons is basically an intersection between the individual protocols

[median tangle]

yes but that should still be fine, it's enough that either of those types in the union is used as the type there

.

[dull lance]

I guess the type checker isn't capable of expressing this "fallback" behaviour in case one of the operator definitions are missing

[median tangle]

so, a pyright bug?

[dull lance]

I had a similar problem with iterables that only implement __len__ and __getitem__ but not __iter__, they fail to satisfy the Iterable protocol even though they are iterable at runtime

[median tangle]

so, would it be safe for me use the supportsallcomparisons? what if my type only declared __gt__ and __lt__?

such types should still work with that func, but probably wouldn't meet supportsallcomparisons

[dull lance]

it would give false positive (fake errors) so it's safe

the code inside heapify itself is sound

[median tangle]

well okay yeah, it would be safe, but it wouldn't be a nice experience for users, they'd have to type ignore or make type wrappers

didn't expect this to be so annoying

[dull lance]

maybe you can wrap your function

from collections.abc import Callable, MutableSequence
from typing import TYPE_CHECKING, TypeVar

if TYPE_CHECKING:
    from _typeshed import SupportsAllComparisons, SupportsRichComparison


_T = TypeVar("_T", bound=SupportsAllComparisons)
def _heapify(lst: MutableSequence[_T]) -> None:
    """Convert a list into a **max** heap."""
    parent: Callable[[int], int] = lambda index: (index - 1) // 2  # noqa: E731

    # Iterate through the list in reverse order, skipping the first (index 0)
    # (note: we can't use enumerate here, as it wouldn't work with reversed)
    for idx in range(len(lst) - 1, 0, -1):
        elem = lst[idx]
        parent_idx = parent(idx)
        parent_elem = lst[parent_idx]

        if elem > parent_elem:
            lst[idx], lst[parent_idx] = parent_elem, elem

T = TypeVar("T", bound=SupportsRichComparison)
def heapify(lst: MutableSequence[T]) -> None:
    return _heapify(lst)  # type: ignore

def main() -> None:
    x = [5, 8, 1, 3, 6, 10, 55]
    heapify(x)
    assert x[0] == 55

[median tangle]

heapq.heapify just does list[Any] lol

[dull lance]

that way the # type: ignore is in your own code (instead of the users' code) while you still perform proper type checking inside the main logic. you can also add a comment explaining why this is the case

[wary summit]

whats going on here

[median tangle]

well yeah, that would work

[dull lance]

discussing how to work around type checkers failing to consider fallback behavior of some dunder methods

[median tangle]

specifically when it comes to comparisons, like __gt__, __lt__, ...

this was my initial q: #type-hinting message

[median tangle]

I think this makes the most sense, it's quite annoying, but oh well

[rough sluice]

:incoming_envelope: :ok_hand: applied timeout to @peak karma until <t:1716393878:f> (10 minutes) (reason: burst spam - sent 8 messages).

The <@&831776746206265384> have been alerted for review.

[dull lance]

How do I annotate a context manager that always suppresses exceptions? This doesn't seem to work

from __future__ import annotations

from types import TracebackType
from typing import Literal, Protocol, overload
from typing_extensions import Self, assert_never

class ContextManagerAlwaysReraise(Protocol):
    def __enter__(self) -> Self: ...

    @overload
    def __exit__(self, __type: type[BaseException], __value: BaseException, __tb: TracebackType) -> Literal[False]: ...
    @overload
    def __exit__(self, __type: None, __value: None, __tb: None) -> None: ...
    @overload
    def __exit__(self,  __type: type[BaseException] | None, __value: BaseException | None, __tb: TracebackType | None) -> Literal[False] | None: ...

class ContextManagerAlwaysSuppress(Protocol):
    def __enter__(self) -> Self: ...

    @overload
    def __exit__(self, __type: type[BaseException], __value: BaseException, __tb: TracebackType) -> Literal[True]: ...
    @overload
    def __exit__(self, __type: None, __value: None, __tb: None) -> None: ...
    @overload
    def __exit__(self,  __type: type[BaseException] | None, __value: BaseException | None, __tb: TracebackType | None) -> Literal[True] | None: ...

class ContextManager(Protocol):
    def __enter__(self) -> Self: ...

    @overload
    def __exit__(self, __type: type[BaseException], __value: BaseException, __tb: TracebackType) -> bool: ...
    @overload
    def __exit__(self, __type: None, __value: None, __tb: None) -> None: ...
    @overload
    def __exit__(self,  __type: type[BaseException] | None, __value: BaseException | None, __tb: TracebackType | None) -> bool | None: ...

def foo(ctx: ContextManagerAlwaysReraise):
    with ctx:
        raise ValueError

    assert_never(0)

def bar(ctx: ContextManagerAlwaysSuppress):
    with ctx:
        raise ValueError

    assert_never(0) # This line should fail

def baz(ctx: ContextManager):
    with ctx:
        raise ValueError

    assert_never(0)

(I'm using Pyright btw)

[oblique urchin]

assert_never(0) should never fail, 0 is not of type Never

[dull lance]

Actually that is why it should fail, since int cannot be assigned to an argument of type Never.

But that's besides the point, the idea is that I want that line to be interpreted as "reachable" by Pyright

currently it's interpreted as being unreachable

[dull lance]

Seems that this works

class ContextManagerAlwaysSuppress(Protocol):
    def __enter__(self) -> Self: ...

    def __exit__(self,  __type: type[BaseException] | None, __value: BaseException | None, __tb: TracebackType | None) -> Literal[True]: ...

Not sure why allowing __exit__ to return None when None is passed to each argument would break it though. From my understanding, None is passed to each argument only when no exception is being thrown inside the context manager, so it should have nothing to do with whether the context manager suppresses exceptions.

so something like this should still work in theory

class ContextManagerAlwaysSuppress(Protocol):
    def __enter__(self) -> Self: ...

    @overload
    def __exit__(self, __type: type[BaseException], __value: BaseException, __tb: TracebackType) -> Literal[True]: ...
    @overload
    def __exit__(self, __type: None, __value: None, __tb: None) -> None: ...

[tranquil turtle]

i dont think you can, and dont think you should

i mean, it probably will work with some typechecker that supports this, but might not work with other

[dull lance]

contextlib.suppress could also be potentially defined as this:

# Need to update _ExitT_co to support TypeGuard
_T_co = TypeVar("_T_co", covariant=True)
_ExitT_co = TypeVar("_ExitT_co", covariant=True, bound=bool | None | TypeGuard[object], default=bool | None)

@runtime_checkable
class AbstractContextManager(Protocol[_T_co, _ExitT_co]):
    def __enter__(self) -> _T_co: ...
    @abstractmethod
    def __exit__(
        self, exc_type: type[BaseException] | None, exc_value: BaseException | None, traceback: TracebackType | None, /
    ) -> _ExitT_co: ...

E = TypeVar("E", bound=BaseException)
class suppress(AbstractContextManager[None, TypeGuard[type[E]]], Generic[E]):
    def __init__(self, *exceptions: type[E]) -> None: ...

    def __exit__(
        self, exctype: type[BaseException] | None, excinst: BaseException | None, exctb: TracebackType | None
    ) -> TypeGuard[type[E]]: ...

(Whether it's worth the effort for type checkers to support this is another question altogether xD)

[carmine phoenix]

I just found out Ruff isnt a type checker

which do you guys recommend?

[trim tangle]

If you're starting in a vacuum, I'd pick pyright

[carmine phoenix]

I do have experience with type-hinting for a few years, I'm just tired of using PyCharm's builtin linter

[trim tangle]

pyright does have an extension for pycharm, apparently https://github.com/InSyncWithFoo/pyright-for-pycharm

I don't use pycharm too much, but from anecdotes and questions in forums the type checker is kind of a letdown, especially for such a powerful editor

[carmine phoenix]

thats neat, thank you

[carmine phoenix]

yeah thats exactly why I want to change ...

while I'm at it, does PyRight have any bugs when type checking?
cuz PyCharm has many false warnings

[trim tangle]

Definitely put a type checker somewhere in your CI pipeline so that you don't forget to read its complaints

[rustic gull]

Guys can you explain to me what annotations are? I read docs but can't get it

[grave fjord]

Which docs did you read?

[rustic gull]

Python official docs

[grave fjord]

Which page specifically?

https://mypy.readthedocs.io/en/stable/ are another good place to start

[rustic gull]

I read PEP 484

[rustic gull]

Oh thanks : )

[grim pumice]

is thre any way to make an enum of functions?

I want to inform the type system that I have a fixed number of functions of the same type

[undone saffron]

you can, but I question whether or not you should. it should be sufficient to just provide a protocol that matches that same type

from typing import Protocol

class Something(Protocol):
    def __call__(self, arg1: int, arg2: int) -> int:
        ...

def foo(arg1: int, arg2: int) -> int:
    ...

def bar(arg1: int, arg2: int) -> int:
    ...

def function_taking_foo_or_bar(fn: Something):
    ...

presumably if these functions are all being used the same way, this is sufficient and more flexible, but

from enum import Enum

def _foo(arg1: int, arg2: int) -> int:
    ...

def _bar(arg1: int, arg2: int) -> int:
    ...

class FnEnum(Enum):
    FOO = _foo
    BAR = _bar

def function_taking_foo_or_bar(fn: FnEnum):
    ...

works fine

[terse sky]

is there a way to get a "fully typed" set of overloads in both interface and implementation without duplicating signatures?

@overload
def process(response: None) -> None:
    ...
@overload
def process(response: int) -> tuple[int, str]:
    ...
@overload
def process(response: bytes) -> str:
    ...
def process(response):
     if response is None::
          return process_impl_none()
    if isinstance(response, int):
        return process_impl_int(response)
    return process_impl_bytes(response)

def process_impl_int(response: int) -> tuple[int, str]:
    # implement

def process_impl_bytes(response: bytes) -> str:
    # implement

[undone saffron]

I don't think so currently (if someone has a neat trick, I'd love to see it). If intersections and the thing to do TypeOf[fn] were to exist, you probably could since these dont have overlapping domains.

[terse sky]

couldn't the overload decorator have helped us with this

@overload("_none")
def process(response: None) -> None:
    # actually implement this now

def process(response):
     if response is None:
          return process_none()
    # etc

[oblique urchin]

are you looking for functools.singledispatch?

[terse sky]

no

that's a) limited to single dispatch, and b) it's going to start processing type signatures at runtime

I actually got interested in this specifically because I was helping someone who started with singledispatch

they had a member function of Foo that returned a Foo

which seemingly broke singledispatch, probably because singledispatch was trying to process that Foo return type annotation eagerly before it made sense

Is there any reason though, even now, that an argument couldn't be added to the overload decorator, to solve this issue?

It even seems backwards compatible

[oblique urchin]

It could be, but do you envision this doing something like sys._getframe(1).f_globals[func.__name__ = argument] = func or something? that seems pretty confusing

[terse sky]

hmm, right, decorators cannot "easily" change function names, right?

[oblique urchin]

they can't control what name the function gets assigned to

[terse sky]

@foo
def bar()
is syntactic sugar for
bar = foo(bar)

[undone saffron]

I don't particularly see this duplication as a huge issue (I have a bit of it myself), but if we're talking hypotethical fixes for this, I'd rather there be a way to register multiple concrete implementations for a set of overloads, which ends up extremely close behaviorally to functools.singledispatch (I understand the difference...) so it's kindof hard to justify

[terse sky]

I was going to say next that registration seems like the obvious choice.
personally I think this is very different from singledispatch. singledispatch has a lot of limitations and it can easily get confusing. This is just a way to handle typing, and retain control of the dispatch

[undone saffron]

something like

@overload
def process(response: None) -> None:
    ...
@overload
def process(response: int) -> tuple[int, str]:
    ...
@overload
def process(response: bytes) -> str:
    ...
def process(response):
     if response is None::
          return process_impl_none()
    if isinstance(response, int):
        return process_impl_int(response)
    return process_impl_bytes(response)


@process.overload_implementation(bytes)
def process_impl_bytes(response):  # no types needed
    ...

[terse sky]

hmm there is get_overloads now

[oblique urchin]

you could get this out of typing.get_overloads() yes

[terse sky]

Yeah so you can already solve this problem

fair enough!

[undone saffron]

I guess with the funny decorator that overwrites the signature of something + typing.get_overloads you could, but I don't think typing.get_overloads works statically does it?

[terse sky]

it doesn't, but the point here isn't to work statically

[oblique urchin]

what do you mean by "statically" here? in your example you could write something like:

def process(response):
    overloads = get_overloads()
    if response is None:
        return overloads[0](None)

[terse sky]

you're still writing the logic in the main process function, by hand, to decide which function in get_overloads to call

[undone saffron]

statically: type checkers being able to seeing the type

[oblique urchin]

I see, yes you're right

[terse sky]

I mean that's not really possible, afaics

get_overloads would have to have a tuple that changes as things are called on it

[undone saffron]

This is one of the things I definitely prefer in some other languages, and singledispatch (As well as things like plum-dispatch) don't quite fill the void

[terse sky]

the return type of get_overloads would need be tuple[Callable[None, [None]], Callable[tuple[int, str],[int]], Callable[str, [bytes]]]

and similarly to keep changing as you add things

[oblique urchin]

that seems theoretically possible with type checker special-casing

[terse sky]

maybe, assuming you ignore "sketchy" things like writing overloads in other files or things of that nature

is get_overloads guaranteedd to return in the same order as they were registered?

So, here's a thought: it would be nice IMHO if overloads took an optional string argument that let it be placed in a dict, rather than a sequence

[undone saffron]

what would the key for the dict be? parameters are slightly annoying to be precise with given that there are 5 kinds of parameters

[terse sky]

@overload("none"
def process(response: None) -> None:
    ...


def process(response):
    ols = typing.get_overloads_dict()
    if response is None:
        return ols["none](response)
    ...

[oblique urchin]

Overloads have to be contiguous, at least in the static type system

[terse sky]

this would already be a lot less error prone

[oblique urchin]

get_overloads() will do funny things if you don't follow that rule, I think we clear the list if the linenos aren't monotonically increasing, so the list doesn't keep growing if you reload a module

[terse sky]

just a name. maybe it's just a string but in practical terms people are a lot less likely to get a name wrong that they chose than an ordering index

[undone saffron]

ohhh

I see you're naming them in the overload deco itself, thanks for that example

[terse sky]

if isinstance(response, int):
    typing.get_overloads_dict()["none"](response)

definitely has some smell to it

whereas typing.get_overloads()[1](response) 🤷‍♂️

and this is a pretty trivial, and backwards compatible change

anything without a name simply doesn't get entered into the dict

[undone saffron]

This seems viable and non-problematic to me, but I'm not sure how much support you'll get for it.

[terse sky]

yeah I admit it's probably not some kind of crazy win

[undone saffron]

probably needs to come with spelling out some of the itnernal details to ensure it matches: #type-hinting message

[terse sky]

what's the easiest way to float it?

[oblique urchin]

typing category in discuss.python.org

but yeah I'm not sure I'd support it, the use case doesn't feel very common, the resulting syntax is not attractive, and it would need some type checker special casing to support

[terse sky]

for what I'm suggesting, you don't need any special casing to support it

[oblique urchin]

I guess you need special casing to make it type-safe

[terse sky]

but it's already not type-safe

this is no worse

[oblique urchin]

(Type checkers would have to infer some sort of TypedDict for the return of get_overloads_dict.)

[terse sky]

you can get partial type safety by having _impl functions that duplicate the type signatures, and calling them

and also duplicating the type signature a third time

[oblique urchin]

with that you still have type safety inside the implementation of the overload body

of course the type checker doesn't enforce that you actually obey the overload signature

[terse sky]

sorry, when you say "with that" - are you referring to my idea, or the approach of calling _impl functions?

[oblique urchin]

with the existing approach

[terse sky]

you have it in my approach too? I don't see why that needs special casing

[oblique urchin]

you call get_overloads_dict()["none"](None)

but get_overloads_dict()["none"] would be Callable[..., Any]

[undone saffron]

calling typed impl function retains type safety inside the implementation. calling the actual provided overloads would require type checkers to synthesize some knowledge (typeddict is likely the best canidate for it, but realistically, it could be something that is only statically allowed from inside the implementation so that type checkers can choose not to and special case it) for it to remain type safe in the body

https://pypi.org/project/plum-dispatch/

you might want to take a look at this and search discourse for it a bit.

there's been discussions about better overloads/dispatch, and this was one of the libraries that has shown battle testing (and been brought up in these discussions)

[terse sky]

okay, maybe the phrase "overload body" is confusing to me then -

@overload("none")
def process(response: None) -> None:
    return None

this to me is an "overload body" and I don't see why it needs special casing to be type checked

[terse sky]

but i'm not interested in the dispatch problem 🤷‍♂️

[oblique urchin]

I guess I meant the overload implementation, the one that doesn't have @overload

[terse sky]

right, so it has partial type checking only, provided that you are willing to duplicate the type signature a third time

actually, this started partially because someone was pointing out that you don't get proper type checking if you implement everything inline

@overload
def process(response: None) -> None:
    ...
@overload
def process(response: int) -> tuple[int, str]:
    ...
@overload
def process(response: bytes) -> str:
    ...

def process(response: None | int | bytes) -> None | tuple[int, str] | str:
    # whole implementation here

this doesn't get type checked correctly, of course

[undone saffron]

yes it does?

[terse sky]

well, then we differ in "correct" 🤷‍♂️

[oblique urchin]

the type checker doesn't enforce that you return None on a none argument

[terse sky]

right

[oblique urchin]

but it does check the body of the type checker the same as it would for a non-overloaded function

[undone saffron]

I'm pretty sure pyright does, 1 sec

[terse sky]

That's why I initially suggested to that person that they factor out individual implementation functions; process_none etc

because those functions can actually be type checked correctly

[terse sky]

that would be pretty cool

[undone saffron]

it doesnt, but I could have sworn I've seen an error for this before from some tooling

🤔

[terse sky]

yeah. anyhow, so my point is I guess that I'm not asking with this change for any special support from the tooling - even without any special support from tooling, it's a trade-off

[oblique urchin]

I think it does for contrained TypeVars, maybe you're confused with that

[undone saffron]

oh

hold on, I didnt notive pyright-play was stuck waiting for server

lemme load this up locally... edit: and yeah, isn't checked

[terse sky]

• with my approach, the "dispatch" code does not get type checked at all
• but the bodies are type checked fully correctly, with no duplication of signatures
• with current approaches, you duplicate signatures 2x to get "partial" type checking, and 3x to get "full" type checking

[oblique urchin]

why 3x?

[terse sky]

because: once on the overloads, once on the _impl's, and once on the actual implementation function in union|ed form

@overload
def process(response: None) -> None:
    ...
def process_none(response: None) -> None:
    # implement
def process(response: None | int | bytes) -> None | tuple[int, str] | str:
    if response is None:
        return process_none(response)
    # impelment rest

the None appears three times

response: None appears three times, i should say

and note: this still isn't fully, fully type checked correctly

type checking the dispatch is imperfect anyway

sacrificing that type checking to avoid repetition while having fully type checked bodies seems like a good trade-off

[undone saffron]

hmm

I wonder if we Should specify that inside the body of an overloaded function, when the input type is known, any return must be consistent with the overloads

cause this is a tractable problem

[rustic gull]

I have a subclass of TypedDict

class InfoResponse(TypedDict):
  # This class in total has around 40 keys
  account: str
  accountType: Any
  country: str
  clientIp: str
  ...

I would need another TypedDict for typehinting a return type of some function. The thing is that this new TypedDict will have same keys as InfoResponse but won't have all of it's keys. In fact it will also have some new keys. Thats what it would look like

class NewTypedDict(TypedDict):
  # This class has around 39 keys
  # clientIp from InfoResponse is removed
  # accountType from InfoResponse is removed
  # Those both keys don't exist anymore

  account: str
  country: str
  region: str
  # Rest of the keys from InfoResponse
  ...

  # NOTE: "region" key doesn't exist in the InfoResponse

Im curious if I would have to rewrite ALL of those keys & types and place them in NewTypedDict or is there any faster way to create a TypedDict like this one?

[rustic gull]

I created a temporary "solution" by creating new classes that inherit from TypedDict

def __modify_info_response(name: str, base: Type["InfoResponse"], remove: List[str], add: Dict[str, Type[Any]]):
  base_keys = {key: type_ for key, type_ in base.__annotations__.items() if key not in remove}
  new_fields = {**base_fields, **add}
  return type(name, (TypedDict,), new_fields)

Im open for any ideas if it needs some improvements (it probably does)

[brisk hedge]

You can subclass typeddicts, so you can define the common fields in a base class and add the specific new fields in the subclasses

[trail kraken]

Are there usable alternatives to TypedDict in 3.6?

[brisk hedge]

typing_extensions backports typing features to previous versions, but 3.6 is below the minimum officially supported version for Python or for the library so your experience is not guaranteed

[rustic gull]

I haven't thought of that

tysm

[trim tangle]

3.6? 💀

my condolences

[trail kraken]

I'll probably conditionally assign to the type based on sys.version, if there are no alternatives. typing_extensions would be a good choice for the future, but for now... I don't get to change the dependencies.

And at least whoever set up the project chose 3.6. Default would've been 3.5...

Could've been worse!

[brazen jolt]

ah yes, can't ignore unnecessary type ignore comment, because it's an unnecessary ignore, thanks pyright! ```
Pyright: Unnecessary "# pyright: ignore" rule: "reportUnnecessaryTypeIgnoreComment" [reportUnnecessaryTypeIgnoreComment]

[trim tangle]

you should get some good running shoes

[trim tangle]

just add an error to that line

🙂

[supple bay]

slightly offtopic to this channel but if i have an init declaration such as

class Error:
    def __init__(
        self,
        code: str,
        status: int,
        message: Optional[
            Union[FormattableString, LocaleStrings, str]
        ] = None,
        *,
        _property: Union[FormattableVar, str] = "global",
        **attributes: Any
    ) -> None:

is there a preferred way to annotate this? im putting the * to block positional arguments past the message because i feel like the style of that would work well with message being a positional argument but im countering myself a lot on whether it actually is, especially considering message can be type string so it would be

Error("failed", 500, "it failed!")

which might look better as

Error("failed", 500, message="it failed!")

is there anything i can reference on where to use the *? personally i already always specify arguments with defaults with keywords in my codebase so adding these * wont require too much more work out of me, but the question is whether for usability/style to change every function to require arguments with defaults to be specified or to use the * tastefully (though thats kind of difficult to judge)

[supple bay]

im translating much of the codebase into typescript which wont allow me to specify arguments in both ways and its probably easier to make every argument with a default specified by keyword like

class ErrorTemplate {
    constructor(
        code: string,
        status: number,
        { message }: { message?: string | FormattableString }
    ) {

[trim tangle]

using keyword arguments is almost always better than positional ones

This is my compass, kinda ```py
fn() # chef's kiss
fn(a) # great, it's probably obvious what the argument means
fn(a, b) # sometimes it's obvious, sometimes not. use your judgement
fn(a, b, c) # a bit iffy.
# re.sub(pattern, repl, string) (or is it re.sub(pattern, string, repl)?)
fn_with_varargs(a, b, c, d, e, f, g) # as many as you want. though maybe consider just accepting a list
fn(True) # probably not, but there will be exceptions
fn(None) # probably not, but there will be exceptions
fn(a, True) # very likely bad
fn(False, True) # NO
fn(False, None, None) # you are fired
fn(a, ignore_errors=True, batch_size=4) # good
fn(a, ignore_errors=True, batch_size=4, <10 more options>) # now you have a bigger problem

[supple bay]

tysm for sharing

ill try to clean that stuff up a bit in my code

[wild valley]

@oblique urchin do you suppose the typing conformance suite could add support for run-time type checkers? I think it'd mostly need to add calls to functions defined in the modules, and .pyi files should not factor into the score, at least directly

frankly I was completely oblivious to the existence of the suite before the typing summit

[tacit sparrow]

is it possible to define abstract attribute somehow (not property)?

from abc import ABC, abstractmethod

class Base(ABC):
    @property
    @abstractmethod
    def name(self):
        pass

# Expression of type "Literal['test']" is incompatible with declared type "property"
#   "Literal['test']" is incompatible with "property"
class Child(Base):
    name = "test"

child = Child()

[tacit sparrow]

found it, it's not possible

https://discuss.python.org/t/abc-add-abstract-attributes-via-abstract-type-hint/26164/13

[dull lance]

this is not really a good example. you're defining a class attribute instead of an instance method so it will fail the type checker regardless of whether it's abstract

if you want to have an "abstract attribute", consider using Protocols to specify the interface that's needed. this separates the interface from the implementation

[tacit sparrow]

that actually works, thank you

from typing import Protocol

class BaseProtocol(Protocol):
    name: str

class Base(BaseProtocol):
    # name = "25"
    pass

# Cannot instantiate abstract class "Base"
#   "BaseProtocol.name" is not implemented
child = Base()

though it's a bit annoying that this code doesn't show any issues

from typing import Protocol

class BaseProtocol(Protocol):
    name: str

class Base(BaseProtocol):
    # name = "25"
    pass

is there way to get similar error defining the class, not instantiating it?

[rare scarab]

Try using __init_subclass__

[oblique urchin]

I wouldn't be opposed but I'm not sure what it'd look like and I think runtime type checkers have a pretty diverse set of interfaces

[heavy pond]

if have an c extension module and you want to add type hints for it, do you just need a module_name.pyi file, or do you need a py.typed file too?

I saw this but it only has pure python so I wasnt sure https://dev.to/whtsky/don-t-forget-py-typed-for-your-typed-python-package-2aa3

[tacit sparrow]

__init_subclass__ can help ensure that subclass will have certain property but it doesn't help in showing a type hint if it doesn't

[wild valley]

I can only speak for my own project, but typeguard's import hook based instrumentation would handle that fairly easily, the test suite just needs to call the functions to trigger the checks

IIRC Cecil is working on a similar mechanic in Beartype if it's not there already

[wild valley]

yes, you would need both

[tranquil ledge]

I think both are necessary.

Ope, didn't see the other response. My bad.

[rare scarab]

Py.typed is only not required if the package is named types-xxx

[heavy pond]

oh interesting ok

[arctic cloud]

says

says

[carmine phoenix]

how can I put the comment to surpress the warning above the line, instead of on the side?

cuz then I get the warning of "line too long (pep8)"

[carmine phoenix]

I never understood the whole point of the py.typed file

is it only to say to the linter that its type-hinted so to analyze it?

for example pandas dosent seem to have it: https://github.com/pandas-dev/pandas/tree/main/pandas

[rare scarab]

Yeah. If it doesn't exist, mypy doesn't even bother reading the package.

[carmine phoenix]

that seems like a bad design on mypy's part

why dont they scan the package one time when its installed

[rare scarab]

though pandas provides .pyi files in pandas/_libs

[carmine phoenix]

and mypy will read them if there is no py.typed file?

[oblique urchin]

the original reasoning was that mypy may crash on untyped code, so it should only look at code that is meant to be type checked

I'm not convinced this is still a good reason

[rare scarab]

And for the record, pyright always type checks all packages, whether a py.typed marker is present or not

[hallow flint]

i think type checkers should still issue diagnostics when importing from packages without py.typed (so you know you're getting shitty types), and then go and do the best it can

[cunning plover]

turn_types.py

import importlib
import pathlib
import sys

lib_names = sys.argv[1:]

for lib_name in lib_names:
    try:
        lib = importlib.util.find_spec(lib_name)  # type: ignore[attr-defined]
        for loc in lib.submodule_search_locations:
            typed_file = pathlib.Path(loc) / "py.typed"
            print(f"{loc=}, {typed_file.exists()=}, touching file")
            typed_file.touch()
    except Exception as err:
        print(f"{err=}")
        continue

      - python3 turn_types.py requests dill django_filters setuptools

my favourite hack

without effort to install and keep up with stub libs 😄

[rough sluice]

:incoming_envelope: :ok_hand: applied timeout to @rustic gull until <t:1717086164:f> (10 minutes) (reason: duplicates spam - sent 4 duplicate messages).

The <@&831776746206265384> have been alerted for review.

[undone saffron]

I think there's good reasons against this.

Allows a project to incrementally add types to things internally without the issues of that being exposed to it's users until they are ready. Very much works well with the spirit of gradual typing

py.typed says "We're ready enough to take bug reports from our users for types now"

[rare scarab]

Which is why .pyi files exist for public apis.

[hardy linden]

Am I wrong to expect that reveal_type(bar) in the below example should yield str instead of str | None? (pyright, if it matters.)

def foobar(foo: str | None, bar: str | None):
    if foo is None and bar is None:
        raise ValueError("foo and bar can't both be None!")
    if foo is None:
        reveal_type(bar)  # Expected `str`, got `str | None`

[terse sky]

I mean you're wrong in the sense that you're expecting too much

[grave fjord]

Yes, for example if bar is not none

[terse sky]

like, obviously control flow like that can be arbitrarily complex - how much do you expect the type checker to work through?

[grave fjord]

Oh I'm missing the logic

[terse sky]

yeah

[undone saffron]

I'd expect that to narrow

[terse sky]

the problem is that basically, the more complex you make the narrowing/auto-casting logic, people will depend on it in progressively more complicated situations, and then it can break in more surprising ways when it does break - either the logic actually changing so it's not guaranteed, but worse, eventually you hit the "limit" of how much logic the type checker processes

[hardy linden]

Yeah, that makes complete sense, I'm just kind of used to how well the narrowing normally works

[terse sky]

it's also inconsistent with how any mainstream statically typed language works - languages that have smart casting (Kotlin) or pattern matching (Rust) also have this "limitation" - you can hit points in the code where its logically guaranteed that a type has been narrowed in a certain way, but the static type system isn't aware of it.

[hardy linden]

But I guess narrowing based on multiple different variables is a bit of a jump

[terse sky]

yeah, I think usually the narrowing works purely off of that variable being mentioned in the condition, though there could be counter-examples to that

[undone saffron]

It's possible to handle any amount of chaining of things that verify a type across any number of values soundly

[terse sky]

well, to do that you need a rigorous theory of what counts as a "static" expression

[undone saffron]

it isn't possible for arbitrary expressions, but for combinations of conditionals which have narrowing behavior, it is. We have a definition of such conditionals since the acceptance of TypeIs

[terse sky]

you could do it for this case and similar cases - the question is whether the benefits are worth the downsides 🤷‍♂️

[undone saffron]

There aren't that many downsides here, and quite a few benefits to having narrowing behavior follow logically from control flow, and there are some open discussions right now, some recently accepted things, and some in progress specification updates that either would benefit from or contribute to better narrowing from control flow.

you're right that we would need to be careful to be clear on the rules of what could narrow like this is to avoid it being unclear to users or being broken though

main benefit is when people write "obvious" code, the type checker that's supposed to be good enough to be checking this over your shoulder for you doesn't tell you to write less obvious (And worse!) code making people feel like they are fighting with tools meant to help them

[terse sky]

🤷‍♂️ I just don't agree; it's easier IMHO when the type inference rules are very simple.
i don't think it's a coincidence that other languages with similar features also don't implement this.

@hardy linden the actual way to go here probably, is to use pattern matching, I would think.

I haven't used it much in python, but that's how you'd do similar things in other languages and get the "right" type automatically

[radiant jetty]

Hello, I want to have function that will have similar signiture to loop.run_in_executor

mypy is able to detect when I pass function with args that are different from args that I passed along with given function.

I am able to achieve similar behavior with ParamSpec but this need both args and kwargs.

Can you advice what typing feature I need to use to have function that will be checked and fail the same way like loop.run_in_executor in example below?

import asyncio

def x(a: int, b: str) -> None:
    pass

async def main() -> None:
    executor = ProcessPoolExecutor()
    loop = asyncio.get_running_loop()
    loop.run_in_executor(executor, x)
    """ mypy is able to detect that int and str args are missing
    mypy error: 
       Argument 2 to "run_in_executor" of "AbstractEventLoop" has incompatible type "Callable[[int, str], None]"; 
       expected "Callable[[], Any]"
    """```

[dull lance]

Perhaps you can check the type definitions for loop.run_in_executor, and copy that? Otherwise, could you explain the issue that you are facing right now?

[grave fjord]

It's Unpack

With TypeVarTuple

[trim tangle]

Honestly I'd just accept a Callable[[], T] with no arguments

if you want to pass arguments to an existing functions, use a lambda or a partial

[radiant jetty]

    pass```
works like a charm, thanks

[lunar dune]

It looks like you're on Python 3.11+, which means you actually don't need Unpack here at all, you can just use:

def run_in_executor(func: Callable[[*Ts], None], *args: *Ts) -> None:
    pass

Unpack[Ts] is the backwards-compatible version of *Ts, for versions of Python where you can't yet use * in annotations

[radiant jetty]

not always possible, I want it for example as a wrapper for creating new processes def create_process(target: Callable[[*Ts], Any], args: tuple[*Ts], daemon: bool) -> multiprocessing.Process: return multiprocessing.Process(target=target, args=args, daemon=daemon)

[trim tangle]

partial will work here

(functools.partials are picklable)

of course if you want to respect some existing API, you don't have a choice

[low escarp]

Is it possible to do postponed evalutations of annotations
Ie.

class Test:
  x: list["Test"]

but for actual typing.Annotated?
So like how to make

class Test:
  x: typing.Annotated[list["Test"], something]

work?

[tranquil turtle]

from __future__ import annotations

[low escarp]

isn't that only required if you are on <3.7?

[tranquil turtle]

this is impossible if you are on <3.7

on >=3.13 this should no longer be required

[low escarp]

I am on 3.9

actually, 3.10

[tranquil turtle]

on <3.13 just slap from __future__ import annotations in literally every file, it wont hurt

[low escarp]

haha ok sure

this seems to ruin my code which is inspecting the __annotations__ attribute of the class

with the future import of annotations it seems to be changing the types to strings (when I loop over the .items() of this dictionary

which, looking at the PEP is intended...

but I am using that info...

[tranquil turtle]

anyway, there is a function in typing module that gives you typehints for an object

[low escarp]

!typing.get_type_hints

[tranquil turtle]

!d typing.get_type_hints

[rough sluice]

typing.get\_type\_hints

typing.get_type_hints(obj, globalns=None, localns=None, include_extras=False)```
Return a dictionary containing type hints for a function, method, module or class object.

This is often the same as `obj.__annotations__`. In addition, forward references encoded as string literals are handled by evaluating them in `globals`, `locals` and (where applicable) [type parameter](https://docs.python.org/3/reference/compound_stmts.html#type-params) namespaces. For a class `C`, return a dictionary constructed by merging all the `__annotations__` along `C.__mro__` in reverse order.

The function recursively replaces all `Annotated[T, ...]` with `T`, unless `include_extras` is set to `True` (see [`Annotated`](https://docs.python.org/3/library/typing.html#typing.Annotated) for more information). For example:

[low escarp]

oh haha ok thanks

[tranquil turtle]

use this instead of .__annotations__

[low escarp]

ah, this also resolves the MRO! Great! Was wanting that too

thanks!

[oblique urchin]

3.14, not 3.13

[low escarp]

ok... I have ran into another maybe more obscure problem...
So this will fall over:

from __future__ import annotations
from typing import Annotated, get_type_hints

class Student():
    name: Annotated[str, 'some marker']

    def test(self):
        for k, v in get_type_hints(self, include_extras=True).items():
            print(k, v)

print(get_type_hints(Student, include_extras=True))
Student().test()

Commenting out the top line will work but then I will lose the ability to have forward-referencing

Sorry, I should have specified that it raises this error:

"typing.py", line 551, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'Annotated' is not defined

which originates from the get_type_hints call inside the test method

the get_type_hints call works just fine when called from the second from last line both times

[oblique urchin]

passing an explicit globals() arguments to get_type_hints() may fix this

[low escarp]

I tried this but then I think I lost the forward-referencing as it then couldn't find the class itself...

actually, I think it works when the code is in the same file, but in my case I am defining this in a subclass which is then used in a different file

from __future__ import annotations
from typing import Annotated, get_type_hints

from tst2 import Base

class Student(Base):
    name: Annotated[list[Student], 'some marker']

print(get_type_hints(Student, include_extras=True))
Student().test()

And Base as a class just has

class Base():
    def test(self):
        for k, v in get_type_hints(self, globalns=globals(), include_extras=True).items():
            print(k, v)

this raises the error NameError: name 'Student' is not defined
Again, this works when calling from outside the class, but not inside

[oblique urchin]

try get_type_hints(type(self)) instead of get_type_hints(self)?

[low escarp]

ah good idea thanks

nope

[cinder bone]

Say I have

t1: tuple[int, ...] = ...
# based on some external conditions I now know that t1 is tuple[int, int]

def foo(t: tuple[int, int]):
    ...

foo(t1) # tuple[int, ...] incompatible with tuple[int, int]

Is there any clean way to change the type of t1 without a # type: ignore?

[tacit sparrow]

cast?

[grave fjord]

TypeGuard?

[trim tangle]

But it will be a different kind of... future, right?

[oblique urchin]

that's not the current plan

[trim tangle]

I mean, it will be the PEP 649 one, right?

!pep 649

[rough sluice]

**PEP 649 - Deferred Evaluation Of Annotations Using Descriptors**

Status

Accepted

Python-Version

3.14

Created

11-Jan-2021

Type

Standards Track

[lunar dune]

this will be the future but not the __future__

the current plan is just to implement it in 3.14 as the default semantics

[trim tangle]

Yeah, as usual my attempt at inserting a pun made my message unclear

[oblique urchin]

the future will look like this: ```Python 3.14.0a0 (heads/pep649-inspect:11a96f9a61, Jun 1 2024, 08:54:47) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

def f(x: undefined): pass
...
f.annotations
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
f.annotations
File "<python-input-0>", line 1, in annotate
def f(x: undefined): pass
^^^^^^^^^
NameError: name 'undefined' is not defined
undefined = int
f.annotations
{'x': <class 'int'>}

[cinder bone]

Just looked it up, seems like a cool feature, but then I assume I have to define a function and run it through that? Seems a little tedious

[grave fjord]

Well what are your external conditions?

[cinder bone]

I don't have a concrete example (it was just floating around in my head) but if you need an example, maybe something like checking if a degree of a bezier curve is 2

(which would happen a lot in one of my programs)

bez_deg = 2

def calc_bezier_property(n_deg_bezier) -> tuple[tuple[float, ...], ...]:
    ...

def do_stuff(second_deg_bezier) -> None:
    ...

for curve in calc_bezier_property(...):
    if bez_deg == 2:  # would I use a TypeGuarded function here?
        do_stuff()

[grave fjord]

What are the possible values for bez_deg?

Pretty sure you can do if len(curve) == 2: do_stuff(curve)

[tranquil ledge]

iirc, a type-checker like pyright can narrow the type of a tuple based on a len() check. Would help here if so.

EDIT: this works with the original example at least —

t1: tuple[int, ...] = ...

def foo(t: tuple[int, int]): ...

if len(t1) == 2:
    reveal_type(t1)  # Type of "t1" is "tuple[int, int]" - Pylance
    foo(t1)  # valid in the eyes of the type checker

[cinder bone]

huh cool

didn't know that

thanks!

[stiff acorn]

how should I deal with mypy complaining about missing return?

from pathlib import Path
from shutil import which

def exe(app: str) -> Path:

    if executable := which(app):
        return Path(executable).resolve()
    else:
        Exception(f"Executable not found in PATH: {app}")

[dull lance]

you need to use the raise keyword to actually raise an exception

[stiff acorn]

oh god

think i need to go to sleep

thank you

[dull lance]

glad to help xD

[stiff acorn]

haha i cannot believe that's the mistake i was making

[restive rapids]

whats the progress on higher kinded types support, maybe there's a thread with discussions about it or something like that? is that even something thats being worked on?

[tacit sparrow]

Is there way to ensure type checkers will assume some function's __doc__ is a str and not str | None if doc-string is present?

from typing import assert_type

def test():
    """hello"""
    pass

# shouldn't raise an error
assert_type(test.__doc__, str)
# shouldn't raise an error
s: str = test.__doc__

[feral wharf]

typing.cast?

[trim tangle]

Why though

Also, there's this

$ python -OO
Python 3.12.0 (main, Oct 10 2023, 20:10:02) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo():
...     """Hmm"""
... 
>>> foo.__doc__ is None
True
>>> 

[tacit sparrow]

to reuse the doc-string elsewhere?

[trim tangle]

🤔

[dull lance]

why not just handle the case where it is None?

[tacit sparrow]

sure, it's possible to just cast to str for every doc

[trim tangle]

You can just do func.__doc__ or ""

[tacit sparrow]

oh, that actually simple enough, thank you

[oblique urchin]

as far as I know nobody is working on this

[sacred spindle]

Is there a Python 3.10+ compatible way to annotate *args and **kwargs? For annotating **kwargs I was going to use def func(**kwargs: typing.Unpack[TypedDictInstance]) but saw that Unpack is not in 3.10.

[oblique urchin]

use typing_extensions.Unpack

[sacred spindle]

so this relates to another question I'm confused about. Do I need typing_extensions if I'm going to only run a static type checker on python 3.12, but need the code itself to run on 3.10?

[oblique urchin]

if you're importing it at runtime, yes

[sacred spindle]

can I avoid that by doing if typing.TYPE_CHECKING: import typing.Unpack? ...but then I suppose I sould have the unresolved Unpack in the function signature

[tranquil turtle]

you can, but you also would need from __future__ import annotations to prevent NameErrors

[grave fjord]

Can you do **kwargs: "*TypedDictSubclass"?

[sacred spindle]

is that "legal" code? ...like, would type checkers verify that?

[sacred spindle]

doesn't look like it 😦

[grave fjord]

Weird I'd have expected it to work

[tranquil ledge]

afaik, it's not being pushed or developed beyond one mypy plugin made a while back.

[tranquil turtle]

does it work without "" ?

[sacred spindle]

nope

[tranquil turtle]

then the problem is probably that your python version is too low, and pyright doesnt allow this syntax, even in strings

[sacred spindle]

library I maintain conforms to NEP-29, so going to be a bit before I can bump it up to 3.11 ... considering the lack of annotations elsewhere in the library, we probably will just work elsewhere in the library and circle back to *args and **kwargs usage when we are at 3.11 as a minimum

[tranquil ledge]

You could make a placeholder for Unpack in your else block so that it's a viable runtime annotation.

Your case isn't quite what this example is for, but something like this (https://discuss.python.org/t/using-get-type-hints-with-cyclic-imports/44828/6) might serve your needs in a way that doesn't break at runtime while making sure your type-checkers understand your intent at type-checking time.

[azure yarrow]

chevapchichi

[viscid spire]

Technically it would be **TypedDictSubclass, but I think that was rejected or postponed or something

because mapping unpack vs iterable unpack

They are both unpacking so Unpack makes sense, but just doing a single star on kwargs type doesn't make sense

[trail kraken]

How useful is adding type hints to *args and **kwargs? Type checkers seem to raise warnings about missing arguments anyway, based on where they are used, though sometimes as false positives.

[grave fjord]

Nope it's still one star for both

[viscid spire]

https://peps.python.org/pep-0692/#introducing-a-new-syntax

it is currently only Unpack for typed dict

and possibly in the future **

[grave fjord]

Oh I see

[viscid spire]

quite important

as important as typehinting any other parameters

[trail kraken]

In what way are they important? What does adding them change? Is there an example of how to type hint them I can follow?

[sacred spindle]

I think it's useful, the alternative is I have a bazillion keyword args specified in my function signature

[grave fjord]

Well it depends what you do with them. Sometimes you need paramspec sometimes you need unpack with a TypeVarTuple and sometimes you need unpack with a typeddict

[sacred spindle]

https://pyqtgraph--3039.org.readthedocs.build/en/3039/api_reference/graphicsItems/plotdataitem.html#pyqtgraph.PlotDataItem I mean, I know all the keyword arguments and the types getting directed to ``**kwargs`

[grave fjord]

Sometimes you need to call super with passthrough args and kwargs and that's just not possible yet

[sacred spindle]

(scroll down a bit to see the tables of recognized keyword arguments)

(personally I hate how **kwargs is used in the library here, but this PR is about documentation, not looking to change things up too much)

[viscid spire]

I would say the most common way of typehinting *args is homogenous

[trail kraken]

Is one of the links supposed to be a GitHub issue?

Mm. Sounds like TypeVarTuple would be the way to go if you care about how the positional arguments. Haven't really had those cases yet.

The main use case for me is indeed forwarding calls, most frequently in __init__.

[sacred spindle]

No, a link to the docs built as part of a PR I’m making to upgrade the documentation on this class and add a bunch of type annotations.

I posted it as all the expected arguments for **kwargs are listed

[trail kraken]

Ah I see. It certainly has quite the number of arguments.

Default pylint settings would not be happy.

[sacred spindle]

Considering I got sphinx to build w/o warnings, numpydoc to have no (relevant) errors, I'm happy with that 😂 definitely not looking to add pylint into the mix 😂

this is indeed an older codebase tho... going to be a while before I can fully modernize it; will be tough to do without breaking some of the pulbic API :/

[river valley]

Does anyone know why I should use
type UserId = int

over
UserId = int

[oblique urchin]

For simple aliases like that it doesn't make too much difference, though it makes your intent more explicit

The advantage of the type statement is that it lets you have explicit generic parameters and use forward references

[trim tangle]

https://github.com/python/typing/discussions/1761
"Self type for a method parameter is an automatic LSP violation"

[soft matrix]

Doesn't that defeat the whole point of the parameter

And surely you can say the exact same thing about a type var bound to self?

[trim tangle]

Yep. It has exactly the same problem.

It's okay in the return position, but not as a parameter (ignoring the self parameter)

[carmine phoenix]

why do people go trough the hassle of making type-hints backwards compatible (i.e. using typing.List, or using typing_exstensions) when you can just use future imports (from __future__ import annotations) to avoid evaluating them on runtime?

(and you can hide declarations and imports under if TYPE_CHECKING: ...)

[oblique urchin]

Thee's a few reasons:

• People may want to support runtime type checkers, which won't work well with the future import
• Some types appear outside of annotations (e.g., in cast, TypeVar bounds, etc.) and aren't helped by the future import
• Some people may still support versions of Python that don't have the future import, or haven't updated their practices since they dropped support for those versions

[carmine phoenix]

People may want to support runtime type checkers, which won't work well with the future import
so if I create a library that uses future imports, will the user not be able to use runtime checkers with other libraries because my future imports disabled it?

[oblique urchin]

Possibly; if the runtime type checker jumps through enough hoops it might still work

[carmine phoenix]

interesting

first time I hear about that

I think MCoding said smth along the lines that you should always use future imports

[oblique urchin]

I don't know who that is

People may have different opinions based on what they use annotations for

[undone saffron]

pairing "always use the annotations future import in files with annotations" with "always have imports that can be resolved at runtime" is an actual guideline I argue for, and enforce in a relatively large codebase. The first without the second creates some problems for runtime introspection. Jelle is right that some parts of that might not matter if your code never interacts with other code that uses annotations in a way you aren't aware of.

[silk harness]

anybody try to use typechecked from typeguard? what you are using to check if your types are correct 🙂 ?

[trim tangle]

Why do you want runtime checking?

[silk harness]

maybe it's overkill

I want to be strict as possible 😄

[trim tangle]

https://decorator-factory.github.io/typing-tips/faq/runtime-checking/

In short, you can only really "check" trivial types like str, int or str | bool

If you're using a type checker reasonably well, you shouldn't really encounter issues when you provided a boolean where a string was expected

[silk harness]

thanks @trim tangle 🙂

[trim tangle]

That's not really a limitation of the tools though, it's kind of a fundamental problem with runtime checking. It's not possible to check whether a function conforms to Callable[[int], str] at runtime

And everything else stems from that (e.g. Iterator[int] is pretty much a Callable[[], int])

[bright tangle]

Why is tuple[Literal['a','b'], *T] not considered equivalent to tuple[Literal['a'], *T] | tuple[Literal['b'], *T] (in an invariant position)? Surely those types should always be equivalent?

[trim tangle]

Can you show more code perhaps?

also what is T?

[bright tangle]

This is my exact return type: list[tuple[Literal["tile", "wall"], str, str, int, int, int]]
This is the code that creates the returned object:

return sorted(
        [
            *(("tile", *tile) for tile in best_tiles),
            *(("wall", *wall) for wall in best_walls),
        ],
        key=lambda tile: (r - tile[3]) ** 2 + (g - tile[4]) ** 2 + (b - tile[5]) ** 2,
    )[:5]

This is the exact error:

Expression of type "list[tuple[Literal['tile'], str, str, int, int, int] | tuple[Literal['wall'], str, str, int, int, int]]" is incompatible with return type "list[tuple[Literal['tile', 'wall'], str, str, int, int, int]]"
  "list[tuple[Literal['tile'], str, str, int, int, int] | tuple[Literal['wall'], str, str, int, int, int]]" is incompatible with "list[tuple[Literal['tile', 'wall'], str, str, int, int, int]]"
    Type parameter "_T@list" is invariant, but "tuple[Literal['tile'], str, str, int, int, int] | tuple[Literal['wall'], str, str, int, int, int]" is not the same as "tuple[Literal['tile', 'wall'], str, str, int, int, int]"
    Consider switching from "list" to "Sequence" which is covariant

[bright tangle]

I just factored out equivalent tuple members because I didn't feel it was important

[trim tangle]

If you have e.g. list[str | int], that's not the same as list[str] | list[int]. The first one can mean ["a", 42, 3], while the second one can't

[bright tangle]

Yes, but I'm making a union of two tuples, a single member of which differs, which should be equivalent to a tuple, a single member of which is a union

[trim tangle]

Hmm yeah

This seems like a bug, you should report it to the pyright repo

Also... maybe use dataclasses or NamedTuple if you want to differentiate between different cases. Plain tuples with literals will have a lot of these inference issues

[stiff acorn]

Is there any typing documentation that covers type hinting various dunder methods for classes?

I looked at both typing and mypy docs but couldn't find anything

Mypy did cover a few like init and slots but I'm interested in the rest like comparators, hash, etc

[oblique urchin]

there isn't much to say in general, each is different and mostly the types flow from the general meaning of the dunder

my autotyping tool can automatically add annotations to some, so that could be helpful: https://github.com/JelleZijlstra/autotyping?tab=readme-ov-file#usage

and you can look in typeshed for how these dunders are annotated on standard library classes

[stiff acorn]

Thank you, I'll take a look at both

[lunar dune]

The CPython data model docs aren't geared specifically towards typing users, but they often say things like "this dunder method should return a bytes object" etc. https://docs.python.org/3/reference/datamodel.html#special-method-names

[stiff acorn]

Thanks, figured some stuff but I still have some doubts

For example, I looked at a bunch of code and found slightly varying type hints for __eq__

    def __eq__(self, other: Any) -> bool:
        if not isinstance(other, Path):
            return NotImplemented
        
        return self.file.resolve() == other.file.resolve()

    def __eq__(self, other: object) -> bool:
        if isinstance(other, Path):
            return self.file.resolve() == other.file.resolve()
        return False

from what i can gather, returning NotImplemented is the "correct" way from data model docs

but the type hint only indicates bool?

mypy isn't complaining so i guess it's fine

[oblique urchin]

__eq__ is tricky. I think object is generally the safest annotation

the type system generally ignores NotImplemented (we pretend it's Any)

[stiff acorn]

That explains it, thank you

[lunar dune]

Yeah, for binary-operation dunders, we generally do the type hint "for the operator the dunder is implementing" rather than strictly trying to type hint the actual return type of the dunder itself. So whereas int.__lt__(42, "42") will strictly-speaking succeed at runtime (it just returns NotImplemented), 42 < "42" will not succeed. So instead of trying to faithfully represent in the stub that int.__lt__ sometimes returns NotImplemented, we simply don't allow int.__lt__(42, "42") as a possibility. Because, as Jelle says, type checkers don't really understand NotImplemented.

[weak oriole]

How do people validate schema on inputs/outputs. Ex: my function accepts a dataframe with columns A, B, C (and no others) and produces an output json structured as {A': str, B': list[str], C': "tz aware timestamp"}? (not sure if this should be in #type-hinting but it seems fairly close)

[warped lagoon]

is the concept of type constructors well-defined in python from the type system side?
I'm sorry that the higher-kinded types PEP was stale for a long while, but I'm revisiting it from time to time and can't give an exact answer to the question above

one of the examples, if you don't mind the syntax is python def magic_map[T, U, I: Iterable[T]](function: Callable[[T], U], iterable: I[T]) -> I[U]: return type(iterable)(map(function, iterable))

it feels like we need to work with __new__ rather than __init__?

i.e. some way to define/check that the constructor can be called with Iterable[T] and will return Self[T]

(iirc parametrizing Self was already discussed as a part of HKT)

the idea is still pretty raw to put it somewhere like python discussions

[restive rapids]

is it I: Iterable[T] and not just I: Iterable? you're parametrizing it later

[warped lagoon]

hm, that's fair yeah

but I'm moreso worried about backwards compatibility

specifically Self

[warped lagoon]

and this is the main cornerstone

[restive rapids]

im not sure if just type(thing)(...) is even a good idea if you bind stuff like Iterable - its not guaranteed that you can make it from another iterable, yeah. i was thinking of something more like Iterable + From[Iterable] maybe? but.. uh, thats an intersection type or something

[warped lagoon]

that is in fact an intersection type

though basedmypy already implemented generic typevars to some degree

[restive rapids]

hmh, maybe...
i never implemented type systems but in my mind it works like:

def f[A, B, I: Iterable, some_syntax_for_a_constraint(From[[Iterable[B]], I[B]])](I[A], (A -> B)) -> I[B]:
  ...
# (the From is like, detailed..ly? parametrized because its not guaranteed that the I constructor is generic over all iterables, but it could be just for B and should still work.. is that even a thing in python? i guess thats ad-hoc polymorphism, and typing.overload is kinda that?)

f(list[A], (A -> B))
  list: Iterable # I: Iterable, satisfies
  list(Iterable[B]) -> list[B] # satisfies From[[Iterable[B]], I[B]] constraint
# ->
list[B]

# maybe even...
f[A, B, C, I: Iterable, From[[Iterable[B]], I[C]]](I[A], (A -> B)) -> I[C] # ?? if I(Iterable[B]) is not I[B] but I[something else (C)], like, fully expressing the type(xs)(Iterable[B])
# in that case..
f(list[A], (A -> B))
  list: Iterable # ^
  list(Iterable[B]) -> list[B] # -> C = B , but could be not that
# ->
list[B]

# and what about cases when, let's say

class T[A]:
  ...

class U[A, B](T[A]):
  ...

def f[A, B, C: T](C[A], (A -> B)) -> C[B]:
  ...

f(U[A, V], (A -> B)) # well, U: T, U[A, V]: T[A], but it also has the V part, what to do with it? leave it the same? so
-> U[B, V] # ?

[muted iron]

Trying to type this beauty in the absence of PEP-661. Client code is fix.

class _MissingInt(int): pass
_MISSING_INT = _MissingInt()

class _MissingStr(str): pass
_MISSING_STR = _MissingStr()

class C:
    x = 0
    a = ""

    def __init__(self, x: int = _MISSING_INT, a: str = _MISSING_STR) -> None:
        if x is not _MISSING_INT: self.x = x
        if a is not _MISSING_STR: self.a = a

# client code
c = C(1, "a")
assert c.x == 1 and c.a == "a"

c.x.as_integer_ratio  # these need to pass
c.a.casefold

class D(C):
    x = 1

d = D(a="a")
assert d.x == 1 and d.a == "a"

How horrible is this? I'm out of ideas otherwise.

[undone saffron]

you can just do

_MISSING: Any = object()

class C:
    x = 0
    a = ""

    def __init__(self, x: int = _MISSING, a: str = _MISSING) -> None:
        if x is not _MISSING: self.x = x
        if a is not _MISSING: self.a = a

...

[tranquil turtle]

class C:
    x = 0
    a = ""

    def __init__(self, x: int = _MISSING_INT, a: str = _MISSING_STR) -> None:
        if x is not _MISSING_INT: self.x = x
        if a is not _MISSING_STR: self.a = a


# why not this? :
class C:
    def __init__(self, x: int = 0, a: str = "") -> None:
        self.x = x
        self.a = a

[undone saffron]

^ as well if that's reasonable, but I've seen situations where it isn't and examples given to avoid sharing confidential stuff end up leaving out why the "obvious" solution can't work.

[muted iron]

This seems to work and is so much better. I missed the obvious Any.

[muted iron]

I would've made it this way, but the default values are class attributes in the client code, and they will change with subclasses.

[rotund talon]

Hi, I wanna make that "hello world" program i saw on instagram, basically all the alphabets keep printing, until the word's letter matches, then that letter stays and next letter's column same alphabets keep printing till that letter matches
basically I want this out put for the word "ice":
a
b
c
d
e
f
g
h
i
ia
ib
ic
ica
icb
icc
icd
ice
ice

how to do this?

[trim tangle]

This doesn't seem related to type annotations, you should see #❓｜how-to-get-help and make a help post in #1035199133436354600

[rotund talon]

ah oke sry ;-;

[tranquil turtle]

interesting

x: bool
if x == 0: ...

[viscid spire]

How is that related to hello world??

[viscid spire]

Pywrong

[rotund talon]

https://discord.com/channels/267624335836053506/1248317299140591646

[restive rapids]

@trim tangle ello
we ready to cook?

[trim tangle]

perhaps

I am not completely awake yet

[restive rapids]

mh, its hard to discuss about seriously writing something, im not sure what to start with
who are we writing for (what knowledge of python we can assume of readers)? what should we cover? with what do we start? how to make examples that would not feel useless?

i think too much yapping at the start like i did in thing will likely make people quit because it starts with some useless for them at the moment stuff
but with what to start so it wont feel like it? how to even introduce typehints to people who havent used them before? maybe make an example of untyped code, and show that by using typehints the typechecker can help us figure out some errors?

[trim tangle]

Have you seen https://diataxis.fr/?

[restive rapids]

nope

[trim tangle]

I think this structure would make the most sense:

• a "central tutorial" that reads like a book chapter and gets you from 0 to basic type checking, including:
  • a brief explanation of type annotations
  • setting up your editor and/or running a CLI type checker
  • annotating function parameters and return types
• a bunch of self-contained how-to guides for specific topics, aimed at people who already understand the basics (e.g.: how to type a function decorator)
• branching tutorials/explainers on more complex topics (like the generics section I already have)

[restive rapids]

hmh
what about like, builtin types? where would that go? people are obviously familiar with int and str or whatever, but lets say we want to introduce list or dict, and they have generic params
do we explain generics, or at the start just say "list[T] means list of T's" or something? or would that only be in the "generics" section?

[trim tangle]

Basic stuff like list and dict will probably be in the initial tutorial

Actually maybe it's better off like this:

• have a main "tutorial"
• have subsequent "tutorials" on broader topics, like generics or protocols or whatever
  because some topics are not that self-contained and might benefit from an overarching narrative structure

[trim tangle]

Because this is very basic material, we can just say "list[int] means a list of integers <insert some examples>" and not go into more detail

[restive rapids]

i think thats kinda what i tried to do in https://lambda-abstraction.github.io/lambda-abstraction/python-typehints/builtin-types/ and it seemed like a good idea

[trim tangle]

I think the mainland tutorial should cover:

• using classes as types (like str and int and YourOwnClass)
• unions (especially X | None)
• basic built-in generics (list, dict, set)

[restive rapids]

yeah that seems good so far
regarding uh, being a "modern" (guide/tutorial), would we use python3.12 syntax for typevars? should we then still explain typing.TypeVar? a lot of stuff still uses those

[trim tangle]

I think we should ignore the 3.12 syntax completely for now

[restive rapids]

hmh, fair i guess, although i really like it, keeps the scope of typevars clear, specially nice for type aliases with the type keyword

[trim tangle]

Maybe we'll figure it out by the time the decision is needed

maybe we'll give up after int

[restive rapids]

"int means int, but, well, bool is also an int, here is liskov substituion principle, uh, inheritance, yes, thanks, we love subtypes"

[trim tangle]

Don't forget about int being a subtype of float

[restive rapids]

even though issubclass(int, float) is false

mh, alright, do you have a repo or something? how would we handle the uh, collaboration aspect? does maybe using something like liveshare make sense in the case of just writing "articles"?
would we probally need some consistent style guide? perhaps about that we should worry later

[trim tangle]

I think I'm going to post it on https://decorator-factory.github.io/typing-tips/

I think I'll try making a very brief draft of the mainland tutorial

[restive rapids]

we could both make a couple drafts and maybe merge them together incase some information missing

[trim tangle]

yes

[restive rapids]

alrighty ill try to cook something

[tranquil turtle]

what are you doing? writing a book about typing?

[restive rapids]

sort-of? i think its more like a tutorial mostly intended for people completely new to typehints

[tranquil turtle]

imo, mypy's cheatsheet is perfect, even for newbies

[restive rapids]

but it doesnt explain why use them & what do they mean, and its well, a cheatsheet, not really structured - just a bunch of simple examples and thats it

[trim tangle]

Yeah, it's a cheat sheet, not an introduction

[restive rapids]

thing
i started writing something i guess, im not sure of the order of things here though, and specially the "wording" of stuff, im not really.. a writer

[trim tangle]

https://decorator-factory.github.io/typing-tips/start-here/

[restive rapids]

haha i love how we both included the "other developers include yourself"

hmh, using screenshots is nice

[trim tangle]

If you're in VSCode, you can Ctrl+V an image into a markdown file and it will just work

[restive rapids]

hmmh, the re.Pattern example is a bit weird for the first introduction to me
although i dont know what first example would be good either

[trim tangle]

yeah probably

Maybe

def find_match(pattern: str, strings: list[str]) -> tuple[int, str] | None:

and then do something like find_match(b"[0-9]", ["100", "hmm"])

[restive rapids]

i think something simple as

def double(number: int) -> int:
    return number + number

print(double("42"))

might be a bit easier? with this one you're introducing generics & unions right away, sort of a big jump from zero
the

Error checking

Type hints don't do anything at runtime: you're free to call find_match(42, socket.socket()) and get a nasty error like TypeError: 'socket' object is not iterable.

still applies good here, just in the other way - you can typehint something too strict / close minded, that passing a valid thing wont work

[restive rapids]

with the last point you could specify then that you'd want to typehint arguments as generic as possible, while keeping the return type as specific as possible
thats a good rule to go by when typing functions imo

[trim tangle]

Maybe something with two arguments, like ```py
def censor(string: str, stars: int) -> str:
return string.replace("java", "*" * stars)

I also want to reuse this example for autocompletion

[restive rapids]

LMAO
ahhaah

this is good

[trim tangle]

Well, this is kinda mean to Java

[restive rapids]

i think its funny, and articles being funny makes them like, more fun to read to readers

[trim tangle]

Well, maybe it's not that funny if you love Java

[trim tangle]

I think I'll go this route

I like this example because it showcases a bunch of things at once. There are more basic examples later

[restive rapids]

hmh now when i think of it
perhaps it is fine? the "target audience" knows that a list can store different things, and that a function can return different types, so thats basically just showing most of the syntax they'll need in one go to express those things

[trim tangle]

Yeah the re.Pattern[str] part is kinda stinky

[restive rapids]

hmh, i think those changes + a small explanation of subtypes would make a good introduction

[trim tangle]

What about subtypes?

[restive rapids]

class Dog:
    ...

def create_dog(height: int) -> Dog:
    dog = Dog()
    dog.grow(height)
    return dog

something like this example, but with the animal being a parameter, and a common Animal parent could work, so people would understand they dont need Cat | Dog, just Animal

[trim tangle]

I think that's good material for the next chapter

[restive rapids]

hmh
the typing docs already use 3.12 syntax

[trim tangle]

type var defaults are still not in any released python, so if you do use pep695 syntax in your tutorial, you'll need to switch a lot of gears when explaining defaults

[restive rapids]

fair

[trim tangle]

(inheritance is not used that often)

[restive rapids]

maybe an int | float example? common too

[trim tangle]

Remember, this is supposed to be the first thing someone reads about type hints

int | float is a very particular... quirk

Btw, I pushed https://decorator-factory.github.io/typing-tips/start-here

[restive rapids]

yeah but people use numbers frequently dont they? like int(input()) and float(input()), would make them think they need int | float, maybe a !!! note would be good

[trim tangle]

Using int | float isn't actively wrong, I think. It will work

It might be good to mention this in one of the sequels, as i said

[tranquil ledge]

There’s two giant recent threads on float | int in the Python discourse forum under the typing category, if y’all want reference for that.

[trim tangle]

There's more stuff to cover, like annotating stuff in classes, dataclasses, Callable, other abstract classes from collections.abc and why list[str] is usually a bad parameter annotation...

[grave fjord]

Love to see mypy complain about redundant unions

[trim tangle]

python/mypy#17223

[mighty linden]

GitHub

[mypy] #17223 int inconsistently considered a subtype of float

[tranquil ledge]

It’s a bit of a weird one, where float isn’t technically a super type of int, but more like “syntactic sugar” for float | int.

At least, from a type checker’s perspective.

[grave fjord]

They just just make the runtime catch up with the type system here IMHO

[tranquil ledge]

How do you mean?

[restive rapids]

was the first just supposed to be a should? would make sense then probally

[trim tangle]

Btw, this would highly benefit from a review by a native English speaker. Because the writing is shit

[tranquil ledge]

If you mean that float and int should be completely compatible by having the same attributes and methods, I don’t disagree.

The status quo is more confusing than it already is otherwise.

[restive rapids]

||are we both russians writing an article in english||

[trim tangle]

Ah yes, int.is_integer 💀

Still no int.hex though

[tranquil ledge]

Currently, hex and fromhex are missing, iirc?

Yeah, that.

And a type checker won’t warn you of that in many cases.

[trim tangle]

I don't know what's worse honestly, int.hex missing or (42).hex() returning 0x1.5000000000000p+5

[tranquil ledge]

Wouldn’t it make more sense to say type-hints are informal documentation?

As opposed to formal?

At least for me, I imagine formal being a docstring following a specific style guide.

[restive rapids]

typehints follow a certain style too dont they

[trim tangle]

"informal documentation" is something like "call this function with a string"

it's formal because type hints have precisely defined meaning, as opposed to arbitrary English words

[tranquil ledge]

That is sorta what a str type hint on a single-parameter function would mean though, but fair enough.

[trim tangle]

A docstring can also contain formal documentation, like

:type foo: str
:type bar: str

(or whatever format you prefer)

[tranquil ledge]

This might be a personal opinion thing for me, where if there isn’t any prose in the documentation, it feels informal. Regardless, that was my attempt at a minor nitpick. I’ll read the rest

[restive rapids]

ill try to write something on collections.abc & being generic in parameters & strict in returns

should abcs and protocols be in one page? maybe next to each other? they kinda deal with the same problem (when typing, atleast)

[tranquil ledge]

Just curious, would you guys consider contributing this guide to the official typing page, or are you set on it being entirely independent? Feels like it would be a good fit for the first entry in the “Guides” section on the front page.

[trim tangle]

I think we should separate the "teaching the basics and the fundamental concepts" stuff from the "explaining individual items and their quirks" stuff

E.g. we could make an article that explains how and why you can define "interfaces" to accept types you don't directly know about. It could use Protocol, probably.

And then we could have a separate article in the style of object vs any to explain ABC vs Protocol

[restive rapids]

but you did say "other stuff to cover: abstract classes from collections.abc"
is this an idea you had just now

[trim tangle]

I meant using Sequence, Iterable, Collection instead of list

[restive rapids]

yeah

[trim tangle]

well, we don't need to explain ABCs to show how to use Iterable

[restive rapids]

fair

[trim tangle]

And these ABCs actually work like protocols because... because

i.e. you don't need to inherit from Iterable to make your class iterable to a type checker

[trim tangle]

Yes, it could be... annexed into that site. Somewhere in the future. Or maybe it's going to be a standalone thing, just like "automate the boring stuff" isn't part of docs.python.org

||also I hate ReST 😩 ||

[tranquil ledge]

I feel like if you first explain Protocols and how that correlates with Python’s duck typing, explaining collections.abc becomes a lot easier.

Just an idea.

[tranquil ledge]

Fair enough.

Mainly suggested it because I really liked the idea in the moment of a first-party beginner-friendly tutorial for annotations.

[trim tangle]

@rustic gull Don't ask for help with that in this server

[restive rapids]

duck typing is what i have right now
im not sure how to name the headers

[trim tangle]

||maybe we should start a patreon||
||to buy you a shift key ||

[restive rapids]

capital letters look bad

[trim tangle]

I will think about it tomorrow

[restive rapids]

whats your timezone?

[trim tangle]

horizontally or vertically

My sleep schedule is so crooked that it doesn't really matter, maybe in 14-20 hours

[foggy shard]

why isn't something like this supported by the callable type 😦

def add_partial_params(
    functions: list[
    Callable[ [ "GPTChatbot", "Request_data", "IndexSearchSharedState", ..., ] str, ] ,
    gpt_chat_bot: "GPTChatbot",
    request_data: "Request_data",
    shared_state: "IndexSearchSharedState",
):
    return [partial(func, gpt_chat_bot, request_data, shared_state) for func in functions]

i want all function params to have the first 3 params and other remaning parms dont matter

[restive rapids]

, Unpack[tuple[Any, ...]] maybe?
(Also functions don't need to be a list in your example, just iterable)

[tranquil ledge]

typing.Concatenate might be helpful here as well.

[trim tangle]

@restive rapids I added a short section on type inference and added the page to the index

[restive rapids]

i think you should move the empty list to it

Whenever you have an empty collection assigned to a variable, you need to give it an annotation:

names: list[str] = []

this one

[trim tangle]

hmmm

I'm on the fence

[restive rapids]

and maybe should make hyperlinks for all things like ``x``? or thats for later

[trim tangle]

wdym

[restive rapids]

For example, json.loads() and pickle.loads() both return Any.
something like that
maybe there's a mkdocs thingy for that instead of manually writing hyperlinks

[trim tangle]

that's possible, yes

I could just link it manually

[restive rapids]

could add Literal and Enum to Frequently asked questions as a how to annotate that a value is one of a certain set of values or something like that (words are hard)
and Final for how to annotate a "constant" maybe

and maybe go over older articles and switch to new style annotations? atleast for unions
i guess i could do that and pr

hmh, while going over older stuff

@dataclass
class _RawMovie:
    name: str
    director: str
    year: int
    genres: Optional[list[str]] = None
    genre: Optional[str] = None


@dataclass
class _RawMovies:
    movies: list[_RawMovies]

this is probally a typo, right? should be list[_RawMovie] i think?

[trim tangle]

yes

[restive rapids]

alright, ill fix that in my pr i guess
will be my first not readme.md pr lmao

from typing import Union

AnyString = TypeVar("AnyString", bound=Union[str, bytes])

def triple(string: AnyString) -> AnyString:
    return string * 3

unicode_scream = triple("A") + "!"
bytes_scream = triple(b"A") + b"!"

this shouldnt be a bound union, should it? should be a constraint or how its called

TypeVar("AnyString", str, bytes)

i wonder if typechecking could be ran over code in markdown

[trim tangle]

Yeah this is just completely wrong

[restive rapids]

https://github.com/decorator-factory/typing-tips/pull/6 i dont know how pull requests work
but i think this works (besides the place where Union is mentioned outside of code, that was probally a bad idea to change it, the Sometimes inheritance works better, sometimes you're better off with a (`A | B`) (`Union`). one.. yeah, thats aids)

might want to iterate over that too, change noreturn to never, and.. maybe, maybe change optional to | None, but i myself am not sure what to use, optional kinda reads better, but is sometimes confused that it means something magical and not just | None, which is more explicit

[restive rapids]

probally too early to accept the pr, i'll go change that and use Never instead of NoReturn
oss is hard

hmh, well, the NoReturn doc in tips uses screenshots and the theming will be different if i screenshot from my vscode.. urgh, sucks to suck, ill guess ill only change some of them, or just not care about it for now. thats probally better
ill go over other stuff for now and maybe fix it too

[trim tangle]

Maybe I should publish my theme 🙂

[restive rapids]

im good with my one

[restive rapids]

ok i think besides that union change its fine

if this one gets uh. merged? idk, would i delete it? if i want to commit later would i make another branch? i never really committed

[trim tangle]

I left a small comment

When I merge the changes you can "sync" your fork and then make a new branch

or you can just commit to the master of your branch and make a PR from there

[restive rapids]

did you? i cant find maybe im blind

[trim tangle]

oh I forgot to press the button

stupid github

[restive rapids]

maybe replace optional everywhere? and maybe even delete the Union[T, None] == Optional[T] part then
and maybe even delete the "Optional" is not optional thing then whatsoever

[trim tangle]

Nah the optional article is still kinda relevant

because people will encounter Optional in code from time to time

[restive rapids]

hmh

well this is a lesson to not do changes and remove files in one commit

[trim tangle]

yeah, you should usually not put unrelated changes in a single PR

[trim tangle]

and since people will use Optional less often they're more likely to be confused about it

[restive rapids]

how could i "fix" that now? would i remove that commit or? is that even a thing?

i've heard somewhere that just making another commit fixing the old one is better
is that preferred

[trim tangle]

git revert creates a commit that does the opposite of another commit

You can use git revert --no-commit <commit-hash> to "partially revert" the commit

[restive rapids]

so i'd still need a new commit

i guess just fixing this one by adding the deleted thing back would be better

[trim tangle]

Yes, it's usually better to add a new commit

[restive rapids]

||or well i could delete the fork and make a new one to not embarass myself with this one 💀||

[trim tangle]

however, if it's just you working on some git history, you can do various history changes

with git rebase -i and other arcane things, and then do git push --force-with-lease

[restive rapids]

well in the end that doesnt show up in the diff so i think thats good

[trim tangle]

actually let's leave the bound part alone, the examples need to be reworked

but I'll merge it

[restive rapids]

regarding this
we cook?

if i write that myself it will probally look very different from other stuff which is not nice
how to solve that

[trim tangle]

I think we should first make some kind of plan, or something like that

[restive rapids]

well a good first example (for Literal / Enum) is open()
it does accept a string mode but not all strings, only some specific ones

[trim tangle]

I mean a more high-level plan, for the whole website

[restive rapids]

ah

is that even a thing for an article collection

[trim tangle]

Well, I'd hope that the stuff under the "Start here" section will serve as a thing you can read in order

[restive rapids]

hmh

perhaps there will need to be a lot of
moving around of current stuff then
and some stuff cant really be in order, you dont need everything right away, some is fairly specific - like the anti patterns one

[trim tangle]

I have an idea

let's not hog this channel too much lol

[merry cradle]

i am constantly getting an error in a function from mypy, peelpreter/evaluator/mbuiltins.py:80: error: Incompatible types in assignment (expression has type "str", variable has type "list[Object]") [assignment],
the function is this,

def m_tail(fname: str, args: list[obj.Object]) -> obj.Object:
    if len(args) != 1:
        return arg_error(fname, 1, len(args), "tail")
    if args[0].type == obj.OBJ_ARRAY:
        assert isinstance(args[0], obj.Array)
        sequence = args[0].elements
        _, *tail = sequence
        return obj.Array(tail)
    elif args[0].type() == obj.OBJ_STRING:
        assert isinstance(args[0], obj.String)
        string = args[0].value
        tail = string[1:] #This is the line mypy is going crazy
        return obj.String(tail)
    else:
        return obj.Error(error.UnsupportedType(fname, args[0], "tail", (-1, -1)))

tail is literally undefined, i dont get how its getting that tail is list[Object]

[trim tangle]

mypy doesn't allow variables with different types to have the same name in the same scope

that's just kinda how mypy works

[merry cradle]

even if they are seperated by if and elif?

[trim tangle]

yep

[merry cradle]

like they are never gonna conflict since only one can be executed at a time

[trim tangle]

yep

def foo() -> bool:
    raise NotImplementedError
    
def bar() -> None:
    if foo():
        hmm = 1
        print(hmm + 2)
    else:
        hmm = "yes"
        #^ error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
        print(hmm + "!")
        #^ error: Unsupported operand types for + ("int" and "str")  [operator]

It's mildly annoying and it's not likely to be changed, but you could open an issue/look for existing issues

[merry cradle]

oh well

a user suggested me to type hint the var on the second mention/elif, gonna try that now

[trim tangle]

mypy will complain about that since you're "redefining" hmm

I think the simplest workaround is to rename the second variable, like rest or tail2 ||💀||

[merry cradle]

yeah

[trim tangle]

in your case I'd just do return obj.String(args[0].value[1:])

[muted iron]

pylance wouldn't complain about this ^ right?

[lyric hull]

<?php

/**
 * @template TX of FooInterface
 * @template TY of FooInterface
 * @phpstan-param ?TX $x
 * @phpstan-param ?TY $y
 * @phpstan-return ?(TX|TY)
 */
function fooInterfaceSwitched(
    ?FooInterface $x, ?FooInterface $y, bool $switch
): ?FooInterface
{
    return $switch ? $x : $y;
}

function fooConcreteSwitched(
    ?FooConcrete $x, ?FooConcrete $y, bool $switch
): ?FooConcrete
{
    return fooInterfaceSwitched($x, $y, $switch);
}

in php using phpstan, this (specifically, the return type of fooConcreteSwitched v.s. fooInterfaceSwitched) typechecks because phpstan can infer that, inside fooConcreteSwitched, fooInterfaceSwitched's return type can be narrowed from FooInterface to the specific FooConcrete, while of course both the code and phpstan would also support different combinations of interface implementers in other functions/uses.

i'm pretty familiar with some (casual) aspects of python's typing, but i'm not sure if something like this would typecheck correctly or what annotations would be needed. is this supported in python?

[trim tangle]

def foo_switched[T: Foo](x: T, y: T, switch: bool) -> T:
    return x if switch else y

This would be more similar to the PHP equivalent:

def foo_switched[TX: Foo, TY: Foo](x: TX, y: TY, switch: bool) -> TX | TY:
    return x if switch else y
``` but is not necessary, since if you call it with e.g. `FooOne` and `FooTwo`, `T` will be inferred as `FooOne | FooTwo` (maybe that's also the case in PHP?)

Before Python 3.12 you'd do ```py
F = TypeVar("TFoo", bound=Foo)

def foo_switched(x: F, y: F, switch: bool) -> F:
return x if switch else y

[lyric hull]

yeah this is pretty close to fooInterfaceSwitched, but i'm wondering if python's typing or generics features allow for stronger narrowing. my guess is "no", i know TypeGuard is pretty lame; i feel like maybe pyright would support this but as a bonus feature rather than anything in python's annotations system

[trim tangle]

what do you mean by stronger narrowing?

this would work exactly like your example

[lyric hull]

in the php code above, phpstan can verify fooConcreteSwitched's return type is correct even though it technically returns FooInterface because it knows the types of x and y and can thus narrow fooInterfaceSwitched from FooInterface to FooConcrete

[lyric hull]

oh, ok. neat!

[trim tangle]

pyright playground

[lyric hull]

indeed, this works too, cool. i'll also try the mypy playground

[trim tangle]

the docstring syntax in addition to PHP annotations kinda terrifies me...

mypy also used type annotations from comments in 2.x

[oblique urchin]

still works in 3.x

[trim tangle]

Yeah

[lyric hull]

i prefer the comments style, actually, but might be due to limitations of my knowledge. i recall doing something weird where mypy could understand a classmethod[...] annotation from a comment, and had such a construct internally, but nothing like that exists in typing

the "reflection" you get from having the types in-interpreter is cool, but it also means we've been playing typing musical chairs since at least 3.6 as what/where the typing bits are defined has changed

[tranquil ledge]

classmethod is generic in typeshed, so type-checkers will understand a parameterized version of it as a type annotation or type comment, AFAIK.

One downside of the annotation is that at runtime, parameterizing classmethod like that is an error, so the annotation can only get away with it via quoting, conditional alias assignment, or something like that, whereas comments don't have to be run at runtime and can get away with whatever.

[undone saffron]

which is really something that should be fixed in all supported python versions before type comments become unsupported since that's still a goal

[tranquil ledge]

If the docs are any indication, it's not a difficult PR based on implementation (https://docs.python.org/3/c-api/typehints.html#c.Py_GenericAlias).

Haven't looked at the internals of classmethod/staticmethod/whatever else is builtin and only generic in stubs currently, so maybe it's not that simple.

[merry cradle]

maybe, but i dont use vscode so....

[jade viper]

Why is the type DateT = TypeVar('DateT', bound=Union[datetime, date]) not being recognized by my linter?:(

[trim tangle]

show the code

[jade viper]

@staticmethod
def first_day_of_next_month(date: DateT) -> DateT:
    return date.replace(day=1) + relativedelta(months=1)

[trim tangle]

What do you mean by not being recognized?

[tranquil turtle]

typevar is not a typealias

[jade viper]

I mean, for instance, this Django method uses a T but when my linter "looks" at it, it shows the type that is being passed as T

Don't know if I've made myself clear

[trim tangle]

where's the typevar here?

I don't understand

[jade viper]

From the Django stub:

It declares it as _T = TypeVar("_T", bound=models.Model)

And then uses it as Generic[_T]

The Position class it the specific T

[trim tangle]

I'm assuming the class where the first_day_of_next_month is is not generic

[jade viper]

Let me try to explain better

DateT is either datetime or date

Here I'm calling a method with a date, is there a way so instead of DateT@... it just shows datetime.date or something like that?

[trim tangle]

if you do x = TemporalAdjuster.first_day_of_month(dt), what does hovering on x show?

[tranquil turtle]

you can make an overload
pyright lsp will show the appropriate overload in this context

[jade viper]

I tried that, that shows the correct type

[tranquil turtle]

otherwise it shows the only available signature of the function

[trim tangle]

yep

In your Django example you're hovering over a method that's already specialized

[median tangle]

is it possible to override the types in a class depending on the value of its generic parameter?

i.e. something like:

@override
class Foo[True]:
  x: int
  y: int

@override
class Foo[False]:
  x: str
  y: str

class Foo[T: bool]:
  x: int | str
  y: int | str

  ...

[trim tangle]

You can do that with methods ```py
class Foo[T: (Literal[True], Literal[False])]:
@overload
def x(self: "Foo[Literal[True]]") -> int: ...
@overload
def y(self: "Foo[Literal[False]]") -> str: ...

why not```py
class Foo[T: str | int]:
x: T

[median tangle]

because I have a bool flag that affects which type several attributes are

and I don't really feel like making 3 generic params

since the class can only be in one of the 2 states

I know it might be better to just do 2 classes here, but that doesn't really play well with the codebase I have, since the classes represent packets, and the system I work with need to have a single class for the packet

right now I just have them as unions, but it needs a bunch of casts when working with the values

which is annoying

like: ```py
if self.my_flag:
do_stuff(cast(int, self.x), cast(int, self.y))

the internal state does guarantee that if this flag is set, the values will be of those types, I just want to avoid the casts here

[trim tangle]

If this flag can change for a single object during runtime, then no, you cannot use a generic parameter in this way

[median tangle]

python technically allows it, but it won't change in the way it's used

[trim tangle]

in that case you can use methods

[median tangle]

so like, store the received data in unions and retrieve them with overloaded getters?

[trim tangle]

Well, I don't know how you're getting the int or the str

[median tangle]

it's received over a socket from a server, this class just has the logic to read that data and represents it in python

that bool flag affects the reading logic

it's received first

[median tangle]

this will probably be good enough

[jade viper]

Can I overload functions by parameter type alone?

[soft matrix]

Yes

[rare scarab]

you may be able to use a union type instead to make things simpler

[plain dock]

is there an easy way to differentiate between theoretically-infinite-length containers like dict[str, int], list[int], or tuple[int, ...] as opposed to fixed-size ones like tuple[int]? perhaps some kinda of Appendable protocol?

[tranquil turtle]

tuple[int, ...] is probably not Appendable

i dont think there is a way at all

[plain dock]

fair point. once it's made, it's not appendable. i meant during creation, where you can modify how many elements it initially has at will

in other words, t: tuple[int, ...] allows initial definitions of both t = (1, 2, 3) and t = (1, 2), etc

[undone saffron]

No, tuple is unique in the type system here, and only really by neccessity

I don't think we'd have tuple[int, ...] if there was a different builtin data type which was an immutable sequence for homogenous data (because people wouldnt use tuple for that then)

[dull lance]

maybe you can use tagged unions to help with type narrowing

[hallow pollen]

I have a function for which I've decided that dict and list will serve as sentinel values, and you can't convince me not to do this. Is there a better way to type hint this than Literal[dict, list]?

[trim tangle]

Literal[dict, list] is not valid

You can use type[dict] | type[list], though that will also match any subclass of dict or list

[oblique urchin]

the closest you can do is type[dict] | type[list]

[hallow pollen]

it has to be exactly dict or list, so I'll have to stick to the invalid approach.

[trim tangle]

You can stick to that approach, but that's just an invalid annotation. Type checkers will simply not understand it

It's like annotating something as "oo oo aa aa"

or with "use list or dict exactly" (which is also invalid but clearer)

[hallow pollen]

I guess I could do type[dict] | type[list] and then change if value is dict (and etc with list) to if issubclass(value, dict)

[trim tangle]

||or you could stop using dict as a sentinel 💀 ||

[plain dock]

new pep idea tbh

[hallow pollen]

the alternative is creating an enum, and I don't want to introduce another thing that has to be imported.

[trim tangle]

is there more context to the code?

[hallow pollen]

def traverse(data, keys: Sequence[int | str | Literal[dict, list]], raise_exception: bool = False, default: Any = None, allowed_exceptions: tuple[Exception, ...] = ALLOWED_EXCEPTIONS):
    """
    For traversing JSON-like structures.
    Equivalent to data[a][b][c][d], etc. where each a, b, c, d in `keys`. if not `raise_exception`, a missing key
    (or index) will return `default`.
    """
    keys_iter = iter(keys)

    for k in keys_iter:
        if k is list:
            keys_tuple = tuple(keys_iter)
            return [
                traverse(elem, keys_tuple, raise_exception, default)
                for elem in data
            ]
        elif k is dict:
            keys_tuple = tuple(keys_iter)
            return {
                k: traverse(v, keys_tuple, raise_exception, default)
                for k, v in data.items()
            }

        try:
            data = data[k]
        except allowed_exceptions as e:
            if raise_exception:
                raise e
            return default

    return data

I haven't updated the docstring.

[trim tangle]

What does dict or list mean in the first argument?

And do you need to keep this for backwards compatibility?

wait, I didn't read correctly

[trim tangle]

And why do you not want to import enum?

[hallow pollen]

I mean that I don't want users of the code (me) to have to import or use an enum for this

[trim tangle]

They already import your function

[hallow pollen]

idc

[trim tangle]

Well, then you can use type[dict] | type[list] if you really hate importing things from the module you've already imported 🙂

Also tuple[Exception, ...] should be tuple[type[Exception], ...]

[trim tangle]

Everyone (and @restive rapids), I made a new beta chapter for the type hints tutorial:
https://decorator-factory.github.io/typing-tips/main-tutorial/1-working-with-classes/
Any thoughts/suggestions/insults?

a bit of a concoction of various topics, but it is what it is

also very sad 😿

Also, does anyone know a decent way to track the view counts and how people follow the links?

[restive rapids]

the repl examples missing syntax highlighting
frozen prevents you from mutating the object after creation. maybe should be clarified? that like, its not that you cant "mutate" it, its that you cant reassign the attrs (but if its e.g. a list you can append to it)

[trim tangle]

Something that isn't... google analytics or other creepy-ish thing.

[trim tangle]

I don't think there's a language specifier for repls

[restive rapids]

is python-repl not a thing?

>>> print("test")

in discord that works

[trim tangle]

not in mkdocs-material

I think that's okay, the default python repl doesn't have highlighting either

[restive rapids]

maybe just use py? should work besides the >>> ... stuff

although seems like its not ready for uh, class reprs

[trim tangle]

yeah

[restive rapids]

hmh, maybe should also leave a small note on what the _something convention means

[trim tangle]

Is there a good source that explains it?

[restive rapids]

https://docs.python.org/3/tutorial/classes.html#private-variables maybe?

[lunar dune]

You want "pycon" (it stands for "Python console" in this context)

[restive rapids]

overall i like it, simple, shows what problems you can have and how to solve them the practical way, and everyone loves kittens so the examples will surely make people not leave them cold and hungry

[trim tangle]

Not sure I like it over the monochrome

[restive rapids]

although regarding the "internal" __annotations__ attribute you should probally leave a note that those shouldnt be used directly for introspection and reference inspect.signature and typing.get_type_hints

[trim tangle]

The intended audience for this tutorial will probably not be using annotation metadata. This is more of a fun fact, and the rationale behind this being an error

though a footnote might be good

[tranquil ledge]

Could go the extra mile and link to the official best practices guide for accessing __annotations__ in the footnote as optional further reading: https://docs.python.org/3/howto/annotations.html.

[tacit sparrow]

Is there any reason for keeping :type x: and :rtype: in a function sphinx doc-string if there is already typing for the arguments and the return value?
It seems it's just duplicating the same information.

def add(a: int, b: int) -> int:
    """
    Add two integers together.

    :param a: The first integer to add.
    :type a: int
    :param b: The second integer to add.
    :type b: int
    :return: The sum of the two integers.
    :rtype: int
    """
    return a + b

[trim tangle]

there's no reason, no

[viscid spire]

I think it will in next update? or maybe that's just REPL in idle shell

[trim tangle]

yeah I've seen something about the new repl

something to the effect of it being based

[viscid spire]

I mean, if you read it, it must be based

[trim tangle]

??

[viscid spire]

(based ppl only read based things)

[trim tangle]

I read all sorts of crap

[viscid spire]

but it's based-crap by proxy

[trim tangle]

I will not judge you if you are into crap

[trim tangle]

https://docs.python.org/3/library/inspect.html#inspect.get_annotations

Always, always, always returns a freshly created dict.
damn they really drove the point home

!e

print((lambda:0).__code__.__doc__)

[rough sluice]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

Create a code object.  Not for the faint of heart.

[trim tangle]

Help on code object:

class code(object)
 |  code(argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize, flags, codestring, constants, names, varnames, filename, name, qualname, firstlineno, linetable, exceptiontable, freevars=(), cellvars=(), /)
 |
 |  Create a code object.  Not for the faint of heart.

[tranquil ledge]

That's a fun warning, lol.

[lunar dune]

The REPL has colour on py313 (the >>> is lilac). But it doesn't have syntax highlighting like rich or mkdocs.

Tracebacks (inside and outside the REPL) have syntax highlighting everywhere on py313, however!

[viscid spire]

I c

[lunar dune]

And because it's not a REPL feature per se, you get colourised tracebacks in the REPL even if you pass PYTHON_BASIC_REPL=1 as an environment variable to get the old py312 REPL back.

Though I think there's another environment variable that disables all colour, everywhere. If you really like black and white.

[normal locust]

class A:
    def b(self, data: dict) -> dict:
        ...

How should I typehint A.b as Callable?
Callable[(A, dict), dict]?

[tranquil turtle]

Callable[[A, dict], dict]

[trim tangle]

why though

[normal locust]

well, I want to store a method of a class, without initialize the class

dict[str, list[tuple[Route, Callable[[Route, type_Data], type_Data]]]]

[viscid spire]

you don't need to typehint it or I don't understand

[viscid spire]

type_Data 🤔

[viscid spire]

that list with 2-tuples is sus. why not make a Route hashable and put in a dict?

[normal locust]

Because a route can point to multiple string

And a string can point to multiple route

[normal locust]

A typealias of a stricter dictionary

[viscid spire]

a TypedDict?

why did you name it like that

[normal locust]

Well technically, it just a typehint as a json compatible dictionary

Without any conversion

[viscid spire]

ok, should be PascalCase

[cinder bone]

I love Concatenate

from collections.abc import Callable
from typing import Concatenate

def foo(x: list[float], y, z):
    print(x)

MY_ALIAS = Callable[Concatenate[list[float], ...], None]
def call(f: MY_ALIAS):
    pass
 
call(foo)

Is this an idiomatic way to express this though? Or does it do something weird (I want any argument after the first to be of type Any)

[soft matrix]

Why do you want it to take kw/var args here? You know the types for sure

[rare scarab]

^ you probably should do Callable[Concatenate[list[float], P], None] instead.

You should also add ```py
def call(f: MY_ALIAS, *args: P.args, **kwargs: P.kwargs) -> None: ...

[jade viper]

What is the best way to type hint "you can pass either a singular int or a sequence of int, and you will get back either a float or a sequence or float depending on the multiplicity of the input"?

Using TypeVars?

[oblique urchin]

use overloads

[jade viper]

How so?

[frigid jolt]

import typing as t

@t.overload
def uwu(x: int) -> float:
    ...

@t.overload
def uwu(x: t.Sequence[int]) -> t.Sequence[float]:
    ...

def uwu(x: t.Union[int, t.Sequence[int]]) -> t.Union[float, t.Sequence[float]]:
    # actual implementation here

[trim tangle]

Or you can choose to only accept a sequence, and have the user call uwu([42]) instead of uwu(42)

[trim tangle]

!pep 747

[rough sluice]

**PEP 747 - TypeExpr: Type Hint for a Type Expression**

Status

Draft

Python-Version

3.14

Created

27-May-2024

Type

Standards Track

[mortal cipher]

hey all! is there a way for me to turn this typescript expression into its python equivalent?

enum Foo {
    A,
    B,
};

type Bar = { a: string; b: number; } & (
    | { tag: Foo.A, value: { a: number, b: number } }
    | { tag: Foo.B, value: {} }
    | { tag: Exclude<string, keyof Foo>, value: { key: string, value: string }[] }
);

in essence Bar is a tagged union with some common parameters. there should be a few well-known types with type-safe representations of value, and a fallback with a generic value type

[trim tangle]

from dataclasses import dataclass

@dataclass(frozen=True)
class Common:
    a: str
    b: int

@dataclass(frozen=True)
class CaseA(Common):
    value: Something

@dataclass(frozen=True)
class CaseB(Common):
    value: SomethingElse

@dataclass(frozen=True)
class CaseC(Common):
    value: None = None

[mortal cipher]

hmm, but then i would have to manually cast into the given type, right? the tag is present as a value in the data i receive from an external service

[trim tangle]

You can use TypedDict if you want to trust the data with no validation/parsing

But I'm not sure TypedDict will work with the same quality of type inference as records in TypeScript

[mortal cipher]

i suppose i'd still want to do validation, but i'd need to write custom deserialization to turn it into those classes, right?

[trim tangle]

You can use packages like pydantic, cattrs, adaptix to turn JSON data into "typed" classes. They're similar to zod in TypeScript

[mortal cipher]

i see, do you know if they have a feature to turn my tagged data into those classes automatically? that was my main question

[trim tangle]

Not sure, you'll have to experiment with https://adaptix.readthedocs.io/en/latest/loading-and-dumping/specific-types-behavior.html#union

[mortal cipher]

thank you :)

[fervent oracle]

Isn't this only type checkable if Iterable requires a certain signature for __init__?

[rare scarab]

def is_iterable(obj) -> TypeGuard[Iterable[Any]]:
  try:
    iter(obj)
  except TypeError:
    return False
  else:
    return True

[echo knot]

Let's say I have a TypedDict called MyDict with key1 and key2. Is there a way to indicate that a variable key can only be Literal["key1", "key2"] without defining a this new type?

I do not think there is some kind of LiteralKeys[MyDict]

But maybe there is a workaround

[trim tangle]

no

There might be some unsavory hacks depending on why you need that

[echo knot]

Not a specific use case. It was jjst quite annoying that within my repo, I have the problem of indexing my TypedDict with a string (infered type due to function signature) when I know that this string can only take the corresponding key-str values for my TypedDict

[rustic gull]

I love python 3.10

[trim tangle]

that's weirdly specific

[tranquil turtle]

3.10 is the last version that had no opcode caches in bytecode
3.11+ have them, and it makes working with bytecode a lot harder

[tacit sparrow]

can I somehow make a test file for typing?
e.g. something like this so it would show a typing error only if test("5") will work without errors
(kind of like pytest.raises but for type checking)

def test(a: int) -> None: ...

# should fail if the expression doesn't have any typing issues
with typing.ensure_typing_issues:
    test("5")

[trim tangle]

test("5")  # type: ignore
```  and then enable `--warn-unused-ignores` in mypy (or the equivalent in pyright)

[tacit sparrow]

nice, thank you!

# pyright: reportUnnecessaryTypeIgnoreComment=error

def test(a: int) -> None: ...
test("5") # type: ignore [reportArgumentType]

def test1(a: str) -> None: ...
# Unnecessary "# type: ignore" comment
test1("5") # type: ignore [reportArgumentType]

[pastel egret]

You may also want to use typing.assert_type().

[half isle]

X-Posting here in case anyone is a typing expert with sqlalchemy.

#databases message

[tacit sparrow]

but assert_type doesn't help in defining the cases that should fail, only cases that should work

[rare scarab]

well python doesn't have any concept of a Not type

[winged palm]

hey yall, having an issue with dynamic classes and type hinting that I can't seem to figure out. Instead of typing out everything I'll just link to my post in the python help channel #1253389046030401626 message Figured people in this channel might have some idea

[rare scarab]

So nothing like Sequence[str] & ~str

[undone saffron]

lol. Good luck with Sequence[str] & ~str This one's only a problem because of a type ignore in the typeshed and str subclassing Sequence when it doesn't conform to Sequence

[trim tangle]

Wait, why doesn't it conform to sequence

Oh, __reversed__

wait, sequences aren't required to have that

[soft matrix]

Sequence takes object for contains, str only takes str

[trim tangle]

ohh

So str isn't a sequence all along 🤯

So why is it forced to be a Sequence?

causing so much anguish to people expecting a Sequence[str] where a string is probably a mistake

[undone saffron]

You also cant just use the thing in useful_types for this, because str -> Sequnce[str] ->their thing

Which avoids it being detectable due to the original lie

Ultimately, lying about a type creates unsolvable siituations since there's no "total program wide consistent use" check requirement

[trim tangle]

int.hex my beloved

[tranquil ledge]

It seems more fundamental than that. str is documented as a "text sequence" in the main Python docs (link), and it's mentioned that it implements "all the common sequence operations"; I do think there's value in teaching it that way. The runtime registration of str as a virtual subclass of Sequence kinda mirrors that as well.

Basically, imo, this doesn't seem like typeshed's fault; it's just respecting a (seemingly) longstanding part of runtime as best as it can.

[undone saffron]

lying about the type isnt useful to people who care about the type.

teaching "str is implemented so that it behaves very similar to collections" doesnt preclude the type being accurate

This is very much a "who is typing for?" moment

[tranquil ledge]

True. I guess I'm just trying to come up with reasons that make sense for why it's the way it is, because I doubt it was a simple choice to lie for no benefits. At the very least, I'd guess that this virtual subclassing was happening before PEP 484 was accepted.

[trim tangle]

Do you think this line applies to int/float as well?

[tranquil ledge]

Oh, so this has been a thing since pre-2.7, if you count basestring being a virtual subclass as well. Damn. https://github.com/python/cpython/commit/3172c5d263eeffff1e89d03d79be3ccc1d60fbde#diff-163c1525f292b112e15d4d78952fb8bc6a4b80e6ec8d608eea37edad2cfed28e

Feels like it would be easier to bend *str.whatever to accept object at this point and have __reversed__ for consistency /hj

[undone saffron]

I think that in the case of a gradual type system which isn't enforced at runtime, it is better to leave things untyped or partially typed with accurate structural information than it is to lie about the type

Its also hard to fix retroactively, as some of this predates the type system

its interesting you bring up float/int though. For the same reasons the numbers module is unsafe to use as typehints, so are some of the collections in collections.abc

[trim tangle]

I bring this up because for some reason that design choice seems particularly strange to me. I might be hyper-focused on it a bit (and some people love it)

[undone saffron]

It costs quite a bit in an on-demand computing situation to have ints where floats were expected. It has the appearance of making sense right up until the difference matters

[trim tangle]

I don't think I quite understand you

[undone saffron]

native code where float math is cheaper than arbitrary precision integer math, dispatching on the types provided

[trim tangle]

ah

well it has to do type checking anyway, right? 🤔

though converting an int to a float might eat up a lot

[undone saffron]

Yes, but the native code cant know the type it got was "unepexpected" in a situation where the native code is translating code from math majors, not CS majors. $work added some temp workarounds for that, but the longterm was dropping python for this usecase 😦

You cant typecheck numerics from python being "actually a float", theres no way to exclude ints statically

Anyhow, i got asked by a few people "how do you fix it" and ill write up my thoughts more constructively at a later date

The lying about types thing though just breaks the ability to do logic, from false premises you can claim anything

[trim tangle]

Should there be an asyncio.gather2 that actually returns a tuple?

Well, not sure if that's super relevant since we have task groups nowadays

but in general, when you have an older API that's impossible to type, and it would cost time/memory to do some conversion

[trim tangle]

I mean, my main gripe was not with accepting an int or a float in many cases, that can already be done with float | int ||(maybe math majors don't run type checkers?)||. It was with the virtual subtyping that doesn't match the runtime

[undone saffron]

probably not, you can type it accurately enough for the usecases that dont fit into task groups (rare, but exist) with typevar tuples

[trim tangle]

asyncio.gather returns a list at runtime, that's what I meant

[undone saffron]

🤔 yeah, but what's the issue with that?

[trim tangle]

In the typeshed it's typed as returning a tuple when you're calling it with a fixed number of arguments

[undone saffron]

all the uses for gather require you keep a reference to original coro/tasks anyhow

[trim tangle]

Why?

At least the examples in the asyncio docs suggest otherwise

[undone saffron]

gather takes varargs. Presumably you do have an actual source for what goes into those. Theres not actually a reasonable case where you would care about the task results, and have heterogenous task result values, and didnt construct tasks ahead of time (now that task wrapping automatically isnt a thing)

List[Task[Any]] would be a perfectly acceptable return type too

[trim tangle]

From the asyncio.gather docs;

If any awaitable in aws is a coroutine, it is automatically scheduled as a Task.

async def main():
    # Schedule three calls *concurrently*:
    L = await asyncio.gather(
        factorial("A", 2),
        factorial("B", 3),
        factorial("C", 4),
    )
    print(L)

maybe I'm misunderstanding what you mean

[undone saffron]

Deprecated since version 3.10: Deprecation warning is emitted if no positional arguments are provided or not all positional arguments are Future-like objects and there is no running event loop.

so, list[Future[Any]]

could do list[Future[Union[*Ts]]] if union is undprecated and gains unpack support

[trim tangle]

oh

[rare scarab]

Has typing-extensions ever had a breaking change?

[oblique urchin]

yes. mostly unintentionally but the 4.0 release removed some stuff

[rare scarab]

That's good. It means my response on this discussion isn't wrong.
https://github.com/tiangolo/fastapi/discussions/11746#discussioncomment-9840852

[oblique urchin]

Yeah I try to never make any more breaking changes in typing-extensions because users' lives become very difficult if libraries start upper-bounding typing-extensions

So I don't want to give people reasons to use upper bounds

[tacit sparrow]

what's the simplest way to narrow down the type of the list elements?

def ret() -> list[int | None]: return []

a = ret() # list[int | None]

# something like assert all(isinstance(i, int) for i in a)
...

reveal_type(a) # list[int]

[trim tangle]

It is impossible

You can make a copy of the list with only ints in it

The reason it's impossible is (by the means of an example):

things: list[int | None] = [1, 2, 3]

def ret() -> list[int | None]:
    return things

a = ret()  # list[int | None]
# do some kind of checking, now a: list[int]

things.append(None)  # completely valid action
# but now `a` has a `None` in it. but we think it's a `list[int]`. That's very bad.

Why are you returning some Nones in that list by the way?

[tacit sparrow]

I meet the same case from time to time when I have something like lst = [get_object_part(o) for o in elements] .

Where get_object_part returns either some type A or None. Typically, in 95% cases it's always just list[A] but I wanted to guard myself somehow from those Nones and to make sure they won't be just skipped if I do lst = [o_:=get_object_part(o) for o in elements if o_]

[jade viper]

How can I copy the type hints of a Callable? For instance: Callable[..., T] , what shuold go in ...?

[trim tangle]

def not_null[T](item: T | None) -> T:
    if item is None:
        raise ValueError("Unexpected None value")
    return item
...
lst = [not_null(get_object_part(o)) for o in elements]
``` or something like ```py
lst = filter(None, (get_object_part(o) for o in elements))
``` if the object is otherwise always  truthy

[trim tangle]

Can you show an example?

[tacit sparrow]

Yeah, I was just wondering if there is some general tool already available from typing / typing_extensions that I'm missing, I was imagining for it to work something like a = [i for i in a if assert_(i, int)] or a = [assert_(i, int) for i in a]

[trim tangle]

typing provides very little in terms of runtime checking (because it's for static typing)

I think filter(None is kinda useful

!d filter

[rough sluice]

filter

filter(function, iterable)```
Construct an iterator from those elements of *iterable* for which *function* is true. *iterable* may be either a sequence, a container which supports iteration, or an iterator. If *function* is `None`, the identity function is assumed, that is, all elements of *iterable* that are false are removed.

Note that `filter(function, iterable)` is equivalent to the generator expression `(item for item in iterable if function(item))` if function is not `None` and `(item for item in iterable if item)` if function is `None`.

See [`itertools.filterfalse()`](https://docs.python.org/3/library/itertools.html#itertools.filterfalse) for the complementary function that returns elements of *iterable* for which *function* is false.

Similar names: django.filter, label.filter, 2to3fixer.filter

[trim tangle]

Though if you have ints in the actual list, it's no good

[tacit sparrow]

but filter has the same problem as adding if i to the comprehension - some None values might slip in

[trim tangle]

how?

None is falsey

[tacit sparrow]

no, I mean that they will be skipped

[trim tangle]

Ah, so you want to raise an error if you get a None?

[tacit sparrow]

when they might indicate bugs

yes

[trim tangle]

Maybe it's something you should do in the get_object_part function then?

[tacit sparrow]

well, this is the problem 😄

this function was added a while ago and codebase sometimes relying on it able to return None

I'm thinking that maybe I should just add get_object_part_safe

that will raise an exception

[trim tangle]

yeah that's definitely an option

[tacit sparrow]

or get_object_part(obj, safe=True)

[trim tangle]

that will create some nasty overloads

[tacit sparrow]

and using @overload make typing correct

are overloads bad?

[trim tangle]

Yeah, they make the function more complicated. And type checkers can't verify that your overloads are correct

Also, if you make a new function, it's easier to deprecate the old function and slowly replace it, if you think raising an exception is a better default

If it's a local comprehension then #type-hinting message clearly doesn't apply. So you could just do ```py
if None in lst:
raise ValueError(...)
lst_safe = typing.cast(list[Obj], lst)

As always, if it's just one time, you can do anything you want 🙂 but you need to assess whether it will happen a second time

(or just note it somewhere)

[tacit sparrow]

it's interesting option, I haven't thought about it

[jade viper]

Sure!

def sequenceable(target: str) -> Callable[[Callable[..., T]], Callable[..., Union[T, Sequence[T]]]]:
    """
    This decorator processes a sequence of values passed as an argument to a function.
    The function is called for each value in the sequence, and the result is stored in the same position in the sequence.
    """

[trim tangle]

The problem with all of the "non-invasive" options so far is that you don't know which o caused an object part to not be found. It's going to make debugging a misery @tacit sparrow

[restive rapids]

!d typing.ParamSpec

[rough sluice]

typing.ParamSpec

class typing.ParamSpec(name, *, bound=None, covariant=False, contravariant=False)```
Parameter specification variable. A specialized version of [type variables](https://docs.python.org/3/library/typing.html#typevar).

In [type parameter lists](https://docs.python.org/3/reference/compound_stmts.html#type-params), parameter specifications can be declared with two asterisks (`**`):

```py
type IntFunc[**P] = Callable[P, int]
```  For compatibility with Python 3.11 and earlier, `ParamSpec` objects can also be created as follows:

```py
P = ParamSpec('P')
```  Parameter specification variables exist primarily for the benefit of static type checkers. They are used to forward the parameter types of one callable to another callable – a pattern commonly found in higher order functions and decorators. They are only valid when used in `Concatenate`, or as the first argument to `Callable`, or as parameters for user-defined Generics. See [`Generic`](https://docs.python.org/3/library/typing.html#typing.Generic) for more information on generic types.

[trim tangle]

How would you call this function? Can you show a more concrete example?

[restive rapids]

this will have a problem with strings wont it

[jade viper]

What do you mean by more concrete? But here's an example call

@sequenceable(target='date')
def last_day_of_next_week(date: DateT) -> DateT:

[trim tangle]

What type should the resulting function have?

[jade viper]

(date: DateT) -> Sequence[DateT]:

[trim tangle]

So maybe ```py
parts: list[Part] = []
for o in elements:
if part := get_object_part(o):
parts.append(part)
else:
raise RuntimeError(f"Part not found for {o!r}")

[restive rapids]

not -> DateT | Sequence[DateT]?

[trim tangle]

I don't see how that matches the docstring. Shouldn't the resulting function accept a sequence?

Can you show the implementation?