type-hinting

[rustic gull]

so idk how to avoid saying type:ignore

[restive rapids]

you could typing.cast it when passing, maybe, and make a pr to pytorch if it doesnt mutate but uses list / tuple

[undone saffron]

I don't see pytorch being wrong here. It accepts either of those and does not accept all sequences

[restive rapids]

sometimes typehints impose additional requirements that arent needed for the thing to work. i dont know if thats the case here

[undone saffron]

it doesn't appear to be

[rustic gull]

yes

but

torch.nn.Parameter is a subtype of torch.Tensor

and since those functions are typed as list[Tensor], and when I pass list[nn.Parameter] VSCode screams at me because list is invariant so it doesnt support subtypes

and I have no idea what to do about it

this thing

      Type parameter "_T@list" is invariant, but "Parameter" is not the same as "Tensor | Parameter"

wtf

then whats the point of using |

if it doesnt work

I put # type:ignore everywhere and it already hid 1000 other errors

[undone saffron]

if you pass a tuple of nn.Parameter it will work

[rustic gull]

why not list?

[undone saffron]

because it isn't typed to accept that

[rustic gull]

but it accepts that and I need to type my thing in the way that doesnt make everything red

[undone saffron]

no, it accepts list[Tensor]

[rustic gull]

but it works with list[nn.Parameter] I just need it to not say red things to me

[undone saffron]

without knowing the internals of pytorch, I can't tell you if it working with list[nn.Parameter] is intentional, and something pytorch would accept a modification to their typings to allow, or something that only works by coincidence and not something they would want to guarantee

[rustic gull]

it is intentional because they use it

its mainly used in optimizers where they pass lists of parameters to it

[undone saffron]

great, then your next step is updating their typings to match what they intend to accept because if it is intentional and the types aren't allowing it, there's an issue in types specified

[rustic gull]

how do I type it so that it accepts list[Tensor] and list[any child of tensor]

[undone saffron]

TensorTypeVar = TypeVar("TensorTypeVar", Tensor, nn.Parameter, ...)
type TensorSequence[TensorTypeVar] = list[TensorTypeVar] | tuple[TensorTypeVar, ...]

something like that would work

[trim tangle]

If it's not mutating the argument, it should probably accept Sequence[Tensor] or Iterable[Tensor] and not specifically list

[undone saffron]

it's not, but it also doesn't accept arbitrary iterables/sequences (that was checked earlier)

so that doesnt work

[rustic gull]

I just tried a deque and it only works with list and tuple othewise it says

TypeError: _foreach_add() received an invalid combination of arguments - got (collections.deque, collections.deque), but expected one of:
 * (tuple of Tensors self, Number scalar)
      didn't match because some of the arguments have invalid types: (!collections.deque!, !collections.deque!)
 * (tuple of Tensors self, tuple of Scalars scalars)
      didn't match because some of the arguments have invalid types: (!collections.deque!, !collections.deque!)
 * (tuple of Tensors self, Tensor other, *, Number alpha)
 * (tuple of Tensors self, tuple of Tensors other, *, Number alpha)

[trim tangle]

ah

[undone saffron]

TensorTypeVar = TypeVar("TensorTypeVar", bound=Tensor, default=Tensor)
type TensorSequence[TensorTypeVar] = list[TensorTypeVar] | tuple[TensorTypeVar, ...]

^ might work too,

[rustic gull]

i think its because it calls C code

[undone saffron]

there's a few options around this, but I would really need a better understanding of the internals to reccomend the right thing

[trim tangle]

Would it even be legal to use a TypeVar only once in the signature?

[undone saffron]

it's not there only once, and that's a type alias for a generic type, so yes

I haven't gotten around to the proposal about not requiring aliasing like this to pull some tricks, right now TypeVars offer constraints, bounds, and defaults and of those, constraints can't be expressed without them

but the limitation on needing multiple typevars is entirely artificial, not essential to the type system

[rustic gull]

okay so this second line is python 3.12 + unfortunately

hmmm

and without it I can say [1,2,3] and it doesn't show anything red

ok they actually changed pytorch into 3.12 now I need to update it

[undone saffron]

it's actually 3.13 due to the default there if you don't import typevar from typing_extensions (if you do, then it's 3.12)

[rustic gull]

hmmmmmmmmmmmm

I tested this and this

from typing_extensions import TypeVar
TensorTypeVar = TypeVar("TensorTypeVar", Tensor, nn.Parameter, ...)
type TensorSequence[TensorTypeVar] = list[TensorTypeVar] | tuple[TensorTypeVar, ...]

x: TensorSequence = [1,2,3]

from typing_extensions import TypeVar
TensorTypeVar = TypeVar("TensorTypeVar", bound=Tensor, default=Tensor)
type TensorSequence[TensorTypeVar] = list[TensorTypeVar] | tuple[TensorTypeVar, ...]

x: TensorSequence = [1,2,3]

neither say any errors

It doesn't seem to actually check what the list is of

[undone saffron]

sounds like a bug in whichever typechecker, that should be rejected in both cases unless Tensor is defined in a way that 1 2 and 3 each count as Tensors

[stiff acorn]

why do i get Any when I've explicitly annotated the possible return types?

[trim tangle]

I think pylance/pyright generally ignore the explicit annotation when they can fully infer the type from the expression

[rare scarab]

Try typing file with a typeddict

[stiff acorn]

typed dict is more code than I wanna write to fix this, so I went with a simple cast instead

[cinder bone]

How would I successfully do something like this?

S = TypeVar("S")
class SupportsOp(Protocol):
    def __add__(self: S, o: S) -> S: ...

N = TypeVar("N", bound=SupportsOp, default=float)

class ValueTracker(Generic[N]):
    # error: Incompatible default for argument "value" (default has type "float", argument has type "N")  [assignment]
    def __init__(self, value: N = 0.0, **kwargs): ...

[viscid spire]

(btw, why use dict() but then []?)

[stiff acorn]

for whatever reason vs code doesn't highlight the next .get() in the chain with {} instead of dict() when both should be identical

[viscid spire]

weird

[viscid spire]

the default of a TypeVar has to be compatible with the bound, which pylance says is not adhered to in your code

so you would have to make the bound a union to fix it I believe

~~```py
from typing import Protocol, Generic, Self
from typing_extensions import TypeVar

class SupportsOp(Protocol):
def add(self, o: Self) -> Self: ...

N = TypeVar("N", bound=SupportsOp | float, default=float)

class ValueTracker(Generic[N]):
def init(self, value: N = 0.0): ...

@cinder bone I found a better solution

your protocol is incorrect, the arguments should be positional-only

from typing import Protocol, Generic, Self
from typing_extensions import TypeVar


class SupportsOp(Protocol):
    def __add__(self, o: Self, /) -> Self: ...


N = TypeVar("N", bound=SupportsOp, default=float)


class ValueTracker(Generic[N]):
    def __init__(self, value: N = 0.0): ...

this now works

[cinder bone]

🤦‍♂️of course! Thanks so much!

[leaden oak]

src/pip/_internal/utils/misc.py:132: error:
"Callable[[FunctionType, Path, BaseException], Any] | Callable[[FunctionType, Path, tuple[type[BaseException], BaseException, TracebackType]], Any]" not callable 
[misc]
        handler: OnErr = partial(

Wait, what?

[trim tangle]

link?

[leaden oak]

It's a local run, but this is the relevant line of code.

[rough sluice]

src/pip/_internal/utils/misc.py line 132

handler: OnErr = partial(```

[trim tangle]

Sounds like the error is buggy in mypy.

[leaden oak]

And now you're about to tell me how this code is janky/stupid/can be refactored :P

I didn't write this, heh.

[trim tangle]

The call to partial here is not legal. If you have something like foo: Callable[[int], str], you can't do foo(quack=42), you can only use a positional argument: foo(42). If you want to specify a callback with keyword arguments, you need a Protocol with a __call__ method

Also, the union is strange. If you have an fn Union[OnExc, OnErr], how do you know what to pass as the third parameter? You can't know if the function accepts a BaseException or an ExcInfo

Actually this comment is wrong

        # `[func, path, Union[ExcInfo, BaseException]] -> Any` is equivalent to
        # `Union[([func, path, ExcInfo] -> Any), ([func, path, BaseException] -> Any)]`.

(A, B, C | D) -> E is equivalent to the intersection of (A, B, C) -> E and (A, B, D) -> E, not their union

e.g. if you have: ```py
def f(x: Callable[[int | str], None]) -> None:
if random.random() > 0.5:
x(42)
else:
x("hmm")

def g1(x: int) -> None: print(x ** 5)
def g2(x: str) -> None: print(x + "!")

g = random.choice((g1, g2))

g: Callable[[int] | None] | Callable[[str] | None]

``` obviously you can't call f(g), because f requires that (x must be able to process str AND x must be able to process int)

[trim tangle]

@leaden oak does this make sense?

[cinder bone]

wait I just tested and it still errors out

[viscid spire]

that's just MyPy being bad presumably

passes pylance/pyright

[trim tangle]

If you have a ValueTracker[str], the __init__'s default value will stil be 0.0, which is definitely not a str

you might still have to add some janky overloads (to ensure that if float is not compatible with the N, the argument is mandatory)

[viscid spire]

this is perfectly fine

[trim tangle]

hm, strange

[viscid spire]

mypy just bad in this respect

[trim tangle]

does the default on typevars somehow interact with default parameters?

[viscid spire]

I mean, that's kinda what it's supposed to be used for...

[trim tangle]

can you post the code as text?

wait, it's available above

[viscid spire]

from typing import Protocol, Generic, Self, reveal_type
from typing_extensions import TypeVar


class SupportsOp(Protocol):
    def __add__(self, o: Self, /) -> Self: ...


N = TypeVar("N", bound=SupportsOp, default=float)


class ValueTracker(Generic[N]):
    def __init__(self, value: N = 0.0): ...


x = ValueTracker()
reveal_type(x)

y = ValueTracker("abc")
reveal_type(y)

z: ValueTracker[str] = ValueTracker()

*edited btw

[trim tangle]

I thought the only thing TypeVar defaults changed was allowing you to spell ValueTracker() instead of ValueTracker[float](). Am I missing something in the PEP about method/function defaults?

Wait, it's not related to typevar defaults at all

[viscid spire]

ValueTracker[str]()

is invalid type-wise, if that helps

[trim tangle]

Why?

[viscid spire]

because you are saying that the type of value must be a str, but a float is assigned to it by default

str and float are not compatible

[trim tangle]

Yeah it seems like pyright does not like this code, but only the last line, not value: N = 0.0 ```py
from typing import Protocol, Generic, Self, TypeVar

class SupportsOp(Protocol):
def add(self, o: Self, /) -> Self: ...

N = TypeVar("N", bound=SupportsOp)

class ValueTracker(Generic[N]):
def init(self, value: N = 0.0):
self._value = value

def value(self) -> N:
    return self._value

v: ValueTracker[str] = ValueTracker()

It seems strange to me that value: N = 0.0 is allowed. Is there some part of the spec that allows this?

[viscid spire]

It seems very expected to me

from typing import reveal_type


class MyClass[T]:
    def __init__(self, value: T = 0.0): ...


x = MyClass()
reveal_type(x) # MyClass[float]

[trim tangle]

and it's only allowed in the __init__. Pyright does not like this method:

    def set_value(self, value: N = 0.0):
#                                  ^^^
# Expression of type "float" cannot be assigned to parameter of type "N@ValueTracker"
#   Type "float" is incompatible with type "N@ValueTracker"
        self._value = value
``` which is what I expected to happen in the `__init__`

[viscid spire]

that's because you can't bind the typevar after creation

because after the object exists, you have to know what type it is

[trim tangle]

Maybe it's pyright's own inference extension, in which case mypy isn't violating anything

[rustic gull]

hi someone know where i can get help for a program that im making ?

[viscid spire]

did you read the channel topic

[trim tangle]

If it's not related to type annotations, see #❓｜how-to-get-help

[viscid spire]

__init__ can determine the value of the typevar because it is part of creating the object

[trim tangle]

maybe

[viscid spire]

toy around with this in pyright

from typing import reveal_type


class MyClass[T]:
    def __init__(self, a: T = 1, b: T = "abc"): ...

reveal_type(MyClass()) # MyClass[int | str]
reveal_type(MyClass(True)) # MyClass[bool | str]
reveal_type(MyClass(b=2)) # MyClass[int]

[trim tangle]

yeah I understand that pyright does this, I'm asking if this kind of inference is "required" by the spec

[viscid spire]

no clue

[trim tangle]

We do a lil unsoundness ```py
from typing import Protocol, Generic, Self, TypeVar

class SupportsOp(Protocol):
def add(self, o: Self, /) -> Self: ...

N = TypeVar("N", bound=SupportsOp)

class ValueTracker(Generic[N]):
def init(self, foo: N, value: N = 0.0):
self._value = value + foo

def value(self) -> N:
    return self._value

class ValueHaverNco: SupportsOp:
def value(self) -> Nco:
...

class Hmm(Protocol):
def call(self, x: N, /) -> ValueHaver[N]:
...

def mischief(hmm: Hmm) -> None:
nasty = hmm("Hello, world")
print(nasty.value())

hmm1: list[Hmm] = [ValueTracker] # pyright says it's alright
mischief(hmm1[0]) # this as well (but will add a str and a float at runtime)

hmm2: Hmm = ValueTracker # this is alright
mischief(hmm2) # ...but this is not alright?!

[viscid spire]

that seems alright to me

[trim tangle]

Hm, is there a way to cause this without using Self in the parameter position? Since maybe that's the questionable part

[trim tangle]

Traceback (most recent call last):
  File "/tmp/foo/main.py", line 31, in <module>
    mischief(hmm1[0])             # this as well (but will add a str and a float at runtime)
    ^^^^^^^^^^^^^^^^^
  File "/tmp/foo/main.py", line 27, in mischief
    nasty = hmm("Hello, world")
            ^^^^^^^^^^^^^^^^^^^
  File "/tmp/foo/main.py", line 13, in __init__
    self._value = value + foo
                  ~~~~~~^~~~~
TypeError: unsupported operand type(s) for +: 'float' and 'str'

[viscid spire]

I think there is a misunderstanding

I'm gonna rewrite these names to make sense to me, and also eliminate any irrelevant code

[cinder bone]

hm interesting

[trim tangle]

Just the fact that hmm2: Hmm = ValueTracker is fine, but mischief(hmm2) is not fine seems problematic to me

[viscid spire]

and it says it's not fine?

[trim tangle]

Yes, it only flags the last line

But it thinks these are fine ```py
hmm1: list[Hmm] = [ValueTracker] # pyright says it's alright
mischief(hmm1[0]) # this as well (but will add a str and a float at runtime)

[cinder bone]

that feels like a bug in pyright?

[trim tangle]

exactly

[viscid spire]

from typing import Protocol, Self, Any


class SupportsAdd(Protocol):
    def __add__(self, o: Self, /) -> Self: ...


class ValueTracker[T: SupportsAdd]:
    def __init__(self, a: T, b: T = 0.0, /):
        self._value = a + b

    def value(self) -> T:
        return self._value


class ValueHaver[T: SupportsAdd](Protocol):
    def value(self) -> T: ...


class SomeTypeOfCallable[T: SupportsAdd](Protocol):
    def __call__(self, x: T, /) -> ValueHaver[T]: ...


def mischief(cls: SomeTypeOfCallable[Any]) -> None:
    x = cls("Hello, world")
    print(x.value())


hmm1: list[SomeTypeOfCallable[Any]] = [ValueTracker]  # pyright says it's alright
mischief(hmm1[0])  # this as well (but will add a str and a float at runtime)

hmm2: SomeTypeOfCallable[Any] = ValueTracker  # this is alright
mischief(hmm2)  # ...but this is not alright?!

if I write this, no flags

[trim tangle]

No, this code is different

Note that in here: ```py
class Hmm(Protocol):
def call(self, x: N, /) -> ValueHaver[N]:
...

[viscid spire]

that seems like the same thing but just implemented poorly

but I'll change it and see what occurs

hmm, it does change

[trim tangle]

class IdentityFunction(Protocol):
    def __call__(self, x: T, /)  -> T:
        ...

def f(x: T) -> T:
    print(x)
    return x

def g(y: int) -> int:
    return y + 1

f is an IdentityFunction, but g is not

[viscid spire]

🤷‍♂️

my brain is shutting down from all this type theory and bug pondering

thus, I'll leave

GL

[trim tangle]

👍

I think I won't be reporting this

[cinder bone]

wait why not?

[trim tangle]

maintainer will probably mark it as designed

[viscid spire]

it's also really hard to put in words what exactly is wrong

[cinder bone]

I mean at least there would be a record then

[trim tangle]

Think of it this way: it's a fun easter egg for someone else to find and get nerd sniped 🙂

[cinder bone]

lol

for now is there any workaround for mypy/pyright d'you think?

[viscid spire]

well pyright is working properly for what I posted originally

[cinder bone]

well yeah but mypy isn't

[trim tangle]

Maybe something like this?

@overload
def __init__(self: ValueTracker[float], value: float = ...) -> None: ...

@overload
def __init__(self, value: N) -> None: ...

[cinder bone]

hmm I'll try that

yeah that works if I do N | float in the real __init__'s signature

thanks!

[tranquil turtle]

def f() -> list[str]:
    return ''.split()  # Expression of type "list[LiteralString]" is incompatible with return type "list[str]"

good job, pyright

feels more like pywrong

[rustic gull]

that's bad.

[finite stirrup]

From https://github.com/python/typeshed/issues/10887 , maybe Iterable[str] would work.

[rare scarab]

You can always do [*''.split()]

or change the return type to Sequence[str]

[tranquil ledge]

Or do str.split("").

[pale fox]

neither Iterable[str] nor Sequence[str] will tell a str from list[str]. do they?

[oblique urchin]

correct

[undone saffron]

Depending on what you want to support, there's some options up here #type-hinting message not much that's accepting of things broadly that won't catch str though. There's also a protocol based solution in useful types, but it's fragile, and won't prevent str -> Sequence[str] -> useful_types.SequenceNotStr[str]

[main cargo]

in PEP742 (TypeIs), there's a comment in an example:

def f(x: Awaitable[int] | int) -> None:
    if isawaitable(x):
        # Type checkers may also infer the more precise type
        # "Awaitable[int] | (int & Awaitable[Any])"
        assert_type(x, Awaitable[int])
    else:
        assert_type(x, int)

What is the syntax (int & Awaitable[Any])? specifically the &? I can't search for it, and I can only find information on | for union types

[rare scarab]

It's fake syntax added by pyright. It basically means it is both types

[main cargo]

ah

on another note, are either assert isinstance(...) or cast() considered more idiomatic for type narrowing when necessary? I realise that only the former will affect runtime behaviour, but is that what people aim for?

[oblique urchin]

assert isinstance() is better because it gives you runtime safety

[main cargo]

makes sense, thanks

[undone saffron]

eh... just dont rely on the asserts raising if wrong and only use them in the same places you'd use a cast, when unconditionally it should be True. you don't get runtime safety, you may get occasional alarms when failures happen in non-production environments, they get removed with -O / PYTHONOPTIMIZE=1, both of which are commonly set

[tepid glade]

Hi. Is there a way to duplicate typehints of a method to another?
I wrote an function which overwrites fastapi.APIRouter.get in a different class, and the types of .get are too complex to copy paste one by one into the new function.

[tranquil ledge]

If you haven’t changed the parameters at all or only inserted new positional-only ones at the front, you can sorta do it with ParamSpec. A decorator like this might fit your use case?

[rough sluice]

src/__experimental__/_misc.py line 103

def copy_annotations(original_func: Callable[_P, _T]) -> Callable[[Callable[..., object]], Callable[_P, _T]]:```

[tranquil ledge]

Oh, that’s helpful. Uh, let’s try a better link.

Damn.

[tepid glade]

Yeah that seems to work.. But I just realized I will have to omit some params from between so I'll have to redefine every param anyways

Thank you

[tranquil ledge]

No problem.

[tepid glade]

It seems to copy all the annotations of the function.. I was looking for something like ```py
class Base:
def init(self, p1: int, p2: float, p3: bool) -> None: # Many parameters
...

class MyClass(Base):
def init(self, my_custom_param: str, *args, **kwargs) -> None: # args kwargs get the signature of base init
...

It doesn't have to be a class' init method, just an example

[tranquil ledge]

You might be able to get that to work with typing.Concatenate.

[tepid glade]

Okayy I'll have a look at it

[frozen matrix]

which group do i go to for help

[grave fjord]

#❓｜how-to-get-help

[frozen matrix]

thank you

[trail kraken]

Is this error expected behaviour? https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=38c376ba8f14274d43c0c6f0a4c89c07

class Base1:
    pass

class Base2:
    def is_same(self, other: "Base1") -> bool:
        #  error: Non-overlapping identity check (left operand type: "Base2", right operand type: "Base1")  [comparison-overlap]
        return self is other

class Merged(Base1, Base2):
    pass

merged = Merged()
merged.is_same(merged)

This is with --strict.

[cinder bone]

I've never actually seen -O being used

[oblique urchin]

that seems right to me. The comparison overlap check would be a lot less useful if it took into account the possibility of multiple inheritance

[trim tangle]

I wish I could mark a class as "cannot take part in multiple inheritance"

[grave fjord]

You can probably do it with __init_subclass__

[trim tangle]

well I mean, at type checking level

like @final

[trim tangle]

That seems inconsistent with

class Base1:
    pass

class Base2:
    pass

class Merged(Base1, Base2):
    pass

def f(x: Base1) -> None:
    if isinstance(x, Base2):
        reveal_type(x)  # Revealed type is "__main__.<subclass of "Base1" and "Base2">"

[trail kraken]

Thanks for confirming. I will just stick a type ignore in there then.

[trail kraken]

That's an interesting workaround. So isinstance(other, Base2) and self is other would technically work.

[trim tangle]

Yeah, it's strange

[buoyant ridge]

hey y'all, i was wondering how dataclasses manage to wrangle the type checker into dynamic behaviour. for example, how come this doesn't work?

from dataclasses import dataclass


def mydataclass(cls: type):
    class MyDC(cls):
        pass

    MyDC.__annotations__ = cls.__annotations__
    return dataclass()(MyDC)


@mydataclass
class MyDC:
    x: int


@dataclass
class DC:
    x: int


MyDC() # passes
DC()  # type error

i kind of want to get this to work so that either a decorator or base class naturally confers the dataclass magic

but ill listen if someone tells me that's bad practice lol

[grave fjord]

https://peps.python.org/pep-0681/

[buoyant ridge]

oh perfect! tysm!

[stiff acorn]

how come there's a difference between vscode's mypy extension and mypy

❯ mypy --version
mypy 1.11.0 (compiled: yes)

❯ mypy .\test.py --strict
Success: no issues found in 1 source file

Using the official extension https://github.com/microsoft/vscode-mypy

[oblique urchin]

looks like vscode is using an old version of mypy

[stiff acorn]

https://github.com/microsoft/vscode-mypy/releases/tag/v2023.6.0 if this is anything to go by then it's really old

[cinder bone]

How hard exactly would it be to implement a Proxy type in mypy/pyright

[tranquil turtle]

type Proxy[T] = T

def make_proxy(obj: T) -> Proxy[T]:
    if TYPE_CHECKING: return obj
    else: return _make_actual_proxy(obj)

[cinder bone]

No I need a distinction between Proxy[T] and T

#type-hinting message

Thinking that if I can get a general hold on how difficult it is to implement then maybe I can consider making a case for a PEP

[rain crow]

hello guys, i've been using python for 4 months now and I want to learn about type hinting indepthly on how can I leverage it for my existing projects is there a learning material or resources that tackles type hinting indepthly in python ?

[stiff acorn]

the stdlib docs (and the ones linked there) are pretty good https://docs.python.org/3/library/typing.html

[stiff acorn]

How many overloads would I need to add to also support CompressionLevel | int because CompressionLevel is a literal of 1-9?

    @overload
    def __init__(
        self,
        file: StrPath,
        mode: OpenArchiveMode = "r",
        *,
        password: str | None = None,
        compression_type: CompressionType | None = None,
        compression_level: CompressionLevel | None = None,
        **kwargs: Any,
    ) -> None: ...

    @overload
    def __init__(
        self,
        file: StrPath,
        mode: str = "r",
        *,
        password: str | None = None,
        compression_type: CompressionType | None = None,
        compression_level: CompressionLevel | None = None,
        **kwargs: Any,
    ) -> None: ...

    def __init__(
        self,
        file: StrPath,
        mode: OpenArchiveMode | str = "r",
        *,
        password: str | None = None,
        compression_type: CompressionType | None = None,
        compression_level: CompressionLevel | None = None,
        **kwargs: Any,
    ) -> None:

OpenArchiveMode is a literal of open modes

I'm trying to add their bases (str for OpenArchiveMode and int for CompressionLevel) to avoid type checkers complaining at times

Would I need to overload every possible combination?

OpenArchiveMode, CompressionLevel
literal, literal
literal, int
str, literal
str, int

Thinking of disabling the formatter for the overload section to make it more bearable

[stiff acorn]

    # fmt: off
    @overload
    def __init__(self, file: StrPath, mode: OpenArchiveMode = "r", *, password: str | None = None, compression_type: CompressionType | None = None, compression_level: CompressionLevel | None = None, **kwargs: Any) -> None: ...

    @overload
    def __init__(self, file: StrPath, mode: OpenArchiveMode = "r", *, password: str | None = None, compression_type: CompressionType | None = None, compression_level: int | None = None, **kwargs: Any) -> None: ...

    @overload
    def __init__(self, file: StrPath, mode: str = "r", *, password: str | None = None, compression_type: CompressionType | None = None, compression_level: CompressionLevel | None = None, **kwargs: Any) -> None: ...

    @overload
    def __init__(self, file: StrPath, mode: str = "r", *, password: str | None = None, compression_type: CompressionType | None = None, compression_level: int | None = None, **kwargs: Any) -> None: ...
    # fmt: on

[grand furnace]

With python 3.8, I get this error with mypy

    def verify_if_command_fails(cmd_output: CompletedProcess[str]) -> None:
E   TypeError: 'type' object is not subscriptable

It's normal because CompletedProcess[str] is only valid with python 3.9 and more
So, I tried to do this

if version_info >= (3, 9):
    CompletedProcessType = CompletedProcess[str]
else:
    CompletedProcessType = CompletedProcess
...
def verify_if_command_fails(cmd_output: CompletedProcessType) -> None

But, now I get this error:

video_timestamps\ffprobe\ffprobe.py:16: error: Cannot assign multiple types to name "CompletedProcessType" without an explicit "Type[...]" annotation  [misc]
video_timestamps\ffprobe\ffprobe.py:77: error: Argument 1 to "verify_if_command_fails" of "FFprobe" has incompatible type "CompletedProcess[str]"; expected "type[CompletedProcess[str]]"  [arg-type]
video_timestamps\ffprobe\ffprobe.py:79: error: Incompatible return value type (got "CompletedProcess[str]", expected "type[CompletedProcess[str]]")  [return-value]

What should I do to support the typing on python 3.8 and more?

[cinder bone]

from __future import annotations and if typing.TYPE_CHECKING is what I usually do
It avoids having the whole 3.8/3.9 split

so something like

from __future__ import annotations

def verify_if_command_fails(cmd_output: CompletedProcess[str]) -> None: ...

[grand furnace]

I decided to drop python 3.8
CompletedProcess[str] isn't valid for python 3.8, so I replaced it with CompletedProcess, but then mypy told me to use CompletedProcess[str]...

[viscid spire]

did you try with the future annotations import?

[grand furnace]

Yes

[viscid spire]

any code sample? and what issue popped up when you tried that?

[grand furnace]

No, I decided to drop it
I don't care anymore

[viscid spire]

fair enough

3.13 is coming out in October anyways

[lethal delta]

It's a reasonable choice imo.

[cinder bone]

Yeah it goes EOL in a few months

A lot of projects (e.g. sphinx) have even dropped 3.9

[viscid spire]

that is unequivocally based

[cinder bone]

fr

[grand furnace]

Honestly, I don't understand why project drop python version that aren't EOL

[rare scarab]

pytest-httpx dropped support for python 3.8, but httpx still supports it

[oblique urchin]

Supporting more versions is more work. Dropping versions makes maintainers' lives easier.

[grand furnace]

Sure, but it isn't that bad. I guess all my package are too small, so I don't get too much problem with supporting old python version

[rare scarab]

It mostly effects compiled packages.

and packages that want to use newer features

[oblique urchin]

https://numpy.org/neps/nep-0029-deprecation_policy.html#all-cpython-supported-versions for some relevant motivation

[exotic plover]

Is there a reason why we can't use TypedDict subclasses in place of existing TypeVars?

https://github.com/microsoft/pylance-release/issues/6110

I'd like to be able to do something like:

class ListMixin[R, F]:
    """List R-type resources using an F-type filter"""

    def list(**kwargs: Unpack[F]) -> R:
        ...

@dataclass
class MyResource:
    active: bool
    created: datetime.datetime
    body: str

class MyResourceFilters(TypedDict):
    active: NotRequired[bool]
    older_than: NotRequired[datetime.datetime]
    ....

class MyResourceManager(ListMixin[MyResource, MyResourceFilters]):
    pass

But I can't have the (otherwise valid) MyResourceFilters be passed in generically (as an implied TypeVar). It works if I explicitly use Unpack[MyResourceFilters] but then I can't use it with generics

[muted iron]

@functools.cache
def f() -> None: ...

f.__name__  # works, but Cannot access attribute "__name__" for class "_lru_cache_wrapper[None]"

This seems wrong to me.

[trim tangle]

Sounds like there's no __name__ attribute in typeshed, you could make an issue/PR: https://github.com/python/typeshed/blob/ea14235f74a0c8c6e58dac06c1d554f6bac74a4e/stdlib/functools.pyi#L54-L64

[rough sluice]

stdlib/functools.pyi lines 54 to 64

@final
class _lru_cache_wrapper(Generic[_T]):
    __wrapped__: Callable[..., _T]
    def __call__(self, *args: Hashable, **kwargs: Hashable) -> _T: ...
    def cache_info(self) -> _CacheInfo: ...
    def cache_clear(self) -> None: ...
    if sys.version_info >= (3, 9):
        def cache_parameters(self) -> _CacheParameters: ...

    def __copy__(self) -> _lru_cache_wrapper[_T]: ...
    def __deepcopy__(self, memo: Any, /) -> _lru_cache_wrapper[_T]: ...```

[trim tangle]

Or you could do f.__wrapped__.__name__

[tranquil wave]

what are the chances of an intersection type ever been added?

i've found myself wanting to be able to dynamically set attributes a number of times, it's unfortunate to set it as Any or add # type: ignore

[restive rapids]

its being worked on. for a long time, though
hard problem, somewhat trackable progress available at https://github.com/CarliJoy/intersection_examples and discussion that it originated from https://github.com/python/typing/issues/213

[tranquil wave]

discussion link?

oh boy, since 2016

[restive rapids]

even 2014 https://github.com/python/typing/issues/18

[tranquil wave]

the repo doesn't look very active :(

[restive rapids]

mhm. damn special cases like inheritance order and intersection with Any make it hard

[tranquil wave]

is anyone actively working on it?

carlijoy has only done a few things on github in the past several months

[undone saffron]

I've diverted the time I was placing into that elsewhere. There's not enough appetitite to fix soundness issues that make specifying this a problem, and I don't really want to write a half-baked specification that wouldnt be able to cover my own needs for the feature just to get it existing now.

not sure if anyone else is still actively working on it

[gilded cedar]

unrelated but 2016 was 8 years ago 👀

[languid aspen]

Let's say I had the following type variables and type aliases py A = TypeVar("A") B = TypeVar("B") Result: TypeAlias = tuple[A, B] Function: TypeAlias = Callable[[A], Result[A, B]] the following class ```py
class Wrapper(Generic[A, B]):
def init(self, func: Function[A, B]):
self.fn = func

def call(self, inp: A) -> Result[A, B]:
return self.fn(inp)
and the following codepy
T = TypeVar("T")

def foo() -> Wrapper[Sequence[T], T]:
def inner(bar: Sequence[T]) -> Result[Sequence[T], T]:
blah blah # whatever this doesn't matter

return Wrapper(inner)
``` my problem is that when I do something like foo().call("hello"), it doesn't seem like my type checker can infer that the type of foo() is Wrapper[Sequence[str], str], I need to type hint it explicitly. Now, maybe I'm going about this wrong, but is there a way for me to meet my end goal? What I suppose I want is for a type checker to be able to tell what the types of A and B are depending on what call is called with.

[restive rapids]

i think foo() tries to resolve the T typevar, but it has nothing to bind it from, so you end up with Wrapper[Sequence[Unknown], Unknown]

[languid aspen]

I see

hmm

I'm not sure how I'd avoid it here

[restive rapids]

what do you want to actually do? why do you need to call foo with no args to then get something which works forall T?

the Wrapper[A, B] implies that it will work only from A to B, i think you just want

def inner[T](bar: Sequence[T]) -> Result[Sequence[T], T]:
  blah blah

?

[languid aspen]

so I'm writing parsers and a bunch of combinators for these parsers, and Wrapper is essentially just a way for me to wrap any function which takes a generic input and returns a result in the form of a tuple of the remaining input after parsing, and the parsed output. I don't necessarily need to call foo with no args here, I just need to wrap inner with Wrapper and have it accept any Sequence[T].

[languid aspen]

if I were to wrap this with Wrapper, it still wouldn't work.

[restive rapids]

hmh. well, the A and B type parameters need to be resolved, you cant make a generic wrapper (that'd need.. existential types or whatever. ∀T. Wrapper[Sequence[T], T])

[languid aspen]

I see

thanks for the insight, I think I have a workaround.

[trim tangle]

Python's type system doesn't allow specifying the type of a "generic object". For example, list[T] or Callable[[T], T] doesn't make sense if T is not bound to anything.

(this would be a kind of existential universally quantified type if you're familiar with Haskell)

[languid aspen]

I see

I found a way around it, thanks for the insight.

[trim tangle]

I think I had a long discussion about it with erictraut but we didn't understand each other

[muted iron]

Message = Literal["FORWARD", "BACKWARD", "UP"]

def switch(message: Message) -> None:
    match message:
        case "FORWARD":
            assert_type(message, Literal["FORWARD"])
            print("go!")
        case "BACKWARD":
            assert_type(message, Literal["BACKWARD"])
            print("back...")
        case "SIDEWAYS":
            assert_type(message, Literal["SIDEWAYS"])  # "assert_type" mismatch: expected "Literal['SIDEWAYS']" but received "Never"
            print("woahh!?")
        case other:
            assert_never(other)  # Type "Literal['UP']" is incompatible with type "Never"

Is this a good use case for Literal and assert_type?
I like this behaviour. However, it seems repetitive. Can this be done better?

[rare scarab]

Is it not a good case for match

But yes, literal can be used for that.

[muted iron]

Well yeah, if this were all of it, it could be a dict. But this just a MRE, and the real one is better of a match.

But having to write assert_type every time seems repetitive.

[rare scarab]

Why would you need to assert the type?

Try ```py
case "FORWARD" as msg:
assert_type(msg, Literal["FORWARD"])

[muted iron]

Why would you need to assert the type?
So the type checker tells me that SIDEWAYS does not exist.

[oblique urchin]

Type checkers often have options for warning about unreachable code, if you turn that on it should hopefully catch this

[muted iron]

Even without the assert_type's?

[oblique urchin]

yes

I haven't tried this particular case but pyright for example tells me if I do ```def f(x: int):
if not isinstance(x, int):
print("unreachable")

[muted iron]

def f(x: int):
   if not isinstance(x, int):
      print("unreachable")  # Code is unreachablePylance

def g(x: Literal[0]):
   match x:
      case 1:
         print("unreachable")  # no warning

hmm

Doesn't look like it works with match case.

[rare scarab]

Those aren't the same

[muted iron]

Yeah, but they're both unreachable.

def f(x: Literal[0]):
   if x == 1:
      print("unreachable")  # Code is unreachablePylance

def g(x: Literal[0]):
   match x:
      case 1:
         print("unreachable")  # no warning

Those are the same.

Same curiousity.

[oblique urchin]

Might be worth reporting a bug, try to see if it's been reported before though

[rare scarab]

Try with mypy

is it pylance or pyright doing the unreachable check?

[muted iron]

It says Code is unreachable - Pylance

So Pylance I guess.

[oblique urchin]

Probably comes from pyright

[rare scarab]

I get errors with basedpyright http://basedpyright.com/?typeCheckingMode=all&code=GYJw9gtgBALgngBwJYDsDmUkQWEMoBUAsAFCkAmApsFMABQAeAXFADJIyUgCGANgNoAGALoBKJqShSkNBlAC88qAEYJJKRqgIQqGHQBEAVxQhK3AMYALbgCNelfaKkBiKAGEwVTAGcox0xbWdpQACnC83CjmlKQU1FBojCzsnDwCIuKSUhDcMFZQzFma5tzelCpqmlXaugb%2BZla29o4uUChgUADu3CAoqGhFVVBAA

Pattern will never be matched for subject type "Literal[0]"

So pyright doesn't warn about unreachable code, that's pylance. Basedpyright does report on unreachable though

[trim tangle]

based

[rare scarab]

basedpyright is based for implementing pylance features into upstream that microsoft refuses to

[muted iron]

I can't get MyPy to warn me about either as unreachable, but I'm probably missing some setting to enable it looking for that.

[rare scarab]

With strict?

You need to configure it with warn_unreachable=true

[oblique urchin]

--warn-unreachable I believe

[rare scarab]

https://adamj.eu/tech/2021/05/19/python-type-hints-mypy-unreachable-code-detection/

[trim tangle]

mainland pyright does complain about something on strict

[rare scarab]

I see, it just doesn't mark it as an error

basedpyright does the same with strict mode instead of all

[trim tangle]

all is too based for me

[rare scarab]

Why is it the default?

[trim tangle]

no idea

Maybe it assumes it's used by mostly typing enthusiasts that tend to want all the pain there ir

[minor nimbus]

class Basic:
    pass

class Specific(Basic):
    pass

def func(arg: ???):
    ...

Given the above, is there a way to type the arg so that Basic is accepted, but Specific isn't? arg: Basic will accept both, base class and subclass, while I need base class only.

[oblique urchin]

no, there is not. There are workarounds, such as an overload for Specific, but they won't work in all cases.

[tranquil turtle]

x: Basic = Specific()
func(x) # ok

[oblique urchin]

Right, the general problem is that any variable with a static type of Basic may actually hold an instance of Specific at runtime

[minor nimbus]

Right, I see. Kinda sad there's no way to do that.

I have a subclass that changes the behavior of the base class slightly, so only the base class would ideally be accepted there.

But oh well.

[void panther]

Why does this fail with No parameter named "context" in pyright?

class GraphQLErrorsWithContext(ExceptionGroup):
    def __init__(self, *args, context):
        super().__init__(*args)
        self.context = context

GraphQLErrorsWithContext("", [Exception()], context={})

[restive rapids]

I'm on phone so can't test properly but it might be because ExceptionGroup has a custom __new__

[void panther]

yeah defining a new that just does a super call fixes it

[languid aspen]

so let's say I've got the following ```py
C = TypeVar("C")
D = TypeVar("D")
E = TypeVar("E")
F = TypeVar("F")

class Foo(Generic[C, D]):
def init(self, fn: Callable[[C], tuple[C, D]]):
self.fn = fn

def bar(self, fn: Callable[[D], Foo[C, E]]) -> Foo[C, E]: ...

def baz(self, fn: Callable[[D], F]) -> Foo[C, F]:
    return self.bar(lambda i: test(fn(i)))

Z = TypeVar("Z")

def test(value: Z) -> Foo[Sequence[Z], Z]:
def inner(inp: Sequence[Z]) -> tuple[Sequence[Z], Z]:
return (inp, value)

return Foo(inner)

I get the following error with pyrightpy
Argument of type "(r: D@Foo) -> Foo[Sequence[F@baz], F@baz]" cannot be assigned to parameter "fn" of type "(D@Foo) -> Foo[C@Foo, E@bar]" in function "bar"
Type "(r: D@Foo) -> Foo[Sequence[F@baz], F@baz]" is incompatible with type "(D@Foo) -> Foo[C@Foo, E@bar]"
Function return type "Foo[Sequence[F@baz], F@baz]" is incompatible with type "Foo[C@Foo, E@bar]"
"Foo[Sequence[F@baz], F@baz]" is incompatible with "Foo[C@Foo, E@bar]"
Type parameter "C@Foo" is invariant, but "Sequence[F@baz]" is not the same as "C@Foo

hm

[trim tangle]

Do you know what variance is?

[languid aspen]

not extensively, no.

I was thinking it might help, I'm in the process of reading through the guide you have on your website.

[trim tangle]

are you python3.12 or later?

[languid aspen]

yes, but I'd prefer to support older versions.

so everything should be compatible with at least 3.9

[trim tangle]

can you explain what you're trying to do?

(maybe try replacing the lambda with a def function with explicit types)

test() is returning a Foo[Sequence[Z], Z], but the code in Foo.bar or Foo.baz never mentions any kind of sequence

[languid aspen]

so, bar is a function which calls self.fn, takes the output of that (which in the tuple (C, D) is D), and calls the fn passed to it on D. fn is any callable which returns a new instance of Foo by modifying the given D, and turning it into something of type E. bar then returns this new instance of Foo. It's implemented as follows py def bar(self, fn: Callable[[D], Foo[C, E]]) -> Foo[C, E]: def inner(inp: C) -> tuple[C, E]: result = self.fn(inp) new = fn(result[1]) return new.fn(result[0]) return Foo(inner) and baz is something that uses bind to create a new instance of Foo where the type of the output of self.fn is mapped to something else by the fn passed to it.

[trim tangle]

Do you have more human names for the methods maybe?

[languid aspen]

bar is bind and baz is map

[languid aspen]

right I guess what I probably want is to ensure that all Cs here are sequences.

[trim tangle]

why though

If you do want that, you might want to add a bound: C = TypeVar("C", bound=Sequence)

[languid aspen]

Sequence needs a type variable though doesn't it

[trim tangle]

C = TypeVar("C", bound=Sequence[object])

[languid aspen]

hrm

[grim pumice]

I'm getting this error from my nvim pyright LSP, but I can't figure out how to reproduce manually. D:

for test_case in ["foo", b"bar"]:
    print(urllib.parse.parse_qsl(test_case))
    # └╴E  Argument of type "str | bytes" cannot be assigned to "AnyStr@parse_qsl | None" Pyright (reportArgumentType)

It reproduces in pyright playground though!
I've filed a bug: https://github.com/microsoft/pyright/issues/8623

[tranquil turtle]

that is not an issue in pyright

if it is an issue at all, then it is a typeshed's issue

[grim pumice]

how so

[tranquil turtle]

but i dont think there is a problem, error message is reasonable

[grim pumice]

str isn't a valid member of AnyStr |None?

It checks fine if i unroll the loop. Normally type errors aren't affected by loop unrolling like that.

[tranquil turtle]

str | bytes is not assignable to AnyStr, which is a typevar constrained to (str, bytes)

what type do you expect as a result of urllib.parse.parse_qsl(test_case) ?

[grim pumice]

str|bytes, since it's passed str|bytes, which are individually valid and defined to return the type it's passed.

that's what a pre-typing intuitive read of this code would expect, does expect

[tranquil turtle]

the function doesnt return str|bytes, it returns list of name-value pairs

!d urllib.parse.parse_qsl

[rough sluice]

urllib.parse.parse\_qsl

urllib.parse.parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace', max_num_fields=None, separator='&')```
Parse a query string given as a string argument (data of type *application/x\-www\-form\-urlencoded*). Data are returned as a list of name, value pairs.

The optional argument *keep\_blank\_values* is a flag indicating whether blank values in percent\-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included.

The optional argument *strict\_parsing* is a flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) exception.

[grim pumice]

sure i misspoke

but i'd expect it to return list[tuple[AnyStr@parse_qsl, AnyStr@parse_qsl]] with AnyStr@parse_qsl having type-value str|bytes

I'd love to define my test_case as list[AnyStr] but that's not allowed

I guess I don't see why/how str|bytes is incompatible with a constraint of str, bytes

certainly at runtime, it's not

[tranquil turtle]

why list[tuple[str|bytes, str|bytes]]?
why not list[tuple[str, str]] | list[tuple[bytes, bytes]] ? it is what you will get at runtime

[grim pumice]

that would be the correct simplification of substituting the typevar, agreed.
It's why I never said list[tuple[str|bytes, str|bytes]].

[trim tangle]

🥴

import urllib.parse

for test_case in ["foo", b"bar"]:
   if isinstance(test_case, str):
      value = urllib.parse.parse_qsl(test_case)
   else:
      value = urllib.parse.parse_qsl(test_case)
   print(value)

[grim pumice]

to me, that should obviously have equivalent typecheckness

[trim tangle]

Yep. It is very silly IMO

I'd expect a constrained typevar like AnyStr work exactly like an overload.

From a theory point of view, if f(A) = A' and f(B) = B' then f(A | B) should be A' | B', not an error

[grim pumice]

is that missing from the spec? is that the issue here?

I'll go update typeshed to use overloads instead of AnyStr everywhere real quick

[tranquil turtle]

i believe it is an incorrect use of constrained typevar in typeshed

[grim pumice]

how so

what's incorrect here that's correct in some other case?

[tranquil turtle]

you can replace AnyStr with regular typevar, and it will work

[grim pumice]

it's an incorrect implemetation (or is it specification?) of typevar constraints imo

[tranquil turtle]

i think you dont understand typevar constraints
they are different from typevar bounds

[grim pumice]

lol yes i had absolutely no idea there were two extremely similar (sounding?) conceptws

please feel free to point me to the doc i've not read. i don't mean to waste your time

and thank you

[trim tangle]

A constrained typevar must be "solved" as exactly one of the constraints. e.g.:

S = TypeVar("S", str, bytes)

def concat(s1: S, s2: S) -> S:
    return s1 + s2

concat("foo", "bar")  # valid, S resolves to "str"
concat(b"foo", b"bar")  # valid, S resolves to "bytes"

class MyStr(str):
    ...
class ColorChannel(StrEnum):  # StrEnum inherits from Enum and str
    red = "red"
    green = "green"
    blue = "blue"

concat(MyStr(), ColoChannel.red)  # valid, S resolves to "str"
concat(MyStr(), MyStr)  # valid, S resolves to "str" (NOT MyStr)

concat("foo", b"bar")  # invalid, S can't resolve to either str or bytes

x1: str | bytes
x2: str | bytes
concat(x1, x2)  # invalid for the same reason

See more examples here #type-hinting message

[grim pumice]

but i think it's worth noting that i've been in the top 1% of typing adopters in my peer group for ~5 years and I still didn't know this.

[trim tangle]

There's some information in the first section of https://typing.readthedocs.io/en/latest/spec/generics.html#introduction

[tranquil turtle]

https://typing.readthedocs.io/en/latest/reference/generics.html#type-variables-with-constraints
https://typing.readthedocs.io/en/latest/reference/generics.html#type-variables-with-upper-bounds

[grim pumice]

okay, but let's consider relaxing must/exactly-one. what would go wrong?

[trim tangle]

oh yeah that

[trim tangle]

What do you mean?

[grim pumice]

you said "must ... exactly one ...". is that necessarily true or can we amend it to produce a more usable system

[trim tangle]

In this case concat must not be called with a mix of str and bytes. You can only add str + str or bytes + bytes, that's why a constrained typevar is the right choice here.

[grim pumice]

sure. and in my case parwse_qsl is never called with a mix

they're trivially fully collated, since there's just one arg

[trim tangle]

Yeah, it's probably an issue with how parse_qsl is typed and not with typevars in general

[tranquil turtle]

that is why it is not good to use constrained typevar for this one arg

[grim pumice]

But we'd have the same problem anywhere else -- even the "right" usages -- of AnyStr|None, no?

[trim tangle]

It might be better to use an overload here. Or introduce a new typevar, like ```py
AnyStrButCool = TypeVar("AnyStrButCool", str, bytes, str | bytes)

[tranquil turtle]

def concat(s1: S, s2: S) -> S:
    return s1 + s2

x: str | bytes
y: str | bytes
concat(x, y) # this is invalid

[grim pumice]

the proposal was to treat constrained typevars as equivalent to overloads. Is that not a good change?

[tranquil turtle]

T = TypeVar('T', bound=str | bytes)
def parse_qsl(
    qs: T | None,
    ... # same
) -> list[tuple[T, T]]: ...
``` this should fix the problem

[grim pumice]

fix in the sense of avoid, sure

but not in the sense of eliminate

[trim tangle]

This is invalid though. If I do parse_qsl(MyCoolStr()), i will get back list[tuple[MyCoolStr, MyCoolStr]] which is wrong (they will be normal strings at runtime)

[grim pumice]

tha'ts a wrong interpretation of substituting that typevar

[trim tangle]

That sounds like a good idea to me, if the constrained typevar only appears once in the parameters

[grim pumice]

correct would be list[tuple[AnyStr, Anystr]] [ AnyStr := (str|bytes) ] -> list[tuple[str, str] | tuple[bytes, bytes]]
which happens to match what an intuitive reading of the code would give

[tranquil turtle]

meh, builtin subclasses strike back

[grim pumice]

let's drop MyCoolStr for the moment? I don't see that it's necessary.

[trim tangle]

Yeah, I think if you have

T = TypeVar("T", A, B, C)

def f(t: T) -> SomeGeneric[T]:
    ...
``` then calling `f` with `A | B | C` should produce `SomeGeneric[A] | SomeGeneric[B] | SomeGeneric[C]`

(but is not allowed by type checkers currently)

[grim pumice]

yes i'm trying to consider that change

[trim tangle]

Maybe start a thread at https://discuss.python.org/c/typing/

[grim pumice]

seems like it may need to vary on co/contra/in variance?

oh but [T] and [A]] will have the same variance

so that's already naturally correct?

@trim tangle would you help me? I don't know those guys and I had enough problem getting my idea across synchronously

[trim tangle]

I think your parse_qsl example is good enough

[grim pumice]

denball doesn't agree

says it's a misuse of constraint var

but i don't see why all uses of AnyStr wouldn't hve the same issue

[trim tangle]

I think it's a mix of different things. The current typeshed of parse_qsl is not great because of this issue, and the inference of calling a function with a constrained typevar can also be improved

[trim tangle]

All functions where AnyStr is only mentioned once in the parameters will have this issue, yes

[grim pumice]

i guess i mean to ask: can we produce the same issue with an obviously-necessary use of AnyStr?

[trim tangle]

I'd say parse_qsl does want to use a constrained typevar. But it should be constrained to str, bytes, str | bytes, not just to str, bytes

[grim pumice]

why would more than one parameter make typechecking less strict?

[oblique urchin]

Is any usage of a constrained TypeVar in a function definition necessary? I think you can always replace it with overloads

[grim pumice]

OTOH isn't that true of all TypeVar constriaints?

in which case yo'ure just arguing for removcing constraints

[trim tangle]

In the case of concat above, it would be invalid to allow a union in the constraints.

[oblique urchin]

That would mean the function could return list[tuple[str | bytes, str | bytes]], which is imprecise; it only ever returns a list of strs or a bytes

[trim tangle]

that is true

[grim pumice]

agreed, that's just wrong

[trim tangle]

In user code (not a stub), an overload it less "type safe", while a constrained typevar is always synced in the implementation

[oblique urchin]

that's true

[trim tangle]

Do you think this should be allowed in general @oblique urchin? (if T only appears in one parameter)

[grim pumice]

tell me why i'm wrong? It looks like allowing constrained typevars to take on a union of its constraints, and defining its type-variable-expansion as distributive would make everything strictly better. (e.g. list[tuple[AnyStr, Anystr]] [ AnyStr := (str|bytes) ] -> list[tuple[str, str] | tuple[bytes, bytes]])

[oblique urchin]

yes, I think that makes sense to me

[grim pumice]

i don't think the only-one is relevant

[oblique urchin]

Wouldn't that make a list of mixed strs/bytes allowed?

[grim pumice]

no, look at the right-hand-side

[oblique urchin]

but what if I pass in [("a", b"b")]

if AnyStr could be solved to str|bytes, then that would get allowed

[grim pumice]

that's only true if you beleive the current spec is complete

[trim tangle]

The main use-case for a constrained typevar is this from the above:

S = TypeVar("S", str, bytes)

def concat(s1: S, s2: S) -> S:
    return s1 + s2
``` it's a feature that you're not allowed to do ```py
def f(x: str | bytes, y: str | bytes):
    concat(x, y)
``` because `concat` demands that either both arguments are `str`, or both arguments are `bytes`. A real example might be functions from `re`: `bytes` patterns only work with `bytes` inputs, and `str` patterns only work with `bytes` inputs

[grim pumice]

i had assumed otherwise

[oblique urchin]

I don't follow

[grim pumice]

it doesn't follow that just because something is correct that it would be admitted by the current typesyste

[trim tangle]

list[tuple[str, str]] | list[tuple[bytes, bytes]] would be correct here, not list[tuple[str, str] | tuple[bytes, bytes]]

(in the only-one case)

[oblique urchin]

I'm saying that something is incorrect and not admitted by the current type system, and your proposed change would make it so the incorrect thing would get allowed

[grim pumice]

ah i think the distributive expansion only applies to invariant typevars? :thinkies:

[trim tangle]

covariant maybe

[oblique urchin]

Don't think so. If you take a Sequence[AnyStr], you should be able to rely on it containing either only str or only bytes

[grim pumice]

yes.

[trim tangle]

I meant on the output side

[trim tangle]

Suppose that parse_qsl did something like ```py
_strs = [("s1", "s2")]
_bytess = [(b"b1", b"b2")]

def parse_qsl(s: AnyStr) -> list[tuple[AnyStr, AnyStr]]:
if isinstance(s, str):
return _strs
else:
return _bytess
``` then if the return type of this function got transformed to list[... | ...], you'd be able to append a tuple[str, str] to _bytess and vice versa

[grim pumice]

yes, I agree your list|list correction is correct

I don't see that the number of arguments matters for this discussion: Code sample in pyright playground

[trim tangle]

Deducing that this call is valid requires the knowledge that the two xs must have the same type

They do in this case, but I don't think the current type checkers support that kind of deduction

[grim pumice]

that's eactly what the constrained typevar is doing. we're talking past each other somehow.

i mean to say the problem we're discussing isn't somehow particular to one-argument typevars.

[trim tangle]

I meant that if you had two absolutely unrelated x: str | bytes and y: str | bytes, it would be rightfully invalid to call f(x, y)

[grim pumice]

agreed.

I feel like you're making a conclusion from that

but i'm not sure what it is

[trim tangle]

One-argument is different in that it doesn't require adding this new kind of deduction to type checkers

[oblique urchin]

From the type checker's perspective, this is the same case. It sees two arguments of type str | bytes

[grim pumice]

okay okay gotcha

[oblique urchin]

It doesn't do the tracking to know that if one of them is str, the other one must also be str

[grim pumice]

it's not trying to ionfer whether the two types are correlated

[oblique urchin]

right

[grim pumice]

and so assignable to equal typevars

so that's why you were proposing this is solvable only in the case of one-arg typevars

[oblique urchin]

Right. And even then I think only if the TypeVar is the entire param annotation

[grim pumice]

no i think i see it working with AB|None. checking...

[tranquil turtle]

combinatorial explosion goes brrr ```py
T0 = TypeVar('T0', str, bytes)
T1 = TypeVar('T1', str, bytes)
...
T9 = TypeVar('T9', str, bytes)

def f(x0: T0, x1: T1, ..., x9: T9) -> SomeGeneric[T0, T1, ..., T9]: ...

x0: str | bytes
x1: str | bytes
...
x9: str | bytes

f(x0, x1, ..., x9) # -> union of 1024 types

[trim tangle]

I think if you have py f(x: SomeGeneric[AnyStr]) -> AnyStr then calling y: str | bytes = f(x) on x: SomeGeneric[str] | SomeGeneric[bytes] should be valid (but not with SomeGeneric[str | bytes], because e.g. f might be doing the work of "".join or b"".join)

[grim pumice]

but if those are all unique, then they effectively aren't typevars

[tranquil turtle]

then you will get 1024 overloads for f

[trim tangle]

ship it

[grim pumice]

if we solve the halting problem then this problem goes away, i think

the type is just the program itself

😦

thanks for discussing. Is there anything actionable here?

[trim tangle]

I think the one-argument case like this #type-hinting message is a good suggestion

[oblique urchin]

right, so it can't be just a matter of allowing str | bytes as a solution for the Typevar, you have to do something like expanding into overloads

[trim tangle]

Yep

more generally, if this would be allowed:

if (branch):
    f(x)
else:
    f(x)
``` I think `f(x)` should be allowed

[grim pumice]

++

as a generic python-typing policy

[trim tangle]

actually, is that always true? i'm not sure

[grim pumice]

Still I feel like my chances are small of forming this proposal in a way that will be well-recieved. Will one of you help author it?

[tranquil turtle]

https://aphyr.com/posts/342-typing-the-technical-interview
https://www.richard-towers.com/2023/03/11/typescripting-the-technical-interview.html

[grim pumice]

@trim tangle pyright guy will know

[trim tangle]

wdym

[trim tangle]

enter the horrors of TypeIs

[grim pumice]

eric i think his name is? he always knows whether these things are true or false offhand.

https://github.com/erictraut

[trim tangle]

I don't think I have enough knowledge of the spec and the things that aren't specced to help with any proposal

[grim pumice]

@strange ivy ^ is this a valid proposal?

[trim tangle]

I don't think he's very active here. But you might just shoot a message to the typing thead

[tranquil turtle]

or open an issue

[grim pumice]

typing thead? issue where?

[rare scarab]

#type-hinting ?

[grim pumice]

that's where i am currently, no?

[trim tangle]

1. https://discuss.python.org/c/typing
2. discussion on https://github.com/python/typing

They stilll haven't banned this guy https://discuss.python.org/t/i-propose-a-security-audit-of-the-python3-standard-library/58948/5 so I think you're in great shape if you have a serious proposal 😄

Though the typing threads can get very heated.

[grim pumice]

well they should realize i'm right, first off

[trim tangle]

unless you're wrong

[grim pumice]

well, no. see point 1

[trim tangle]

you might have a long reply to the effect of "this is a threat to the entire architecture of <type checker> so we can't add this to the spec"

[grim pumice]

i can sympatize with that

when are we getting py4000 with types that aren't just shoehorned on top

[trim tangle]

may I suggest <different programming language>?

[grim pumice]

but python is nice. except for that bit.

[trim tangle]

Dynamic typing is one of the defining feature of Python. Its ecosystem would be entirely different with static typing

If you like gradual typing, you might want to check out Dart or Julia

[grim pumice]

no i'm not proposing static typing

but if the language was dessigned with gradual types day one, various things would be non-issues

@trim tangle didn't dart die?

... is there a less off-topic channel where we can discuss this? or am I fine?

[trim tangle]

Flutter is still pretty popular, though I'm not an expert in that

[trim tangle]

type system design seems fitting here

[grim pumice]

i like rust a lot but it's so hard 😦

[trim tangle]

Dart seemed kinda similar to Kotlin for me. I personally don't enjoy the idea of a language designed with gradual typing from the ground up

[grim pumice]

... you'd rather it was slapped in post-hoc?

[trim tangle]

I mean... I'd rather it have static types from the ground up 🙂

[grim pumice]

oh, you want to cancel the whole typing ovement

ah okay

I dream of a world with a language quite similar to python with static types

some dude had a python->C compiler...

[trim tangle]

Slapping types after the fact seems to result in a bit of a mess

[grim pumice]

pypy has one too

[trim tangle]

hmm

https://mypyc.readthedocs.io/en/latest/introduction.html

(Nuitka/Cython also works, but it doesn't consider type annotations)

[grim pumice]

nuitka was the one i was trying to remember. yep.

[languid aspen]

oop

[trim tangle]

py

[languid aspen]

@trim tangle I had to jump for a while; for some more context, I'm using this to write a parser and a bunch of parser combinators, this is what the actual class looks like: ```py
Input = TypeVar("Input")
Output = TypeVar("Output")
ParseResult: TypeAlias = "Result[Input, Output] | Error"
ParseFunction: TypeAlias = Callable[[Input], "ParseResult[Input, Output]"]
BindOutput = TypeVar("BindOutput")
MappedOutput = TypeVar("MappedOutput")

class Parser(Generic[Input, Output]):
def init(self, parser_fn: ParseFunction[Input, Output]) -> None:
self.parser_fn = parser_fn

def parse(self, inp: Input) -> ParseResult[Input, Output]:
    return self.parser_fn(inp)

def bind(
    self,
    binder: Callable[[Output], Parser[Input, BindOutput]],
) -> Parser[Input, BindOutput]:
    def parser_fn(inp: Input) -> ParseResult[Input, BindOutput]:
        result = self.parse(inp)
        if isinstance(result, Error):
            return result

        bound_parser = binder(result.output)
        return bound_parser.parse(result.input)

    return Parser(parser_fn)

def map(self, map_fn: Callable[[Output], MappedOutput]) -> Parser[Input, MappedOutput]:
    return self.bind(lambda res: pint.primitives.result(map_fn(res)))

and the type error issue is with map

the whole types don't match thing

I'm not exactly sure what I should do to resolve it, I could make all Inputs just be bound to like, a Sequence or something

[trim tangle]

What is pint.primitives.result?

[languid aspen]

oh, it's basically just the test function from earlier: ```py
def result(value: T) -> Parser[Sequence[T], T]:
def parser_fn(inp: Sequence[T]) -> "ParseResult[Sequence[T], T]":
return Result(inp, value)

return Parser(parser_fn)

[trim tangle]

I'm not sure why this requires a sequence at all

can you show examples of how bind and map are supposed to be used?

[languid aspen]

I guess it was just for consistency, most of the primitive parsers will only operate on sequence types.

[languid aspen]

sure, this is a parser defined with bind: ```py
def just(expected: T) -> Parser[Sequence[T], T]:
return take_any().bind(lambda c: result(c) if c == expected else fail("Unexpected item"))

take_any just takes one of anything from the input sequence

[languid aspen]

result doesn't need one, but most every other parser does.

[trim tangle]

You might want to rebrand your Parser as Parser[T, Output] and use Sequence[T] instead of Input

[languid aspen]

I see

[trim tangle]

Also, instead of Sequence you should probably define a type with the stuff that you actually need, like ```py
class InputT:
def getitem(self, index: int, /) -> T: ...
def len(self) -> int: ...

[languid aspen]

oh that might be better, thanks

[viscid spire]

What the skull

oh btw can't you omit the [T] of the Protocol when using 3.12 syntax?

[trim tangle]

not sure

[viscid spire]

I thought I read somewhere that you should

No source sorry

[trim tangle]

https://tenor.com/view/armstrong-made-it-up-gif-25653819

[viscid spire]

most opinions are like that it seems

ppl don't base their opinions on enough fact

[languid aspen]

sooo ```py
class Parser(Generic[T, Output]):
def init(self, parser_fn: ParseFunction[Sequence[T], Output]):
blah blah

[trim tangle]

Yep. If the parsers are supposed to work on arbitrary sequences

[languid aspen]

I see, gotcha, thanks.

[trim tangle]

If you only need seq[i] and len(seq), I'd recommend this though

[languid aspen]

yeah I'll look into that

if I refactor Foo from earlier to instead look like ```py
C = TypeVar("C")
D = TypeVar("D")
E = TypeVar("E")
F = TypeVar("F")

class Foo(Generic[C, D]):
def init(self, fn: Callable[[Sequence[C]], tuple[Sequence[C], D]]):
self.fn = fn

def bar(self, fn: Callable[[D], Foo[Sequence[C], E]]) -> Foo[Sequence[C], E]: ...

def baz(self, fn: Callable[[D], F]) -> Foo[Sequence[C], F]:
    return self.bar(lambda res: test(fn(res)))

I still getpy
Argument of type "(res: D@Foo) -> Foo[Sequence[C@Foo], Sequence[C@Foo]]" cannot be assigned to parameter "fn" of type "(D@Foo) -> Foo[Sequence[C@Foo], E@bar]" in function "bar"
Type "(res: D@Foo) -> Foo[Sequence[C@Foo], Sequence[C@Foo]]" is incompatible with type "(D@Foo) -> Foo[Sequence[C@Foo], E@bar]"
Function return type "Foo[Sequence[C@Foo], Sequence[C@Foo]]" is incompatible with type "Foo[Sequence[C@Foo], E@bar]"
"Foo[Sequence[C@Foo], Sequence[C@Foo]]" is incompatible with "Foo[Sequence[C@Foo], E@bar]"
Type parameter "D@Foo" is invariant, but "Sequence[C@Foo]" is not the same as "E@bar"

hrm, maybe

oh nevermind, I'm stupid

I see the issue

what I've got now is ```py
class Foo(Generic[C, D]):
def init(self, fn: Callable[[Sequence[C]], tuple[Sequence[C], D]]):
self.fn = fn

def bar(self, fn: Callable[[D], Foo[C, E]]) -> Foo[C, E]: ...

def baz(self, fn: Callable[[D], F]) -> Foo[C, F]:
    def mapper(res: D) -> Foo[C, F]:
        mapped = fn(res)
        return test(mapped)

    return self.bar(mapper)

``` but now the problem is with the definition of test

[forest vessel]

I think this is not generally true. To take an example based on the constrained-typevars case:

x: str | bytes
y: str | bytes
if isinstance(x, str) and isinstance(y, str):
    concat(x, y)
else:
    concat(x, y)

There is a matrix of four possible type combinations for x and y; the if/else here narrows those four possibilities down to only two possibilities ((str, str) and (bytes, bytes)), which happen to be the possibilities for which the concat call is valid. A plain concat(x, y) without the if/else should be a type error.

[trim tangle]

The concat(x, y) in the else branch would not be allowed

[languid aspen]

so I think I've been going about how test/result is typed wrong; thinking more about it now, it should be something like def result(value: Z) -> Parser[object, Z], since it makes a parser that returns any object, and it doesn't matter what the input is.

so maybe ```py
def test(value: Z) -> Foo[object, Z]:
def inner(inp: Sequence[object]) -> tuple[Sequence[object], Z]:
return (inp, value)

return Foo(inner)

[trim tangle]

I meant that something is allowed in both branches, it doesn't make logical sense that it is not allowed without a branch

[languid aspen]

although pyright doesn't seem to like that either ```py
Expression of type "Foo[object, F@baz]" is incompatible with return type "Foo[C@Foo, F@baz]"
"Foo[object, F@baz]" is incompatible with "Foo[C@Foo, F@baz]"
Type parameter "C@Foo" is invariant, but "object" is not the same as "C@Foo"

[trim tangle]

Well, a Parser[object, Z] is definitely not a Parser[C, Z]

[languid aspen]

right

I'm not sure how to type that, though

since a typevar can't only appear once in the definition can it.

[forest vessel]

you're right, never mind 🙂

[trim tangle]

was actually thinking whether there's some usage of TypeIs or whatever where this would be formally violated

or TypeGuard or whatever

well... that might be a bad TypeIs implementation

[oblique urchin]

for TypeGuard this property should always hold, since the type of the variable in the else branch is the same as if you never used TypeGuard

for TypeIs, it doesn't hold in the constrained TypeVar case

[forest vessel]

it doesn't in the current type system, but I think it's true that in principle it should hold

[languid aspen]

eh, I guess I could do ```py
def test(value: Z) -> Foo[Any, Z]:
def inner(inp: Sequence[Any]) -> tuple[Sequence[Any], Z]:
return (inp, value)

return Foo(inner)

and py
def baz(self, fn: Callable[[D], F]) -> Foo[C, F]:
def mapper(res: D) -> Foo[C, F]:
mapped = fn(res)
p: Foo[C, F] = test(mapped)
return p

    return self.bar(mapper)

[grim pumice]

Do these need to be incompatible? Intuitively they are compatible, at least.

from typing import NamedTuple
class _NetlocResultMixinBase(object):
    @property
    def netloc(self) -> str:
        raise NotImplementedError
class _ParseResultBase(NamedTuple):
    netloc: str
class ParseResult(_ParseResultBase, _NetlocResultMixinBase):
    # └╴E  Base classes for class "ParseResult" define variable "netloc" in incompatible way Pyright (reportIncompatibleVariableOverride) [10, 7]
    __slots__ = ()

The @property is not settable, but neither is the namedtuple attr.

related but separately, is there a cleaner way to indicate netloc: str in the mixin class?

[tranquil turtle]

cant you write netloc: str in mixin body?

[undone saffron]

If you have an unsound TypeIs function or LSP violations somewhere that are also not currently enforced, you could create a situation where that's the case.

Generally speaking

if f(T) is always okay
then narrowing on T shouldn't effect if f(T) is okay.

If you have a case where it does, you don't have one of f, T, or sound narrowing

[trim tangle]

Yeah, that makes sense. This seems like a basic property that should always hold unless there are logic holes

Btw @grim pumice eric replied in the thread and said that the typeshed is misusing a constrained typevar

Yes, this is a blob pain moment for sure

I personally don't like an overload here because you're losing the implementation type safety that you had with the typavar. But oh well, these are not real type hints anyway

[undone saffron]

I really don't like that the typeshed exists to begin with (if it's not good enough for standard lib, then why should anyone use it?), but if it's going to, it having incorrect types isn't helpful.

seems like just using gradual typing as gradual typing (leave it Any till it can be typed correctly) would have been better than the typeshed's current state, but there's some bias there in the kinds of issues I've run into as a result of it

[trim tangle]

I think we are at polar opposites philosophically. I like TypeScript, which is not sound, it's barely even coherent. But it lets you express a lot of stuff

(like a type level brainfuck interpreter, or converting snake_case to camelCase)

[undone saffron]

Oh, I'd love for more expressibility in python as well, but I don't like where we get a mix of false positives and negatives due to incorrect types

false negatives constrained to only where gradual typing exists is a very nice thing

and I think these "correct for most people" types prevents some features needed for better expressibility from gaining traction

doesn't need to come with any increase to false positives either, simply not placing a type there until you have a correct one lets gradual typing do its job

[trim tangle]

You might be able to sell these typeshed modifications to basedpyright (iirc pyright just ships its own typeshed?)

[undone saffron]

doesn't need additional support, both pyright and basedpyright support the typeshedPath configuration option

[trim tangle]

Wait no, pylance does. Or maybe I'm confused

[undone saffron]

to override typeshed

I've actually started on redoing the typeshed for just the standard library based on those principles for my own use (as well as how it interacts with another project)

[trim tangle]

basedtypeshed

[tranquil ledge]

Can’t wait for basedpython.

[fervent girder]

I wonder, where and how do you use type hinting

It looks hard

[cinder bone]

It's not (most of the time)
https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html

[languid aspen]

is there any nice way I can type hint a tuple of an arbitrary number of callables, all of which are typed such that Callable[[str], T] where T is something different for every callable?

[cinder bone]

On another note, is it possible to achieve something like types.MethodType[MethodClass]
My actual use case

P = ParamSpec("P")
M = TypeVar("M", bound=Mobject)


def is_mobject_method(method: Callable[..., Any]) -> TypeIs[types.MethodType]:
    if not inspect.ismethod(method):
        return False
    mobject = method.__self__
    return isinstance(mobject, Mobject)


def always(method: Callable[Concatenate[M, P], object], *args: P.args, **kwargs: P.kwargs) -> M:
    if not is_mobject_method(method):
        raise ValueError("always must take a method of a Mobject")
    mobject = cast(M, method.__self__)  # workaround
    func = method.__func__
    mobject.add_updater(lambda m: func(m, *args, **kwargs))
    return mobject

[cinder bone]

if you don't care about the value of T just replace it with object

Iterable[Callable[[str], object]]

[languid aspen]

oh right, I suppose that'd work. Thanks.

[cinder bone]

I suppose I could do something like

class MobjectMethod(Protocol):
    __self__: Mobject

but that feels like overkill

[fervent girder]

Ok, but where and why, do you use it

[cinder bone]

everywhere you can

[fervent girder]

I don't understand a single line of code

[cinder bone]

there will be some cases where python is too dynamic to type, just don't use it there

[fervent girder]

You are too smart for me

[cinder bone]

#type-hinting message
Maybe that will help

[viscid spire]

why is object the better choice here, instead of Any?

[cinder bone]

https://mypy.readthedocs.io/en/stable/dynamic_typing.html#any-vs-object

[viscid spire]

I guess it depends on how they want to treat the callables (?)

[cinder bone]

well yeah
In general Any is supposed to be an escape hatch for when stuff is too dynamic for mypy/pyright to understand

You're supposed to use object when you don't care about the return value of a function for example

But if you have the function return something where you know for sure it has certain properties (that mypy/pyright cannot check with e.g. a Protocol or something) I would use Any

[cinder bone]

bump btw.

[rustic gull]

guys pyright or mypy which one would you recomment for type checking?

[fervent girder]

Eyes

Use eyes

[rare scarab]

why not both? mypy for cli, pyright for vscode

mypy also has a vscode plugin

[rustic gull]

ohh thanks

[rustic gull]

pyright can be used with pre-commit and pyproject as well

[trim tangle]

Why not both? because they will fight each other

it's like formatting the code with black locally and with ruff in CI

[cinder bone]

ruff better

[trim tangle]

hey, you're here as well

[cinder bone]

lol hi

[viscid spire]

Rust is a Scala copy, imagine using a tool written in a FAKE (/j)

[stiff acorn]

mypy simply because I can declare it as a dev dependency in my project

[rough sluice]

Star / Wildcard imports

Wildcard imports are import statements in the form from <module_name> import *. What imports like these do is that they import everything [1] from the module into the current module's namespace [2]. This allows you to use names defined in the imported module without prefixing the module's name.

Example:

>>> from math import *
>>> sin(pi / 2)
1.0

This is discouraged, for various reasons:

Example:

>>> from custom_sin import sin
>>> from math import *
>>> sin(pi / 2)  # uses sin from math rather than your custom sin

• Potential namespace collision. Names defined from a previous import might get shadowed by a wildcard import.
• Causes ambiguity. From the example, it is unclear which sin function is actually being used. From the Zen of Python [3]: Explicit is better than implicit.
• Makes import order significant, which they shouldn't. Certain IDE's sort import functionality may end up breaking code due to namespace collision.

How should you import?

• Import the module under the module's namespace (Only import the name of the module, and names defined in the module can be used by prefixing the module's name)

>>> import math
>>> math.sin(math.pi / 2)

• Explicitly import certain names from the module

>>> from math import sin, pi
>>> sin(pi / 2)

Conclusion: Namespaces are one honking great idea -- let's do more of those! [3]

[1] If the module defines the variable __all__, the names defined in __all__ will get imported by the wildcard import, otherwise all the names in the module get imported (except for names with a leading underscore)
[2] Namespaces and scopes
[3] Zen of Python

[dull lance]

can't you also do that with pyright?

[cinder bone]

From what I've seen mypy as a dev dependency is much more common

I personally use both - pyright for lsp and mypy for CI

[stiff acorn]

There's a third party pyright pypi package that installs node and a fork called basedpyright

I don't really know much about either

It's just easier for me to use the official mypy package

[fervent girder]

is it Manim?

animation engine?

[cinder bone]

Yeah

[ashen remnant]

Probably a basic question:

    def __init__() -> None:
      # Omitting
    
    def dist(self, point: Point) -> float:
      # Omitting ```

I get an error with the following code stub:

``NameError: name 'Point' is not defined.``

How do I resolve this? Dist should take in an object of type: Point because its going to calculate the distance between 2 points.

[oblique urchin]

put the name in quotes ("Point") or add from __future__ import annotations, or use Python 3.14

[ashen remnant]

That worked, thanks a ton!

[grim pumice]

I recommend setting isort to add that line to all your python ^

[oblique urchin]

I don't. It will be deprecated.

[cinder bone]

By PEP 649 or by python 3.14?

[oblique urchin]

By PEP 649 and 749 in Python 3.14.

[grim pumice]

help me de-conflict these base classes?
https://paste.pythondiscord.com/VAHA

@oblique urchin ah TIL! so we're edging back closer to dependent types

[oblique urchin]

what

what makes you say that?

[grim pumice]

there's some hope that x: Array[int, 3] will do something useful, someday

since it's just python code once more

[oblique urchin]

that's very remote from what's actually changing in Python 3.14

[grim pumice]

yes nevermind me 🙂

request above?

[oblique urchin]

The issue is that pyright doesn't like that it's a property in one case and a namedtuple attribute in the other. No good solution comes to mind immediately.

[grim pumice]

same 😦
I agree they're incompatible inasmuch as netloc: AnyStr is settable
except that namedtuple attributes aren't settable...
is pyright incorrect to say they're incompatible at that point?

perhaps it just needs to learn that namedtuple attrs have no setter?

[oblique urchin]

no, I think it still sees the problem as being that if you access on the class (_NetlocResultMixinBase.netloc) you get a property object

but on the namedtuple base class you don't

[grim pumice]

I believe i recall that being a "fixed" problem in some other equivalent use case. I'll try to find it...

but i believe >90% of pre-typing python programmers would call those base classes compatible no?

[oblique urchin]

pyright is very strict here, maybe unnecessarily so

[grim pumice]

i do have "strict" turned on, so perhaps it's self-inflicted =X

pyright has no complaint here:

# pyright: strict
class A:
    @property
    def i(self) -> int:
        raise NotImplementedError
class B: i = 3
class C(A, B): pass

https://github.com/microsoft/pyright/discussions/8679

[cinder bone]

Ah sorry to distract, but I had some time to go back to a discussion about a Proxy type, and I was wondering what the next steps were. Should I paste a draft PEP in the discussion, or just wait for more feedback/interest?

[grim pumice]

Does this error make sense to you? These seem compatible, to me.

Argument of type "tuple[AnyStr@urlunsplit, ...]" cannot be assigned to parameter "args" of type "tuple[str, ...] | tuple[bytes, ...]" in function "_coerce_args"

[tranquil turtle]

that is exactly the same error you had several days ago

[grim pumice]

really? my bad.

no, that error was Argument of type "str | bytes" cannot be assigned to "AnyStr@parse_qsl | None" Pyright (reportArgumentType)

this is the other way round

the workaround has the same shape though...

    if isinstance(components, ParseResult):
        (scheme, netloc, url, params, query, fragment), _coerce_result = _coerce_args(components)
    else:  # ParseResultBytes
        (scheme, netloc, url, params, query, fragment), _coerce_result = _coerce_args(components)

[cinder bone]

I think they're different because for the argument you can have tuple[str, bytes] but the parameter only accepts tuples of only strings or tuples of only bytes

[grim pumice]

tuple[AnyStr, ...] is homogeneous too

[rare scarab]

Would you really consider list[Any] to be homogeneous?

[oblique urchin]

that's very different

my intuition is also that what @grim pumice posted should be allowed

[rare scarab]

!d typing.AnyStr

[rough sluice]

typing.AnyStr

typing.AnyStr```
A [constrained type variable](https://docs.python.org/3/library/typing.html#typing-constrained-typevar).

Definition:

```py
AnyStr = TypeVar('AnyStr', str, bytes)
```  `AnyStr` is meant to be used for functions that may accept [`str`](https://docs.python.org/3/library/stdtypes.html#str) or [`bytes`](https://docs.python.org/3/library/stdtypes.html#bytes) arguments but cannot allow the two to mix.

For example...

[rare scarab]

Just checking that it was a typevar instead of a union

[pure prawn]

unalivejoy

[oblique urchin]

The choice of the name AnyStr is questionable

[pure prawn]

did you know that whenever ur hand is itchy money is coming towards you

[dry portal]

!mute 1261923006754721865 1d You've been posting off-topic nonsense for a bit. Stop, read our #rules and #code-of-conduct

[rough sluice]

:incoming_envelope: :ok_hand: applied timeout to @pure prawn until <t:1723066578:f> (1 day).

[trim tangle]

erictraut argues in discuss that this is not a valid use of constrained typevars

[oblique urchin]

That's a slightly different case, no?

[trim tangle]

seems pretty similar

[trim tangle]

Can you show the type of _coerce_args?

[oblique urchin]

it's about passing AnyStr to a union, while Eric's post is about passing a union to AnyStr

[trim tangle]

ah, yes

[grim pumice]

@oblique urchin did you understand erics reply? I didn't.
https://discuss.python.org/t/proposal-relax-un-correlated-constrained-typevars/59658/3

[rustic gull]

hello

I have this issue that keeps appearing

when I say

for i in x:
  b = d[i]

when I then say

b

it keeps saying

"b" is possibly unboundPylancereportPossiblyUnboundVariable

is there any way to tell it that x is never an empty iterable?

bro put it 1000 times better than me https://github.com/microsoft/pyright/discussions/2033

[soft matrix]

id do something like

b = None
for i in x:
 ...
assert b is not None

[viscid spire]

or just give a default value to b that's "real", and thus no assertion (assertion in production is bad anyways)

[viscid spire]

wait why do you want to do this anyways

[soft matrix]

yeah fwiw

b = d[collections.deque(x, maxlen=1)[0])```

[grim pumice]

I believe I've found a bug in typeshed?
This definition throws away the params to the callable:
https://github.com/python/typeshed/blob/main/stdlib/functools.pyi#L69C1-L70C1

The result is that both mypy and pyright will always choose the first @overload of any lru_cache'd function.

[rough sluice]

stdlib/functools.pyi line 69

def lru_cache(maxsize: Callable[..., _T], typed: bool = False) -> _lru_cache_wrapper[_T]: ...```

[oblique urchin]

It's known but difficult to fix, there's some open issues and PRs

[grim pumice]

oh. it seemed easy to me...
Any idea why I'm getting "bound is uknown parameter to ParamSpec" ?

[oblique urchin]

ParamSpec doesn't support bounds in the type system

As for lru_cache, it's easy for functions but causes issues with methods I believe

I haven't personally looked into it a lot but people have definitely tried

[grim pumice]

but in what way is bound an uknown param?
https://docs.python.org/3/library/typing.html#typing.ParamSpec

[oblique urchin]

yeah it works at runtime but it has no specified meaning in the type system

[grim pumice]

well this is where we'd need it, if you haven't found a use case before

lru_cache can only wrap a paramspec with lower bound Hashable

currently the args and kwargs of lru-cache call are Hashable. I want to make them paramspec without losing that...

remind me why product types are hard?

[tranquil turtle]

Hashable is a fancy alias for object currently

[oblique urchin]

not quite https://pyright-play.net/?code=GYJw9gtgBAxmA28CmMAuBLMA7AzgOgEMAjGKdCABzBFSgAkCcALY5AKABMlgpgAKAB4AueoxZFkAShEVGONm34BtALqSgA

[grim pumice]

not at all? or were you thinking of mypy only

[oblique urchin]

Hashable has some problems that make it not fully safe (e.g., the fact that object is hashable)

But it's not completely useless

[grim pumice]

but object is hashable. i don't get it.

[oblique urchin]

https://pyright-play.net/?code=GYJw9gtgBAxmA28CmMAuBLMA7AzgOgEMAjGKdCABzBFSgAkCcALY5AKABMlgpgAKAB4AueoxZFkAShEVGONp25QA5oJFgiAKxSppbKAd6DJC1QG0AupKA

[grim pumice]

haha

[tranquil turtle]

all unhashable types violate LSP

[grim pumice]

this is one of those co/contravariance things... subclasses of a hashable are sometimes not hashable

sadge

but isn't that true of protocols gnerally? My subclass can break the protocol that's honored by my superclass.

[oblique urchin]

sure, and then you break LSP

[grim pumice]

language server protocol?

[oblique urchin]

the special thing about Hashable is that any class that is not hashable is necessarily a subclass of something Hashable

Liskov Substitution Principle

essentially, the idea that subclasses should be compatible with the superclass

[grim pumice]

sure.... let's pop stack back to OP

is it impossible to take what's currently defined as Hashable and make it use paramspec without removing the Hashable check?

[oblique urchin]

correct

[grim pumice]

poops. so is it worse to lose overloads or worse to lose hashability constraint

i'm currently more annoyed by losing overloads...

but also this seems to be a clear use case for the currently-undefined Paramspec bound

[oblique urchin]

yeah agree. As I recall the problem is that the ParamSpec-based solution does bad things when the decorator is applied to methods

you can search for lru_cache in typeshed issues and PRs for more

[grim pumice]

i wonder why..

oh does mypy not apply the method descriptor because it's no longer a funciton?

that's a bug if so...

pyright fixed something similarish to that just now: https://github.com/microsoft/pyright/commit/efda709120abd8c4cceaf01bb6fd0c5c3fbbe63b

okay yea this undoes exactly what I'm doing 😦
https://github.com/python/typeshed/pull/6356/files

actually this seems like my guess was right? mypy is no longer applying the method descriptor likely because it doesn't believe this object is a function (and it's not but that shouldn't matter). So it seems like they need something similar to eric's patch above?

edb/server/tenant.py:138:27: error: Missing positional argument "self" in call to "__call__" of "_lru_cache_wrapper"  [call-arg]

[tranquil turtle]

(x: int)->str is assignable to Callable[[int], str] but not other way around

[grim pumice]

sure. i'm saying i don't see the harm in typing methods as simply Callable.

it would be closer to how python actually works in practice too

@oblique urchin FWIW pyright only shows new errors that seem to be valid problems that were previously missed (in a spot check of edgedb)

we do lose one (valid) error about using lru_cache on uncacheable types

so the current situation is that I want to revert typeshed#6356 but can't due to a bug in mypy (but not pyright) 😭
how do I proceed to try to correct this situation?

[fathom osprey]

Hi. Should I be able to get a squiggly line in VSCode with Pylance, when I write:

def fun(a: str | int):
  pass

and do

fun({})  # Should I get a squiggly line here?

?

[trim tangle]

Yes

If you don't, you need to go to settings and switch the type checking mode in Pylance from "off" to "standard"

[fathom osprey]

you mean: ?

[trim tangle]

Search for "typeCheckingMode"

It should let you select between "off", "basic", "standard" and "strict"

[fathom osprey]

now I got some errors. Is there any way that it was previously done in some other way?

I mean I think I had this on previously and didn't have errors

[rare scarab]

They need to change the default.

[fathom osprey]

?

[rare scarab]

What error are you getting now?

[fathom osprey]

ohh related to pandas typing for example where I'm using it

[trim tangle]

apparently, typing pandas is an unsolved problem 🙂

[rare scarab]

is polars any more type-safe?

[trim tangle]

Not sure if I agree. Everyone using the Python extension will get the default, not everyone wants type checking. If someone does want this stuff, they can enable it

[rare scarab]

prompt to enable it so people know it exists?

[trim tangle]

Yeah, that's better

[cinder bone]

https://github.com/pandas-dev/pandas-stubs?

[grim pumice]

I have an untyped program that has a superclass with an abstract attribute (self.x is used but not intialized).
What's the most straightforward option for getting this typed?
If I simply add x:T then it complains that x is never initialized.
Adding "Protocol" as a superclass "fixes" it but it has a bunch of runtime implications that I don't want.

[grim pumice]

(separately)
I'd really like to be able to do this. Thoughts?

    args: tuple[*P.args],
    kwds: TypedDict[**P.kwargs],

[soft matrix]

Probably just type ignore and move on if you can't give it a sentinel value

[grim pumice]

I don't love "don't" as an answer to "how?" -- the premise is that I'm converting this code to be typed. Thank you though 🙂

[viscid spire]

i get this from pylance

"args" attribute of ParamSpec is valid only when used with *args parameter

I wonder if it's spec'd to not be allowed, or if it's pyright's choice

[viscid spire]

can you provide an MRE?

[oblique urchin]

That's part of the spec. You can only use .args together with .kwargs

[viscid spire]

err ok... that's not what pylance was highlighting tho

[trim tangle]

Maybe we're missing a "thing" that captures both args and kwargs

like, make Arguments(*args, **kwargs) and pass this around

though if you ignore typing, that would be pretty useless

[cinder bone]

My one annoyance with ParamSpec+Concatenate is the args and kwargs of paramspec have to be next to each other
Like this won't work

def foo(f: Callable[P, T]) -> Callable[Concatenate[P, int], T]
    def wrapper(*args: P.args, foo: int, **kwargs: P.kwargs):
        ...
    return wrapper

It's not a big deal just a tad annoying

[viscid spire]

Concatenate[P, int]?

pylance doesn't like this

from what I'm understanding, you can't put a ParamSpec before other arguments in Concatenate, because of its variadic nature

basically that int, in the Concatenate, is required to be usable as positional

whereas in your wrapper, it is keyword-only, as it comes after a *

so my guess would be that the reason you can't make that foo: int at all is because it is unexpressable in the type system

[cinder bone]

Yeah that's what I don't like

If you change it to positional and reverse the Concatenate it typechecks

[viscid spire]

and this is because Concatenate[P, int] itself is invalid

[cinder bone]

Ig it makes sense thought because whatif one of the parameters of P was kw only and called foo

[viscid spire]

if we had keyword arguments in subscription, then we could express this

[cinder bone]

It wouldn't be worth the complexity idt

ParamSpec is good enough 99.9% of the time

[viscid spire]

Concatenate[P, foo=int]

this would be the only real solution, but I think it was rejected or something

wait this doesn't even make sense

because P.kwargs comes after the foo param

[cinder bone]

Yeah

[viscid spire]

so you would then have to do...

Concatenate[P.args, foo=int, P.kwargs]

which is horrendous

[cinder bone]

Indeed

Honestly as it is ParamSpec is a genius move ngl

[stiff acorn]

Why is mypy failing on just 3.10? https://github.com/Ravencentric/archivefile/actions/runs/10333258087/job/28605398844#step:10:40

[ornate ermine]

I have a question regarding typing with Literal and dict:

COOL = Literal['a', 'b']
COOL_DICT: dict[COOL, int] = {'a':0, 'b':0} 

Gives me an error, that [str, int] does not match dict[Literal['a', 'b'], int]
How can I fix that?

[trim tangle]

who gives you an error?

[ornate ermine]

mypy sry

[trim tangle]

Works fine for me https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=77b55219dfdb9cc87d09bb7afa6e912d

What version of mypy are you using?

[ornate ermine]

i was on 1.11.0, updated to 1.11.1, now it passes

thanks for the quick feedback 🙂

[trim tangle]

mypy 1.11.0 also passes 🤔

maybe you didn't save the file. or maybe there's some stateful bug in mypy

[ornate ermine]

it autosaves before running pre-commit... hmm weird cant reproduce it now

[stiff acorn]

still lost on this, would appreciate any pointers

src/archivefile/_adapters/_zip.py:69: error: Argument "mode" to "ZipFile" has
incompatible type "str"; expected "Literal['r', 'w', 'x', 'a']"  [arg-type]
                mode=self._mode,
                     ^~~~~~~~~~
src/archivefile/_adapters/_zip.py:73: error: Unused "type: ignore" comment 
[unused-ignore]
            )  # type: ignore
            ^~~~~~~~~~~~~~~~
src/archivefile/_adapters/_zip.py:113: error: Argument "size" to
"ArchiveMember" has incompatible type "int"; expected "ByteSize"  [arg-type]
                size=zipinfo.file_size,
                     ^~~~~~~~~~~~~~~~~
src/archivefile/_adapters/_zip.py:114: error: Argument "compressed_size" to
"ArchiveMember" has incompatible type "int"; expected "ByteSize"  [arg-type]
                compressed_size=zipinfo.compress_size,
                                ^~~~~~~~~~~~~~~~~~~~~
src/archivefile/_adapters/_zip.py:125: error: Argument "size" to
"ArchiveMember" has incompatible type "int"; expected "ByteSize"  [arg-type]
                    size=zipinfo.file_size,
                         ^~~~~~~~~~~~~~~~~
src/archivefile/_adapters/_zip.py:126: error: Argument "compressed_size" to
"ArchiveMember" has incompatible type "int"; expected "ByteSize"  [arg-type]
                    compressed_size=zipinfo.compress_size,
                                    ^~~~~~~~~~~~~~~~~~~~~
src/archivefile/_adapters/_zip.py:217: error: Unused "type: ignore" comment 
[unused-ignore]
            return self._zipfile.read(name, ***  # type: ignore
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 7 errors in 1 file (checked 27 source files)

these only pop up in 3.10 but not in later versions

im using mypy 1.11.1 on all of them

[tranquil turtle]

first error is self-explanatory, others seem to be related to pydantic

[stiff acorn]

yea i get that

the issue is why are they only popping up in 3.10

and pass in 3.11

if i fix the error above

it fails on newer python

poetry run mypy .
src/archivefile/_adapters/_zip.py:73: error: Unused "type: ignore" comment 
[unused-ignore]
            )  # type: ignore
            ^~~~~~~~~~~~~~~~
Found 1 error in 1 file (checked 27 source files)

seems to be some multi line issue?

# passes on mypy 1.11.1 with Python 3.10, 3.11, 3.12
        self._zipfile = ZipFile(
            self._file,
            mode=self._mode, # type: ignore
            compression=self._compression_type,
            compresslevel=self._compression_level,
            **kwargs,
        )

# passes on mypy 1.11.1 with Python 3.11, 3.12
# fails on mypy 1.11.1 with Python 3.10
        self._zipfile = ZipFile(
            self._file,
            mode=self._mode,
            compression=self._compression_type,
            compresslevel=self._compression_level,
            **kwargs,
        ) # type: ignore

3.11+

poetry run mypy .
src/archivefile/_adapters/_zip.py:113: error: Unused "type: ignore" comment 
[unused-ignore]
                size=zipinfo.file_size, # type: ignore
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 1 error in 1 file (checked 27 source files)

3.10

poetry run mypy .
src/archivefile/_adapters/_zip.py:113: error: Argument "size" to
"ArchiveMember" has incompatible type "int"; expected "ByteSize"  [arg-type]
                size=zipinfo.file_size,
                     ^~~~~~~~~~~~~~~~~

how am i supposed to handle this when fixing one version breaks another

[trail kraken]

May be ByteSize(file_size) just to make mypy happy.

[cinder bone]

Hey guys, I was thinking about PEP 649, and thought of an example like this

from django.contrib.auth import get_user_model

def something(x: get_user_model()) -> None:
    ...

def something2(x: "get_user_model()") -> None:
    ...

print(something.__annotations__)
print(something2.__annotations__)

How would the resulting __annotations__ differ? Would this raise an error at compile time?

[oblique urchin]

the first one will call get_user_model(). the second one will return a string

[cinder bone]

So django requires the settings to be set up before get_user_model returns something - would python just ignore that error or what?

[oblique urchin]

Python just calls the function (but under PEP 649, only when __annotations__ is evaluated). It doesn't know about what that function does.

by the way, you can already try this if you compile CPython from main, I implemented it

[cinder bone]

ooh might try that

seems like a ton of fun :)

[rustic gull]

from numbers import Number
a: Number = 3

Expression of type "Literal[3]" is incompatible with declared type "Number"
  "Literal[3]" is incompatible with "Number"PylancereportAssignmentType

wtf

[restive rapids]

yeah, ints are not Number's for typecheckers (well, with the standard typeshed that is, i guess), though at runtime isinstance(3, Number) is true :d the whole numbers thing is pretty weird, i dont think it gets used much

[undone saffron]

If you want type checking to work with generalized numeric types properly, you either have to write a lot of things, or use something like: https://github.com/beartype/numerary/ where that work has already been done. The numbers module isn't type checking friendly.

[upbeat wadi]

Just wondering, is this behaviour intentional? (where the type of the global variable isn't narrowed down in the function)

[viscid spire]

You aren't calling the function at the time, so it can't know if it will be narrowed later

[grim pumice]

how would I annotate an argument that is a (unknown) subclass of NamedTuple?

[undone saffron]

tuple[Any, ...] / type[NamedTuple] depending on if you mean an instance of the type or the type itself

you can't rely on anything unique to it being a named tuple subtype with it being unknown

[grim pumice]

type[NamedTuple] is close but isn't quite it, as calling the object thinks it wants the two-argument procedural call to NamedTuple. the first argument should be the namedtuple class name
I suspect there's no way to specify non-inclusive subclass.

[undone saffron]

you could do type[tuple] if that's the intent

[grim pumice]

but these objects have _fields: tuple[str, ...]

[undone saffron]

Are you using _fields ? if not, it doesn't matter

[grim pumice]

of course.

to be clear i'm more interested in the typesystem capabilities and the right-way to do things than a quick fix to this problem

[undone saffron]

the right way to do this is probably not do it. Not all things that can be done in python should be done as part of the type system/

instead type the structure you need it to have (via a protocol), and document that the expectation is a named tuple use

[grim pumice]

I'm holding it wrong. Gotcha. let's see if anyone else has a way though

[undone saffron]

might be possible to do slightly better than that in the future if intersections come to be, cause then you just need a tuple subclass that also has _fields (And whatever else you rely on)

[oblique urchin]

I think mypy accepts : typing.NamedTuple as an annotation, though personally I'm not fond of it

[grim pumice]

type[typing.NamedTuple] is close but isn't quite it, as calling the object thinks it wants the two-argument procedural call to NamedTuple. "the first argument should be the namedtuple class name"

[undone saffron]

I had something that looked like:

class RecordLike(Protocol):
    _fields: tuple[str, ...]
    _field_defaults: dict[str, Any]
    def __call__(self, *args: Any, **kwargs: Any) -> Any:  ...

for this at one point, but I ended up going with a different design just based on type[T] -> T and documenting what T needed to provide without it being part of the type

as well as providing detailed exceptions when recieving a type the function couldnt handle

[upbeat wadi]

Makes sense, thanks

[buoyant ridge]

yeah I would've thought @grim pumice 's situation calls for a generic constrained to NamedTuple

or i suppose if the argument is the class itself, then [T: Type[NamedTuple]]? i'm not sure how to then type the return of an instance of T though

nevermind, that seems to be the same pitfall of the type system expecting the creation of a new namedtuple

[leaden ember]

I'm getting started with mypy and typing an existing codebase, but I'm getting “library types not installed” (ie., import-untyped error), for scipy.spatial, sklearn.cluster, psutil, zarr, mpi4py, tqdm, cgen, netCDF4, and cftime. I know that I can install stubs/types for a couple of these packages (namely tqdm which I’ve already found), but I can’t find any stubs for most of these including scipy and sklearn.

I don’t really want to write my own stubs for these as I feel that would be a big undertaking. I’m also surprised that sklearn and scipy don’t have these stubs (or is it only partially supported?). How are people handling missing stubs/types for scipy/sklearn and in general? I couldn’t find an option in the documentation for disabling this import-untyped error on a module basis (the defaults for disallow_untyped_calls and related settings are set to false, so I don’t think these settings will help https://mypy.readthedocs.io/en/stable/config_file.html#untyped-definitions-and-calls). It would be great to have something similar to ignore_missing_imports on a module basis but for untyped stuff

[[tool.mypy.overrides]]
module = "sklearn.spatial"
ignore_missing_imports = true

[frigid jolt]

@oblique urchin just wanted to let you know, if it's important for you, that i may have found a real scenario use case for PEP-702, https://github.com/DisnakeDev/disnake/issues/1149#issuecomment-2286030511

[cinder bone]

[mypy-tqdm]
ignore_missing_imports = True

[cinder bone]

woah PEP 702 sounds amazingly useful

[leaden ember]

Yeah, that doesn't answer my question :/ . Its about untyped imports not missing ones. Also using a pyproject.toml config

[cinder bone]

yes - and I believe that ignore_missing_imports does ignore untyped imports
And a pyproject.toml the only difference is it's tool.mypy instead of just mypy

or on a permodule basis

[tool.mypy-manim.mobject.*]
ignore_missing_imports = True

[oblique urchin]

thanks, there are plenty of use cases of PEP 702 already

[frigid jolt]

oh alright, about that issue, if you did read it do you see any other way?

[lunar dune]

type[NamedTuple] is very unsafe FWIW, because NamedTuple is actually a function at runtime -- it doesn't have any of the attributes you'd expect on an instance of type (__name__, __mro__, etc.)

[tranquil turtle]

(__name__ exists on functions (and some other callables))

[cinder bone]

yeah but it's not a class and shouldn't be treated as such

[undone saffron]

I'm aware, which is why I explained here: #type-hinting message That even though that's what the obvious and correct option should be, it's not really what should be done.

The fact that NamedTuple is a function at runtime is not actually useful to typing users IMO, but I don't want to get into another argument about how things that arent actually types at runtime create problems

could have been implemented as a type with __call__ defined

[lunar dune]

true, thanks. __mro__ and __subclasses__ are better examples here.

[leaden ember]

oh, awesome! Didn't know that ignore_missing_imports also covered untyped stuff

This is the format I settled on which works

[[tool.mypy.overrides]]
module = [
    "scipy.spatial",
    "sklearn.cluster",
    "zarr",
]
ignore_missing_imports = true

Thanks for the help!

[lunar dune]

could have been implemented as a type with __call__ defined

It actually originally was, but this was changed, as it was considered confusing that class X(NamedTuple): ... would not create a class where NamedTuple was present in X's mro. Making NamedTuple a function emphasises that it is not itself a type; it is instead a factory function for creating new types, even when you "inherit" from it. See https://github.com/python/cpython/pull/19371

Anyway, I agree with you that probably the best way to type "a thing that could be an instance of any class created by the NamedTuple factory function" is to use a protocol, but that really you should probably be trying to avoid that situation in the first place

[undone saffron]

I really disliked this change, but I'm an advocate of not having some of the special cases that have been created.

[lunar dune]

I'm aware that in general you think that the typing system as it is today has far too many special cases and that you would have made many different decisions.

For the specific case of typing.NamedTuple: In an ideal world, would you just not have made that change, and kept NamedTuple as it was prior to that PR? Or would you disallow inheriting from NamedTuple altogether, and only allow creation of typed namedtuple classes by calling typing.NamedTuple as a function?

[undone saffron]

neither!

[undone saffron]

since it's clear that NamedTuple not being in the mro yet it being a type was considered too confusing, I'd have changed the way _make_nmtuple functions and insert the type into the mro.

I don't personally think it was unsolvable with documentation rather than a code change, but... that's much more subjective.

[lunar dune]

I see, so you would have had typing.NamedTuple work completely differently to collections.namedtuple? It was conceived as a drop-in replacement that would work with type checkers

[undone saffron]

yeah, but I think the moment that it was shown that that was confusing and people had other expectations....

another viable path on this would have been more similar to how dataclasses actually work and just say type checkers need to understand the fucntional use of collections.namedtuple

(dataclass_transform being the special casing in type checkers associated)

[lunar dune]

true, most people probably don't think of inheriting from NamedTuple as "now I shall create a custom tuple subclass with convenient property accessors that type checkers will understand"

[undone saffron]

I'm generally okay with things that arent perfect drop-in replacements, but for the vast majority of users function as one so long as the differences exist for good reason and are documented.

[lunar dune]

They do, as best they can (or at least, mypy does). You just end up with a lot of Anys if you can't type the fields: https://mypy-play.net/?mypy=latest&python=3.12&gist=d64bcaef648dd1bfa0587445cb3bcc28

[undone saffron]

oh, I think that's better than it was in the past there at least.

I forget the last time I had an issue with that though, I think it dates back to when I couldnt assume typing.NamedTuple would even exist (library supporting multiple python versions, but a long time ago)

[lunar dune]

They do also understand the functional use of typing.NamedTuple, though (or, again, at least mypy does), and then you end up with much less Any being inferred everywhere -- which is why I was wondering if "only allow the functional form of typing.NamedTuple" would appeal to you there

[undone saffron]

dont think so, I think the class based definition is definitely more intuitive for how most people intend to use it

I've been working on something that's a bit of a "What if" situation for python. "what if we could redesign the syntax and type behavior knowing what we do now", someone whose seen more of it thought it was kindof like typescript is for js.

The interesting thing with it from my perspective is how little I've deviated on what users need to do. Unfortunately, I think the changes overall are definitely too disruptive to do it in python right now. I've come across a few things that are typing soundness issues that are rectifiable with breaking, and a few I think could be rectified without breaking but need to look into further.

mostly as a bit of an exploration tool, potentially actually usable like typescript in the future, but it may never reach that point and people shouldnt be hoping it will right now

an interesting one that I may write a pep for later is an opaque version of Any

where the caller may provide anything without regard for variance, but the consumer of the type must verify the type before using

would actually assist with a few cases like above stuff with paramspec where people want to be able to accept anything, but not have the type checking disabled on their use

https://discuss.python.org/t/allow-p-args-as-a-type-annotation/60419/4 being a concrete place such an opaque any type could assist with

[warm salmon]

What is type-hinting about? any small words?

[trim tangle]

small words?

If you want an introduction, I have a WIP tutorial https://decorator-factory.github.io/typing-tips/main-tutorial/0-start-here/

size of words may vary

[solemn sapphire]

class Base:
    def __init__(self, attr: int | str) -> None:
        self.attr = attr

class Derived(Base):
    def __init__(self, attr) -> None:
        super().__init__(attr)
        if isinstance(self.attr, str):
            raise TypeError("can't be str here")

    def add1(self) -> int:
        return self.attr + 1 # type error adding to a str

What's a nice way to help mypy narrow self.attr to be an int in Derived?

One way I know is to do assert(isinstance(self.attr, int)) before using self.attr but that's annoying to add on all methods of Derived

[trim tangle]

I don't think there's a way to specify this, because this probably violates substitutability

(assuming this attribute is writable)

[solemn sapphire]

Oh the attribute is not writeable

by substitutability, you mean this, https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides ?

[trim tangle]

Yes, something like this

[trim tangle]

class Base:
    def __init__(self, attr: int | str) -> None:
        self._attr = attr

class Derived(Base):
    def __init__(self, attr: int) -> None:
        super().__init__(attr)
        if isinstance(self._attr, str):
            raise TypeError("can't be str here")
        self._attr = self._attr

    def foo(self) -> int:
        return self._attr      

this does work in pyright, but not in mypy

Do you have other methods that use attr?

[soft matrix]

Could make it bound with a default type var over the union

[solemn sapphire]

ye

[trim tangle]

If this is your own class, I'd consider rethinking the design to avoid subclassing (in general)

It seems like there's no way to make mypy accept this (besides asserts or # type: ignore)

[solemn sapphire]

I'm sorry, I do not understand this 😓

[trim tangle]

class Base[Attr: (str, int, str | int)]:
    def __init__(self, attr: Attr) -> None:
        self._attr = attr

[solemn sapphire]

This is homework that I'm unnecessarily fancifying xD
I have a Cipher base, with multiple implementations that derive from it that provide do_encode and do_decode methods. The base provides methods to convert alphabets to numbers, printing it all nicely and some more things

The problem is the ceaser cipher, that takes a stupid numeric key rather than a string, everything else (atleast the ones required to be done) take a string as a key

[trim tangle]

Can you post the code?

!paste

[rough sluice]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[solemn sapphire]

sure!

from typing import Self, Iterable, TypeAlias

Chars: TypeAlias = Iterable[str]
Nums: TypeAlias = Iterable[int]

def to_numbers(message: Chars) -> Nums:
    return map(lambda c: ord(c) - ord("a"), message)


def to_alphabets(message: Nums) -> Chars:
    return map(lambda c: chr(c + ord("a")), message)


class Cipher:
    def __init__(self) -> None:
        self.key: int | Nums
        self.message: Nums

    def set_key(self, key: int | str) -> Self:
        self.key = list(to_numbers(key.lower())) if isinstance(key, str) else key
        return self

    def set_message(self, message: str) -> Self:
        self.message = list(to_numbers(message.lower()))
        return self

    def encrypt(self) -> str:
        return "".join(to_alphabets(self.do_encrypt()))

    def decrypt(self) -> str:
        return "".join(to_alphabets(self.do_decrypt()))

    def do_encrypt(self) -> Nums:
        raise NotImplementedError

    def do_decrypt(self) -> Nums:
        raise NotImplementedError


class Caeser(Cipher):
    def __init__(self) -> None:
        if not isinstance(self.key, int):
            raise TypeError("Cannot use non int key for Ceaser cipher")

    def do_encrypt(self) -> Nums:
        return map(lambda c: (c + self.key) % 26, self.message)

    def do_decrypt(self) -> Nums:
        return map(lambda c: (c - self.key) % 26, self.message)

Usage would be something like

ccipher = Caeser().set_key(5)
encrypted = ccipher.set_message("Hello").encrypt()
print(encrypted)
decrypted = ccipher.set_message(encrypted).decrypt()
print(decrypted)

[trim tangle]

In this case making key: int would indeed break substitutability, because set_key would no longer accept a str

[solemn sapphire]

Oh

[trim tangle]

Do you have an example of how you'd use Cipher and Caesar?

[solemn sapphire]

Caeser would be used like

ccipher = Caeser().set_key(5)
encrypted = ccipher.set_message("Hello").encrypt()
print(encrypted)
decrypted = ccipher.set_message(encrypted).decrypt()
print(decrypted)

[trim tangle]

do you have a different Cipher?

[solemn sapphire]

lemme make one rq

from itertools import cycle, starmap

class Vignere(Cipher):
    def do_encrypt(self) -> Nums:
        return starmap(lambda a, b: (a + b) % 26, zip(self.message, cycle(self.key)))

    def do_decrypt(self) -> Nums:
        return starmap(lambda a, b: (a - b) % 26, zip(self.message, cycle(self.key)))

[trim tangle]

but does your program need a second cipher?

[solemn sapphire]

Ye, I need to implement a couple (playfair, railfence) and I'd like to not duplicate code around

I'll be honest I didn't think very deeply into how I should design this, my idea was to not repeat code

maybe it should just be free functions

[trim tangle]

The thing is, different ciphers have different requirements that this doesn't fully account for. For example, different ciphers are configured differently.

• Caesar cipher is configured with a number (which must be between 0 and 25)
• Your variant of Vigenere is configured with a string containing lowercase latin letters
• Playfair will need a string of exactly 25 lowercase latin letters

and, for example. some other cipher will need a key and the number of rounds

do you have the code where you select the different ciphers?

[solemn sapphire]

What I'm thinking is, cipher implementations take in Iterable[int] and give out Iterable[int] for their encrypt and decrypt methods.
Rounds I could implement as a separate method on whatever cipher requires it with

def rounds(self, rounds: int) -> Iterable[int]:
  message = self.message
  for _ in range(rounds):
    message = self.encrypt(message)
  return message

The constrains could be checked in the constructor of the implementing class?

Hmm, now that I think about it, maybe I should compose A particular cipher implementation in a Cipher class, and that implementation needs to follow a protocol which is has two methods, one for encrypting and one for decrypting

[cinder bone]

(hello, nice to see another Manim dev around here)

[solemn sapphire]

Its not very nice code, I am open to rewriting everything to make it type safe

[solemn sapphire]

xD
Its been quite a while since I worked on manim
Hello!

[trim tangle]

Well, eventually your code structure will terminate in something like: ```py
def main():
user_input = input()
if user_input == "caesar":
# caesar-specific code
elif user_input == "vigenere":
# vigenere-specific code
elif user_input == "playfair":
# you get the ide
else:
# print error message

[solemn sapphire]

ye that's what it is currently

[trim tangle]

you only need this kind of polymorphism when you can reasonably handle elements of different concrete types with the same code. In this case it's not really possible, because they require different shapes for configuration.

As in, you can't write a function that takes an unknown cipher and asks the user to configure it (currently)

You might want to use polymorphism for the part that is possible to handle generically, which is the do_encrypt and do_decrypt methods.

So you might have something like: ```py
def main():
user_input = input()
if user_input == "caesar":
# ask user for Caesar-specific configuration
cipher = Caesar(...)
elif user_input == "vigenere":
# ask user for Vigenere-specific configuration
cipher = Vigenere(...)
elif user_input == "playfair":
...

action = input()
if action == "encrypt":
    message = string_to_nums(input())
    print(nums_to_string(chiper.encrypt(message)))
elif action == "decrypt"
    message = string_to_nums(input())
    print(nums_to_string(chiper.decrypt(message)))
elif ...:

[solemn sapphire]

Ah! that makes sense
I didn't really have a deep understanding of when and how to use polymorphism, this gives more clarity, Thank you so much! I shall structure my program like this then

[trim tangle]

In this case the Caesar and Vigenere classes don't need a shared base class, they just need to have encrypt and decrypt methods with the appropriate contract

[solemn sapphire]

Indeed, they just need to adhere to some protocol

[trim tangle]

Well, you might need to make a Protocol to make mypy understand this. Pyright is smarter about this

[solemn sapphire]

pyright makes my editor lag a lot :/

hence I'm using mypy

[tacit sparrow]

Hi! Any advice on typing for version agnostic API? See example below.

The only thing I'm missing currently - is it possible to make get_agnostic_object to share somehow signature with the objects it's instantiating? I mean so that someone using get_agnostic_object("v2") would be able to see the arguments of Object_V2 and Object_V3?

class Object_V2:
    def __init__(self, name: str): ...
class Object_V3:
    def __init__(self, name: str): ...

Object_Agnostic = Object_V2 | Object_V3

# Need to be able to use it in typing.
def agnostic_method(obj: Object_Agnostic): ...

# And instantiate it.
def get_agnostic_object(v: str, *args, **kwargs) -> Object_Agnostic:
    if v == "v2":
        return Object_V2(*args, **kwargs)
    else:
        return Object_V3(*args, **kwargs)

[cinder bone]

You could do it with literals

from typing import Literal, overload

@overload
def get_obj(v: Literal["v2"]) -> Object_V2: ...

@overload
def get_obj(v: object) -> Object_V3: ...

# actual implementation

Sorry I'm on my phone

[tacit sparrow]

sure, literals is an option to make get_agnostic_object be more specific if correct literal is provided
but I'm looking for name from Object signature to appear in get_agnostic_object signature, so user would know what they're specifying using *args, **kwargs.

[cinder bone]

Why not just return the class and have users instantiate it