type-hinting

[trim tangle]

python/mypy#11941

[mighty linden]

GitHub

[mypy] #11941 Signatures of "ior" and "or" are incompatible when assigning or with compatible method

[vivid ore]

yes

[trim tangle]

Also why do you want to make maybe mutable?

[vivid ore]

just wanna experiment a bit
also thought it could possibly somehow be useful or be an interesting exercise

[trim tangle]

I am unable to reproduce your issue

Can you show the whole code?

[vivid ore]

https://gist.github.com/schuelermine/50241788d9febe5d7eede03bcfe536a8#file-mmaybe-py-L249

[trim tangle]

Hmmmm

Any reason you use G instead of T?

Also there's no need to type self if you use T as the type var

The issue seems to go away if you use T

The error message is strange though

[vivid ore]

it’s a habit I acquired from working with the covariant type variable because I would get errors saying covariant type variables can’t be used as arguments

thank you for the tip

[slender timber]

Descriptor protocol (type hints) when? 😩

[elfin nexus]

a)dict[TYPE1 : TYPE2]
b)dict[TYPE1, TYPE]

if i want i type hint a dictionary, are both methods correct, (do they mean the same thing), and if so which is preferred (and why)
Thanks

[oblique urchin]

a is wrong, b is right

[median ledge]

I have a descriptor whose __get__ can either return itself or a value stored in the instance, depending on whether it's accessed from the class or an instance, respectively.

Is it possible to hint it so that a type checker can tell if it should return the Descriptor or T@Descriptor ?

[soft matrix]

yes

use overloads

[vivid ore]

from typing import overload
...
class MyDescriptor(...):
    ...
    @overload
    def __get__(self, obj: type[MyClass[T]]) -> MyDescriptor:
        ...

    @overload
    def __get__(self, obj: MyClass[T]) -> T:
        ...

    def __get__(self, obj: type[MyClass[T]] | MyClass[T]) -> MyDescriptor | T:
        if isinstance(type(obj), type):
            return self
        return ...

@median ledge

[median ledge]

Shouldn't obj be of type None | MyClass[T]?

[vivid ore]

I might be misunderstanding how descriptors work

isn’t it that when you get Klass.descriptor, it calls Descriptor.__get__(object.__getattr__(Klass, "descriptor"), Klass)?

[median ledge]

If I read the docs correctly, the signature is

class Descriptor:
    def __get__(self, obj: MyClass | None, objtype: type[MyClass]):...

[rare scarab]

obj can be none

[median ledge]

I forgot to mention, but this descriptor is an abstract class, which is subclassed to have descriptors of different types.
This is basically what I have:

# python 3.10, so typing.Self not available, but I'll upgrade if this workaround doesn't work
Self = TypeVar("Self", bound="Descriptor")

class Descriptor(Generic[T], metaclass=ABCMeta):
    @overload
    def __get__(self: Self, obj: None, objtype: FooMeta | None = None) -> Self:
        pass

    @overload
    def __get__(self, obj: Foo, objtype: FooMeta | None = None) -> T | None:
        ...

    def __get__(self: Self, obj: Model | None, objtype: FooMeta | None = None) -> Self | T | None:
        if obj is None:  # called from class
            return self
        return getattr(obj, self.name, self.default)

class IntDescriptor(Descriptor[int]):...

class Foo:
    bar = IntDescriptor()

Foo().bar # int
Foo.bar # IntDescriptor

[median ledge]

oops, forgot that

[rare scarab]

why not import Self from typing_extensions?

[median ledge]

because I didn't know about that 😅
Is that like __future__ but for typing?

[rare scarab]

!pip typing-extensions

[rough sluice]

typing-extensions v4.4.0

Backported and Experimental Type Hints for Python 3.7+

[median ledge]

oh, nice

[rare scarab]

if you use future imports and only import it through a check for if typing.TYPE_HINTING:, you don't even have to install it.

from typing import TYPE_CHECKING
if TYPE_CHECKING:
  from typing_extensions import Self

[median ledge]

Is it not a built-in?

[rare scarab]

no. it's backports

[vivid ore]

does mypy support Self at this point?

[median ledge]

hmm, must be a dependency from another package because I already have it in my venv

[rare scarab]

Looks like yes? https://github.com/python/mypy/issues/11871

[median ledge]

the python extension for vscode seems to, at least

[vivid ore]

that’s pyright

[rare scarab]

pyright always implements the latest experimental typing features

at least pyright is open source (unlike pylance, grr)

I want pylance on openvsx

(off topic)

[vivid ore]

How do I correctly type a method on an invariant generic that can work on any instance whose type variable is an instance of a specific generic protocol?
As I understand it, this would constrain foobar to only work on Bars with exactly Protocol[T]] and not any subclasses.

class Foo(Protocol[T]):
    ...
class Bar(Generic[T]):
    def foobar(self: Bar[Protocol[U]]) -> U:
        ...

and I can’t do this because type variables cannot be generic

...
I = TypeVar("I", Foo[U])
...

[rare scarab]

You're not supposed to use Protocol directly.

Try Foo[U]?

Protocol means nothing.

[vivid ore]

Oh yeah sorry

I made a mistake

Still that doesn't work

Because type variables cannot be generic

[rare scarab]

remove the generic

[vivid ore]

but as you can see, I want the return type to follow the type parameter

[rare scarab]

Could you use Literal[type[Bar]]?

would that even work?

[oblique urchin]

no

[rare scarab]

I guess make an enum representing your class

[vivid ore]

how would that solve it?

[rare scarab]

I kind of have a hard time understanding your usage. What does the implementation look like?

[vivid ore]

I want foobar to work for every Bar[T] where T is a subtype of Protocol[U], even though Bar is invariant

[rare scarab]

but Protocol[U] doesn't mean anything.

it's like saying Generic[T]

[vivid ore]

oh sorry I made the same mistake again

I meant Foo[U]

sorry

[vivid ore]

let’s say

T = TypeVar("T")
T_co = TypeVar("T", covariant=True)
U = TypeVar("U")

class Foo(Protocol[T_co]):
    def getfoo(self) -> T_co:
        ...

@dataclass
class Bar(Generic[T]):
    x: T
    def foobar(self: Bar[Foo[U]]) -> U:
        return self.x.getfoo()

[rare scarab]

you mean def getfoo(self) -> T_co?

[vivid ore]

yes

[rare scarab]

so is Foo a mixin?

[vivid ore]

I’m not familiar with the concept, but if it describes what’s happening here, yes

[rare scarab]

multiple inheritance

so does self need to be a Foo, or self.x?

[vivid ore]

self.x needs to be a Foo

but for this to happen the T must be Foo

so self must be a Bar[Foo[...]]

in the end I want this to work:

class Boo(Foo[tuple[()]]):
    def getfoo(self) -> tuple[()]:
        return ()

y: Bar[Boo] = Bar(Boo())
x = y.foobar()

[rare scarab]

T_co = TypeVar("T_co", covariant=True)

class Foo(Protocol[T_co]):
  def getfoo(self) -> T_co: ...

class Bar(Generic[T_co]):
  x: Foo[T_co]
  
  def foobar(self) -> T_co:
    return self.x.getfoo()
```Like this?

[vivid ore]

that would allow Bar to only have Foos in it

I want Bar to be able to hold other values

but the foobar function to only be correct when it holds a subtype of a certain type

I’m assuming it’s just not possible with the current type system

[wanton urchin]

So, I'm writing a main method that takes the arguments as a parameter, but I'm also accounting for if the main method is given the Namespace object from argparse. So I'm trying to create a new "ArgumentParser" object by joining the strs in a list[str]. But my editor doesn't like that and marks it as a syntax error. How would you solve the problem:

def main(args: list[str] | Namespace) -> int:
  if args is list[str]:
  parser = ArgumentParser(" ".join(args)) # args has red squigly underline because list[str] | Namespace cannot be converted to list[str]

[soft matrix]

you cant compare args to a list[str]

you need to use isinstance

!d isinstance

[rough sluice]

isinstance

isinstance(object, classinfo)```
Return `True` if the *object* argument is an instance of the *classinfo* argument, or of a (direct, indirect, or [virtual](https://docs.python.org/3/glossary.html#term-abstract-base-class)) subclass thereof. If *object* is not an object of the given type, the function always returns `False`. If *classinfo* is a tuple of type objects (or recursively, other such tuples) or a [Union Type](https://docs.python.org/3/library/stdtypes.html#types-union) of multiple types, return `True` if *object* is an instance of any of the types. If *classinfo* is not a type or tuple of types and such tuples, a [`TypeError`](https://docs.python.org/3/library/exceptions.html#TypeError "TypeError") exception is raised. [`TypeError`](https://docs.python.org/3/library/exceptions.html#TypeError "TypeError") may not be raised for an invalid type if an earlier check succeeds.

Changed in version 3.10: *classinfo* can be a [Union Type](https://docs.python.org/3/library/stdtypes.html#types-union).

Similar names: 2to3fixer.isinstance

[wanton urchin]

so...

if isinstance(args, list) and isinstance(args[0], str):

it doesn't like

if isinstance(args, list[str]):

[soft matrix]

just isinstance(args, list) should be fine

cause Namespace should never be a subclass of list im assuming?

id generally stay away from checking this stuff at runtime

[wanton urchin]

Correct

[soft matrix]

this doesnt work cause its a pain in the ass to make work correctly

and not have awful time complexity

[wanton urchin]

Well, the behavior has to be different depending on what is passed. If a list of strings is passed, then it has to set up argparse and create the Namespace object and then do all the same things.

[soft matrix]

but the actual type of the elements of the list doesnt matter at runtime cause you just assume that if its a list it'll be a list of str

unless you do actually accept list[int] or w/e

[wanton urchin]

No. You're right. It should be a list of strings

I was responding to what you said about "checking this stuff at runtime"

In any case, I still need to pass a list[str] to ArgumentParser()

And the type of "args" is Union[list[str], Namespace]

or really list[str] | Namespace

[soft matrix]

it shouldnt be inside of the if block

i though type checkers were smart enough to avoid this

[wanton urchin]

pylint doesn't seem to be.

I think

[soft matrix]

pylint isnt a typechecker is it?

[wanton urchin]

wait... somehow "if isinstance(args, list)" seemed to fix it somehow?

[soft matrix]

what did you have before?

[wanton urchin]

def main(args: list[str] | Namespace) -> int:
  if args is list[str]:
    parser = ArgumentParser(" ".join(args))
    # Arguments go here

    args = parser.parse_args()
  return 0

[soft matrix]

yeah you have to use isinstance as i said

args identity comparing to list[str] is a bad idea

[wanton urchin]

vs...

def main(args: list[str] | Namespace) -> int:
  if isinstance(args, list):
    parser = ArgumentParser(" ".join(args))
    # Arguments go here

    args = parser.parse_args()
  return 0

before, args was red-underlined, now it's not

I had come up with the solution of args.copy()

but I didn't know if it was right

[analog fable]

I want to define a typealias for a union of some constants, what's the best way to do it?

A = 'a'
B = 'b'
C = 'c'
LETTERS: TypeAlias = A | B | C

[oblique urchin]

Literal["a", "b", "c"]

[analog fable]

I would like to have A, B, and C as constants though and would like to reuse them instead of redefining them in LETTERS

if I changed A = "z", then I'd also have to change LETTERS = Literal["z", "b", "c"]

just curious

I guess I can define it this way and still have the typesafety I want

LETTERS = Literal['a', 'b', 'c']
A: LETTERS = 'a'
B: LETTERS = 'b'
C: LETTERS = 'c'

[pastel egret]

The best way would be to use enum.Enum, then you get the right behaviour.

[analog fable]

ah, that's good. Thank you

[viral spade]

which is preferred a: int = 0 or a :int = 0 ?

[pastel egret]

The first, same as dict literals.

[slender timber]

Do u guys annotate locals with type hints also?

Looks quite verbose

[pastel egret]

Usually you don't need to, only if it's not something that can be inferred. In particular you want to annotate empty lists/sets/dicts etc, because the types aren't present.

[atomic lodge]

Does it make sense to typehint like this?

frontend: dict[str, Any] = tomllib.load(file) or do all type checkers automatically know that tomllib.load() provides such a dict?

[leaden oak]

Looks unnecessary https://github.com/python/typeshed/blob/main/stdlib/tomllib.pyi#L9 they should be able to infer that using its return type

[rough sluice]

stdlib/tomllib.pyi line 9

def load(__fp: SupportsRead[bytes], *, parse_float: Callable[[str], Any] = ...) -> dict[str, Any]: ...```

[runic sleet]

can overloads allow a "permissive base case"?

from typing import overload, Any

@overload
def view(obj: int) -> int: ...

@overload
def view(obj: object) -> str: ...

def view(obj):
    if isinstance(obj, int):
        return 1000
    else:
        return "1000"

mypy >> error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]

obj: typing.Any is also the same error

I'd like to define multiple actual overload mappings and then one type that will be returned if no previous overload matches (any other type essentially)

[vivid ore]

No

[oblique urchin]

if you type ignore that error it will work as you expect

[runic sleet]

hm.. yeah

reveal_type does seem to get everything correctly

[oblique urchin]

the problem is that this will work wrong if you have something like x: object = 1; reveal_type(view(x))

that's going to be an int at runtime but mypy will infer str

[trim tangle]

yep

[runic sleet]

ah hm

I guess there's no way around that for now

[trim tangle]

yeah, there isn't really a "negation" here

I guess you could do (obj: object) -> str | int

[runic sleet]

was there a pep for negation typevar

or was it rejected can't remember

[oblique urchin]

there hasn't been one

people have talked about wanting that feature

but nobody has actually written a PEP

[runic sleet]

also apparently mypy infers this correctly but pycharm adds a union for the base overload 😔

[runic sleet]

oh right there was this other thing, is returning the current subclass type in a generator possible? It seems the new typing.Self type does not support use as a subscript type?

from collections.abc import Generator
from contextlib import contextmanager
from typing import Self

class BaseView:
    @contextmanager
    def context(self) -> Generator[Self, None, None]:
        yield self

class View(BaseView):
    ...

with View().context() as x:
    reveal_type(x)

[oblique urchin]

yeah that should work

[runic sleet]

error: Variable "typing.Self" is not valid as a type  [valid-type]
note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
note: Revealed type is "Self?"

[oblique urchin]

what mypy version are you on?

I think only 0.990+ supports Self

[runic sleet]

--version shows

mypy 0.991 (compiled: yes)

and using typing.Generator seems to be the same

[oblique urchin]

seems you found a mypy bug

[runic sleet]

❯ mypy --version && mypy main.py
mypy 0.991 (compiled: yes)
main.py:7: error: Variable "typing.Self" is not valid as a type  [valid-type]

[oblique urchin]

oh hm, maybe Self support actually didn't go into 0.991

[runic sleet]

ah

!pep 673 also actually... looking at the pep, is it stated anywhere that stuff like Iterator[Self] will be supported?

[rough sluice]

**PEP 673 - Self Type**

Link

Status

Accepted

Python-Version

3.11

Created

10-Nov-2021

Type

Standards Track

[oblique urchin]

confirmed this works correctly on mypy master, the reveal_type() says View

[oblique urchin]

https://peps.python.org/pep-0673/#valid-locations-for-self has examples with list[Self]

[runic sleet]

ah cool 👍 yeah that'd be nice

[runic sleet]

this seems to work for now 😔

from typing import TypeVar, ContextManager

_T = TypeVar("_T")

class BaseView:
    def context(self: _T) -> ContextManager[_T]:
        class Context:
            # noinspection PyMethodParameters
            def __enter__(self_) -> _T:
                return self

            def __exit__(self, exc_type, exc_val, exc_tb) -> None:
                pass

        return Context()

class View(BaseView):
    ...

class SubView(View):
    ...

with View().context() as x:
    reveal_type(x)

main.py:25: note: Revealed type is "main.View"
Success: no issues found in 1 source file

though maybe I'll just stick with Self and wait for mypy support when it comes

[oblique urchin]

should also work with @contextmanager and def c(self: _T) -> Generator[_T, None, None]

[runic sleet]

I.. yup, that works the same, don't know what I was doing there

[still ridge]

I'm trying to write types for a class like tensorflow.keras.Model where subclasses implement a call method that the superclass __call__ delegates to. So as the simplest case I just want to have type deduction figure out that in subclasses __call__ has the exact same signature as the subclass's call it would be nice if I could apply some transformations to the signature as well, but that's not required.

Is something like that possible? I would like something kind of like self.__call__: TypeOf[Self.call]

[soft matrix]

this just seems a little redundant surely just implement call in every case?

[still ridge]

The class needs to add wrapping logic around the inputs and outputs of the call method so it doesn't work with just super

[soft matrix]

maybe you need a metaclass tbh

or just init subclass

have it magically wrap the method appropriately

[still ridge]

How would the typing work with a metaclass?

I haven't used typing with metaclasses before

[soft matrix]

call will get the type checker to report if the signatures are incompatible

wait actually it should do it for this case already 🤔

[still ridge]

An example of what I'm talking about is roughly

class Model:
    def build(input_shapes):
        ...

    def __call__(self, *args, **kwargs):
        if not self.built:
            self.build(recursively_get_shapes(args))
            self.built = True
        with name_scope(self.name):
            t_args = self.recursive_to_tensor_of_type_as_needed(args, type=self.dtype)
            t_kwargs = self.recursive_to_tensor_of_type_as_needed(kwargs, type=self.dtype)
            return self.call(*t_args, **t_kwargs)


class SubModel(Model):
    def call(self, x: Tensor, mask: Tensor | None = None) -> Tensor:
        if mask is not None:
            return apply_mask(x, mask)
        else:
            return x


submodel = SubModel()
x = Tensor.ones((5, 3), type=float)
mask = Tensor.ones((5, 3), type=bool)
# Want this to have correct type inference
result = submodel(x, y)

So in this case the signature of SubModel should be deduced to be def __call__(self, x: ConvertibleToTensor, mask: ConvertibleToTensor | None = None) -> Tensor

[soft matrix]

yeah ok i think my suggestion of using init subclass is the best way to do this

Model has a writable dict right?

[still ridge]

I'll note I come from C++ where I'm used to being able to do lots of tricks to get good type inference with things like decltype

yeah

I'm not sure I understand what an init subclass is

[soft matrix]

you could theoretically use generics here but youd lose the parameter names of call

!d object.init_subclass

[rough sluice]

object.\_\_init\_subclass\_\_

classmethod object.__init_subclass__(cls)```
This method is called whenever the containing class is subclassed. *cls* is then the new subclass. If defined as a normal instance method, this method is implicitly converted to a class method.

Keyword arguments which are given to a new class are passed to the parent’s class `__init_subclass__`. For compatibility with other classes using `__init_subclass__`, one should take out the needed keyword arguments and pass the others over to the base class, as in:

```py
class Philosopher:
    def __init_subclass__(cls, /, default_name, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.default_name = default_name

class AustralianPhilosopher(Philosopher, default_name="Bruce"):
    pass

[soft matrix]

i think this does a decent job explaining what it does

although im happy to answer any questions about it

[still ridge]

oooh I hadn't seen this before. Does it interact well with type checking?

[soft matrix]

it doesnt really touch type checking

[still ridge]

The code I posted above works for execution. The problem is that the last line doesn't type check well. It can't check that the arguments are correct (or do autocomplete) or tell me what the type of result is.

[soft matrix]

class Model:
    def __init_subclass__(cls):
        orig_call = cls.__call__
        @functools.wraps(orig_call)
        def wrapper(self, *args, **kwargs):
            if not self.built:
                self.build(recursively_get_shapes(args))
                self.built = True
            with name_scope(self.name):
                t_args = self.recursive_to_tensor_of_type_as_needed(args, type=self.dtype)
                t_kwargs = self.recursive_to_tensor_of_type_as_needed(kwargs, type=self.dtype)
                return orig_call(self, *t_args, **t_kwargs)
        cls.__dict__["__call__"] = wrapper

    def build(input_shapes):
        ...


class SubModel(Model):
    def __call__(self, x: Tensor, mask: Tensor | None = None) -> Tensor:
        if mask is not None:
            return apply_mask(x, mask)
        else:
            return x


submodel = SubModel()
x = Tensor.ones((5, 3), type=float)
mask = Tensor.ones((5, 3), type=bool)
# Want this to have correct type inference
result = submodel(x, y)```

that sshould work

[still ridge]

oh I see

[soft matrix]

that was what i was imagining atleast

[still ridge]

that doesn't handle transforming the types, but I wasn't expecting that to be possible

[soft matrix]

wdym transforming the types?

oh wait ik

um

yeah thats gonna be a lot harder

you could probably just do it with overloads but thats annoying

[still ridge]

if you look in my example the types got transformed to ConvertibleToTensor. I wasn't thinking that was going to be possible though.

[vivid ore]

I made a proposal related to a recent question I asked here: https://github.com/python/typing/issues/1311

[fathom river]

How would I go about overloading variadics?
I've got this rough setup

@overload
async def send(*args: str, handler: Literal[None] = ..., **options: Any) -> Message:
    ...

@overload
async def send(*args: str, handler: Handler = ..., **options: Any) -> tuple[Message, Handler]:
    ...

async def send(*args: Any, **options: Any) -> Any:
    pass

but pyright doesn't like it

Overload 2 for "send" will never be used because its parameters overlap overload 1

[oblique urchin]

don't annotate your kwargs as Any

though actually, it should still be able to figure out the overloads don't fully overlap; this might be a pyright bug

[fathom river]

it's kinda either that or a huge list of unions

[fathom river]

welp

type: ignore it is then

[oblique urchin]

report the bug to pyright too

[runic sleet]

Why does mypy require a list specifically for ctypes.Array setitems?

from ctypes import c_ssize_t

Arr = c_ssize_t * 3
arr = Arr()

arr[:] = (1, 2, 3)

error: No overload variant of "__setitem__" of "Array" matches argument types "slice", "Tuple[int, int, int]"  [call-overload]

note: Possible overload variants:
note:     def __setitem__(self, int, Union[c_ssize_t, int], /) -> None
note:     def __setitem__(self, slice, List[Union[c_ssize_t, int]], /) -> None

https://github.com/python/typeshed/blob/main/stdlib/ctypes/__init__.pyi#L294-L295

[rough sluice]

stdlib/ctypes/__init__.pyi lines 294 to 295

@overload
def __setitem__(self, __s: slice, __o: Iterable[Any]) -> None: ...```

[runic sleet]

the typeshed for ctypes.Array shows Iterable[Any]

[oblique urchin]

I think they have a plugin that sets some precise types

https://github.com/python/mypy/blob/master/mypy/plugins/ctypes.py#L165

[rough sluice]

mypy/plugins/ctypes.py line 165

def array_setitem_callback(ctx: mypy.plugin.MethodSigContext) -> CallableType:```

[fathom crown]

why does it show Any when i have hinted the type?

[oblique urchin]

looks like you annotated self.sleeping, not sleeping

[fathom crown]

uhh

how should it be done

[trim tangle]

Can you show more code maybe?

[runic sleet]

Is there a way to annotate as returning a TypeVar's generic type?

_T = TypeVar("_T")

class Foo(Generic[_T]):
    def __init__(self, x: _T) -> None:
        self.x = x

    def get(self, index) -> (_T's generic type?):
        return self.x[index]

assuming f = Foo([1, 2, 3])
_T should be list[int]
so method get() should return the generic subtype of list, which is int

[oblique urchin]

you return _T

[runic sleet]

hm? but _T is list right

get would return an element at index of the list, an int in this case

[oblique urchin]

oh I see. in this case you should make your class generic over Sequence[_T] (or a more permissive equivalent, like a generic Protocol with __getitem__)

the current code is unsafe because x may not even support subscripting

[runic sleet]

ah hm

do I have an option of not supporting the actual protocol of the object?

I'm aware typing.get_args can runtime inspect the "subtypes" of a generic but something like that is not available in type checking?

[oblique urchin]

no, and it wouldn't be type-safe here

since you don't know that the generic argument is what __getitem__ returns

[runic sleet]

well essentially it's just this which currently works fine for inferring get

class Foo(Generic[_T]):
    def __init__(self, x: list[_T]) -> None:
        self.x = x

    def get(self, index) -> _T:
        return self.x[index]

but instead of list, a TypeVar with a bound of list

[fathom crown]

it happens if I use __new__ as constructor instead of __init__ as you can see their methods being colourless (1st pic) in contrast to the 2nd pic

[trim tangle]

Ah, I think the attribute annotations only really work in __init__ (or instance methods?..)

What you can do is
a) add an explicit sleeping: asyncio.Future annotation to the class
b) just use __init__

Why do you need __new__ btw?

[fathom crown]

i subclassed a class that used __new__

while also learning about the __new__ method

[fathom crown]

how does a) work?

[trim tangle]

class Foo:
    bar: Amogus

This is an attribute annotation, most notably used in dataclasses

Most of the time you don't really need __new__. But, well, sometimes you do have a library that does some magic

[runic sleet]

btw I think the main issue is

can't actually find the documentation for that but I don't think type hinting non self attributes is supported

[trim tangle]

Yeah

Something like that

The variable name self might be deceptive 🙂

[runic sleet]

but as long as you hint it in the class header it should be fine

class Number(int):
    text: str

    def __new__(cls, value, base=10) -> Number:
        instance = super().__new__(cls, value, base)
        instance.text = str(value)
        return instance

n = Number(10)
value = n.text

reveal_type(value)

note: Revealed type is "builtins.str"

[fathom crown]

i see

interesting

[fathom river]

How are these incompatible?

@overload
async def send(ctx: commands.Context, *args: MessageContent, paginator: Paginator | Literal[None], **options: Any) -> tuple[discord.Message, Paginator | None]: ...

@overload
async def send(ctx: commands.Context, *args: MessageContent, **options: Any) -> discord.Message: ...

async def send(ctx: Any, *args: Any, paginator: Any = ..., **options: Any) -> Any:
    pass

Overload 1 for "send" overlaps overload 2 and returns an incompatible type

[trim tangle]

If options in the second overload include a paginator key, it's actually the first overload.

...In which case a tuple will be returned. So if you call this with some arbitrary kwargs, you don't know what you'll get back

Can you provide more context perhaps?

[fathom river]

Well I've got this helper function, *args and **options are arguments that will be forward to another function, but I also want to add this other key word argument (paginator) that changes the behavior of the helper. And I'm trying to express that change of behavior through overloads

Simply passing in paginator, regardless of the value, should change the behavior. But I guess setting it as paginator: Any will get me nowhere

A Not[Paginator | Literal[None]] would be cool here lmao

[rare scarab]

Build it with conditionals first

You can do that in typescript without any special type. ```ts
type Not<T, E> = T extends E ? never : T
type NotPaginatorOrNull<T> = Not<T, Paginator | null | undefined>

(not in python though 😦 )

[fathom river]

Can you even do that in python?

[rare scarab]

[fathom river]

Oh, I thought you meant "without any special type", welp

Tbf, pyright can somewhat get the return types right, so I guess I'll just type: ignore it

[rare scarab]

use type guards instead.

[fathom river]

Time to figure out how those work then

How would that be applicable to the current case?

[tranquil turtle]

are there any differences between Literal[None] and None?

[soft matrix]

no

[heady flicker]

Do we have a way to overload with return type changing based on the argument not being passed while the default value for the argument is an empty sequence instead of None? I.e. Pyright is fine with the following code while mypy outputs: error: Overloaded function signatures 1 and 2 overlap with incompatible return types

from collections.abc import Sequence
from typing_extensions import Never, overload, Any


@overload
def foo(bar: Sequence[str]) -> list[dict[Any, Any]]:
    ...


@overload
def foo(bar: Sequence[Never] = ...) -> None:
    ...


def foo(bar: Sequence[str] | Sequence[Never] = ()) -> list[dict[Any, Any]] | None:
    if bar:
        return [{}]
    return None


foo([123])  # error, as expected
foo([])  # no error, as expected
foo([""])  # no error, as expected

[trim tangle]

() is also a valid Sequence[str], so that won't really work

Maybe consider making None the default value?

(in which case the whole function isn't really needed)

[heady flicker]

Yeah, I know :(

That's the main issue.

[fathom river]

you don't have to type hint the implementation, just the overloads iirc

[heady flicker]

This is just a dummy example ;)

[heady flicker]

You do for typechecking within the function

[heady flicker]

I hoped that there was a better way but I guess not.

Thank you!

[hallow flint]

mypy’s overlap with incompatible return type warning can be safely type ignored if it’s intentional; mypy will do what you want

[heady flicker]

I would really prefer to stay within mypy's boundaries, especially for such simple cases.

And if mypy disagrees with me — I'm either doing some nasty magic or I'm wrong. That's how I think of it. It's similar to Alex Traut's view where he prefers to expand the type system instead of making plugins or putting in type ignore comments

[sonic fox]

class ExampleTypedDict(t.TypedDict):
    name: ExampleTypedField

In this example, with TypedDict or other python typing features, is it possible to make name a dynamic key that can be any string? I haven't found any evidence that suggests this is possible :/. Please note that I'm typing an API response and would prefer to type it like this, even if it's a little hacky, rather than modifying the response data. thanks! :)

[rose root]

I don't think it's possible

You only got the static keys in TypedDict, the whole point of it

[tranquil turtle]

Wdym by static/dynamic keys?

[trim tangle]

like py class Thing(TypedDict): name: str age: int ArbitraryKeys and {"name": "foo", "age": 42, "money": 42069, "cool": false} would be a valid example

[tranquil turtle]

Ok, got it
*_: ArbitraryKeys[Any]

[cosmic plinth]

is there a way for a property to return a TypeVar?

from typing import TypeVar

class Builder:
    pass

BuilderBound = TypeVar('BuilderBound', bound=Builder)

class BuilderConfig:
    def __init__(self, builder: BuilderBound) -> None:
        self.__builder = builder

    @property
    def builder(self) -> BuilderBound:
        return self.__builder

I get:

error: A function returning TypeVar should receive at least one argument containing the same TypeVar  [type-var]
        def builder(self) -> BuilderBound:

[fathom river]

make BuilderConfig a Generic

[cosmic plinth]

oh ok, how?

[fathom river]

import Generic from the typing module, and have BuilderConfig "subclass" it. Then pass the TypeVar to Generic's parameters

class BuilderConfig(Generic[BuilderBound]): ...

[cosmic plinth]

you're a wizard, thank you

[cosmic plinth]

what about this?

interface.py:

from abc import ABC
from typing import Generic, TypeVar

from .config import BuilderConfig, BuilderConfigBound

class BuilderInterface(ABC, Generic[BuilderConfigBound]):
    @classmethod
    def get_config_class(cls) -> type[BuilderConfigBound]:
        return BuilderConfig

BuilderInterfaceBound = TypeVar('BuilderInterfaceBound', bound=BuilderInterface)

config.py:

from typing import TYPE_CHECKING, Generic, TypeVar

if TYPE_CHECKING:
    from .interface import BuilderInterfaceBound

class BuilderConfig(Generic[BuilderInterfaceBound]):
    def __init__(self, builder: BuilderInterfaceBound) -> None:
        self.__builder = builder

    @property
    def builder(self) -> BuilderInterfaceBound:
        return self.__builder

BuilderConfigBound = TypeVar('BuilderConfigBound', bound=BuilderConfig)

I get:

interface.py: error: Incompatible return value type (got "Type[BuilderConfig[Any]]", expected "Type[BuilderConfigBound]")  [return-value]
            return BuilderConfig

[fathom river]

nvm. What're you trying to do? could you give a bit more context?

I thought you'd subclass the ABC, but I guess not

[cosmic plinth]

I need the method get_config_class on BuilderInterface and any subclass to return the class type BuilderConfig or a type derived from that

so WheelBuilder's might return WheelBuilderConfig

[fathom river]

you don't need to make BuilderInterface generic then

[cosmic plinth]

same error

[fathom river]

you don't need the BuilderConfigBound TypeVar to be in config, you can make it in interface. You're also gonna get an error at runtime because BuilderInterfaceBound won't be defined in config.py

[cosmic plinth]

sorry, what should I do?

[fathom river]

oh, silly me, make get_config_class return type[BuilderConfig]. You don't need the TypeVars

[cosmic plinth]

& subclasses that have alternative configs would return type[AlternativeBuilderConfig]?

[fathom river]

type-wise, it'll be ok to return AlternativeBuilderConfig (assuming it subclasses BuilderConfig) with the function's return annotation as type[BuilderConfig]

[cosmic plinth]

@fathom river ty! so close, last question:

class BuilderInterface(ABC, Generic[BuilderConfigBound]):
    def __init__(self) -> None:
        self.__config: BuilderConfigBound | None = None

    @property
    def config(self) -> BuilderConfigBound:
        if self.__config is None:
            self.__config = self.get_config_class()('...')

        return self.__config

    @classmethod
    def get_config_class(cls) -> type[BuilderConfig]:
        return BuilderConfig

interface.py: error: Incompatible types in assignment (expression has type "BuilderConfig[Any]", variable has type "Optional[BuilderConfigBound]")
[assignment]
                self.__config = self.get_config_class()(
                                ^
interface.py: error: Incompatible return value type (got "Optional[BuilderConfigBound]", expected "BuilderConfigBound")  [return-value]
            return self.__config
                   ^~~~~~~~~~~~~

[fathom river]

your type checker isn't inferring that self.__config has been changed if it's None, so create a variable that points to self.__config, and use that as the return value instead (also use it to change the value of self.__config too)

[cosmic plinth]

interface.py: error: Incompatible types in assignment (expression has type "BuilderConfig[Any]", variable has type "Optional[BuilderConfigBound]")
[assignment]
                config = self.get_config_class()(
                         ^
interface.py: error: Incompatible return value type (got "Optional[BuilderConfigBound]", expected "BuilderConfigBound")  [return-value]
            return config
                   ^~~~~~

[fathom river]

show code

[cosmic plinth]

    @property
    def config(self) -> BuilderConfigBound:
        config = self.__config
        if config is None:
            config = self.get_config_class()(
                self, self.root, self.PLUGIN_NAME, self.build_config, self.target_config
            )
            self.__config = config

        return config

[fathom river]

you shouldn't return BuilderConfigBound, you should return BuilderConfig. And self.__config should be BuilderConfig | None. You don't need typevars nor a generic in this case

[cosmic plinth]

that's what master has rn and is broken

backend\src\hatchling\builders\sdist.py:166:20: error: "BuilderConfig[Any]" has no attribute "support_legacy"  [attr-defined]
                    if self.config.support_legacy:
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~
backend\src\hatchling\builders\sdist.py:184:17: error: "BuilderConfig[Any]" has no attribute "core_metadata_constructor"  [attr-defined]
                    self.config.core_metadata_constructor(self.metadata, extra_dependencies=build_data['dependencies']),
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
backend\src\hatchling\builders\sdist.py:188:16: error: "BuilderConfig[Any]" has no attribute "support_legacy"  [attr-defined]
                if self.config.support_legacy:
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~
backend\src\hatchling\builders\sdist.py:203:16: error: "BuilderConfig[Any]" has no attribute "strict_naming"  [attr-defined]
                if self.config.strict_naming
                   ^~~~~~~~~~~~~~~~~~~~~~~~~

https://github.com/pypa/hatch/blob/6322daba750f8db73361fa1aaa965bc99605303b/backend/src/hatchling/builders/sdist.py#L344-L346

[rough sluice]

backend/src/hatchling/builders/sdist.py lines 344 to 346

@classmethod
def get_config_class(cls) -> type[SdistBuilderConfig]:
    return SdistBuilderConfig```

[spice locust]

two questions:
how would i type-hint a non-empty list or tuple? i.e. list[str] except it should not accept empty lists

how would i recursively check the types of nested objects against a provided type-hint to verify that the object conforms to the type-hint? for example, if i have a list object [1, 2, 3, "hello"] and a type hint list[int] how would i 1) check that the main object (the list in this case) is a list, and 2) check that every object inside the main object is an int (while also working with an arbitrary number of nesting levels)? in other words, do what a type checker does, preferably without a large amount of code

[fathom river]

you can't do the former, for the latter there's a message somewhere that explains that it isn't ideal to check types are runtime.
It had to do with the amount of stuff that can be checked and how you'd check it

[fathom river]

that's a lot of new stuff

[acoustic thicket]

for the second question https://decorator-factory.github.io/typing-tips/faq/runtime-checking/

[analog glacier]

is there a way you can force a function to only accept a type of data?

like only iterable

[pastel egret]

What do you mean by "force"? Add a type hint for that type? Or do you mean at runtime? Or do you want to constrain it in multiple ways, which you could do via Protocol?

[analog glacier]

add a type hint for that type

like raise an exception if its not that type

[pastel egret]

There are several runtime type checkers out there, that check arguments for you. There are limitations though, since not all types can actually be checked ahead of time. For instance there's no way to be sure what type a generator would produce, without actually fetching a value from it...

[analog glacier]

so you have to use if statements anyways?

[peak echo]

Hey need a little help on enum typehinting
I have this enum

class MyEnum(Enum):
  VALUE = 0

And I have a function that works like this

my_func(MyEnum) -> MyEnum.VALUE```

How do I typehint `my_func` properly to work with any Enum?

[soft matrix]

!d typing.Literal

[rough sluice]

typing.Literal

typing.Literal```
A type that can be used to indicate to type checkers that the corresponding variable or function parameter has a value equivalent to the provided literal (or one of several literals). For example:

```py
def validate_simple(data: Any) -> Literal[True]:  # always returns True
    ...

MODE = Literal['r', 'rb', 'w', 'wb']
def open_helper(file: str, mode: MODE) -> str:
    ...

open_helper('/some/path', 'r')  # Passes type check
open_helper('/other/path', 'typo')  # Error in type checker
```...

[pastel egret]

You'd want to use a TypeVar bound to Enum:

from typing import TypeVar, Type
from enum import Enum
EnumT = TypeVar('EnumT', bound=Enum)

def my_func(cls: Type[EnumT) -> EnumT: ...

[soft matrix]

It shouldn't need a typevar right?

[peak echo]

Thought of that, thanks

[soft matrix]

Oh wait any enum

[peak echo]

Literal would only work for that concrete enum and that concrete value

[soft matrix]

Can't read yeah sorry

[pastel egret]

No, you use libraries like typeguard or beartype which generate the checks based on your type hints.

[trim tangle]

I have a short piece on that 🙂
https://decorator-factory.github.io/typing-tips/faq/runtime-checking/

[devout barn]

How am I supposed to know if a given annotation is typing.Annotated without accessing typing's private impl details.
Say I wanted to refactor the following function

# _AnnotatedAlias is from stdlib typing
def get_metadata(obj, default: T | Literal[UNSPECIFIED] = UNSPECIFIED) -> T | X:
    if isinstance(obj, _AnnotatedAlias):
        return obj.__metadata__
    if default is UNSPECIFIED:
        raise ValueError()
    return default

How do I go about doing it without importing private symbols

[soft matrix]

you cant really

the best i think you can do is AnnotatedAlias = type(Annotated[int, 0])

metadata isnt documented either is it

[devout barn]

good idea!

[soft matrix]

!d typing.Annotated

[rough sluice]

typing.Annotated

typing.Annotated```

[soft matrix]

oh wait you use get_args dont you

[devout barn]

To my surprise, alot of the typing internals seem to be undocumented which was troubling as I was creating an introspection tool which accounted typing in runtime.

[devout barn]

Yes I can, with include_extras to be true, but I wanted to have a special case with Annotated which is why I opted for accessing __metadata__

Is typing lib prone to making changes to internal implementation details without prior notice? I sure hope not.

[soft matrix]

yes :)

it was basically all changed in 3.7

but i dont think thatll happen again

[devout barn]

I remember my testing failing with ForwardRef on 3.7 so yeah.

they didn't have is_class or something

[devout barn]

I am guessing not but is there is a flavor of TypeGuard which somehow does require that the function does not return a bool, like how could this example implement a guard?

def check_and_return(
    obj: T,
    expected: type[T],
    exc: type[BaseException] = TypeError,
    template: str = "Expected object `{obj}` to be of type {expected} got {got} instead",
) -> T:
    if not isinstance(obj, expected):
        msg = template.format(obj=repr(obj), expected=expected, got=type(obj))
        raise exc(msg)
    return obj

it's intended usecase is something like this:

valuex: TypeX = check_and_return(should_return_valuex_most_likely(), TypeX)

so that the annotation here can be simply inferred

[soft matrix]

i think youre looking for an asserting type guard which currently isnt a thing

but you can do this with overloads

[wraith linden]

Any idea why both Pyright and MyPy reject the following? ```py
max(map(abs, (1, 2, 3)))

[summer berry]

https://github.com/python/mypy/issues/6697 it's a known issue

[runic sleet]

is there a way to "alias" the overloads of another function without manually doing it all over again?

from typing import overload

@overload
def convert(x: str) -> int: ...

@overload
def convert(x: int) -> str: ...

def convert(x):
    if isinstance(x, str):
        return int(x)
    else:
        return str(x)

def double(x):
    return convert(x) * 2

reveal_type(convert(15)) # str
reveal_type(convert("15")) # int
reveal_type(double(15)) # Any <---

[spiral fjord]

hmm, according to pylance that's always int

[trim tangle]

That's because it does some bidirectional inference. It basically substitutes the 15 in

[trim tangle]

Nope. It's a bit of a pain

[runic sleet]

[trim tangle]

I usually try to avoid overloads altogether

They don't compose well, as you demonstrated

Can you tell more about your use case maybe?

[runic sleet]

so I have a function, view which, depending on the type of the source object, returns a specialized subclass of View, and currently this types fine:

from einspect import view

v1 = view("Hello!")
reveal_type(v1) -> "StrView[str]"

v2 = view(500)
reveal_type(v2) -> "IntView[int]"

v3 = view(["a", "b"])
reveal_type(v3) -> "ListView[list[str]]"

but View has an instance method move_from which moves memory from another view, and returns a new instance of the other view onto the current address.
This also types fine:

V = TypeVar("V", bound=View)

def move_from(self, other: V) -> V:
    ...

str_view = view("Hello!")
new_view = str_view.move_from(view(500))

reveal_type(new_view) -> "IntView[int]"

but essentially I want to make move_from call view on the provided object if it's not already a view, so these will work the same:

def move_from(self, other: ???) -> ???:
    if not isinstance(other, View):
        other = view(other)
    ...

str_view.move_from(view(500))
str_view.move_from(500)

but it seems not possible to just type the return type of move_from as "whatever the return type from view(arg) is"

[trim tangle]

Why would one use view(42) and not, say, IntView(42)?

Also, do you really have to return IntView and not something like View[int]?

[runic sleet]

those would be the same yeah, but I guess the overloaded view is convenient to return the associated View without needing to specify it

[trim tangle]

What is a view in your case?

[runic sleet]

since the subclasses offer additional properties and methods based on the PyObject struct of that object

[trim tangle]

I suppose what you essentially want is a "type map" mapping key types to value types

Which is not something Python has (yet?)

Oh wait wait

Nope

[soft matrix]

There's stuff on typing sig discussing it

[carmine phoenix]

what IDE is that?

[soft matrix]

looks like vsc

[plain dock]

pretty sure it's pycharm

[carmine phoenix]

does anyone know why I'm getting a warning from PyCharm saying: Expected type 'list[tuple[str, int]]', got 'list[str]' instead when I'm returning list(dict.item()), which should be a list of tuples...
this is the code:

def get_smth() -> dict[str, int]:
    return {'a': 3, 'b': 2}
def blah_blah() -> list[tuple[str, int]]:
    dct = get_smth()
    return list(dct.items())  # warning: Expected type 'list[tuple[str, int]]', got 'list[str]' instead

>>> blah_blah()
... [('a', 3), ('b', 2)]

[soft matrix]

last i checked pycharm didnt have inlay typehints

but i always turn it off so who am i to question that

[carmine phoenix]

really? I havent seent that feature in Pycharm that it auto-writes the type hintts

[soft matrix]

actually yeah i think they are right

[oblique urchin]

looks like a bug

[carmine phoenix]

altough I've seen it on other JetBrains products (for Java)

[carmine phoenix]

did you also get that warning?

[oblique urchin]

I don't use pycharm

[carmine phoenix]

and what type-checker do you have ?

oh

so is the issue with PyRight or Python's ItemView object?

[oblique urchin]

the issue is with pycharm's type checker I would expect

just tried with pyright and it handles it fine

[carmine phoenix]

what? isnt that PyCharms default type-checker?

[plain dock]

pyright (VSC's type checker) doesn't even complain on strict mode, which usually yells at everything

[oblique urchin]

no, pycharm has its own builtin typechecker

pyright comes with vscode

[carmine phoenix]

Whats the name of PyCharm's builtin type-checker?

I just want to know who to blame LOL

[oblique urchin]

I don't think it has its own name

[carmine phoenix]

so where can I open a (GitHub) issue?

[oblique urchin]

I think pycharm's issue tracker is some JIRA instance

[soft matrix]

you need to use youtrack

[carmine phoenix]

I cant stand those youtrack ads

every time I open youtube...

[rare scarab]

It's a first party solution, youtrack

[runic sleet]

pycharm

[runic sleet]

https://plugins.jetbrains.com/plugin/19538-python-inlay-params

also it's not really auto writing a type hint it's inlay parameter hints, the first parameter of that function is named obj

though it does also give inlay type hints for function returns

[rare scarab]

Vscode has that too

[runic sleet]

does it matter what "object" a type var is? Or are they the same if they're the same name?

for example importing a T = TypeVar("T") or creating a new one in the module, are there any differences?

or I guess is importing typevars already a bad idea?

[runic sleet]

also what's up with this for mypy

error: Function "ctypes.pointer" is not valid as a type  [valid-type]

and also

error: Function "ctypes.POINTER" is not valid as a type  [valid-type]

neither of those can be used for typing?

[slender timber]

I'd rather not import them if they don't have any bound i e. not special, since they are super ambiguous names like T and T_co. Its more like languages that directly support <T>, in the future Python's getting the [T] syntax making imports useless anyway

[slender timber]

Yes I just use Any, ctypes typing ie fucked

[runic sleet]

is there really a point to type checking if you use Any 🥴

[worldly bramble]

how do you properly type overload something like this?

from dataclasses import dataclass

@dataclass
class Square:
    length: int | float
    def area(self) -> int | float:
        return self.length * self.length

based on the type of length, area can output the corresponding type

[tranquil turtle]

Make Square generic
Or just use float everywhere because int is subclass of float

[stray summit]

im wondering, is anyone working on meta typing in the form of TypeOf[somevariable] , ReturnTypeOf[somecallable], FirstValid[a, b]

i have a function with dozens of overloads, and i cant get the overloads of something that defers to it right

i'd really like to be able to spell it like

def get(
    key: str,
    default: _D|None = None,
    convert: Callable[[str], _T]|None = None,
) -> FirstValid[ReturnTypeOf[convert], str] | TypeOf[default]:
    ....

[worldly bramble]

thanks, that was an over simplified example
but generics makes sense 👍

[carmine phoenix]

thats pretty cool!

[mortal cipher]

is there a way to achieve something akin to declaration merging in typescript, wherein you can declare additional methods and properties on a class in order to have the type checker recognize them when that typing file is included in the project? context: i have a module system where some modules add additional properties to objects based on whether that module received the request or not, and right now it doesn't typecheck

[trim tangle]

I don't think that's possible

[runic sleet]

why doesn't this work 😔 , can I not nest a generic with typevars in another generic?

class PyDictObject(PyObject[dict[_KT, _VT]]):
    ...
    @classmethod
    def from_object(cls, obj: dict[_KT, _VT]) -> PyDictObject[dict[_KT, _VT]]:
        """Create a PyDictObject from an object."""
        return super(PyDictObject, cls).from_object(obj)

d = {"a": "dog", "b": "cat"}

obj = PyDictObject.from_object(d)
reveal_type(obj)
>> Revealed type is "PyDictObject[Any, Any]"

for context PyObject is defined as such

class PyObject(ctypes.Structure, Generic[_T]):

[tranquil turtle]

(you dont need from_object method because you are not doing anything in it)

[runic sleet]

well, I'm trying to grab the KT, VT for typing

but yes it functionally doesn't do anything

[tranquil turtle]

Just make good annotation on PyObject.from_object

[runic sleet]

it's annotated with Self

which isn't currently supported by mypy

[runic sleet]

if I remove this from_object method:

note: Revealed type is "Self?"
error: Self? has no attribute "GetItem"  [attr-defined]

[slender timber]

Well not everywhere

Just problematic

Plus mypy always has some bugs lying around

[rare scarab]

I had a thought. Is there a flake8 extension that controls what imports you can use from typing and typing_extensions? Just to make sure you don't accidentally import a type in the wrong python version.

[oblique urchin]

type checkers (and the runtime when you try to import the module) will tell you when you import from typing before you can

the reverse doesn't really matter but probably someone has written a flake8 plugin

[rare scarab]

So the solution is to use tox

[lethal solar]

How can I type my protocol like "the two methods return the same type" but without need to define this type when typing using my protocol

L = TypeVar('L', covariant=True)

class LanguageImplementation(Protocol, Generic[L]):  # or Protocol[L]
    @staticmethod
    def from_emote(emote: str) -> L | None:
        ...

    @staticmethod
    def from_discord_locale(locale: Locale) -> L | None:
        ...

t: LanguageImplementation    = Smthg()
#                       ^^^ dont want to specify any type

[rustic gull]

im making a GUI that has 3 windows and they can go to each window and have radiobuttons at 2nd and go to third but it says "local variable ___ referenced before assignment" wat do i do

[serene raft]

Not ask in #type-hinting lol

[spare cloud]

class SetOwnerMixin:
    def perform_create(self, serializer):
        owner = self.request.user
        try:
            serializer.save(
                owner=owner,
            )
        except IntegrityError as e:
            raise DuplicateObject

Pylance says: Cannot access member "request" for type "SetOwnerMixin"
Member "request" is unknown

How can I fix that?

[harsh lantern]

define request

[trim tangle]

typing mixins is kind of an open question

[tranquil turtle]

You should annotate self.request in class body

[slender timber]

!pip flake8-typing-imports that's something

[rough sluice]

flake8-typing-imports v1.14.0

flake8 plugin which checks that typing imports are properly guarded

[lethal solar]

Anyone?

[trim tangle]

If you don't specify which type it is, how do you use the type?

[lethal solar]

I want something like "it can be any type, doesn't matter what it is, but this two method must return the same type"

[trim tangle]

why do you want this?

[lethal solar]

In fact I changed my code, so I don't need something like this anymore
but I was just curious if it was possible ? It's a protocol, this kind of things should be possible, right?

[elfin nexus]

Hey

are typing.Optional[TYPE] and TYPE | None the same thing, if so which one is preferred and why

[trim tangle]

Yes they're the same, but the latter is only available since 3.10

I'd use the latter because it's shorter and more explicit

[lethal solar]

TYPE | None is prefered for python version > 3.10 ig
it is more concise

[spiral fjord]

I think Optional[TYPE] or TYPE | None is mostly down to preference, if on the correct version.
I'd avoid Union[TYPE, None]

[elfin nexus]

so if i want to support older python versions eg 3.7 i have to use Optional[TYPE]

[trim tangle]

Your two methods always return an object of the same type, whatever they do you could just say: "L is actually object" and since everything is an object, your methods' results will always share a common type

[trim tangle]

yes

[trim tangle]

or e.g. if one method returns a string and the other returns an integer, you could say they both return int | str

[lethal solar]

the two methods can be implemented with all sort of thing, but I want that this two methods return the same type, no matter what it is

[trim tangle]

well, suppose I have this:

class LanguageImplementation:
    @staticmethod
    def from_emote(emote: str) -> str:
        ...

    @staticmethod
    def from_discord_locale(locale: Locale) -> int:
        ...
``` does it fit your protocol? (let's ignore the None for now)

[lethal solar]

no it doesn't, "str" and "int" should be just a single unique type

[trim tangle]

Both methods return str | int though

[lethal solar]

but I want them to return like only str

[trim tangle]

what about this?

class LanguageImplementation:
    @staticmethod
    def from_emote(emote: str) -> object:
        ...

    @staticmethod
    def from_discord_locale(locale: Locale) -> object:
        ...

[lethal solar]

this should match the protocol yes

[trim tangle]

but the previous example is a subtype of this

because from_emote can return a string, and from_discord_locale can return an integer

(as they're both objects)

[lethal solar]

yes ofc
but I want the protocol to be accepted only if this two methods return the same type
and str != int

[spiral fjord]

t: LanguageImplementation    = Smthg()
#                       ^^^ dont want to specify any type```
You don't specify the type there, you specify the type on the return of Smthg

[trim tangle]

what do you mean by "the same type"?

the annotations should be the same?

[lethal solar]

with generic it does what I want, but I don't need to know what it is

[lethal solar]

yes

[trim tangle]

could you perhaps explain the larger problem you're solving here?

[lethal solar]

nothing more than I explained
just want a protocol that describe two methods that should have the same return type annotation
but I think I need to use Generic since I will also need to know the return type when I will use this methods

[trim tangle]

This is generally not possible because of "subtyping". Something like this:

class Foo(Protocol):
    def bar(self) -> str:
        ...

    def baz(self) -> int:
        ...

class Bar(Protocol):
    def bar(self) -> str | int:
        ...

    def baz(self) -> int | list[int]:
        ...
``` here `Foo` is a "subtype" of `Bar` because if some object matches the `Foo` protocol, then it definitely matches the `Bar` protocol.

So if you have something like this:

class Foo(Protocol):
    def bar(self) -> str:
        ...

    def baz(self) -> int:
        ...
``` it will always be a subtype of this: ```py
class Bar(Protocol):
    def bar(self) -> object:
        ...

    def baz(self) -> object:
        ...

but yeah generally you just want to know what type both of the methods return

[tranquil turtle]

I think Optional is deprecated

[spiral fjord]

It's not listed here.
https://docs.python.org/3/library/typing.html#deprecation-timeline-of-major-features

[tranquil turtle]

Yes, it is not deprecated, im wrong

[runic sleet]

if you're using pycharm the built-in version compatibility inspection works quite well

[languid magnet]

how do we get that in vscode?

pyright?

pylance does the same?

[rare scarab]

It's a setting. Search for inlay

[hallow flint]

mypy will do this; just make sure to specify eg --python-version 3.7 if you're dev-ing with a higher python version than what you claim to support

[peak echo]

def f(cls: T) -> ?```
It returns this so basically
```py
type(cls.__name__, (OtherClass,), dict(something=something))```
Keeps attributes of `cls` but inherits `OtherClass`, how do I make a correct typehint for function `f()`?

Would it be fine to use T | Type[OtherClass]?

[soft matrix]

no youd need an intersection type (which doesnt exist)

[peak echo]

💀

[carmine phoenix]

is there a type hint for a Sorted Iterable?

[tranquil turtle]

no

i dont think it would be useful

you can try to write your plugin for mypy

[carmine phoenix]

if you have a function that takes a sorted iterable wouldnt it be nice to annotate the parameter like that?

[soft matrix]

use a NewType

thats the best thing i can suggest

[carmine phoenix]

the tricky part is verifying the Iterable is actually sorted

[soft matrix]

SortedIterable seems like a really hard thing to test

[carmine phoenix]

I just did this for now

SortedList: TypeAlias = list

[soft matrix]

if you want a sortedlist thats not too bad

you could actually just have a subclass of list that calls the super init and then just self.sort()

[carmine phoenix]

how am I supposed to use NewType?

[soft matrix]

and that is then a concrete type that can be tested for

have you read the docs for it?

if they dont make sense im happy to explain it

[carmine phoenix]

I have used it before, but I'll have a look at the docs...

*havent lol

[carmine phoenix]

wouldnt it be nice if the second time we call list it would give a warning like: "Iterator is already exhausted"...

x: Iterator = (x for x in range(5))
print(list(x))
print(list(x))

[soft matrix]

probably but that sounds ridiculously hard to check, there might be a flake8 plugin for it

[carmine phoenix]

so everything is too hard to check 😅

[oblique urchin]

I think this is "linear types"

[carmine phoenix]

I guess somebody's got to implement it ...

[tranquil turtle]

I think it is possible if vars can change their types after some operations

[runic sleet]

is there any way to re-typehint an inherited parent method without doing anything runtime

class Base(Generic[_T]):
    def __init__(self, obj: _T) -> None:
        self.obj = obj
    def to_object(self) -> _T:
        return self.obj

class ListBase(Base[list[_V]]):
    def get(self, index: int | slice) -> _V:
        return self.obj[index]

it seems I'd have to retype the init to get access to the _V typevar?

also it seems the nested generic in the other generic isn't inferable by mypy

any way to do this typing in current python?

I also tried

class ListBase(Base[list], Generic[_V]):
    ...

but mypy gives an error for incompatible generic

[fathom river]

what's self.x?

[runic sleet]

typo, self.obj

[fathom river]

I'm not too sure if I understand what you're trying to do. Say I've got this setup

lb = ListBase(["foo", "bar"])

What should obj be? What should get return? index can also be a slice, so get can also return list[_V], should it?

[runic sleet]

obj would be ["foo", "bar"], so list[str], and bound to TypeVar("_T")

[runic sleet]

well slice support is simple with overloads, provided this works

class ListBase(Base[list[_V]]):
    @overload
    def get(self, index: int) -> _V:
        return self.obj[index]

    @overload
    def get(self, index: slice) -> list[_V]:
        return self.obj[index]

    def get(self, index: int | slice) -> _V:
        return self.obj[index]

the issue is there seems to be no way to get at the _V typevar

[solemn sapphire]

from typing import Callable
from operator import invert

a_func: Callable[..., int] = invert # ERROR: _T_co@invert incompatible with int

Aren't ints invertible?

How would I bind that _T_co@invert to an int?

[trim tangle]

a_func: Callable[int, int] = invert
```?

[solemn sapphire]

sorry that was a bad minimal example, this one should be more accurate

from typing import Callable
from operator import invert, add

cond: bool = True
a_func: Callable[..., int] = invert if cond else add # ERROR: _T_co@invert incompatible with int

[trim tangle]

well, this says that it accepts any arguments and returns an integer. which isn't right (what if you pass in a float?)

[solemn sapphire]

I had before a TypeVarTuple with both the constrains being int, something like
Ints = TypeVarTuple("Ints", int, int)
and then did
a_func: Callable[*Ints, int]

But then the type checker complained that there was only one Generic argument present, so I resorted to Callable[..., int], hoping that the return type would inform the input types as well.

For example invert's signature is,
(__a: _SupportsInversion[_T_co], /) -> _T_co
So in this case specifying the return type would be sufficient right?

[trim tangle]

oh hmmmm

not sure

[solemn sapphire]

Oh well :(

[oblique urchin]

TypeVarTuple can't have constraints

[solemn sapphire]

Oh, you're indeed correct, I think that was an incorrect refactor of me changing TypeVar to a TypeVarTuple

Do you have an idea on how do I solve it?

[oblique urchin]

Not sure what you're trying to achieve

[solemn sapphire]

type a function that takes a variable number of integers and returns an integer

so could be either add, xor, invert and so on

[oblique urchin]

maybe a callable protocol with def __add__(self, *args: int) -> int: ...

though what's the use of such a type if you don't know how many ints it will accept?

[solemn sapphire]

It will either accept one or two ints
So invert will accept one while add accepts two

Is that bad design?

[oblique urchin]

it's not type safe

like there's no type-safe way to call a function of this type

[solemn sapphire]

To not make it an XY problem, I have a gate class that looks like

class Gate:
    def __init__(self, gate: Callable[..., int], arity: int, name: str) -> None:
        self.arity = arity
        self.gate = gate if arity<=1 else partial(reduce, gate)
        self.name = name

so either it could be some n input logic gate or a not gate.

I see, so I shouldn't try to make invert be a part of this gate class?

[oblique urchin]

maybe you should have separate classes for 1-input and 2-input gates? It still seems awkward to use this Gate class

[solemn sapphire]

maybe you're write, I am going from writing random scripts to properly structured code 😅.
Hence this abomination

[solemn sapphire]

Wouldn't that be limiting?

How would you approach it?

[oblique urchin]

Still not entirely sure what the Gate classes are for, but I do think I might use separate UnaryGate and BinaryGate classes

[solemn sapphire]

I wanted some nice way to generate an n input logic gate, I was doing something like

and3 = partial(reduce, and_)
or3 = partial(reduce, or_)

and then these things could accept 3 numbers and give one output (though they could accept more)

yeah this doesn't look great
I'll think more about this. I think having a UnaryGate and a NGate for an n input gate would be a good idea

Thanks!

[solemn sapphire]

I am sorry for replying a little late.
Would you mind elaborating what is the type-unsafety in having a function accept a variable number of integer arguments?

Do you mean its as good as not having any type annotation for such a function in the first place?

[oblique urchin]

you don't actually know whether any call to that function will work

[solemn sapphire]

I am a little confused, let's say for the sake of argument that the function is

>>> def f(*a):
...     print(sum(a))
... 

Then the following calls work

>>> f(1)
1
>>> f(1,2,3,4)
10
>>> f()
0

what would it mean in this context that this function is not guaranteed to work, given the information that it accepts a variable number of integer arguments?

[oblique urchin]

your functions aren't like that though. They are either def f(a) or def f(a, b)

so if you see a call like f(1) or f(1, 2), you don't know that it will work, because the function may accept either one or two arguments

[solemn sapphire]

ah!

right, and that's why I was storing the arity as well in the Gate class to special case arity=1 and properly provide the arguments.
I have since moved on from that terrible approach and have it separated into def f(a) (A unary gate) and def f(a, b, *c) (a binary and above arity gate) and that seems to work much more nicely as well.

I actually recognise you from https://github.com/python/typeshed/issues/9143#issuecomment-1327016840 :)

Thanks for you help!

[novel elbow]

How can I use mypy to parse python code and print a dict with all variables along with their types

[soft matrix]

with lots of suffering

https://github.com/Gobot1234/steam.py/blob/main/docs/extensions/annotations.py probably comes pretty close to what you need

[novel elbow]

@soft matrix thanks, it demonstrated perfectly how to use build

I ended up doing it like this:

from mypy.build import BuildSource, build
from mypy.options import Options

result = build([BuildSource("<string>", __name__, "import typing;var: typing.Any = 5;print(var)")], Options())
print(result.files[__name__].names["var"].type)```

[novel elbow]

now the challenge is going to be linking this to the ast

[soft matrix]

hence ^

[novel elbow]

why are there no docs on it 💀

[soft matrix]

cause thatd be too easy

[tranquil turtle]

reveal_locals() exists

you will get something like that

[heady flicker]

#TIL

[runic sleet]

@oblique urchin do you know of PEPs to support subscripting generic typevars in type hints? Since it's inferable, I don't see why it couldn't be type hinted

or was there previous discussions against this

from typing import TypeVar, overload

T = TypeVar('T', list)
V = TypeVar('V')

class ListWrap(list[T]):
    def __init__(self, ls: T[V]): # <- subscripting a generic typevar
        self.ls = ls
        
    @overload
    def get(self, i: int) -> V: ...
    
    @overload
    def get(self, i: slice) -> list[V]: ...
        
    def get(self, i):
        return self.ls[i]

[soft matrix]

There's going to hopefully be a pep for higher kinded typevars which might make this possible

[stray summit]

It's unlikely to happen soon, the type system is pretty static and they rather add hacks like the dataclass transforms than ways to safely generate annotations in code

[soft matrix]

idk im hopeful it will be a 3.13 feature

https://github.com/python/typing/issues/1311 and https://github.com/python/typing/issues/548

[runic sleet]

is there a reason mypy type hints generic ctypes with a notation that is not actually possible to evaluate at runtime?

x = ctypes.py_object(1)
reveal_type(x)
>> Revealed type is "ctypes.py_object[builtins.int]"

If assigned at runtime:

    t = py_object[int]
        ~~~~~~~~~^^^^^
TypeError: type 'py_object' is not subscriptable

[tranquil turtle]

stubs dont match runtime

or bad plugin

[livid viper]

hey 👋 just a quick question: suppose I construct a set of classes like those below, such that I know MyClassInt is the only subclass of MyClass[Int].

class MyClass(Generic[T], ABC):
    def do_something(x: T) -> T:
        ...

class MyClassInt(MyClass[int]):
    def do_something(x: int) -> int:
        return x + 1

class MyClassBool(MyClass[bool]):
    def do_something(x: bool) -> bool:
        return (not x)

Given this condition, MyClassInt and MyClass[Int] should be usable as type aliases for each other.

I'm wondering if there's a way (possibly through a MyPy plugin) to declare such equivalences / rewrite rules? Essentially this would amount to allowing downcasts from MyClass[Int] to MyClassInt (which would always be safe as MyClassInt is the only non-abstract class under MyClass[Int]).

yes, I miss typeclasses :(

[trim tangle]

I don't think that's possible. Maybe you could explain why you need this?

[tranquil turtle]

MyClassInt: typing.TypeAlias = MyClass[int]

[trim tangle]

Oh, you mentioned plugins. Yes, I think that's theoretically possible

I have played around with mypy plugins for a bit, I was in great pain 🙂

maybe it's just a skill issue

[livid viper]

the issue is I want ad-hoc polymorphism (i.e. defining custom implementations of functions specified in MyClass based on the type of the generic)

[runic sleet]

is there a reason mypy errors on using a more permissive overload later in the chain?

@overload
def fn(x: list) -> list:
    ...

@overload
def fn(x: Any) -> str:
    ...

error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]

it doesn't seem to have issues with inference, despite the error

reveal_type(fn([1, 2, 3]))
>> Revealed type is "builtins.list[Any]"
reveal_type(fn(100))
>> Revealed type is "builtins.str"

[viral birch]

looks ambiguous to me even though the inference seems to work, if you pass a list it can be either of the overloads

[soft matrix]

i was under the impression that overloads were selected by the first that matched

[trim tangle]

that Any could still be a list, in which case the function returns a list

like ```py
x: list = []
y: object = x
fn(x) # list
fn(y) # str? but actually a list

the issue is that you can't have an "any-but-definitely-not-a-list"

[tranquil turtle]

Any & ~list[Any]

[runic sleet]

yeah but, if we define overload to be sequential, surely list matches before Any?

I'm not sure if the pep for overload actually talks about that affirmatively or otherwise

but essentially simulating this runtime pattern

if isinstance(x, list):
    return ...
else:
    return ...

pylance/pyright seems to follow this and infers the first matching overload without warnings

[tranquil turtle]

fn(Any) should be str | list

[hallow flint]

this error is a little bit of a lint. ignoring it isn't the worst and mypy will still basically infer types the way you want.

the reason mypy is warning about this code is because it's unclear how you'd create an implementation that discriminates between those types at runtime. your if isinstance(x, list): isn't right, because you could have a list that is type erased to Any, and mypy will think it's a str:

def calls_fn(x: Any):
    fn(x) + "some str"

calls_fn([])  # oops!

[runic sleet]

hm... fair point

[runic sleet]

so uh, appears I have 2 options

• Patching the function/class's __annotations__ by regex at runtime before calling get_type_hints
  or
• Using fishhook to patch on a __class_getitem__ to ctypes.pointer in a if not TYPE_CHECKING: conditional

not sure which one is less cursed 😔

or option 3. PR typing-extensions to add a generic subscript-able pointer type that evaluates in runtime

and option 4. PR cpython/ctypes to make ctypes.pointer support __class_getitem__

though not sure what the latter 2 would entail, does ctypes typing additions require a PEP?

[soft matrix]

No

[tranquil turtle]

class Descriptor(Generic[T]):
    @overload
    def __get__(self: TDesc, inst: None, owner: type[Owner], /) -> TDesc:
        ...
    @overload
    def __get__(self, inst: Owner, owner: type[Owner] | None, /) -> T:
        ...
    def __get__(self: TDesc, inst: Owner | None, owner: type[Owner] | None, /) -> T | TDesc:
        if inst is None:
            return self
        # actual descriptor logic
        return ...
``` why it is so painful to annotate descriptors... 😭

class Descriptor(Generic[T]):
    @descriptor_get[Owner, T] # some type-checker magic
    def __get__(self, inst, owner, /): # it is automatically annotated, all overloads are defined implicitly
        if inst is None:
            return self
        # actual descriptor logic
        return ...

[soft matrix]

macros 🤤

[blazing nest]

Any suggestions for how I might accomplish this? ```python
class Wrapper(Generic[P, RT]):
callback: Callable[P, RT]

def __init__(self, callback: Callable[P, RT]) -> None:
    # NOTE: Obviously my real __init__ has more parameters which
    # get more complicated, this is just a repro
    self.callback = callback

@classmethod
def from_callback(cls, callback: Callable[P, RT]) -> Self[P, RT]:
    return cls(callback)

Expected no type arguments for class "Self"

[summer berry]

Can you use Wrapper instead of Self there

[soft matrix]

wait a couple of years :)

[blazing nest]

Hmm, yeah that could work as a work-around. I think the benefit of parametrization will outweigh loosing ability to subclass

[soft matrix]

if someone subclasses they can realistically just re-implement the method in if TYPE_CHECKING

and thats pretty easy

[blazing nest]

Yeah I suppose so, it is not that much of a headache

[soft matrix]

i recommend following this https://github.com/nekitdev/peps

so much so that i felt the need to paste that twice

[trim tangle]

-> Self?

[runic sleet]

why is get_type_hints on 3.8 so painful 😔 https://github.com/ionite34/einspect/actions/runs/3793135458/jobs/6449968102

number of TypeError: 'type' object is not subscriptable I've gotten...

also need to runtime replace collections.abc imports with typing so they're usable as generics

having to choose between breaking 3.8 or using a "deprecated" import 😩

[trim tangle]

I think the latter is much better

it's not going to be removed any time soon, and it's not a very difficult change to do

[runic sleet]

journey to fix typing continues 😔

I don't even know what's wrong anymore

https://github.com/ionite34/einspect/actions/runs/3794924052/jobs/6453525797#step:8:56

everything works except 3.8 windows and macos

[brazen jolt]

seems like some pytest issue

maybe one of the dependencies doesn't play well with win/mac

though that's super weird

are you using some pytest extensions? or some other optional dependencies?

[wicked scarab]

class A: ...

class B:
    val = 1

class C: ...


def get_B_positions(arg: list[A | B | C]) -> list[int]: ...


def foo(arg: list[A | B | C]):
  for i in get_B_positions(arg):
    arg[i - 1] = arg[i].val + 2

How do I typehint so that arg[i] would always be B? Currently, the type-checker (pyright) sees arg[i] as A | B | C

[brazen jolt]

other than explicitly casting it as B, you can't do this. Pyright isn't able to infer that nth element in a list is of specific type. This would be a bit easier with tuples, which can have different elements for different positions, however even with those, you'd need to define all of them in a single type-hint, you can't do it dynamically like this i.e. every element on these dynamically obtained indices is of type B

you could also consider just checking each element directly, rather than obtaining the indices to them, in that case, pyright would be able to infer that that element is of the checked type

i.e. ```py
from typing_extensions import reveal_type

elements: list[A | B | C]
for element in elements:
if isinstance(element, A):
reveal_type(element) # Type inferred as A
elif isinstance(element, B):
reveal_type(element) # Type inferred as B
...

If for some reason you insist on using the list of indices pointing to which elements are of specific type, you could just explicitly cast them into that type though.

from typing import cast
from typing_extensions import reveal_type

elements: list[A | B | C]
for i in get_B_positions(elements):
    element = cast(B, arg[i])
    reveal_type(element)  # element is of type B

@wicked scarab ^

[wicked scarab]

ooh, okay I see, thanks

[wicked scarab]

another question,

class A: ...
class B: ...
class C: ...

class FooIdentity:
  identity1: bool = False

def get_identity(classes: list[A | B | C]):
  identity = FooIdentity()
  for i in classes:
    if isinstance(i, B):
      identity.identity1 = True

  return identity

def main(classes: list[A | B | C]):
  identity = get_identity(classes)
  if not identity.identity1:  # This should basically means the same as "if the class B not in classes" right?
    reveal_type(classes)
``` Why does `classes` still `list[A | B | C]` here?

[trim tangle]

that's a very deep level of analysis that would be required to deduce that

probably something around "dependent typing"

And in fact, that wouldn't be a correct judgement. Consider this:

things: list[A | C] = []

def main(classes: list[A | B | C]):
    global things

    identity = get_identity(classes)
    if not identity.identity1:
        things = classes

hmm: list[A | B | C] = [A(), A(), C()]
main(hmm)
hmm.append(B)  # oops, now "things" has a B

[wicked scarab]

any workaround for that?

[trim tangle]

# type: ignore, I guess?

depends on your particular case

[wicked scarab]

hm yeah, but too many type ignores is bad right?

[trim tangle]

In most programs yes, it's not the best thing.

Maybe you could explain more about the bigger problem you're solving?

[pliant vapor]

is there a way to type hint nested lists of arbitrary nesting level?
like [1, 2, 3] or [1, [2], [[3, 4]]] would both work, but [1, [2], [[3, 'a']]] wouldn't
i was thinking something like

N = list[N|int]

but that doesn't work

[trim tangle]

N = list[Union["N", int]]

I don't remember if mypy supports this already

[wicked scarab]

it's something like this

def no_B(classes: list[A | C]): ...

def main(classes: list[A | B | C]):
  identity = get_identity(classes)
  if not identity.identity1:
    no_B(classes)
``` But if I pass it directly to `no_B` it will still produce an error: `error: Expression of type "list[A | B | C]" cannot be assigned to declared type "list[A | C]"`

[trim tangle]

Oh yeah it does
https://mypy-play.net/?mypy=latest&python=3.11&gist=e89464b642b35696da9e21a82664d7dd

[wicked scarab]

or maybe if there's a way I can pass list[A | C] directly to list[A | B | C], that could also be a solution

[trim tangle]

Can't you do something like ```py
def ensure_no_bs(classes: list[A | B | C]) -> list[A | C]:
return [x for x in classes if not isinstance(x, B)]

[pliant vapor]

oh cool thx

[wicked scarab]

yeah I can, but it's kinda slow, isn't it? I have gotten myself a lot of loops already

[trim tangle]

If it turns out too slow for your needs, benchmark and improve performance 🙂

Maybe you could explain the real case where you need this? The A/B/C make it kinda abstract

and, well, you're doing a loop in any case

[wicked scarab]

class EquationIdentity:
  variable_count: dict[Variable, int] = {}
  prove: bool = False


def get_equation_identity(parsed_groups: list[Group | Operator | Equals | ParenthesizedGroup]) -> EquationIdentity:
  identity = EquationIdentity()
  for group in parsed_groups:
    if isinstance(group, Equals):
      identity.prove = True
    if isinstance(group, (Equals, Operator, ParenthesizedGroup)):
      continue

    if (var := group.variable) is not None:
      identity.variable_count[var] = identity.variable_count.get(var, 0) + 1
    
    return identity


def determine_equation_type(parsed_groups: list[Group | Operator | Equals | ParenthesizedGroup]) -> int:
  identity = get_equation_identity(parsed_groups)
  if len(identity.variable_count) == 0 and not identity.prove:
    return solve_basic(parsed_groups)  # type: ignore
``` this is a part of my code, this basically determines the type of a parsed math equation. `solve_basic` here takes `list[Group | Operator | ParenthesizedGroup]` without the `Equals` class. But I can't pass `parsed_groups` to it directly as it will produce pyright's error that I've shown before.

[trim tangle]

What is prove?

[wicked scarab]

which one

[trim tangle]

the attribute of EquationIdentity

[wicked scarab]

identity.prove? It's to check if the equation has an equal sign in it

and I might need it somewhere else, so I don't check it directly inside determine_equation_type. I put it in the identity instead

[trim tangle]

Well if there's no equal sign, it's not an equation, is it?

[wicked scarab]

uh yeah, it should be an equation solver. But it supports basic problems also like "1 + 1"

[trim tangle]

So you have two kinds of problems: equations and computations?

[wicked scarab]

That's for later, for now just on the typehints, they're confusing

why can't I just pass list[A | B] for example to an argument that accepts list[A | B | C] as they both have the same A | B

[trim tangle]

def hmm(things: list[A | B | C]) -> None:
    things.append(C())

only_ab: list[A | B] =  []
hmm(only_ab)

Generally, that's because lists are "invariant"
I have a WIP tutorial on all this: https://decorator-factory.github.io/typing-tips/tutorials/generics/ maybe it will be helpful

[wicked scarab]

I'll walk through it, thanks

[wicked scarab]

yeah this raises Argument of type "list[A | B]" cannot be assigned to parameter "things" of type "list[A | B | C]" in function "hmm"

[trim tangle]

Right, because if that was allowed, there would be a C inside of the list

[wicked scarab]

I got it, thanks

[wraith linden]

Hello. I was wondering if there is a way around the following situation. Consider this file: ```py
from functools import cached_property
from typing import Protocol

class A(Protocol):
@property
def x(self) -> int: ...

class B:
@property
def x(self) -> int:
return 42

class C:
@cached_property
def x(self) -> int:
return 42

a1: A = B()
a2: A = C() # Pyright doesn't like this.
``` MyPy considers cached_property compatible with property, but Pyright does not (https://github.com/microsoft/pylance-release/issues/1469). Pyright is usually my preferred type-checker out of the two (because it's faster). I'd like to be able to experiment with swapping out property for cached_property on different attributes of an implementation of the protocol, to see what effect it has on performance. What can I do that doesn't involve ignoring types, or switching to MyPy?

[acoustic thicket]

if TYPE_CHECKING: cached_property = property maybe?

i suppose that is kinda sorta ignoring types but i think thats necessary given

In particular, cached_property allows writes but property does not.

[wraith linden]

Yeah, I don't really understand that though, because this is ok apparently: ```py
class D:
x: int = 123

a3: A = D()

[wraith linden]

Ah right nice. That might be what I end up doing. Thanks!

[hallow flint]

Yeah, I’m not sure that erictraut got that one right

[novel elbow]

How do I add extensions to an abstract class?

[soft matrix]

Wdym by an extension?

I'd personally just subclass the abc and add the new attributes and methods

Nothing else will type check correctly

[novel elbow]

https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods

Like extension methods to instances in c#

[novel elbow]

So it would be a mix of abstract classes and default classes?

[brazen jolt]

python doesn't have support for adding such extension methods (at least not without some really cursed code)

any instance of a class like str is it's own complete object, and you can't define a method elsewhere to act as one of string's instance methods

you'll just have to use regular functions, and pass the object as first argument

or subclass it and use the custom type, which has that method added

[novel elbow]

So it's possible to have an abstract class with some functions already defined? Can you give an example of your proposals?

[brazen jolt]

well if you're just subclassing, you don't really need an abstract class, you can just do: python class MyString(str): def word_count(self) -> int: return len(self.split(" ")) It's certainly possible to use a mixin class and define these methods elsewhere, but these mixin classes aren't something you see very commonly used in python and unless you have a good reason to use them, you should probably stick to just simple subclassing like above, but here's how that could look: ```python
class StringMixin:
def word_count(self: str) -> int:
return len(self.split(" "))

class MyString(str, StringMixin):
pass

x = MyString("hello there")
x.word_count() # 2

As for actual abstract classes in python, they're meant more like a base for another concrete class, which can implement all of it's abstract methods, abstract base classes aren't really used as mixins. ```python
from abc import ABC, abstractmethod

class Animal(ABC):
@abstractmethod
def sound(self) -> str:
raise NotImplemnetedError

def __repr__(self) -> str:
    # You can call the abstract sound method here, as ABCs require it to be present,
    # otherwise initialization won't be possible. Child classes will then inherit
    # this repr method, calling their `sound` function.
    return f"{self.__class__.__name__}(sound={self.sound()})"

my_animal = Animal() # TypeError: can't initialize Animal class, missing implementation for the abstract sound method

class Dog(Animal):
def sound(self) -> str:
return "Woof"

my_dog = Dog()
print(my_dog) # Dog(sound=Woof)

This is useful when you have several types that share similar interface. ABCs also easily allow you to have multiple classes that all inherit some functions that internally call some unimplemented method, and so each of the subclasses can provide it's own implementation, and the ABC class itself won't be initializable. You can then use this ABC in type-hints too:

def process_animal(x: Animal):
    print(f"This animal makes {x.sound()} sound")

[novel elbow]

So you can have predefined functions in an ABC class?

[brazen jolt]

sort of yeah, they're just abstract functions that can have some docstring, and type hints, so that type-checkers will recognize these methods, even if they're not actually expected to be used without being overridden. It's often the case that you can end up needing a template class which has a lot of functions that rely on some abstract function, which doesn't have any sensible default implementation, but needs to be present, so you make it an abstract method.

Here's a bit more real-world example: ```python
from abc import ABC, abstractmethod
import socket
import struct

class Writer(ABC):
@abstractmethod
def write(self, data: bytes) -> None:
raise NotImplementedError

def write_utf(self, value: str) -> None:
    self.write(value.encode("utf-8"))

def write_long(self, value: int) -> None:
    self.write(struct.pack(">L", value))

def write_double(self, value: float) -> None:
    self.write(struct.pack(">d", value))

...

class TCPSocketWriter(Writer):
"""Writer sending data over a TCP socket."""
def init(self, socket: socket.socket) -> None:
self.socket = socket

@classmethod
def make_client(cls, address: tuple[str, int], timeout: float) -> Self:
    """Construct a client connection to given address."""
    sock = socket.create_connection(address, timeout=timeout)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    return cls(sock)

def write(self, data: bytes) -> None:
    self.socket.send(data)

class BufferWriter(Writer):
"""Writer storing data into an internal buffer."""
def init(self) -> None:
self.data = bytearray()

def write(self, data: bytes) -> None:
    self.data.extend(data)

class AsyncTCPWriter(Writer):
"""Writer sending data over an asynchronous connection."""
...

[brazen jolt]

In that example above, it clearly doesn't make sense to provide any default/fallback implementation, as you can create writers that write to different things (you could also have a FileWriter for example) and all of the concrete classes just have to implement the write method to be actually usable.

[novel elbow]

Thanks that's basically what I wanted to achieve

[heady flicker]

Sorry for offtop, but I want to wish everyone a happy new year.

(Got this shirt as a present from a coworker)

[tidal fern]

Nice

[carmine phoenix]

pretty nice

is this the correct way to specify how the return type varies based on if the value is True/False

    @overload
    def extract_keywords(self, sentence: str, span_info: bool = False) -> list[str]:
        ...

    @overload
    def extract_keywords(self, sentence: str, span_info: bool = True) -> list[tuple[str, int, int]]:
        ...

    def extract_keywords(self, sentence: str, span_info: bool = False) -> ...:
         ...

The issue is that the type is the same, so I dont think the type checkers will understand (or even check?) the value

[soft matrix]

No that's not how you do this

Use Literal[True] and Literal[False] for the annotation instead of bool

[carmine phoenix]

im all ears

I'll try this, thanks

now it works!

btw should I be concerned that a decorator will rename or do some funky stuff with my method ?

    @overload
    def extract_keywords(self, sentence: str, span_info: Literal[False]) -> list[str]:
        ...

    @overload
    def extract_keywords(self, sentence: str, span_info: Literal[True]) -> list[tuple[str, int, int]]:
        ...

    @normalize_parameter('sentence')
    def extract_keywords(self, sentence: str, span_info: bool = False) -> ...:

[heady flicker]

Depends on the decorator. Do you have its source? Would you be able to provide it?

[carmine phoenix]

sure:

def normalize_parameter(*params: str) -> Callable:
    """
    Instead of doing this at the beginning of each function:
    if not self.case_sensitive:
        word = word.lower()

    We can just decorate the function with: `@normalize_parameter('parameter_name')`

    :param params: parameter names
    """
    def decorator(func):
        def wrapper(*args, **kwargs):

            kwargs.update(dict(zip(func.__code__.co_varnames, args)))  # convert all args to kwargs
            self: TrieDict = kwargs['self']
            if not self._case_sensitive:
                for p in params:
                    value = kwargs.get(p)
                    if value is not None:
                        kwargs[p] = value.lower()

            return func(**kwargs)
        return wrapper
    return decorator

I ask because I heard that a decorator can rename the function and I dont want @overload to get confused or anything like that

[heady flicker]

There are multiple things this decorator could break. Let me give you the correct way to write these:

[carmine phoenix]

🤔

[heady flicker]

from collections.abc import Callable
from functools import wraps
from typing import TypeVar, overload, Union

from typing_extensions import Concatenate, Literal, ParamSpec, Protocol

P = ParamSpec("P")
T = TypeVar("T")


class TrieDict(Protocol):
    _case_sensitive: bool


S = TypeVar("S", bound=TrieDict)


def normalize_parameter(*params: str) -> Callable[[Callable[Concatenate[S, P], T]], Callable[Concatenate[S, P], T]]:
    """
    Instead of doing this at the beginning of each function:
    if not self.case_sensitive:
        word = word.lower()

    We can just decorate the function with: `@normalize_parameter('parameter_name')`

    :param params: parameter names
    """

    def decorator(func: Callable[Concatenate[S, P], T]) -> Callable[Concatenate[S, P], T]:
        @wraps(func)
        def wrapper(self: S, *args: P.args, **kwargs: P.kwargs) -> T:

            kwargs.update(dict(zip(func.__code__.co_varnames, args)))  # convert all args to kwargs
            if not self._case_sensitive:
                for p in params:
                    value = kwargs.get(p)
                    if value is not None:
                        kwargs[p] = value.lower()

            return func(self, **kwargs)

        return wrapper

    return decorator


@overload
def extract_keywords(self, sentence: str, span_info: Literal[False]) -> list[str]:
    ...


@overload
def extract_keywords(self, sentence: str, span_info: Literal[True]) -> list[tuple[str, int, int]]:
    ...


@normalize_parameter("sentence")
def extract_keywords(self, sentence: str, span_info: bool = False) -> Union[list[str], list[tuple[str, int, int]]]:
    pass

However, I advise you to use simpler approaches such as simply normalizing the first argument.

And I advise you to use inspect.signature instead of __code__.co_varnames -- it is a less hacky approach

[carmine phoenix]

Thanks for posting this

Youre type hints are too overpowered 😅

[heady flicker]

Just like my shirt states :DD

[carmine phoenix]

I tried to use inspect.Signature(func).bind() but I was getting some weird TypeError (I'm on Python 3.10), and also I dont think this was available in previous Py versions, so I gotta be caerful if I want to make this package available to a wide range of ppl

[heady flicker]

Which versions are you targeting?

I used inspect.signature on 3.6 last year.

[carmine phoenix]

and btw would you recommend using Union[str, int] instaed of str | int?

[carmine phoenix]

oh damn, than I should use that

[carmine phoenix]

idk, I have this dilemma where im not sure how far back I should go to make my package compatible, I think Python 3.9 is ok, but ideally if I could include 3.7 and 3.8 it will work for all the boomers as well

[heady flicker]

Definitely not. The only reason I used Union was because I wasn't sure which python version you've had so I went for the lowest common denominator: 3.9, because you use typed generics (PEP 585).

[heady flicker]

I usually make all of my libraries compatible with 3.7+, i.e. with all versions that haven't reached end of life.

[carmine phoenix]

Oh damn, then I should use typing.Union and all that stuff to make it backwards compatible. And do you know from whats the latest version that dosent support type hints? i.e: wrap them with quotes

[heady flicker]

Need to take a look into pep 484

[heady flicker]

3.4, I think.

[carmine phoenix]

3.5 it says

but how do you remember the PEP codes if there are like a few hundreds of them

[heady flicker]

3.4 is the latest version that doesn't support type hints :))

[heady flicker]

I only remember like 3. Pep 8, pep 20, and pep 484 :))

[carmine phoenix]

my bad, 3.5 it was introduced

oh lmao

is there like a list of the important ones?

[heady flicker]

Nope. I just like these three.

[carmine phoenix]

because there all multiples of 4?

[heady flicker]

You could do the following too:

from __future__ import annotations


def hello() -> str | None:
    return "Hello world!"


print(hello())

def hello() -> "str | None":
    return "Hello world!"


print(hello())

P.S. Works on 3.7

[heady flicker]

Nah. Because PEP 8 is incredibly important to write consistent and clean code, because PEP 20 is funny and provides some cool guidance to newbies, and because PEP 484 is about typehints which I am really passionate about.

[carmine phoenix]

I'm surprised that discord recognizes the type hints and dosent show it as a string

[heady flicker]

Not sure if that's discord or the markdown rendering library they are using.

[carmine phoenix]

anyways thanks for everything, I will try inspect.Signature() and see if it works now that we use @wraps()

[heady flicker]

Just remember that it's inspect.signature(). inspect.Signature is a bit different

[carmine phoenix]

oh fuck, that might have been the issue

so I do: inspect.signature(func).bind()?

[heady flicker]

Yeah, but you gotta place your arguments into bind()

[carmine phoenix]

oh right inspect.signature(func).bind(*args, **kwargs)

[heady flicker]

Yep. Don't forget about self too if you split self from args

Gotta run now.

[carmine phoenix]

Thanks again brother, have a nice day

[novel elbow]

When using abstract classes, how can I make it inherit the documentation?

[soft matrix]

How are you viewing the documentation?

[novel elbow]

I use the Google style doc strings

Will parsers automatically inherit the documentation?

[soft matrix]

Sphinx I don't think does

Vsc does if you hover over the definition

It was something I was actually going to make an issue about for autodoc to have an option to use inherited doc strings

[autumn glen]

I have a type in a C extension. I used mypy to make stubs, but now mypy insists my type is abstract. I don't know how to convince it it's not. I can share the whole project, but it's complex.

[sleek sluice]

Hey @autumn glen, sorry not the answer you're looking for. I wanted to know if there is a possibility I can take a peek at the code, more because of curiosity. Always wanted to see code written by someone who has probably been programming for more years than I was alive.

Can't meet Robert C. Martin so you're the next best thing

[autumn glen]

The code is public: https://github.com/nedbat/coveragepy/tree/nedbat/mypy-part-3

[sleek sluice]

Nice! Thank you 🙏

[soft matrix]

Can you share the error?

[autumn glen]

https://gist.github.com/nedbat/0bd1666ea4a3f5887a58bf32c767e0fd

I'm adding files one at a time to the list of "ok to check" files. collector.py isn't OK yet, but if I check it solo like this, I get this error. I'm only concerned now about the first error

[sleek sluice]

First error is saying that you are trying to assign something to a variable that is expected to be a class (concrete or otherwise), but the thing you are assigning is not a class.

Second error is saying that you are trying to assign a dictionary with keys and values of an unknown type (<nothing>) to a variable that is expected to be a dictionary with specific key and value types.

[autumn glen]

right, that's the error I need to solve: CTracer is a class, it's defined in a C extension.

[soft matrix]

Does the stub work?

Honestly it's probably just a mypy bug, I remember having issues with conditional expressions and types

[autumn glen]

what does it mean for a stub to work?

[sleek sluice]

I don't know, might be stupid but

from typing import Type

tracer_class: Type[TTracer] = CTracer

This should allow you to assign an instance of CTracer to the tracer_class variable without getting a type error.

[soft matrix]

Like they don't show up as Any?

[soft matrix]

That shouldn't work

[autumn glen]

sorry, can you be more explicit? Where would something show up?

[soft matrix]

TTracer doesn't have any meaning here

[autumn glen]

TTracer is a protocol I defined.

[soft matrix]

Oh

I thought it was a type var

If you do reveal_type(CTracer.some_defined_attribute) does it work?

[autumn glen]

putting that on the line before the error gives, Revealed type is "def (self: coverage.tracer.CTracer) -> coverage.types.TTraceFn" which is right.

[autumn glen]

I'm torn between defining protocols as TSomething or ISomething

[soft matrix]

I generally don't recommend prefixing anything

[autumn glen]

that's the third direction I'm torn in 😄

[sleek sluice]

is TTracer an abstract class?

[autumn glen]

it's a Protocol

[sleek sluice]

try importing TypeVar from typing to use as:

T = TypeVar('T', bound=TTracer)

class TTracer(Protocol):
  # Protocol def

c_tracer: Type[T] = CTracer

This will allow any class that implements the TTracer protocol to be assigned to the c_tracer variable

supposedly

[soft matrix]

That's what I thought you were trying to do when I said that shouldn't work

[sleek sluice]

Woops

Aight guess i'll just stick to TypeScript

[autumn glen]

I appreciate the help

[novel elbow]

How can I export classes to the outside in my __init__.py file but only do that when it gets called/imported from outside my package, since I can't use the exported names in my package and it will cause a circular import?

[soft matrix]

Just don't use the name from the root package, import the name from the actual file itself

[novel elbow]

that wont cause issues? I always thought I had to start from the root package

[leaden oak]

import package.module
c = package.module.MyClass()

# OR

from package.module import MyClass
c = MyClass()

You can always import a package submodule directly. You do it when you import os.path.

[novel elbow]

I assume you switched from MyClass import package.module by accident?

[soft matrix]

Yeah that looks like a typo

[leaden oak]

oops 😅

fixed now, maybe too much JS? I dunno how that happened haha

[noble thunder]

from __future__ import annotations
from typing import Awaitable, Generic, TypeVar, overload

T = TypeVar('T', int, str)

class Foo(Generic[T]):
    @overload
    def bar(self: Foo[int]) -> int: ...

    @overload
    def bar(self: Foo[str]) -> Awaitable[int]: ...

    def bar(self) -> int | Awaitable[int]:
        ...

    def baz(self):
        return self.bar()


f = Foo[int]()
reveal_type(f.bar())  # Type of "f.bar()" is "int"
reveal_type(f.baz())  # Type of "f.baz()" is "int | Awaitable[int]"

How come the type of f.baz() is different than the type of f.bar()? I have a situation similar to this and I don't want to make a bunch of overloads for each function

[trim tangle]

what bigger problem are you trying to solve?

generally, type inference is not very strictly defined, and it varies from typechecker to typechecker

[noble thunder]

so I have this

SyncAsyncT = TypeVar('SyncAsyncT', Literal[False], Literal[True])
class Client(Generic[SyncAsyncT])

How its supposed to work is Client[Literal[False]] is the synchronous client and Client[Literal[True]] is the asynchronous client so all methods would return an awaitable.

I then have this request method

@overload
def request(self: Client[Literal[False]], method: str, url: str, **kwargs: Any) -> Dict[str, Any]: ...

@overload
def request(self: Client[Literal[True]], method: str, url: str, **kwargs: Any) -> Awaitable[Dict[str, Any]]: ...

def request(self, method: str, url: str, **kwargs: Any) -> MaybeAwaitable[Dict[str, Any]]:
# MaybeAwaitable: TypeAlias = Union[T, Awaitable[T]]

I then have this other method

def generate_endpoints(self):
    return self.request('GET', '/generate')

I want Client[Literal[True]]().generate_endpoints() return type to be shown as Awaitable[Dict[str, Any]] instead of Dict[str, Any] | Awaitable[Dict[str, Any]]

[trim tangle]

ah

yeah that thing is generally a bit of a problem

I don't think there's a good way of doing this

why do you need this client to be both sync and async though?

[noble thunder]

Not any specific reason

I guess I want it to be flexible

[soft matrix]

Might be easier with 2 classes

[latent swift]

I have an issue with Django drf's serializer and mypy.

class ProfileSerializer(serializers.Serializer[Profile]):
    data = serializers.JSONField()

The issue is that there's a data property on the Serializer superclass which leads to Mypy giving Incompatible types in assignment (expression has type "JSONField", base class "Serializer" defined the type as "ReturnDict")

[soft matrix]

That error is perfectly valid if ReturnDict is a dict

We need more information if you want us to help fix the error

Like what the ReturnDict actually is, is this a class variable or an instance variable?

[wraith linden]

Is it possible to place a constraint on the relationship between the types of the keys of a dictionary, and the types of their corresponding values? For example, if I have a dictionary of callbacks, where each key is a type, and the corresponding value is a list of callback functions, with each callback taking a value of that type as its only argument.

[spiral fjord]

dict[T, list[Callable[[T], Return]] is not an option?

[void panther]

it's not valid, but the last time I needed something like that I placed a typehint like that into a comment above

[wraith linden]

Nah I think this will mean the callables callbacks all have the same type right?

I guess I could just assert the type of the callback before calling it?

[spiral fjord]

yeah, it doesn't work 😦

[void panther]

yeah you can do a getter for the dict with an assert on the types to have the typing

[wraith linden]

Actually whatever, I'll just not type this dict.

[trim tangle]

the python type system experience

[tranquil turtle]

I didnt check it, but it might work: ```py
if TYPE_CHECKING:
class MyDict(dict):
# class MyDict(dict[type, list[Callable[[Any], Any]]]): # maybe this
# for example, this dict returns value of key's type:
def getitem(self, key: T) -> T: ...
# in your case you should do something like that:
# def getitem(self, key: type[T]) -> list[Callable[[T], Any]]: ...

else:
MyDict = dict # at runtime it is a regular dict

x = MyDict[Any, Any]({1: 2, 'a': 'b'})
x[1] # int
x['a'] # str
x[float()] # float
x[[]] # list

you probably need # type: ignore at def __getitem__ line

i don't think it is possible to annotate .items() in this case

[soft matrix]

yeah that actually does work lmao

atleast with pyright

although the x = MyDict[Any, Any]({1: 2, 'a': 'b'}) should just be x = MyDict({1: 2, 'a': 'b'})

[waxen spade]

So I have a list of objects, and I'm looping through the list with a simple for loop. But i'm loosing my autocomplete. Does somebody know how to type-hint this for loop variable? Something like this:

list = [obj1, obj2, obj3]

for item in list:
  item.help <- can't autocomplete

[trim tangle]

Is the variable named list?

that might be the problem

[waxen spade]

no, its just in this example

[trim tangle]

Can you do reveal_type(your_list)?

what are the types of obj1, obj2 and obj3?

[waxen spade]

it are objects of a class

[trim tangle]

everything in Python is an object of some class

can you show what types your editor infers for them?

[waxen spade]

I guess so, sure gimme a sec

<__main__.image object at 0x00000284FFEFF808>
<__main__.image object at 0x00000284FFF847C8>
<__main__.image object at 0x00000284FFC1D7C8>
<__main__.image object at 0x00000284FFC1D888>

it's a custom class i made names image

[trim tangle]

That's what they are at runtime. But what does your IDE show as the types of obj1, obj2 and obj3?

And can you show the real code?

the whole file would be better

!paste

[rough sluice]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[waxen spade]

https://paste.pythondiscord.com/aguvurabub

sure, posted it

So in line 67 im appending the img object to the list. And in line 71 looping through the list

[trim tangle]

ah, that's different form the example you posted

[waxen spade]

68 and 72*

[trim tangle]

in that case you should do ```py
images: list[image] = []

(also, you should name your classes in PascalCase, like Image)

[waxen spade]

Thx, will try and keep a consistent naming style for once 🥲

[trim tangle]

If you haven't seen already:

!pep8

[rough sluice]

PEP 8 is the official style guide for Python. It includes comprehensive guidelines for code formatting, variable naming, and making your code easy to read. Professional Python developers are usually required to follow the guidelines, and will often use code-linters like flake8 to verify that the code they're writing complies with the style guide.

More information:
• PEP 8 document
• Our PEP 8 song! :notes:

[waxen spade]

wow thx, will look into it

i got an IDE error. But it doesn't effect running the file*

[trim tangle]

What Python version are you running?

[waxen spade]

3.7.9

[trim tangle]

that's a pretty old version, 3.11 was recently released

In 3.7.9 you'll have to do from typing import List and use List[Image]

[waxen spade]

perfect, it works. Can't go any higher with python for compatibility issue's. But it works now so thanks

[carmine phoenix]

Hey guys, when I call extract_keywords(my_sentence) without the optional parameter (span_info), I get the following warning:

Parameter(s) unfilled Possible callees:
KeywordProcessor.extract_keywords(self: KeywordProcessor, sentence: str, span_info: Literal[False])
KeywordProcessor.extract_keywords(self: KeywordProcessor, sentence: str, span_info: Literal[True])

The code:

@overload
def extract_keywords(self, sentence: str, span_info: Literal[False]) -> list[str]:
    ...

@overload
def extract_keywords(self, sentence: str, span_info: Literal[True]) -> list[tuple[str, int, int]]:
    ...

@normalize_parameter('sentence')
def extract_keywords(self, sentence: str, span_info: bool = False) -> Union[list[str], list[tuple[str, int, int]]]:
    # actual code...
    ...

extract_keywords(my_sentence)  # gives me a warning
extract_keywords(my_sentence, True)  # warning gone
extract_keywords(my_sentence, False)  # warning gone

[heady flicker]

In the False overload — put false as the default value

[trim tangle]

Who's giving you the warning?

unrelated, but you might want to just split this into two functions. can you show the implementation?

[carmine phoenix]

PyCharm's builtin linter

[carmine phoenix]

it wouldnt be very practical, but this is the code:

    def extract_keywords(self, sentence: str, span_info: bool = False) -> Union[list[str], list[tuple[str, int, int]]]:
        keywords = []
        trie_dict = self.trie_dict
        sentence_len = len(sentence)
        prev_char = None

        for idx, char in enumerate(sentence):
            # DEBUG: recognizes the start of each word (including the first)
            if prev_char not in self.non_word_boundaries and char in self.non_word_boundaries:
                keywords_found: list[tuple[str, int, int]] = []  # can you have two keys of the same size?
                node = trie_dict

                for i, c in enumerate(sentence[idx:]):
                    node = node.get(c)
                    if node is None:
                        break
                    else:
                        kw = node.get(KeywordProcessor.keyword)
                        next_char_idx = idx + i + 1
                        # if kw AND (it's the last char in the sentence OR the next char is not in non_word_boundaries)
                        if kw and (next_char_idx == sentence_len or
                                   sentence[next_char_idx] not in self.non_word_boundaries):
                            keywords_found.append((kw, idx, idx + i + 1))

                # if we have found any keywords, select the keyword with the most amount of characters
                if keywords_found:
                    # TODO: get word or clean_word with the highest len()?
                    longest_keyword_tup = max(keywords_found, key=lambda tup: len(tup[0]))
                    keywords.append(longest_keyword_tup) if span_info else keywords.append(longest_keyword_tup[0])

            prev_char = char
        return keywords

[carmine phoenix]

now it works, thanks!

    @overload
    def extract_keywords(self, sentence: str, span_info: bool = False) -> list[str]:
        ...

    @overload
    def extract_keywords(self, sentence: str, span_info: Literal[True]) -> list[tuple[str, int, int]]:
        ...

[trim tangle]

Well, you could yield just longest_keyword_tup and then have a second function to extract the first value

I suppose a NamedTuple would also be better, since tuple[str, int, int] is not very obvious

the second function would definitely be simpler than a flag parameter and overloads

    def extract_keywords(self, sentence: str) -> Iterator[Keyword]:
        prev_char = None

        for idx, char in enumerate(sentence):
            if prev_char not in self.non_word_boundaries and char in self.non_word_boundaries:
                longest_keyword: Keyword | None = None
                node = self.trie_dict

                for i in range(idx, len(sentence)):
                    node = node.get(sentence[i])
                    if node is None:
                        break
                    kw = node.get(KeywordProcessor.keyword)
                    if kw and (i == len(sentence) - 1 or sentence[i + 1] not in self.non_word_boundaries):
                        if len(kw) > longest_keyword.word:
                            longest_keyword = Keyword(kw, idx, i + 1)
            
                if longest_keyword:
                    yield longest_keyword
                    
            prev_char = char
        return keywords

unrelated to typing, but you might want to replace enumerate(sentence[idx:]) with something that doesn't copy a lot of characters: ```py
def enumerate_shifted(seq, start):
for i in range(start, len(seq)):
yield i, seq[i]

Or use range and index it directly, replacing c with sentence[i]

[carmine phoenix]

I was wondering if there was a better alternative (and if str[slice] really creates a copy and not just a view)

[trim tangle]

yeah, see my version above with range

[trim tangle]

there's no built-in version, but it's a simple function

there's itertools.islice but it will still go through the start of the sequence anyway

[carmine phoenix]

also I added thse two at the top of the loop as I dont want to call getattr('trie_dict') 50k times for a large document, and same with the len(). what do you think about that?

trie_dict = self.trie_dict
sentence_len = len(sentence)

[trim tangle]

You can measure the impact by benchmarking 🙂

[carmine phoenix]

but youre refactoring the code 😅 so I just want to know youre toughts on it

but enumerate(sentence[slice]) might have been the reason that this is not as fast as expected

[trim tangle]

My thoughts are: 1. don't do micro-optimizations without benchmarking, and before you've done bigger optimizations (i.e.: figure out what your bottleneck is)

2. do macro-optimizations first, like improving your algorithm or avoiding things like enumerate(large_think[idx:]) (also with benchmarking)

3. figure out how fast it needs to be. if your code is fast enough, don't waste time optimizing it, especially if that harms code quality

[carmine phoenix]

its probably my first time saving an attribute to avoid getting it everytime, but I saw in the Collections.Counter:
https://github.com/python/cpython/blob/e6d44407827490a5345e8393fbdc78fd6c14f5b1/Lib/collections/__init__.py#L525

[rough sluice]

Lib/collections/__init__.py line 525

mapping_get = mapping.get```

[trim tangle]

I think it's slowly being phased out, because it's largely unnecessary with the new performance improvements in attribute lookup

[carmine phoenix]

But I completetly agree with you. Speaking of which, are there any resources that explain how the data strcutures work specifically in Python and whats the overhead for all these common operations?

[trim tangle]

i don't know to be fair, I would ask in #internals-and-peps

[tranquil turtle]

cls.attr is still slow IIRC, idk why

[carmine phoenix]

honestly for this NLP stuff there really isnt fast enough because when youre processing ten thousands documents with each 50k chars, each microsecond counts

and this is gonna be used (if its fast enough) for a search engine

[trim tangle]

Have you tried pypy?

if it is really too slow... maybe a fast search engine isn't quite a Python project 🙂

🦀

[carmine phoenix]

not yet, but I have to sooner or later

btw if you think what we have is messy, than have a look at the original function 😅 :
https://github.com/vi3k6i5/flashtext/blob/5591859aabe3da37499a20d0d0d6dd77e480ed8d/flashtext/keyword.py#L450

[rough sluice]

flashtext/keyword.py line 450

def extract_keywords(self, sentence, span_info=False):```

[trim tangle]

I've definitely seen worse

and yes it is a big improvement

[carmine phoenix]

is it just me or is that completely unreadble (and garbage imo)

it took me a few days to understand how it actually works

[trim tangle]

Yeah it's pretty bad.

anyway... we might want to move to #software-architecture

[carmine phoenix]

once you move it, it dies.

[trim tangle]

we're all going to die one day

[carmine phoenix]

but holy shit when I looked at youre code I realize there is alot that can improve

before I go, can you recommend and resources for learning DS in Python?
(not the useless stuff like linked trees, etc. but what happens when you call in on a hasmap vs a sequence, or when you do a shallow copy, or get a slice of a sequence, etc..

)

[rare scarab]

A sequence isn't really a data structure. It's more of a class of data structures that can be ordered arbitrarially

[carmine phoenix]

Fair enough, do you have any recommendations for where I can learn the inner workings of these objects?

I feel like the only way is to learn some C and go trough the source code lol

[rare scarab]

There's some tutorials and guides that use other languages, like javascript.

[heady flicker]

I don't think it is the best way to typehint this return type

I think Generator would be more suitable because it makes more sense to use the most specific type hints for return type while using the most general type hints for parameter types

[oblique urchin]

another way to look at it is that the type hint specifies the contract with your callers. If you say Generator, callers can rely on the fact that it returns a generator. But if you say Iterator, you're free to change the implementation to use some other kind of iterator

[heady flicker]

I agree, that also makes a lot of sense. Though I would recommend to use specifics in the general case, especially if we're not playing with inheritance

[trim tangle]

why?

As Jelle said, it's an implementation detail that it's a generator

I think you should keep your interfaces as small as possible. If you include send() and throw() in the signature of the return type, that's a bigger interface.

[heady flicker]

It gives your users (or you) maximum versatility in terms of how they use the return value of your function.

I.e. if you return sequence[str] instead of list[str], then your user is constrained in terms of appending.

The reverse is true for parameter types.

[trim tangle]

On the other hand, it constrains you in changing the function. If the return type is list[str], you can't change it to a tuple or a custom sequence in the future without breaking your promises to the users

[carmine phoenix]

which do you guys prefer for a generator function that dosent take any args from gen.send()/gen.throw()

    def func_that_yields_tuples() -> Generator[tuple[str, int], None, None]:
    def func_that_yields_tuples() -> Iterator[tuple[str, int]]:
    def func_that_yields_tuples() -> Iterable[tuple[str, int]]:

[trim tangle]

I'm pretty sure we're discussing that right now 🙂

[carmine phoenix]

oh lol

so whats the verdict?

oh this is about the code I wrote yesterday 😅

[trim tangle]

Contrived example: py def get_version() -> Literal["v1.0.3"]: return "v1.0.3" vs ```py
def get_version() -> str:
return "v1.0.3"

And what extra options does returning a Generator give in this case?