type-hinting

[undone saffron]

Back to the impact on documentation, I'd be willing to say the number of places the average developer encounters where the type would cause visible confusion because it appears to contradict the documentation in some way is much smaller, and the cases as clear as asyncio.gather is for this are much rarer (I have 7 of these in total so far, and all of them have associated typeshed issues and could be revisited later given better tools to express some things, but probably can't be currently)

Outside of those cases many times, this inaccuracy is very minor and something that would not lead to user confusion. The user confusion if non-overload, non-complex protocol types were the only considered signatures would be much fewer.

in terms of readable documentation, I would say the other issues I pivoted to to put as a first line in the argument against are significantly more important.

as for the philosophical difference: consistently choosing imprecision over inaccuracy with gradual typing as a philosophy leads to 2 things I find extremely valuable

1. zero false positives.
2. remaining gradual types act as a roadmap for what can't be expressed in a more useful manner yet.

[solemn sapphire]

This code

def list_dicts() -> list[dict[str, int]]:
    return [{'one': 1}, {'one': 1}, {'two': 2}]

print(set(list_dicts()))

passes type checking, but dict is not hashable

Looking at the issue tracker, I came across https://github.com/python/typeshed/blob/bfdb87b9fd5aaa54f671a75560d993fb6aeafa9e/stdlib/typing.pyi#L177-L183 this PR, but I don't understand how a subtype of hashable might not be be hashable?

Like why can't _T here have a bounds=Hashable?
https://github.com/python/mypy/blob/master/mypy/typeshed/stdlib/builtins.pyi#L1047

[rough sluice]

stdlib/typing.pyi lines 177 to 183

@runtime_checkable
class Hashable(Protocol, metaclass=ABCMeta):
    # TODO: This is special, in that a subclass of a hashable class may not be hashable
    #   (for example, list vs. object). It's not obvious how to represent this. This class
    #   is currently mostly useless for static checking.
    @abstractmethod
    def __hash__(self) -> int: ...```
`mypy/typeshed/stdlib/builtins.pyi` line 1047
```py
def __init__(self, iterable: Iterable[_T], /) -> None: ...```

[cinder bone]

To give an example of how a subtype of a hashable doesnt have to be hashable, just look at object. object is Hashable, the following is not

class Foo(object):
    def __eq__(self, other: object) -> bool:
        return True

[solemn sapphire]

Oh. Then how about having the signature be

def __init__(self, iterable: Iterable[Hashable[_T]], /): -> None

?

[tranquil ledge]

Mypy and pyright don't understand Python's hashability rules, afaik, so that may play a part. I could see this maybe causing an increase in "false negative" type-checker issues, which typeshed has a tendency to avoid.

[solemn sapphire]

I see, so hashability doesn't just mean presence of __hash__

[tranquil ledge]

Well, it does. Fun fact: defining only __eq__ erases __hash__ from your class.

[tranquil ledge]

That's what this is meant to demonstrate.

[cinder bone]

Hashable isn't generic, so this wouldn't work regardless

[stiff acorn]

from __future__ import annotations
from enum import StrEnum

class Test(StrEnum):
    A = "A"
    B = "B"
    C = "C"

    @classmethod
    def get(cls, key: str, default: Test = Test.A) -> Test:
        ...
        

is this not possible?

    def get(cls, key: str, default: Test = Test.A) -> Test: ...
                                           ^^^^
NameError: name 'Test' is not defined

[lunar dune]

It's the default value that's causing the NameError there rather than the annotation (since you have __future__ annotations enabled)

[stiff acorn]

Yea

[restive rapids]

yes, thats where the error is pointing to :d maybe do the usual | None = None and assign a default in the body

[stiff acorn]

That means I cannot reference the class in the default value i guess?

[lunar dune]

Correct, as default parameter values (like parameter annotations) are eagerly evaluated in a function's enclosing scope when the function is defined rather than being evaluated in the scope created by the function when the function is called. You could do default: Test | None = None in the signature and do default = default or Test.A as the first line in your function

[stiff acorn]

I think I can also just get by with a Literal["A", "B", "C"] to achieve nearly the same thing

and make default a string instead

[cinder bone]

You might also be able to do like = A

[trail kraken]

As an aside, it might be preferable to use just Enum in some cases. Getting the string: Enum.A.name. The other direction: Enum["A"].

[vivid ore]

Is there a way to type a value to be whatever the type of an entry in a dictionary is, for the case that it might be a TypedDict?

Say I have a class that wraps a dictionary. It has a method for generating a copy of itself with some function applied to its values.
Can I type that class in a way that its set of keys and corresponding types are statically typed?

[rare scarab]

python has no keyof keyword like typescript. Instead, you have to manually specify each key using Literal["x", "y", "z"]

[vivid ore]

What if I want to be generic over this?

[rare scarab]

can't do that.

[vagrant granite]

hi everyone

[stiff acorn]

class APIWrapper:
    def __init__(self, api_url: str = "https://example.com/api/", ratelimit: int = 2, **kwargs: Any) -> None:
        self.api_url = api_url
        self.ratelimit = ratelimit
        self.kwargs = kwargs

    def post(self) -> str:
        with httpx.Client(**self.kwargs) as session:
            session.post(self.api_url)

        return "POST"

is there a more succint way to type this or is Unpack[TypedDict] my only option?

I mean the **kwargs which should be identical to httpx.Client.__init__

[trim tangle]

that's the only way

Perhaps a better way would be accepting an httpx.Client in the __init__. The whole point of a client is to not create a new instance on every call

class APIWrapper:
    def __init__(self, *, api_url: str = "https://example.com/api/", ratelimit: int = 2, client: httpx.Client) -> None:
        self.api_url = api_url
        self.ratelimit = ratelimit
        self.kwargs = kwargs
        self.client = client

    def post(self) -> str:
        self.client.post(self.api_url)
        return "POST"

[stiff acorn]

Oh wow, I didn't think of that

Although guess I'll have to define dunder enter and exit and .close for a better experience

[trim tangle]

with httpx.Client(...) as client:
    wrapper = ApiWrapper(client=client)
    # use `wrapper`

But yes, you could just create a client inside __init__ and then make APIWrapper a context manager

[stiff acorn]

Makes sense, thank you

[void panther]

My dependencies are doing basically this in their typing because of stubs, causing pyright to shout when I pass the expected type from init. Is there a way of getting it to ignore the wrong new or should I just set an ignore?

class A:
    def __new__(cls, b: str):
        return super().__new__(cls)

class B(A):
    def __init__(self, a: int):
        pass
B(4)  # error, not compatible with new's signature

[viscid spire]

you can use if TYPE_CHECKING to fix their types up

[hasty phoenix]

Given the function def fn(factory: Type[T] | None = None) -> T. This works when factory is not None, however it is Unknown when factory is None. Is there a way to specify a default return type T when factory is omitted (None)?

[viscid spire]

you can use overloads, or TypeVar from typing extensions

[cinder bone]

Also if your python version supports it you can use type[T] instead of Type[T]

[hasty phoenix]

^ That was a typo. pun indended

[tacit sparrow]

hi!

can someone please help me understand why from __future__ import annotations adds Module(".bar") to the typing? is there some workaround to avoid it?

[tranquil ledge]

iirc, there's at least one issue in the pyright issue tracker about the type-checker getting confused about such a case where a type exported by a module and that module's name are the same, and the maintainer's recommendation was to rename either the file or the type. Consider looking for yourself to double-check that though.

[tacit sparrow]

I see, couldn't find anything in the pyright issues, will try to start a discussion there too

[tranquil ledge]

👍 I also tried looking for it and couldn't find it again. Maybe Eric'll point you to it or re-enumerate his reasons.

[tranquil furnace]

Hi all, new to this server. Came to ask about mypy behavior that I don't understand: https://mypy-play.net/?mypy=latest&python=3.12&gist=36722ae3721f8ff7e0494edec101b785

In this example I expect the return value to comply to the return type hint, since cls is used, guaranteeing the returned object is of type Self. What am I missing here?

(Hope there are some typing or mypy experienced devs here to make this clear.)

[rustic gull]

Hi, I wonder what are your thoughts about the else clause especifically? Do you use smth like this?

from typing import TYPE_CHECKING, Any

if TYPE_CHECKING:
  from x import _MyType
else:
  _MyType = Any

[trim tangle]

If you don't have from __future__ import annotations (which will be deprecated) or quoted annotations, this is even necessary

[rustic gull]

interesting, so are you agreeing right? im using 3.10 so no __future__ annotation imports
im quite new and sm insecure

[pastel egret]

There's a similar problem in mypy#16558. This code is only safe because the class is an enum. Although cls is accessed, you're just looking up class attributes, which have a fixed type. If a subclass existed, calling the classmethod would claim to produce the subclass, but return the same attributes still. However in your actual code that can't happen because enums are not subclassable. Mypy just doesn't take that fact into account here.

It's probably better and more accurate to just use the class name directly, that will solve the problem.

[trim tangle]

from __future__ import annotations is not automatically applied in 3.10 (and never will, instead there's going to be a new thing with PEP 649)

Yes, it's probably a good idea to have that else branch, so that the _MyType global is always defined

[rustic gull]

exactly, also important, one can't use None

bc it clashes with other None if you have _MyType | None

[pastel egret]

Your else clause only has effect at runtime, I'm not sure why you chose Any here specifically. I'd probably just do "_MyType", or object. It only needs to be something sufficiently valid to work for code at runtime.

[rustic gull]

and runtime-errors. that's my experience at least

[trim tangle]

object() | int does not work

so Any is probably correct

[pastel egret]

I mean the object class.

[trim tangle]

ah

[rustic gull]

idk whether object | object would work

[trim tangle]

it will

[trim tangle]

unless you have a generic type, in which case you'll have fun stuff like ```py
if TYPE_CHECKING:
from tkinter import Event as EventType
else:
class EventType[T]:
...

(if you're on 3.11 or below you'll need to use TypeVar and Generic instead)

[rustic gull]

yeah atm i only have TypeDicts, i'd use TypeVar but only once im comfortable w the rest

i think the runtime criticism is quite fair, and makes sense

i wonder why does Any work ?

[trim tangle]

work in what way?

(also object should also work as TeamSpen suggested)

[rustic gull]

bc Any would be the value at runtime, not for typechecking right?

nvm i think i just got confused, both should work since both are types, and python isn't removing them at runtime

[tranquil furnace]

This makes sense, @pastel egret ! Thanks! cls() is always Self, but cls.APPLE is not. It's always a relief to figure out why mypy disapproves something. 🙏

[rustic gull]

ppl prefer mypy? ive used pyright atm, i think it was better experience when jus testing myself

?

[trim tangle]

mypy is older and more popular, so more real world projects probably use mypy

pyright is not just a type checker though, it also provides autocompletions

if you're in VSCode you're probably using Pylance which is not quite pyright...

I'm using basedpyright which is like unmicrosofted pylance

[rustic gull]

i use pyright

read its README.md and issues, unsure it's worth switching.

[trim tangle]

One benefit of basedpyright is that it provides an official PyPI package. So you don't need to install Node.js yourself

there is a pyright PyPI package, but it's maintained by a different person and is not that great. (it installs node when you first run it instead of bundling node)

[rustic gull]

yeah i read the readme

idk whether it will be stable, maintained etc though; ig im waiting till they get more usage

although it was like 19M downloads iirc.

[cosmic plinth]

when does from __future__ import annotations become the default, 3.12 or 3.13?

[restive rapids]

!pep 649

[rough sluice]

**PEP 649 - Deferred Evaluation Of Annotations Using Descriptors**

Status

Accepted

Python-Version

3.14

Created

11-Jan-2021

Type

Standards Track

[cosmic plinth]

3.14?

[restive rapids]

yep

[cosmic plinth]

wow that import will live on for a while, okay thanks!

[oblique urchin]

never

[viscid spire]

(And actually it won't be the default, but rather something better... eventually)

[oblique urchin]

Eventually is right now. Just clone the CPython repo and build.

[viscid spire]

Oh?

I guess build 3.14

makes sense

[terse sky]

from typing import TypeVar

_T = TypeVar("_T")

def unwrap(t: _T | None) -> _T:
    if t is None:
        raise RuntimeError("Unexpected empty optional unwrapped!")
    return t
    
def blub(x: str | int):
    pass
    
x: str | int | None = "hello"
y = unwrap(x)
blub(y)

this makes me pretty sad

this will not type check

on mypy, at least

[restive rapids]

seems like a mypy issue

[oblique urchin]

In https://github.com/hauntsaninja/useful_types/pull/7 I tried to make a solution that is compatible with mypy and pyright, but got stuck and lost motivation to finish. If you only care about one type checker, there will be a variant there that works though.

[terse sky]

Thanks! This is definitely motivating me to switch us from mypy to pyright; pyright is basically "baked in" for us since our whole team uses vscode

the overload trick is clever

honestly though, I understand pylance's objection

python doesn't have any rules saying which overload is "more specialized", those kinds of rules are very complex

so I'd say in general you can't really have @overload 's that overlap?

[cinder bone]

What do you mean by overlap?

[terse sky]

you can't have calls that work for both

overlap in terms of the sets of calls/types they cover

@overload
def not_none(obj: _T, /, message: str | None = ...) -> _T:
    ...


@overload
def not_none(obj: None, /, message: str | None = ...) -> Never:
    ...


def not_none(obj: _T | None, /, message: str | None = None) -> _T:
    """Raise TypeError if obj is None, otherwise return obj.
    Useful for safely casting away optional types.
    """
    if obj is None:
        if message is not None:
            raise TypeError(message)
        else:
            raise TypeError("Object is unexpectedly None")
    return obj

here's the code in question

the two overloads "overlap", not_none(None) works for both

[cinder bone]

Oh I think usually the first overload that matches is the one that is used

But I'm not sure

[terse sky]

apparently that doesn't work here for some reason

[cinder bone]

What do you mean it doesn't work?

[terse sky]

well, Jelle mentioned that in the issue

although.... it does seem to work for me

I'm not sure what Jelle meant

I think might work with both if you reorder overloads?
...
No, that way it doesn't narrow away the None at all, because None matches the first overload.

🤷‍♂️

mypy and vscode (which is using pylance under the hood) both seem to be okay with this

[cinder bone]

Oh this was about the PR
Yeah I'm not sure

[terse sky]

JsonType = None | str | int | float | Sequence[JsonType] | Mapping[str, JsonType]
MutableJsonType = None | str | int | float | MutableSequence[MutableJsonType] | MutableMapping[str, MutableJsonType]

def foo(x: MutableJsonType) -> JsonType:
    return x

a fun stress test for covariance 😛

(this does pass mypy)

I wonder if it's better to do MutableSequence/MutableMapping here, or just list/dict

I'm not actually sure what "extra" API list/dict have compared to MutableSequence/MutableMapping

[rare scarab]

List/dict is a specific implementation

You can always implement your own mutable mapping. Ex you could make a multimapping

https://github.com/encode/starlette/blob/fa7b382a66cd99e3dc18f3baa44dae5ec68be76b/starlette/datastructures.py#L317

[rough sluice]

starlette/datastructures.py line 317

class MultiDict(ImmutableMultiDict[typing.Any, typing.Any]):```

[rare scarab]

Httpx actually implements mutablemapping.
https://github.com/encode/httpx/blob/49d74a2e7f773f88a84dd4af900d7a8543636ae2/httpx/_models.py#L59

[rough sluice]

httpx/_models.py line 59

class Headers(typing.MutableMapping[str, str]):```

[rare scarab]

So it's for objects that behave like a full dict, but aren't actually backed by a dict.

Multidict and headers is backed by a list

[terse sky]

Yes, I'm aware. I wondered what the difference in API is

List implements MutableSequence so obviously lists API is a superset

Question is whether list has API that's not part of MutableSequence, and if so what

Same with dict and MutableMapping

[rare scarab]

I suggest testing things in the mypy or pyright playgrounds

https://mypy-play.net/

[terse sky]

I don't really see how that relates to what I'm talking about

I think maybe you didn't understand what I was asking

[tough bolt]

I'm experiencing a type hinting issue related to multiple inheritance. Here's the code:

class B:
    pass

class C:
    pass

class SubB(B, C):
    pass

class A:
    def b(self) -> B:
        return B()

class SubA(A):
    def b(self) -> SubB:  # Mypy raises an error here
        return SubB()

I'm convinced that this issue is caused by multiple inheritance. Am I correct in thinking this?

[jolly cipher]

what error are you seeing?

[tough bolt]

The return value of the function differs from the expected value of the supertype.

Refer to the Liskov Substitution Principle

[jolly cipher]

can't reproduce. what mypy version are you using?

[tough bolt]

1.11.2

app/domains/requirement/infrastructure/database/repositories/sql_requirement_repository.py:21: error: Argument 1 of "_save" is incompatible with supertype "RequirementRepository"; supertype defines the argument type as "RequirementModel" [override]

[jolly cipher]

i'm not sure the repro is right -- i can't reproduce. make sure your repro represents all the key details of the actual code

if you copy and paste that code you sent ^ in mypy playground @ https://mypy-play.net/?mypy=1.11.2, you don't get an error

[tough bolt]

I had no idea this kinda thing
ty

[keen pawn]

Is there some way to assert that mypy should error in a test case? or is the common practice just # type: ignore
(within a test case that should raise an actual exception one way or the other ; relevant code and test case for context)

[rough sluice]

py-polars/tests/unit/test_config.py lines 741 to 743

def test_config_raise_error_if_not_exist() -> None:
    with pytest.raises(AttributeError), pl.Config(i_do_not_exist=True):
        pass```

[oblique urchin]

yes, the standard way is --warn-unused-ignores

[spice locust]

is there a way to specify only a partial set of keys for a dict, while not disallowing other keys?

in other words, can i have a typeddict that allows keys not defined in it? preferably with a type set for all 'other' keys

something like

class MyDict(TypedDict):
  key: str
  *: str | int | float | bool

[restive rapids]

currently no

[restive rapids]

!pep 728
if this gets accepted then uh would be cool

[rough sluice]

**PEP 728 - TypedDict with Typed Extra Items**

Status

Draft

Python-Version

3.13

Created

12-Sep-2023

Type

Standards Track

[rustic gull]

I found how make function inherit another functions arguments with IDE suggestions and I can add and remove any number of rightmost arguments

_P = ParamSpec("_P")
_T = TypeVar("_T")
_S = TypeVar("_S")
_R_co = TypeVar("_R_co", covariant=True)

class FuncWithArgs1(Protocol, Generic[_P, _R_co]):
    def __get__(self, instance: Any, owner: type | None = None) -> Callable[_P, _R_co]:...
    def __call__(self, __: Any, *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ... # pylint:disable=E1101,E0213 #type:ignore
class FuncWithArgs2(Protocol, Generic[_P, _R_co]):
    def __get__(self, instance: Any, owner: type | None = None) -> Callable[_P, _R_co]:...
    def __call__(self, __1: Any, __2:Any, *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ... # pylint:disable=E1101,E0213 #type:ignore
class FuncWithArgs3(Protocol, Generic[_P, _R_co]):
    def __get__(self, instance: Any, owner: type | None = None) -> Callable[_P, _R_co]:...
    def __call__(self, __1: Any, __2:Any,__3:Any, *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ... # pylint:disable=E1101,E0213 #type:ignore
class FuncWithArgs4(Protocol, Generic[_P, _R_co]):
    def __get__(self, instance: Any, owner: type | None = None) -> Callable[_P, _R_co]:...
    def __call__(self, __1: Any, __2:Any,__3:Any,__4:Any, *args: _P.args, **kwargs: _P.kwargs) -> _R_co: ... # pylint:disable=E1101,E0213 #type:ignore

heres a decorator to remove 3 arguments and add 2 arguments

def f2f_remove3_add2(_: FuncWithArgs3[_P, _T]) -> Callable[[Callable[_P, _S]], FuncWithArgs2[_P, _S]]:
    def _fnc(fnc: Callable[_P, _S]) -> FuncWithArgs2[_P, _S]:
        return fnc # type:ignore
    return _fnc

let me write an example

hmmm

it doesnt let me add new args but it does let remove args

here is example of removing

def person(
    color: float,
    size: float,
    shape: list[int],
    position: list[float] = [4,5],
    rotation: float = 0.0,
    weight: float = 0.0,
    name: str = 'default',
    description: str = 'default',
    occupation: str = 'job',
):
    return 1

@f2f_remove3(person)
def redman(
    *args,
    **kwargs,
): ...

remove1_add1 can be used to move one methods hints to another method because it makes self work somehow

[mint iris]

Yikes...

[stiff acorn]

I'm getting a strange error from mypy:

❯ mypy .\mybot\ --strict
mybot\config.py: error: Source file found twice under different module names: "config" and "mybot.config"
Found 1 error in 1 file (errors prevented further checking)

structure:

mybot
├── config.example.toml
├── config.toml
├── poetry.lock
├── pyproject.toml
├── README.md
└── mybot
    ├── __main__.py
    ├── commands
    │   ├── __init__.py
    │   ├── ban.py
    │   ├── mute.py
    │   └── kick.py
    ├── config.py
    ├── helpers.py
    ├── mybot.py
    └── version.py

config.py has another variable called config

# mybot/config.py
from pathlib import Path
from pydantic import BaseModel

class Config(BaseModel):
    blah: str
    bla2: int
    etc: ...

workspace = Path(__file__).parent.parent
config_file = workspace / "config.toml"

if not config_file.is_file():
    raise FileNotFoundError(f"Cannot load configuration from {config_file} because it doesn't exist!")

config = Config.from_file(config_file)

which is then used in every other module:

from mybot.config import config

[stiff acorn]

There's no runtime error fwiw

[cerulean pecan]

is it possible to enforce this "eight-digit base62 ID" with a type hint

or can i somehow put a regex

[trim tangle]

no

You'll very likely be getting these values at runtime, so a runtime check will be needed anyways. So a type hint would only really help if you're using string literals with these values a lot.

[cerulean pecan]

I want both runtime and compile check

[trim tangle]

What would be the benefit of a static check?

[cerulean pecan]

so if a user enters get_project("abc") they will get a warning in their IDE

[trim tangle]

If you want to ensure that only validated values are propagated downstream, you can make a wrapper class like this ```py
class BookId:
def init(self, raw: str) -> None:
if re.fullmatch("...", raw) is None:
raise ValidationError(...)
self._raw = raw

def __repr__(self) -> str:
    return f"BookId({self._raw!r})"

def raw(self) -> str:
    return self._raw

[cerulean pecan]

thx very useful

[trim tangle]

Maybe your system is very unusual, but typically you don't put these nasty IDs in the code itself. It either comes from some kind of database or config file.

You can, of course, ensure that a value is validated and is kept validated with a wrapper class like the above

[cerulean pecan]

I guess i will probably just end up using NewType(str)

I was looking for more something like @Pattern annotation from java
https://github.com/JetBrains/java-annotations/blob/master/src/jvmMain/java/org/intellij/lang/annotations/Pattern.java

[crystal sequoia]

Is there a reason that type checkers don't consider implementing __iter__(self) -> Iterable[T] to make a type Iterable[T] itself? Specifically, I'm getting a warning when passing an instance of the type to the enumerate function (which works correctly in practice). To be honest, I actually thought this was a bug in the PyCharm type checker, but I just noticed today that mypy has the same behavior, so maybe there's a good reason for it?

[pastel egret]

You want __iter__(self) -> Iterator[T]:, not Iterable. That's likely the problem.

[crystal sequoia]

Ahh, yes, that makes sense, and that does resolve the warning in both tools. Thanks!

[proud yew]

I've noticed that with dataclasses, when you do this:

@dataclass
class SomeClass:
    var_a: int | None = None

x = SomeClass(var_a=)```
When hovering over var_a in VSCode with Pylance, it simply shows:
 (variable) var_a: None

Does anyone know of a workaround to show the actual type, not just the default value?

I can override the init and that works, but thats not practical for a large dataclass because one would have to copy each value into the init and keep track of 2 different default values. I suppose you could write up the variables without defaults and then only define the default in the overwritten init like this:
```py
@dataclass
class SomeClass:
    var_a: int | None
    def __init__(self, var_a: int | None = None):
        self.var_a = var_a```
but I'm not sure if that would break something with how dataclasses are supposed to work.
Its a bug tho, no?

[terse sky]

Did you mean to write x = SomeClass(var_a=None)

[cinder bone]

Are you sure you need a dataclass? My general rule of thumb is if I need a runtime computed default value, a dataclass isn't the right fit

Also you'd want to override __post_init__ or pass init=False to the @dataclass decorator

[viscid spire]

It's inferring the current type based on a known value

[proud yew]

no, Im talking about the popup when you hover over the var_a during instantiation

[proud yew]

thats valid

[proud yew]

but it should know the possible types through the type hint

[viscid spire]

yes it does know them

It also knows that at that moment, the value is definitely None

And therefore doing an integer operation on the value will error

[terse sky]

but there isn't a runtime computed default value here? And if there was one, why would that make a dataclass the wrong fit?

[terse sky]

"hover over" - you mean you stop typing in the middle of initializing some class, and put your mouse over var_a?

here's what I see

sorry, if I add the = sign, then the "Undefined name" thing vanishes, and I'm just left with the part above

[viscid spire]

You did not replicate the situation

put a default value

[terse sky]

ah you're right good catch

The thing is though even still, I don't see that unless I deliberately stop typing and move my mouse over the variable

In "normal use" this is what I see

[proud yew]

interesting, I dont get that

you using pylance?

[terse sky]

yeah, I think so. Just the default vscode/python setup

do you see that kind of completion for functions?

not completion I guess, pop-up

this kind of pop-up is a lot less impressive on subprocess.run 😂

[proud yew]

with functions you get this, but I also dont see proper typing when I hover the arg= part. But, with normal non-dataclass classes, Im assuming since they're members, you actually do. just not with dataclasses, which is why i thought it could be a bug and fixable

[terse sky]

what do you mean by "hover the arg=" part?

like, you type, stop typing in the middle, then move your mouse cursor so that it's over what you just typed, and wait?

If I do that, then yes, I also see the same thing you did. but that's not really a bug in vscode/pylance IMHO, that's just not really an intended workflow. You're stopping in the middle of the popup that's helping you write valid python code, while your python code is invalid, and hovering over it

[proud yew]

that popup doesnt come when ur instanciating a class

[halcyon gyro]

hey all 👋

I'm having trouble annotating a generic function. Here is the reduced example:

from typing import TypeVar

def print_all(data: list[str|int]) -> None:
    for d in data:
        print(d)

T = TypeVar('T')

def concat(lsts: list[list[T]]) -> list[T]:
    return sum(lsts, start=[])

data: list[list[int]] = [[1, 2, 3], [4, 5], [6]]

print_all(concat(data))

mypy gives me this error:

scratch_48.py:18: error: Argument 1 to "concat" has incompatible type "list[list[int]]"; expected "list[list[str | int]]"  [arg-type]

I have a vague recollection that list is invariant for some reason, which would explain why mypy reject list[list[int]] as being compatible to list[list[str | int]]. Still, I can't figure out what the proper generic way to annotate this.

[oblique urchin]

The general solution is to use a more general, covariant type like Sequence or Iterable in parameter annotations

That might not work for your concat() function; I'd probably rewrite it using itertools.chain.from_iterable anyway because sum() is quadratic here

[halcyon gyro]

I just realised that indeed Sequence is covariant and is accepted by mypy, but that does force me to explicitly convert my input with list(...) even though I "know" these are list already. (Good call for itertools, though I'm still very much curious about how to solve that typing thing ;))

[oblique urchin]

Where does it force you to convert your input?

[halcyon gyro]

Consider this alternative, naive implementation of concat:

def concat(lsts: list[list[T]]) -> list[T]:
    result: list[T] = []
    for lst in lsts:
        result = result + lst
    return result


def concat2(lsts: Sequence[Sequence[T]]) -> list[T]:
    result: list[T] = []
    for lst in lsts:
        result = result + lst  # fails
    return result

The second one has convariant hints, so it solves the original issee. However, it currently fails because of + lst (Sequence doenst have __add__). To fix this, I need to change it to an explicit + list(lst). But this is superfluous since I know from the get go that I only have lists. (I know, deep in nitpick territory here.)

[oblique urchin]

Right, the solution is to not use +, which is a good idea independent of typing because again this code is quadratic

[halcyon gyro]

Agreed. I should have a better example with a legitimate use of, e.g., append, which list gives me but not Sequence.

[viscid spire]

MutableSequence

[oblique urchin]

Is still invariant and doesn't fix the original problem

[viscid spire]

ah

[terse sky]

the reason why list has to be invariant is because if it was covariant, people could push super-types into the list, "ruining" the original list.

def print_all(data: list[str|int]) -> None:
    data.append("hello")
    data.append(5)

data: list[int] = [1, 2, 3]
print_all(data)

as for concat - the actual solution is to use extend rather than + list(lst)

Sequence isn't really the right annotation for concat either

def concat3(inputs: Iterable[Iterable[T]]) -> list[T]:
    result: list[T] = []
    for inp in inputs:
        result.extend(inp)
    return result

For function inputs you really want to annotate with the absolute minimum you need. Since you should almost never be mutating arguments to a function, you should not be annotating function arguments as list or even MutableSequence very often

[vale lantern]

cool

[halcyon gyro]

This makes sense, thanks!

[spice locust]

  Type "type[Any]" is not assignable to type "type[ItemT@Base] | type[Any] | None"
    "type[type]" is not assignable to "type[None]"```

this seems odd given the type type[ItemT | Any] | None and the assignment value being literal Any

[trim tangle]

Can you show the code?

That does seem like a bug, but it could be just a misleading error message

[spice locust]

from __future__ import annotations

from typing import Any

foo: type[str | Any] | None = Any```

[trim tangle]

Ah. type[Foo] requires a class, but you are trying to assign Any to it, which is not a class

What do you want to do with this foo later?

[spice locust]

use it as a runtime type check

as in if foo is not Any: do something, else do nothing

[trim tangle]

If you're just doing isinstance, you can use object (because everything is an instance of object)

[spice locust]

not isinstance

if not self.item_type is Any:
return TypeAdapter(self.item_type).validate_json(response.content)

using it to disable pydantic validation

at runtime

by skipping it if the passed type is None or Any

[trim tangle]

you can change it to if self.item_type is not object:

[spice locust]

why object?

[trim tangle]

Why not?

[spice locust]

idk

its not really the if thats the issue

its the pyright message on the arg type hint

[trim tangle]

The issue, as I said, is that type[...] only accepts class objects -- instances of type. And Any is a special form that is not a class

The error message is definitely wrong

[spice locust]

how can i accept either a type[...] or Any or None?

[trim tangle]

If you really want to keep this interface, the best you can do today is foo: Any (or foo: object)

[spice locust]

if i annotate with Any by itself instead of type[Any] ruff warns about type being too generic and whatnot

but ig its right since it doesnt give me an error...

from __future__ import annotations

from typing import Any

foo: type[str] | Any | None = Any

[trim tangle]

This is (almost) equivalent to foo: Any = Any

as in, you can put anything at all to the right of =

[spice locust]

😬

i only want literal Any object

but ig just None is the next best thing

i wanted Any for semantic reasons

[trim tangle]

Can you show a more realistic version of what you're doing? not as a variable definition

[spice locust]

class Base(Generic[ItemT]):
    @overload
    def __init__(
        self: Self,
        name: str,
        /,
        *,
        item_type: type[ItemT | Any] | None = Mapping[str, DataType],
        base_token: str | None = None,
        project_id: str | None = None,
        host: str | None = None,
        api_version: str = "v1",
        json_decoder: type[json.JSONDecoder] = json.JSONDecoder,
    ) -> None: ...

where item_type is a type that pydantic will validate incoming and outgoing data against during runtime

unless the user chooses to disable it, so thats where my idea for None or Any comes in

logically if i say Base(item_type=Any) that means no validation and Any is returned from the db

[trim tangle]

Yeah, in this case None seems appropriate. py item_type: type[ItemT] | None = Mapping[str, DataType], in which case Mapping[str, DataType] is not an appropriate default

actually, what is the ItemT | Any for?

[spice locust]

originally it was just type[ItemT]

[trim tangle]

If you need to describe something that will work as a type-form-that-pydantic-understands, things might get better with pep 747

[spice locust]

3.14...

i need to support 3.9

[spice locust]

why isnt Mapping an appropriate default?

ItemT = TypeVar("ItemT", bound=ItemType)```

sorry forgot to include this

[trim tangle]

It's a draft, it's not even accepted

[trim tangle]

Suppose that you do py my_base: Base[ThingModel] = Base("my_base") the type of the item_type argument is supposed to be type[ThingModel] | None, but the provided default is Mapping[str, DataType]

It's like this:

T = TypeVar("T")

class Box(Generic[T]):
    def __init__(self, initial_value: T = "lemon") -> None:
        self.value = value

box: Box[int] = Box()  # the default value does not work for this `T`

[spice locust]

oh right

i guess typevar doesnt do that automatically

it goes one way value -> type not type -> value

[trim tangle]

Even if ItemT is a subtype of Mapping[str, DataType], then the default might not be right, e.g. ```py
class MyItem(TypedDict):
foo: int
bar: str

my_base: Base[MyItem] = Base("my_base")

[elfin junco]

Hello,

I'm having trouble with some annotating (python3.12).

class Example:
    @property
    def GenericClass(self):
        class G[M]: ...
        return G

e = Example()

Here type of e.GenericClass shown by pylance is G[Unknown] where I would like it to be a generic class and to be able for instance to do:

class Example2(e.GenericClass[int]): ...

I hope my example is understandable. Thank you in advance for any help you might provide.

[soft matrix]

Things like this are generally very hard to annotate

It might be possible if you define a protocol which has the same interface but I'm not really sure why you'd do this to yourself in the first place

[elfin junco]

Thank you for your response. I'm trying to code a Flask extension that would produce a class like flask_sqlalchemy does (db = SQLALchemy(); db.Model). But I wanted this class to be generic. Also I'm trying to put it in a property rather than just assigning it to self attribute in order to use functools.cached_property to compute the class only once.

On a related note, is it possible to pass a generic class as a method arg and keep it's generic aspect ? e.g.

class Example:
    def __init__(self, generic_base_cls: type[GenericBase]):
        self.generic_base_cls = generic_base_cls

    def init_app(self, app: Flask):
        class MyGeneric[T](self.generic_base_cls[T]): ...
        self.MyGeneric = MyGeneric

Right now I can't make the type checker acknowledge that self.generic_base_cls is generic and thus that self.generic_base_cls[T] is valid. Also, generic_base_cls: type[GenericBase] is flagged by the type checker with "Expected type arguments for generic class"

[terse sky]

Make the function generic

[stiff acorn]

    @xt.setter
    def xt(self, value):
        value = str(value)
        if self._HASH_REGEX.match(value):
            self._hash = value
        else:
            match = self._XT_REGEX.match(value)
            if match:
                self._hash = match.group(1)
        if not hasattr(self, '_hash'):
            raise ValueError(value)

How would a setter like this be typed?

since the code calls str() on the value, i think it's gonna be object?

    @xt.setter
    def xt(self, value: object) -> None: ...

[stiff acorn]

on one hand it can be object because the function calls str() over it, on the other, anything other than a string matching the specific regex will raise the value error

so realistically it cannot be anything other than a string

[cinder bone]

Then just type it as a string

If you type it as an object people will be confused why they can't set it to an int or something

[stiff acorn]

I'll do that, thank you

[spice locust]

there is another issue. if i have a function that takes foo: Thing[SomeType] parameter, and I pass it foo=Thing() which is of type Thing[Unknown], the type checker does not warn

[stiff acorn]

    def write_stream(self, stream, validate=True):
        """
        Write :attr:`metainfo` to a file-like object

        :param stream: Writable file-like object (e.g. :class:`io.BytesIO`)
        :param bool validate: Whether to run :meth:`validate` first

        :raises WriteError: if writing to `stream` fails
        :raises MetainfoError: if :attr:`metainfo` is invalid
        """
        content: bytes = self.dump(validate=validate)
        try:
            # Remove existing data from stream *after* dump() didn't raise
            # anything so we don't destroy it prematurely.
            if stream.seekable():
                stream.seek(0)
                stream.truncate(0)
            stream.write(content)
        except OSError as e:
            raise error.WriteError(e.errno)

how should this function be typed?

is BytesIO appropriate for stream?

define a custom protocol that covers the methods used?

IOBase also looks appropriate

(i think)

https://docs.python.org/3/library/typing.html#typing.BinaryIO this looks the most appropriate option to me

[soft matrix]

its recommended to use your own protocol for things like this

so

class _Stream(Protocol):
  def seekable(self) -> bool: ...
  def seek(self, pos: int, /) -> object: ...
  def truncate(self, pos: int, /) -> object: ...
  def write(self, data: bytes, /) -> object: ...

[rare scarab]

Why not IO[bytes]?

[oblique urchin]

The standard IO classes are poorly defined and large; they're not protocols

[rare scarab]

So we'd make an IOLike[T: (str, bytes)] protocol like Gobot did?

[oblique urchin]

You should write a protocol that's more tailored to your specific needs

[rare scarab]

mhm

[oblique urchin]

Likely not one that's generic over str/bytes, because in concrete applications you probably only need one

[stiff acorn]

Okay, I'll write a custom protocol then. thanks!

[stiff acorn]

How do I handle a dependency that's only used by the *.pyi files (typing-extensions)?

Do I treat it as any other normal dev dependency

Take typing_extensions.Generator for an example

The package I'm writing the stubs for supports >=3.8

So to leverage the new type defaults in the pyi file, do I have to add typing extension as a dev dependency and then import that in my pyi file

[trim tangle]

Yes, if typing-extensions is only used in your pyi files to describe something not available in base Python (i.e. not used at runtime), you can just skip typing-extensions from the package

If necessary, you can mention in the package description that typing-extensions is required if you want accurate type checking information. But don't quote me on that, perhaps mypy/pyright special-case this package?

[stiff acorn]

Adding a typing only dependency would be a bummer

I guess one way to cope with it would just be using 3.8 typing.Generator[something, None, None] instead of the fancier one with defaults

[trim tangle]

right, you don't need to do that

though if Generator is the only thing you're using from typing-extensions, I'd switch to the plain collections.abc.Generator or collections.abc.Iterator

also, why do you have pyi files instead of just adding annotations in the implementation?

[stiff acorn]

Generator was just an example since that changed several times between 3.8 to now, but I'm also using Self for example

[stiff acorn]

It's not my project

[trim tangle]

ah

[stiff acorn]

It's just a pypi project that I happen to use semi regularly so I decided to write hints

Once I'm done I'll PR it to get the author's opinions

If they end up being positive to adding it directly to the implementation, i would happily do that

[trim tangle]

Self can be replaced with a typevar if you really need it

[stiff acorn]

For the first step, I just typed the public API with typing.* variants since I'm on 3.12 and the package supports >=3.8

[trim tangle]

oh right

collections.abc stuff is not subscriptable in 3.8

[stiff acorn]

I was thinking for the next step I'll have to go through the imports to import them from the appropriate modules

Hence my question regarding how dependencies work in pyi files

[trim tangle]

From a quick test, mypy and pyright both seem to assume that typing-extensions is installed. So the stubs should work even if the person running the type checker doesn't have the module installed

actually, mypy has typing-extensions as its own dependency

[stiff acorn]

that's nice, thank you for testing

[stiff acorn]

what's the appropriate way to run mypy over pyi files?

I've got mypy .\pkg\ --strict --exclude '.*\.py$'

oh i guess i don't need to

mypy automatically checks the pyi instead of py

but I have some .py files that don't have an accompanying .pyi file

these aren't part of the public API so i skipped them

[stiff acorn]

so mypy .\pkg\ --strict does complain about them

[oblique urchin]

In stubs it is

[trim tangle]

interesting

[stiff acorn]

is this documented somewhere that I missed?

does that mean I can get rid of these in my pyi files:

if sys.version_info >= (3, 9):
    from collections.abc import Iterable, MutableSequence
else:
    from typing import Iterable, MutableSequence

[oblique urchin]

https://typing.readthedocs.io/en/latest/spec/distributing.html#syntax second paragraph

[oblique urchin]

yes

[stiff acorn]

thank you!

I'm guessing that also applies to re.Pattern?

[oblique urchin]

yes

[stiff acorn]

Also about typing_extensions

is that implicitly assumed to exist by type checkers?

[oblique urchin]

yes

[stiff acorn]

that's amazing, thank you

[tranquil turtle]

if i want to make stub-only package, how do i nest the files?
there is an imgui package (pip install pyimgui -> import imgui) that i want to provide stubs for

i currently have the following:

pyimgui-stubs/ # git repository
    pyproject.toml # pyimgui-stubs package name
    imgui/
        __init__.pyi
``` is this the correct way? it doesnt seem to work right now

oh, now it works
i have no idea what i am doing, but i am happy
thanks

[trim tangle]

I appreciate your dedication to not learning JavaScript

[rare scarab]

First you learn JavaScript, then comes typescript. Eventually you're writing full stack frameworks with vite

[tranquil turtle]

that is a coincidence 😄
imgui has nothing to do with replacing javascript

im looking into justpy to do that

[trail kraken]

Is justpy like dash but with Vue instead of React?

[raw frigate]

Good morning everyone, I'd like to use a Protocol to tell my IDE which attributes are available for a object type. I know a bit about how I could do that, but I don't know if it's possible to make this Protocol be based on a dictionary which contains the attributes.

I'm working with the Invoke library, which has a config class that can be __getattr__'d though the task's first arugment once you collection has been configured.

[raw frigate]

So far, I have to repeat the keys from my configuration dict in my Protocol

config = {"OP_VAULT": ""}


class ContextProtocol(Protocol):
    OP_VAULT: str


class BaseContext(ContextProtocol, Context): ...


@task(autoprint=True, help={"name": "The secret name"})
def get_item(
    ctx: BaseContext,
    name="",
    field="",
    auto_select=False,
    vault="",
):
    """Gets an item from op using fzf"""

[slender iris]

So ctx will be an instance, but this instance is configured with config mapping?

It sound like what your trying to do is say "the keys in the config need to exist in the Protocol". But since dicts dont support dot notation access for values, I dont think theres a straightforward way to do this. In other words, Protocol says this object will have attributes "X, Y, Z", but the keys of python mappings are not attributes.

[stiff acorn]

Can sphinx docs use type annotations from stub files?

[cinder bone]

try it and see

I would be (happily) surprised if that was true though

[fierce ridge]

maybe it's just late in the day and i'm tired, but can someone help me understand why this doesn't work?

from typing import Annotated, TypeAlias, TypeVar, Union

from pydantic import AnyUrl, UrlConstraints


# TODO: Keep an eye on https://github.com/pydantic/pydantic/issues/7353
S3Url = Annotated[AnyUrl, UrlConstraints(host_required=True, allowed_schemes=["s3"])]
r"""An S3 URI. The "hostname" is the S3 bucket name."""


AnyPath: TypeAlias = Union[Path, S3Url]
AnyPathT = TypeVar("AnyPathT", bound=AnyPath)


def joinpath(prefix: AnyPathT, suffix: str) -> AnyPathT:
    match prefix:
        case Path():
            return prefix / suffix
        case AnyUrl(scheme="s3", host=s3_bucket, path=s3_path) if s3_bucket is not None:
            s3_path = (s3_path or "/").rstrip("/")
            suffix = suffix.lstrip("/")
            return AnyUrl.build(scheme="s3", host=s3_bucket, path=f"{s3_path}/{suffix}")
        case _:
            raise ValueError("Input is not a supported path type.")

.../anypath.py:19: error: Incompatible return value type (got "Path", expected "AnyPathT")  [return-value]
.../anypath.py:24: error: Incompatible return value type (got "Url", expected "AnyPathT")  [return-value]

(AnyUrl is a type alias for Url in the pydantic source code)

[trim tangle]

class FancyUrl(S3Url):
    ...

p = joinpath(FancyUrl(....), "/home/")
# p should be a FancyUrl, but it isn't

Why can't you have joinpath(prefix: AnyPath) -> AnyPath? (or make two separate functions?)

If you want to keep this as a single function, you will have to use "constraints" instead of bound=:

AnyPathT = TypeVar("AnyPathT", Path, S3Url)
``` (one reason I dislike this feature is that it doesn't have an obvious name and is impossible to search for)

[fierce ridge]

oh, that was a silly mistake

thanks

the whole pydantic "url" framework is pretty bad and wack anyway

i just didn't want to spend the time to add all the pydantic magic methods to hyperlink's URL classes

fortunately fixing it is on the pydantic v2.10 roadmap https://github.com/pydantic/pydantic/issues/7353

[prisma talon]

How would I typehint a inherited property?
I need to do the typehinting in B as I cannot access A

class A:
  def __init__():
    self.val = 5
  @property
  def val():
    return self.val
class B(A):
  ...
  self.val: int

The solution I came up with was just making a new property (If thats not how super works I can fix that later, im tired RN 😅 )

@property
def val(self) -> int:
    return super().val

[cinder bone]

class B(A):
    val: int

[tranquil turtle]

if TYPE_CHECKING:
  @property
  def val(self) -> int: ...

[prisma talon]

I don't think his worked because it is a property

[cinder bone]

oh I didn't see property oops

[prisma talon]

I will try this tomorrow thanks

[thin trout]

hi, anyone able to help me in https://discord.com/channels/267624335836053506/1293530615295250453 ?

[boreal patrol]

Hey guys, when I run tox -e typing in my project, i got these error:
error: Returning Any from function declared to return "dict[Any, Any]" [no-any-return]
error: Returning Any from function declared to return "dict[Any, Any]" [no-any-return]
and the source code is:

def _load_raw_data(location: str = 'json') -> dict[t.Any, t.Any]:
    """
    Load raw data from the specified location in the request.
    @param location: Specifies the source of the data to load, which can be one of the following:
        - `json` : Load data from the request's JSON body.
        - `form` : Load data from the request's form.
        - `file` or `form_and_files`: Load data from both the request's form and files
            or request's files
        - `json_or_form` : Preferably load data from the request's JSON body or request's form.
        - `query` : Load data from the request's query parameters
    @return: A dictionary containing the data loaded from the specified location.
    @raise ValueError: If the provided location argument is not one of the expected values.
    """
    if location == 'json':
        if request.is_json:
            return request.get_json()
        return {}
    if location == 'form':
        return request.form.to_dict()
    if location in ('files', 'form_and_files'):
        return {**request.files.to_dict(), **request.form.to_dict()}
    if location == 'json_or_form':
        # can't provide both json and data
        if request.is_json:
            return request.get_json()
        return request.form.to_dict() or {}
    if location == 'query':
        return request.args.to_dict()
    raise ValueError(f'Invalid location argument: {location}')

Can someone give some ideas to solve the type error?

[stable fjord]

those errors should be specifying specific lines which should help you narrow down which of the return's are to blame.

[boreal patrol]

sorry, i forget to mark the lines . there is code after i specificed lines .

def _load_raw_data(location: str = 'json') -> dict[t.Any, t.Any]:
    """
    Load raw data from the specified location in the request.
    @param location: Specifies the source of the data to load, which can be one of the following:
        - `json` : Load data from the request's JSON body.
        - `form` : Load data from the request's form.
        - `file` or `form_and_files`: Load data from both the request's form and files
            or request's files
        - `json_or_form` : Preferably load data from the request's JSON body or request's form.
        - `query` : Load data from the request's query parameters
    @return: A dictionary containing the data loaded from the specified location.
    @raise ValueError: If the provided location argument is not one of the expected values.
    """
    if location == 'json':
        if request.is_json:
            return request.get_json() # type check error
        return {}
    if location == 'form':
        return request.form.to_dict()
    if location in ('files', 'form_and_files'):
        return {**request.files.to_dict(), **request.form.to_dict()}
    if location == 'json_or_form':
        # can't provide both json and data
        if request.is_json:
            return request.get_json()  # type check error
        return request.form.to_dict() or {}
    if location == 'query':
        return request.args.to_dict()
    raise ValueError(f'Invalid location argument: {location}')

Flask.request.get_json() return Any | None type, so i shoudle convert Any into dict[Any, Any] ?

[stable fjord]

only if you know it's guaranteed to return a dict. But trying to type json's (or any source of external data) is a nightmare

if request.is_json guarantees it's a dict, then you can # type: ignore[no-any-return safely

[boreal patrol]

Thanks for the reply. I should check whether the return value is None, and wrap it into a dict type or return an empty dictionary directly.

[terse sky]

if you're interested, you can type json most correctly

*more

Json = None | str | int | float | Sequence[Json] | Mapping[str, Json]
MutableJson = None | str | int | float | MutableSequence[MutableJson] | MutableMapping[str, MutableJson]

[hasty phoenix]

Is there a way to copy the type hints from another function into a wrapper function f(*args, **kwargs)?

[oblique urchin]

Depends on your exact use case but ParamSpec may be what you want

[hasty phoenix]

That's a good thing to know, thanks.

My use-case is somewhat more complicated: I have a class A with lots of methods with rather large method arguments. Common to them all is that they carry the argument "key" as the first argument. E.g. def something(key: int, b: int, ...). I want to make a class B which contains key as an attribute. I'd like to mimic the methods of A, but without the key arguments. E.g. def something(b: int, ...). Is there a simple way to construct this for typing? I'd hoped to DRY.

[oblique urchin]

The closest you can come probably is Unpack-with-a-TypedDict

!pep 692

[rough sluice]

**PEP 692 - Using TypedDict for more precise \*\*kwargs typing**

Status

Final

Python-Version

3.12

Created

29-May-2022

Type

Standards Track

[final eagle]

I want to create a decorator that takes a function definition like:

def myfn(a: Input, b: Output, c: Output, d: Input) -> None: ...

and transforms the function definition to:

def myfn(a: Input, d: Input) -> tuple[Output, Output]: ...

is there a way to do that?

(where the location of Inputs and Outputs is arbitrary)

[viscid spire]

is the type of Input the same?

[final eagle]

no

[viscid spire]

then you need 4 typevars

[final eagle]

I think that's going to presume that I know the location of outputs ahead of time

[viscid spire]

def dec[A, B, C, D](f: Callable[[A, B, C, D], None]) -> Callable[[A, D], tuple[B, C]]: ...

[viscid spire]

location?

[final eagle]

e.g. it should work generically for:

def myfn(a: Input, b: Input, c: Output)
def myfn(a: Output, b: Input)

etc.

[viscid spire]

ah

yeah don't think that is possible

that is also exceptionally weird

[final eagle]

hah

here's my use case: https://github.com/atoms-org/cuda-ffi

you can only pass inputs to Nvidia CUDA functions... for outputs of a function, you pass a pointer to the output and it copies data into it

if there were some way to filter out specific types from a TypeVarTuple or a ParamSpec, I could get it to work... but I don't think there's any way to programmatically alter generics

[viscid spire]

Anyone got some good examples of using type defaults? I had read some but I forgot them all

[cinder bone]

from typing_extensions import Generic, TypeVar, assert_type

X = TypeVar("X")
Y = TypeVar("Y", default=X)

class Foo(Generic[X, Y]):
    x: X
    y: Y
    
def wrapper() -> Foo[int]:
    return Foo[int]()
    
assert_type(wrapper().y, int)

That's the only thing I've found them useful for, but you can probably read the PEP for more

[viscid spire]

ooooo, yeah that makes sense

I'm gonna try and think of when this could be useful

class Foo[T, *Ts=*tuple[T, ...]]:

oh also... should type parameter defaults be formatted with spaces around the equals or nah?

[cinder bone]

I would just run ruff or black on it and accept whatever behavior they have

[hearty grove]

Is it possible to specify something like x: type[A & B] to mean "x is a type that inherits from both A and B"?

[oblique urchin]

No. That requires intersections, an idea that has had some traction but hasn't been added to the type system.

[brazen jolt]

best you can do here is probably a protocol, so that you have a class which has specified methods/attributes

[hearty grove]

hmm okay i don't think this solves the problem i'm having regardless

trying to build something that lets me do this:

class Record:
  age: DWORD # this is a subclass of int which encodes serialization info
  name: bytes[4]

r = Record()
r.age = 7 # mypy HATES this
print(r.serialize())

[terse sky]

You could get fancy, but really you should just do r.age = DWORD(7)

or - use a different approach for encoding your serialization info

[trim tangle]

If you want this DWORD only as an annotation for serialization, you could do age: Annotated[int, Dword()]

[terse sky]

^

[trim tangle]

(or make DWORD an alias for that Annotated[int, Dword()])

[hearty grove]

hmm when I do that (inside a TYPE_CHECKING block):

DWORD: typing.TypeAlias = typing.Annotated[int, None]
DWORD(0) # "Annotated" cannot be instantiated

oh, it's pylance that i'm using not mypy

[trim tangle]

right, you can't call a type alias

[hearty grove]

[trim tangle]

But you should be able to do r.age = 7 now

[hearty grove]

(you can call it... but i see what you mean)

[trim tangle]

!e

import typing
X = typing.Annotated[int, None]
print(X(42))

[rough sluice]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

42

[trim tangle]

ok that's cursed

[hearty grove]

and yeah that did work

[oblique urchin]

NewType may also be appropriate for what you want

though I guess that wouldn't allow you to write r.age = 7 as you want

[terse sky]

I kinda think the annotated approach is the way to go here, tbh

[hearty grove]

annotated is probably right yeah

[terse sky]

AFAICS, age is just exactly an int in terms of how it behaves in the program. You just want it to be serialized in a particular way

it's pretty common to use some form of annotation for this rather than type

[hearty grove]

DWORD has a lot of methods (all serializable types do) such as .serialize() -> bytes and .size() -> int

[trim tangle]

the annotated approach also allows for more complex configuration, e.g. ```py
class Record:
age: Dword
name: Annotated[str, PascalString(len_bits=1)]

though I don't know how that will work with delayed annotation evaluation

[hearty grove]

but that is probably nicer

[terse sky]

i guess the only caveat to that is that if your serialization for int is such that you need to bound the value of int

then you may want to consider making it a separate type still

[trim tangle]

you could also use a descriptor

[terse sky]

that was the "getting fancy" approach I was thinking of before

[hearty grove]

my library is more complex and supports silly things like:

class Packet:
  header: DWORD
  data: ByteVector['header']

for VLAs

which mypy absolutely throws a fit at, but that's kind of on me

[oblique urchin]

type checkers are not going to like that

[hearty grove]

yeah noted

[terse sky]

VLAs is not something I expected to hear in a python discord 😛

[oblique urchin]

they'll think header is a forward reference

[hearty grove]

ohhh

right

because that's a thing you can do

[restive rapids]

whatif ByteVector = Literal
but at runtime actually no

[oblique urchin]

yes, type checkers basically assume that subscripting types always means generics

[oblique urchin]

Then type checkers aren't going to let you construct packets that contain any string other than header

Would probably write this as Annotated[bytes, ByteVector("header")] or something similar

[hearty grove]

yeah i was playing with that, let me see what was annoying me about it

i think it was mainly the "everything needs to know how to serialize itself" idea. Maybe at some point i'll retool this to use annotations like a good girl

[tranquil turtle]

~~```py
if TYPE_CHECHING:
ByteVector = Annotated[T, ...]
else:
ByteVector = actual_impl

[thick saffron]

The latest Pyright release (v1.1.384) seems to have broken something in my project... again. Thankfully it's rather small however I do not understand how could I make it happy.

Given this:

ResponseExtractorField = Literal["status_code", "headers", "body", "cookies"]```

I can no longer do
```py
RESPONSE_FIELDS: Sequence[ResponseExtractorField] = ["status_code"]```
as it complains that a `str` is not assignable to these `Literal`s as the list I create is of type `list[str]` which makes sense, but then how would I define a list of literals? ![ducky_concerned](https://cdn.discordapp.com/emojis/1178032077514477629.webp?size=128 "ducky_concerned")

[oblique urchin]

Yeah that seems wrong, would recommend to report a bug to pyright

[hallow flint]

if your code is open source, lmk and i can add it to the mypy_primer regression suite that will flag issues in pyright's ci

[thick saffron]

Looks like it's somewhat more complicated and might have something to do with Pydantic?
https://github.com/microsoft/pyright/issues/9205

[hasty phoenix]

given a function that may return a lot of different types: def get_from_db() -> int|float|str|None. Using this in a specific context where the datatype is known and expected: designed_to_be_int: int = get_from_db("something_int") However this returns typing error due to the other datatypes it might return. Is there a way to state that in this usage it will return int in this case? Casting?

I've been toying with def get_from_db(useful_args, _: type[T]) -> T It can be used with designed_to_be_int: int = get_from_db("something_int", int). Where _ is a fake arg just to give the type hint T to the type checker to deduce the return type. However is just as close as you can get to cast which is kinda smelly.

[oblique urchin]

That sort of interface is inherently not type-safe, so you either need to cast away the error or add a wrapper that adds type safety. For example, a variation on your suggestion could insert an assert isinstance(result, int) call to enforce the value is of the expected type.

[hasty phoenix]

Yeah, thats an idea. Thanks.

def get_int_from_db(*args, **kwargs) -> int:
    result = get_from_db(*args, **kwargs)
    assert isinstance(result, int)
    return result

[viscid spire]

<@&831776746206265384>

tenkiu

[pastel egret]

What I've done in my own code for this situation is create an OptionKey[T] class, instantiated with the expected type object and key name. Store that as a bunch of constants, then pass in to fetch. It reduces the risk of typos since the name is only specified once. By introspecting globals(), you can also fetch a list of all of them, then verify the database up-front.

[terse sky]

_T = TypeVar("_T", bound=int|float|str|None)

def get_from_db() -> int|float|str|None:
    return 5
    
def get_from_db_as(t: type[_T]) -> _T:
    x = get_from_db()
    
    if isinstance(x, t):
        return x
        
    raise RuntimeError("Wrong type!")
    
x: int = get_from_db_as(int)

there's nothing smelly about this, fwiw

This is exactly how you solve this kind of problem in statically typed languages

[grave fjord]

how do I do this: https://github.com/python-trio/trio/pull/3110/files#diff-af569bb81b2e1341afd03b5f42557c76cabed50c418114b3097c29a1cb0515d2R83 on 3.8?

dropping 3.8 is probably happening next release so it's not that important but I'd like to know

oh it's like this https://github.com/python/mypy/issues/4148#issuecomment-338493891

oh no that doesn't work - it has a concrete generic parameter not a TypeVar

[proper nimbus]

confused why the return type is unknown:

T = TypeVar("T", bound=msgspec.Struct|list[msgspec.Struct])

def query2(struct_type: type[T], *args: str) -> T:
    s = send_unchecked("query", *args)
    data = s.recv(4096)
    s.close()
    return msgspec.json.decode(data, type=struct_type, strict=False)


window = query2(Window2, "--windows", "--window")
windows = query2(list[Window2], "--windows", "--space") # lsp warning here

Argument of type "type[list[Window2]]" cannot be assigned to parameter "struct_type" of type "type[T@query2]" in function "query2"
  Type "list[Window2]" is not assignable to type "Struct | list[Struct]"
    Type "list[Window2]" is not assignable to type "Struct | list[Struct]"
      "list[Window2]" is not assignable to "Struct"
      "list[Window2]" is not assignable to "list[Struct]"
        Type parameter "_T@list" is invariant, but "Window2" is not the same as "Struct"
        Consider switching from "list" to "Sequence" which is covariant

[restive rapids]

have you read the last part about variance? list[Struct] means a list that is fine with storing any struct
even though Window2 is a Struct, list[Window2] means a list that is only fine with storing Window2's, not any Struct's
if its decode function doesnt accept Sequence, i dont think you can get a solution without overloads, but, honestly, you could also just not introduce bounds / constraints to the typevar - msgspec itself doesnt do that

[proper nimbus]

I did read that though I'm new to python typing. without the bound, decode takes type[T] without warnings

[proper nimbus]

@restive rapids so I was confused why bound=list[msgspec.Struct] would confuse it since that is what seems to be happening without the bound

[chrome hinge]

Because a list[Window2] isn't a list[Struct] (even though a Window2 is a Struct), because list is invariant

What you probably want here is to use Sequence instead of list in the typevar.

[proper nimbus]

which module provides Sequence?

[oblique urchin]

collections.abc

[proper nimbus]

thanks yall

[iron vapor]

Some typing help needed (or a better solution). In my FastAPI app, I created the following:

class globals:
    def __setattr__(self, __name: str, __value: Any) -> None:
        object.__setattr__(self, __name, __value)


g = globals()

Issue I'm facing is when I use the object, I get a typing error on any added properties that it's not defined, which makes sense in a strictly defined scenario. What's the best solution? Should I be storing to a private object maybe? Or is there a solution to help with the typing? Or a better way to do globals?

[oblique urchin]

If you want type safety, it's better to define an object with a defined set of attributes instead of allowing arbitrary attributes. For example, you could define a dataclass.

If you don't, you could add something like if TYPE_CHECKING: def __getattr__(self, attr: str) -> Any: ... and type checkers will allow getting arbitrary attributes from this object

[iron vapor]

Yah, that's where I was leaning next. I was thinking of a global object to carry values in the project. Heck, I would have just done it with a single variable in this case (to hold the current user), but when I tested that, it doesn't modify the value between modules, which I'm led to understand is the correct behavior.

[spice iris]

Does anyone know of any large(ish) open source projects using pyright for their type checking?

[oblique urchin]

https://github.com/hauntsaninja/mypy_primer/issues/43 lists some

[hybrid warren]

import functools
from typing import Callable, Any, Concatenate


class A:
    def __init__(self, name: str) -> None:
        self.name = name

    @staticmethod
    def decorator[
        **P, T
    ](fn: Callable[Concatenate[Any, P], T]) -> Callable[Concatenate[Any, P], T]:
        def wrapped(self, *args: P.args, **kwargs: P.kwargs) -> T:
            print(f"calling a wrapped function for {self.name}")
            return fn(self, *args, **kwargs)

        return wrapped

    @decorator
    def foo1(self, m: int) -> int:
        """f(m) = m + 10"""
        return m + 10

    @decorator
    def foo2(self, a: float, b: float) -> float:
        """f(a, b) = a * b + 10"""
        return a * b + 10


a = A("my class")
print(a.foo1(1))
print(a.foo2(0.5, 4))

Is there a way to do this but without Any?

can't use Self in staticmethod ofc

but the Any bothers me

[restive rapids]

use a typevar like you did for the parameters and the return type? i think you'd want something like

    @staticmethod
    def decorator[
        C: A, **P, T
    ](fn: Callable[Concatenate[C, P], T]) -> Callable[Concatenate[C, P], T]:
        def wrapped(self, *args: P.args, **kwargs: P.kwargs) -> T:
            print(f"calling a wrapped function for {self.name}")
            return fn(self, *args, **kwargs)

        return wrapped

if the decorator actually relies on the function being a method of A (so it could use self as an A)

[terse sky]

Maybe I'm being dense but why can't 'A' be used? is it a question of working with inheritance?

Or lambda's suggestion seems good too. Would probably be less confusing if it were a generic free function

[hybrid warren]

It does really on self (more so than the example), thank you

[cinder bone]

yes, it's mostly that then if you have a subclass B(A) then the callable you get returned takes Callable[Concatenate[A, P], T] which in reality it could be Callable[Concatenate[B, P], T]

[terse sky]

Sure, I was just curious if OP actually cares about this case

since the original post didn't show it at all

[cinder bone]

well I also just realized it physically isn't allowed lol

if you have something like Callable[[A], T] as a parameter it can't be overriden in inheritance to be Callable[[B], T]

[terse sky]

isn't that exactly what happens when you inherit, in some sense?

the type of self changes

[cinder bone]

yeah but in inheritance it's not hardcoded to be the base class

[terse sky]

Sure? I admit I don't totally understand exactly what the "not allowed" thing is.

[cinder bone]

wait I'm wrong mypy seems okay with it

[terse sky]

decorator is not participating in inheritance at all. if B(A) had foo1 and foo2 overrides, then lambda's solution works just fine.
my solution works just fine as long as you only have A, and not B(A).

[cinder bone]

that's true

[terse sky]

I think in part it's a little less confusing if decorator is a free function, which is why I suggested it

since staticmethods aren't really part of the class at all anyhow

(in terms of things like how they interact with inheritance and so on)

[spice iris]

How might one go about typing the following function so that a type-checker can correctly infer that a defaultdict/nested defaultdict passed to it is now a dict?

def defaultdict_to_dict(dd):
    if isinstance(dd, defaultdict):
        dd = {k: defaultdict_to_dict(v) for k, v in dd.items()}
    return dd

[tranquil ledge]

I don’t think that’s possible to annotate explicitly currently, since Python doesn’t yet have standardized ways to annotate type negations and intersections. In a world where those exist, I’m imagining a valid signature would look like the following:

@overload
def default_dict_to_dict[K, V](
   dd: defaultdict[K, V],
) -> dict[
   K & ~defaultdict,
   V & ~defaultdict,
]:
   ...

@overload
def default_dict_to_dict[T: ~defaultdict](
   dd: T,
) -> T:
   ...

def default_dict_to_dict[T](
   dd: T,
) -> T | dict[
   ~defaultdict,
   ~defaultdict,
]:
   ...

That could probably get more specific with an overload for the non-defaultdict case and an overload for the defaultdict case.

Edit: for clarity and specificity.
Edit2: to with overloads and representation for the passthrough nature of the function.

That probably isn’t enough, actually, since the annotations don’t handle the recursion. Bah.

[fierce ridge]

you can annotate the return type as dict, no?

ah but you want a dict of dicts only, not a dict of defaultdicts?

[spice iris]

Well no, because if it's not a defaultdict it returns whatever it was before

[fierce ridge]

oh

[spice iris]

So it basically returns Any - defaultdict infinitely nested

[fierce ridge]

maybe try writing two functions?

seems like you need to insert a dict annotation somewhere in there

so maybe you can split that into a separate helper function somehow

[trim tangle]

this is not possible to annotate as is because we don't have negation or subtraction

[fierce ridge]

right, but there's no way to work around that in this case?

[spice iris]

I wondered if it can work with an overload but I don't think it's possible because of the recursion

[trim tangle]

Let's set aside recursion for a moment and pretend that it's always {str -> int}

Suppose that you have a th: Thing. defaultdict_to_dict(th) would return Thing if and only if th is not a defaultdict.
If it's definitely a defaultdict[str, int], the return type is dict[str, int]. Otherwise it's a dict[str, int] | Thing
But there are no tools in the type system that would add that knowledge that some th is not a defaultdict, which is required to do this

You can use overloads where defaultdict is first, but that would be a lie

[spice iris]

When you say using an overload is a lie what do you mean exactly?

[tranquil ledge]

Due to the order-dependent evaluation of overloads by popular type-checkers currently, you can sorta use their order to mimic a negation by placing more specific overload cases first. However, without a real way to represent negation, the overloads aren’t technically true? I could be misunderstanding what they’re referring to, but that’s all that comes to mind right now.

[trim tangle]

!d typing.overload

[rough sluice]

typing.overload

@typing.overload```
Decorator for creating overloaded functions and methods.

The `@overload` decorator allows describing functions and methods that support multiple different combinations of argument types. A series of `@overload`\-decorated definitions must be followed by exactly one non\-`@overload`\-decorated definition (for the same function/method).

`@overload`\-decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non\-`@overload`\-decorated definition. The non\-`@overload`\-decorated definition, meanwhile, will be used at runtime but should be ignored by a type checker. At runtime, calling an `@overload`\-decorated function directly will raise [`NotImplementedError`](https://docs.python.org/3/library/exceptions.html#NotImplementedError).

[spice iris]

Right, but when you say "it's a lie" is that because as Thanos says, it's just due to the behaviour of definition order of overloads or more generally overloads are a lie because they are not checked against the method body?

[trim tangle]

Because of what Thanos says, kind of. Suppose you have this:

@overload
def foo[K, V](th: defaultdict[K, V]) -> dict[K, V]: ...
@overload
def foo[T](th: T) -> T: ...
``` if you have this:
```py
def f(th: Thing):
    bar = foo(th)  # bar is inferred as "Thing"
    assert not isinstance(bar, dict)
    assert isinstance(bar, Thing)

class DictThing(Thing, defaultdict):
    ...

f(DictThing(...))  # both asserts fail

Multiple inheritance is something you probably should use very rarely, but it messes up the whole thing

[spice iris]

Oooft yeah that's grim

[terse sky]

fun example

[spice iris]

@trim tangle
I don't know what would happen in the multi inheritance pattern but the following achieves the desired result and mypy is happy:

from collections import defaultdict
from functools import singledispatch
from typing import Any


@singledispatch
def dd_to_d(arg) -> Any:
    return arg


@dd_to_d.register(defaultdict)
def _(arg: defaultdict[str, Any]) -> dict[str, Any]:
    return {k: dd_to_d(v) for k, v in arg.items()}


def func_that_returns_d() -> dict[str, dict[int, int]]:
    indict: defaultdict[str, defaultdict[int, int]] = defaultdict(
        lambda: defaultdict(int)
    )
    for i in "abc":
        for j in range(4):
            indict[i][j] += j
    return dd_to_d(indict)

Mypy strict mode isn't happy
Pyright marked as strict seems to like it though

[light quartz]

Hey! So I'm using classes for type annotation, like so:

from dataclasses import dataclass

@dataclass
class DictionaryItem:
    a: str
    b: int

@dataclass
class Dictionary:
    data: dict[str, DictionaryItem] = {}

    def add(key: str, item: DictionaryItem):
        self.data[key] = item

Does anyone know from the raw compiling speed/efficiency point of view, is there any difference where I create an instance of a class or not?

I mean performance differences in Dictionary.add("testKey", {a: "testVal", b:1})
vs
Dictionary.add("testKey", DictionaryItem({a: "testVal", b:1})) ?

[oblique urchin]

The latter does more work so it'll almost certainly be slower. Whether the difference in speed is important (or even measurable) depends on your use case. It's usually not useful to worry about small speed differences without first profiling your application.

[light quartz]

Thats what I thought too. thanks!

[terse sky]

maybe you already know this but just to be safe - you should never default mutable types in a dataclass that way

it's actually one of the ugliest gotchas in python

hmm is there no python evaluation bot here?

[trim tangle]

should be

!e

from dataclasses import dataclass

@dataclass
class Foo:
    bar: list[int] = []

[terse sky]

actually the nice thing is that it errors now

that's a relatively new thing

[rough sluice]

:x: Your 3.13 free threaded eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 3, in <module>
003 |     @dataclass
004 |      ^^^^^^^^^
005 |   File "/snekbin/python/3.13t/lib/python3.13t/dataclasses.py", line 1305, in dataclass
006 |     return wrap(cls)
007 |   File "/snekbin/python/3.13t/lib/python3.13t/dataclasses.py", line 1295, in wrap
008 |     return _process_class(cls, init, repr, eq, order, unsafe_hash,
009 |                           frozen, match_args, kw_only, slots,
010 |                           weakref_slot)
... (truncated - too many lines)

Full output: https://paste.pythondiscord.com/TAOZIX4BNKA4E2DJ2NBTLFPAEM

[terse sky]

but if you are on an older python Gremlin, make sure you do not do that

[trim tangle]

fancy button

[light quartz]

I actually didn't know that. Thank you

[terse sky]

the reason why is because when you create multiple instances of Foo (in older versions of python), they will all share the same dict instance

so you need to do from dataclasses import dataclass, field

and then bar: list[int] = field(default_factory=list), or similar to that for dict

[light quartz]

Hold on. Are you saying that all instances of classes declared with the dataclass decorator, share a single instance of an attribute?

Declared in the said class

[terse sky]

yeah

I mean, if you're on a newer version of python, then the code doesn't even work

I'm actually confused why this code doesn't work in python 3.8 lol, I wonder if this fix to dataclasses was actually backported to earlier python versions

I know with certainty it used to work

so in terms of vaguely modern pythons, the code just doesn't work so it doesn't matter really

from dataclasses import dataclass

@dataclass
class Foo:
    bar: list[int] = []

would have generated

   def __init__(self, bar = []):
      self.bar = bar

for the init function

so you can see it's just the good old crazy python mutable default problem - just biting you in a less familiar form

[still radish]

I have a few questions about a python code I'm supposed to make.. If anyone can help me pls add me if thats okay

[light quartz]

Well, tbh theres nothing 'familiar' about the mutable default problem to me. This is the first time i hear about it.

So, when using the dataclass decorator, if i want to assign a default value to an attribute, I should do this via the post_init() method?

Would this make the attribute bound to the class instance?

[oblique urchin]

use default_factory

like you'd write quicknir's code as bar: list[int] = field(default_factory=list)

[terse sky]

you'd do this (same as what Jelle wrote)

sorry, the mutable default problem in functions is a common sore point that people come across in python

!e

def foo(x = []):
    x.append(1)
    print(x)

foo()
foo()
foo()

[rough sluice]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | [1]
002 | [1, 1]
003 | [1, 1, 1]

[terse sky]

most people find this behavior really unintuitive

[light quartz]

Ok, just to be clear. The issue is me assigning the default value to an attribute upon declaration?

Im confused because the first example in the dataclass docs provides an examlpe of an int that is assigned a default vale right away quantity_on_hand: int = 0

[oblique urchin]

ints are immutable, so this isn't an issue

[terse sky]

depending what language you're coming from, you might find the statement "ints are immutable" really confusing, because x = 7 sure looks like you're mutating x. Or even moreso, x += 2. But you're actually reassigning x in those cases.

[light quartz]

Huh. Well...I learned something today here. Thank you guys @terse sky @oblique urchin

[terse sky]

!e

x = []
y = x
x.append(5)
print(f"{x=}, {y=}")

xi = 5
yi = xi
xi += 2
print(f"{xi=}, {yi=}")

[rough sluice]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

001 | x=[5], y=[5]
002 | xi=7, yi=5

[light quartz]

Yeah, i just finished reading on that. The inta being immutable was confusing indeed

[terse sky]

this kind of already illustrates the difference

lists are mutable, so when you assign multiple variables to teh same list - when you mutate - it affects all variables "attached" to the list

with integers, they're immutable - even when you do xi += 2 - you're actually reassigning xi to point at a different integer, that holds the value 7. So you don't affect yi.

when you have a default argument in python - every time the function is called, it's always pointing to the same instance of the default.

[light quartz]

Yeah, great examples actually

[terse sky]

the thing is that when you have immutable types, you can't really tell if they're pointing to the same thing or not - how could you tell? it's just an implementation detail.

for mutable types, you have a way to test - mutate one, and see if the rest are mutated or not

fwiw this is worth understanding because you'll see very similar issues to this in many many garbage collected languages. java, kotlin, etc

C/C++/Rust don't have these issues in the same way (they have their own issues 😛 )

(to be clear - other GC languages won't have this weird issue with argument defaults, that's a python special mostly. But they will have the same kind of example with x/y/xi/yi)

[light quartz]

Good to know anyway. And its always nice to be warned about something like this ahead of time instead hitting a bug and sifting over the web to figure it out 🤝

[paper salmon]

apparently since its inception in python 3.7, dataclasses did include a check specifically for lists, dicts, and sets:
https://github.com/python/cpython/blob/v3.7.0/Lib/dataclasses.py#L695-L697

[rough sluice]

Lib/dataclasses.py lines 695 to 697

if f._field_type is _FIELD and isinstance(f.default, (list, dict, set)):
    raise ValueError(f'mutable default {type(f.default)} for field '
                     f'{f.name} is not allowed: use default_factory')```

[paper salmon]

and only in 3.11 did they expand this restriction to forbid all unhashable values:
https://docs.python.org/3/library/dataclasses.html#mutable-default-values

Changed in version 3.11: Instead of looking for and disallowing objects of type list, dict, or set, unhashable objects are now not allowed as default values. Unhashability is used to approximate mutability.

[terse sky]

that's just super duper weird

the only thing I can think is maybe I"m thinking of when we were on 3.6

and I was getting dataclasses from a third party package?

[oblique urchin]

there was a dataclasses backport for 3.6. I expect it worked the same though

[terse sky]

but I distinctly remember hitting this error at some point

[oblique urchin]

maybe you were thinking of some other mutable type that wasn't in the hardcoded list of rejected ones, or you were using some other dataclasses-like library

[paper salmon]

wait a minute, commit e029c53's message says list/dict/set was (first) prohibited in 3.10? but the source i linked earlier was tagged at v3.7.0...

[terse sky]

it definitely wasn't the latter... I don't recall it as being the former but

hmm

[paper salmon]

maybe the commit author simply wrote 3.10 as meaning "current python", i.e. the latest version at the time of development, rather than specifically describing when the feature first existed

[terse sky]

maybe

but I am quite certain that I remember this changing at some point

maybe it was nested dataclasses though

i.e. an inner dataclass that was non-frozen (so unhashable by default), having a default value in an outer dataclass

in fact yeah, I think that was it

so when we upgraded to 3.11, that broke a ton of code

[cinder bone]

Out of curiosity, is typing.dataclass_transform typically special cased in typecheckers or does it to some magic to get typecheckers to believe the right thing

[oblique urchin]

special cased

very heavily

[terse sky]

what are the main things left in typing that typical python devs would use at least somewhat regularly

given python 3.11 and all the various deprecations that have taken place

TypeVar is one that I can think of

[oblique urchin]

I don't know about typical Python devs, but maybe Any, Callable, Literal, Final, overload, TypedDict

TypeVar is not needed in 3.12+

[terse sky]

and Any on occasion

Callable is in abc.collections is it not

[oblique urchin]

right

[terse sky]

what happens in 3.12?

oh, we get the special syntax or something like that?

[oblique urchin]

!pep 695

[rough sluice]

**PEP 695 - Type Parameter Syntax**

Status

Final

Python-Version

3.12

Created

15-Jun-2022

Type

Standards Track

[terse sky]

very nice

The annoying thing is I can't get on 3.12 immediately though lol

at least, probably not.

Final would be a great thing to use in principle, the problem is that it's very verbose atm, afaics

there's no type inference right, so you could go from writing x = some_func() to x: Final[Dict[str, List[People]]] = some_func()

[oblique urchin]

No, you can just write Final

[terse sky]

oh, dang, you're right

I will try to make a point to start using that then

[oblique urchin]

https://typing.readthedocs.io/en/latest/spec/qualifiers.html#syntax

[terse sky]

I'm a little sad because my "typing prelude" idea succumbed to linters/checkers 😕

one star import and things like ruff complain, and it affects their ability to check for other errors, which is really surprising

[oblique urchin]

It's because they do single-file analysis, so after an import * they have no idea what's going on any more. Might change in the future if they rebuild Ruff on top of a typechecker.

[terse sky]

I didn't realize ruff was single file

but I guess that makes sense, since they aren't proper typecheckers

The thing is that it's honestly just a lot for people to keep track of. which things are supposed to come from typing, and which from abc.collections

so just doing it right, once, in a file that has nothing but that, and then letting people star import that file seems worthwhile, at least for all the common annotations

[stable fjord]

I think all linters are single-file

[terse sky]

All python linters you mean?

[stable fjord]

sure

[oblique urchin]

I don't think pylint is

[terse sky]

This is my sort of "transition guide" I wrote for my team; we adopted a lot of the static type checking stuff around 3.6-3.8, and this stuff has evolved pretty quickly since then. Would welcome any feedback - either corrections, suggestions for inclusion (though this is going in an email and it cannot be infinitely long or include every typing best practice - this is just the biggest stuff that's done systematically incorrectly in our codebase mostly as a result of older python versions):

1. Do not use Dict, List, Tuple, Type. The builtins (dict, list, set, tuple, type) now serve this purpose without needing to import anything. Do not use typing.Set for a similar reason - I'll come back to Set.
2. Do not use Union or Optional - just use the | syntax. We could have a longer convo about Optional[Foo] vs Foo | None, but basically the former implies there's actual nesting happening when there isn't - Optional[Optional[Foo]] is the same as Optional[Foo] (this is not the case in e.g. C++ with std::optional).
3. Function arguments should not be annotated with list/dict/set, almost ever. In most cases functions should not mutate their arguments - Use Sequence/Mapping/Set. These are abstract interfaces found in collections.abc that don't allow mutation. If mutation is needed, you can use MutableSequence/MutableMapping/MutableSet. For function returns list/dict/set is fine. For class members it's case by case; try to enforce immutability when it makes sense though (e.g. classes that exist to load data from disk and not have it mutated again).
4. Building on the above - function arguments should generally have the least restrictive annotation that makes sense. In particular, that means you can often use Iterable even, instead of Sequence (just one example).
5. Many things in the typing module that were aliases to things in collections.abc are now deprecated. This includes all the collection interfaces mentioned in 3, but also Iterable and Callable, in particular. Some of the relatively few things you might still find useful in typing: Any, Final, overload, TypeVar (before 3.12).

[stable fjord]

ah, you're right

[stable fjord]

why not rely on linter/type checker rules to enforce this? Or are you doing both

[oblique urchin]

Note that points 1/2/5 can be enforced by a linter. For 2, I'd add that Optional is confusing because an "optional" argument to a function is not the same as an Optional argument.

[terse sky]

well, I can't just naively turn it on

we have tons of code that violates this

and nobody is going to sit down and spend a week just fixing this

[oblique urchin]

Ruff/pyupgrade can fix these automatically

[terse sky]

or even run a script over the whole codebase

most likely we will fix things opportunistically when we're touching a file anyway

[stable fjord]

I'd totes make somebody run pyupgrade / ruff --fix for a couple hours

[terse sky]

i will take a look at using Ruff to fix automatically, being able to do that to a whole file at once when messing with it should save a little time

oh, there was a 6 which is that I introduceda proper recursive union Json type, that I'll be encouraging people to use. Just not that easy to switch to it all at once.

[stable fjord]

in the long run that would surely save a lot of time. You can also make use of per-file-ignores to make it partial

[terse sky]

eh, I'm not convinced that will save time, but lets not argue about that

[young zealot]

Is there a way to typehind a @ancient thistle on a Protocol which returns an instance of the class that is implementing that protocol?

[terse sky]

weird I'm running ruff check <filename> --select UP and not getting anything 🤔

[young zealot]

Something like:

class Something(Protocol):
  @staticmethod
  def make() -> Self:
    ...

[oblique urchin]

your default Python version may not be set right

[terse sky]

yeah you're right

Have to specify the minimum python version

I wonder if there's a way to specify that on the command line

there is a way now I just have to figure out the quoting syntax and everything 😂

so far I have ruff check 'target-version = "py311"' <filename> --fix

it runs but doesn't quite work

very strange. Putting project.requires-python in my pyproject.toml file works. Putting tool.ruff.target-version does not

[cosmic plinth]

say I have the following base class:

class Interface:
    @cached_property
    def config(self) -> BaseConfig:
        return self.config_class()()

    @classmethod
    def config_class(cls) -> type[BaseConfig]:
        return BaseConfig

is there a way to automatically have the config property hinted as the type returned by the classmethod or does every subclass have to define the config property?

[oblique urchin]

you could make the base class generic over the return type of the config_class property

[cosmic plinth]

interesting, I think I vaguely remember doing that somewhere in another situation years ago 😅 do you have an example?

[oblique urchin]

ConfigT = TypeVar("ConfigT", bound=BaseConfig)

class Interface(Generic[ConfigT]):
    @property
    def config(self) -> T: ...
    @classmethod
    def config_class(cls) -> type[T]: ...

class Concrete(Interface[ConcreteConfig]):
    @clasmethod
    def config_class(cls) -> type[ConcreteConfig]: ...

[terse sky]

btw Jelle, I managed to get ruff running, but I cannot seem to make it, or pyupgrade, convert Unions automatically

List, Optional, work

[oblique urchin]

Odd. Optional and Union should have the same logic for conversion (since both were replaced in 3.10)

[terse sky]

yeah, I'm not sure. Have you ever tried this?

[oblique urchin]

If you're relying on from __future__ import annotations, note it will only convert in actual annotations

[terse sky]

I'm not

[oblique urchin]

Then it should only replace Optional/Union in 3.10

[cosmic plinth]

awesome thanks! if I'm on 3.12 is there built-in syntax or is that 3.13?

[oblique urchin]

yes, built-in syntax in 3.12

[terse sky]

I'm running pyupgrade with python 3.11 as the argument

[oblique urchin]

class interface[ConfigT: BaseConfig]:

[terse sky]

and I have a Foo = Union[Bar, Baz] not getting replaced 🤷‍♂️

it's at top level scope (i.e. not in an annotation)

[cosmic plinth]

so sorry to ask again but I want to make sure what I'm doing is conventional: my base class/interface defines not only abstract methods but also makes it so the default config class is BaseConfig. is that okay or should I have the generic class be very minimal just for typing and then my actual base class would inherit from that?

nevermind based on your example that is not okay and indeed I have to have a class just for typing if I don't want to allow for a default config class

[oblique urchin]

You can use defaults, PEP 696

[cosmic plinth]

it doesn't work in my IDE, maybe I have to upgrade VS Code one moment

oh 3.13

[oblique urchin]

In 3.12 you have to use typing.TypeVar(default= ...) for that

[cosmic plinth]

it works thank you so much! I'm fortunate that this project is an application and we can determine the version, it's a bummer this isn't available for libraries until 3.11 is gone

@oblique urchin actually this isn't working on 3.12.6

[oblique urchin]

sorry use typing_extensions.TypeVar

[cosmic plinth]

VS Code gave me completion but running it errors

[cosmic plinth]

do you know what Python version I should limit that dependency to if I want defaults?

[oblique urchin]

3.13

[cosmic plinth]

out of curiosity is that also when the syntax accepts defaults or is that even higher?

[oblique urchin]

no, it's the same

PEP 696 was accepted and implemented in Python 3.13

[cosmic plinth]

superb, thank you again very much

oh bummer, our application is a CLI and typing_extensions introduces a large import cost

❯ python -m timeit -n 1 -r 1 "import typing_extensions"
1 loop, best of 1: 20.3 msec per loop

[oblique urchin]

can potentially put this in if TYPE_CHECKING:, though that's a bit hard with generic classes

[cosmic plinth]

yeah I was trying to think of a way but wouldn't that be impossible with generic classes?

[oblique urchin]

Also I don't think we've tried yet to optimize import cost of typing_extensions; feel free to open an issue and we can see if there's ways to speed it up

[oblique urchin]

Not easy, but there's probably ways to get some gains

[cosmic plinth]

would this work?

if TYPE_CHECKING:
    import typing_extensions ...
    G = Generic[...]
else:
    G = object

it does not

ah it does!

class DeveloperEnvironmentConfig: ...

if TYPE_CHECKING:
    from typing_extensions import TypeVar
    ConfigT = TypeVar("ConfigT", bound=DeveloperEnvironmentConfig, default=DeveloperEnvironmentConfig)
else:
    from typing import TypeVar
    ConfigT = TypeVar("ConfigT")

class DeveloperEnvironmentInterface(ABC, Generic[ConfigT]):

[proper nimbus]

how do I find the inner type of a list?

[rare scarab]

You can't.

[oblique urchin]

Could you clarify what you mean?

[rare scarab]

What's the inner type of []?

[proper nimbus]

I managed to do it this way but I'm wondering how get rid of the warnings

[oblique urchin]

That's what I'd suggest (get_origin/get_args). But yes, that will involve a lot of Any; the type system has great introspection capabilities, but it's difficult to typecheck the introspection code itself

[proper nimbus]

I was hoping since the type var is passed in I could read the concrete type

T = TypeVar("T")

def query(type_parsed: type[T], *args: str) -> T:
    if get_origin(type_parsed) is list:
        fields = msgspec.structs.fields(get_args(type_parsed)[0]) # Argument type is Any
    else:
        fields = msgspec.structs.fields(type_parsed) # Argument of type "type[T@query]" cannot be assigned to parameter "type_or_instance" of type "Struct | type[Struct]" in function "fields"

    names = ",".join(field.encode_name for field in fields)
    print("names:", names)

    with send_unchecked("query", args[0], names, *args[1:]) as s:
        data = s.recv(4096)
        return msgspec.json.decode(data, type=type_parsed, strict=False)

windows = query(list[Window], "--windows", "--space")

[rare scarab]

Maybe have a separate function for lists

[proper nimbus]

can I overload if it's a list?

[rare scarab]

Overloading won't help with types inside

[proper nimbus]

still get the second issue:

T = TypeVar("T")
buf_query = bytearray(4096)

def query_list(type_parsed: type[T], *args: str) -> list[T]:
    fields = msgspec.structs.fields(type_parsed) # Argument of type "type[T@query_list]" cannot be assigned to parameter "type_or_instance" of type "Struct | type[Struct]" in function "fields"
    names = ",".join(field.encode_name for field in fields)
    print("names:", names)

    with send_unchecked("query", args[0], names, *args[1:]) as s:
        read = s.recv_into(buf_query)
        return msgspec.json.decode(buf_query[:read], type=list[type_parsed], strict=False)


windows = query_list(Window, "--windows", "--space")

[rare scarab]

Maybe you need a bound=Struct on your TypeVar

[proper nimbus]

bingo

thanks!

unrelated question (since the context is here), does slicing the bytearray allocate?

since it returns bytearray...

[oblique urchin]

yes, it returns a new bytearray with a copy of the data

[terse sky]

is there a reason why collections.abs.Set does not have union and intersection

but has the operator version of those

seems like the reason is just to make the interface more minimal

actually, more interesting might be why Mapping doesn't support |

I feel like I've come across this before and learned the reason and forgotten 🤔

[lunar dune]

that's because | for mappings was added in Python 3.9, but the Mapping ABC long predates 3.9. It's unfortunately hard to add methods to existing ABCs for backwards compatibility reasons

[terse sky]

hmm really, why is that?

oh, because existing types that implemented the ABC would no longer satisfy it? but what if it was a mixin?

[trim tangle]

stuff in collections.abc is weird because for the purposes of typing, it acts as a protocol

i.e. it's structurally typed

so you could have your own class that only inherits from object, and it would pass as an Iterator if it has the right methods

[lunar dune]

see e.g. https://github.com/python/cpython/issues/77700 for some discussion 🙂

[terse sky]

Hmm, but you only get the mixin implementations if you actually inherit? Is that what you're saying?

[trim tangle]

Yep

[terse sky]

That would indeed be an issue

It's just unfortunate though that there's this backwards compatibility issue, and pressure to be minimal

The main real world use of e.g. Mapping, IMHO is - I want to pass a dict but read only

[trim tangle]

defaultdict bouta end your whole career

[terse sky]

But the way Mapping is evolving is not really consistent with that

[terse sky]

defaultdict isn't a mapping 🙂

[lunar dune]

counterpoint: it is

[oblique urchin]

warning: you just entered one of quicknir's pet peeves 😛

[lunar dune]

great

[terse sky]

Syntactically, yes. Semantically, no

[lunar dune]

I'm strapping in

[terse sky]

It's just a limitation of structural typing

[terse sky]

I feel seen ❤️

I will make one concession - if defaultdict implemented == so that two defaultdicts could compare equal while having different keys then you could argue it's a Mapping

But it doesn't. So operator[] mutates a defaultdict, which violates the semantics of Mapping

[viscid spire]

what does __contains__ do for defaultdicts?

[oblique urchin]

I'll say that on at least one occasion I had the choice between using defaultdict and .setdefault(), then thought of your previous arguments and picked the latter

[oblique urchin]

>>> d = defaultdict(int)
>>> 'x' in d
False
>>> d['x']
0
>>> 'x' in d
True

[viscid spire]

Hmm

[terse sky]

that's another good example where I'd personally argue defaultdict violates the implied Mapping semantics

[viscid spire]

yeah that does seem wrong

I would think any hashable key should return True

[terse sky]

fwiw this has actual practical consequences for your code because given:

def foo(x: Mapping[str, str]):
    ...

d = ...
foo(d)
...

You would expect that whether or not you call foo does not affect anything afterwards, because foo cannot mutate its argument

[trim tangle]

are the semantics for Mapping actually formally specified anywhere?

[terse sky]

but if d is a default_dict then that's obviously not the case

[terse sky]

they're not. I would say it's pretty clear given the existence of MutableMapping that the whole point of Mapping is that the API does not mutate anything - but obviously some people will argue with that.

[viscid spire]

defaultdict should be "a mapping of any possible hashable object of the specified type" to objects of another type

but implemented lazily

(You know... instead of using infinite memory)

[terse sky]

the problem with this is that it makes == very expensive potentially

[viscid spire]

I see

Because then to compare equal you actually create all those missing keys

(?)

[terse sky]

I agree with you that the concept you summarized is the only really correct and consistent thing that defaultdict can represent

[viscid spire]

well I guess __contains__ is O(1)

[terse sky]

but it's just not very practical, so defaultdict doesn't do it, and instead defaultdict is just not very correct or consistent

[lunar dune]

oh I see your point... hmm that is interesting!

[idle ingot]

i am a beginer can anyone help me to solve this error

[terse sky]

the problem isn't creating keys, the problem is calling the default function over and over

lets say A and B are defaultdict

Well, I guess you could assume that the default function always returns something that is equal

[oblique urchin]

You may have accidentally reassigned the print function to some other value. For example, this could happen if a previous line contained print = 0.

[viscid spire]

That would be wrong

[terse sky]

right

so in that case, you'd need to call the default function once for every single key in the symmetric difference of A and B

[viscid spire]

ok that's what I meant

[terse sky]

sorry I misunderstood

another practical issue with defaultdict is this - in my overwhelming experience, people want the "get or insert" behavior in one particular piece of their code, but they still want to get the "get or throw" behavior somewhere else

It's very common that people have some kind of loop where they are "building" the dictionary they want - defaultdict has useful behavior for them, so they use it.
once they're done "building" though they just want to access it like a normal dict.

[idle ingot]

oh thank u very much ❤️

[cunning plover]

i am trying to type this to pydantic and a bit confused.

parameters:
  - $ref: ../../components/parameters/path/organization_id.yaml

What kind of element is it? 🤔

class Smth(BaseModel):
  parameters: list[i don't get it]

https://gema.long2ice.io/
Nvm, yaml to pydantic online service exists 😋

Oh.. right i am stupid

it is regular list of dictionaries

we can define multipe keys and values in a single line of yaml.

havne't recognized it right away

NVM. found

[tacit sparrow]

class A:
    x: int
class B:
    x: int
class C: ...

class Object:
    data: A | B | C | None



class Geometry:
    SUPPORTED_TYPES = A | B

    @classmethod
    def test(cls, obj: Object):
        # Cannot access attribute "x" for class "C"
        if ((data:=obj.data) and isinstance(data, cls.SUPPORTED_TYPES) and data.x == 5):
            pass

Any idea if it's a bug in pyright?

[trim tangle]

type aliases cannot be created at the class level

do you want this to be configured in subclasses?

Actually that's not true, they can be created at class level

!e

class Foo:
    type BAR = 42
    print(repr(BAR))

[tacit sparrow]

no, there will be no subclasses for Geometry

[rough sluice]

:white_check_mark: Your 3.12 eval job has completed with return code 0.

BAR

[trim tangle]

I'm not sure if this is a bug or just a case that hasn't been considered. If you hover over cls.SUPPORTED_TYPES you'll see that the type is inferred as UnionType. But if you change cls.SUPPORTED_TYPES to Geometry.SUPPORTED_TYPES, it is inferred as A | B.

Part of the problem is that you cannot put any annotation on SUPPORTED_TYPES:

class Geometry:
    SUPPORTED_TYPES: ??? = A | B
``` since `SUPPORTED_TYPES` is something-that-can-be-used-in-isinstance, and there's no way to name a union type with specific classes

Depending on why you extracted this into a variable, I'd do one of:

• use isinstance(data, Geometry.SUPPORTED_TYPES)
• replace SUPPORTED_TYPES = A | B with SUPPORTED_TYPES = (A, B)
• use isinstance(data, A | B)

[tacit sparrow]

The idea is that it can be both used for typing (e.g. some other variable being a: Geometry.SUPPORTED_TYPES) and for runtime isinstance. So isinstance(data, Geometry.SUPPORTED_TYPES) is the best workaround for this.
I think it's safe to report this to pyright as a possible bug.

[trim tangle]

I'd suggest opening a discussion in python/typing on github and asking what the behaviour for both mypy/pyright should be

mypy does understand this construct, so it could be a bug in pyright or it could be mypy's own thing that's not allowed by the spec

well, "spec"

[trim tangle]

(1. because it could be a bug in mypy instead, or just a missing use case for the typing system; 2. because pyright issues are prone to being closed as as-designed)

[tacit sparrow]

because pyright issues are prone to being closed as as-designed

True 😄 I reported it to pyright for now, let's see how it will go https://github.com/microsoft/pyright/issues/9277

[burnt scaffold]

Is it possible to achieve the following?

T = TypeVar("T")

# No objects ever get passed to this class, so T is unknown
# It's only used to hold types and is checked against to confirm origin
class Test(Generic[T]):
  ...

test: Test[int] # make it so that int is the typehint, if that makes any sense..

[trim tangle]

Yes, you can use type parameters as markers

class Key(Generic[T]):
    def __init__(self, name: str) -> None:
        self._name = name

DB_KEY: Key[Database] = Key("database")
RATE_LIMITER_KEY: Key[RateLimiter] = Key("rate_limiter")

class Storage:
    def store(self, key: Key[T], value: T) -> None: ...
    def get(self, key: Key[T]) -> T | None: ...

though it would be more convenient to write ```py
class Key(Generic[T]):
def init(self, name: str, type_: type[T] = ...) -> None:
self._name = name

DB_KEY = Key("database", Database)
RATE_LIMITER_KEY = Key("rate_limiter", RateLimiter)

[burnt scaffold]

That's kind of the problem here. The class that stores the type is simply just there for the typehinting, and so is the inner type 😭. I do some witchery with the annotations to generate a pack and unpack function to (de)serialise dataclasses. So it'd be used like this:

class Test(Generic[T]):
  ...

test: Test[int] = 0

My problem with that is that it loses the typehint.

I know that there's a little workaround that makes it work just fine, but it's just kind of ugly and tiring to do.

@packable
@dataclass
class Test: 
  test: Pack[Int] = 0
  
  def __post_init__(self) -> None:
    self.test: int

Is it still possible to achieve what I want?

[terse sky]

I guess it's still not clear to me, fwiw, why you need a class to store a type at all

maybe what you wanted is some kind of test: Annotated[int, Test] or something like that instead?

now test is an int, but it also carries some metadata 🤷‍♂️

[viscid spire]

That sounds correct

[burnt scaffold]

That's exactly what I'm looking for tysm

[leaden ember]

I have a variable cast_dtype that should either be np.float32 (the callable) or np.float64 (the callable). Does anyone know how I can type annotate that?

cast_dtype: np.float32 | np.float64 would treat them as float objects

All good, found out I can do Type[np.float32] etc

[chrome hinge]

yup, or Literal[np.float32, np.float64]

[oblique urchin]

That's not allowed by the type system

[oblique urchin]

You can use just type instead of typing.Type

[chrome hinge]

ah, annoying, I didn't realise Literal was only for a shortlist of types

Type arguments for "Literal" must be None, a literal value (int, bool, str, or bytes), or an enum value

[harsh lantern]

why does pylance says mpf is a variable

[viscid spire]

is mpmath faster than decimal?

Is there proper typing provided for that package?

my Pylance doesn't find a stub file for mpmath

[plain dock]

it's untyped. mp.mpf appears to be defined as an attribute of an MPContext() instance. It appears that this class creates the mpf attribute dynamically, but i'm not sure

[rough sluice]

mpmath/__init__.py line 21

mp = MPContext()```
`mpmath/__init__.py` line 190
```py
mpf = mp.mpf```
`mpmath/ctx_mp.py` line 42
```py
class MPContext(BaseMPContext, StandardBaseContext):```

[viscid spire]

tho it is a type

I checked using print(type(mpf))

[plain dock]

yeah it does stuff funny but that's similar to how random works too

[harsh lantern]

https://github.com/mpmath/mpmath/blob/master/mpmath/ctx_mp_python.py#L702

[rough sluice]

mpmath/ctx_mp_python.py line 702

ctx.mpf = type('mpf', (_mpf,), {})```

[harsh lantern]

💀

[viscid spire]

what the skull

[harsh lantern]

this is the mpf type apparently

maybe its created with the type function so pylance didn't think its a type

[plain dock]

curious if mypy has similar complaints, and if its error message is more helpful

[cinder bone]

You might be able to do something like

class Mpf(mpf):
    ...

?

Actually a better alternative (if you're willing to replace usage of mpf everywhere) might be

Mpf = typing.NewType("Mpf", mpf)

[harsh lantern]

ok

thanks

it worked

[cunning plover]

i remember ther eis a type that says: It is not yet typed

could someone remind me which one it is?

so i could use it instead of Any

and mention that it is actually just not typed because of my laziness, but it is not Any

i could make NewType("TODOTypeLater", Any) as a silly alternative 😄

i saw this type when i used stub lib mypy autogen feature i think

an array of types remained untyped but it was named smth specific

i'll just try to run stubgen and see

[lunar dune]

In typeshed we use _typeshed.Incomplete for this, which is just an alias for Any. Mypy uses this symbol in stubs generated with stubgen, because it's fine for any stubs to import symbols from the _typeshed module. The _typeshed module doesn't exist at runtime, however, so it'll be a bit more complicated to use Incomplete in a .py file

[cunning plover]

Thanks. Good enough to me. So i could just create my own alias for Any and call it Incomplete. Mostly name was interesting to me, how to name it

[lunar dune]

yup

it's a readability thing more than anything else

[cunning plover]

Lazy to type heavily nesting types (for OpenAPI schema generation via pydantic)

decided to just mark them to type later

i probably better to use Enum named somehow like Incomplete though

for the purpose of openapi schema being generated

that says the msg that it is not typed

that will be better than Any, that could be printing misguided info in my generated openapi schema

[forest rampart]

Hello,

I want to override a method in a subclass (in my case FastAPI's APIRouter.get() method) and add a new keyword parameters. Is there a way to doing this without loosing typing ? If I do it by defining my method like this: def get(self, *args, *, errors, **kwargs) I will loose the typing for all FastAPI parameters.

[rare scarab]

That's one of the downsides of overriding methods

[terse sky]

Adding additional defaulted keyword-only parameters in an overriding function makes conceptual sense

[terse sky]

what exactly is the issue? For me this type checks just fine

from typing import override


class Foo:
    def bar(self):
        pass
    
class Bar(Foo):
    @override
    def bar(self, *, is_funny: bool = True):
        pass

[cinder bone]

I'm guessing it's when you do something like *args/**kwargs to pass in arguments to the super method

[forest rampart]

That's easy because you don't have any parameters in parent method. I want to add a parameter to method in the subclass. The base methods have like dozen of params

[terse sky]

It should still be fine? Can you show an example that doesn't type check to clarify what you mean?

Instead of us guessing

[forest rampart]

It's not that it doesn't type check, it's that I have to define all parameters and types in the subclass method, if I use *args, **kwargs then it doesn't type check

`class Foo:
def bar(self, a, b, c=None, d=None, e=None):
pass

class Bar(Foo):
def bar(self, *args, **kwargs, is_funny: bool = True):
# Do something with is_funny
super().bar(*args, **kwargs)`

[terse sky]

Have you looked at ParamSpec?

I don't have much experience with it but I think it is aimed at this kind of problem

[forest rampart]

Yes, and Concatenate but I have not been able to use it in this case, it seems to exists mostly for decorators

[terse sky]

When I have been in this situation in the past I've just repeated the signature fwiw

[forest rampart]

Yes that's a solution, but it's verrrrryyyyyyy verbose 😦

[terse sky]

it's not amazing. but that's sometimes the reality of static typing

do you have multiple derived classes or just the one?

[forest rampart]

Just one, but 5 methods to override with a lot of parameters

[cinder bone]

Is there a way to typehint a method of a specific class?
I'm trying to typehint something like this

def is_foo_method(m: types.MethodType[object]) -> TypeGuard[types.MethodType[Foo]]:
    return isinstance(m.__self__, Foo)

[restive rapids]

uhh.. Callable[Concatenate[Foo, ParamSpec], ReturnType]? not sure where the params & return type would come from
maybe define a protocol with [T]: __self__: T?

[cinder bone]

yeah I'll probably just end up doing that

Is there a reason types.MethodType isn't generic?

[summer sable]

I am a bit confused. What is the equivalent of this code in python 3.12+ bracket notation?

tried a bunch of things with [**T, U] to no avail

None of this i s generating any type hints in vscode either

want to get some retrying/logging wrapper scripts going with respective type hinting but i am having some real trouble doing so

[sullen oak]

what do i do after clicking these?

can someone help me please?

@lone sparrow sorry for the ping but i don't know what to do now to the link that you send me

[lone sparrow]

I think you've made your way to the wrong channel, probably best to head to #python-discussion

But you just pick any of the resources that appear

Any start learnig

[sullen oak]

oh sorry

[summer sable]

think i got it

[harsh lantern]

what should i do with this warning

oh just add return before assert_never

the type checker doesn't say anything

[viscid spire]

I would disable the rule

type checking catches it

[harsh lantern]

it doesn't show up on vscode

[viscid spire]

yeah cuz it's fine

typechecking catches if it's an issue

[harsh lantern]

👍

[plain dock]

a low-quality meme post in another channel has arisen a actual question: when creating type hints for things inside containers, is there anything close to a standard about the order? for some containers, like tuples, order has meaning, but what makes one decide the ordering when there is no meaning? e.g.,

dict[str, bool | str | float | list[float]]
``` i moved the `float` later, to put it next to the similar `list[float]`, and might also prefer subcontainers later for readability (especially large ones), but what of the others? alphabetical? something else?

[vast olive]

not a standard AFAIK, nor does it need one IMO
I just… let the formatter handle it

[raven nova]

hello

[trim tangle]

A | B should be exactly the same as B | A

at least in terms of meaning

If you have a lot of similar types, I'd at least try to be consistent (e.g. pick str | int and stick to it, instead of sometimes doing int | str)

[stable fjord]

is there any formatter that handles this? I've wanted them to be ~sorted to enforce consistency but not found any tool for it

[terse sky]

Yeah it's an interesting point

Maybe some formatter will handle it in the future

The issue is that "technically" it couldn't affect code behavior

[cinder bone]

Hmm maybe it would break with the usage of Annotated though

[stiff acorn]

is __all__ in the package's __init__.py[i] required for type checkers?

[oblique urchin]

No, but if it's present, type checkers pay attention to it

[stiff acorn]

❯ mypy .\hello.py --strict
hello.py:1: error: Module "torf" does not explicitly export attribute "Torrent"  [attr-defined]
Found 1 error in 1 file (checked 1 source file)

This is the error I get but and here's the package's __init__.py

__version__: str = ...

from ._errors import *
from ._magnet import Magnet
from ._stream import TorrentFileStream
from ._torrent import Torrent
from ._utils import File, Filepath

adding __all__ does fix it

but the original author isn't keen on __all__ and I'm just writing stubs for it

[oblique urchin]

from ._torrent import Torrent as Torrent should also fix it

[stiff acorn]

it did! Success: no issues found in 1 source file

Thank you

What was issue, I would like to understand this if you don't mind

[oblique urchin]

Mypy wants to guard against using attributes that are just incidentally imported into a module. For example, this works at runtime because the typing module happens to import sys: ```>>> import typing

typing.sys
<module 'sys' (built-in)>

So we use some heuristics to decide what objects that are imported into a module are meant to be visible externally

One such heuristic is the presence of __all__, the other is the X as X thing I suggested above

[stiff acorn]

that makes sense, thank you!

[stiff acorn]

Do star imports behave differently? I didn't get the same attr-defined error if I imported one of _errors.*

[viscid spire]

without an __all__, star import imports everything except same-alias imports, and variables that begin with single underscore

same-alias being like import xyz as xyz

[stiff acorn]

I'm aware, my question was the Exception classes imported in __init__.pyi get recognized by mypy and pylance without them being in __all__ or X as X

# torf/__init__.pyi
from ._errors import *
from ._torrent import Torrent

# some user importing torf
from torf import Torrent # unknown attr from mypy and pylance
from torf import TorfError # no complaints

[stiff acorn]

how do i typehint these?

    @property
    def creation_date(self) -> datetime | None: ...
    @creation_date.setter
    def creation_date(self, value: int | float | datetime | None) -> None: ...

the setter takes 4 types. First two will get converted to datetime, datetime and None will be kept as is. Which means .creation_date will always be datetime or None

but mypy isn't happy with this

test.py:6: error: Incompatible types in assignment (expression has type "int", variable has type "datetime | None")  [assignment]
    t.creation_date = 199999999999
                      ^~~~~~~~~~~~
Found 1 error in 1 file (checked 1 source file)

implementation:

    @property
    def creation_date(self):
        """
        :class:`datetime.datetime` instance or ``None`` for no creation date

        :class:`int` and :class:`float` are also allowed and converted with
        :meth:`datetime.datetime.fromtimestamp`.

        Setting this property sets or removes
        :attr:`metainfo`\\ ``['creation date']``.
        """
        date = self.metainfo.get('creation date', None)
        if isinstance(date, (float, int)):
            return datetime.fromtimestamp(date)
        else:
            return date

    @creation_date.setter
    def creation_date(self, value):
        if isinstance(value, (float, int)):
            self.metainfo['creation date'] = datetime.fromtimestamp(value)
        elif isinstance(value, datetime):
            self.metainfo['creation date'] = value
        elif not value:
            self.metainfo.pop('creation date', None)
        else:
            raise ValueError(
                'Must be None, int or datetime object, '
                f'not {type(value).__name__}: {value!r}'
            )

[restive rapids]

you might need to use a descriptor
when assigning to a property, it checks if the type matches the property's value type, rather than the one in the setter

[stiff acorn]

What does that mean? Sorry never heard of it before

[restive rapids]

descriptors are the underlying thing for hooking into getting & setting, propertys just wrap around it

from datetime import datetime
from typing import Self, overload, reveal_type


class DateDescriptor:
    @overload
    def __get__(self, instance: None, cls) -> Self:
        ...
    @overload
    def __get__(self, instance: "Thing", cls) -> datetime | None:
        ...
    def __get__(self, instance: "Thing | None", cls) -> Self | datetime | None:
        if instance is None:
            return self
        date = instance._date
        if isinstance(date, (float, int)):
            return datetime.fromtimestamp(date)
        else:
            return date
    def __set__(self, instance: "Thing", value: int | float | datetime | None) -> None:
        if isinstance(value, (float, int)):
            instance._date = datetime.fromtimestamp(value)
        elif isinstance(value, datetime):
            instance._date = value
        elif not value:
            ...
        else:
            raise TypeError

class Thing:
    date = DateDescriptor()
    def __init__(self) -> None:
        self._date: int | float | datetime | None = None

thing = Thing()

reveal_type(Thing.date) # DateDescriptor
reveal_type(thing.date) # datetime | None

thing.date = 0 # fine, type(type(thing).date).__set__ accepts an int

ok well.. after the thing.date = 0 pyright thinks that thing.date: Literal[0], thats.. pretty cursed, not sure how that even works. that might even be a bug worth reporting, and then maybe the property setting thing could be fixed too. on mypy that part is fine, though the property approach isnt

[stiff acorn]

what would be the type of cls here?

[restive rapids]

type, or, well, more specifically, here only type[Thing] would be used, but we never actually rely on it

[stiff acorn]

I see, I've never used descriptors so this is pretty new to me

appreciate the help 😄

[cinder bone]

A fun exercise is to then implement your own property/staticmethod/classmethod ;)

[cosmic plinth]

I can't get this pattern to work, is something like this possible? https://mypy-play.net/?mypy=latest&python=3.12&gist=338e86a591bb22129f23dbf77bebb61f

[trim tangle]

class Foo(Interface[FooConfig]):
    @classmethod
    def config_class(cls) -> type[FooConfig]:
        return FooConfig