type-hinting

[cosmic plinth]

ohhhhh I did misunderstand, that is cool thank you

I'm also investigating this now https://scientific-python.org/specs/spec-0001/ that might be the way to go

[abstract mural]

Nice

[cedar sundial]

I think this is a typing problem. What is the best way to solve this?

try:
    some_var = ...
    CONSTANT = False
    ...
except ValueError:
    CONSTANT = True

if CONSTANT:
    data = ...
else:
    data = some_var[0] # some_var can be unbound

The code will always have some_var be bound to a value in this case, so is there a recommended way of getting rid of the squiggly red lines?

I'd rather not have to do any casts or assertions or define some_var in the except clause as an empty variable, unless that is the only option.

[tranquil turtle]

i would do this: ```py
try:
some_var = 42
CONSTANT = False
except ValueError:
some_var = None
CONSTANT = True

now some_var is of type int|None

if CONSTANT:
data = ...
else:
assert some_var is not None
data = some_var[0] # some_var is bound, inferred to be int

but that violates 2 of your requirements :(

[cedar sundial]

For now, I've just set them as empty lists as that seems the cleanest, even though they won't be used.

[tranquil turtle]

try:
    some_var = ...
    CONSTANT = False
    data = some_var[0]
except ValueError:
    CONSTANT = True
    data = ...

[cedar sundial]

I probably should have provided a better mre. The data is in a loop, so to prevent code duplication, data is pulled out of the try/except

[tranquil turtle]

yeah
this change also would not be possible if there is some code between tr-except and if-else that is required to make data and it depends on CONSTANT

[cedar sundial]

Yeah, I'm either pulling the data from AWS if available, or creating it if not. Anyway, the empty lists seem to work fine 🙂

[tranquil turtle]

there is no way to express relation between two variables such as if X is True, then Y is Unbound, and if X is False, then Y is list[...]

i guess it is called something like "dependent types"

[cedar sundial]

I see. Pythons typing is too young for that right now?

[tranquil turtle]

yes

i guess you can create tuple of CONSTANT and some_var

t: tuple[False, list[...]] | tuple[True]
try:
    some_var = ...
    CONSTANT = False
    t = (CONSTANT, some_var) # tuple[False, list[...]]
    ...
except ValueError:
    CONSTANT = True
    t = (CONSTANT,) # tuple[True]

if t[0]:
    # t is tuple[True]
    data = ...
else:
    # t is tuple[False, list[...]]
    data = t[1][0]
``` (note: use `Literal[True/False]` instead of `True/False` in type context)

[trim tangle]

try:
    some_var = ...
except ValueError:
    some_var = None

if some_var:
    data = some_var[0] 
else:
    data = ...

[rare scarab]

I would use else. ```py
try:
some_var = ...
except ValueError:
CONSTANT = True
else:
CONSTANT = False

[grave fjord]

Can I get someone to look at my PR ? https://github.com/python/mypy/pull/16154

[thick saffron]

somewhat off:
is there any way to validate/lint if a hardcoded string matches a set pattern/format? or is it --currently-- still only possible at runtime?

[proud brook]

Using RegEx?

When writing syntax highlighting extensions for Visual Studio Code, it is possible to use RegEx, so it comes to mind

[trim tangle]

Making a lint rule is definitely possible with something like a flake8 plugin. Could you give an example maybe?

[thick saffron]

gave it some more thought and i think linting it would be somewhat problematic.

ideally i would want to do something like:

SlugStr = StringPattern(r"^[a-z0-9]+(?:(?:-|_)+[a-z0-9]+)*$")  # this is made up

@dataclass
class DummyObj:
    dummy: str
    slug: SlugStr = "default-"  # error
```to have the IDE/editor/CI scream at me right away

[soft matrix]

this is something a linter could still do

but i think the idea of this has been brought up before on python/typing

[thick saffron]

lol, yup; #1202

[undone saffron]

while not directly supported right now, you can do this with newtype + a typeguard and a validation function

[trim tangle]

class Slug:
    def __init__(self, raw: str) -> None:
        if not re.fullmatch(r"[a-z0-9]+(?:(?:-|_)+[a-z0-9]+)*", raw):
            raise ValueError
        self._raw = raw

    def raw(self) -> str:
        return self._raw

or ```py
@dataclass(frozen=True)
class Slug:
value: str
def post_init(self) -> None:
if not re.fullmatch(r"[a-z0-9]+(?:(?:-|_)+[a-z0-9]+)*", self.value):
raise ValueError

Do you really only want to validate the hardcoded strings?

or yes, a newtype

[thick saffron]

in this case, yes

a dataclass or such wouldn't be used, bad example tbh, my bad

[undone saffron]

import re
from typing import NewType, TypeGuard

SlugStr = NewType("SlugStr", str)

def is_slug_str(s: str) -> TypeGuard[SlugStr]:
    return bool(re.fullmatch(YOUR_PATTERN_HERE))
# required use is now:

my_str = "..."
if is_slug_str(my_str):
    # only detected as safe here if things accept SlugStr as their type, forcing it to be checked first.

oh

well, the rest other than the dataclass applies

That's runtime enforcement though

typesystem checks that it's checked at runtime at least, but it's not just checked statically and won't feel good in some places

[trim tangle]

If you have some kind of hardcoded string, you can extract the check to the module level. So if it's wrong it should be caught in the most basic automated smoke test

[undone saffron]

^ if these are hardcoded, moving it to module level and writing a test that anything hardocded conforms is probably your best bet. If they aren't, or it's to be extended by users, the new type + type guard is the best you'll get for static use right now, and if it's for other people, since it has to be checked at runtime anyhow... it's not going to be very ergonomic if they are responsible for it.

[rustic zodiac]

good to know

what is ahwrdrdcode

*hardcoded

[reef yacht]

When you type a specific value into the source code. Instead of it being a variable or configuration or whatever.

[trim tangle]

I think the person was just spamming

[reef yacht]

Who?

[trim tangle]

samthegod
(they were sending a bunch of random messages in a lot of channels, presumably to get access to voice)

[reef yacht]

Weird

[empty hull]

_VoicePlayerT = T.TypeVar("_VoicePlayerT", bound=VoicePlayer)

class VoiceConnectionImpl(T.Generic[_VoicePlayerT]):
    ...
    @classmethod
    async def init(
        self,
        ...,
        *,
        player_cls: type[_VoicePlayerT] = VoicePlayer,
        **kwargs: T.Any,
    ):
    pass  # todo

I get the following error in my classmethod during type-hinting if I try to set VoicePlayer as default arg. I tried setting covariant=True but then it complains that covariant can't be used. I am well aware that I can make a secondary typevar for the same but it feels a bit wasteful to do that. What can the possible solutions besides that be? As a secondary question, is it possible to disable generic scope inside certain parts of, classmethods, or static-methods of the class?

[rare scarab]

generics with a default value are tough. I've solved it in the past by using an overload.

btw, classmethod's first arg should be cls

[cold turtle]

I have a simple script that basically loads a list of website URLs from a file and opens them in separate tabs in a web browser:
https://paste.pythondiscord.com/XMIA

The code works fine, but when editing in VS Code, I get the following warning on line 12:

  Type "OrderedDict | list[Unknown] | str | Unknown" cannot be assigned to type "list[Website]"
    "OrderedDict" is incompatible with "list[Website]" PylancereportGeneralTypeIssues
(property) data: OrderedDict | list[Unknown] | str | Unknown```

If I remove the type hint from `sites`, then that warning gets replaced with a different one on line 15:
```Argument of type "Literal['url']" cannot be assigned to parameter "__key" of type "SupportsIndex | slice" in function "__getitem__"
  Type "Literal['url']" cannot be assigned to type "SupportsIndex | slice"
    "Literal['url']" is incompatible with protocol "SupportsIndex"
      "__index__" is not present
    "Literal['url']" is incompatible with "slice" PylancereportGeneralTypeIssues```

If I instead try to coerce the results with `list(loadyaml(settings.read_text()).data)`, I get yet another error:
```Expression of type "list[str]" cannot be assigned to declared type "list[Website]"
  "list[str]" is incompatible with "list[Website]"
    Type parameter "_T@list" is invariant, but "str" is not the same as "Website" PylancereportGeneralTypeIssues```

I can get the errors to go away if I add `# type: ignore` to line 12, but I would like to know the proper way to pass type checking here. I had assumed that it would automatically recognize that `list[Website]` belongs to `list[Unknown]`, which is a possible return type of `strictyaml.load`. What is the correct way to satisfy Pylance that my types are sufficiently hinted at?

[viscid spire]

for the first warning, you will need to cast the return value with from typing import cast, as you are basically saying which keys are present and the type, without being 100% sure it will match. That may fix your second warning.

...
from typing import TypedDict, cast

Website = TypedDict("Website", {"url": str, "name": str, "nick": str})

def cli():
    ...
    sites = cast(list[Website], loadyaml(settings.read_text()).data)
    ...

[oblique urchin]

I think the first thing to understand is that your code is unsafe, because there is nothing checking that the things you get out from the yaml file are actually of the shape you claim they are. You can work around that with a cast() as suggested above if you are confident that they will be of the right type, or you can write code that explicitly validates the types

[cold turtle]

Thanks, that is a convenient solution, and it does get rid of the warnings.

[cold turtle]

So if I check the input with try-except, the type checker will know what I am doing and allow the assignment to list[Website]?

[oblique urchin]

the type of signal handlers is defined in typeshed here: https://github.com/python/typeshed/blob/1ac5b7b1ac90210eeb51cfa69b5bd600120625c2/stdlib/signal.pyi#L67. The first argument is an int and the second is types.FrameType | None

[rough sluice]

stdlib/signal.pyi line 67

_HANDLER: TypeAlias = Callable[[int, FrameType | None], Any] | int | Handlers | None```

[oblique urchin]

No, try-except doesn't affect type checkers, and in any case the assignment is not checked at runtime. You'd need to write some code that goes over the objects and checks they're of the right type. I wouldn't recommend doing this; either use a cast as already recommended, which may mean you get runtime errors later if the types are wrong, or use something like Pydantic to do the type checking for you at runtime

Not for the standard library, the types for the standard library come with your type checker

you don't need to import anything from that module explicitly, I am just linking it to show the expected types

[heady flicker]

Not necessarily true. I believe, try-except affects "Never" behavior in pyright, though I'd need to double check.

But yeah, in the current case it doesn't ;D

[undone saffron]

try except shouldn't effect never unless you're blind excepting or otherwise catching BaseException, both of which are generally frowned upon. if it is that's probably not safe to do from just type info.

The extent to which it could would be

try:
    something_that_always_raises()  # () -> Never
except:  # can't even be as specific as Exception, 
 # type system doesn't know the aboce isn't something like asyncio.CanceledError
  # do something other than return or re-raise here

pass  # this is no longer unreachable code.

could be one of those "pragmatic decisions" where pyright just assumes you're handling the appropriate exception type given it isn't expressible by the type system

[queen grotto]

how could i create a typing.Protocol subclass that inherits from another class? or is this not the right way of typing this scenario?

my use-case is a discord bot using a commands.Bot subclass to add my own attributes- this subclass is passed through other objects and my typechecker thinks that it's the superclass rather than my subclass, meaning the vsc type checker cant verify it has those attributes

[rare scarab]

you can only inherit from another protocol

[queen grotto]

i think i figured out a workaround, i think commands.Bot has a field for arbitrary data

[dense karma]

can i get mypy to tell me if i missed hinting a variable or return value

[grave fjord]

strict will do this

[grave fjord]

Specifically is disallow_untyped_defs and disallow_incomplete_defs

[trim tangle]

!mute 871562705897664522 spamming a whatsapp advert

[rough sluice]

:incoming_envelope: :ok_hand: applied timeout to @crisp holly until <t:1696105068:f> (1 hour).

[soft matrix]

does anyone know if theres a reason that Generator and Coroutine etc arent protocols at type time considering they behave the same?

[grave fjord]

They collections.abc.Generator and Coroutine are protocols at type time I'm pretty sure

Oh indeed they are not protocols

https://github.com/python/typeshed/blob/main/stdlib/typing.pyi#L409

[rough sluice]

stdlib/typing.pyi line 409

class Generator(Iterator[_YieldT_co], Generic[_YieldT_co, _SendT_contra, _ReturnT_co]):```

[grave fjord]

Probably all the gi_frame and co_frame properties

I think they kinda ruined it really, Generator should have been the bare minimum protocol and GeneratorType should have had all the fancy gi_frame properties

But it's too late to fix it now

@soft matrix that explains why I used register

Ok the fix is to subclass Coroutine and make passthrough properties to the wrapped coro

[soft matrix]

well im just trynna see if theres a better solution in pyright

[soft matrix]

yeah youre definitely right here

ill see if i can fix this actually

[grave fjord]

Too much code is -> Generator: and not -> GeneratorType: and async defs don't get typed as CoroutineType

[lunar dune]

yeah, the whole thing is a mess. lots of stuff in mypy is hardcoded to expect generator functions to return Generator, when they should "probably" be returning GeneratorType if you're being "principled" about things. but trying to fix that now would be a whole lotta effort.

[soft matrix]

I'm willing to give it a go

From my glance it didn't look that bad

[past prism]

I'm the lead maintainer of https://github.com/poljar/matrix-nio
The codebase is full of type comments like this: https://github.com/poljar/matrix-nio/blob/b00c9b8632d3adfb3dfa9b0b115bf74dbd64a234/nio/http.py#L298-L299

Considering that I only intend this project to support versions of python that aren't EOL'ed (so >=3.8) for now, I'd like to replace all of these type comments with proper type hints/annotations as appropriate.
Are there any tools that can do this automatically? Bonus points if it's available as a pre-commit hook
Thanks to anyone who has ideas!!

[rough sluice]

nio/http.py lines 298 to 299

self._message_queue = deque()  # type: Deque[HttpRequest]
self._current_response = None  # type: Optional[HttpResponse]```

[lunar dune]

https://github.com/ilevkivskyi/com2ann is the most popular tool for this, I think (created/maintained by one of the most active mypy maintainers)

[soft matrix]

should this also mention the typing meaning of this https://docs.python.org/3.13/glossary.html#term-generic-function considering generic class is done right below?

[oblique urchin]

yes, and the generic type entry should also mention PEP 695

Honestly that definition just seems wrong to me

The one for generic functions, I mean. I haven't heard of singledispatch being described as a "generic function".

[soft matrix]

neither

[past prism]

Oooh, this looks really interesting, I'll check it out! Ty 🙂

[cunning plover]

is there a way in pydantic to have optional key? 🤔
that is not rendered when doing model_dump()?

oh yeah, there is exclude syntax. neat.

[cunning plover]

except not NotRequired not supported to be defined in pydantic BaseModels🤔

https://github.com/pydantic/pydantic/discussions/3563

kind of supported though, but only in nested values https://github.com/pydantic/pydantic/pull/3687

[rare scarab]

set a default?

class NotSetType:
  pass
NotSet = NotSetType()

class MyModel(BaseModel):
  model: str | NotSetType = NotSet

[cunning plover]

py", line 781, in match_type
    return self._unknown_type_schema(obj)
  File "/home/naa/repos/flowey-api/.venv/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 377, in _unknown_type_schema
    raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'NotSetType'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.

For further information visit https://errors.pydantic.dev/2.3/u/schema-for-unknown-type

Crashes right away, some unkonwn amount of errors ahead

[rare scarab]

I guess you need to use a type known to pydantic.

None will work.

or maybe ...

[reef bronze]

hi i am new

[rare scarab]

or set arbitrary_types_allowed=True in model_config

[reef bronze]

in this server

[rare scarab]

hi, this is the channel for help with type hinting.

[reef bronze]

i learn python coding

so where i start ?? anyone tell me ??

[rare scarab]

For general help, ask in #python-discussion

keep this channel open for discussion about type hinting.

[reef bronze]

okay thnks

[grave fjord]

Will copy.replace get a typing_extensions backport?

[oblique urchin]

No current plans but the thought crossed my mind. It feels a little out of scope for typing-extensions as it's not specifically related to typing

[grave fjord]

But copy.replace will get special support in typing?

Feels like it's worth a backport

[oblique urchin]

Will it? I don't think any type checkers special-case it yet.

[grave fjord]

Seems useless without typing support 😭

[oblique urchin]

Open an issue on typing-extensions and see what people say

[grave fjord]

Maybe it doesn't need special support with

class SupportsReplace[P, T]:
    def __replace__(self, *args: P.args, **kwargs: P.kwargs) -> T: ...

def replace(o: SupportsReplace[P, T, **kwargs: P.kwargs])-> T: ...

[oblique urchin]

no that's illegal, you can't have a paramspec with only kwargs

[grave fjord]

You could type it as supporting args

And rely on __replace__ never having args

The only issue then would be supporting dataclasses on <3.12 that don't have __replace__

I don't think you can @overload your way into supporting regular objects

[oblique urchin]

Also, your type checker would still need to special-case the __replace__ method

[grave fjord]

Or does everything have a __replace__ in 3.13?

[oblique urchin]

No. From grep seems like it was added to dataclasses, namedtuples, some datetime things, CodeType, SimpleNamespace, and some things in inspect

[grave fjord]

But there's a default behavior?

Of batch assigning the kwargs?

[oblique urchin]

    func = getattr(cls, '__replace__', None)
    if func is None:
        raise TypeError(f"replace() does not support {cls.__name__} objects")
    return func(obj, **changes)

So no

[undone saffron]

Is there a reason param spec requires both args and kwargs? This seems to keep coming up and also seems an entirely artificial limitation

[oblique urchin]

I don't recall it being discussed at the time. I think it fits with ParamSpec's general idea that it lets you forward a signature exactly

I feel a kwargs-only ParamSpec would need some marker when it's defined. For example, let's talk about the signature in https://peps.python.org/pep-0612/#id2. If the inner function used only kwargs, that would affect the signature of the outer function, but it's not visible in the outer function's signature.

[undone saffron]

I don't really see an issue there, if both aren't used, it should be possible to infer from that that passed args aren't used (and vice verse), but this is a frequently recurring thing. I don't think most functions I write take both args and kwargs, generally having both means the function is doing too much

[oblique urchin]

It means you have to look at the body of the function to infer its signature, which isn't ideal

And ParamSpec is meant for generic decorators, which should support functions with both args and kwargs

[undone saffron]

That's arbitrarily limiting it to only being good for that instead of also handling every other case where args or kwargs are passed through, would type checkers be able to use args/kwargs hinted as Never here?

[oblique urchin]

Not sure how you'd use Never. If you don't accept *args or **kwargs, wouldn't you just not put those in your signature?

[grave fjord]

It's possible to support just *args with TypeVarTuple

Seems sad there's no way to do kwargs only

[undone saffron]

Well, you said type checkers can't infer from a lack or using p.args/p.kwargs, so next best would be to accept it but error or type it this way in an if type checking block

I don't see why they couldn't, that's still static type info

[trim tangle]

Isn't there TypedDict support for kwargs now?

[grave fjord]

There's no TypeVarDict though

[oblique urchin]

Not sure I understand. The PEP's motivation has an example def with_request(f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]: .... How would that signature line change with a kwargs-only ParamSpec?

Maybe an approach could be to support bounds on ParamSpec, with the semantics being that the signature has to be a subtype of the signature of the bound (probably a callback protocol)

Then you could express a kwargs-only paramspec by setting your bound to a callback protocol with def __call__(self, **kwargs: Any):

[undone saffron]

Or just add a kwargs only way of handling this and deprecate param spec since kwargs + typevar tuple do what's needed

[oblique urchin]

They don't, because of positional-or-keyword arguments

[trim tangle]

oh yeah... sadgeful

[undone saffron]

Wouldn't change. Concatenate is implicit use of args

[trim tangle]

somehow TypeScript manages to have just one "kind of type variable" for everything
it doesn't have kwargs of course, but it does support passing arbitrary arguments to functions

[undone saffron]

ts also allows covariant containers, is only concerned with structural subtyping, and has many other things that can't translate to python. probably isn't a good idea to just try and copy

Use of .args or not should be visible in static typing, concatenate implies it, p.args explicitly uses it

[oblique urchin]

It's implicit use of one positional arg, not of *args. That function could create a new function that accepts only **kwargs.

[undone saffron]

Which is still visible in static type info in the inner scope

If that's too complicated though, I'd point out this is the exact kind of "tooling not being smart makes real code not play well with typing" being discussed elsewhere

[undone saffron]

Actually, it's worse. Both mypy and pyrighy detect using one but not the other, and instead of saying "oh, this isn't used" error

[hardy aurora]

With Django how do you get type hinting for foreign keys?

[grave fjord]

Do you have

[tool.mypy]
plugins = ["mypy_django_plugin.main"]

[trim tangle]

That won't help VSCode

because it uses Pylance

I don't think mypy can provide these hover things at all

[hasty hull]

Does anyone know if it's possible to retain overload signatures while using a decorator that returns a custom class?

This is my current setup:

P = ParamSpec("P")
R = TypeVar("R")
_T = TypeVar("_T")

class Response(Generic[_T]):
    ...

class APIMethodWrapper(Generic[P, R]):
    fn: Callable[Concatenate[Any, P], Response[R]]

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

def decorator(
    metadata: str | None = None,
) -> Callable[
    [Callable[Concatenate[Any, P], Response[R]]],
    APIMethodWrapper[P, R],
]:
   ...

Which works perfectly for non overloaded methods

class Example:
    @decorator(metadata="example_metadata")
    def create(self, *, x: int, y: int) -> Response[int]:
        ...

But breaks down when used with overloads as the response type is turned into a union

class Example:
    @overload
    def create_overload(self, *, x: int, stream: Literal[False] = False) -> Response[int]:
        ...

    @overload
    def create_overload(self, *, x: int, stream: Literal[True]) -> Response[str]:
        ...

    @decorator(metadata="example_metadata")
    def create_overload(self, *, x: int, stream: bool = False) -> Response[int | str]:
        ...

e = Example()
res: int = e.create_overload(x=1)  # error: int | str cannot be assigned to int

The return type is inferred correctly to Response[int] if the APIMethodWrapper[P, R] usage in the decorator is replaced with Callable[P, R].

I suspect this just isn't possible but I'm interested to see if anyone has any ideas!

[carmine phoenix]

is there a way to type-hint an Iterable of characters (string of len() == 1), other than Iterable[str]?

[soft matrix]

sort of

check out shantanu's useful types repo

oh wait no

you can do the inverse

[carmine phoenix]

wdym?

[tranquil turtle]

i guess Gobot means that you can annotate something as Iterable[str] and not str using that repo
Iterable[str] & ~str

[carmine phoenix]

interesting, I've never seen the and/or syntax in type-hints

but Iterable[str] can still be ('afse', 'fes', ...) for example,
there is no guarantee/hint that the string is of length 1

[tranquil turtle]

Iterable[str] & ~str - this is not a real syntax (now)

[tranquil turtle]

yes, there is no way to annotate length of iterables (except for tuples)

[carmine phoenix]

the only thing I could think about is doingchar: TypeAlias = str
the issue is that the linter will remove the alias and it will just be str

[tranquil turtle]

you can do ```py
Char = Literal['a', 'b', 'c', ...] # all 1-char strings
CharIterable = Iterable[Char]

[tranquil turtle]

then maybe NewType can help you

[carmine phoenix]

I think thats more appropriate for this case

[tranquil turtle]

Char = NewType('Char', str)
CharIterable = Iterable[Char]

s: CharIterable = 'abc' # error, str is not compatible with Char

[trim tangle]

my_char = Char("yep this is totally a character")

[viscid mango]

hi

[young zealot]

I have a paramspec P = ParamSpec('P') that is used in a Generic[P] class, and then used in the class with a Callable[P, None] property. In one case I want to use that class where the callable has no paramaters. If I do my_inst: MyClass[None], it forces me to manually pass None to the callable like my_inst.my_prop(None). is there any way to make it so it doesn't need the None to be passed?

[oblique urchin]

MyClass[[]] I believe

[young zealot]

oh yeah that seems to work, thank you!

[trim tangle]

I find it kinda disappointing that we don't have other "paramspec literals"

like, with kwargs

[vocal shore]

https://mypy-play.net/?mypy=latest&python=3.11&gist=a8f0a9d3a79b99f32e0c9d784272cb3f i am trying to apply type annotations to a file that uses class decorators to dynamically add static and instance methods to an enum. i've gotten the instance methods to work, but for some reason mypy thinks that the type of the static method is MyEnum rather than Callable[[], Iterable[T]] as it is declared on the base class. as a result, mypy says MyEnum is not callable when i try to call that static method. any pointers? the playground link highlights some unexpected type deductions in this whole thing

[rare scarab]

No Unpack[T] where T: dict[str, Any]?

[tranquil turtle]

why?
just do **k: Any

[rare scarab]

The point is for T to be a typevar

[trim tangle]

nope

[undone saffron]

seems like an oversight in what should be allowed by taking an overly narrow view. There's no reason Unpack[T] shouldn't work so long as the bound on T is a dict with string keys.

That is to say, the PEP suggests that it should only work for TypedDicts, but a TypeVar presents just as much knowledge about the types from the perspective of someone using it generically, because the use then determines what the string keys and the types of the associated values are

[agile lodge]

hello all, I have quick que
My only need is this code works while excel file is open. how can I do this ?

import time
import openpyxl

file_path = "C:/Users/itopya/Desktop/yield.xlsx"
workbook = openpyxl.load_workbook(file_path)
sheet = workbook.active

value_to_paste = sheet["B2"].value

while True:

next_row = sheet.max_row + 1


sheet[f"D{next_row}"].value = value_to_paste
sheet[f"E{next_row}"].value = time.strftime("%Y-%m-%d %H:%M:%S")


print(f"Pasted value: {value_to_paste}")


workbook.save(file_path)


time.sleep(5)

[trim tangle]

You should see #❓｜how-to-get-help and open a help thread. This channel is for discussing type annotations, not general debugging help

[tranquil turtle]

!pep 729

[rough sluice]

**PEP 729 - Typing governance process**

Status

Draft

Python-Version

3.13

Created

19-Sep-2023

Type

Process

[agile lodge]

ty sir

[languid leaf]

hi folx

i'm causing myself much anguish by trying to type hint something

i'm using the ldap3 library. there's a Connection object that can take a different client_strategy kwarg value and based on that, the search method returns different values

for example an operation id or a four tuple, if it's async (not python async, just you ask the server for the result with your id) or sync

[soft matrix]

Are you using overload?

[languid leaf]

i looked at that

i'm not sure how I would get from kwarg in class constructor to changing the return value of a instance method

I guess I could write my .pyi stubs and just pretend I'm always one client_strategy value, which I actually am

here's some code

import ldap3
server = ldap3.Server("ldaps://example.com")
connection = ldap3.Connection(Server, "cn=manager", "password", client_strategy=SAFE_RESTARTABLE)
status, result, response, _ = connection.search("ou=Discord", "(uid=kronos)", attributes=["uid", "hasNitro"])

[cunning plover]

class ActionBinder(ABC):
    def __init__(
        self,
        _func: Callable,
    ):
        self._func = func

class BasicActionBinder(ActionBinder):
    _func: BasicFunctionType

    # code that uses return self._func(view, serializer)

class QuerysetActionBinder(BasicActionBinder):
    _func: QuerySetFunctionType

   # code that uses return self._func(view, serializer, queryset)
   # code uses functions from BasicActionBinder

i get mypy error in QuerysetActionBinder

Incompatible types in assignment (expression has type "QuerySetFunctionType", base class "BasicActionBinder" defined the type as "BasicFunctionType")  [assignment]mypy
(variable) _func: QuerySetFunctionType

any idea how to fix it in a not awkward way? 😅

Clarification, solution should be friendly to python3.8+

we are probably going to deprecate soon 3.8 and 3.9 and leave only 3.10+ but that's not super confident if already ready to happen 🤔

It feels like it is Generic situation. 🤔

i am just missing something to make it happen

ergh, restructured to

class ActionBinder(ABC):
    def __init__(
        self,
        _func: Callable,
    ):
        self._func = func

    # moved shared functions to here

class BasicActionBinder(ActionBinder):
    _func: BasicFunctionType

    # code that uses return self._func(view, serializer)

class QuerysetActionBinder(ActionBinder):
    _func: QuerySetFunctionType

   # code that uses return self._func(view, serializer, queryset)

then it works okay

no inheritence = no problems 😁

[grave fjord]

You can use _func: Callable = BasicFunctionType

[cunning plover]

where?

[grave fjord]

Where you have _func = BasicFunctionType

[cunning plover]

i don't understand how it would work 🤔

[grave fjord]

Callable defaults to Callable[..., Any]

[cunning plover]

BasicFunctionType is not existing function or type, it is protocol only
assignment happens in def __init__ only

[grave fjord]

Your restructuring is better

[cunning plover]

wait a moment, may be it will work 😅 a moment, will try

nah, it will not work in your idea, i'll keep just restructured version

if i try reassigning type, i just get new errors

[whole quiver]

If I have a dictionary with its key being the class of its values, how would I annotate it? It's like this but compatible with normal dictionary ```py
class MyDict(tp.Protocol):
def getitem(self, item: tp.Type[_T]) -> _T: ...

Currently I'm doing like ```py
class MyDictProtocol(tp.Protocol):
def getitem(self, item: tp.Type[_T]) -> _T: ...
class MyDict(MyDictProtocol, dict):
pass

foo: tp.Any = {}
bar: MyDict = foo

[trim tangle]

_T = TypeVar("_T")

class TypeStore:
    def __init__(self):
        self._store: dict[Any, Any] = {}

    def __setitem__(self, key: type[_T], value: _T) -> None:
        self._store[key] = value

    def __getitem__(self, key: type[_T]) -> _T:
        return self._store[key]

I would do it like this at least

The pattern here is "untyped core, typed shell" or something like this. Use the public interface to ensure some invariant holds

[whole quiver]

I want it to be typing-only though, so it won't affect runtime

[trim tangle]

ah

[trim tangle]

then maybe py class TypeStore(Protocol): def __setitem__(self, key: type[_T], value: _T) -> None: ... def __getitem__(self, key: type[_T]) -> _T: ... def get(self, key: type[_T]) -> _T | None: ... without inheriting from dict

because dict will add a bunch of methods like update which you can use to "compromise" the object, like bar.update({str: 42})

[whole quiver]

Oh I didn't think about those methods compromising the object. I inherited from dict so it can have those methods

[trim tangle]

Yeah, you're essentially inheriting from dict[Any, Any]. So all the dict methods you want don't automatically understand your contract

If you want something like update... it's gonna be pretty challenging to type

well... ```py
def update(self, source: TypeStore, /) -> None: ...

[whole quiver]

There's still the problem with assigning a dict to that will gives a warning but ig I can #type: ignore that part

[trim tangle]

Yeah, at assignment point you'll have to use some escape hatch

[trim tangle]

If you want something like items, it's going to be a bit more challenging. You can, of course, type it as Iterable[tuple[type[Any], Any]]

it all depends on what methods you really need

[whole quiver]

Mostly just set, get and pop, I don't see myself needing items or update in near future

[tranquil turtle]

you cant inherit from non-protocols while defining protocol

i also was thinking about dict[type[T], set[T]] in my project
i think defining protocol with only setitem/getitem is the best option

[trim tangle]

Oh right

[lavish oriole]

I have screenshot of problem

I will send

[fierce ridge]

i wish, imagine if you could write

class FooProtocol(Protocol, Foo):
    ...

and inherit all the annotations of Foo on a protocol?

[royal lagoon]

I just had a thought. Are Python's Protocols in any way relatable to Go's way of handling interfaces?

[trim tangle]

Yes, they are both elements of structural typing

[cunning plover]

Python Protocols are interfaces pretty much. especially if you turn on strict mypy that makes them enforceable almost absolutely like Golang interfaces 😅
(difference at that point only that when Golang Interface accept objects, object has type of the interface and u need explicitely casting back to specific object types in order to call anything that is not declared in interface. Which u should never do pretty much anyway. And python interfaces do not change type of an object, it remains the same, enforcing is done purely at static typing level before runtime. So with using Python interfaces u can break the abstraction much easier and call smth out of object that is not declared in in its interface)

[primal lantern]

Hi everyone! I feel really dumb. The super().foo() call in B.foo() should work, no?

from typing import TypeVar, Generic

T = TypeVar("T", str, int)

class A(Generic[T]):
    def foo(self, object_id: T) -> None:
        pass

class B(A[T], Generic[T]):
    def foo(self, object_id: T) -> None:
        super().foo(object_id)

mypy 1.5.1 says:

main.py:11: error: Argument 1 to "foo" of "A" has incompatible type "str"; expected "T"  [arg-type]
main.py:11: error: Argument 1 to "foo" of "A" has incompatible type "int"; expected "T"  [arg-type]

Playground: https://mypy-play.net/?mypy=latest&python=3.11&gist=da07bb9c6f6c8641ca58d211aa532db5

My understanding is that within B.foo(), T (on parameter object_id) and the T on super().foo() should be the same type, as B passes T to A. What am I doing wrong and how can I make this code work (i.e. I want to override a method and pass arguments with generic type to the overridden method)?

[crisp kestrel]

I want to define a TypeVar for async functions like this: async something(ctx: dict, *args, **kwards) - I tried a Protocol with async def __call__(self, ctx: dict, *args: Any, **kwargs: Any) -> Any: but it doesn't quite work, any hints as to how to do it?

[tranquil turtle]

Generic[T] in class B(A[T], Generic[T]): is not needed
but i dont think it can cause any problems

[oblique urchin]

That's a limitation of how mypy handles TypeVars with constraints

Generally it's better to avoid TypeVars with constraints (as opposed to bounds)

[primal lantern]

Hmmm, okay. The constraint is coming from farther up the hierarchy and a think I can't get rid of it. But knowing that it's a limitation helps. I was worrying that I was misunderstanding something fundamental about signature compatibilty and subclassing and stuff. Thanks!

[primal lantern]

Thanks for the hint!

[ruby ember]

How can I create a test for a protocol ? If i have something such as :

class Func(Protocol):
    def __call__(self, *, df: pd.DataFrame, alter : str) -> pd.DataFrame:
        ...

which is used as

def process(func : Func, data : dict[str, str]): ...

I'm not sure how to write a pytest test to cover the Func protocol 🤔

[cunning plover]

static typing requires no testing.
static typing is part of testing itself. Just enforce protocol with strict mypy (may be strict pyright can do it too) onto all your code

[cunning plover]

All you relly need to enforce it... is doing smth like this in any part of your code
any_var: Func = func

if func is also typed... like def process(func : Func, data : dict[str, str]): ... the one
mypy should not give errors when u assign it to protocol

any_var: Func = process just do anywhere without pytest (but with turned on strict mypy)

[ruby ember]

Yeah - i have mypy strict running and it's fine - but I don't have test coverage for the protocol when running with pytest coverage so thought I might as well add it, and was then unsure how 😅

[cunning plover]

there is no point to test protocol 😛 they are meant to enforce the... the interface, if they work with strict mypy, then they work
just ensure to have in your CI mypy job to check submitted commits

[ruby ember]

sure - it felt a little like testing the language rather than the program, but I was unsure how to do it all the same.

I guess in this case I'd have to ignore the lines from coverage rather than testing them

[cunning plover]

interfaces are kind of static typing test in a nature 😅

[trim tangle]

What do you mean by "testing the protocol" ?

[ruby ember]

i just ignored it in coveragerc... I meant creating a pytest test which would ensure that the protocol was correctly preventing methods with incorrect signatures being passed to it though

[tranquil turtle]

i usually make function like def __typecheck()->None somewhere and put a bunch of x: T = y statements in it to make sure that y is of type T
(i guess you can get the same result with typing.assert_type(y, T))

[oblique urchin]

not quite, assert_type() asserts that two types are exactly the same, not that they're compatible

[tranquil turtle]

oh yeah, i guess that's why i dont use it for "type testing"

[rapid cosmos]

Is this a safe pattern, using @overload, in order to avoid using "PEP 696 – Type defaults for TypeVarLikes" (https://peps.python.org/pep-0696/), which is still a draft and still being implemented in Pyright and Mypy?

from __future__ import annotations
from typing import Generic, reveal_type, overload
from typing_extensions import TypeVar

# T1 = TypeVar('T1', default=None)
# T2 = TypeVar('T2', default=None)
T1 = TypeVar('T1')
T2 = TypeVar('T2')


class MyClass(Generic[T1, T2]):
    @overload
    def __init__(self: MyClass[None, None]) -> None:
        pass

    @overload
    def __init__(self: MyClass[T1, None], type1: type[T1]) -> None:
        pass

    @overload
    def __init__(self: MyClass[T1, T2], type1: type[T1], type2: type[T2]) -> None:
        pass

    def __init__(
        self: MyClass[T1, T2],
        type1: type[T1] | None = None,
        type2: type[T2] | None = None,
    ) -> None:
        pass


myclass1 = MyClass()
reveal_type(myclass1) # Revealed type is "generic5.MyClass[None, None]"
myclass2 = MyClass(float)
reveal_type(myclass2) # Revealed type is "generic5.MyClass[builtins.float, None]"

[trim tangle]

I think that's the way to go, yes

well, it's up to you to decide whether this additional complexity is required so that you can do MyClass() instead of MyClass(None, None)

[rapid cosmos]

@trim tangle I must create a staircase of overloads for a decorator that theoretically deals with variadic type variables. We still don't have variadic type variables in practice and we still don't have typing.Map-like transformations.

[trim tangle]

Ah yeah that's a massive pain in the ass

[rapid cosmos]

I'm basically trying to create types for this thing: https://stackoverflow.com/questions/76234660/type-hinting-instances-vs-types-with-variadic-generics

For Pyside6 (Python for Qt6)

[trim tangle]

We still don't have variadic type variables
what do you mean by this?

[rapid cosmos]

We can't apply t.Type to t.TypeVarTuple
https://github.com/python/typing/issues/1216

[trim tangle]

Ah

[rapid cosmos]

That would eliminate the need for such staircase of overloads

It is an interesting problem for practicing, python is not my main language, but it is such a pain in the ass

[trim tangle]

Yeah it's currently a total pain https://github.com/python/typeshed/blob/main/stdlib/builtins.pyi#L1474-L1512

[rapid cosmos]

Yeah

[trim tangle]

honestly I don't understand that much what the point of TypeVarTuple is, without mapping

[rapid cosmos]

Indeed...

Such a limitation

Even though they said it was super complex to implement in that PEP

[trim tangle]

Have you considered using a single argument instead? If you want, you can use a tuple/NamedTuple/dataclass

[rapid cosmos]

Yes, yesterday I thought about it. I even started to look at the mypy source code and thought about creating a plugin, but I want to avoid it. The problem with using a single argument is that the typings must interface with the Python bindings that the generator (Shiboken) for PySide6 uses to expose it to C/C++ Qt6

Current QtCore.pyi:

class Slot(object):
    def __init__(self, *types: type, name: Optional[str] = ..., result: Optional[str] = ...) -> None: ...
    def __call__(self, function: Callable[P, T]) -> Callable[P, T]: ...

Usage:

from PySide6.QtCore import Slot

class C:
    @Slot(str, int)  # these arguments must match number and type of arguments of foo()
    def foo(self, text: str, num: int) -> None:
        print(f"{text.lower()}, {num.__abs__()}")

C().foo("ABC", -123)

Slot calls the PySide module bindings, passing (str, int), which informs Qts MetaObject system that provides the signals and slots mechanism for dynamic inter-object communication that these are the types that are going to be passed around.

I can create an alternative interface in python that deals with passing around single arguments and then sets up the call to Slot accordingly. Like some sort of @SlotTyped alternative.

[rustic island]

Does anyone know how I can "extend" typing.Annotated?
I want to be able to inject metadata.
That is:

>>> Annotated[int, 1]
typing.Annotated[int, 1]
>>> Annotated[int, 1, 2]
typing.Annotated[int, 1, 2]
>>> WithOne = Annotated[T, 1]
>>> WithOne[int]
typing.Annotated[int, 1]
>>> WithOne[int, 2]
Traceback (most recent call last):
...
TypeError: Too many arguments for typing.Annotated[~T, 1]; actual 2, expected 1

I'd like to be able to do WithOne[int, 2] and get typing.Annotated[int, 1, 2].

[soft matrix]

the only way to do this is to use a type statement

from typing import Annotated, reveal_type

type WithOne[T] = Annotated[T, 1]

reveal_type(WithOne[int])  # type[int]

wait this doesnt seem right ```py
from typing import Annotated, Literal, reveal_type

type WithOne[T, *Ts] = Annotated[T, 1, *Ts]

print(WithOne[int, Literal[2]].value) # typing.Annotated[T, 1, typing.Unpack[Ts]]

[rustic island]

The type statement is new in Python 3.12, right?
Is there a way to do this with Python 3.11?

I actually don't even mind if it comes with some transformation, as long as I can extract all the information.
For example, if I can write WithOne[int, 2] and get typing.Annotated[int, 1, typing.Annotated[NoneType, 2]], that's still good for me since I can extract the information of int, 1 and 2 here.

[soft matrix]

not really no

[rustic island]

Bummer. =(
Thanks for your help, regardless.

[tropic viper]

Hmm, is it possible to typehint decorators in a way that adds return types? Would LSPs even support that?

[trim tangle]

What do you mean by adding return types?

[tropic viper]

def decorator():
    def wrapped(func) -> func_types | dict:
        if some_condition:
            return func()
        else:
            return { "dict": True }
    return wrapped

@decorator
def func() -> list | str | None:
    ...

As an example, typehint this decorator so that it'd include all the possible return types of func, plus dictionary

(also that's entirely whipped up on a whim so if the decorator syntax is off that's just my shaky memory of decorators and how they work)

[trim tangle]

from collections.abc import Callable
from typing import TypeVar, ParamSpec

P = ParamSpec("P")
T = TypeVar("T")

def decorator(func: Callable[P, T]) -> Callable[P, T | dict]:
    def wrapped(*args: P.args, **kwargs: P.kwargs) -> T | dict:
        if some_condition:
            return func(*args, **kwargs)
        else:
            return { "dict": True }
    return wrapped

[tropic viper]

Cool. Now to uh. Figure out how that syntax expands to a decorator with args 😿

Thanks ❤️

[trim tangle]

That's called a "decorator factory" https://stackoverflow.com/a/5929165/10295729

[tropic viper]

Oh yeah, I know how to do the factory, just not how to typehint it in a way that doesn't make my brain melt

Especially when you want to support optional parameters and the decorator syntax without parens, which means you might be returning from the inner or the outer decorator

[trim tangle]

I think supporting both @foo() and @foo is unnecessary complexity

just always require the @foo() version

Yes, typing a decorator factory is extremely verbose, sometimes requiring you to create a Protocol if you're working with generic functions

[plain dock]

i once hinted a decorator that took arguments (thus, a factory) designed to enforce image and domain constraints. it's, uh, not pretty

[tropic viper]

Eh, I suppose. Empty parens decorators feels odd, and it's definitely not the syntax most developers default to when they're recalling how to decorate something as long as there's no parameters. I think it's worth supporting both, especially for a public library or something widely used, but it's definitely a headache.

(aka definitely not this project so that's someone else's problem for the time being)

[tranquil turtle]

did you name it after yourself? 🙂

[tranquil turtle]

it is not always convenient (or possible) to support both ways
how would your decorator know what to do if you pass a single function to it? is it an argument or function being decorated? so you have to disallow callable first args or disallow args (and enforce kwargs)

on the other hand, if you support only @deco(...) and not @deco you might get weird error if you forgot put () after it

[tropic viper]

I mean, if you're using a decorator that takes callables as positional args, I'd argue that's your mistake right there.

Especially if they're optional

[tranquil turtle]

fishhook.hook and einspect.impl take classes as posargs, for example (but arguments are mandatory, so there is no confusion)

i cant recall any other example...

[urban imp]

if i wanted to type-hint cls in a classmethod, what would i type it as?

class Foo:
    @classmethod
    def bar(cls: "???") -> None:
        print("help")

i'm adding flake8-annotations to my project and it's complaining about lack of a hint there. Should i just ignore that one?

[heady flicker]

type[Self]

Or "type[Foo]"

Though the first one is preferable

[trim tangle]

Yes, you shouldn't add annotations to cls and self except some very specific cases. I would use mypy or pyright to complain about missing annotations, and remove flake8-annotations completely

[whole quiver]

How can I delay the evaluation of Self? Like this: ```py
from typing import Self, Callable, TypeAlias

ReturnSelf: TypeAlias = Callable[[], Self]

class Foo:
bar: ReturnSelf

[trim tangle]

I think that's not possible

I guess you can do ```py
from typing import Self, Callable, TypeAlias, TypeVar

T = TypeVar("T")
ReturnSelf = Callable[[], T]

class Foo:
bar: ReturnSelf[Self]

[whole quiver]

Damn

[urban imp]

OK, that's fair, thanks!

one other one, if i have a method that returns a class that is defined within the method, how do i type that?

def foo() -> "???":
    class Bar:
        # do some cool stuff in here

    return Bar

[trim tangle]

Why do you want to do this?

[dull lance]

You can define a Protocol which your local class Bar implements, and type hint the function as returning that Protocol

[crisp kestrel]

How do I add type hints for a partially specified callable? I tried Protocol with def __call__(self, ctx: dict[Any, Any], *args: Any, **kwargs: Any) -> Any: but it is not doing what I want: https://gist.github.com/aucampia/b9f2fd2a627d0ca11790b49b3071a450

task: [mypy] poetry run python -m mypy --show-error-context --show-error-codes 
tests/test_callable_protocol.py: note: In function "test_callable_protocol":
tests/test_callable_protocol.py:22: error: Incompatible types in assignment (expression has type "Callable[[dict[Any, Any], NamedArg(str, 'name')], None]", variable has type "WorkerFunctionT0 | None")  [assignment]
Found 1 error in 1 file (checked 2 source files)
task: Failed to run task "validate": exit status 1

[crisp kestrel]

Maybe I should rather state the problem, I want to create a type variable that only accepts functions which has ctx: dict[Any, Any] as their first argument, but I don't particularly care what the rest of their arguments are, it can be anything

[dull lance]

You can create more protocols to include all the cases of whether args/kwargs exist:

class WorkerFunctionT_NoArgsKwargs(Protocol):
    def __call__(self, ctx: dict[Any, Any]) -> Any: ...
class WorkerFunctionT_ArgsOnly(Protocol):
    def __call__(self, ctx: dict[Any, Any], *args: Any) -> Any: ...
class WorkerFunctionT_KwargsOnly(Protocol):
    def __call__(self, ctx: dict[Any, Any], **kwargs: Any) -> Any: ...
class WorkerFunctionT_ArgsKwargs(Protocol):
    def __call__(self, ctx: dict[Any, Any], *args: Any, **kwargs: Any) -> Any: ...

WorkerFunctionT: TypeAlias = WorkerFunctionT_NoArgsKwargs | WorkerFunctionT_ArgsOnly | WorkerFunctionT_KwargsOnly | WorkerFunctionT_ArgsKwargs

[frigid jolt]

can't he use an overload in the protocol?

[whole quiver]

Using overload means the function must be compatible with all the overloads, this means it needs to support args and kwargs. So no, he can't

[undone saffron]

Do you care about it being bound to the name "ctx" or just that that's the first positional parameter? The latter is doable with just paramspec + concatenate

P = ParamSpec("P")
def passes_context(f: Callable[Concatenate[dict[Any, Any], P], Any]):
    ...


def pass1(ctx: dict[Any, Any]):
    ...

def pass2(ctx: dict[Any, Any], /):
    ...

def pass3(diff_name: dict[Any, Any]):
    ...

def pass4(ctx: dict[Any, Any], something_else: int):
    ...

def pass4(ctx: dict[Any, Any], something_else: int, *args: str, **kwargs: int):
    ...

def fail(not_ctx: str):
    ...

[urban imp]

it's for testing, but... now that i think about it, maybe i don't need to do that. I'll take a look.

[trim tangle]

If you want to do this, a Protocol is the way to go probably

But usually you can avoid creating classes at runtime

[urban imp]

I tried this, but when i instantiate the returned class, mypy complains that the protocol isn’t callable. Maybe it needs to be Type[MyProtocol]?

[trim tangle]

Yeah, if you're actually returning the class

though not sure if it will work

[summer breach]

Anyone know of an existing thing for detecting unnecessary type: ignores -- specifically for pyright?

[soft matrix]

reportUnnecessaryTypeIgnores = true

or something to that effect

[summer breach]

Huh, interesting, one of the pyright issues made me think that only worked for type: ignore without a specific error, but it seems that indeed may work, thanks.

[vale glen]

Hi 👋

I think I asked this before a while back and there wasn't a good solution, but figure it's worth asking again since things are always changing
Bit of a clunky use case, we have a class that looks something like this

class View:
  def fields(self) -> list[str]:
    # Some default logic here

  def render(self) -> str:
    if callable(self.fields):
      fields = self.fields()
    else:
      fields = self.fields

This class provides a default implementation for fields but also supports subclasses overriding the method with an attribute, eg:

class CustomView(View):
  fields = ["a", "b", "c"]

Pylance doesn't like that though as it expects fields to be a method
The error is:

Expression of type "list[str]" cannot be assigned to declared type "(self: Self@FormView) -> list[str]"
  Type "list[str]" cannot be assigned to type "(self: Self@FormView) -> list[str]"PylancereportGeneralTypeIssues

Is there a good way to handle this? Or even disable the type checking without throwing type: ignore everywhere?
We're in a bit deep with this so renaming things would be quite a bit of hassle but that's the only solution I can think of (eg. rename the property to fixed_fields, or just use methods everywhere)

[trim tangle]

Well, using a class where fields is either a field or a method is a bit of a pain

I would suggest just making a method

This is breaking substitutability, because according to the annotations, you should be able to write this function:

def print_fields(view: View) -> None:
    for field in view.fields():
        print(field)

You could make fields a property:

class Foo:
    @property
    def fields(self) -> list[str]:
        return ["a", "b", "c"]
        
class Bar(Foo):
    fields = ["d", "e", "f"]

works with mypy at least ^

I can't check on pyright playground because my cloud provider is experiencing "minor issues"

[soft matrix]

I think Pyright would complain

With incompatible variable override on

[vale glen]

I agree we should probably just stick to using methods in the subclasses, saving that tiny bit of typing doesn't seem to be worth the hassle

Unfortunately using a property doesn't work with pylance, thanks though

[trim tangle]

that's hella cringe

[vale glen]

The error I get is

Expression of type "list[str]" cannot be assigned to declared type "property"

Which I suppose makes sense

[trim tangle]

I guess it would matter if you were accessing Foo.fields, but I don't think it is a helpful thing to mark as an error

[grim garden]

When does it make sense not to have runtime_checkable for a protocol? Performance reasons? but then you wouldnt be calling isinstance yourself anyways, right? It just seems like a thing that is useful and should be included by default to me, but I only read about it for like 2 min

[undone saffron]

It matters for substitutability because of type[Foo]. There are other issues with soundness of type[...] like __init__ that are currently not checked by pyright & mypy, but that doesn't mean that the lack of it being checked has those being correct.

[trim tangle]

soundness is overrated 🙂

[undone saffron]

if people would stop over-using inheritence, they wouldn't have as hard a time with sound typing 🙂

[trim tangle]

I don't think checking __init__ will ever work because then every __init__ would need to work with 0 arguments to satisfy object.__init__

Well, you could make a special case for object

[undone saffron]

object being special cased could be done soundly. that isn't even the only good case for special casing object, see __eq__ and how orms and data science libraries want to use that dunder

just needs to be clear that it is the special case due to not being able to avoid inheriting it with python's object model

[trim tangle]

Do you think having a field x should also not satisfy a protcol with an x property?

[undone saffron]

I think currently yes, but that we would benefit from more clearly delineating protocols and what they are actually defining, and give a way to express both "I just want to be able to do .x and get y" and the way we have currently.

if the type system were more composable and could express more, while remaining internally consistent with only documented exceptions, I believe typing would get both easier and simpler, but that's not a widely shared opinion of what would happen.

[trim tangle]

Do you have some examples maybe?

[undone saffron]

if the type system were fully consistent (and we don't ever add higher kinded types) only publicly exported types that you wanted to "lock" the interface of for reasons of Hyrums's law would ever need to be typed*. everything else could be inferred.

* Well, also native code that wasn't visible to the type checker would still need stubs.

[trim tangle]

Also what do you mean by "soundness"?

[undone saffron]

"That which is defined is checked to be consistent with how it is defined, or in the absence of a definition, that there is a possible consistent type which matches all uses of it"

[trim tangle]

I am not smart enough to parse that

[undone saffron]

A more rigidly defined system that has clearly defined consistent rules, allows tools to infer more without being explicitly told. You might still want to explicitly type certain things to avoid them being changed by other things.

This doesn't mean that devs would need to type more, it would mean tools would need to be smarter about inference.

There have been concerns raised counter to this, but many of those concerns don't apply. Some people have brought up how haskell's inference is great, but then horrible when it can't infer something, but that only ever comes up with higher kinded types. Other languages have shown these aren't neccessary and the costs to things like inference aren't worth it.

but it's hard to get buyin on such a massive change to the way typing is handled. There would be a lot of work involved in creating such a spec, then type checkers implementing it.

[trim tangle]

Have you considered implementing your own type checker that would be based on (some of) these principles?

[undone saffron]

No, I've just been writing less python. After the way many interactions have gone on this topic, I don't see a future for this in python.

[trim tangle]

Maybe people who really like sound type systems just don't tend to write Python 🙂

[undone saffron]

for context, I did consider it. The problem is it requires things that current type checkers allow to be disallowed, and would allow certain things current type checkers don't even have a means to understand. This would conflict with existing user tooling, including that of common IDEs, so there's no way for such an example to ever gain traction without ecosystem buyin

[trim tangle]

Maybe I'm just bad at imagination, but for me all of this is really abstract, and I don't understand what the big changes that you mention would be

I definitely don't enjoy Python's type system in its current shape, and I'm happy to consider something nicer and more expressive

(I'm a typescript fanboy (which is also not sounds though))

[undone saffron]

In my ideal, In a pure python library, you'd only want to type a few protocols/types for what's exported to users and let the type system infer everything else, checking that the exposed boundary that's typed is consistent with all of what the library code does.

(but with changes that power and enable that inference capability)

google's pytype is somewhat closer to my ideal here, but there's a lot they dont infer (and last I checked, they still only supported python 3.8)

[trim tangle]

so like, infer argument types as structural types based on usage inside the function?

[trim tangle]

Well, mypy didn't create a big overarching spec, then implement it, then show to Python users which proceeded to immediately adopt it

It started out as a tool speaking its own language, and then people saw that it's pretty useful

[oblique urchin]

It doesn't work very well (necessarily). For example, it doesn't work with protocols with data fields, and it doesn't check the signature of methods, only their existence

[trim tangle]

Yeah, it's a bit strange

[oblique urchin]

I don't really understand why this is a blocker. From what you said on Discuss, the changes to the existing type system are fairly minor. I don't see why a new tool with a slightly different interpretation of the type system couldn't be started and see use if people find it useful.

[undone saffron]

Because what's essentially being said is that even though I can show that is is provably an outcome of such changes from mathematical theory alone, there's no interest in it from decision makers. Why would I spend the time on such a prototype in that situation, instead of spend that time moving the code that I want that level of safety in to a language that supports it, and only after that, consider if I have the time?

[oblique urchin]

I don't see why you need "decision makers". You can write this tool, and show that it is useful. It is not reasonable to expect others to jump on something and standardize it without proof that it works in practice.

And of course, it's your right to use another language instead

[undone saffron]

I'm not going to spend my free time making something for python that requires changes to the type system to work if there's no buyin on even the idea. There was actively expressed disdain for the academic proofs provided that all of the outcomes I claimed were not only possible, but practical.

At the end of the day, any work on that only has value if there's a chance of adoption, and I just don't see that happening given the discussion.

unless you view "showing people it was possible out of spite" providing value, which I don't, spite isn't a motivator for me. I've been a little disappointed by people in those discussions, but I'll keep using python where it provides value for me, and I'll use other tools that better work for me when I need other tools.

[oblique urchin]

Adoption relies on end users though, not necessarily what the standard library decides

[undone saffron]

conflicting with existing tools isn't a realisticly useful path for what I want from this, and those conflicts are intertwined with decisions of the standard library and other offical projects like the typeshed.

if my type checker allows almost no types to be needed, but users complain that their IDE says something isn't typed, did that help me?

it really does require more than just having a prototype, and I don't see a realistic path forward given the individual stances of all involved.

[rapid cosmos]

I have been using both pyright (pylance) and mypy for awhile. I think the pyright experience when it comes to typing has been slightly smoother.

Reading the docs that explains the differences between them

[trim tangle]

I actually stumbled upon this problem right now... I wanted to specify that I accept functions of this shape: py def foo(world: World, entity: Entity, foo: Foo, bar: Bar, baz: Baz) -> None def foo(world: World, entity: Entity, foo: Foo, bar: Bar) -> None def foo(world: World, entity: Entity, foo: Foo) -> None def foo(world: World, entity: Entity) -> None basically (world: World, entity: Entity, *args: Any) -> None, and then I was planning to inspect annotations and do interesting things

But in Python's type system, it means that the functions I accept must accept any kind of arguments, not that the signature to the right of these two arguments doesn't matter

oh well, maybe I'll just go all in and make World and Entity optional in case the handler doesn't need them.

but kinda sad

[trim tangle]

sigh

[tranquil turtle]

case _: assert False

[trim tangle]

😩

[plain dock]

what if the type hints are not followed?

[trim tangle]

Then there will be a runtime error

the type checker assumes the type annotations will be respected, otherwise it wouldn't be able to infer anything

[oblique urchin]

you might be aware but this has previously been reported to pyright and Eric rejected it

it just doesn't work well with how they do type narrowing

[trim tangle]

loud sigh

[plain dock]

eric traut rejects a good idea for fixing pylance, what a surprise

[soft matrix]

https://github.com/microsoft/pyright/issues/5824 that was my issue for it

[trim tangle]

maybe it is really time to make a new type checker 🙂

[trim tangle]

There are actually no bugs in pyright, everything is by design

Question of the day: let's assume we made a new type checker that can infer the type definition of a function based on the implementation.
What does the checker infer as the type of this function?

def quadratic(a, b, c, x):
   return a*x*x + b*x + c

[oblique urchin]

some hellishly complicated generic function

[trim tangle]

yeah

[oblique urchin]

even for def f(a, b): return a + b the signature would be quite complicated

[bleak imp]

A protocol with mult, that has a return of a protocol with add?

[oblique urchin]

what about __rmul__

[bleak imp]

Interesting. I suppose then it would be splitable into a bunch of different possible ones, for each workable configuration of mul, rmul, add, and radd.

[trim tangle]

Well, it would be something like f[A, B, C where CanAdd[A, B, C]](a: A, b: B) -> C where CanAdd is some new entity representing an N-way protocol (which is probably built-in because it has to know the hardcoded rules about __add__ and __radd__)

can you even express this in the current type system?

[oblique urchin]

ignoring __radd__ for now, I guess you'd need a protocol class SupportsAdd[T, U]: def __add__(self, other: T) -> U:

[tranquil turtle]

i would like arg types and return type to be inferred from use cases
if you only pass ints to it - it accepts ints and returns int
if you pass floats - now it accepts ints and floats and returns int or float depending on arg types
if you pass some nonsense to it - typechecker should raise error

(now when i think more about it, it is actually very similar to what Jelle said)

[oblique urchin]

and then def f[B, C](a: SupportsAdd[B, C], b: B) -> C:

[bleak imp]

I suppose it could be something like f(a:add, b:match a's type) or f(a:match b's type, b: radd) but I don't remember anything letting that sort of cross type hint options work.

[oblique urchin]

for radd I think you'd need an overload

[bleak imp]

The main issue is that this isn't expressible with unions, because a: add|match_b, b:match_a|radd would have a:match_b, b:match_a and a:add, b:radd be valid configurations, when they shouldn't be.

[trim tangle]

so for the original... ```py
def quadratic[
A, B, C, X,
T1, T2, T3, T4, T5,
where
CanMul[A, X, T1],
CanMul[T1, X, T2],
CanMul[B, X, T3],
CanAdd[T2, T3, T4],
CanAdd[T4, C, T5],
](a: A, b: B, c: C) -> T5

So my hypothesis is: if we take a classic approach to type inference, we end up with most type signatures being way too complicated and just replicating the implementation

[oblique urchin]

it's like Haskell where you can write a type and the implementation can be generated for you

[trim tangle]

Maybe I'm asking the wrong question. Maybe the type checker will have some other way of tracking this information

[trim tangle]

Well not quite, but in Idris and Agda you can get close sometimes

[bleak imp]

That really puts the algebra into algebraic typing system

[tranquil turtle]

imagine reading error messages about this function

[trim tangle]

What errors, I write correct code every time

[tranquil turtle]

i guess that will contradict the Zen

!zen errors

[rough sluice]

The Zen of Python (line 9):

Errors should never pass silently.

[tranquil turtle]

oh wait, not this one
upd: i think i imagined a zen line :(

[bleak imp]

I can image being able to implement nicely reading errors, but the thought of implementing a system for automatically generating them seems impossible.

[soft matrix]

pyright cant even tell me why 2 dicts are incompatible if theyre too complicated

[trim tangle]

I think the reason Haskell and others have pretty much full type inference is because of the very simple type system

as you start adding GHC extensions, you lose some of that

[tranquil turtle]

i think this is a pretty bad idea for several reasons:

• your function may accept more than you intended because of typechecker being too smart
• typechecker can infer some nonsense if your implementation is too complicated (just imagine signature for function that does x += y in a loop)
• some problems can arise when trying to combine "autotyped" functions with manually typed ones (like use one from another)
• errors may be raised in placed that are not close to the source of problem at all
• error messages will be like in C++
• performance problems (?)

[trim tangle]

Yeah I'm not saying it's something we should do. I'm more trying to say that we might need a completely new paradigm if we want to combine inference with very complex behaviour

Or maybe there should be some kind of trigger that infers more monomorphic types, like (float, float, float, float) -> float (or is it (int, int, int, int) -> int?).
But it's not clear how that would work with user-defined types

Maybe the type system will simply not permit creating such confucktions in user code

[tranquil turtle]

maybe we need some chatgpt-based plugin to infer types from docstring and implementation

[trim tangle]

that's called an intern

A bigger problem might arise when we add conditionals, loops, and mutable state... there's going to be a lot of undecidable stuff perpahs

[bleak imp]

One other fun thing is that parenthesis can change the signature. If you had a*(x*x) then it would be CanMul[X, X, T1], CanMul[A, T1, T2]

One thing that does strike me is that this is getting very close to just being compiled, since that type hint is basically the functions ast.

[trim tangle]

!e

def quadratic(a, b, c, x):
    return a*x*x + b*x + c

print(quadratic("O", "o", "f", 2))

[rough sluice]

@trim tangle :white_check_mark: Your 3.12 eval job has completed with return code 0.

OOOOoof

[tranquil turtle]

is there a proposal to add exception information to type system?
like @typing.can_raise(ValueError, MyEroor)

i guess this is not that simple, because basically every operation can raise something

[bleak imp]

Another good question is when does typing become the wrong tool? I feel like at a certain level of complexity, you might just need to write tests instead of types.

[deep saddle]

why am I getting this error? ```py
def wrap(f: Callable[P, T]) -> Callable[P, T]:
def wrapped(*args: P.args, **kwargs: P.kwargs) -> T:
print("blah blah")
return f(*args, **kwargs)

return wrapped

f: Callable[[int], float] | Callable[[float], int] = import("however the function is defined")
wrap(f)
Argument of type "((int) -> float) | ((float) -> int)" cannot be assigned to parameter "f" of type "(**P@wrap) -> T@wrap" in function "wrap"
  Type "((int) -> float) | ((float) -> int)" cannot be assigned to type "(float) -> float"
    Type "(int) -> float" cannot be assigned to type "(float) -> float"
      Parameter 1: type "float" cannot be assigned to type "int"
        "float" is incompatible with "int"```

the function f is obviously compatible with the generic wrapper, why is pylance so worried about it

is there a way for me to resolve this without editing the typing on f itself? (since I'm getting that function from a library, and its typing is accurate already)

obviously i have this above py P = ParamSpec("P") T = TypeVar("T")

i am fine to adjust my typing on wrap - I just want it to be "f is anything, as long as it's callable, and the wrapped return value has the same signature"

T = TypeVar("T", bound=Callable) doesn't really work because then I can't correctly type the return value for the inner wrapped() -> ???:

[trim tangle]

C = TypeVar("C", Callable)

def wrap(f: C) -> C

and then slap type-ignores in the definition of wrap

actually, what would you do with a value of Callable[[float], int] | Callable[[int], float]?

You can't check which of these variants the function is

[deep saddle]

is that the only way, trough 3.12? i'd prefer not to

[trim tangle]

A union of different callables generally does not make sense

[deep saddle]

it's more-so a difference between self, *args and *args which is pretty simple to detect

[trim tangle]

[deep saddle]

this is a wrapper which injects logging functionalities into discord.py interaction callbacks

there are probably like 20 different types of signatures for interaction callbacks

[trim tangle]

Ah, if you mean truly dynamic signatures

then this is the way to go, yes

[deep saddle]

alright, bummer

thanks

[rapid cosmos]

In all of these potentially fixed cases (if merged), Pyright respects the explicit hint for inference. But Mypy simply ignores valid, PEP-respecting, python code https://github.com/python/mypy/pull/16020

I keep seeing Mypy being this kind of overly opinionated, acting more like a linter than type checker.

[green gale]

I think it's more that HM was literally designed around being able to do this

oh that's an old comment >.>

[whole quiver]

If args and kwargs are optional (so calling foo(world, entity) won't raise a runtime error), then you can do it like ```py
class AcceptWorldAndEntity(Protocol):
def call(self, world: World, entity: Entity) -> None: ...

def foo(world: World, entity: Entity, foo: Foo = None, bar: Bar = None): ...

f: AcceptWorldAndEntity = foo
f(world, entity)

[trim tangle]

they are not optional

[bleak imp]

I wonder if you could use something like py def foo(world: World, entity: Entity, *args: tuple[Foo]|tuple[Foo, Bar]|tuple[Foo, Bar, Baz]) -> None

[trim tangle]

That's going to be very cumersome to use.

Right now I swtiched to a different scheme completely:

def foo(world: World, q1: Query[Foo, Bar], q2: Query[Baz]) -> None:
    ...

there can be as many queries as you want

Though it's still not possible to describe this statically. I decided to just check the signature while I'm parsing the annotations

[tranquil ledge]

Just out of curiosity, do you think TypeVarTuple would help here?

[trim tangle]

it would not

[tranquil ledge]

Fair enough; I might be misunderstanding what you're trying to do. Never mind.

[trim tangle]

actually it might work, but now that I switched to accepting q1: Query[...], q2: Query[...], ... it won't work because you can't make TypeVarTuple items "bound" by a type

[tranquil ledge]

Gotcha. It was a bit in the shot in the dark, as I still don't have a proper understanding of PEP 646, but tuple packing/unpacking seemed related.

[trim tangle]

Yeah it might work with (world: World, *things: Unpack[*Ts]), but idk if type checkers will understand that

[rare scarab]

Isn't it just Unpack[Ts]?

[trim tangle]

hmmm maybe

[oblique urchin]

yes

[rare scarab]

The *Ts syntax was rejected

[soft matrix]

No it wasn't

[oblique urchin]

It's been in Python since 3.11

What was rejected was **TD for kwargs (part of PEP 692)

[soft matrix]

maybe if PEP 637 wasnt rejected itd be a thing

[whole quiver]

Why does this not work? ```py
class Foo(Protocol):
bar: int | Callable[[], int]
class Bar:
bar = 1
baz: Foo = Bar()

Typing in python feels really shitty sometimes

I have to do this ```py
class Foo(Protocol):
bar: int | Callable[[], int]
class Bar:
bar: int | Callable[[], int] = 1
baz: Foo = Bar()

[undone saffron]

Because of variance. Mutable classvars and instance fields are invariant. If they werent, this would be allowed:

class Foo(Protocol):
  bar: int | Callable[[], int]

class Bar:
  bar = 1  # right here, the type checker only knows this is a ClassVar[Literal[1]], Foo above is unrelated to this class


def the_problem(foo: Foo):
    foo.bar = lambda: 1  # protocol allows this assignment

b = Bar()  # what type is b.bar
the_problem(b)
# how about now?

[whole quiver]

Yeah I get that. Still kinda annoying though, but ig explicit defining it to be int|Callable[[], int] is better than the typechecker assuming that

[rapid cosmos]

I understand that Mypy tends to be conservative with its type inferences. However, I'm unclear as to why it doesn't utilize the information from the type variables to infer the type as Signal[builtins.tuple[str, int]] (a fixed-length tuple) like Pyright does.
Instead, it infers it as Signal[builtins.tuple[_T_co1, ...]] (a variadic tuple). Why does it prefer this generalization?

from __future__ import annotations
from typing import TypeVar, Generic, reveal_type

T1_signal = TypeVar('T1_signal')


class Signal(Generic[T1_signal]):

    def __init__(self: Signal[T1_signal], __t1: type[T1_signal]) -> None:
        return
   
   
   
test = Signal(tuple[str, int])
reveal_type(test)  
# mypy: Signal[builtins.tuple[_T_co1, ...]]
# pyright: Signal[builtins.tuple[str, int]]

https://mypy-play.net/?mypy=latest&python=3.11&flags=new-type-inference%2Cstrict&gist=5ee508bb32ae745f6a4e41415d345c2b
https://pyright-playground.decorator-factory.su/?gzip=H4sIAK65J2UC_02PMQuDQAyF9_yKbCrYwVVo125dKl1EjsPm5ODMyV0s-O97iooZAkm-PN4zwY-olJllDqQU2nHyQVAze9FiPUcwKyLLZHk4zs0y0UeHEp_EFGxfYqAfaacSRgBNpaIdWDu8H2iencusAIDe6RjxvS3yXaU9ka6oATDVl0xyZ9mKUnkkZ-r958KWiZCqXi3SVQJvD3x5pnpTWitQSsnbeGlCUZLR3YvMk6M2SgpnWboCLsnyFS3gD1XOPu80AQAA

[lyric ingot]

How do I annotate nodes for a tree structure? I tried this but it doesn't work: ```py
def init(self, value: str | None = None, left: "Node" | None = None, right: "Node" | None = None):
...

[tranquil ledge]

I assume this is the init of your Node class. When you say "doesn't work", what do you mean? Did you get some kind of runtime error?

[lyric ingot]

Yeah right. The error is : TypeError: unsupported operand type(s) for |: 'str' and 'NoneType'

[tranquil ledge]

Ah. What Python version are you running this in? This looks like a case of you either not having a high enough Python version, not using from __future__ import annotations at the top of your file, or both.

Python doesn't turn your quoted types into to-be-evaluated strings for type-hinting purposes without that future import, iirc, though I may be forgetting some details.

[bleak imp]

You can do from __future__ import annotations to directly use Node in your type hints inside Node as long as you are on 3.7+

[lyric ingot]

I see. Yeah I think this only works on 3.12 and I'm using 3.10 or 11

[lyric ingot]

Without quotes? I'm on 3.10 and it gives me an error.

[bleak imp]

Then it would probably depend on your ide/plugin settings, but it works for me in 3.9

It also works in my pycharm on 3.10

[tranquil ledge]

Try showing all of your code so far?

[bleak imp]

Also what error is it?

[lyric ingot]

Yeah. It works now with __future__ anyway so that's good. I will look more into this import, it seems interesting

[lyric ingot]

Here it is: https://pastebin.com/Wuya5LK1

[lyric ingot]

NameError: name '_MerkleNode' is not defined

[bleak imp]

Luckily the magic that is future just works, I’m mostly clueless on why

[lyric ingot]

Yeah, maybe the feature was added and then removed and then added again in 3.12 haha? Btw if a function raises an exception it's return value should be annotated with typing.NoReturn?

[trim tangle]

If it always raises an exception, yes

or if it runs an infinite loop

[bleak imp]

I think it can also be Never in 3.11+ according to the docs

[trim tangle]

yeah it's exactly the same IIRC

[lyric ingot]

Thanks, but I think I heard that importing typing is very slow, is there a way to do without importing it?

[trim tangle]

Where did you hear that?

If you're using any libraries at all, there's a high chance someone already imports typing. And imports are cached

[lyric ingot]

I don't remember honestly, but i'm fairly confident I saw it somewhere, mayne maybe stackoverflow or in some python docs

[trim tangle]

For me it takes about 9ms

If that's critical for the startup of your program, then Python is definitely not the right tool

!e

import time

start = time.perf_counter_ns()
import typing
end = time.perf_counter_ns()

print((end - start) / 1000000, "ms")

[rough sluice]

@trim tangle :white_check_mark: Your 3.12 eval job has completed with return code 0.

145.163513 ms

[trim tangle]

wow what kind of potato does snekbox run on

That's probably an HDD

[lyric ingot]

Got it. I probably didn't understand what I read then. Thanks!

[oblique urchin]

Nothing in the code you showed depends on 3.12. The | syntax is supported at runtime since 3.10.

from __future__ import annotations (supported I think since 3.7) makes it so your annotations are not evaluated at runtime, which suppresses many errors

[lyric ingot]

So what was the problem ?

[oblique urchin]

Evidently you weren't running 3.10+ or didn't have from __future__ import annotations

[lyric ingot]

I didnt have the import statement, but I'm definitely running either 3.10.9 or 3.11
5, these are the only versions I have on my machine.

[oblique urchin]

oh sorry, it's because of the "Node" | None. We don't support | for strings + None

https://github.com/python/cpython/issues/107271

[lyric ingot]

Yeah. But it works when I use future, maybe it was added in 3.12.

[oblique urchin]

No, that's because with the future your annotations are not evaluated

[lyric ingot]

Oh. I feel like there are lots of things I need to learn about this subject 😅. Can I ask why are they evaluated by default?

[oblique urchin]

When annotations were introduced it was not purely for type hinting. I'm not sure it was an explicitly considered decision at the time

However, several important use cases (e.g. dataclasses) rely on evaluating annotations

[lyric ingot]

Oh yeah I forgot about them. But still I dont see why they would need to be evaluated with dataclasses. I mean they do not enforce any type checking, and I seem to be able to put any valid python expression after the 2 dots not just types! I think I don't understand what you mean by evaluated ?

[oblique urchin]

dataclasses needs to know whether the annotation is a ClassVar or InitVar

[lyric ingot]

Yeah I'm way out of my league here haha. I need to learn more. Thanks for the informations!

[tranquil turtle]

there is also KW_ONLY

[young zealot]

I think doing it all in a string would work? like ”Node | None”

[hallow flint]

in general, i think mypy is reluctant to treat type as applying to special forms instead of instances of type. and it thinks heterogeneous tuples are special forms, so i'm guessing it falls back on some random homogenous tuple here.

see this kind of thing: https://github.com/python/mypy/issues/9003
and: https://mypy-play.net/?mypy=latest&python=3.11&gist=42af11a786c233a1b9974f3a444e30c9

[tropic viper]

Hmm, odd question. Is it possible to typehint a module/class without actually importing that class at runtime?

E.g. I have a central module that works differently and has a few different functions depending on which runtime/core module you're using, but they're pretty hefty imports and this particular use case hould be as lean as possible so I don't really want to force every user to import all those modules, so they're just imported in-function instead of top-of-file. Only problem is I can't seem to properly type-hint return/input values for those functions without importing at top of file.

[tropic viper]

Found it, typing.TYPE_CHECKING is a lovely little flag.

👍

[tropic viper]

Nope, nevermind, that fails at runtime. Hmm

[oblique urchin]

You may also need from __future__ import annotations or to enclose the type in quotes

[sonic dagger]

If I'm using a library that doesn't come with type stubs, what's the best way to ignore any type errors related just to that library when running mypy on commit?

Argument 1 to "color_popover" becomes "Any" due to an unfollowed import [no-any-unimported]

[young hawk]

I'm hitting whats starting to feel like a weird mypy issue...

P = ParamSpec('P')
R = TypeVar('R')
K = TypeVar('K')

def cached(
    *,
    cache: MutableMapping[K, R] | None,
    key: Callable[P, K] | None = None,
    lock: AbstractContextManager[Any] | None = None,
) -> Callable[[Callable[P, R]], _SyncCached[P, R, K]]:
    """Memomize the results of this function."""
    key_maker = key or cast(Callable[P, K], keys.hashkey)

    def decorator(func: Callable[P, R]) -> _SyncCached[P, R, K]:
        wrapper = _SyncCached(func, cache=cache, key=key_maker, lock=lock)
        return functools.update_wrapper(wrapper, func)

    return decorator


@cached(cache={})
def _test(val: int) -> str:
    return str(val)

and mypy says:

Mypy: Argument 1 has incompatible type "Callable[[int], str]"; expected "Callable[[VarArg(<nothing>), KwArg(<nothing>)], <nothing>]" [arg-type]

[trim tangle]

Seems like it bound the P variable to some default value in the cached(...) call

I'm not sure if it's a mypy bug, but you could make an overload

[young hawk]

so thinking P is undefined because of the key default

[young hawk]

yeah it looks like by using P and R in the parameter typings but not having the callable signature in there it doesn't like it much

[trim tangle]

Actually yeah, it's not a mypy bug

or maybe it is

[young hawk]

not sure if it's a bug or just an ambigious corner case

generics in python seem a wip still

[trim tangle]

There's going to be the same problem with K if you set cache to None btw

[young hawk]

hrm, it seems to like this variation ```python
def cached(
*,
cache: MutableMapping[K, Any] | None,
key: Callable[..., K] | None = None,
lock: AbstractContextManager[Any] | None = None,
) -> Callable[[Callable[P, R]], _SyncCached[P, R, K]]:
"""Memomize the results of this function."""

def decorator(func: Callable[P, R]) -> _SyncCached[P, R, K]:
    key_maker = cast(Callable[P, K], key or keys.hashkey)
    wrapper = _SyncCached(func, cache=cache, key=key_maker, lock=lock)
    return functools.update_wrapper(wrapper, func)

return decorator

@cached(cache=None)
def _test(val: int) -> str:
return str(val)

[trim tangle]

yeah, but now you don't check what key accepts

[young hawk]

yeah, it's not perfect by any means

and I had to make cache technically capable of storing anything

but binding P or R in any attributes seems to break the return typings

I think they're getting precedence

now to figure out how to type async functions properly for use here too

[young hawk]

Hrm, I find that I'm needing to use something akin to typeshed's IdentityFunction so that the decorator doesn't mess with typing where it's used in the code base. But if I do that then the type information of the helpers on wrapper are lost, any ideas?

[rapid cosmos]

Makes sense. @ilevkivskyi replied to my issue and pointed out that the solution's starting point is already marked as TODO in checkexpr.py (https://github.com/python/mypy/issues/16253).

[young hawk]

is there a way to type a Callable with attributes?

i.e. ```python
def test() -> None:
pass

test.cache = {}

[tranquil turtle]

class CallableWithAttr(Protocol):
    cache: dict[..., ...]
    def __call__(self) -> None: ...

[lunar dune]

Yeah, this is the only way, unfortunately

[rare scarab]

Maybe consider using a callable class instead of a function

[young hawk]

sadly it is a callable class, but I can't get the typings on the decorator to behave properly if I return it directly

I have to return the decorator typed similar to this ```python
class _CachedDecorator(Protocol):
def call(self, __x: _Wrapped) -> _Wrapped:
...

[rapid cosmos]

@young hawk Are you testing it using more than one type checker?

[young hawk]

just mypy

when I went to return the cacher object directly in the typings all the usages of the decorated functions broke throughout the codebase :/

[trim tangle]

||you have unlocked the true typed python experience||

[rapid cosmos]

So true

[fossil nest]

I've created a schedule function for a custom event loop implementation but I need help annotating the function parameter. It can take a regular function, a coroutine or a coroutine function as argument:

def schedule(function: typing.Callable[..., typing.Coroutine[Event, None, None] | None] | typing.Coroutine[Event, None, None]):
    pass

I feel like I've gone wrong somewhere but my type checking software doesn't complain 😅

Alternatively:

Coroutine = typing.Coroutine[Event, None, None]

def schedule(function: typing.Callable[..., Coroutine | None] | Coroutine):
    pass

[soft matrix]

as a design point it should probably only take callables or coro funcs

but also Coroutine's first arg is useless

[fossil nest]

What do you mean?

[soft matrix]

It's the send type

It's not the return

[fossil nest]

I thought it was the yield type
Coroutine[yield, send, return] right?

all my coroutines yield a custom Event object at some point

[soft matrix]

Oh sorry yeah it's yield

Oh ok if you know what you're doing

Well yeah your codes fine

[fossil nest]

I don't really xD
heres an example of my coroutine

async def example(message: str):
    print(message)
    await Sleep(2)

where Sleep is an Event:

class Event:
    def __await__(self) -> typing.Generator[Event, None, None]:
        return (yield self)

[young hawk]

I have had to do some weird thigns to let both async and non async functions pass through to some code

I did find that Awaitable[int] is a useful annotation though

[rare scarab]

It doesn't work with asyncio.run, asyncio.create_task, or asyncio.TaskGroup.create_task

and possibly asyncio.gather

[nimble glade]

I am using Pyright. Why am I getting this error?

[frigid jolt]

you typed it as a Type

so a class

[nimble glade]

how do i typehint that playable can be .Source instance and its subclasses' instance too?

[oblique urchin]

with just pyglet.media.Source

annotating something as a class always includes subclasses

[nimble glade]

ohh

SO confused me a bit, thanks!

[young hawk]

Does mypy have support for annotated integer ranges like this? googling this is proving hard...

FourHundreds = Annotated[int, annotated_types.Interval(ge=400, lt=500)]

[oblique urchin]

no

[young hawk]

that's what I was thinking, thanks for confirming

[oblique urchin]

as far as I'm aware pyanalyze (of which I am the author) is the only static type checker that supports such types: https://pyanalyze.readthedocs.io/en/latest/annotated_types.html

[young hawk]

neat, right now I only have experience with mypy, but I've been growing frustrated with it as we get more complex typings (especially generics...)

[oblique urchin]

pyanalyze probably isn't going to be better at that than mypy. It's better at understanding highly dynamic code (because it imports modules)

[young hawk]

gotcha

some of the generic improvements in 3.12 seem promising to at least simplify using the types, but I wonder how long it's going to take for the whole ecosystem to level up

[trim tangle]

I don't know a (practical) language that can actually do this, because this is extremely hard to do statically (unless you only check literals)

[young hawk]

so would you recommend running something like pyanalyze as a secondary type checker or would it still be best suited as a replacement?

[trim tangle]

oh wait ,pyanalyze does actually check those

that's actually based

Is there anything in mypy that pyanalyze doesn't really support?

[young hawk]

that's fair, in this specific case it's mostly literals or constants being passed in, obviously there are some scenarios where the variables would be near impossible to identify the underlying value

[oblique urchin]

Honestly I'd probably stick with pyright or mypy by default. Pyanalyze might work for you but it has some limitations (and I don't have a ton of time to work on it)

[oblique urchin]

Variance checks, generally type checking definitions (as opposed to uses) of generics, PEP 646, a lot of ParamSpec stuff

[trim tangle]

and it doesn't have a language server I assume

[oblique urchin]

true

[trim tangle]

I really like the ability to define custom type-checking rules

TypeScript can actually do this, but it's a bit esoteric. And it doesn't support a "type-level throw"

[young hawk]

Argument 2 to "register_api_exception_handler" of "Service" has incompatible type "Callable[[Request, HasuraActionException], Coroutine[Any, Any, JSONResponse]]"; expected "Callable[[Request, Exception], Awaitable[Response]]"

I'm confused, HasuraActionException subclasses Exception (few steps removed) and JSONResposne subclasses Response...

hrm, it appears it's the paramspec there that's the issue, I change it to _ExceptionHandler = Callable[..., Awaitable[Response]] and it's okay with it

[trim tangle]

You're expected to give a function that can handle any exception. But you provided a function that can only handle HasuraActionExceptions

Like if you have a function like set_mammal_handler(fn: Callable[[Mammal], None]), you can provide a Callable[[Animal], None]. but Callable[[Dog], None] will not work, because you also need to handle Cat, Rat etc.

[young hawk]

oh duh, I guess I need a generic here then :/

[trim tangle]

y tho

[young hawk]

appreciate the help on that one 😂 for some reason I just wasn't in the right mindset

[nimble glade]

how to deal with this? in the TypedDict of self.config, if timeline is a list, it is list[int] (idk if I can typehint it better), but actually timeline will always have two items if it is a list, so in LocativeAudio I accept timeline as tuple[int, int]

[oblique urchin]

left, right = timeline[:2], then pass timeline=(left, right)?

(or better variable names depending on what the two numbers mean)

[nimble glade]

that will work and works, thanks!

[nimble glade]

I can't figure out how to properly typehint this function. My function basically converts args like ("foo.bar", "3") to dict(foo=dict(bar="3"))

def parse_config_address(key_address: str, value: str) -> dict[str, str]:
    """Convert a string address and its value to a dictionary."""

    base_dict: dict[str, str | dict[str, str]] = {}
    current_nested_dict = base_dict
    dict_keys = key_address.split(".")
    last_dict_key = dict_keys.pop()

    for dict_key in dict_keys:
        current_nested_dict: dict[str, str | dict[str, str]] = current_nested_dict.setdefault(dict_key, dict())

    current_nested_dict[last_dict_key] = value

    return base_dict```

[eager vessel]

I don't think you can

[nimble glade]

probably

[eager vessel]

Most accurate thing you could do is probably make it return a recursive type/dict

[nimble glade]

I think this function is pretty self-explanatory since the for loop part, but I'll try your idea, thanks

[nimble glade]

from __future__ import annotations

ConfigDict = dict[str, str]
RecursiveConfigDict = dict[str, str | ConfigDict]


def parse_config_address(key_address: str, value: str) -> RecursiveConfigDict:
    """Convert a string address and its value to a dictionary."""

    base_dict: RecursiveConfigDict = {}
    current_nested_dict = base_dict
    dict_keys = key_address.split(".")
    last_dict_key = dict_keys.pop()

    for dict_key in dict_keys:
        current_nested_dict: RecursiveConfigDict = current_nested_dict.setdefault(dict_key, dict()) # type: ignore

    current_nested_dict[last_dict_key] = value

    return base_dict

Something like this?

[eager vessel]

It's not really recursive

Can't you do

ConfigDict = dict[str, str | "ConfigDict"]

[nimble glade]

i get an error

ConfigDict = dict[str, t.Union[str, "ConfigDict"]]

pyright accepted this one

[eager vessel]

Yeah, makes sense

You wanted to support any depth (e.g. "a.b.c.d.e") is fine?

[nimble glade]

yeah any depth is fine

[eager vessel]

Then that should be correct

[nimble glade]

from __future__ import annotations
import typing as t

ConfigDict = dict[str, t.Union[str, "ConfigDict"]]


def parse_config_address(key_address: str, value: str) -> ConfigDict:
    """Convert a string address and its value to a dictionary."""

    base_dict: ConfigDict = {}
    current_nested_dict = base_dict # type: ignore
    dict_keys = key_address.split(".")
    last_dict_key = dict_keys.pop()

    for dict_key in dict_keys:
        current_nested_dict: ConfigDict = current_nested_dict.setdefault(dict_key, dict()) # type: ignore

    current_nested_dict[last_dict_key] = value

    return base_dict

did something like this

pyright cried about two statements, so had to type ignore them

[lavish plinth]

A question regarding Pydantic, are there any specific naming conventions on the models? I am using FastAPI and SQLAlchemy and for a table called Customer it seems one would add CustomerModel(BaseModel) with all shared columns, CreateCustomer(CustomerModel) with any additional parameters needed to create a customer and then GetCustomer(CustomerModel) which then has additional items that is to be returned. Is that the way to do it?

[trim tangle]

I would keep all the models separated. In the future they're very likely to diverge

[lavish plinth]

Hm ok, i have one table with 30 columns though. Keeping them all separated would have a lot of lines for those models

[trim tangle]

30 columns? what are you storing in there?

[lavish plinth]

its an old existing database, built by someone else a long time ago

online bookings

You cant create models where you set the orm mode to the swlqlchemy and just say which columns to exclude?

[trim tangle]

If you have some blob of information that you need to share, maybe extract it into its own field and use composition?

class CustomerDetails(BaseModel):
    address: str
    married: MarriedStatus
    is_rich: bool
    ...

If that's not an option, I think making a base class is okay for a start

[trim tangle]

There's stuff like SQLModel but it's truly horrifying to me. It couples your API schema to your database model, and makes your API clients depend on your implementation details

Very small changes to the database or the API might require you to scrap the whole thing and just write it out long-form. So I'd do that for a more clear separation

A small change like: suppose you want to store all personal data (emails, phone numbers, physical addresses etc.) in a separate service. It's a pretty common requirement and might be required by law in some contexts. In your database you'll store an anonymized identifier instead of the real phone number or SSN.
Now you don't have a 1-1 mapping between your API and your database

[lavish plinth]

True true

Thank you for the input, I will review and figure out how I will proceed

[lavish plinth]

I am using from sqlalchemy import Date, DateTime, Index in SqlAlchemy to map to the database, how do I use that in my pydantic models?
Getting this error: ```_unknown_type_schema
raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'sqlalchemy.sql.sqltypes.DateTime'>. Set arbitrary_types_allowed=True in the model_config to ignore this error or implement __get_pydantic_core_schema__ on your type to fully support it.

If you got this error by calling handler(<some type>) within __get_pydantic_core_schema__ then you likely need to call handler.generate_schema(<some type>) since we do not call __get_pydantic_core_schema__ on <some type> otherwise to avoid infinite recursion.

For further information visit https://errors.pydantic.dev/2.4/u/schema-for-unknown-type```

[lavish plinth]

Was able to get it working with simply adding Optional

[eager vessel]

You should use standard python types with pydantic

like date or datetime

[lavish plinth]

Had to use those mysql specific ones in order to map to the existing database though

[eager vessel]

You don't have to

Can you share your model? Probably #databases would be a better place for that

[lavish plinth]

        DateTime(), nullable=False, server_default="0000-00-00 00:00:00"
    )```

that is one for example, it is using from sqlalchemy import Date, DateTime, Index

[eager vessel]

What version are you using?

1.4 or 2.0?

[lavish plinth]

I also have blnShowInList: Mapped[TINYINT] = mapped_column( TINYINT(unsigned=True), nullable=True, server_default="1" ) which is using from sqlalchemy.dialects.mysql import INTEGER, TINYINT, VARCHAR

2.0

[eager vessel]

You can simplify that to

dtmServiceStart: Mapped[datetime] = mapped_column(server_default="0000-00-00 00:00:00")

[eager vessel]

This to

blnShowInList: Mapped[int | None] = mapped_column(TINYINT(unsigned=True), server_default="1")

[lavish plinth]

Oh cool, let me give it a try

[eager vessel]

And in your pydantic models just use python types

Also you don't have to use camelCase for field names

It can be passed into mapped_column

[lavish plinth]

ok a bunch of good info, let me give it all a try

nullable is false default on that one? so only need it if the value is true?

[lavish plinth]

Seems to work just fine, thanks!

[eager vessel]

Default value depends on what annotation you use with Mapped

If it's nullable (| None) then it's nullable 🙂

Makes sense

[lavish plinth]

Yeah I just realized thanks 🙂

[eager vessel]

@lavish plinth You can even wrap standard types using a TypeDecorator

For example if you need a pathlib.PurePath or something else

https://docs.sqlalchemy.org/en/20/core/custom_types.html

class PathType(TypeDecorator[PurePath]):
    impl = String
    cache_ok = True

    def process_bind_param(self, value: PurePath | None, _: Dialect) -> str | None:
        if value is None:
            return None  # pragma: no cover
        return value.as_posix()

    def process_result_value(self, value: str | None, _: Dialect) -> PurePath | None:
        if value is None:
            return None  # pragma: no cover
        return PurePath(value)

[lavish plinth]

All seems to work fine for the datetime ones but I also have columns of type Date so not sure how to replicate those without the use of mysql specific dialects?

Nevermind, was able to figure out!

[eager vessel]

default date should work with all dialects

[lavish plinth]

Still getting an error on the following though: blnShowInList: Mapped[int | None] = mapped_column(TINYINT(unsigned=True), server_default="1") when I run pytest: ERROR app/v1/tests/test_alley.py - sqlalchemy.exc.CompileError: (in table 'alley', column 'blnShowInList'): Compiler <sqlalchemy.dialects.sqlite.base.SQLiteTypeCompiler object at 0x000002AD08EAA990> can't render element of type TINYINT

I guess I should just use mysql for testing also and keep life easier

[eager vessel]

Maybe sqlite doesn't support tinyint? 🤔

I don't know though

[lavish plinth]

Yeah seems so :/

[grave fjord]

Is there a way to disallow untyped defs inside typed defs?

Not yet it seems https://github.com/python/mypy/issues/1324

[eager vessel]

I guess you could use a separate tool for that

Like ruff

https://docs.astral.sh/ruff/rules/#flake8-annotations-ann
Or a corresponding flake8 plugin

[grave fjord]

Does that look at nested types?

[fierce ridge]

https://mypy-lang.blogspot.com/2023/10/mypy-16-released.html

Don’t Consider None and TypeVar to Overlap in Overloads

@overload
def f(x: None) -> None: ..

@overload
def f(x: T) -> Foo[T]: ...

Before this change, this was considered unsafe because the type of T could include None. Was there some way to specify that T could be "anything other than None"?

i definitely ran into this problem before, but didn't realize why it was happening, and now that i see the explanation of why & how it's been fixed, i want to know how it could have been fixed the "right" way, without loosening the type checker rules

[undone saffron]

In some distant future where we have intersections and negation, you could do:

def f[T: object & ~None](x: T) -> Foo[T]:

right now? There's not a way to do this other than relying on overload ordering having a meaning to mypy and that change being made.

[fierce ridge]

i see, so this was legitimately necessary to support what i think is a common use case

(barring the introduction of type intersections and negation)

would there be other problems with interections and negations? i've never seen that before in a type system so i assume it's uncommon for a good reason

[grave fjord]

Allegedly Dotty has it

[fierce ridge]

that's the scala 3 compiler?

[grave fjord]

No

Scala 3 is too controversial never going to happen

[fierce ridge]

https://dotty.epfl.ch/ not this, then?

[grave fjord]

They think it's scala 3

[fierce ridge]

i have no idea what the state of scala is, i haven't looked into it since before scala 2

ah okay, it's not "official"?

[grave fjord]

Officially it's Scala 3

But practically everyone is on Scala 2

Not sure if anyone has ever had a radical 2-3 divide before

[fierce ridge]

hah, i see

[undone saffron]

Negation can't compose with all things. It can't compose with gradual types or unknown types. There are a few type systems that have it, and it's generally (As in there are no mathematically sourced issues with it) fine in type systems, not raising new issues, including remaining decidable.

[trim tangle]

iirc python's type system is already undecidable

there was some example with mypy at least

[undone saffron]

There's nothing about python's type system that requires it be undecidable. The closest feature that people want that would force undecidability is HKTs, which aren't here.

python's type system can't express all valid runtime types, but that's a different issue.

(any type system that could would be undecidable)

[grave fjord]

Mypy plugins are undecidable

[fierce ridge]

i see. so it could be introduced to mypy, but with some limitations on where it can be used?

[undone saffron]

it could be introduced to the type system, where Negation doesn't compose with gradual typing.

so ~str is meaningful but ~Any would need to not work

T -> ~T is fine so long as it requires the types passed as T to have valid non-gradual types.

Edited to link out to a comment on the typing issue for Not that explains the why of that in better detail than I'm going to repeat here.

(T: type[~ProtoFoo]) -> type[ProtoFoo & T] is an example of why you might actually want this, this being the signature of a decorator that augments a type with what's provided by ProtoFoo, but wants to type check that this is compatible.

[reef yacht]

~Any could he the return type. Basically NoReturn.

[undone saffron]

~Any isn't Never

The linked discussion and the papers linked therin go into why that can't be the case

~object on the other hand would be Never

[rare scarab]

I'd be more interested in Sequence & ~(str | bytes)

[oblique urchin]

That's an interesting example, thanks. I think so far every example use case I have seen for ~ has been in combination with &, which made me think that perhaps what we need is only a difference type A - B, instead of A & ~B. TypeVar bounds are probably one of the main places where ~ without & makes sense.

[undone saffron]

I have a couple other examples that would benefit from ~ without &, but I think the largest benefits of type negation strongly relate to intersections. Even my example there still involves an intersection, it's just interesting due to where it shows up different from other cases

Also, ~T is just object - T, so I would think negation to be more usefully expressive since otherwise people who want it will just do that to get it anyhow. Since chaining together positive and negative groups with "This and that and not these" leads to natural logical forms (DNF) it would be easier to teach (only marginally for simple cases though)

~None as a type hint or a typevar bound is useful as one in an overload set, though the reasons for that are somewhat obscured to end-users by the recent mypy change special casing that None doesn't overlap with typevars when both are options for overload resolution.

The intersections discussion had the point where overloads could become unordered by specification with negation because negation allows properly expressing the precedence by such intentional exclusion. It could then be made an error to have overlapping overloads (probably only in strict) so that there is no complex heuristic (and no accidental resolutions to the wrong overload). The negations required can be computed automatically from an overload set and order, so an IDE could have a right click action for this.

There's also the stuff like Sequence[str] & ~ str, though this one in particular could be solved by changing the semantics of Sequence[T] such that if T is a Sequence, it has to be a non-T Sequence containing T. Then if you want the old behavior Sequence[str] | str is valid. There are some questions of this and recursive types (similar to properly typing json.load, though that uses a concrete list) though that make me hesitate on is such a semantic change would actually be good.

[grave fjord]

mypy doesn't allow more than one Unpack in a def, eg:

from typing import Callable, Awaitable, TypeVar, TypedDict
from typing_extensions import TypeVarTuple, Unpack
from datetime import timedelta

T = TypeVar("T")
Ts = TypeVarTuple("Ts")

class IOLoop: pass

class EmptyDict(TypedDict):
    pass

def sync(
    loop: IOLoop,
    func: Callable[[Unpack[Ts]], Awaitable[T]],
    *args: Unpack[Ts],
    callback_timeout: str | float | timedelta | None = None,
    **kwargs: Unpack[EmptyDict],
) -> T:
    ...

I get an error on the **kwargs: Unpack[EmptyDict] line with error: More than one Unpack in a type is not allowed [misc]

[oblique urchin]

report a bug

[grave fjord]

https://github.com/python/mypy/issues/16285

[rustic island]

I'd like to have a type hint for a value representing a type - int, str, MyClass, Any.
That last one is a problem though.

from typing import Any

def boo(t: type) -> None:
    print(t)

boo(Any)

Running mypy on this gives the error:

error: Argument 1 to "boo" has incompatible type "object"; expected "type"  [arg-type]

What can I put instead of type in t: type in order to allow it to accept Any as an option?

[grave fjord]

t: object

[rustic island]

That would make it work for boo(3) too though.
I want to allow types. Every type, and only types.

[tranquil turtle]

Any or list[int] are not types at runtime, they are special forms

so using them in runtime context as types doesnt make a lot of sense

[grave fjord]

Any is a type at runtime

since 3.11 you can use class Whatever(typing.Any): ... there's a backport in typing_extensions

looks like this is actually a typeshed bug

https://github.com/python/typeshed/blob/d14c5f82c881af92bd52519193f2b64aa4403999/stdlib/typing.pyi#L138

[rough sluice]

stdlib/typing.pyi line 138

Any = object()```

[grave fjord]

should be class Any on 3.11

[soft matrix]

mypy will be special casing it anyway

[rustic island]

Bummer.
This is for something that's used by something that deals with annotations, but that something also goes through mypy checks and that specific part should enforce a type at that location.
Any ideas for a workaround at least?

[grave fjord]

show the actual code you have?

[tranquil turtle]

i didnt know that :)

[rustic island]

It's more complex (uses a Protocol actually and not a direct function), but the core issue is that it expects type | None and Any doesn't fit in there.

[grave fjord]

@oblique urchin ^

@rustic island you'll have to take object on 3.10 anyway

[rustic island]

And 3.11?

[grave fjord]

I've just opened a typeshed issue https://github.com/python/typeshed/issues/10906

you should post your example code there if you want it fixed

[rustic island]

OK, let me see if I can isolate it well enough.

[oblique urchin]

Just closed that. It's not a bug and Any should not be compatible with type[].

[grave fjord]

this isn't Any in a type context, it's Any as a runtime value

[oblique urchin]

What @rustic island wants is not possible with the current type system. You need something like TypeForm: https://github.com/python/mypy/issues/9773

[grave fjord]

but that's not what Any is here, it's a genuine class you can subclass

[tranquil turtle]

mypy doesnt like subclassing Any: https://mypy-play.net/?mypy=latest&python=3.11&flags=strict&gist=bd4eacc674289eb6ec8c682e94d48154

[rustic island]

class _ResponseDecoratorSignature(Protocol):
    def __call__(
        self,
        model: type | None,
        description: Optional[str] = None,
        default: bool = False,
    ) -> SimpleDecorator:
        ...

def _make_response_decorator(
    status_code: int | str,
) -> _ResponseDecoratorSignature:
    def actual_decorator(*args: Any, **kwargs: Any) -> SimpleDecorator:
        return _response(status_code, *args, **kwargs)

    return cast(_ResponseDecoratorSignature, actual_decorator)

http_200_ok = _make_response_decorator(200)

@http_200_ok(Any, "All is well in the world.")
def world_check() -> Any:
    pass

I wasn't sure what to put in place of irrelevant parts (SimpleDecorator, _response), so please just ignore those.

[grave fjord]

it works on pyright https://pyright-playground.decorator-factory.su/?gzip=H4sIAJYEMGUC_0srys9VKKksyMxLV8jMLcgvKlFwzKvk4krOSSwuVnDLz9cAcjWtuBSAoAAoxMXFlaZgC5bQ5ErTy0jMBQCPFga2QgAAAA%3D%3D

[grave fjord]

I can't find the ticket to add mypy support for subclassing Any

[tranquil turtle]

i guess thats because it is not a planned feature

hmm

[oblique urchin]

huh that seems like a bug. I think we allow it in stubs

[tranquil turtle]

D:\>mypy _.py
Success: no issues found in 1 source file

D:\>mypy --strict _.py
_.py:3: error: Class cannot subclass "Any" (has type "Any")  [misc]
Found 1 error in 1 file (checked 1 source file)

i guess subclassing Any is ok, but strict mode disallows it (to prevent subclassing classes from unresolved modules i guess)

D:\>mypy --disallow-subclassing-any _.py
_.py:3: error: Class cannot subclass "Any" (has type "Any")  [misc]
Found 1 error in 1 file (checked 1 source file)

maybe mypy should special-case that

[oblique urchin]

oh yes, didn't notice you had --strict on. We should probably have --disallow-subclassing-any not use the "misc" code

[rare scarab]

Should probably depend on the target python version.

well... unless it's just all misc codes get applied to --strict

[rustic island]

Well, logically speaking, I'd expect Any to be possible to use where types are expected, so I do hope it's added.
That said, any idea on how to make this work right now?
Should I just disable that check? Switch it to object? I don't really like that I can put non-types there, but I guess I can check them at runtime instead.

[oblique urchin]

I would use Any. Also, what do you mean by "types"? Anything valid in a type annotation?

[oblique urchin]

No, it's appropriate to give an error for it on all versions. It's allowed on newer versions at runtime, but it's still unsafe and should be flagged in strict mode.

[tranquil turtle]

Well, logically speaking, I'd expect Any to be possible to use where types are expected, so I do hope it's added.

from typing import Any

def f(t: type) -> None:
   print(t())
f(int)
f(list[int])
f(Any) # ok statically, but fails at runtime: TypeError: Any cannot be instantiated

[rustic island]

Yeah, pretty much.

[oblique urchin]

Then I would use Any. type refers to concrete types, not anything valid in an annotation

[rare scarab]

?```py
if TYPE_CHECKING:
BaseClass = Any
else:
BaseClass = object

class Foo(BaseClass): pass

[rustic island]

That also wouldn't work if you put a class there that has required arugments in its constructor.

[oblique urchin]

Any is appropriate to use in cases where the type system isn't powerful enough to express what you want

[rustic island]

Alright, thank you. =)

[solemn sapphire]

Is there a way I can make mypy understand that the reversed[Any] can be the Callable it expects?

names = {
    1: "one",
    2: "two",
    3: "three",
    4: "four",
}

a = map(reversed, names.items())

/private/tmp  ζ mypy test.py
test.py:1: error: Argument 1 to "map" has incompatible type "type[reversed[Any]]"; expected "Callable[[tuple[int, str]], reversed[_T]]"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

or would that be a limitation of the stub files?

[oblique urchin]

Probably a limitation of mypy. Does it work with --new-type-inference?

[solemn sapphire]

it passes!

thank you for the magic option :)

[oblique urchin]

It's experimental for now but is planned to be the default under 1.7. If you see any bugs with it, please report them

[solemn sapphire]

sure will definitely do!

[grave fjord]

It's available in typing_extensions.Any

[rare scarab]

Never thought I'd see Any in the typing_extensions module

why not make typing_extensions a drop in replacement for typing?

[grave fjord]

https://github.com/python/typing_extensions/blob/main/CHANGELOG.md#release-440-october-6-2022

[grave fjord]

I think it is now

[oblique urchin]

we did that in one of the recent releases

[rare scarab]

cool.

[grave fjord]

https://github.com/python/typing_extensions/blob/main/CHANGELOG.md#release-470rc1-june-21-2023

[grave fjord]

I just import from wherever and let linting fix it

[rare scarab]

hooray for pyupgrade

It's very helpful when vscode wants to from typing import Iterable and pyupgrade fixes it to from collections.abc import Iterable

[grave fjord]

It's no help when I try importing Irritable

[solemn sapphire]

you gotta do from friend import Irritable

[solemn sapphire]

I wonder if that's typo?

[soft matrix]

no that error is fine

*a = ... is fine

[tranquil turtle]

it is not

!e *a = b

[rough sluice]

@tranquil turtle :x: Your 3.12 eval job has completed with return code 1.

001 |   File "/home/main.py", line 1
002 |     *a = b
003 |     ^^
004 | SyntaxError: starred assignment target must be in a list or tuple

[tranquil turtle]

*a, = b is fine

a = *b is not good
and mypy incorrectly says that error is in assignment target, while it is in the right side of assignment (how is it called?)

[solemn sapphire]

expression?

(excuse the horrible code)

[oblique urchin]

it's not saying it's in an assignment target, it's saying you can use a starred expression only in an assignment target

[solemn sapphire]

Ye I missed that

[soft matrix]

oh sorry thats what i meant

[oblique urchin]

but yeah that message isn't great, please file an issue to improve it

[tranquil turtle]

a,b=*a, is this fine?

[oblique urchin]

mypy should probably emit a similar message to Python

[tranquil turtle]

oh, right
i misread that
i think this error message is a bit confusing

[solemn sapphire]

What should I classify this as?

feature or bug?

[oblique urchin]

bug I guess, it's a usability bug

[solemn sapphire]

alrighty!

[rare scarab]

why not wrap it with tuple()?

[spark otter]

Hey everyone, I'm using pyright and pylint (no pylance in my environement sadly) and I want there to be strict with enum type checking and I can't find how
I'm posting a screenshot instead of text code because it'll show there's not red squiggly line where I want it

But as expected running the above code yields this result, I just want pylint and or pyright to catch that for me if at all possible ? Searching google for "pylint strict enum type checking" has not given me any result

[lavish plinth]

In my pydantic (model for patch endpoint) I had Name: Optional[str] which I thought would make it optional and not required but that did not work, Name: Optional[str] = None worked though

Why is that?

[void panther]

Optional is just a union with None, but I'm not sure if pydsntic does (or is supposed to do) something special

[trim tangle]

You have to set this option to "basic" or "strict"

oh wait, it's not pylance

But there might be a similar setting in pyright

[spark otter]

I found it ! Sorry it's my bad I changed the pyrightconfig.json and I didn't see tha.. strict is, very strict though like :

The whole type is unknown despite being defined right above ? Mode basic is working perfectly though thanks 🙂

[proud brook]

Type hinting a name as optional does not automatically assign it None, so if it weren't defined beforehand, it is undefined

[solemn sapphire]

Would it be possible for a's type to be resolved to tuple[int, int]?

a = tuple(iter((0, 0)))
from typing import reveal_type
reveal_type(a)

/private/tmp  ζ mypy --new-type-inference test.py
test.py:25: note: Revealed type is "builtins.tuple[builtins.int, ...]"
Success: no issues found in 1 source file

[solemn sapphire]

No one told me python also had two sentinels like js /s

[trim tangle]

I don't think so

(In this case you can just do a = (0, 0), so I'm assuming you have something more complex)

[lavish plinth]

So I updated all my sql alchemy models with the name parameter:```class Alley(Base):
tablename = "alley"

id: Mapped[int] = mapped_column(INTEGER(unsigned=True), primary_key=True, index=True, name="lngAlleyID")``` to have a better looking variable name rather than what the database column is actually called

It seems then I am also forced to do the same updates on my pydantic models ```class AlleyGet(BaseModel):
model_config = ConfigDict(from_attributes=True, populate_by_name=True)

id: int = Field(alias="lngAlleyID")``` or it crashes

Do I need to do that on those also (seems like quite the double work) or can pydantic somehow pull it of the sql alchemy model?

[solemn sapphire]

Indeed, it's the case where I have some collection, which I transform in some way and turn it into the same collection again. But the types get changed in the process

[undone saffron]

inference here is totally possible, here's what pyright infers:

from typing_extensions import reveal_type
from typing import Sequence

a = tuple(iter((0, 0)))
from typing import reveal_type
reveal_type(a)  # Type of "a" is "tuple[Literal[0], ...]"

def get_seq() -> Sequence[int]:
    return (0, 0)

b = tuple(get_seq())
reveal_type(b)  # Type of "b" is "tuple[int, ...]"

If you have your own collections, if they are generic over a typevar tuple and implement __iter__ using that typevartuple, it will work too

seems like mypy just not understanding as much

[proud brook]

What are the two sentinels about

[brisk hedge]

a joke about how you said that if something is not defined it is "undefined" as if undefined is a value in python

[proud brook]

None of that madness in Python

[young zealot]

I have a generic class Test(Generic[X]):, is there a way to make it so X must conform to a protocol, but it still knows what the concrete type of X is so I can access its non-protocol defined properties?

[hallow flint]

set the protocol as the bound of the X TypeVar

[trim tangle]

X = TypeVar("X", bound=MyProtocol)

or class Test[X: MyProtocol]: in 3.12 syntax

[young zealot]

ah great, I wasn't sure if I could use bound with a protocol, but makes sense, thanks

[soft matrix]

https://github.com/python/cpython/pull/110212 would someone mind reviewing/merging this?

[lavish plinth]

Anyone know anything about this one? Would prefer not having to do the name/alias in both db models and pydantic models

[grave fjord]

Reviewed

Are there generally tests for these things? (I hope so)

[oblique urchin]

I think there's one test file with a long list of classes that should be subscriptable

[grave fjord]

Sweet

I do love how rejecting a PR puts it in the core review queue

[grave fjord]

Are there tests for too many and too few type arguments? Cc @soft matrix

[oblique urchin]

We don't check that in builtins

list[int, str, float, bool]

[soft matrix]

one day :P

[rare scarab]

That's how pyright implemented inline typeddict. dict[{"foo": str}]

[grave fjord]

CoroutineType and GeneratorType need to count to 3 no more no less. Three shall be the number thou shoult count and the number of the counting shall be three

[soft matrix]

well hopefully after pep 696 it should only be 1 for gen

[oblique urchin]

and nobody knows what the first two arguments mean

[rare scarab]

1, 2, 9, um.. 7

[grave fjord]

You shall snuff it

[soft matrix]

hmm that does actually pose a problem for the thing i was working on

[grave fjord]

Honestly I sometimes I want Coro[RValue] and sometimes Gen[SValue]

And sometimes Gen[RValue]

[rare scarab]

It's only an experimental feature. It came about from a typing-sig suggestion.

[soft matrix]

yeah ik, might kindly ask when it gets pepified to use TypedDict[{"foo": str}] which i think is also supported

[grave fjord]

I'd like to see

class TD(TypedDict, dict[str | K, V]):
    ham: Spam

[soft matrix]

typealiases™️

[rare scarab]

There's a bug where it might try to resolve foo as a type

[soft matrix]

or PEP 637

[grave fjord]

__extra__ is redundant IMHO

And doesn't let you add non str keys

[grave fjord]

class TD(TypedDict[str | K, V]):
    ham: Spam

Would work too

Is there a discourse for __extra__?

https://peps.python.org/pep-0705/#use-a-reserved-extra-key

https://github.com/python/peps/pull/3441 seems there's no discourse yet?

[soft matrix]

should use Mapping probably not dict

[grave fjord]

Maybe for TypedMapping

[soft matrix]

unfortunately the idea got dropped

but TypedDict isnt a subclass of dict

cause pop isnt always allowed

[stoic warren]

Playing around with the 3.12 type params, and ive got this: ```py
class ContainerK, V:

@abstractmethod
def map[V2](self, f: Callable[[V], V2]) -> Container[K, V2]:
    ...

But this signature isn't right. `map` returns the *same* `Container` as `self` is, not a generic version

I cant use the Self type because you cant give it parameters

How are you supposed to annotate this?

[tranquil turtle]

you cant

[stoic warren]

ah the lack of HKTs foils me again 😭

[undone saffron]

This idea of mapping one type to another is typeable, but not as an instance method (which is tied to a type), have you considered a functional approach?

def make_transformer[K, V, R](f: Callable[[V], R]) -> Callable[[tuple[K, V]], tuple[K, R]]:

    def kv_transform(kv: tuple[K, V]) -> tuple[K, R]:
        k, v = kv
        return (k, f(v))

    return kv_transform

items = {1: "0xff", 2: "0xfe"}
new_items = dict(map(make_transformer(lambda v: int(v, base=16)), items.items()))

You can swap out the dict on the last line for your own container or make other tweaks, inference works fine.

(make_transformer here is only here to illustrate this can work generically over any function that transforms values, your specific case could just be the inner function)

[fierce ridge]

can you return a protocol-ized version of Container?

is there a theoretical reason why we can't have something like SelfType[K, V2]?

[undone saffron]

It can be used as a backdoor to HKTs, so you'd significantly increase computational complexity of static analysis and force there to be answers about HKTs with python's type system that haven't been decided.

[lavish plinth]

@router.get("/", status_code=status.HTTP_200_OK)
def get_alleys(db: Session = Depends(get_db)) -> list[AlleyGet]:
    alleys = db_get_alleys(db)
    if alleys is None:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND, detail="No alleys found"
        )
    return alleys

Why do I get ```py
Variable not allowed in type expressionPylancereportGeneralTypeIssues
(class) Session

[eager vessel]

Where do you import Session from?

Should be from sqlalchemy.orm import Session

[lavish plinth]

from ..db.session import Session, get_db
``` which has ```py
Session = sessionmaker(autocommit=False, autoflush=False, bind=engine)
``` in it

[eager vessel]

Session is indeed a variable here, not a type

name it something more appropriate

session_factory

[lavish plinth]

Ah nice catch, thank you!

Do you remember the other day you mentioned that I could use the name parameter in my sql alchemy models to avoid using "lngAlleyID" as the model name (that is the column name in the existing db)

[eager vessel]

Yes

[lavish plinth]

That was super nice, I did however notice that I then also needed to do the same with all my pydantic models

[eager vessel]

class Model(Base):
    __tablename__ = "model"

  id: Mapped[int] = mapped_column("uglyColumnNameInDatabase")

[eager vessel]

Hm, not really?

[eager vessel]

pydantic models would work with id field in this case

[lavish plinth]

Hm ok, I will need to double check then as I did get errors when I tested the getAllAlleys endpoint

Yeah seems to be working so must have been something else then, thanks!

Sorry for all the questions but have one more 🙂

def db_get_alleys(db: Session):
    sql = select(Alley)
    return db.scalars(sql).all()
``` Should one add a return type on database/all functions? ```py
def db_get_alleys(db: Session) -> list[SqlAlchemyAlleyModel]

[rare scarab]

it's suggested yes

[lavish plinth]

Ok cool thank you!

[grave fjord]

Is there a changelog for mypy 1.6.1?

I can just use the diff https://github.com/python/mypy/compare/v1.6.0...v1.6.1

But a CL would be nice

[frank field]

so ive never really bothered to use type hinting before, but how does it work? does it raise an error if you use the wrong type?

the 'hinting' part suggests that it doesn't, but just curious

[oblique urchin]

Type hinting is generally ignored at runtime. You run a separate tool, a type checker, on your code to check that the type hints are correct

The most commonly used type checkers are mypy and pyright

[frank field]

kk

so it's just like commenting in a sense then?

like to say 'hey, if you're using this, this is the type it takes'

[oblique urchin]

In some sense, but with the option to have a tool validate that your comments are right

[trim tangle]

If you're on desktop you can try some stuff on https://pyright-playground.decorator-factory.su/

[frank field]

kk

good to know