type-hinting

[jolly cipher]

this really boils down to i think any fully static type that is only assignable a single runtime value

so that would be all types consisting of None or Literal or fixed-size tuples with None or Literal members?

[oblique urchin]

it could also be Never

or a newtype

[jolly cipher]

oh, right, Never!

Literal[True] | Never is not assignable to Never

although that never happens... does it

but about NewType...

NewType('OnlyTrue', Literal[True]) seems like an interesting thing

[oblique urchin]

sorry, NewType is not actually possible in the current spec, NewTypes over literals are illegal

[jolly cipher]

so yeah, Never would beat this if Never branches were checked in the first place :-)

key takeaway is to not use a bucket to carry a single drop of water

[restive rapids]

context: https://github.com/microsoft/pyright/issues/10738
so, mypy does indeed do that narrowing of a: A, but i cant actually call unsound as the typevars resolve to Never, so i cant make an absurd function
does anyone have an idea of whether its possible to rewrite that a bit for it to "work" in mypy too, or this issue doesnt happen in it at all?

[thorn osprey]

i have a lot of types like int | float is there a way to make it have an alias for simplicty but when you hoverover it for type linters it would still show the int | float?

[brazen jolt]

yeah, you can just do type Number = int | float and then use the Number annotation

if you need support for older python versions, you can instead do Number: TypeAlias = int | float (with from typing import TypeAlias)

[thorn osprey]

but now my type hinter is showing the typeAlias and not the types int|float

[oblique urchin]

That may not do what steel wants in terms of hovering. I think there's some divergence between type checkers over whether aliases get expanded in hover output or not

And different opinions over whether they should

[brazen jolt]

ah I see, I thought you just wanted a reusable alias, not one that gets expanded, yeah, there isn't a standard way to do that unfortunately

[thorn osprey]

ooh shucks

[rare scarab]

To support 3.9, use Number: TypeAlias = "int | float" or Number: TypeAlias = Union[int, float]

[trim tangle]

For this particular case, you can use float. Type checkers consider int to be a subtype of float

(they might display the types int | float and float differently though)

[brazen jolt]

Not all of them do, and while it's usually fine, it's not entirely justified to say that int is a float subtype

[trim tangle]

https://typing.python.org/en/latest/spec/special-types.html#special-cases-for-float-and-complex

Python’s numeric types complex, float and int are not subtypes of each other, but to support common use cases, the type system contains a straightforward shortcut: when an argument is annotated as having type float, an argument of type int is acceptable; similar, for an argument annotated as having type complex, arguments of type float or int are acceptable.

Which type checkers don't support this?

[brazen jolt]

iirc there was a setting to disable this behavior in basedpyright

[trim tangle]

I don't think so

[brazen jolt]

maybe it was in something else, but I'm pretty sure I saw the setting somewhere

[trim tangle]

That's not something you can change with a setting unfortunately. Consider this: you're using a library that has a function with this type: ```py
def get_number() -> float:
...

If you turn off the `int->float` promotion in your type checker, should this function be treated as returning a `float`? Or `int | float`?

[brazen jolt]

hm, good point, but I still recall seeing it somewhere

oh I guess basedpyright just wanted to implement this, but didn't end up doing so yet

DetachHead/basedpyright#319

[mighty linden]

GitHub

[basedpyright] #319 option to disable fake subtypes (promotions)

[oblique urchin]

Yeah it doesn't make much sense to do this in a type checker since typeshed is written assuming that the promotion is used

so if you just disable the behavior in a type checker, you'll get a ton of false positives in user code

[indigo halo]

I am wondering now if I am insane…

1. Argument of type "Entry" cannot be assigned to parameter "entry1" of type "BaseEntry" in function "set_entries"
     "Entry" is not assignable to "BaseEntry" [reportArgumentType]

The function set_entries takes BaseEntry instances. Class Entry derives from BaseEntry. Thus, an Entry "is-a" BaseEntry.

Where is my logic error?

It works, of course. Python is okay with it. Maybe Pyright and the folks at Microsoft have not heard of Polymorphism?

[soft matrix]

can you show more of the code here

its probably a variance issue

[indigo halo]

Pyright is pretty whacky at times I find anyway. reveal_type(a<b) ● Pyright: Type of "a < b" is "Any"

[indigo halo]

certainly variance. I just don't understand that term yet.

I will post some code. Gimme a sec.

Thanks for engaging! ♥️

Oh man, now of course I cannot reproduce the problem.

[soft matrix]

love that

[indigo halo]

Meanwhile, why would the type of a<b be Any?

[soft matrix]

one of a or b is Any

without more code i cant tell

[indigo halo]

Thanks for your readiness to help. I cannot figure this out, I've been staring at it too long. I will give it a break and come back and will hopefully encounter someone else as helpful as you! ♥️

[jade viper]

How would one ago type hinting a Django JSON Field?

This is +- my case:

import uuid
from typing import TypedDict

from django.db import models


class Team(models.Model):
    ...


class Player(models.Model):
    ...


class DraftPick(models.Model):
    ...


class TradeOfferItem(TypedDict):
    players: list[Player]
    picks: list[DraftPick]
    sender: Team
    receiver: Team


class TradeOffer(TypedDict):
    uuid: uuid.UUID
    offers: list[TradeOfferItem]
    accepted: bool


class Trade(models.Model):
    offers: list[TradeOffer] = models.JSONField(default=list)  # This is what I'm not sure what to do, it's not actually a dict, I just want to make clear the shape of the JSON
    ...

[jade viper]

On another note, is this not the correct way to indicate a subclass of some class? i.e. subclass of A -> type[A]

[feral wharf]

No, just do arg: A and it'll allow subclasses too

Doing type[A] means you're expecting a class, not an object

[stiff hollow]

Is it possible to create a generic type parameter that is bound so that it satisfies the following two constraints:

• It is a subclass of django.db.models.Model
• It has an attribute state (preferably that attribute is also a django field, but I can live without that constraint)
  Of course both things apart are easy to do: for the first one I can just do class MyClass[T: Model] and fir the second one I can create a Protocol. But is there a way I can constrain my type to two those constraints simultaneousy? Something like rust's +?

[brazen jolt]

There isn't, python doesn't have support for type intersections (at least not yet, but there is some talk on it), one thing you could do is write a protocol class that mimics django model and has the state attribute, or make a concrete class that inherits from model and defines state, letting the other classes that you need to support inherit from it, if that's something you can do

[stiff hollow]

Ah okay, that is what I thought already

[brazen jolt]

you could still make something like a StatefulModel class and manually cast if you at least want to have the type expressible, but it still won't allow you to pass it in directly:

from typing import Protocol, TypeGuard, reveal_type, cast
from abc import ABC
from django.db.models import Model

class HasState(Protocol):
   state: str  # or whatever type you want


class MyStatefulModel(Model):
   state: str

# abstract class just for your internal use
# (it would of course be better to have the classes you need
# actually inherit from this, but if you don't have control 
# over that, you can "cheat" with a cast)
class StatefulModel(Model, HasState, ABC): ...  


m = MyStatefulModel()
m = cast(StatefulModel, m)
reveal_type(m)  # StatefulModel (even though it's a lie)

Or maybe make it a little safer and make the protocol runtime_checkable and write a type-guard ```python
def is_stateful_model(x: Model) -> TypeGuard[StatefulModel]:
return isinstance(x, HasState)

m = MyStatefulModel()
if is_stateful_model(m):
reveal_type(m) # StatefulModel

But again, this is more about just allow you internally to work with that type, if you're writing a lib and you want to expose the function to your users, who should pass in the correct type, this won't really help you much, they'd need to use the cast/typeguard check on their type first.

[stiff hollow]

Hmm okay, thank you, I will investigate this

I do have another question. Suppose I have this:

class MyGenericClass[T]:
    def __init__(self, var: T):
        self.var = var

I have a subclass of this that will bind the generic T to a type:

class MySubclass(MyGenericClass[int]):
    pass

Now, somewhere else, I need to have a type that contains instances of subclasses of MyGenericClass with the type T bound to some other generic type X. How do I do that? This does not work:

my_instance = MySubclass(10)
assert_type(my_instance, MyGenericClass[int])
assert_type(my_instance, type[MyGenericClass[int]])

will result in

test_file.py:14: error: Expression is of type "MySubclass", not "MyGenericClass[int]"  [assert-type]
test_file.py:15: error: Expression is of type "MySubclass", not "type[MyGenericClass[int]]"  [assert-type]

so I need some type that will say that a given object is an instance of a subclass of my generic class, with the generic type bound to a specific type.

[rare scarab]

MySubclass is of type MySubclass, not MyGenericClass[int]

[brazen jolt]

yeah, assert_type looks for an exact match

I'm not exactly sure what you're trying to achieve here

[stiff hollow]

me neither tbh

[brazen jolt]

if you just want a function that accepts any instances of MyGenericClass with the generic type bound to int, you can just annotate that: ```python
def foo(x: MyGenericClass[int]): ...

[stiff hollow]

Ah, that just works 🤔 I thought it wouldn't

[vivid nimbus]

what would be the correct typing for a callable that takes varargs:

def foo(*names: str) -> bool:
   ...

f: Callable[[<here>], bool] = foo
#  what goes---^

list[str] gets squiggles for me, as does tuple[str], iterable[str], etc

[spiral fjord]

You'll need to use a Protocol for that

[vivid nimbus]

was afraid so :/

just something like:

class Foo(Protocol):
   __call__(self, *names: str) -> bool:
       ...

?

[spiral fjord]

Yes

[brazen jolt]

yeah, though you can generalize it if you need it more than once: ```python
class ArgsCallableT, R:
def call(self, *args: T) -> R: ...

and then use f: ArgsCallable[str, bool]

[restive rapids]

Callable[[*tuple[str, ...]], bool]

[brazen jolt]

oh, that looks a bit cursed, had no idea that was a thing, lol

[vivid nimbus]

at least in .13, it doesn't like the * in that

oh wiat!

no it does take taht...why does it need the * and the tuple to convey varargs?

[restive rapids]

just tuple[str] would mean a tuple with a single string,
tuple[str, ...] means a variadic tuple with n strings (as ... in tuples repeats the last type)
unpacking a variadic tuple = varargs

[vivid nimbus]

I get the semantics, I guess more implementation wise, why does *Iterable[str] not work as well

[restive rapids]

because you can only unpack tuple types

[vivid nimbus]

then why won't *list[str] work

[restive rapids]

list is not tuple?

[vivid nimbus]

you can unpack a list

[restive rapids]

in expressions, yes, but not the typesystem

[vivid nimbus]

well....I don't like it 😆

but it works, thank you!

[frigid jolt]

Hi, I need to typehint a function that can take a set of kwargs, for that reason I have defined some TypedDict(s)

now if I typehint it like this t.Callable[[t.Unpack[AnyOfMyTDict]], t.Any] it doesn't work because in that context I don't think you can use Unpack like that, are there any other options?

[feral wharf]

Use a protocol!

https://docs.python.org/3/library/typing.html#typing.Protocol

[frigid jolt]

so instead of returning a callable I'll return the protocol right?

[feral wharf]

Yup

[frigid jolt]

idk why I didn't thought about that

ty

[frigid jolt]

is it not possible to use t.Unpack on a type var bounded to a typed dict? because it's not letting me do it

e.g

class BaseAttrs(t.TypedDict):
  ...


class RegisterDecorator[T: BaseAttrs](t.Protocol):
    def __call__(self, **kwargs: t.Unpack[T]) -> t.Any:
        ...

[feral wharf]

Uhh I haven't tried that yet, sorry

[frigid jolt]

KwargT = TypeVar("KwargT", bound=BaseAttrs, default=BaseAttrs, covariant=True)


class RegisterDecorator(t.Generic[KwargT], t.Protocol):
    def __call__(self, **kwargs: t.Unpack[KwargT]) -> t.Any:
        ...

even setting a default doesn't work 😭

[feral wharf]

Yeah nothing seems to work

That's interesting

[frigid jolt]

😔

I'm limited by the technology of my time

I think it's a valid use case

[feral wharf]

Same

@ jelle @ erictraut

[gilded pine]

hello people

[frigid jolt]

I'm thinking whether to write a PEP for it

[feral wharf]

Do it

[frigid jolt]

I'm scared

[frosty flume]

😅 why @frigid jolt ?

[frigid jolt]

I just started to look at the PEPs that explains how to write a PEP and there's a lot to do before even getting to write it

[frosty flume]

The PEPs that explain how to write PEP... that's an inception 😅

[feral wharf]

I don't even know how a PEP works lol

[rare scarab]

!pep 1

[rough sluice]

**PEP 1 - PEP Purpose and Guidelines**

Status

Active

Created

13-Jun-2000

Type

Process

[rare scarab]

!pep 2

[rough sluice]

**PEP 2 - Procedure for Adding New Modules**

Status

Active

Created

07-Jul-2001

Type

Process

[frigid jolt]

!pep 13

[rough sluice]

**PEP 13 - Python Language Governance**

Status

Active

Created

16-Dec-2018

Type

Process

[frigid jolt]

all of these PEPs guide you to write a PEP

pep 0 as well but it's more an index to see the useful peps

[frigid jolt]

Hi, I have two files

in one file, server.py, I have defined a Server class, in the other, context.py I have defined a Context class;
I have made the Context class generic over Server because it has an attribute (server) which holds a reference to Server (so I have imported Server, TypeVar inside Generic must be resolved at runtime)
now in server.py I also need Context (not for typing) and so I have imported it

How can I end this limbo?

In code:

# server.py
from .context import Context

class Server: ...

# context.py
import typing_extensions
import typing as t

from .server import Server

ServerT = typing_extensions.TypeVar("ServerT", bound=Server, default=Server)

class Context(t.Generic[ServerT]):
  server: ServerT

[trim tangle]

The easiest way is to move them to the same module, since they're coupled together anyway.

If you don't want to do that, decouple them so that e.g. Context depends on an interface that Server matches instead of hard-coding the Server class.

If you don't want to do any of the above, you can use import modules instead of individual items, and use from __future__ import or stringified annotations:

# <3.13
from . import server
ServerT = typing_extensions.TypeVar("ServerT", bound="server.Server", default="server.Server")

class Context(t.Generic[ServerT]):
    server: ServerT

# 3.13+
from __future__ import annotations
from . import server

class Context[S: server.Server = server.Server]:
    server: S

If you don't want any of that, the last resort is to use an if typing.TYPE_CHECKING: block.

Circular dependencies between modules can usually be avoided. Can you provide more details? Why do these classes need references to each other?

[frigid jolt]

you mean defining the TypeVar there? that doesn't work

[trim tangle]

Sometimes you need to get creative with an else block to keep the runtime working ```py
if TYPE_CHECKING:
from . import Server
ServerT = typing_extensions.TypeVar("ServerT", bound=Server, default=Server)
else:
ServerT = TypeVar("ServerT")

though as I said that's a last resort, try other options first

[frigid jolt]

the answer to that is really long and I don't think I'm able to explain it without you looking at the code, which can be completely avoided, but the reason is not weak

[trim tangle]

Importing modules instead of items, as well as using stringified annotations, should fix most issues

[frigid jolt]

I tried it now, I am also using from __future__ import annotations, it still gave me the error

[trim tangle]

you'll need to show the code and the error

[frigid jolt]

from __future__ import annotations

import typing as t
from typing_extensions import TypeVar


ServerT = TypeVar("ServerT", bound=server.BaseDiscordMCPServer[t.Any], default=server.BaseDiscordMCPServer[t.Any])

class Context(t.Generic[ServerT]):
  ...

ImportError: cannot import name 'get_context' from partially initialized module 'discord_mcp.core.server.common.context' (most likely due to a circular import) (C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\common\context.py)

get_context is another member in the context.py file

the typical circular import error

[trim tangle]

Where's get_context called?

[frigid jolt]

it's imported and used in the server.py file

[trim tangle]

You probably need to quote the bound and the default:

ServerT = TypeVar(
    "ServerT",
    bound="server.BaseDiscordMCPServer[t.Any]", 
    default="server.BaseDiscordMCPServer[t.Any]",
)

from __future__ import annotations only impacts annotations

[frigid jolt]

tried, didn't work, same error

[trim tangle]

Then you'll need to show more code

do you have it on github?

[frigid jolt]

I am pushing to github yes

server file: https://github.com/FallenDeity/discord-mcp/blob/feat/autocomplete/src/discord_mcp/core/server/mcp_server.py#L54-L60
context file: https://github.com/FallenDeity/discord-mcp/blob/feat/autocomplete/src/discord_mcp/core/server/common/context.py
the reason to have that server attribute on Context: https://github.com/FallenDeity/discord-mcp/blob/feat/autocomplete/src/discord_mcp/core/plugins/manifests.py#L65-L100
autocomplete is a similar concept as you may already know in discord bot development (if you have experience in that field)

[trim tangle]

Can you show the full traceback for the import error?

[frigid jolt]

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\__main__.py", line 1, in <module>
    from discord_mcp.cli import cli_main
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\cli.py", line 11, in <module>
    from discord_mcp.core.server.http_server import run_server as run_http_server
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\http_server.py", line 9, in <module>
    from discord_mcp.core.server.common.context import DiscordMCPContext
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\common\context.py", line 31, in <module>
    ServerT = TypeVar("ServerT", bound=BaseDiscordMCPServer[t.Any], default=BaseDiscordMCPServer[t.Any])
                                       ^^^^^^^^^^^^^^^^^^^^
NameError: name 'BaseDiscordMCPServer' is not defined

[trim tangle]

!cleanban 1366092950601470063 spam

[rough sluice]

:incoming_envelope: :ok_hand: applied ban to @sinful raptor permanently.

[trim tangle]

#type-hinting message

typing.TYPE_CHECKING is false at runtime, that's how it avoids runtime issues with circular imports. So you need to quote names that don't exist at runtime

also that seems like a different error

[frigid jolt]

yes sorry I undid it before

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\__main__.py", line 1, in <module>
    from discord_mcp.cli import cli_main
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\cli.py", line 11, in <module>
    from discord_mcp.core.server.http_server import run_server as run_http_server
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\http_server.py", line 9, in <module>
    from discord_mcp.core.server.common.context import DiscordMCPContext
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\common\context.py", line 16, in <module>
    import discord_mcp.core.server.mcp_server as server
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\mcp_server.py", line 46, in <module>
    from discord_mcp.core.plugins.manager import DiscordMCPPluginManager
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\plugins\manager.py", line 8, in <module>
    from discord_mcp.core.plugins.manifests import BaseManifest, PromptManifest, ResourceManifest, ToolManifest
  File "C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\plugins\manifests.py", line 15, in <module>
    from discord_mcp.core.server.common.context import DiscordMCPContext, get_context
ImportError: cannot import name 'DiscordMCPContext' from partially initialized module 'discord_mcp.core.server.common.context' (most likely due to a circular import) (C:\Users\snipy\Desktop\discord-mcp\src\discord_mcp\core\server\common\context.py)
``` the issue is shifted but persists

do I need to import the module in server.py as well instead of members

[trim tangle]

Do you understand why these errors occur at runtime with item improts? Maybe I can clear that up

It's similar to this scheme: ```py

foo.py

from bar import b

def a():
return b

bar.py

from foo import a
b = [a]
Imagine that `foo` is executed first. The first line of `foo` needs to import `b` from `bar`. But to populate the `b` item, `bar` needs to execute `foo` and extract `a` from it. However, if you have module imports, there's no such requirement:py

foo.py

import bar

def a():
return bar.b

bar.py

from foo import a
b = [a]
``` since bar.b isn't accessed until a() is called

[frigid jolt]

I know and understand why it happens, what I don't understand is why it's needed to have members of a type variable used in a Generic class imported at runtime if they are useful exclusively when TYPE_CHECKING is True, so not at runtime

[trim tangle]

You can quote the names that are only present in a TYPE_CHECKING block

[frigid jolt]

is this your answer to #type-hinting message

[trim tangle]

yes

They're not strictly needed at runtime, you can quote them if they're "fake"

And with 3.12/3.13 syntax, they will be counted towards from __future__ import annotations or the new lazy evaludation scheme from 3.14

class Foo[S: Server]: ...  # 3.12
class Foo[S: Server = Server]: ...  # 3.13
``` which isn't an option in libraries, but it's at least improving over time

[frigid jolt]

like TypeVar("T", bound="AnyType") or do you mean class A(t.Generic["T"]): ...

[trim tangle]

The first one

[frigid jolt]

they are needed at runtime if you need to import them outside a TYPE_CHECKING

[frigid jolt]

isn't the actual answer to this that __annotations__ needs to be populated by actual types

[trim tangle]

With lazy evaluation (with from __future__ import annotations or the 3.14 mode), this would work: ```py

foo.py

import bar
class Foo:
attr: bar.Bar

bar.py

import foo
class Bar:
attr: foo.Foo

```py
# foo.py
from bar import Bar
class Foo:
    pass

# bar.py
from foo import Foo
class Bar:
    pass
``` So it's primarily due to item imports. There's a dirty trick that can make this work: ```py
# bar.py
class Bar:
    pass

from foo import Foo

since the bar module will already have a Bar member by the time foo tries import it (when bar is a "partially initialized module"), it will all succeed

[trim tangle]

Sounds like that's different from the code on github, line 16 of core\server\common\context.py is now in a TYPE_CHECKING block

[frigid jolt]

yeah, it was the server.Server stringified approach you suggested, and failed with that error

I have decided to use this last resort https://github.com/FallenDeity/discord-mcp/blob/feat%2Fautocomplete/src%2Fdiscord_mcp%2Fcore%2Fserver%2Fcommon%2Fcontext.py#L20-L23

[rough sluice]

src%2Fdiscord_mcp%2Fcore%2Fserver%2Fcommon%2Fcontext.py lines 20 to 23

    ServerT = TypeVar("ServerT", bound=BaseDiscordMCPServer[t.Any], default=BaseDiscordMCPServer[t.Any])
else:
    # to avoid importing BaseDiscordMCPServer at runtime causing circular imports errors
    ServerT = TypeVar("ServerT")```

[trim tangle]

That should be simplifiable to ```py
if t.TYPE_CHECKING:
from discord_mcp.core.bot import Bot
from discord_mcp.core.server.mcp_server import BaseDiscordMCPServer, DiscordMCPStarletteApp, STDIODiscordMCPServer

ServerT = TypeVar("ServerT", bound="BaseDiscordMCPServer[t.Any]", default="BaseDiscordMCPServer[t.Any]")

[frigid jolt]

quoting the default too?

TypeVar is from typing_extensions

[trim tangle]

sorry, that was a typo

[frigid jolt]

I mean I tried that before the stringified and non-stringified server.Server approach

I can try later to see what error is raised but it should fail

[trim tangle]

because of how stuff works in Python, circular imports are often a pain

same with other languages with assignment semantics like JS (maybe Lua as well?)

[frigid jolt]

at least I were able to use that else trick

didn't think about that

ty for your help 🙏

[spare mauve]

that's with |=, it didn't seem to help

all I can figure is that {}.keys technically returns a dict_keys({}) (which isn't instantiable)

[sweet sparrow]

Hello, is it possible to define a type generic to a numeric value rather than a type ? (e.g: a type FixedLengthTuple[t, n] (where t would be a type and n an int) that would be the type n-length Tuple of elements of type t ?

[sour stratus]

I mean, couldnt you just use a list of your type and use len(list) to get the size ?

[sweet sparrow]

I could but I wonder if it is possible. To be honest I'm intending to implement some matrices logic and I wonder if I could define a generic matrix type like Msytic[t: type, ndim: int, dims: tuple of n int].

[sour stratus]

I guess it's for learning purposes,

It looks like you wanna use a class there

[sweet sparrow]

It is.

Yes, I will necessarily use a class but I'd also like to implement a type validation with annotations and 3rd party tools with being just more "Is it a Sequence of Sequences of elements" or similar. I wonder if it's possibility to include the dimensions in the definition of types.

[sour stratus]

Oh,

Yes you can type hint a "dimension" with Typing.Annotated https://docs.python.org/3/library/typing.html#typing.Annotated

But it's not gonna validate it at Run time, it's only checked by a Static Type Checker like Pyright or Mypy

Using Annotated, you'll need to create Annotations yourself like MaxLen or ValueRange

@dataclass
class ValueRange:
    lo: int
    hi: int

T1 = Annotated[int, ValueRange(-10, 5)]

#source : https://stackoverflow.com/questions/68454202/how-to-use-maxlen-of-typing-annotation-of-python-3-9
class MaxLen:
    def __init__(self, value):
        self.value = value

def sum_nums(nums: Annotated[List[int], MaxLen(10)]):
    return sum(nums)

[sour stratus]

The Stackoverflow example also give an example on how to validate at runtime with a simple Decorator

[sweet sparrow]

Would you have a link to that SO example please?

Oh, at runtime.

[sour stratus]

It's the link in the code block

https://stackoverflow.com/questions/68454202/how-to-use-maxlen-of-typing-annotation-of-python-3-9

[sweet sparrow]

Oh, mybad.

[sour stratus]

You need a type checker to get type errors/warnings in your ide

Pyright / Mypy are mostly used

(Pylance extension on vscode uses pyright)

[sweet sparrow]

I usually use mypy, but I guess that what I wan't isn't currently possible for static type analysis.

(if not simply impossible)

[sour stratus]

With Annotated, it IS going to be checked by the type checker at static Analysis, and WILL give your errors

[sweet sparrow]

Sorry, I'm only seeing your message now. At runtime you mean?

[sour stratus]

No, your type checker (that runs either directly in your IDE or by hand by running pyright for example) will check your type hints and tell you there might be an issue with you types.

But if you run the code, it will run without telling you there is a type issue

[sweet sparrow]

I thought Annotated metadata were mostly ignored unless the static type checker had some implemented way to interpret it ? And the SO link seems to be for runtime.

[sour stratus]

They're used by the type checker, but the errors can be ignored and should not cause issues with the code at runtime

They can also be used to implement some sort of Validation at runtime

[sweet sparrow]

If a library or tool encounters an annotation Annotated[T, x] and has no special logic for the metadata, it should ignore the metadata and simply treat the annotation as T.
--- https://docs.python.org/3/library/typing.html#typing.Annotated

I apologize but I frankly don't get it. Are you saying that the StackOverflow link above gives a way to make the type checker (mypy for instance) handle Annotated metadata as additional logic to check and that the said metadata won't be ignored by the tool ?

[sweet sparrow]

Apparently you can cause mypy to fail in annotated if an exception happens but I don't find that really useful. 🙂 https://github.com/python/mypy/issues/19438#issue-3229662446

[restive rapids]

But it's not gonna validate it at Run time, it's only checked by a Static Type Checker like Pyright or Mypy
its the opposite, the typechecker ignores metadata, it only checks the types

[feral wharf]

I wonder if it's possible to access overloads pre 3.11?
Like before https://docs.python.org/3/library/typing.html#typing.get_overloads

[trim tangle]

No, before 3.11 it just returned a dummy with no special effects
https://github.com/python/cpython/blob/3.10/Lib/typing.py#L2009-L2044

some unholy stdlib code https://github.com/python/cpython/blob/3.14/Lib/typing.py#L2622-L2629

[rough sluice]

Lib/typing.py lines 2622 to 2629

# classmethod and staticmethod
f = getattr(func, "__func__", func)
try:
    _overload_registry[f.__module__][f.__qualname__][f.__code__.co_firstlineno] = func
except AttributeError:
    # Not a normal function; ignore.
    pass
return _overload_dummy```

[trim tangle]

I guess that's to be expected

You could theoretically patch the function in typing if you know that the module using it will not be using it before you can patch it, but it sounds like a bad idea

[feral wharf]

Ah shit

Nvm then. Can't patch a stdlib in a library like discord.py lol

[frigid jolt]

The goal justifies the means /j

[tranquil ledge]

If you use typing_extensions.overload everywhere, then typing_extensions.get_overloads() would work.

Forces a strong dependency though, and won’t work on third-party libraries that use the stdlib overload.

[feral wharf]

Makes sense

[gleaming goblet]

Is there a way with Multi-inheritance / mix-ins to basically append new function overloads / single-dispatch implementations?

[violet solar]

def add_class(
        self,
        class_: Optional[T] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> T | Callable[[T], T]:

    def wrapper(class_: T) -> T:
            if self.get_class_name(class_) in self.custom_classes_names:
                raise ClassAlreadyInitialized(
                    f"Class {name} already exists"
                )

            c: _customclass = _customclass(
                name or class_.__name__, class_, from_data, to_data
            )
            self.custom_classes_names.append(c.classname)
            self.custom_classes.append(c)
            setattr(self, c.classname, c)
            return class_

        if class_:
            return wrapper(class_)
        return wrapper```

why is this causing this typehinting error

Untyped class decorator obscures type of class; ignoring decoratorPylancereportUntypedClassDecorator

[trim tangle]

Can you show more code? Is T the parameter of the generic Config class or is it a freestanding type variable?

[violet solar]

T = TypeVar("T", bound=type)

[trim tangle]

Try this ```py
class TestConfig:
...

c = Config.add_class(name="MyTest")(TestConfig)
``` What type is c inferred as?

[violet solar]

this will return the decorator cause class_ is not provided

[trim tangle]

fixed

[violet solar]

this will return the original class

after you edited

[trim tangle]

What type does Pylance show for it?

[violet solar]

c = Config.add_class(name="TestConfig")
---
Type of "c" is partially unknown
  Type of "c" is "(Unknown) -> Unknown"PylancereportUnknownVariableType
(variable) def c(Unknown) -> Unknown```

c = Config.add_class(name="TestConfig")(TestConfig)
---
Type of "c" is unknownPylancereportUnknownVariableType
(variable) c: Unknown

[trim tangle]

I think I see the problem. The call is equivalent to this:

decorator = Config.add_class(name="MyTest")
TestConfig = decorator(TestConfig)

so when you call decorator, Pylance doesn't know what T is supposed to mean. Therefore it assumes that it is Unknown (which is the same type as Any)

[violet solar]

ok so how can i fix this

[trim tangle]

First, let's simplify the code for now so that you always need to use it as a decorator factory. Otherwise you need an overload. (that's an unrelated problem) ```py
def add_class(
self,
*,
name: Optional[str] = None,
from_data: Optional[Callable[..., object]] = None,
to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
) -> Callable[[T], T]:

def wrapper(class_: T) -> T: ...
return wrapper

[violet solar]

i use it for both decorator and non-decorator

previously it has name as a arg and class_ as kwarg so it didnt raise a problem but then i switched it so i can use the decorator without calling anything]

[trim tangle]

...then you need to replace the Callable[[T], T] with a protocol

from typing import Protocol, TypeVar

T = TypeVar("T", bound=type)
class IdentityDeco(Protocol):
    def __call__(self, arg: T, /) -> T:
        ...

# or if you're using Python 3.12 or higher

class IdentityDeco(Protocol):
    def __call__[T: type](self, arg: T, /) -> T:
        ...

...
    def add_class(
        self,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> IdentityDeco:

    def wrapper(class_: T) -> T: ...
    return wrapper

[violet solar]

that is confusing

[trim tangle]

Python's type system doesn't have a better way of expressing "callable that returns the same type as the argument"

[violet solar]

my package is 3.9<= its on pip

[trim tangle]

then use the first variant

[violet solar]

ok

but it also works on higher versions so can that cause problems?

[trim tangle]

yes

wait, I misread

[violet solar]

what do i put in __call__

[trim tangle]

Protocols typically just have ... in the body

[trim tangle]

It should work fine in later versions

I do realize that I use ... to mean both literally ellipsis and also omission

[violet solar]

rest code is same just replace the -> T | Callable[[T], T] with -> Identity Deco and add the identity deco coded

[trim tangle]

If you want the T | Callable[[T], T] thing to work you'll need to add overloads

I'm assuming that needs to be done so that you can use @Config.add_add_class with no parentheses?

[violet solar]

yea

how do i add a overload perhaps

[trim tangle]

If you don't have that in previous versions I'd just require the parentheses, this makes the typing a nightmare because overloads are doo doo

[violet solar]

it was like that before and i had no problems but when i was taking feedback a guy recommended it and i agreed since it looked easy

[trim tangle]

I'd just put an example like ```py
@Config.add_class() # note: parentheses are required even with no arguments
class TestConfig:
...

It's tempting (and dataclasses.dataclass does it) but it kinda just adds unnecessarily complexity IMO. You can keep it if you like it of course

[violet solar]

previusly i had the name as a the standard arg and class_ as a kwarg so you either had to use a decorator and call it or for without decorator add class_=MyClass

[violet solar]

yea thats the example he gave so i said how hard can it be

[trim tangle]

I recommend reading https://typing.python.org/en/latest/spec/overload.html. It's going to be something like this

# ... means ... literally
    @overload
    def add_class(self, cls: T, /) -> T:
        ...

    @overload
    def add_class(
        self,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> IdentityDeco:
        ...

    def add_class(
        self,
        cls: type[object] | None = None,
        /, *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> object:
        # implementation

Above I said that overloads are doo doo because they are not verified by the type checker, and you can end up with an exponential number of overloads easily

[violet solar]

ok let me try

[violet solar]

nope doesnt work

 @overload
    def add_class(self, class_: T, /) -> T:
        ...

    @overload
    def add_class(
        self,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> IdentityDeco:
        ...
        
    def add_class(
        self,
        class_: Optional[object] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> object:```

[trim tangle]

What error do you get?

[violet solar]

its showing Unknown if i dont call the decorator and if i call the decorator its showing the same error i mentioned above

[trim tangle]

Can you show the code you're testing this on and all the revealed types?

[violet solar]

this code

[trim tangle]

Try this ```py
direct = Config.add_class(TestConfig)
decorator = Config.add_class(name="MyTest")
applied_decorator = decorator(TestConfig)

reveal_type(direct)
reveal_type(decorator)
reveal_type(applied_decorator)

I'm assuming Config is not a class? Since it's a regular method and not a class/static method. Non-class variables are typically named with snake_case/SCREAMING_SNAKE_CASE in Python

[violet solar]

i fixed it i am just dumb

prob the 4 hour of sleep with 30mg caffiene and being awake from 4am to 11am

i didnt fking upload the overload from typing :C

[trim tangle]

isn't that like 1/3 of a coffee cup

or did you miss a zero

[violet solar]

i drink chai

[trim tangle]

understandable

you should probably get an error on the missing import

(but sometimes it's easy to miss an error when there's multiple)

[violet solar]

i didnt run the file i just based it of pylance

cause the main issue was from pylance not recognizing

when i ran it i got error and solved the issue

@overload
    def add_class(self, class_: T, /) -> T:
        ...
        
    @overload
    def add_class(
        self,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> IdentityDeco:
        ...

    @overload
    def add_class(
        self,
        class_: Optional[T] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> T:

    def add_class(
        self,
        class_: Optional[object] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
    ) -> object:```

[trim tangle]

pylance is supposed to recognize undefined variables

iirc it does that even in basic mode (when type checking is not done)

[foggy garnet]

I'm considering adding a top-level types.py module to a library I maintain. it will contain type definitions. anyone have experience with collecting all the public types in one place?

[dense summit]

hello, how can i add multiple onboud generics to my interfaces (abc classes) and then define them once I inherit from my child classes, e.G:
i'm using python 3.12

from abc import ABC, abstractmethod

class Foo[T: SomethingA, V: SomethingB](ABC)
  @abstractmethod
  def foo_a(something_a: T) -> T: ...
  @abstractmethod
  def foo_b(something_b: V) -> V: ...

class ChildFoo(Foo[SomethingRelatedToA, SomethingRelatedToB])
  def foo_a(self, something_a: SomethingRelatedToA) -> SomethingRelatedToA:
    # code here
    ...
  def foo_b(self, something_b: SomethingRelatedToB) -> SomethingRelatedToB:
      # code here
      ...

im kinda confused cause im trying something similar as mypy docs explain https://mypy.readthedocs.io/en/stable/generics.html#generic-protocols

This code apparently doesn't work

[restive rapids]

I don't think generics are the problem? why is the first parameter not self

[violet solar]

@overload
    def add_class(self, class_: T, /) -> T:
       
        ...
        
    @overload
    def add_class(
        self,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
        typechecking: bool = False,
    ) -> IdentityDeco:
       
        ...

    @overload
    def add_class(
        self,
        class_: Optional[T] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
        typechecking: bool = False,
        meta_data: Optional[dict[str, dict[str, Any]]] = None,
    ) -> T:
        
        ...
        
    def add_class(
        self,
        class_: Optional[object] = None,
        *,
        name: Optional[str] = None,
        from_data: Optional[Callable[..., object]] = None,
        to_data: Optional[Callable[..., dict[str, AcceptableTypes]]] = None,
        typechecking: bool = False,
        meta_data: Optional[dict[str, dict[str, Any]]] = None,
    ) -> object:```

Pylance is complaining for the 2nd overload 
`Overload 2 for "add_class" overlaps overload 3 and returns an incompatible typePylancereportOverlappingOverload`

https://paste.pythondiscord.com/V5AA

[rustic gull]

Hey everyone, I need a little help.
So, i have to create a pdf from a markdown .md file with a given schema json file with yaml language server.
The following line is at the top of .md file

yaml-language-server: $schema=schemas/page.schema.json

and the schema.json file is available inside a folder in the same directory

[trim tangle]

make class_ and meta_data not optional in the last overload

Actually, why do you need three overloads? Why not merge the third one and the first one?

[violet solar]

but i need it to be optional

[violet solar]

ok so i have 4 conditions for this add_class function to work

1. you use it as @Config.add_class simple decorate uses the 1st overload
2. Calling The Function and still using as a decorator @Config.add_class(name="MyClass") uses the secound overload
3. Using it without a decorator normal passing in class Config.add_class(MyClass) also uses 1st payload or can also use 3rd overload
4. Using it without a decorator and passing in arguments Config.add_class(MyClass, name="MyClass")it uses the 3rd overload

i figured this pylance error is only from the library side not visible to user to i dont have to fix it

[trim tangle]

So you need two overloads: one for the decorator version, one for the non-decorator version

# decorator version
@overload
def add_class(self, *, name: Optional[str] = None, other_parameter: Optional[Foo] = None) -> IdentityDeco: ...

# non-decorator version
@overload
def add_class(self, class_: T, *, name: Optional[str] = None, other_parameter: Optional[Foo] = None) -> T: ...

[violet solar]

i had that before but it messes with the non calling decorator somehow i think i posted the error here aswell

[trim tangle]

Conditions 1 and 3 are exactly the same

[violet solar]

well i might try it later rn just published the v5.0.0a2

[trim tangle]

Remember, this py @foo class MyClass: ... is exactly the same as ```py
class _MyClass:
...

MyClass = foo(_MyClass)

[violet solar]

i know that but pylance messing smh

[trim tangle]

class_ in the third overload cannot be optional

[violet solar]

removing the first overload didnt fix the pylance error i mentioned on 2nd overload

[violet solar]

oh yea that fixed it

[stiff acorn]

import enum
from typing import overload, Self, Any

class IntEnum(enum.IntEnum):

    @overload
    @classmethod
    def get(cls, key: Self, default: Any = ...) -> Self: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: Self = ...) -> Self: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: None = None) -> Self | None: ...
    
    @classmethod
    def get(cls, key: int | str | Self, default: Self | None = None) -> Self | None:
        try:
            cls(key)
        except ValueError:
            return default

class Color(IntEnum):
    RED = 1
    BLUE = 2

a = Color.get("AAA")

reveal_type(a) # Type of "a" is "Color"

how do I type this? a should be Color | None

my reasoning for these overloads:

1. if key is Self then get will always succeed
2. if key is int | str with default = Self, then get will always return Self
3. if key is int | str with default = None, then get will either return Self or None

I guess 2 matches Color.get("AAA") so i get Color instead of Color | None

but if i swap them I get another error

Overload 2 for "get" overlaps overload 3 and returns an incompatible type (reportOverlappingOverload)

[trim tangle]

a = Color.get("AAA") hits this overload ```py
@overload
@classmethod
def get(cls, key: int | str, default: Self = ...) -> Self: ...

(because they are prioritized in definition order)

If I was a typechecker I would reject it because it's confusing (overload 3 has a default argument, but if you don't provide a value for default this overload will never be invoked)

[stiff acorn]

I tried flipping it:

    @overload
    @classmethod
    def get(cls, key: Self, default: Any = ...) -> Self: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: None = ...) -> Self | None: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: Self = ...) -> Self: ...

    @classmethod
    def get(cls, key: int | str | Self, default: Self | None = None) -> Self | None:
        try:
            cls(key)
        except ValueError:
            return default

but I got Overload 2 for "get" overlaps overload 3 and returns an incompatible type

[trim tangle]

    @overload
    @classmethod
    def get(cls, key: Self, default: object = ...) -> Self: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: None = ...) -> Self | None: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: Self) -> Self: ...

The third overload is typically made into this ```py
@overload
@classmethod
def get[D](cls, key: int | str, default: D) -> Self | D: ...

[stiff acorn]

object ended up causing trouble because the implementation is Self | None so i went with this:

class IntEnum(enum.IntEnum):

    @overload
    @classmethod
    def get(cls, key: Self, default: Self | None = ...) -> Self: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: None = ...) -> Self | None: ...

    @overload
    @classmethod
    def get(cls, key: int | str, default: Self) -> Self: ...

    @classmethod
    def get(cls, key: int | str | Self, default: Self | None = None) -> Self | None:
        try:
            return cls(key)
        except ValueError:
            return default

[trim tangle]

Just as a precaution, accepting Self in the parameter position is unsound. Here's an example

class Color(IntEnum):
    RED = 1
    BLUE = 2

class Banana(IntEnum):
    GREEN = 67
    RIPE = 68
    OVERRIPE = 69


def f(t: type[IntEnum], x: IntEnum) -> None:
    c = t.get(x)
    reveal_type(c)  # Type of "c" is "IntEnum"
    assert c is not None

f(Color, Banana.RIPE)  # AssertionError at runtime

actually I guess it's the overload that's unsound? not sure

Just don't accept type[IntEnum] or an abstract IntEnum anywhere and it should be fine

[stiff acorn]

well my plan was to subclass this IntEnum (well StrEnum too, but this is an example) so having .get on the base class was the goal

[trim tangle]

Yeah, I understand

[stiff acorn]

for default, I only wanna allow Self | None

[trim tangle]

when an enum class has members, it's considered final

[stiff acorn]

if Self is unsound, how else can i spell the fact that default should be always be the same type or None?

same with key i guess

[trim tangle]

This is the only way to do this I think

[stiff acorn]

I'll go with that then

thanks

[civic river]

A question of preference/taste for how to make mypy happy. Consider this line of code:

https://github.com/ubernostrum/akismet/blob/67c94eb739316fd0cb5f12662ab12e49ed48ed29/src/akismet/_sync_client.py#L312

which calls this function:

https://github.com/ubernostrum/akismet/blob/67c94eb739316fd0cb5f12662ab12e49ed48ed29/src/akismet/_common.py#L259

kwargs is hinted in _post_request() as being a TypedDict of type AkismetArguments. The goal of _prepare_post_kwargs() is to act as a runtime enforcer of that, making sure you didn't pass in any unsupported arguments. But _prepare_post_kwargs() is hinted as accepting dict, and mypy doesn't like that, because it violates the rules of the Python static type system (https://typing.python.org/en/latest/spec/typeddict.html#assignability):

A TypedDict isn’t assignable to any Dict[...] type, since dictionary types allow destructive operations, including clear(). They also allow arbitrary keys to be set, which would compromise type safety.

OK. So, there are a few ways to do this. One is to hint _prepare_post_kwargs() as accepting AkismetArguments, even though it's there for the case where someone ignored the type hint of _post_request() and passed something that actually isn't an AkismetArguments dictionary. Another is to just # type: ignore at the call site. I'm sure there are probably other ways to make it pass too.

Anyone have a strong preference?

[rough sluice]

src/akismet/_sync_client.py line 312

**_common._prepare_post_kwargs(kwargs, endpoint),```
`src/akismet/_common.py` line 259
```py
def _prepare_post_kwargs(kwargs: dict, endpoint: str) -> AkismetArguments:```

[trim tangle]

You can use collections.abc.Mapping instead of dict

[frosty flume]

are you guys still importing List and Dict from typing?

[frigid jolt]

it really depends what python versions your project should support

if I make a project intended for python 3.10 or higher I just use the built-in types

[frosty flume]

That makes sense >>>>>

[trim tangle]

since 3.9 a bunch of items from typing are deprecated, so it's not really a preference https://docs.python.org/3/library/typing.html#deprecated-aliases

python 3.8 is not supported anymore, so you shouldn't import List or Dict from typing in any project

[frosty flume]

yeah. Thanks.

[stiff acorn]

class RawPasteContent(TypedDict):
    paste: str
    attachment: NotRequired[str]
    attachment_name: NotRequired[str]

is there a way to type this more accurately? This currently only tells you that attachment and attachment_name can be missing. What it doesn't tell you is that they are related. Either both of them are present or both of them are absent. It's not possible for only one of them to be missing.

[brazen jolt]

You could make 2 typed dicts and put them in a union, a type checker should then be able to narrow the type once it's known that one of those fields is present

question is whether that's something you consider worth it, especially if you have more such relations in that type, as this grows very quickly if you have more than 2 variants

[trim tangle]

(they'd need to be the new closed typeddicts for this to work safely though)

[stiff acorn]

I only have this one, but I don't think it's worth it either way. I'll stick with what I have I guess.

[trim tangle]

well, after they are merged

[brazen jolt]

oh yeah, true

ugh, it's annoying that typed dicts aren't closed by default

I always forget

[trim tangle]

pyright actually does let you narrow it this way right now

even though it's unsound

but hopefully that PEP gets accepted

[trim tangle]

and yeah open typeddicts by default are annoying, here's another case
https://github.com/JelleZijlstra/unsoundness/blob/main/examples/typeddicts/typeddict_argument_unpacking.py

[brazen jolt]

isn't using typed dicts in this way even mentioned in the original pep?

did no one take an issue with the fact that they were open?

[trim tangle]

||the type system was ahead of its time by 10 years because it was generated by chatgpt||

[brazen jolt]

definitely the right call, unless you'd have a big reason for modeling this coupling, you shouldn't try to express it in this way

btw, in case you didn't understand what we talked about with the open/closed typed-dicts, basically, right now, by default, typeddicts allow extra keys to be present no matter what you do, so technically, even in the example I sent, it's unsafe to assume that just because "attachment" key is present, the type can be narrowed, since the RawPasteContentA could have any extra keys, and those keys can be of any type, so even with "attachment" present, it could still be RawPasteContentA type.

pyright just doesn't take this into account, so it does work, but it's misleading, with the fact that typeddicts are open by default, and the proposal to allow making closed typedicts still not having been approved yet, there's actually no way to safely represent this relation without relying on the technically wrong behavior that pyright has here

[stiff acorn]

thanks for the nice summary. I'm not too knowledgable about typed dicts since I usually prefer a real class

[brazen jolt]

yeah, same, when possible, I try to deserialize directly into a proper class, maybe with something like pydantic, rather than working with a TypedDict interface

[stiff acorn]

class PrivateBinUrl(msgspec.Struct, frozen=True, kw_only=True):
    server: str
    id: str
    passphrase: str
    @classmethod
    def from_str(cls, url: str, /) -> Self: ...


class DeletablePrivateBinUrl(PrivateBinUrl, frozen=True, kw_only=True):
    delete_token: str
    @classmethod
    def from_str(cls, url: str, /, *, delete_token: str) -> Self: ... # Signature of "from_str" incompatible with supertype "PrivateBinUrl"Mypyoverride

how do i type this without a type: ignore (if possible)

[brazen jolt]

You can't, because it's not type safe. Consider if someone had a sequence of PrivateBinUrl (a type that DeletablePrivateBinUrl is a subtype of) and they called from_str, they'd only see the signature of the base class, so they would call it with that signature in mind, however, if one of the items in your sequence was actually a DeletablePrivateBinUrl, this call would fail, because the override isn't compatible (it would be lacking the delete_token kwarg

here's a more concrete example to illustrate it

from collections.abc import Sequence

def bin_factory(bin_classes: Sequence[type[PrivateBinUrl]]):
    for bin_cls in bin_classes:
        yield bin_cls.from_str("myurl")  # passes the type-checker

bin_factory([DeletablePrivateBinUrl, DeletablePrivateBinUrl, PrivateBinUrl])

[stiff acorn]

That makes sense. I guess I'll have to do without a common base class. I approached it like this because every deletable url is also a valid url but not every url is a deletable url.

[brazen jolt]

you can still use a base class, but make it a base class for both

class Base(msgspec.Struct, frozen=True, kw_only=True):
    ...  # whatever shared definitions, but not from_str

class PrivateBinUrl(Base):
    @classmethod
    def from_str(cls, url: str, /) -> Self: ...

class DeletablePrivateBinUrl(Base):
    @classmethod
    def from_str(cls, url: str, /, *, delete_token: str) -> Self: ...

so you can still have some shared internals if you work with the URLs in the same way

[rancid stratus]

Hey, I've been wondering if there's a better way to narrow it to Group?

right now i've done something like this

rbx_client = RobloxClient(ROBLOX_API_KEY)
_: Group | None = rbx_client.get_group(GROUP_ID)
assert _
group: Group = _

just asserting it doesn't work and pyright complains that in the other files that the type is still Group | None

[rancid stratus]

and this solution seems a bit quirky to me

[brazen jolt]

you shouldn't need an intermediate variable like that

pyright is able to narrow down the type after the assertion and get rid of the None

[rancid stratus]

Yeah but somehow it still is Group | None

in a different file

using this

[brazen jolt]

can you put together a minimal example?

[rancid stratus]

sure

[brazen jolt]

oh you're right, huh, that is odd

[rancid stratus]

it's like pyright just ignores that assert statement

[brazen jolt]

mypy behaves the same way

[rancid stratus]

hmm

[brazen jolt]

I mean you can definitely just do something like this

but I'm quite surprised that the type narrowing didn't work across files

[rancid stratus]

maybe it would be too expensive to check everything so it only gets the type straight from the variable without considering any asserts

that's my guess

[brazen jolt]

it actually ignores any kind of type narrowing, even something like: python if x is None: raise would still make pyright think x is int | None when imported

[brazen jolt]

it might be some lazy evaluation behavior yeah

[rancid stratus]

i think i'll do something like this then

or just change get_group to return Group only

[brazen jolt]

oh, I thought you didn't have control over that, if you do, that might be the better choice yeah, and just raise if you can't get it, but it does depend what that function is doing and why the group can sometimes be none

[rancid stratus]

i don't know why i didn't think of that earlier xd

it's my custom wrapper

alright thanks then anyways 👍

[slender basin]

Hey. In this simplified snippet:

def gen_list() -> list[int | str]:
    return ["q","w","e"]


def take_list_of_str(arg: list[str]):
    ...

def test():
    order = gen_list()
    assert all(isinstance(expr, str) for expr in order)
    return take_list_of_str(order)   # <===

type checkers complain at the last line, similar to "Argument list[int | str] is not assignable to parameter arg with type `list[str]" . Tested on pyright, ty and pyrefly (seems mypy doesn't go deeper than 'list' container).
I hoped the assert would convey the missing extra info to the type checkers. Is there any different way I can achieve that?

[brazen jolt]

you can use a TypeGuard function, like so: ```python
from typing import TypeGuard, reveal_type

def is_str_list(arg: object) -> TypeGuard[list[str]]:
if not isinstance(arg, list):
return False
return all(isinstance(item, str) for item in arg)

def gen_list() -> list[int | str]:
return ["q", "w", "e"]

def take_list_of_str(arg: list[str]): ...

def test():
order = gen_list()
reveal_type(order)
assert is_str_list(order)
reveal_type(order)
return take_list_of_str(order)

that said, it's actually unsafe to narrow a list like this

since lists are what's called invariant

[slender basin]

Thank you! Can you please give an example of a case where this narrowing is unsafe?

[trim tangle]

def fn():
    foo: list[int | str] = ["a", "b", "c"]
    bar: list[int | str] = foo
    assert is_str_list(foo)
    bar.append(42)
    # now `foo` is a `list[str]` that has 42 as one of the elements

[slender basin]

Not sure how is this described as "lists aren't invariant". Maybe invariant == immutable? Anyway that's enough for me. Thanks!

[trim tangle]

list is invariant. It means that even if A is a subtype of B, list[A] and list[B] aren't subtypes of each other. This is opposed to covariant types (e.g. tuple[A, ...] is a subtype of tuple[B, ...]) and contravariant types (e.g. Callable[[B], None] is a subtype of Callable[[A], None])

[slender basin]

Man, typing is hard. Why is this a problem?

def cond() -> bool:
    ...

def func(children: list[int] | list[str]) -> list[int] | list[str]:
    # " Returned type `list[int | str] | list[int] | list[str]` is not assignable to declared return type `list[int] | list[str]`" :
    return [children[0]] if cond() else children

This code cannot return a list[int | str] . What am I missing that can convey this to the type checkers?

[brazen jolt]

simply said, this is the reason

you could do this: ```python
from typing import TypeVar

IntOrStr = TypeVar("IntOrStr", int, str)

def cond() -> bool: ...

def func(children: list[IntOrStr]) -> list[IntOrStr]:
return [children[0]] if cond() else children

[slender basin]

No need to TypeVar for that, I could use list[int|str] directly - but that would mess things down the line.

[brazen jolt]

no, that's not the same

list[int | str] wouldn't let you call the function with list[int] (lists are invariant, remember?)

I would however recommend that you take a look at collections.abc.Sequence type, which list is a supertype of, basically, it allows you to pass in any immutable sequence (or a mutable one, but it won't let you actually mutate it without it being a type error), so you will also be able to use types like a tuple or a set when you call the function. The advantage of a Sequence type is also that unlike lists which are invariant, sequence is covariant, so you can do Sequence[int | str] and it will let you pass in both a list[str] or a list[int], since if you won't be mutating these, it's not an issue

[brazen jolt]

in this case, you could also cast here, you know that [children[0]] will for sure only be list[str] | list[int], the type checker is just too dumb to understand it here, so you can tell it to assume that type:

from typing import cast

child_list = cast("list[str] | list[int]", [children[0]])

[brazen jolt]

the typevar also won't even allow lists of list[int | str] here, a type-var like: TypeVar("IntOrStr", bound="int|str") would, but with the syntax I used above, it will only let you use one of those types, not a union of them

[brazen jolt]

and here's the example with casting, but notice how with the type-var, the return type is better, since it's preserved, while this would only give you back the union, so this is more just to demonstrate it as an option, as sometimes it is useful, but for this, I'd go with the type-var

[slender basin]

Thanks a lot everyone. I'll get the hang of python types (and checkers limitations) eventually.

[brazen jolt]

no worries, this can definitely be a bit confusing when you're just starting to explore using type checkers

[spiral fjord]

def func[T: (str, int)] in modern python

[brazen jolt]

Oh, nice. Wasn't sure it was possible with the new syntax, cool to see that it is

[feral wharf]

TIL

I don't remember the docs mentioning that

Nvm it does https://docs.python.org/3/library/typing.html#typing.TypeVar

[feral wharf]

can we somehow do type checking on list/tuple length?

[trim tangle]

You can specify a heterogenous tuple like tuple[int, int] or tuple[str, str, bool] or tuple[()]

[feral wharf]

oh true

wba lists

[restive rapids]

nope, lists are always variable length

[feral wharf]

not even via some protocol abuse trick?

[restive rapids]

it will not be a list at this point: you wont be able to append, pop, whatever
neither would you be able to "create" a value of such type without a runtime check and a copy, because the typechecker itself has no information on the lists length even in trivial cases, and imagine cases like:

def f(xs: list[int]):
  assert len(xs) == 2
  ys: listN[int, Literal[2]] = xs # "fine" but only if `xs` is not apended/popped to later, because otherwise,
  xs.append(42) # now ys has 3 elements

[feral wharf]

ah welp

[tired pilot]

I'm running into some frustrations when trying to use ruamel.yaml, and it's exposing some of my ignorance regarding generics. I'm dealing with some CommentedMap objects, which is a subclass of OrderedDict to which ruamel.yaml adds additional functionality. In the process of doing this, it also removes the generics from these subclasses using a # type: ignore internally, so for example I cannot specify x: CommentedMap[str, int] = .... It's effectively a dict[Any, Any].

I've found that I can assign CommentedMaps to dicts to get the benefits of knowing the contents, but then obviously the type checker doesn't know that they have the extra attributes of a CommentedMap:

test: dict[str, float] = CommentedMap({"a": 1.0})
test.yaml_set_comment_before_after_key("a", before="\n")  # type: ignore[attr-defined]

Is there a clean way to get the best of both worlds, with type hints for CommentedMap objects which function like the generic dict type while still knowing about the added attributes?

[trim tangle]

I would just live with it being a dict[Any, Any] tbh. It's a bug/deficiency in ruamel if they want to support type annotations well

tbh this ordereddict thing looks cursed

[tired pilot]

It sure is

[trim tangle]

why does it conditionally import from an ordereddict module? that's just weird

[oblique urchin]

Python 2.6 compat?

[trim tangle]

yeah...

ordereddict on PyPI is a backport of ordereddict to Python 2.6 and hasn't had a release since 2010

[oblique urchin]

oh lol I guessed the version right. OrderedDict was added in 2.7 and 3.1

yeah probably time to remove that fallback

[trim tangle]

I mean, it's not doing any harm it seems

but if they want to support Python 2.6, making this class generic will be difficult

[tired pilot]

I would work off the assumption that ruamel.yaml isn't interested in supporting much. I'm just trying to work with what's inside my control.

[tired pilot]

So I saw in the Mypy docs that you can use a parameterized TypeGuard to type-narrow, and that seems like it would be a really useful pattern for some things I'm working on where I want to narrow strings to specific Literal options to be used as keys in TypedDict assignments.

But when I tried it, I found that it doesn't seem to work for Literals - I get errors about 'incompatible type "<typing special form>"'. It works fine for other types like str or float, and it works when the type is simply specified up front (i.e. the non-parameterized equivalent), just not when it's passed in as a parameter. Examples: https://mypy-play.net/?mypy=latest&python=3.12&gist=3fd51d3c17cc596fe339d4edc88cd293

I'd love to be able to narrow a str to a specified Literal programmatically like this, any thoughts?

[oblique urchin]

Literal is not a type[]. You need to wait for PEP 747 (TypeForm)

[tired pilot]

Side note, before finding the parameterized TypeGuard my initial thought was to assert x in ["a", "b"], but alas, were it that simple and intuitive! I then found it had been proposed a few times before: https://github.com/python/mypy/issues/12535

[tired pilot]

Aha, I figured the solution might be to replace type[T] with some equivalent that handled special forms, and that's exactly how this is used. Thanks! Shame it's not already available!

[tired pilot]

So I can presumably start using from typing_extensions import TypeForm today, get things working with Pyright, and as soon as this pull request is merged into Mypy I can use that as well, no?

[stiff acorn]

from pathlib import Path
from typing import cast, reveal_type, Any, Union

OldPathLike = Union[str, Path, None]
NewPathLike = str | Path | None

foo: Any = cast(Any, "foo")

assert isinstance(foo, NewPathLike)
assert isinstance(foo, OldPathLike) # main.py:10: error: Parameterized generics cannot be used with class or instance checks  [misc]

https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=b98d4d9fc80706dae752e9279ec5fd99
Why is mypy treating them differently?

(I'm aware it's not safe to call isinstance with type aliases, I'm curious about the difference here)

[brazen jolt]

probably because Union itself is typed as _SpecialForm, while the new pipe syntax is considered as a UnionType, but this is something that most people would just consider as undefined behavior

with pyright, you don't get an error on either

it could just be that they special-cased it to produce an error to prevent accidental misuse

[oblique urchin]

For what it's worth I'd consider this a low-priority bug in mypy

[hallow flint]

i think it also used to be invalid, maybe changed in 3.10 or something

you probably want something like:

index 9752a5e68..88b3005b1 100644
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -582,7 +582,10 @@ class ExpressionChecker(ExpressionVisitor[Type], ExpressionCheckerSharedApi):
                     and not node.node.no_args
                     and not (
                         isinstance(union_target := get_proper_type(node.node.target), UnionType)
-                        and union_target.uses_pep604_syntax
+                        and (
+                            union_target.uses_pep604_syntax
+                            or self.chk.options.python_version >= (3, 10)
+                        )
                     )
                 ):
                     self.msg.type_arguments_not_allowed(e)

and then you also need to change typeshed, since the stubs for isinstance are wrong

https://github.com/python/mypy/pull/19714

[trim tangle]

Here's another one:

from typing import TypeAlias

type Foo = int
Bar: TypeAlias = int
Baz = int

print(isinstance(42, Foo))
print(isinstance(42, Bar))
print(isinstance(42, Baz))
``` only the first `isinstance` gives an error, in fact it's two errors

main.py:7: error: Parameterized generics cannot be used with class or instance checks [misc]
main.py:7: error: Argument 2 to "isinstance" has incompatible type "TypeAliasType"; expected "_ClassInfo" [arg-type]

The second error is also alarming, what is _ClassInfo?

mypy seems to be a bit trigger-happy with exposing implementation details in error messages

[hallow flint]

it’s what the typeshed stub says isinstance takes (and it is not exactly right)

[summer berry]

Is there a way to express the interesection of two types? My use case is a function that takes a TypedDict and returns a new TypedDict with some additional keys. It's so I can add common API request parameters to different types of requests.

I suppose not https://github.com/python/typing/issues/213

[oblique urchin]

For that sounds like you might be able to use TypedDict inheritance?

[summer berry]

I don't think so because the input is an arbitrary dictionary.

[foggy garnet]

so I went down a rabbit hole of trying to write a runtime type checker for types relevant for parsing JSON documents (int, float, tuple, union, literal, typeddict) things ballooned in complexity with typeddict in particular. I basically got this working but it feels extremely janky. But the experience gave me an idea -- instead of using a function check_type(obj, typ) that introspects typ and compares it to obj (recursing as needed), what about generate_type_check_func(typ), that generates a re-usable function that does a type check specific to typ. Then I could generate my type-checking routines as part of a CI step or in my test suite, and avoid the runtime cost of type introspection. are there any libraries that take this approach?

generate_type_check_func(typ, mode: Literal['fail_fast', 'verbose']) could control how extensively the generated function parses the input object, giving a nice tradeoff between performance and user-friendly exceptions

[trim tangle]

adaptix uses code generation, though at startup instead of as a separate build step

[fierce ridge]

That's an interesting approach, like a performance versus debug toggle

I don't think I've ever seen that

Apart from something very specific like a contract framework

[trim tangle]

(and from some short tests it outperforms pydantic v2 in some cases)

Separate build step CI is a bit of a pain, but it does mean you can easily read the code. I think both options should be available if you go this route

[foggy garnet]

Separate build step CI is a bit of a pain, but it does mean you can easily read the code. I think both options should be available if you go this route
my idea was to skip a build step, and instead declare expectation of the auto-generated function a test

so that if you run the tests, and the expected auto-generated function is not found, the test complains (and provides a copy of the body of the expected autogenerated function)

the developer then has to copy + paste in the expected function, or exempt that function from the tests

this way the entire source tree has the same provenance, as opposed to a source tree that's a mix of human-written and auto-generated code

[trim tangle]

adaptix does code generation the same way dataclasses.dataclass does it, it generates the code as a string and then execs it

so it's not writing anything to disk or anything like that

[foggy garnet]

i specifically want functions that are persisted to the source code, so they can be read and reasoned about

[fierce ridge]

Maybe this is a silly question, but why not control this with a function parameter, contextvar, env var, etc and dispatch between them at runtime?

[foggy garnet]

i'm not sure I understand the question -- what exactly would be controlled with a function parameter?

it's possible that I have not explained the problem correctly

[trim tangle]

If you're making a library, different users probably have different needs. Some people want to always have the code materialized and others don't

[foggy garnet]

the basic functionality would be generating python source code that evaluates to a function definition of a function that does the type check. that functionality could totally be wrapped in a routine that calls eval and binds the result to a variable, which would give you something you could run immediately

but if anything goes wrong inside one of these auto-generated functions, it might not be fun to debug

that's why my preference would be to use the code gen as a developer assistance, instead of something you use at runtime

[foggy garnet]

here's the implementation of the runtime type checker in case anyone wants to have a look. I'm pretty new to this side of python so it's possible that I'm doing something inefficient here

[terse tree]

Wondering why type-checkers don't work well with contravariance

from typing import Generic, TypeVar

T1 = TypeVar("T1", contravariant=True)
T2 = TypeVar("T2", covariant=True)

class A:
    pass

class B(A):
    pass

class C(B):
    pass

class Contra(Generic[T1]):
    def __init__(self, value: T1):
        self.value = value

class Cov(Generic[T2]):
    def __init__(self, value: T2):
        self.value = value

def f1(x: Contra[B]) -> ...: ...
def f2(x: Cov[B]) -> ...: ...

# 1: pass strict subtypes according to variance, should work
a1 = f1(Contra(A()))  # wrongly fails type-check in mypy
a2 = f2(Cov(C()))

# 2: pass strict supertypes according to variance, should fail
b1 = f1(Contra(C()))  # wrongly passes type-check in pyright & mypy
b2 = f2(Cov(A()))

# 3: pass exact types, should work
c1 = f1(Contra(B()))
c2 = f2(Cov(B()))

[oblique urchin]

what I see in mypy doesn't match what you're saying, and some of your expected results are wrong https://mypy-play.net/?mypy=latest&python=3.12&gist=358e3c2a0a3381a94468068739c2f5f5

f2(Cov(A())) is correctly rejected by mypy

f1(Contra(C())) works because of bidirectional inference, it infers the argument as Contra[B]

[terse tree]

Why does f1(Contra(A())) fail?
and why does the argument that gives b1 get inferred as Contra[B] when Contra[C] is a supertype of Contra[B] (because C is a subtype of B)?

btw f1(Contra(A())) passes in pyright (pylance), which I expect, but f1(Contra(C())) also passes in pyright which I don't expect, so it essentially accepts any supertype or subtype of B

[oblique urchin]

oh right, the Contra(A()) could fail. Seems like uses a wrong type context

Contra(C()) should pass because of type context

essentially, C is assignable to B, so if you infer Contra(C()) as being of type Contra[B], it passes

[terse tree]

Maybe I'm not understanding the bidirectional inference, but I thought it shouldn't infer a subtype. So x: bool = 1 shouldn't pass because 1 should not be inferred to bool; bool is a subtype of int. Similarly, x: Contra[B] = Contra(C()) shouldn't pass because Contra[B] is a subtype of Contra[C].

[oblique urchin]

it's more like x: list[int] = [True]

[True] could be a list[bool], but because bool is a subclass of int, it can also be list[int]

similarly, Contra(C()) could be a Contra[C] but could also be a Contra[B]

[trim tangle]

a cool feature for inlay hints in pylance/basedpyright/pycharm would be showing the inferred type variables in calls, especially to classes

[terse tree]

I get the list[int] example; the declared/expected type list[int] is a supertype
In my case, the declared/expected type Contra[B] is a subtype

[oblique urchin]

it's not a supertype

[terse tree]

I think what you described is how it works in a2 which is fine

well thanks, I'll look into it

[vernal verge]

hi

[bleak imp]

Is there a way to make this type check? I'd like to have a Sequence and Interconnected be non-subclasses of Node so that I can check if an object is exactly a Node, but all three should have the same __sub__ implementation. basedpyright playground link

[hoary pagoda]

I don't believe you can express that NodeSub has no other subclasses than the ones listed in NodePlace

type NodePlace = NodeSub does seem to work

[bleak imp]

I see, I was being silly, if I just make NodePlace into a class with the sub it will work, thanks

[delicate kiln]

What is wrong with this code? https://mypy-play.net/?mypy=latest&python=3.12&gist=c35349fa5f91f6644a906bd0015bc96e
Subtypes having wider interfaces does not break LSP, so I assume it's some weird effect of how Python's operator overloading and poor-man's multiple dispatch are interacting that make mypy unhappy, and it's just not giving me enough information on how to resolve it.

[oblique urchin]

I think mypy is just not good at noticing when an overload override is correct

[neon depot]

Is there a strict interpretation of LSP, under which children have to be bug-compatible with their parents?

[feral wharf]

what's the reason that typing-extensions doesn't support the extra_items classparam? https://typing-extensions.readthedocs.io/en/latest/#typing_extensions.TypedDict
https://peps.python.org/pep-0728/#the-extra-items-class-parameter

[oblique urchin]

it does, we forgot to update the docs I think

[feral wharf]

i'm getting TypedDict does not support __init_subclass__ parameter "extra_items" PylancereportGeneralTypeIssues on the latest

[oblique urchin]

works at runtime. maybe pylance is out of date

[feral wharf]

it does support closed

oh i see

https://github.com/microsoft/pyright/issues/10755 it's still experimental

thanks!

[feral wharf]

edit: it's not anymore as of last week, pylance is just behind https://github.com/microsoft/pyright/releases/tag/1.1.404

[oblique urchin]

Yes, the PEP was accepted very recently

[feral wharf]

makes sense

[oblique urchin]

https://github.com/python/typing_extensions/pull/664 updates the typing-extensions docs

[feral wharf]

can we somehow do a union of typeddicts?

or at least overload them

use case: users can provide their own class in a decorator

[delicate kiln]

Union or intersection?

The type system still (AFAIK) still doesn't support ad-hoc intersection.

[rare scarab]

Closest thing to an intersection is multiple inheritance on a concrete class

[feral wharf]

I need it for a overload specifically

[silent bobcat]

I'm trying to write a dataclass decorator for a custom type that synthesizes some additional methods, but I'm having a really hard time type hinting the decorator for mypy.

class MyABC:
  def __rich_repr__(self): raise NotImplementedError
  def __repr__(self): return rich.pretty.pretty_repr(self)

@functools.wraps(datclasses.dataclass)
def my_dataclass(cls: type[MyABC] | None = None, /, *, repr: bool = False, **kwargs):
  dataclass = dataclasses.dataclass(repr=repr, **kwargs)
  
  def decorator(cls: type[MyABC]) -> type[MyABC]:
    cls = dataclass(cls)
    cls.__rich_repr__ = _my_abc_rich_repr(cls)
    return cls

  return decorator(cls) if cls else decorator

@my_dataclass
class Foo(MyABC):
  a: int

Foo(1)  # mypy doesn't recognize the generated constructor here!

Is @dataclass hardcoded into MyPy somehow? Any ideas how to tell MyPy to treat this decorator as though it's exactly @dataclass , or even better to type hint the additionally synthesized methods?

[trim tangle]

!d typing.dataclass_transform

[rough sluice]

typing.dataclass\_transform

@typing.dataclass_transform(*, eq_default=True, order_default=False, kw_only_default=False, frozen_default=False, field_specifiers=(), **kwargs)```
Decorator to mark an object as providing [`dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass)-like behavior.

`dataclass_transform` may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of `@dataclass_transform()` tells a static type checker that the decorated object performs runtime “magic” that transforms a class in a similar way to [`@dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass).

Example usage with a decorator function...

[trim tangle]

dataclasses used to be hardcoded in type checkers, but now we have this

[silent bobcat]

❤️

I feel like google and AI deeply failed me in not finding that lol, thank you

[trim tangle]

I don't think it's possible to specify extra synthesized methods with the decorator syntax. You can probably do that with a dataclass-transform metaclass

[trim tangle]

Strange, dataclass_transform was added in 2021 so should be in the training data. Maybe there's very little code using it

I asked DeepSeek R1

In Python, how do I type-hint a decorator that acts like dataclasses.dataclass?
and it did come up with typing.dataclass_transform

It did however use pre-3.12 typevar syntax and suggested importing Type from typing. Smh

It also suggested importing Union from typing, which doesn't make sense, | was introduced in 3.10 but dataclass_transform in 3.11

[stiff acorn]

@dataclass
class Folder:
    files: Iterables[str | PathLike[str]]

How do I deal with the fact that I want my constructor types wide (e.g., Iterable[str | PathLike[str]) but internally the class will always transform that into a concrete type tuple[Path, ...] and will always return the concrete type when you do folder.files

I can think of two solutions:

1. Write my own constructors
2. use _field in dataclass and write properties

[trim tangle]

Making a normal class instead of a dataclass sounds like the right solution here

(Unless it's an outlier and you have several other fields)

You could also use @dataclass(init=False)

[stiff acorn]

guess I'll do that, thanks

[foggy garnet]

consider also not using __init__ for instance creation that will parse the inputs

you can instead define a separate method that takes whatever parameters you want, parses them into exactly the types that Folder takes, and safely call Folder()

so

@dataclass
class Folder:
    files: tuple[Path, ...]
    
    @classmethod
    def build(cls, files: Iterable[str | PathLike[str]]) -> Self:
        # parse files into a tuple
        return cls(files=files_as_tuple)

maybe from_iterable is a better name for this method

[indigo halo]

I have a couple of dataclasses and matching TypedDicts and it pains me so much to have all this redundancy. Why? Why doesn't there seem to be a way to create a bloody TypedDict from a dataclass, like I can make a TypeAdapter from a Pydantic model?
or at least let me use a TypedDict as the basis for the dataclasses. But no, right now it seems I have to define two classes with identical fields and god forbid I make a mistake and one has a:int and the other a:str, which is a function of redundancy and never a question of "if" but only ever "when".

I searched the Internet and I could not find anything. I understand that TypedDict is not a real class at runtime.

[stiff acorn]

I'm currently stuck on this. I've

from dataclasses import dataclass
from enum import StrEnum

# ========================= FIRST APPORACH =========================

class Genre(StrEnum):
    # "bare" variant
    ALL = "All"
    
    # "bare" variant
    ROCK = "Rock"

    # concrete variants
    ROCK_CLASSIC = "Rock - Classic Rock"
    ROCK_ALTERNATIVE = "Rock - Alternative Rock"
    ROCK_PUNK = "Rock - Punk Rock"

    # "bare" variant
    HIP_HOP = "Hip Hop"

    # concrete variants
    HIP_HOP_TRAP = "Hip Hop - Trap"
    HIP_HOP_BOOM_BAP = "Hip Hop - Boom Bap"
    HIP_HOP_GANGSTA = "Hip Hop - Gangsta Rap"

    # "bare" variant
    ELECTRONIC = "Electronic"

    # concrete variants
    ELECTRONIC_HOUSE = "Electronic - House"
    ELECTRONIC_TRANCE = "Electronic - Trance"
    ELECTRONIC_DUBSTEP = "Electronic - Dubstep"

@dataclass
class Album:
    title: str
    genre: Genre # Wrong since an album will never be one of the "bare" variants


# Genre is valid here because you can search the "bare" variants or the concrete variants
def search(title: str, genre: Genre) -> Album:
    raise

# ========================= SECOND APPORACH =========================

class ParentGenre(StrEnum):
    ALL = "All"
    ROCK = "Rock"
    HIP_HOP = "Hip Hop"
    ELECTRONIC = "Electronic"

class SubGenre(StrEnum):
    ROCK_CLASSIC = "Rock - Classic Rock"
    ROCK_ALTERNATIVE = "Rock - Alternative Rock"
    ROCK_PUNK = "Rock - Punk Rock"

    HIP_HOP_TRAP = "Hip Hop - Trap"
    HIP_HOP_BOOM_BAP = "Hip Hop - Boom Bap"
    HIP_HOP_GANGSTA = "Hip Hop - Gangsta Rap"

    ELECTRONIC_HOUSE = "Electronic - House"
    ELECTRONIC_TRANCE = "Electronic - Trance"
    ELECTRONIC_DUBSTEP = "Electronic - Dubstep"

@dataclass
class Album2:
    title: str
    genre: SubGenre # Corrent this time since an album will never be one of the "bare" variants

def search2(title: str, parent_genre: ParentGenre, sub_genre: SubGenre) -> Album2:
    raise

search2("foobar", parent_genre=ParentGenre.ROCK, sub_genre=SubGenre.ELECTRONIC_DUBSTEP) # Type checks, but runtime error

I've reduced it to an MRE. That's not the real code but it's the exact same pattern.

[restive rapids]

why do you need both ParentGenre and SubGenre in search2 if SubGenre already gives the full information?

ultimately i think what you're trying to do cant nicely be handled by a traditional typesystem, since the actual inputs will be dynamic while type systems ensure correctness from static constructions
like, even if you describe the relationships, the gymnastics you'll have to go through at each call site to actually go from arbitrary strings to arguments that will pass the typechecking will be crazy, likely duplicating the permutations of values
(also the "subtyping" of genres includes "multiple inheritance", say, trap <: hip hop & electronic, and assigning a single concrete genre to a whole album is questionable too)

[stiff acorn]

I've updated the code for more clarity. They don't. If you only have SubGenre, you cannot search for the "bare" variants ("all", "rock", etc)

[restive rapids]

it definitely does give the full information for search2. if the only valid calls to it are search2(a_genre, its_subgenre), and you can determine the "parent" of a subgenre, then its just duplication.

[stiff acorn]

But what if I don't know the exact subgenre. I just wanna search all of rock for "foobar".

[restive rapids]

search2's interface does not allow that, since you need to always provide both the parent and sub genres

[stiff acorn]

If I reduced it to just subgenre then:

# but what if I don't know the exact subgenre and want to search all of ELECTRONIC
search2("foobar", sub_genre=SubGenre.ELECTRONIC_DUBSTEP)

(sorry im not trying to be dense here)

[restive rapids]

then you need a separate thing to search by parent genres, yep

ultimately i dont think using just the python typesystem will be good for determining valid calls, both because describing the relationships using it will be hard, and getting the values that conform to those relationships from user inputs

[stiff acorn]

yea, the ultimate disconnect here is just the fact that search allows more variants than the Album. Album can only be "concrete" genres while search is more lax there

you can search all of "ROCK" but an album will never be just "ROCK"

first approach uses a single enum but ends up polluting Album's definition
second approach seperates them (so we have an accurate definition of Album) but invalid combos no longer get type checked in search

[restive rapids]

i dont agree that the second approach has an accurate definition of album
there are definitely albums with tracks in different genres

[stiff acorn]

As I said, this is just an MRE. In my case it is accurate. Album will never be anything other than SubGenre variants

[restive rapids]

i dont think doing this at the type level makes sense, since in reality inputs will be given by the user, not hardcoded and therefore knowing the type of

[stiff acorn]

in my case, subgenre is known and an exhaustive enum

[restive rapids]

yes, but you literally only have 2 sets this way, which is not enough to express the relationship of "electronic_dubstep < electronic, rock_classic < rock" in a system which only supports subtyping between whole sets

[stiff acorn]

I had a feeling that type system just doesn't support this but I wanted to ask anyway. I'm also open to any other approach that might be type safe if possible

[restive rapids]

how is the user giving the inputs to those searching functions?

[stiff acorn]

The search function I depicted. Currently I'm just using the first approach. a single enum that encompasses it all. So my "Album" definition is wider than reality but search works nicely

[restive rapids]

i might disappoint you, but it becomes "wider than reality" as soon as you use a str

[stiff acorn]

it's not a str though?

i mean i guess StrEnum is str but still

[restive rapids]

your album definition says the title is a str

[stiff acorn]

genre

[restive rapids]

.. what? you have

@dataclass
class Album:
    title: str

in both cases

there are definitely values of type str which are not valid album titles
you're trying to impose too strict requirements in the typesystem. the more you do of that, the harder it will be to get a call-site that can actually do anything with them.

[stiff acorn]

I think we aren't talking about the same thing. title is str in both search and Album. It's not even part of this problem. Only the genre is the relevant part here. Notice the genre field is typed differently in each

[restive rapids]

maybe for your situation search should just take a Parent | Sub, or Sub should inherit from Parent

but if i was actually trying to express the correct relationship between genres in the typesystem while being able to sort them into larger "buckets", just subtyping or unions wouldnt do, or if they would - it would be through combinatorial explosion and repeating a lot of stuff
maybe the "larger buckets" is the problem. just subtyping (& multiple inheritance) alone would actually work for describing, well, subgenres, and subgenres of subgenres...

[stiff acorn]

yea, overloads might work but it'll be impractical

[vapid matrix]

I think this is type hinting question - if not I apologize. I have added a "trace" and a "verbose" levels to the stdlib logging module - how do I get Pylance in vscode to recognize those as valid and not flag them?

(I don't really want to put # typing ignore comment everywhere)

[restive rapids]

you'll have to change the typeshed stubs (https://github.com/python/typeshed/blob/main/stdlib/logging/__init__.pyi#L114), each typechecker gets a copy of them
modifying stdlib is pretty weird though
maybe just subtype Logger instead?

[vapid matrix]

Its a supported API in logger.

[restive rapids]

so what code provokes the "not valid" from pylance?

[vapid matrix]

I just looked at the helper function (I got it from stackoverflow) and ... it calls setattr on logging

So, no its NOT part of the std API.

<Sigh> --- I'll try something else.

https://stackoverflow.com/a/35804945 -- this is code / function I used to setup the extra levels.

log.trace(f"{uptime=}") <-- in vscode, pylance Cannot access attribute "trace" for class "Logger"
  Attribute "trace" is unknown

[restive rapids]

yeah that seems about right, even if you didnt use setattr, all the methods the typechecker should know about have to be defined in the class declaration

[vapid matrix]

I may cut this "helper" fucntion down to just the official API parts and call log() with a textual level

Cut out the nonsense.

[I hate the stdlib logging module but I didn't write it so...]

[restive rapids]

you could subclass Logger

[vapid matrix]

add my own methods there?

Can you link to an example?

[restive rapids]

yeah, then you just need to make the typechecker know that log is of that logger type

[restive rapids]

import logging

TRACE = 9 
logging.addLevelName(TRACE, "TRACE")

class NewLogger(logging.Logger):
    def trace(self, message: str) -> None:
        if self.isEnabledFor(TRACE):
            self._log(TRACE, message, ())

logging.setLoggerClass(NewLogger)
logging.basicConfig(level=TRACE)

logger = logging.getLogger(__name__)
assert isinstance(logger, NewLogger) # to make the typechecker know that its of our new type

logger.trace("uptime: -1s")

something like this

[vapid matrix]

That is much appreciated - let me play with that.

[trim tangle]

this feels like it's more work than having a helper function tbh...

[trim tangle]

I would just do logger.log(TRACE, "other arguments")

[vapid matrix]

Might be - I'm going to try both I think.

But I apprecaite the help

[frigid stone]

if i put that will it be pygame?

imput pygame

[magic wagon]

Can someone help us fix the script

[hasty phoenix]

If I have a generic function def something() that can return anything. It can be annotated with -> Any and its up to the usage of it to reduce it to a specific type. E.g. in one usage its known that it returns int. Using v: int = something() works fine.

Let the function return -> int | str | None. Doing the same type reduction gives type errors (at least with pylance): v: int = something(). Is there a simple way to do the same as with Any for polytyped functions without needing to test all types?

[trim tangle]

# actually perfom the check, can be removed with the -O flag
v = something()
assert isinstance(v, int)

# silence the error
v: int = something()  # type: ignore[specify-error-type-for-your-type-checker]

# typing.cast (may have a relatively high runtime cost), similar effect to `type: ignore`
v = typing.cast(int, something())

Can you tell more about the function? Maybe it is possible to avoid this?

[hasty phoenix]

Hmm. These are all known methods

Perhaps the easiest is to think of this as a KV store dict[str, int | str | None]. It's known by design which key values lead to what type of data.

I have been contemplating a something_as_int(...)

[autumn sluice]

so, you know that the age key, for example would always return an int? and name -> str?

you may need a TypedDict: https://peps.python.org/pep-0589/

[hasty phoenix]

Yes, that's known and I user it in simpler cases

[trim tangle]

(btw, as the pep says, it's better to link the docs: https://typing.python.org/en/latest/spec/, as PEPs are mostly historic documents)

[trim tangle]

Why doesn't TypedDict work for you?

[hasty phoenix]

Because the dataset of name to type mapping would be huge. Like really huge. It's better to do that selection or reduction when the KV is used in the code

[trim tangle]

But how do you know whether a key corresponds to an integer, string, or None? (and why are there keys mapping to None?)

[hasty phoenix]

I'm trying to find the balance between having the untyped code (which works fine) and the full and proper type annotations. IMHO one's in a wrong place if type hinting takes up more space than the functional code.

[hasty phoenix]

Because of an external API that determines this. So the programmer know when using a key what type it is. That's why this works fine with Any.

[trim tangle]

If the keys are dynamic, I can suggest two solutions:

1. use accessor functions for specific types, as you said above (configparser does this: https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.getint)

2. define keys as first-class entities ```py
   class Key[T]: # TODO

KEY_FOO = Key("foo", int) # inferred as Key[int]
KEY_BAR = Key("bar", str) # inferred as Key[str]
KEY_BAZ = Key("bar", str, optional=True) # inferred as Key[str | None]
``` this is what aiohttp does for storing state on an application instance (it uses the name AppKey for this)

3. or maybe it's not worth the effort and you can use Any

[hasty phoenix]

I like no 2. I'll take a look into that. Thanks.

[trim tangle]

I think adding extra code just for static analysis and/or runtime validation is helpful in areas that are harder to test (like external APIs). If all the settings are defined in some sort of struct, it's easy to compare that struct with the API docs instead of hunting for every key across the program. And if the API changes, you know what's affected

[autumn sluice]

Another suggestion would be to use a parse/validation library like pydantic or msgspec and define a schema with only the fields that interest you.

[hasty phoenix]

The beauty with accessor function, like no1, is that the runtime type check can be done in that accessor function, not sprinkled everywhere in the code

[trim tangle]

the keys can also do runtime checking

[hasty phoenix]

yes

[trim tangle]

If you have accessor functions but keep keys as string literals, you can still do getstring("foo") in one place and getint("foo") in another

if you have dozens of keys, I'm probably thinking of a thing that's different from what you have

[hasty phoenix]

Any ideas why pylance is giving error on this code:

from typing import TypeVar
T = TypeVar("T")

class A:
    data: dict = {}
    def __call__(self, obj: T) -> T:
        if isinstance(obj, str):
            for k, v in self.data.items():
                if isinstance(k, str) and k in obj:  # A)
                    obj = obj.replace(k, v)  # B)
        return obj

It returns:

A) Operator "in" not supported for types "str" and "str* | T@__call__"
   Operator "in" not supported for types "str" and "object*"
A) Type "str | Unknown" is not assignable to declared type "T@__call__"
   Type "str | Unknown" is not assignable to type "T@__call__"
B) Cannot access attribute "replace" for class "object*"
   Attribute "replace" is unknown

It's the k in obj test (at A) which is failing, but both k and obj have been type tested and are str.

[trim tangle]

i think it is being silly

[hasty phoenix]

ok, bug report to pyright then

[restive rapids]

its because of the obj = obj.replace(...), you're saying it should be T -> T, so, say, for a subclass of str the isinstance still passes, but replace is not guaranteed to return the same subclass, just an arbitrary str

[trim tangle]

oh yeah

[trim tangle]

Imagine that you have py foo: Literal["banana"] = "banana" the function signature says that A()(foo) will have the return type of Literal["banana"] but it may not be the case

[hasty phoenix]

So str.replace() might not return str?

[trim tangle]

another example would be binding T to a subclass of str, like a StrEnum value

[restive rapids]

(T <: str).replace() might not return T

[restive rapids]

yeah thats a good example
im not sure how to handle stuff like that, almost like you'd want a "infer not-so-concrete type variable"

[hasty phoenix]

Hmm. I would expect that ``(T <: str).replace()would return(T <: str)` in like for like

[trim tangle]

"banana" is a Literal["banana"], but "banana".replace("a", "n") is not a Literal["banana"]

[trim tangle]

If you have a fixed number of types, you can use a "type variable with constrains"

T = TypeVar("T", int, str, int | None, str | None)
``` then `T` is only allowed to resolve to those 4 types

[hasty phoenix]

I'm not sure I follow the logic and argument with Literal here. k in my code isn't specified, but its known runtime that it's an str.

[restive rapids]

no, its an arbitrary subclass of str, not exactly a str
though whats interesting is that if you do type(obj) is str it doesnt work either so runtime-nonexistant types like Literal might also be considered
pretty ugly case tbh

i'd uh do an overload str -> str and [T] T -> T maybe? if you really need it to be generic. the str case should catch all the subclasses too
or just check for str at call site, since it doesnt do anything in other cases
whats your actual usecase here?

[trim tangle]

The signature of __call__ says that it returns the same type as the type of the argument. Consider this call:

a = A()
a.data = {"a": ""}

banana: Literal["banana"] = "banana"
replaced = a(banana)
``` `replaced` is supposed to be the same type as the argument, which is `Literal["banana"]`

[hasty phoenix]

Interesting. There are non-runtime type annotations that only exists in the type checker

[trim tangle]

!e
Or with a StrEnum: ```py
from enum import StrEnum

class Bool(StrEnum):
YES = "yes"
NO = "no"

yeah = Bool.YES.replace("s", "ah")
print(repr(yeah), type(yeah))

[rough sluice]

:white_check_mark: Your 3.13 eval job has completed with return code 0.

'yeah' <class 'str'>

[trim tangle]

Bool.replace will always return a plain str that's not a Bool

[trim tangle]

Yes, a Literal is just a string at runtime. Just like a TypedDict is a plain dict at runtime

[hasty phoenix]

So how to I fix my function?

The runtime behavior should be fairly obvious I hope

[trim tangle]

Will T only have a fixed number of types? Like str, int or float

[hasty phoenix]

No, it can be anything

[trim tangle]

then this is probably the closest solution

@overload
def __call__(self, arg: str, /) -> str: ...
@overload
def __call__(self, arg: T, /) -> T: ...

def __call__(self, arg: Any, /) -> Any: # actual implementation

[hasty phoenix]

Sorry I missed their response further up. I'm well familiar with overload.

Thank you both

[trim tangle]

it is still slightly incorrect though...

@overload
def f(x: int) -> int: ...
@overload
def f[T](x: T) -> T: ...
def f(x: object) -> object:
    "todo: implementation"


def g[X](y: X) -> X:
    z = f(y)
    reveal_type(z)  # z: X
    return z  # no errors
``` pyright only considers the possibility of the second overload when the argument is a typevar, apparently

[hasty phoenix]

This works too

obj = cast("T", obj.replace(k, v))

[restive rapids]

"works" as in "shuts up the typechecker". its unsound. you could cast it to anything.

[hasty phoenix]

or more precisely, it shuts up pyright

heh, yes

But I'm thinking that if the remedy for this proper required 10+ lines of code just for being able to formally specify the type hinting correctly, then the type hinting is excessive.

It is a balance

Many opt out and use Any precisely because of that, and I think that's equally "I don't care about type hints" as cast is.

[spark yoke]

running into this scenario where I have something like this in pydantic models:

class Foo(pydantic.BaseModel):
    # ...

class Bar(pydantic.BaseModel):
    # ...
    some_foo: Foo | None

then in another part of the code I'll have something constructing it like:

def make_foo() -> Foo | None:
   # ...

my_foo = make_foo()
my_bar = Bar(some_foo=my_foo)

Then mypy will review this and say: error: Argument "some_foo" to "Bar" has incompatible type "type[Foo] | None"; expected "Foo | None"

What exactly is going on with that type[]?

[restive rapids]

type[Foo] is the type of the type object Foo itself (like, x: type[Foo] = Foo), you are probably forgetting to construct it

[spark yoke]

as in, the make_foo() is not correctly producing an instance of Foo?

[restive rapids]

the make_foo example in particular is nonsense because make_foo is hinted to return Foo | None, not type[Foo] | None, yet the diagnostic says otherwise. if the problem was in make_foo, it's declaration should error from not matching the return hint, not the usage

[spark yoke]

ah. thanks. it just clicked. here is a more complete picture of the mistaken piece of code: def make_foo[T](value, some_type: T) -> T | None:. Where the desired class would be passed as the second argument. To correct this: def make_foo(value, some_type: type[T]) -> T | None:

[restive rapids]

how exactly do you plan on this function to work? it'd make sense to take a Callable[some_params, T] instead atleast, because type has no information about the constructor

[spark yoke]

it has a pretty specific purpose of converting input text to various string enums with standard edge cases. I have simplified/generalized

in which case, the class itself provides the constructor. basically

[golden barn]

Hey! Not sure where to ask this: Any suggestions on good books about parsers/lexers?

[trim tangle]

this is definitely the wrong channel, this channel is about Python's type annotation system

#algos-and-data-structs is probably the best place to ask

[golden barn]

thanks, I'll jump there

[hasty phoenix]

T = TypeVar("T")
class A:
    data: dict[str, str] = {}
    def fn(self, obj: T) -> T:
        if obj in self.data:
            a = self.data[obj]  # Why is this flagged?
            # Isn't it inferred that since obj is in self.data,
            # it must be a str and thus valid as a key?
        return obj

It's pylance/pyright that complains "Argument of type "object*" cannot be assigned to parameter "key" of type "str" in function "getitem""

[feral wharf]

T is not str technically

Unless you do data: dict[T, str] = {}

[hasty phoenix]

I know, but since obj in self.data, T collapses to an str doesn't it?

But evidently inference doesn't work like that here

[feral wharf]

https://github.com/microsoft/pyright/issues/10381#issuecomment-2833511015 looks like pyright doesn't like that because it's just a comparison

[hasty phoenix]

Interesting. Thanks.

Functionally this code is working and is safe, its "just" the typing formalism that doesn't allow it

[feral wharf]

Ya

Btw, from 3.12, you can define a typevar like this: class A[T]:

[hasty phoenix]

Very informative thread. Thank you. Python object subtleties meeting type correctness is evidently not straight forward.

As for my question my take away is: obj can be __eq__-like with str in obj in self.data. This doesn't guarantee that objis a str.

Learned something new today 💪

[trim tangle]

In this case it would be on the method: ```py
def fn[T](self, obj: T) -> T:

class A[T] replaces class A(Generic[T]), i.e. a generic class

[feral wharf]

Oh yeah my bad

[pallid holly]

Maybe y'all can help me figure out a good type hint here. Although this is not the simplest example that can demonstrate the issue I'm facing, it's more in line with the actual thing I'm trying to do, which is essentially the logic for building a JSON object to be sent to some external API:

from collections.abc import MutableMapping, MutableSequence

JsonT = (
    MutableMapping[str, "JsonT"] | MutableSequence["JsonT"] | int | float | str | None
)

def foo(v: list[int]) -> JsonT:
    x: JsonT = {
        "foo": 10,
        "bar": { "baz": 100 },
    }
    x["qux"] = {"quux": v} # type issue here
    return x

This results in the following error:

Diagnostics:
Argument of type "dict[str, list[int]]" cannot be assigned to parameter "value" of type "JsonT" in function "__setitem__"
  Type "dict[str, list[int]]" is not assignable to type "JsonT"
    "dict[str, list[int]]" is not assignable to "MutableMapping[str, JsonT]"
      Type parameter "_VT@MutableMapping" is invariant, but "list[int]" is not the same as "JsonT"
    "dict[str, list[int]]" is not assignable to "MutableSequence[JsonT]"
    "dict[str, list[int]]" is not assignable to "int"
    "dict[str, list[int]]" is not assignable to "float"
    "dict[str, list[int]]" is not assignable to "str"
    "dict[str, list[int]]" is not assignable to "None" [reportArgumentType]

As I understand the error, it happens because broadening the container type from dict[str, list[int]] to dict[str, list[JsonT]] would cause some form of type safety issue, which is fair. My actual question is, what would be the "best"/"correct" way of type hinting this? So far I've found various workarounds, e.g. making x a dict[str, JsonT], casting v to JsonT, casting the entire argument to __setitem__ to JsonT, just # type: ignore it, etc. Anyone with more knowledge and/or strong opinions who can help me figure out what's best?

[trim tangle]

Does the mapping/sequence have to be mutable?

[pallid holly]

Presumably it has to be mutable if I'm changing it in this function 🤔

[trim tangle]

This is perfectly legal for example: ```py
def make_sequence() -> Sequence[int]:
numbers: list[int] = [1, 2, 3]
numbers.append(4)
return numbers

Sequence doesn't mean that it's an immutable object

it's just that if you have something that you only know to be a Sequence, you cannot mutate it

list[int] is not a MutableSequence[JsonT], because you are allowed to e.g. append a str to a MutableSequence[JsonT]. This is why the type checker is complaining.
But list[int] is a Sequence[JsonT]

[pallid holly]

Yeah, but if I just change JsonT to use Mapping and Sequence I'm now getting this:

Diagnostics:
Argument of type "dict[str, Sequence[int]]" cannot be assigned to parameter "value" of type "int | dict[str, int]" in function "__setitem__"
  Type "dict[str, Sequence[int]]" is not assignable to type "int | dict[str, int]"
    "dict[str, Sequence[int]]" is not assignable to "int"
    "dict[str, Sequence[int]]" is not assignable to "dict[str, int]"
      Type parameter "_VT@dict" is invariant, but "Sequence[int]" is not the same as "int"
      Consider switching from "dict" to "Mapping" which is covariant in the value type [reportArgumentType]

[trim tangle]

Can you show the new code?

[pallid holly]

Which I assume is because I'm using a literal to construct the initial thing, and I can't make the type checker ignore that it knows it's actually a dict.

from collections.abc import Mapping, Sequence

JsonT = (
    Mapping[str, "JsonT"] | Sequence["JsonT"] | int | float | str | None
)

def foo(v: Sequence[int]) -> JsonT:
    x: JsonT = {
        "foo": 10,
        "bar": { "baz": 100 },
    }
    x["qux"] = {"quux": v}
    return x

Edit: Oof, double space

[trim tangle]

This seems to work in this particular case ```py
def foo(v: Sequence[int]) -> JsonT:
x: dict[str, JsonT] = {
"foo": 10,
"bar": { "baz": 100 },
}
x["qux"] = {"quux": v}
return x

[pallid holly]

Yeah, I did notice that too. But is that the "correct" solution? 😛

[vapid viper]

Hi

Does anyone have the real client.command for auto react discord python?

auto_react_users = {}

    @client.commmand()
    async def ar(ctx, user: discord.Member, emoji: str):
        auto_react_users[user.id] = emoji
        await ctx.send( f"Auto react with {emoji} enabled for {user.mention}.")

[trim tangle]

Are you sure this is related to type annotations? Did you mean to ask in #discord-bots ?

[vapid viper]

sorry and yes

too many channels cant understand or see anyt

[trim tangle]

If you don't know where to ask, see #❓｜how-to-get-help and make a post in #1035199133436354600

[vapid viper]

Ok

I posted in discord bots

[trim tangle]

pyright has an extremely unhelpful feature where it will infer a more precise type even if you have an annotation. For example:

bar: Sequence[int] = [1, 2, 3]
reveal_type(bar)  # Type of "bar" is "list[int]"
bar.append(4)  # allowed... because pyright knows this is a list, even though you specifically said that it's a sequence
``` so what actually happens in your function is that pyright disregards the annotation completely and infers its own type to it: ```py
x: JsonT = {
    "foo": 10,
    "bar": { "baz": 100 },
}
reveal_type(x)  # Type of "x" is "dict[str, int | dict[str, int]]"

This type doesn't include list[int] in the value type, so it flags an error. If x was a JsonT, the error would be different: x could be a list, int, float, str, or None.

I'm assuming that you didn't put quux in the literal because you want to assign it conditionally. You could do this, for example:

def foo(v: Sequence[int]) -> JsonT:
    x = {
        "foo": 10,
        "bar": { "baz": 100 },
    }
    if random.random() > 0.5:
        x = {**x, "quux": v}
    return x

[pallid holly]

Yes, that's correct. I left out the condition for brevity 🙂

[trim tangle]

pyright allows you to reassign a variable with a different type, so this is what happens

[pallid holly]

Right, using x = {**x, "quux": v} also gets rid of the problem, but presumably that's less efficient D:

I mean, for this example it's probably negligible, but eventually x could grow enough that it matters at least somewhat. Dunno where that point is though.

[trim tangle]

I think using : dict[str, JsonT] as an annotation is a perfectly acceptable solution

sometimes type checkers are just annoying, especially for local reasoning within a function like the above

[pallid holly]

Mm. Fair enough.

Well, I've spent more than enough time thinking about this, so... I'll settle for being given permission to use that workaround 😄 Thanks for the help!

[jaunty lagoon]

it's been 12 years since i dove deep into a python topic, and now i'm trying to type hint metaclasses with generics and enums. not looking for help (at the moment), just wanted to drop in and say i'm having a blast 🙂

not entirely sure i can do what i want without actual generic metaclasses, but such is life. there are always less elegant ways to do things and still be correct.

mostly just my way of dropping in to say hello to folks who actually care about this stuff.

[foggy garnet]

do people still use enums extensively? I basically always use a Literal

[pastel egret]

I use them quite a lot. In particular, being able to define a value that matches what a user should type, then having a more technical attribute name for code to use, then just calling the enum to parse data is quite handy. You can also define properties on them, which is a nice way to expess conversions between values. On one of mine I have __invert__ defined even to expess a particular swap.

[fierce ridge]

I much prefer enum over literal when possible

The only time I use Literal is when I'm interfacing with an external system and I don't want the Enum overhead, or if I'm type-hinting existing "magic string" code, or if I want to support "magic strings" as a convenience input (but in that case I'll usually just write str and raise at runtime)

[autumn sluice]

I like enums, so I can match over values and use assert_never to tell me when somebody adds a new enum value that needs to be handled in different places.

[foggy garnet]

you can do that with literals too, no?

[autumn sluice]

come to think of it. never tried. 🙂 I reach for an enum whenever I get 3+ options

[foggy garnet]

typing.get_args(Literal['a','b']) returns ('a','b'), so you can handle the values in a match statement easily.

i have also seen this pattern:

MyLiteral = Literal['apples', 'oranges']
MY_LITERAL: Final = ('apples', 'oranges')

this kind of thing is much nicer in typescript

maybe my issues with enums stem from my aversion to OOP, when functions + types are sitting right there 😆

[fierce ridge]

think of enums as more like ADTs than classes

[foggy garnet]

you mean a sum type

i think the problem is that enums are classes

maybe the API has changed, but back when I was using them, given some enum like

class Color(Enum):
    RED = 1
    GREEN = 2
    BLUE = 3

iterating over the names (or maybe it was the values?) required accessing dunder methods

this just feels like python doing OOP contortions to compensate for a lack of simple immutable mappings

also, the requirement that the enum names be valid python identifiers (i.e., no whitespace) is clearly a limitation that only exists so that you can do Color.RED

[oblique urchin]

What limitation? ```In [6]: X = enum.Enum("X", ["a b", "c d"])

In [7]: X
Out[7]: <enum 'X'>

In [8]: list(X)
Out[8]: [<X.a b: 1>, <X.c d: 2>]

[foggy garnet]

oh cool, although this would break any API that expects to invoke enum.<field>

[oblique urchin]

you can still use getattr(). It's impractical in a number of ways but the enum abstraction allows it

[foggy garnet]

"dot notation for some contents, getattr for others" isn't exactly a scintillating design

[oblique urchin]

you can use getattr() for all of them

[foggy garnet]

of course

[oblique urchin]

or subscripting

Out[9]: <X.a b: 1>

[foggy garnet]

yeah, still not great to have a non-uniform API here

but enums are probably very old?

[oblique urchin]

not that old, added during the Python 3 series

[oblique urchin]

not sure what else you'd expect? The only way to avoid this would be to not use attribute access for enums, which would just make everyone's life harder for a rare edge case

[foggy garnet]

i'd expect the API for accessing members of a collection to not depend on the particular values of the contents of that collection

so no attribute access

the __getitem__ syntax is uniform

[pliant vapor]

is there a way to type hint a decorator that takes a function and adds a keyword argument to it

[trim tangle]

nope

[pliant vapor]

sad

[trim tangle]

yep

[pliant vapor]

what if it just added a positional argument at the end

i would've assumed typing.Concatenate but that has to end in a paramspec or ...

so i don't think Concatenate[P, bool] works

[restive rapids]

I thought it does

[pliant vapor]

if it does that would be nice

i haven't tried it I've already turned my laptop off for the night

[trim tangle]

IIRC you can only add to the front or something like that

[pliant vapor]

!d typing.Concatenate

[rough sluice]

typing.Concatenate

typing.Concatenate```
Special form for annotating higher-order functions.

`Concatenate` can be used in conjunction with [Callable](https://docs.python.org/3/library/typing.html#annotating-callables) and [`ParamSpec`](https://docs.python.org/3/library/typing.html#typing.ParamSpec) to annotate a higher-order callable which adds, removes, or transforms parameters of another callable. Usage is in the form `Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable]`. `Concatenate` is currently only valid when used as the first argument to a [Callable](https://docs.python.org/3/library/typing.html#annotating-callables). The last parameter to `Concatenate` must be a [`ParamSpec`](https://docs.python.org/3/library/typing.html#typing.ParamSpec) or ellipsis (`...`).

[restive rapids]

why
i can understand not adding keyword params to generic paramspecs because they can overlap but positional ones..

[pliant vapor]

i guess at this stage in typing Concatenate and ParamSpec are more built with like lambda calculus style functions in mind

where the main thing is just arity and types

vs the more colorful python functions with keywords and all that

[pliant vapor]

oh wait also

to actually make such a function you'd have to do something like func(*args, new_param) i guess

so i guess you can't do that naturally in normal python anyways

[restive rapids]

well, to do Concatenate[Arg1Type, ParamSpecVariable] you already do func(x, *args, **kwargs), how is that any different? just different order

[pliant vapor]

i just mean like

wait what

[restive rapids]

you mean the signature? ah, yeah
but you can do old_args = args[:-1], new_arg = args[-1] or *old_args, new_arg = args

[pliant vapor]

!e

def foo(*args, x): return x
print(foo(1, 2))

[rough sluice]

:x: Your 3.13 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 2, in <module>
003 |     print(foo(1, 2))
004 |           ~~~^^^^^^
005 | TypeError: foo() missing 1 required keyword-only argument: 'x'

[pliant vapor]

ah it's a keyword

yeah

[pliant vapor]

true

[restive rapids]

i dislike how this is inconsistent with sequence assignment targets
*args, x = it works just fine

[pliant vapor]

yeah, an unfortunate side effect of notating keywords as "after the *"

[fierce ridge]

Yes sorry, a sum type specifically

[fierce ridge]

the whole point is that it's syntactic sugar to make enums feel more "static"

they're not members of a collection, or you're not supposed to think of them that way

imo you're "supposed to" think of them as enumerated sum types

the quasi-requirement for syntactically-valid names helps enforce this convention

the same is true for literally any other class -- yes, a class is a dict in a trenchcoat, but you'd never argue that class instance member access should use [] instead of .

[trim tangle]

I think it's also trying to emulate a built-in indented enum Color: block, which should be a familiar concept for Java/C#/TypeScript/C/C++ programmers

[rare scarab]

Don't you dare bring up enums in Typescript

[cyan pond]

What do you think about circular imports in Python? For example, when using relationships in SQLAlchemy? I always use TYPE_CHECKING to fix

[foggy garnet]

this has the most explanatory value to me

they're not members of a collection, or you're not supposed to think of them that way
that runs contrary to very normal use cases like iterating over the variants, and iterating over the names of those variants

and there would be no need for the variants to even have names if the python enum wasn't catering to legacy design choices and was instead a first-class type

[jolly cipher]

this does look bad and that's not how i expected bidirectional inference to work, but on the other hand is that necessarily bad given that what you're getting here is basically closer to truth, and the liskov substitution principle still applies--the problem only appears within the same code range, not across functions with disjoint scopes?

i think this is only problematic if you have a line where you assign e.g. a list and want to prevent mutations of it in the lines to follow by enforcing an immutable sequence type. but in that case you could just use a tuple, right? since it's the list display used here that is the one to blame for the narrowing

how is that problematic otherwise

[jolly cipher]

i see that the "qux" setitem operation (after the line with dict display assignment) is flagged because of that behavior.
however, we also said that that x dictionary is a JsonT type which covers much more runtime objects than just mutable dictionaries, so mutating x like it's just a dictionary is not something you want to do (despite that it seems like it's killing the main purpose of type checking which is preventing runtime errors)

which is not consistent with the behavior of ```py
bar: Sequence[int] = [1, 2, 3]
reveal_type(bar)
bar.append(4)

...yeah that's annoying

but i think you can avoid these problems sometimes by approaching the problem from a different perspective like in that conditional reassignment example

[jolly cipher]

i think i know how.
you may end up passing that somewhere where a list is expected and since you wanted it to be immutable type, it can get mutated somewhere else. but i think that still boils down to just using a tuple.

[jolly cipher]

drawing very theoretical conclusion i think it's a trade-off of deriving decisions from what the code currently is and what is expected that it might become

now i'm seeing that enforcable types are more logical and easier to follow. so yes, it would be nice if pyright supported that instead of "knowing better"

[gray gorge]

do you know if it possible to specify tuple type like tuple[int, ...] but where first argument is int | None and rest is strictly int, amount of elements is dynamic, that's why ... is needed, but tuple[int | None, ...] will produce wrong type and you can't write tuple[int | None, int, ...]?

[restive rapids]

tuple[int | None, *tuple[int, ...]] ? mind that this requires the first value to be present though, so you might want | tuple[()] it to also allow empty ones

[desert delta]

very cool, I had no idea unpacking was allowed in an annotation

[gray gorge]

looks like it

[trim tangle]

It is unhelpful for debugging. You say x: Sequence[int] but then pyright thinks that x is a list, which can lead to confusion. (as exemplified above)

It can also lead to worse error messages. If the right hand side infers a complicated generic type, and you're getting an error when using it, you might get a terrible error message that doesn't even fit into the pop-up window. (I can't make an example right now, but maybe you've seen such occasions). You can't just do foo: SimpleInterface = complex_generic_type, you have to invent some workaround now

It also makes it hard to work around inference bugs. For example:

def foo() -> None:
    secret: int | None = 42

    def mischief() -> None:
        nonlocal secret
        secret = None

    reveal_type(secret)  # Literal[42]
    mischief()
    reveal_type(secret)  # still Literal[42]...

This is a sneaky one: you're making a library that exposes this class with foo and bar attributes being collections. ```py
class Foo:
def init(self) -> None:
collection: Collection[int] = [1, 2, 3]
self.foo = collection
self.bar = collection

in user code:

reveal_type(Foo().foo)
reveal_type(Foo().bar)
``` mypy users will see Collection[int] and Collection[int]. But pyright users will see list[int] and list[int].

[jaunty lagoon]

I ran into this in one of my projects recently, and I ended up creating a separate types module that all the other modules import and inherit from where appropriate. It meant creating a couple extra fundamental types that I wouldn’t otherwise need, so it’s more work than TYPE_CHECKING, but it feels better 😂

[wicked scarab]

TL;DR: Where's the appropriate place to put python stub files (.pyi)?

I’m building a Python library with a C extension. Right now, the C file is inside the package. Is this a good structure? If so, should the stub file (.pyi) also be inside the same package? If not, where should I place the C file and the stub file?

Options:
a. Put the stub file inside the package alongside the C file.
b. Put the stub file in a separate package but keep the C file where it is.
c. Neither

[jolly cipher]

yep, this is very very bad

[jolly cipher]

this sucks as well

[rare scarab]

Put the stubs at the import path, usually next to the dll

[loud zodiac]

So is there a way to typehint it without forcing the cls parameter of observable_dataclass being a subclass of ObservableDataclass? https://paste.pythondiscord.com/WE3A

[rare scarab]

Would dataclass_transform be able to do this?

!d typing.dataclass_transform

[rough sluice]

typing.dataclass\_transform

@typing.dataclass_transform(*, eq_default=True, order_default=False, kw_only_default=False, frozen_default=False, field_specifiers=(), **kwargs)```
Decorator to mark an object as providing [`dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass)-like behavior.

`dataclass_transform` may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of `@dataclass_transform()` tells a static type checker that the decorated object performs runtime “magic” that transforms a class in a similar way to [`@dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass).

Example usage with a decorator function...

[loud zodiac]

no

[jaunty lagoon]

I don’t know about that case, but thanks for the pointer. I hadn’t seen this yet, and it looks like it’ll help with a project I’m working on.

[loud zodiac]

the TypeVar was enough to type the decorator's return as the same as the input type

[rare scarab]

Its probably best to use a base class or meta class

[loud zodiac]

I was trying to avoid metaclasses

[jolly cipher]

what do you guys think

https://discuss.python.org/t/support-suppressing-generator-context-managers/103615
https://discuss.python.org/t/special-semantics-for-generator-context-managers/103616

[jaunty lagoon]

And now that I’ve looked up dataclass_transform and tried a few things with it, I’m convinced that I don’t understand how to use field_specifiers 😂

No matter what I tried, I couldn’t get mypy to recognize anything other than dataclasses.field as a valid field

[rare scarab]

It needs to be directly on the decorator or base class

[jaunty lagoon]

I’m not sure I follow, because I thought those were the only places to put it already

I did just dig through the PEP and I’ll just work outward from their example code

[rare scarab]

Share your code?

[jaunty lagoon]

i didn't keep any of my code, i was just playing around with it. and i can't even manage to get the example in the PEP to work. so i googled for an example and google's AI gave me something that works-ish, but also not: https://paste.pythondiscord.com/KAJA

for some reason, i really can't find any simple, complete examples of something that just works, to build on

ultimately what i'm trying to make is this:

class GIF(Structure):
    tag: bytes = FixedBytes(b'GIF')
    version: bytes = Bytes(size=3)
    width: int = Integer(size=2, endianness='<')
    height: int = Integer(size=2, endianness='<')

but for the life of me, i can't get python and mypy to agree on anything

at the end of the day, i'm okay dropping the type hints and relying on explicit __init__ methods. it'd just be nice if this can work

[rare scarab]

Don't annotate it with Field

x: int = Field() should be fine

[jaunty lagoon]

That’s Mr hope, but mypy keeps telling me there’s a type mismatch. It’s acting exactly like it does without the field specifier. The autocomplete stuff in VS Code does the right thing, though. 🤷‍♀️

[rare scarab]

I don't think you can use a generic

[jaunty lagoon]

I don’t think I got it to work without the generic either. And would that still be the case if the generic is in a base class, but the subclass used as the actual field isn’t generic?

Namely, my actual library is like this at the moment:

class Integer(Field[int]): …

[oblique urchin]

The conformance tests https://github.com/python/typing/blob/main/conformance/tests/dataclasses_transform_class.py pass a field class and a function returning Any, but only use the function. It does seem like there's not really a convenient way to use the field class

[jaunty lagoon]

Thanks, I hadn’t seen that. So yeah, that matches how dataclasses themselves do it. There’s a class and a function that used to create instances of that class. Probably just to get the Any return type that you can’t get from the class alone.

I’ll probably end up with something like

age: int = Integer(size=2).field()

It’s not as pretty as I’d like, but I should be able to make it work

[rare scarab]

Does call chaining work for this?

[jaunty lagoon]

like String.size(20).padding('\x00').encoding('ascii')? i considered something like that, but i really dislike the look of it in general, and i'd still need something for fields that don't need any arguments.

another option would be to just have Integer, String and company be functions instead of classes. or perhaps use a decorator on the classes that turns them into functions for the sake of creating the fields, then convert them back to class instance once the data class is created. all depends on how much dancing i want to do in order to have a little bit nicer of an IDE experience

[rare scarab]

Considered using annotations?

@foo
class Bar:
  baz: Annotated[int, Field()] = DEFAULT

[jaunty lagoon]

yeah, i considered that too. that's probably the most "correct" way of doing it, from the perspective of IDE tooling. but i expect users to have to add a bunch of fields, so i'm also trying to keep it cleaner. i'm not ruling anything out, though!

mostly, it's all just so weird. the only thing that actually seems to be happening is that a function is typed to return Any, which is then cast to the appropriate type when assigning the field. And doing that doesn't even require field_specifiers. so yeah, i just don't get the point. 🤷‍♀️ and i'm willing to leave it at that for now.

[rare scarab]

Make some type aliases

type Biz = Annotated[int, Field()]

@foo
class Bar:
  baz: Biz = BAP

[jaunty lagoon]

but then i can't do any customization of the field when adding it to the class. the annotation is locked when the alias is created.

i appreciate the ideas, i'll think this all over for a while and see where i land. i have much more useful aspects of this library to work on 🙂

[trim tangle]

You can annotate them further: ```py
type Uint8 = Annotated[int, Unsigned(8)]

class Foo:
bar: Annotated[Uint8, Except(255)]
baz: Annotated[Uint8, NonZero()] = 69

!e
multiple layers of Annotated are collapsed

from typing import Annotated

Foo = Annotated[int, "foo"]

print(Annotated[Foo, "bar"])

[rough sluice]

:white_check_mark: Your 3.13 eval job has completed with return code 0.

typing.Annotated[int, 'foo', 'bar']

[trim tangle]

Not sure how to make it work with type statements

because if Foo is written as a proper type alias (type Foo = ...), then it doesn't collapse 🥴

!e

from typing import Annotated

Foo = Annotated[int, "foo"]
Bar = Annotated[Foo, "bar"]
Baz = Annotated[Bar, "baz"]
print(Baz)

type Apple = Annotated[int, "apple"]
type Banana = Annotated[Apple, "banana"]
type Cherry = Annotated[Banana, "cherry"]
print(Cherry)
print(Cherry.evaluate_value())

[rough sluice]

:white_check_mark: Your 3.14 pre-release eval job has completed with return code 0.

001 | typing.Annotated[int, 'foo', 'bar', 'baz']
002 | Cherry
003 | typing.Annotated[Banana, 'cherry']

[trim tangle]

@oblique urchin Is there anything in the new annotationlib that can remedy this? (i.e.: produce Annotated[int, "apple", "banana", "cherry"] to match the non-type version)

[oblique urchin]

no, why is this so bad?

[trim tangle]

I guess it's not bad per se, just inconsistent. Annotated did have this collapsing behaviour after all

[oblique urchin]

I try to design the runtime to preserve information. Users who don't care about some layers can write their own code simplifying the representation

but if we simplified it eagerly the information would be lost to users who did care about the extra layers

[trim tangle]

true

Oh, it even says that Annotated is not flattened with type aliases in the Annotated docs. My bad

[rare scarab]

I think fix is asking for a utility function to get the concrete type of a type alias as well as all the annotations for each layer.

At least that's something I'd like.

kind of similar to get_origin/get_args

Using fix's example: ```py
get_concrete_type(Cherry) # => int
get_flattened_annotations(Cherry) # => ("cherry", "banana", "apple")

[jaunty lagoon]

fair, but if i'm gonna ask users to annotate it inline like that anyway, there's no compelling reason to set up aliases in advance. or at the very least, i could leave the aliases as an exercise for the users to do if they so choose. there's no point doing it in the framework itself.

i can accept that i don't have any ideal answers here. and i also recognize that i'm an outlier because i'm approach my project more as an art piece than a technical endeavor. the tech is just the set of constraints i have; the project is to do something beautiful within them.

[wraith saddle]

Hello?

[restive rapids]

you got the wrong channel :d this is for discussing types, not helping people with typos in their code
you should ask in #ot0-psvm’s-eternal-disapproval and send the full file since the error depends on the code on the lines before it occurs, and currently we dont see the line with the error
you might have mixed spaces and tabs in the same file or something like that

[wraith saddle]

oh sorry

[trim tangle]

!cleanban 1414726884045885583 scam

[jade viper]

Is there any way to indicate a "regex" format for a string as a type hint? More specifically a URL?

I thought of something like this but doesn't look too good

import re
from typing import Annotated

URL_PATTERN = re.compile(
    r'^(?:(?:https?|ftp|file)://)?'  # Optional protocol
    r'(?:'  # Host group
    r'(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?|'  # Domain
    r'localhost|'  # Localhost
    r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|'  # IPv4
    r'\[?[A-F0-9]*:[A-F0-9:]+\]?'  # IPv6
    r')'
    r'(?::\d+)?'  # Optional port
    r'(?:/?|[/?]\S+)$',  # Path, query, fragment
    re.IGNORECASE
)

UrlLike = Annotated[str, URL_PATTERN]

[trim tangle]

Depends on why you want to do this

If it's important to validate the string, you can define a class that would in fact validate it ```py
class Url:
def init(self, raw: str) -> None:
if not URL_PATTERN.fullmatch(raw): raise ValueError
self._raw = raw

def __str__(self) -> str:
    return self._raw

``` or use something from the ecosystem like yarl.URL

If it's not, you can use a NewType (and note the relation to the pattern in a comment)

[jade viper]

No I can't validate the string itself

(and note the relation to the pattern in a comment)
How so?

[trim tangle]

What's the point of the regex then?

[trim tangle]

Url = NewType("Url", str)  # pinky promise that the string matches `URL_PATTERN`!

[restive rapids]

so uh
yearly typing survey!
(direct google forms links are not allowed)

[plain dock]

7-11 years
wat
checks
pep 484 was 11 years ago

[rustic island]

Hello. 🙂
I recent Mypy update (PR #18883 for reference; not sure if it's alright put links here) added an error when an inner class uses a type variable used by an outer class.
This is the test case that was added:

from typing import Generic, Iterable, TypeVar
T = TypeVar('T')
class A(Generic[T]):
    class B(Iterable[T]): pass  # E: Type variable "T" is bound by an outer class

I'd like to know what the correct way to achieve this typing is.
How do we define these two classes, A and A.B, such that for a variable a of type A[int] we would get that a.B is of type Iterable[int]?

[alpine matrix]

hello can someone help me ? i cant find method that will change my emoticons to emojis this is my code def main():
name = input("Write something ")

def convert():
#print()
#:) = (smile_emoji)
#:( = (frown_emoji)

main()

[trim tangle]

This is the wrong channel. You should see #❓｜how-to-get-help and make a post in #1035199133436354600 with a more detailed explanation

[quartz hare]

hi

sup

[trim tangle]

hello

[restive rapids]

I think you might be misunderstanding this? a.B would not be of type Iterable[int] even if this was allowed, it would be a type[Iterable[int]]
neither does it make sense how a single definition of the inner B class would be Iterable[int] for A[int] and Iterable[str] for A[str], since it doesnt even have information about the instance - its just a class declaration that happens to be inside another class
that is, python generics are not reified

[rustic island]

Yes, sorry, naturally a.B would be a subclass of Iterable[int], not an instance. That's what I meant.
Regarding the rest, putting aside ways to read values of generic parameters in some circumstances, I am talking strictly about types.
For example, I can have something like this:

class A(Generic[T]):
    class B(Iterable[T]):
        def f(self, x: T) -> T:
            return x

class StrA(A[str]): pass

sb = StrA.B()
s = sb.f("Hello")  # s would be of type `str` if this worked as I expected

I know this doesn't work though, so I wanted to know if there was a way to do this with Python's type system.
(It's not some unheard-of concept; there are languages that do support it.)

[restive rapids]

i can understand the f since the implementation is the same for all types, but how is it going to become an Iterable[str]?

[rustic island]

Here is one way:

class A(Generic[T]):
    class B(Iterable[T]):
        def __init__(self, value: T) -> None:
            self.values = [value]
            self.iterator = iter(self.values)

        def __next__(self) -> T:
            return next(self.iterator)

Kinda doing this on the spot so I'm not sure if I missed something.

Anyway, the iterator thing itself is besides the point.
My own use case doesn't even use that.
The main question here is if it's possible to make the inner class use the same generic type as the outer class.

[restive rapids]

whats your usecase for this? why does B have to be namespaced inside A? why not just A[T] and B[T]?
nested classes in python might not be what you expect coming from "other languages that support this", you're just namespacing it, it doesnt have any connection to the outer class whatsoever

also, that is not to say that you cant hack the typesystem into doing this if you really want to:

from collections.abc import Iterable
from typing import TYPE_CHECKING, reveal_type

class B[T](Iterable[T]):
    def __init__(self, value: T) -> None:
        self.values = [value]
        self.iterator = iter(self.values)
    def __iter__(self):
        return self
    def __next__(self) -> T:
        return next(self.iterator)

class Descriptor[T]:
    def __get__(self, instance: object, owner: type | None = None) -> type[B[T]]:
        return B

class A[T]:
    B = Descriptor[T]()

def test():
    reveal_type(A[int].B(42))

    A[int].B("") # error, "" not assignable to int

test()

[vernal whale]

Hi,

I've run into an interesting behaviour for both mypy and pyright regarding union type subtraction. I'm wondering if this behaviour is a coincidence or by design.

I've toyed with different strategies for enabling subtracting types from a union generically since this is a use case thats come up for me a few times. I've found that using generic variadics
work both in mypy and pyright.

In a nutshell:

T_co = TypeVar('T_co', covariant=True)

class C(Generic[T_co]):
    pass

def variadic_test(*args: A) -> Callable[[C[A | A2]], C[A2]]:
    raise NotImplementedError()


b: C[int | str | bytes] = C()
reveal_type(variadic_test("")(b))  # revealed type is: C[int, bytes]
reveal_type(variadic_test("", b'')(b)) # revealed type is: C[int]

I was quite surprised by this, since to my knowledge, there is no pep that describes this behaviour (I could be wrong).
What I'm wondering is: is this behaviour by design, or be accident? Can I find information about in a pep? If its incidental, maybe I shouldn't rely on it in my api design?

[rustic island]

Thanks. I think that answers my question - that you can't actually do it (without such hacks, at least).
That's really all I wanted to know.
I did come up with another solution for my own use case that didn't require this, but it made me wonder if this was possible.

[indigo halo]

Hey, given a function like this:

async def react_to_data_update[T](
    updates_gen: AsyncGenerator[T],
    *,
    callback: Callable[[T], Coroutine[None, None, None]],
) -> None:
    try:
        async for update in updates_gen:
            await callback(update)

    except asyncio.CancelledError:
        pass

how can I call this while assisting the type checkers about T?

yield react_to_data_update(tpsrv.itc.updates("tpdata"), callback=callback) doesn't seem to allow pyright to infer T == TPData

yield react_to_data_update[TPData](tpsrv.itc.updates("tpdata"), callback=callback) doesn't work because Python seems to think I want to index into the function.

[brazen jolt]

you could cast the tpsrv.itc.updates("tpdata") as AsyncGenerator[TPData] which should make pyright infer T from that once you pass it in, it's possible that pyright only sees it as AsyncGenerator[Unknown], depending on how it was typed

[indigo halo]

yeah, works with the cast.

Thank you @brazen jolt

[stiff acorn]

@dataclass
class Category:
    type: Literal["media", "literature", "film"]
    format: MediaFormat | LiteratureFormat | FilmFormat

    @staticmethod
    def media(format: MediaFormat, /) -> Category:
        if not isinstance(format, MediaFormat):
            msg = f"Expected a member of 'MediaFormat', but got {type(format).__name__}"
            raise TypeError(msg)
        return Category("media", format)

    @staticmethod
    def literature(format: LiteratureFormat, /) -> Category:
        if not isinstance(format, LiteratureFormat):
            msg = f"Expected a member of 'LiteratureFormat', but got {type(format).__name__}"
            raise TypeError(msg)
        return Category("literature", format)

    @staticmethod
    def film(format: FilmFormat, /) -> Category:
        if not isinstance(format, FilmFormat):
            msg = f"Expected a member of 'FilmFormat', but got {type(format).__name__}"
            raise TypeError(msg)
        return Category("film", format)

is there a way to type the relation between type and format (i can change the code itself if there's another approach that needs runtime changes on top)

[restive rapids]

why do you need the string tag if you already have the type to discriminate on using isinstance?

[stiff acorn]

that makes sense, I guess i can just do a union and then isinstance

[jolly cipher]

or match, perhaps even better on that case

[solemn sapphire]

Is there anyway to import files from typeshed for type checking purposes
Say I define a custom_cache and the type level behaviour is similar to functools.lru_cache, rather than copy pasting quite a lot of code is there a mechanism to resuse it?

[rough sluice]

stdlib/functools.pyi lines 56 to 69

@final
class _lru_cache_wrapper(Generic[_T]):
    __wrapped__: Callable[..., _T]
    def __call__(self, *args: Hashable, **kwargs: Hashable) -> _T: ...
    def cache_info(self) -> _CacheInfo: ...
    def cache_clear(self) -> None: ...
    def cache_parameters(self) -> _CacheParameters: ...
    def __copy__(self) -> _lru_cache_wrapper[_T]: ...
    def __deepcopy__(self, memo: Any, /) -> _lru_cache_wrapper[_T]: ...

@overload
def lru_cache(maxsize: int | None = 128, typed: bool = False) -> Callable[[Callable[..., _T]], _lru_cache_wrapper[_T]]: ...
@overload
def lru_cache(maxsize: Callable[..., _T], typed: bool = False) -> _lru_cache_wrapper[_T]: ...```

[oblique urchin]

you can import under if TYPE_CHECKING

[solemn sapphire]

I'm unable to figure out how would that work

if TYPE_CHECKING:
  from typeshed.stdlib.functools import lru_cache

that doesn't work (obviously)

[rare scarab]

Import it directly from functools

__all__ = ['custom_cache']
if TYPE_CHECKING:
  from functools import lru_cache as custom_cache
else:
  def custom_cache(*args, **kwargs):
    ...

[solemn sapphire]

Thanks!

[rare scarab]

This technically lies and says your custom_cache is lru_cache. So you might not have any static docs like displayed in vscode

[solemn sapphire]

I'm using pylsp with neovim and that feature has been broken on pylsp for quite some time :(

[floral heron]

[project]
name = "w"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "mypy>=1.18.1",
    "numpy==2.2.3",
    "ty>=0.0.1a20",
]
[I] > cat test.py
import numpy as np


def test() -> None:
    d = {"a": np.array([1,2,3])}
    np.savez_compressed("tmp.npz", **d)
[I] > uv run mypy test.py
test.py:6: error: Argument 2 to "savez_compressed" has incompatible type "**dict[str, ndarray[tuple[int, ...], dtype[Any]]]"; expected "bool"  [arg-type]
Found 1 error in 1 file (checked 1 source file)```

I'm at a bit of a loss on this one. Pretty sure I'm not doing anything actually wrong and it's mypy who is confused, but figured it's worth a second set of eyes

[rare scarab]

!d numpy.savez_compressed

[rough sluice]

numpy.savez\_compressed

numpy.savez_compressed(file, *args, allow_pickle=True, **kwds)```
Save several arrays into a single file in compressed `.npz` format.

Provide arrays as keyword arguments to store them under the corresponding name in the output file: `savez_compressed(fn, x=x, y=y)`.

If arrays are specified as positional arguments, i.e., `savez_compressed(fn, x, y)`, their names will be *arr\_0*, *arr\_1*, etc.

[floral heron]

My gut is to say that mypy is just out of this world confused, there's no world in which that's a second positional argument, it's clearly a double splat expanding out into keyword arguments.

[rare scarab]

It can't seem to understand that d only has the key a

[floral heron]

Doesn't actually really matter what's in d, this is minimized from the real code

even an empty dict fails

[rare scarab]

Try passing allow_pickle=True after **d

[floral heron]

np.savez_compressed("tmp.npz", allow_pickle=True, **d)
and
np.savez_compressed("tmp.npz", **d, allow_pickle=True)

both pass

[rare scarab]

What does pyright say?

[floral heron]

Good question. ty was fine with the original snippet, let's see what pyright says.

/Users/rdeaton/w/test3/test.py
  /Users/rdeaton/w/test3/test.py:6:38 - error: Argument of type "ndarray[Unknown, Unknown]" cannot be assigned to parameter "allow_pickle" of type "bool" in function "savez_compressed"
    "ndarray[Unknown, Unknown]" is not assignable to "bool" (reportArgumentType)```

[rare scarab]

You can't trust ty to not give you false negatives yet imo

Can you use a TypedDict?

[trim tangle]

mypy sometimes has very confusing error messages

[floral heron]

Yeah, ty is obviously not a reliable checker yet, but sometimes gives nicer messages. The pyright also failing is interesting

[trim tangle]

Type checkers aren't smart enough to remember that d only has specific keys (that are not allow_pickle). So to mypy/pyright, d is a dict with some arbitrary str keys. That's why mypy and pyright complain

[rare scarab]

Passing allow_pickle explicitly would override any coming from d, so it passes

[floral heron]

I can kinda see that in pyright's error, but boy mypy's error is gross

[trim tangle]

I think you get a runtime error if you pass two of the same kwarg

!e

def foo(bar, **kwargs):
    print(f"{bar=} {kwargs=}")

kw = {"a": 1, "bar": 2}
foo(1, **kw)

[rough sluice]

:x: Your 3.13 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 5, in <module>
003 |     foo(1, **kw)
004 |     ~~~^^^^^^^^^
005 | TypeError: foo() got multiple values for argument 'bar'

[rare scarab]

But as kwargs?

[trim tangle]

wdym?

[rare scarab]

!e

def foo(**kwargs):
    print(f"{kwargs=}")

kw = {"a": 1, "bar": 2}
foo(**kw, a=3)

[rough sluice]

:x: Your 3.13 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 5, in <module>
003 |     foo(**kw, a=3)
004 |     ~~~^^^^^^^^^^^
005 | TypeError: __main__.foo() got multiple values for keyword argument 'a'

[rare scarab]

Yup, you're right

But it won't matter here because allow_pickle won't be in the dict

[jolly cipher]

and ideally after making sure there's a # stable comment near the imported item definition in typeshed :)

but that's if importing from _typeshed

i often like reaching out for OptExcInfo, Unused and StrPath

[stiff acorn]

I really want StrPath alias in typing

[rare scarab]

I want Pattern to have a default generic of Pattern[str]

[trim tangle]

I think adding a default is a breaking change

Pattern used to mean Pattern[Any], now it means Pattern[str]

[rare scarab]

But how popular are bytes patterns?

[oblique urchin]

meaning str | os.PathLike[str]?

[trim tangle]

right, this is python... what's a few breaking changes between friends

[rare scarab]

Python 4's only change: Adding a default generic to Pattern

[oblique urchin]

it would be a typeshed change anyway 😄

[stiff acorn]

yes! I use it quite often.

Sometimes I define it myself, other times i remember it's in typeshed so I import it from there under TYPE_CHECKING

[indigo halo]

Do you guys know of a place to ask Pydantic questions? I particularly need to find out how to write __get_pydantic_core_schema__ for a behaviour-only class, i.e. one that does not have any data. return core_schema.no_info_plain_validator_function(lambda x: x) yields an error: "UserWarning: A custom validator is returning a value other than self."

[brazen jolt]

on python discord, probably just #1035199133436354600, though you might not find that many people able to help you with something this specific. I know that pydantic has github discussions enabled, so you could try asking there: https://github.com/pydantic/pydantic/discussions