#type-hinting

1 messages · Page 41 of 1

indigo halo
#

I am not using github. I got the dependencies covered. The problem is that it runs with different flags.

If I run mypy tptools/util.py, it runs without errors: "Success: no issues found in 1 source file"
I have mypy configured in pyproject.toml with strict = true and warn_unreachable = true.

If pre-commit invokes mypy, it fails with:

tptools/util.py:11: error: Function is missing a type annotation  [no-untyped-def]

this leaves me quite confused, and not only because the function is actually typed:

def json_dump_with_default_x(
    obj: object,
    methodname: str = "json",
) -> object:
#

How can the pre-commit hook report an error like this if mypy --strict does not??

indigo halo
#

Alright, pre-commit clean and recreating it fixed that. Now on to the next problem…

#

I have a dataclass Entry(*, id:int, player1:Player, player2:Player, namepolicy:NamePolicy|None = None) and a factory that creates Entrys. I have pre-created the Player objects in a mapping players={'player1': Player1Instance, 'player2': Player2Instance} and now I am calling Entry(id=123, **players). And mypy complains:

error: Argument 2 to "Entry" has incompatible type "**dict[str, Player]"; expected "NamePolicy" [arg-type]

Yes, I could just unpack that, but I am curious what is going on. Because this is valid, right?

trim tangle
#

you might want a TypedDict, or to just make an Entry instead of a dictionary

#

(i.e. why are you making a dict and not an Entry directly?)

#

alternatively just Entry(id=123, player1=players["player1"], player2=players["player2"])

indigo halo
trim tangle
indigo halo
#

yeah I guess I am potentially doing that.

#

TypedDict it is…

trim tangle
#

Well, you could just have a tuple[Player, Player]

#

You could even expect a tuple[Player, Player] or a Sequence[Player] or a dict[str, Player] in the Entry

indigo halo
#

or that

trim tangle
#

I'd personally avoid TypedDict unless necessary, because it's kinda clunky and verbose and has some soundness issues

indigo halo
#

i agree

#

the thing is that player2 is optional, and so I am creating the dict in a comprehension that has if p["id"] is not None at the end, so the dict is either len==1 or len==2 and using **players is just a convenient way to call the function with varying number of arguments.

trim tangle
indigo halo
#

bloody type checkers 😉

trim tangle
#

If you support a variable number of players, you could have a dict or sequence of players instead of two fixed attributes

indigo halo
#

I mean, hey, I can only fathom the complexity involved in even doing what they are already capable of.

indigo halo
trim tangle
#

tuple[Player,] | tuple[Player, Player]
kinda wild though

indigo halo
#

yikes ;0

trim tangle
# 
type Upto4[T] = tuple[T,] | tuple[T, T] | tuple[T, T, T] | tuple[T, T, T, T]
indigo halo
#

i just had a PTSD flashback to C++ days and specifically this book: https://en.wikipedia.org/wiki/Modern_C%2B%2B_Design

Modern C++ Design

Modern C++ Design: Generic Programming and Design Patterns Applied is a book written by Andrei Alexandrescu, published in 2001 by Addison-Wesley. It has been regarded as "one of the most important C++ books" by Scott Meyers.
The book makes use of and explores a C++ programming technique called template metaprogramming. While Alexandrescu didn't ...

#

like this guys championed the equivalent of for loops using only C++ templates 😉

trim tangle
#

I sometimes watch C++ talks for some reason, he's my favourite speaker

#

(I don't know C++ at all)

indigo halo
#

back in the days, it totally revolutionised the way people wrote and thought about C++

#

A dataclass already specifies typed fields. Can I extract a TypedDict from that? Like if I have dataclass(Foo(a:int, b:str)), I would like to be able to type a dict {"a":1, "b":"string"} without recreating the set of fields in a TypedDict

trim tangle
#

nope

#

You could write a mypy plugin if you're into that, of course

indigo halo
#

no thank you 😉

winter lintel
#

auto-generating a typed dict from __dict__ would be neat

indigo halo
trim tangle
indigo halo
#

databases

#

and no, I am not using an ORM and not able to ☹️

trim tangle
#

how would you use that typeddict?

winter lintel
#

so you want to auto-generate the transform method with dataclass field kwargs

#

not obvious to me how the output dict being typed would be a benefit

#

since you're probably gonna ** unpack it because you like brevity

#

and the only type checked access to a TypedDict is by literal keys

#

and i'd be impressed if access to the db without an ORM is type checked, but i'm far from db wizard

indigo halo
#

the bloody db returns everything as string already

#

I am in a world of pain

#

and mypy is not helping right now ☹️

winter lintel
#

it sounds like a good job for a parent class, to have a transform method that turns self.__dict__ into the db stuff, maybe coerces everything to str because lmao, and respects a ClassVar[tuple[str]] of fields to omit from the transform

#

that way the return type is always dict[str, str]

indigo halo
# 
def dict_key_translator[T](
    indata: dict[str, T], xlate: dict[str, str], *, strict: bool = False
) -> dict[str, T]:
    …

def extract_playerdata_from_entry_query_data(
    data: EntryData, playernr: int
) -> dict[str, Any]:
    return dict_key_translator(
        cast(dict[str, Any], data),
            …
…

and mypy says:

Incompatible return value type (got "Mapping[str, Any]", expected "dict[str, Any]")

trim tangle
#

What's EntryData?

#

I think you're omitting the part of the code that mypy complains about

indigo halo
# 
class EntryData(TypedDict):
    entryid: int
    player1id: int
    firstname1: str
    name1: str
    club1: str
    country1: str
    player2id: int
    firstname2: str
    name2: str
    club2: str
    country2: str
indigo halo
trim tangle
#

right, a TypedDict is not assignable to a dict[str, Any]. Because, for example, dict[str, Any] supports the clear() method.

#

You need a collections.abc.Mapping[str, object]

winter lintel
#

yea it can coerce the TypedDict to dict but not the other way around

indigo halo
#

I updated the code above. I am casting the TypedDict to dict[str, Any]

rare scarab
#

!pip datamodel-code-generator can be useful if you're OK with using json schema

rough sluiceBOT
indigo halo
#

that is cool yeah. Unfortunately, I have no API specs or anything, just this bloody ancient database to work with ☹️

trim tangle
indigo halo
#

nope

trim tangle
#

Then you should use collections.abc.Mapping as an annotation

indigo halo
#

i mean, I use it to create a new dict

trim tangle
winter lintel
#

yeah if you use Mapping it will error if you mutate the dict, makes it read-only during type checking

#

value

indigo halo
#

tptools/entry.py:147: error: No overload variant of "ror" of "dict" matches argument type "MutableMapping[str, Any]" [operator]

#

yeehaa, the gift that just keeps on giving

#

MutableMapping does not provide operator|

#

if foo and bar are both MutableMapping[str, str], then I should be able to do foo |= bar and foo.update(bar)…

#

bar doesn't even have to be mutable for that

#

dict.update seems to work, |= does not

pastel egret
#

That's because | was added much later, MutableMapping was already in use. Requiring it would be backwards incompatible.

rare scarab
oblique urchin
rare scarab
#

Hm, true

oblique urchin
#

This argument basically means you can never add a method to an ABC

#

Which is sad

trim tangle
#

(e.g.. you don't have to inherit or register yourself to be an iterator, that would make it unusable)

oblique urchin
trim tangle
#

huh

oblique urchin
rough sluiceBOT
#

stdlib/typing.pyi line 615

class Sequence(Reversible[_T_co], Collection[_T_co]):```
oblique urchin
#

generally only the ones with a single method are

trim tangle
#

ah, I see

soft matrix
#

and the runtime doesnt have it as protocol like either

#

i remember running into this a while ago

spare yacht
#

I'm trying to type annotate a function that uses mpmath.

mpmath.mpf is a factory function that returns the hidden type mpmath.ctx_mp_python.mpf

Do I annotate using the hidden type? Do I do something with type(mpmath.mpf(0))?

Thanks in advance.

BTW, the discord invite to the pydantic server is broken on their website

grave fjord
spare yacht
#

I'm pretty new to this, so I might be confusing myself.

mpmath has an init.py that creates mp as mp = MPContext()
It later creates mpf as mp.mpf

This makes mpmath.mpf a variable according to pylance, even though it is a type.

grave fjord
spare yacht
grave fjord
#

Which line is the issue?

spare yacht
#

line 66 is where I import in a funny way
line 246 is the first time I use it in a type annotation

spare yacht
#

The funky thing I'm doing is taking an internal class ("_mpf") and importing it as though it wasn't internal ("mpf") just so I can make type annotation not throw any errors. This smells wrong to me, so I'm asking for either assurance or a better way forward.

indigo halo
#

I decided I didn't want to waste time annotating types in my tests, and so I added to pyproject.toml:

[[tool.mypy.overrides]]
module = ["tests.*", "integration.*"]
disallow_untyped_defs = false

My tests under tests/* are all being treated accordingly, but tests under integration/* produce mypy warnings, e.g.:

integration/test_loading_entries.py:11: error: Function is missing a return type annotation [no-untyped-def]

what am I missing?

tranquil turtle
#

perhaps there is another override that overrides integration.*, and that messes with your override?

indigo halo
#

Those aren't modules, they are globs… maybe that is an issue? But it works for tests…

indigo halo
#

grrrr

#

touch integration/__init__.py

#

thatg makes no sense but kinda does. it's wrong anyway

stiff acorn
#

Although I would recommend typing your tests if your library is typed. Helps catch typing errors.

indigo halo
#

Maybe I will add that later. For a start, it cost me way too much time. But maybe that was also learning curve

indigo halo
stiff acorn
#

You should use file overrides instead

indigo halo
#

I don't want "excludes"

#

I want to just override one flag

stiff acorn
#

So you want to typechek the file just not the function signatures?

indigo halo
#

yeah

#

i supposed function sigs for tests are really easy though and I should just

winter lintel
#

you typecheck your tests?

feral wharf
#

You aren't supposed to?

#

Oh wait yeah that kinda doesn't make sense

#

Tests are supposed to catch it

winter lintel
#

i've found a lot of bugs by typechecking my code

#

i don't think typechecking my tests would have found any bugs in my code

#

i've found a lot of bugs by running tests on my code

#

that's just my experience

#

quality of tests does matter but only when it helps you change the tests quickly as you refactor imo

grave fjord
winter lintel
#

make your test cases load from data files and float above these concerns 😂

winter lintel
grave fjord
#

Bugs in the code

grave fjord
winter lintel
#

i should try it and see what shakes loose

terse sky
#

it's really easy for type mistakes to quietly pass in tests because of things like

#

!e

print(5 == "hello")
rough sluiceBOT
terse sky
#

depending who you talk to, a lot of people would say this should be an error instead of simply returning False

#

but this is how it behaves, so you can imagine if you have an assertFalse in a test, and you think you're comparing two integers, but you forgot to parse the string into an int, the test will always pass

#

just one example

stiff acorn
#

You can also catch typing bugs while testing. Maybe you're testing an invalid value and you notice the type checker isn't raisining an error at the call site because your type is too wide. Maybe you're testing a successful value but you find out the type checker is flagging it because your type isn't wide enough.

#

I've found plenty of both in my projects

#

Where typing checking my tests ends up revealing cases where my types weren't as good as they could have been

winter lintel
#

good points, and there's probably some value, but i think there are other practices that minimize it

#

linting the test code with opinionated rules may have almost as much value, and at less cost

#

i'm pretty spoiled by greenfielding something all by myself though at the moment 😂

#

so it's all built with the strictest mypy settings and a pile of ruff rules

terse sky
#

idk, I personally think the cost of adding type checks even before the benefits here are discussed is basically zero

#

any small amounts of time you spend you already get back from improved tooling

#

auto completion, catching errors as-you-type instead of when things run

#

if you're doing green field that's exactly when the cost is zero, really

#

What's expensive is trying to get types retrofitted on an existing codebase

rare scarab
restive rapids
terse sky
#

Incremental is even the default effectively, is it not

#

If you don't pass --strict

winter lintel
#

hmm

from unittest.mock import Mock
import pytest
from pytest_mock import MockerFixture

@pytest.fixture
def mock_handlers(mocker: MockerFixture) -> Mock:
    return mocker.Mock(spec=MessageHandlers)

error: Returning Any from function declared to return "Mock"  [no-any-return]
winter lintel
#

oh, maybe i should lie and say it returns MessageHandlers instead of a Mock

#

nope

indigo halo
#

If I have a class whose constructor takes a type, e.g. __init__(self, cls=dict), how do I type that?

#

it's really a factory argument, so could be a callable, and I guess a constructor is a callable, except not to the linter…

trim tangle
indigo halo
#

Well, I am trying to figure that out

trim tangle
#

If you want cls to be a callable, you need to annotate it as a Callable

indigo halo
#

type Factory[T] = Callable[..., T] | T doesn't work

trim tangle
#

Do you have some examples of how you'd use DataAdapter?

rare scarab
#

It may be a problem with typevar defaults.

indigo halo
#

cls: Factory[T] = dict[str, DataType] means that T would be seen a dict. DataAdapter[T] has T has parameter. So I guess there must be somewhere in the code where T is not a superclass of dict.

rare scarab
#

Try using TypeVar("T", default=dict[str, Any])

trim tangle
rare scarab
#

What if you do DataAdapter[list]([])

#

Using a factory function for the default adapter may be required.

trim tangle
indigo halo
#

I want the argument to be optional, and a dict used by default, unless I provide something else

rare scarab
# 
@dataclass(frozen=True)
class DataAdapter[T](Iterator[T]):
  iterable: Iterator[RowType]
  cls: Factory[T]

  @classmethod
  def default(cls, iterable: ...) -> "DataAdapter[dict[str, Any]]":
    return cls(iterable, dict[str, Any])
# 
adapter = DataAdapter.default([])
list_adapter = DataAdapter[list[Any]]([], list)
trim tangle
trim tangle
rare scarab
#

But this wouldn't work, right? ```py
from typing import Generic
from typing_extensions import TypeVar
T = TypeVar("T", default=dict[str, Any])

@dataclass(frozen=True)
class DataAdapter(Generic[T])
iterable: Iterator[RowType]
cls: Factory[T] = dict[str, Any]

trim tangle
#

You haven't actually shown how you'd use this class. It really looks like you're making an overcomplicated version of map

indigo halo
#

Well, for a start, DataAdapter takes an Iterator of Iterables, e.g. a database table (rowiter over rows, which are tuples, and it returns an iterator of a mapping of column name to cell, so there is a zip(colnames,row) in there. I also allow massaging of cells with callbacks. And some other normalisation.

#

tempted to just type:ignore that line

#

is it works

rare scarab
#

Are you using this instead of an ORM?

indigo halo
#

yes, it's a poor man's ORM because setting up an ORM with this stupid database is not possible.

#

don't ask.

rare scarab
#

ah, badly designed database. Very common.

indigo halo
#

no relational constraints whatsoever, and so much redundancies in the gigantic tables

drowsy gulch
#

Has anyone worked with TypedDict?
Is there any reliable and type-safe data->json->data workflow?
or should i use a proper database approach? i want my data to be readeable and easy to edit by hand

rare scarab
#

Pydantic is a data validation library

drowsy gulch
#

Does it support nesting?

winter lintel
#

yes

rare scarab
#

It even supports custom validation logic

winter lintel
#

or should i use a proper database approach?
is it between database and flat file(s)? ever going over the wire?

drowsy gulch
winter lintel
#

pydantic is very good, it's a lot faster than it used to be, and covers your "reliable" and "type-safe" and "json" reqts perfectly

#

i've worked with TypedDict, it made me not want to work with TypedDict

rare scarab
winter lintel
#

all the cool kids write their python libs in rust now huh?

#

does pyo3 have a better type generation story than pybind11?

terse sky
#

typeddict is for legacy code tbh

#

(at least, mostly)

terse sky
winter lintel
#

there's one place i still use TypedDict, typing my test cases that come out of yaml files

terse sky
#

I think attr + cattr will get you there

#

I guess I don't quite understand the yaml example, but I would probably still use dataclasses/attr/etc for that

grave fjord
rough kettle
#

is it possible to fully type an implementation of functools.partial?

#

at least for positionals

restive rapids
# rough kettle at least for positionals

it would look something like

from collections.abc import Callable

def partial[*Now, *Later, Return](fn: Callable[[*Now, *Later], Return], *now: *Now) -> Callable[[*Later], Return]:
    return lambda *later: fn(*now, *later)

but you cant have multiple unpacks

rare scarab
#

No Concatenate?

#

Ah, you can't concatenate 2 typevartuples

novel notch
#

Sometimes I can use assert isinstance(...), but that requires a regular import. 🤔 So... With a TYPE_CHECKING import I can avoid circular import.

What are my options?

  • override/ignore the typing instead of the assert. How?
  • investigate if the circular import can be avoided. Not sure this is always possible? 🤔
  • other way?
terse sky
#

not sure if I missed some context here

#

I would almost always try to avoid circular imports though

rare scarab
novel notch
trim tangle
#

It's still a circular import (and a circular dependency in general)

rare scarab
#

It just doesn't raise

trim tangle
#

circular imports aren't banned in python, but one module accessing another while both aren't completely executed isn't allowed (in some cases)

#

or, well, sometimes it won't be able to find the required symbol as it hasn't been assigned yet

novel notch
#

I'm trying to do slight improvements without spending a month 😂

novel notch
terse sky
#

Yeah legacy code might not be worth breaking the circle

#

Usually it's not that bad, you just move some code into a third file

#

But crazy things can certainly happen

trim tangle
#

But if you do just import foo and only use foo.Foo when methods of Bar run, this problem doesn't occur

novel notch
novel notch
trim tangle
#

understandable

wooden cipher
#

hi
if a function can return any value, even None
would i type hint it as -> Any or -> Any | None ?

trim tangle
wooden cipher
#

it's a function that gets a callback function and calls it
the callback can return anything, or just return None

trim tangle
#

If the result of the function is discarded, it's probably better to be -> None

#

If it can be anything, -> object

wooden cipher
#

it can be ignored or used
it's library code, the return value is used by user

trim tangle
#

What kind of value can it be? Do you have some examples?

rare scarab
#

Would using generics make sense?

wooden cipher
rough sluiceBOT
#

django-stubs/dispatch/dispatcher.pyi lines 18 to 20

def connect(
    self, receiver: Callable, sender: object | None = None, weak: bool = True, dispatch_uid: Hashable | None = None
) -> None: ...```
wooden cipher
rough sluiceBOT
#

django/core/cache/__init__.py lines 61 to 64

def close_caches(**kwargs):
    # Some caches need to do a cleanup at the end of a request cycle. If not
    # implemented in a particular backend cache.close() is a no-op.
    caches.close_all()```
trim tangle
#

What happens to the value it returns if it's not None?

wooden cipher
#

user gets a list of values
there can be None in it

#

what they do with it is up to them

trim tangle
#

Sounds like it's Callable[..., object]

#

which is the same as just Callable from the user's point of view

#

user = the caller of connect()

wooden cipher
#

i've done some work on the arguments it takes
so i think i have to define a return value as well
also i'm not using Callable
i've defined a protocol

trim tangle
#

can you show the protocol?

wooden cipher
# 
class Receiver(Protocol):
    def __call__(
        self, *, signal: "Signal", sender: Any, **kwargs: Any
    ) -> Any | Awaitable[Any]: ...
#

i have doubts if Awaitable is correct or not tho

trim tangle
#

If it accepts specific kwargs and no positional arguments, then a protocol is correct, yes

trim tangle
#

you could keep | Awaitable[Any] as a reminder, I guess

wooden cipher
#

i put it in there since it can be an async function

trim tangle
#

well, if a function returns an awaitable, that still counts as Any, because everything counts as Any

sullen pecan
wooden cipher
#

that's fair

trim tangle
# wooden cipher that's fair

That could still be useful for documentation purposes. Just like the signature says sender: object | None = None, which is redundant in favour of sender: object = None.

sullen pecan
oblique urchin
#

it means OR

trim tangle
sullen pecan
#

Thanks.

wooden cipher
#

ok
thank you

#

it's been a good day
i learned we have a type keyword now
that's fun

feral wharf
#

Since 3.12 yes

rare scarab
#

MagicStack/MagicPython#262
MagicStack/MagicPython#265
MagicStack/MagicPython#270
MagicStack/MagicPython#271

mighty lindenBOT
grave fjord
wooden cipher
tranquil turtle
terse sky
#

What's a counter example of this? If T is Never? Or is there something more interesting?

#

I'm actually not even sure if Any | Never != Any

oblique urchin
terse sky
#

Ah right, good point

#

It's too easy to mix up Any with the top type

#

It doesn't help that Any is the name of the top type in some other languages

trim tangle
#

Well, in this case, it should be the same from the caller's perspective. These are stubs, and any protocol that fulfils -> Any | Awaitable[Any] fulfils -> Any and vice versa

terse sky
#

If one type checks in situations where the other doesn't then they cannot be equivalent

trim tangle
terse sky
#

Yeah I don't think that's right

trim tangle
#

what's wrong with it?

terse sky
#

It only works in one direction

trim tangle
#

Which direction is wrong?

terse sky
#

f is a super type to g

oblique urchin
#

no I think that's right

terse sky
#

Actually maybe opposite

#

Ah, actually, I see

oblique urchin
#

those types should be assignable in both directions

terse sky
#

Man I hate Any

#

Any breaks all the sane rules of type systems

trim tangle
#

well, such is the business of gradual typing

terse sky
#

Yeah, it just doesn't really make any sense

#

A and B are not equivalent types yet somehow functions returning A and B are

#

It's pretty bonkers

oblique urchin
#

they're not equivalent

terse sky
#

Yes, I knew someone would nitpick that

#

But it can be implemented in either direction

trim tangle
#

well, if you call g, you'll need to account for the fact that the result can be int

terse sky
#

Yeah, the type error comes up internally

#

It just ends up breaking the normal rules, your f/g example

#

Normally it's not possible that two function signatures are mutually assignable without them being the same

#

Two types are supertypes of each other but they're not the same type

#

Any has this relationship with every type so I guess it's not that surprising once you take it all in

stiff acorn
#

What would be the right way to approach to type a param that, despite having a few "popular" set of values, can essentially be any arbitrary string?

cinder bone
#

str

stiff acorn
#

Thought as much, thank you

feral wharf
#

Literal["x", "y", "z"] | str works too right?

trim tangle
feral wharf
#

But will type checkers still "see" the Literal?

trim tangle
#

actually, pyright seems to give suggestions in that case. So it could be useful

image.png
#

@stiff acorn ^

stiff acorn
#

That's very helpful, thank you!

feral wharf
#

Really neat!

solid sleet
# 
    @property
    def size(self) -> Size:
        """Size of gadget."""
        return self._size

    @size.setter
    @bindable
    def size(self, size: Size):

is there a way to get around pylance giving me a reportRedeclaration on size (commenting out @bindable will remove the error)

rare scarab
#

Bug report to pyright?

#

How does mypy feel about it?

solid sleet
#

i dunno

eager vessel
#

Is there a way avoid type-abstract error in mypy with given code?

from typing import Any, Protocol, TypeVar

T = TypeVar("T")

some_cache: dict[Any, Any] = {}


def method(cls: type[T]) -> T:
    return some_cache[cls]


class Interface(Protocol):
    def do_stuff(self) -> str: ...

# error: Only concrete class can be given where "type[Interface]" is expected  [type-abstract]
method(Interface)

I did some searching in python/mypy issues but didn't find anything relevant 🤔

rare scarab
#

Return a callable instead

eager vessel
#

It should return instance of the type, not callable

rare scarab
#

Or accept a callable

#

Making T variant may help too

#

Don't recall which is needed. Probably contravariant

eager vessel
#

Callable seems to work though 🤔

#

Yep, works perfectly, thanks 😃

#

The one thing I don't like about this is that it allows you to pass things like functions inside though

rough kettle
#

is there a protocol for elements that can be part of a set?

#

so far I think only collections.abc.Hashable is required, but I'm not sure where to check that

trim tangle
rough sluiceBOT
#

stdlib/builtins.pyi line 1178

class set(MutableSet[_T]):```
trim tangle
#

The trouble is that e.g. object is hashable, but there are subclasses of object whose instances are not hashable

rough kettle
#

oof I see

#

so I should just take Any then?

trim tangle
trim tangle
rough kettle
#

this would require something like a -> NotImplemented return type similarly to -> NoReturn for type checkers to be able to determine if __hash__ is removed right?

oblique urchin
# 
def make_hashable(x: object) -> Hashable:
    return x  # OK, `object` is Hashable

x = {make_hashable([])}
rough kettle
#

isn't object hashable though?

trim tangle
#

Yes, but lists are not hashable

rough kettle
# 
In [1]: hash(object())
Out[1]: 8770461571815

In [2]: set([object()])
Out[2]: {<object at 0x7fa083f1acc0>}
#

I see

trim tangle
#

since [] is an object, and it's not hashable

rough kettle
#

inheritancen't

terse sky
#

A lot of these headaches are basically caused by the fact that set and dict cannot enforce their keys not changing. So as a workaround the language tries to avoid implementing hash for mutable types

#

I wonder how annoying it would be if object were unhashable

#

I know python uses tons of dicts internally but I thought they basically always had string keys

tranquil turtle
#

hmm, no, I think you're right

grave fjord
#

Or is this a thing where if T is Unknown it quashes an error

terse sky
#

this kind of reasoning is treating Any like it's the top type - I fell into exactly the same trap

#

if you have t: Any you can do t.foo() - that's totally okay
if you have t: Any | str you cannot do t.foo - str.foo doesn't exist

#

Any is this weird opt-out of typing that kind of behaves like a mixture of a top and bottom type.

grave fjord
#

Oh hrm that seems like a bug

#

The checker should warn to force you to use object instead of Any | T here

trim tangle
# grave fjord Oh hrm that seems like a bug

It can be a bit unintuitive, but I'm pretty sure that's intentional. Consider this:

foos: list[Any]
bars: list[int]
if random.random() > 0.5:
    item = random.choice(foos)
else:
    item = random.choice(bars)
``` after the if-else, you won't be allowed to do `print(item + "!")` because it could be an int.
#

(at least in pyright)

#

In other words: for an operation f and x: A | B, f(x) is allowed only if f(A) and f(B) are allowed. This property is preserved when A is Any

hoary pagoda
#

Any propagating less is probably a good thing tbh

grave fjord
terse sky
#

Any can't be object though

#

object has different behavior from Any | T

#

with Any | T, you can for example pattern match or do an isinstance check for T

#

handle the T, and then in the other branch you can do Any-thing you want 😛

#

that will not work with object

grave fjord
#

You can't handle T though

terse sky
#

In any case, you can always for example just return the Any | T back to the user - for the user, T will be some concrete thing

#

so returning Any | T back to the user isn't the same as returning them an object - now T is something concrete like str

#

Any | str isn't the same as object - you can call .capitalize() on an Any | str

eager vessel
oblique urchin
#

Any | T represents an unknown set of static types that is at least as big as T

terse sky
#

but they're still not the same. I mean Any | T doesn't have to be T, so that's a difference right there. If you T is concrete then, again, you can do an isinstnace check and handle the Any and T cases separately

eager vessel
terse sky
#

T isn't a subset of Any - Any is not a top type

#

T is a subset of object

#

so object | T is exactly the same as object

#

Never is a bottom type, so Never | T is the same as T

#

Any eventually gets weird to think about because at the end of the day Any is basically a request to the type checker to leave you alone.
Any | T is basically saying "I either have a T or something else, but if it's something other than a T then please don't worry about type checking it and assume everything will be ok"

eager vessel
terse sky
#

Your reasoning works for object

#

But also if Any is a superset of T, then Any | T would be Any

#

Not T

eager vessel
terse sky
#

I'm not sure what you mean by that

#

Let's look at object since it actually works this way

eager vessel
#

Sequence | Container does not become Sequence

terse sky
#

object | T is object, right?

eager vessel
#

object isn't a superset of T though

terse sky
#

When you say superset are you talking about the API or the types?

eager vessel
#

yes

terse sky
#

object is a superset of T - all types in T are also in object

#

It can't be "yes" because the two things are reversed

#

objects API is a subset of all other types. Object is a superset of all types - all types are in object

eager vessel
#

I'm pretty sure it's discord's fault 😅

image.png
#

I'm mostly talking about structural subtyping it seems, with concrete types/classes it probalby works differently

terse sky
#

I think you use the terminology for API where it's expected to be talking about the types itself basically

#

Type checkers care about the narrowest API

#

Which is the broadest type

terse sky
trim tangle
#

x: Any | T describes the same set of objects as x: Any, but a type checker will allow different sets of operations on those xs

oblique urchin
trim tangle
#

??

oblique urchin
trim tangle
#

What do you mean by sets of sets?

oblique urchin
#

A gradual type represent a set of fully static types that it can materialize to

trim tangle
# oblique urchin https://typing.python.org/en/latest/spec/concepts.html#gradual-types

In the same way that Any does not represent “the set of all Python objects” but rather “an unknown set of objects”, tuple[int, Any] does not represent “the set of all length-two tuples whose first element is an integer”. That is a fully static type, spelled tuple[int, object]. By contrast, tuple[int, Any] represents some unknown set of tuple values; it might be the set of all tuples of two integers, or the set of all tuples of an integer and a string, or some other set of tuple values.
I think I don't undestand that explanation. If Any means some specific but unknown set of objects, we'd have the ability to have foo: Any1 and bar: Any2, which are two unknown sets of objects, which are not assignable to each other. In reality, we have a single Any type

#

or rather, I don't see how that definition helps with anything in the type system

#

Wouldn't it be simpler to say that Any is just a special form such that Any is assignable to any other type, and any other type is assignable to Any?

terse sky
#

That's mostly how I think of it

#

Simultaneously a top and bottom type

#

But not quite either 🙂

trim tangle
# oblique urchin https://typing.python.org/en/latest/spec/concepts.html#gradual-types

The definition in the article seems to suggest that Any is a stand-in for a statically known (possibly infinite) set of objects that just can't be expressed in the type system yet.
But suppose that I have: py def get_attrs(things: Iterable[Any], attr: str, /) -> list[Any]: return [getattr(thing, attr) for thing in things] there isn't a meaningful static type that describes Any №1 or Any №2 in isolation.

#

rather, there's some hidden link between the two that also depends on the value of attr

#

Another example: ```py
class Key[T]:
pass

class TypeMap:
def init(self) -> None:
self._store: dict[Key[Any], Any] = {}

def get[T](self, key: Key[T]) -> T:
    return self._store[key]

def put[T](self, key: Key[T], value: T) -> None:
    if key in self._store: raise KeyError("Key already exists")
    self._store[key] = value

``` there does not exist a hypothetical static type that could replace Anys in dict[Key[Any], Any]

mint inlet
trim tangle
#

Maybe the paragraph I quoted is an oversimplification?

trim tangle
#

I think that got me even more confused

winter lintel
brisk hedge
# trim tangle The definition in the article seems to suggest that `Any` is a stand-in for a st...

the idea is that certain types are too complex to represent in notation or encode in the rules of the type checker

e.g. the (most precise) type for getattr can be mathematically modelled as an infinite intersection of function types that map objects with a specific attribute to the types of the attributes

But in practice it's entirely intractable so it collapses to a single function taking and returning Anys

#

This is castagna et al.'s formalization and I think it's immensely useful, and has already been for developing the elixir type system (another system added on to a dynamic language)

#

Though they prefer the name Dyn to refer to the fact that the precise type can only be known dynamically at runtime; Any has a lot of baggage as a name

#

Fundamentally a static type represents a single set of values (could be any set, but only those representable by a type checker get given notation), and every gradual type represents a set of possible static types (of which 1 is correct at runtime); Any is then a gradual type representing a set of all static types

#

There is subtlety with quantifiers, e.g. gradual types are allowed to represent fresh type parameters, but that is the basic idea

subtle cipher
#

I'm encountering an issue while working with mocks and would like to ask a question.

class A():
    field: MagicMock

def f(obj: A):
    # IDE type hinting not working
    obj.field

In this situation, the field of the parameter obj is not recognized as a MagicMock by the IDE (no type hinting).
Types like int work fine, but mock types are not hinted.
I suspect this happens because Mock inherits from NonCallableMock, but I haven’t been able to find the exact reason.

I'm using VS Code as my IDE and Python 3.12.10.

terse sky
#

How would you want them to be hinted?

#

Am I wrong in thinking that the Mock type allows all method calls, and saves all the calls to a dict for later inspection, or something like that?

subtle cipher
#

Sorry for the confusion. I was talking about the "field" itself's type hinting.

image.png
terse sky
#

My guess is that MagicMock is something too dynamic for the type checker to really do much with. But also the whole point of this type seems to be that it can mock anything, right?

#

If it weren't typed Any then you'll get type errors when using it as intended

#

So on second thought perhaps the Any type is intentional?

#

Like say you have a function def foo(x: Database)

#

You want to be able to pass the MagicMock into foo without a type error

rare scarab
#

Is MagicMock a descriptor? (defines def __get__(self, object, owner))

subtle cipher
rare scarab
#

Usually you save the mock as a local var, mock it, then after it finishes, check its stuff.

#

I think the point is you don't want any of the assert/state of the mock to be exposed to any consumers.

#

mock.assert_any_call(): Ok
obj.mock.assert_any_call(): Bad, won't expose the mock for assertions

#

that's how I understand it at least

#

ex: ```py
def test_foo(monkeypatch):
mock = MagicMock()
monkeypatch.patch("x.y", mock)
x.z()
mock.assert_any_call()

subtle cipher
#

The background behind the issue we encountered was to create a utility object related to mocking.
While running tests, we ended up mocking many objects repeatedly, which caused the number of parameters to grow and led to decreased readability.
As a solution, we decided to create a collection of mocks and pass it around to address the problem of having too many parameters.

def mock_conn_closed_receiver_dependency(func):
    func = patch("CursorHandler.get_cursor", return_value=cursor_a)(func)
    func = patch("remove_cursor")(func)
    func = patch("get_watchers", return_value=cursors)(func)
    func = patch("get_watchings", return_value=cursors)(func)
    func = patch("unwatch")(func)
    func = patch("multicast_cursor_quit")(func)

    async def wrapper(*args, **kwargs):
        return await func(*args, **kwargs)
    return wrapper

class ConnClosedReceiver_TestCase(AsyncTestCase):
    @mock_conn_closed_receiver_dependency
    async def test_normal(
            self,
            get_cursor: MagicMock,
            remove_cursor: MagicMock,
            get_watchers: MagicMock,
            get_watchings: MagicMock,
            unwatch: MagicMock,
            multicast_cursor_quit: AsyncMock
    ):
terse sky
#

I think this is the intended and desirable behavior

grave fjord
trim tangle
trim tangle
brisk hedge
#

Yeah, like a function with two Anys is allowed to have them share type variables and be connected in some way

stray summit
#

anyone aware how to type annotate decorators that add kwonly args


class CallWithCacheFileProtocol[**P, R](Protocol):
    def __call__(
        self, *args: P.args, cache_file: os.PathLike | None = None, **kwargs: P.kwargs
    ) -> R: ...
``` errors claiming that nothing can come after the arguments from the paramspec
stray summit
echo knot
#

Is there a way to create generic type aliases before python 3.12 (before PEP695)?

I would like to have something like:

class HasResult(Generic[T]):
   ResultT: TypeAlias = T
class MyClass(HasResult[int]): ...

So that I can use MyClass.Result as a type annotation?

#

If not, is there any alternative close to it other than defining the type alias (Result = int) inside MyClass?

trim tangle
echo knot
# trim tangle How would it look in 3.12?

I saw this discussion here but since I am not even using 3.12 I did not fully verify if they allowed this:

https://discuss.python.org/t/class-scoped-type-statement-that-references-outer-scoped-typevar/40026

Discussions on Python.org
Class-scoped `type` statement that references outer-scoped `TypeVar`

Using traditional ways to define a type alias, it has been illegal for a class-scoped type alias definition to reference an outer-scoped TypeVar. This makes sense because these traditional type alias mechanisms implicitly bind type variables to their scope, so the use of an already-bound TypeVar is ambiguous. class A(Generic[T]): X1 = list[...

rare scarab
#

I don't think namespaced types are valid outside of modules. If you define a type alias inside a class body, you should use it inside the class.

#

What do you mean "before 3.12"? You posted how it would look. ```py
ResultT: TypeAlias = T

echo knot
#

I kust thought Python 3.12 would allow this with the new "type" sintax introduced by PEP695

type my_type = list[T]

rare scarab
#

Is my_type generic? or is it scoped?

echo knot
# rare scarab Is my_type generic? or is it scoped?

I am not sure about this terminoloy

To make it more clear: I would like it to be defined according to the value of the type variable within the class.

Such that, if I had:

class MyClass(HasResult[int]), MyClass.Result would be a type alias equal to int

rare scarab
#

That's not valid. Per Eric Traut in the thread you linked

I don’t think that’s an issue because a.TypeAlias isn’t a valid type annotation. Variables aren’t allowed in type annotation expressions, so this should already be flagged as an error by a type checker. Likewise, call expressions aren’t allowed in type annotations, so x: A[int]().TypeAlias should result in an error.

echo knot
oblique urchin
#

Very clunky syntax but should behave the same as the 3.12 type statement as far as type checkers are concerned

warped acorn
#

this one looks good

mental ingot
#

How can I write an async function signature that "extracts" the coroutine return type if callback is a coroutine?

import asyncio
import typing

T = typing.TypeVar("T")

async def run(callback: typing.Callable[[], T]) -> T:
    """
    Run the callback and return its output.
    """

    if asyncio.iscoroutinefunction(callback):
        return await callback()  # type: ignore
    else:
        return callback()  # type: ignore

async def main() -> None:
    async def fn_1() -> int:
        return 1

    out: int = await run(fn_1)
    assert out == 1

    def fn_2() -> int:
        return 2

    out: int = await run(fn_2)
    assert out == 2

asyncio.run(main())

I reaaaaaaaaaaaaally want to avoid an overload. I need an overload for something else and I'd like to avoid 4 overloads (2 times 2) 😅

rare scarab
#

Use a union

# 
def run(callback: Callable[[], Awaitable[T]] | Callable[[], T]) -> T:...
mint inlet
#

Note that inference on that might not work well (I’m not sure if mypy gained heuristics for this case)

#

For instance if you pass Callable[[], Awaitable[int]], T can either be int or Awaitable[int]

rare scarab
mint inlet
#

nice, i just remember running into this issue when typing some codebase :'(

mental ingot
#

Nice, that union works with an explicit type (e.g. out: int = await run(fn_1)). But when I use implicit types, I get an error

Need type annotation for "out"

import asyncio
import typing

T = typing.TypeVar("T")

async def run(
    callback: typing.Callable[[], typing.Awaitable[T]] | typing.Callable[[], T],
) -> T:
    """
    Run the callback and return its output.
    """

    if asyncio.iscoroutinefunction(callback):
        return await callback()  # type: ignore
    else:
        return callback()  # type: ignore

async def main() -> None:
    async def fn_1() -> int:
        return 1

    # Error: Need type annotation for "out"
    out = await run(fn_1)
    assert out == 1

    def fn_2() -> int:
        return 2

    out = await run(fn_2)
    assert out == 2
#

reveal_type(out) is Any

trim tangle
main cargo
#

I'm interacting with an external library that has insufficiently narrow type hinting. So I have a lot of:

for parent_sha in commit_parents:
    assert isinstance(parent_sha, bytes)

    parent_commit = repo[parent_sha]
    # Before: parent_commit is a ShaFile
    assert isinstance(parent_commit, Commit)
    # After: parent_commit is a Commit, a subclass of ShaFile

    parent_tree = repo[parent_commit.tree]
    # Before: parent_tree is a ShaFile
    assert isinstance(parent_tree, Tree)
    # Before: parent_tree is a Tree

But there's a lot of this. Is there a way to make this a bit less noisy? Either by having a partial set of types defined for the things I'm using (like a partial stub file?) or by having an inline version of assert isinstance() that just returns the value?

pastel egret
# main cargo I'm interacting with an external library that has insufficiently narrow type hin...

You certainly can. For the types themselves, if you have stubs for the package directly in your repo/path, those will override installed ones, so you could copy and modify the types that way. For the asserts, it's simple enough to put that in a function:

def assert_commit(value: ShaFile) -> Commit:
    assert isinstance(value, Commit)
    return value

def assert_is[Type: ShaFile](value: ShaFile, typ: type[Type]) -> Type:
    if isinstance(value, typ):
        return value
    raise TypeError(f'Expected a {typ.__name__}, but got {value!r}')
main cargo
pastel egret
#

You could mix and match files, but not the whole module. However there's tools that generate a stub from source, you could use those as a starting point.

main cargo
pastel egret
#

For the library itself, by changing the type hints you could make it more egronomic. Define some NewTypes, one for each Git object type - these let you make a "type" for checkers, which is actually just the underlying type at runtime.

#

Then @overload on __getitem__ could have it return the right class depending on those.

main cargo
#

Any reason not to do

assert isinstance(value, typ)
return value

...?

pastel egret
#

Well if you're putting it in a function, why not give a nicer error?

main cargo
main cargo
pastel egret
#

Consistency is good yeah, either works fine.

feral wharf
#

Unless you run with -O

#

Also typing.cast exists

Returns on runtime

echo knot
#

Why is this not allowed by Mypy but allowed by Pyright?

from __future__ import annotations

from typing import TypeVar


class MyClass:
    pass

class MySubClass(MyClass):
    pass

MyClassT = TypeVar("MyClassT", bound="MyClass")

class A:
    def func(self, var: MyClassT) -> MyClassT:
        return var

class B(A):
    def func(self, var: MySubClass) -> MySubClass:
        return var
#

The Typevar is invariant so I should be able give any subclass of the bound as long as both the type of the input argument and the return type are the same, right?

restive rapids
echo knot
#

Would there be a way to do something like that? Allowing subclasses both in input argument and return type

restive rapids
#

are you sure the function is supposed to be generic, not the class?

trim tangle
echo knot
trim tangle
echo knot
echo knot
echo knot
civic hare
#

Does anyone know why I don't get a warning for the return type of the second function? I already have all official python extensions installed (vscode).

image.png
oblique urchin
#

if a dependency isn't installed in the environment pyright is looking in

civic hare
#

Both are imported at the top of the file. Response comes from rest_framework and User is defined by me in another file.

image.png
#

I already made sure rest_framework is installed properly in the virtual environment.

#

this is what the User class looks like

class User(AbstractBaseUser, PermissionsMixin):
    email = models.EmailField(unique=True)
    is_staff = models.BooleanField(default=False)

    objects= UserManager()

    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = []

    def __str__(self):
        return self.email
#

This one does give me a warning. Maybe it has something to do with the User class.

image.png
#

I think it's because User inherits from other classes.

trim tangle
#

If you F12 into Response, does pyright/pylance find where it's located?

civic hare
civic hare
trim tangle
#

What happens if you do ```py
x: Response = 42

or

y: User = 42

civic hare
civic hare
trim tangle
#

and what do you see when you hover over Response and User?

civic hare
trim tangle
#

Who's actually issuing these warnings? Can you hover over one of them?

civic hare
trim tangle
civic hare
#

Oh, you're right, they're from sonarqube.

#

I might not have properly configured pylance.

trim tangle
#

so you probably have Pylance's type checking mode set to "off"

#

it's configurable in Settings under the Pylance section somewhere

civic hare
#

Let me try.

#

Yeah, it was off.

#

Now I have like 100 errors in my project lol.

#

thanks for the help.

visual bone
#

Not sure if the right channel to ask, but what exception should be raised when a function isn't compatible with the platform it is being executed from e.g. a function that will only work on Linux but not windows or others?

trim tangle
#

well, to put it in more useful terms, it just doesn't define the symbols in e.g. os on unsupported platforms

#

!e

import os
print(os.listdrives())
rough sluiceBOT
# trim tangle !e ```py import os print(os.listdrives()) ```

:x: Your 3.12 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 2, in <module>
003 |     print(os.listdrives())
004 |           ^^^^^^^^^^^^^
005 | AttributeError: module 'os' has no attribute 'listdrives'
trim tangle
#

!e
sometimes uhhh

import platform
print(platform.android_ver())

wtf

rough sluiceBOT
trim tangle
#

!e

import pathlib
print(pathlib.WindowsPath())
rough sluiceBOT
# trim tangle !e ```py import pathlib print(pathlib.WindowsPath()) ```

:x: Your 3.13 eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "/home/main.py", line 2, in <module>
003 |     print(pathlib.WindowsPath())
004 |           ~~~~~~~~~~~~~~~~~~~^^
005 |   File "/snekbin/python/3.13/lib/python3.13/pathlib/_local.py", line 860, in __new__
006 |     raise UnsupportedOperation(
007 |         f"cannot instantiate {cls.__name__!r} on your system")
008 | pathlib._abc.UnsupportedOperation: cannot instantiate 'WindowsPath' on your system
trim tangle
#

In 3.12, it was just NotImplementedError apparently

rare scarab
#

Oh, 3.14 removed the pathlib/_abc.py module

rough sluiceBOT
#

Lib/pathlib/_abc.py lines 48 to 51

class UnsupportedOperation(NotImplementedError):
    """An exception that is raised when an unsupported operation is called on
    a path object.
    """```
rare scarab
#

(continuing to read the backlog)

trim tangle
#

devastating breaking change, I was relying on len(UnsupportedOperation.mro()) in my space satellite code

rare scarab
trim tangle
visual bone
visual bone
tranquil turtle
rough sluiceBOT
#
RuntimeError 

exception RuntimeError```
Raised when an error is detected that doesn’t fall in any of the other categories. The associated value is a string indicating what precisely went wrong.
wet prairie
#

we love pyright

image.png
#

or in my use case

image.png
wet prairie
#

or this:

image.png
#

programming languages funky

feral wharf
spiral fjord
#

Looks like that's inherited from object (which also throws the NotImplemented)

#

To narrow like this use TypeIs

cinder bone
wet prairie
#

It's interesting because it did fix it in this case
The screenshot was before PyRight refreshed

image.png
feral wharf
#

that's assert

#

that'll raise an error if not truthy

oblique urchin
feral wharf
#

does mypy?

oblique urchin
feral wharf
#

neat

cinder bone
#

oh wow I did not know that

candid cove
#

In visual studio code (with both the release and pre-release python extension) the type of the source variable is detected as Any even though it should be Source why is that and how to fix it ?

image.png image.png
#

this bug also occurs when using "Source" as the return type of the load function, the Self return type comes from the typing module.

spiral fjord
#

@candid cove what if you annotate the function as being static?

candid cove
#

you can't annotate a function (at least if you declare it with def) you can only annotate the arguments and the return type

#

do you mean using @staticmethod ? I tried and it didn't fix it

spiral fjord
#

Yeah i meant that

candid cove
#

using @staticmethod even seem to override the return type of the function by Anythis is even worse

#

with @staticmethod

#

without it

#

okay i found the fix

feral wharf
#

doing -> Self means you'll return the same instance.. you are creating a new one

spiral fjord
#

I looked up the PEP

candid cove
# 
@staticmethod + typing.Self: The return type is Any
typing.Self: The return type is Source but source doesn't acquires the type
"Source": The return type is Source but source doesn't acquires the type
@staticmethod + "Source": The return type is Source and source acquires the Source type```
spiral fjord
#

And self is not allowed for static methods

candid cove
#

this shit is weird

feral wharf
candid cove
spiral fjord
# 
Note that we reject Self in staticmethods. Self does not add much value since there is no self or cls to return. The only possible use cases would be to return a parameter itself or some element from a container passed in as a parameter. These don’t seem worth the additional complexity.```
#

So they just want you to do -> Source

feral wharf
#

you can use Self in a classmethod if you cls: type[Self]

oblique urchin
feral wharf
#

ah, cool

candid cove
#

-> Source doesn't work because Source is not defined in the body of the Source class but -> "Source" works fine

feral wharf
#

3.14 / from __future__ import annotations moment

candid cove
#

you will be able to just use Source as a type annotation within the Source class body from python 3.14 ?

#

that would be nice

oblique urchin
feral wharf
#

There is no way to overload a TypeDict with Unpack right?

trim tangle
# candid cove ho really i thought it was like `Self` in rust my bad

It does kinda work like in Rust. For example, in a trait you can do rs trait Clone { fn clone(&self) -> Self; } and this means that if impl Klone for Foo and x: Foo, then (&x).clone() is also Foo.
The reason you can use Self like this in Rust: ```rs
struct Foo { bar: u32 }

impl Foo {
fn new() -> Self { Foo { bar: 0 } }
}
``` is that Rust doesn't have inheritance. In your case, if you did class BetterSource(Source): ..., the BetterSource.load method would have to return an instance of BetterSource

#

This is how that would work:

class Source:
    def __init__(self, *, name: str, is_primary: bool, decoded_key: bytes): ...
  
    @classmethod
    def from_json(cls) -> Self:
        ...
        return cls(name=init_data["name"], is_primary=init_data["is_primary"], decoded_key=private_key)
candid cove
#

yhea that's a good way to replicate this behavior

#

i have some ugly function py def to_client(self, client_class: type[ClientSubclass], addr: Tuple[str, int], secure: bool = False) -> ClientSubclass: return client_class(self, addr, secure)

#

now that i know this i can make it into something cleaner thanks ```py
@classmethod
def from_source(cls, source: Source, addr: Tuple[str, int], secure: bool = False) -> Self:
return cls(source, addr, secure)

#

giving a class as an argument is often shitty

#

@classmethod is cool though

trim tangle
#

Self is a shortcut to a type variable. So you could in theory do ```py
@classmethod
def from_source[S: Source](cls: S, source: Source, addr: tuple[str, int], secure: bool = False) -> S:
return cls(source, addr, secure)

trim tangle
#

go to settings and change "type checking mode" in Pylance from "off" to "standard"

candid cove
#

yhea it was set to off

#

This was surely caused by the "Checking of Any type" being disabled

vague oar
#

why in this example it can't infer that y is the same type as x (tuple[Literal[1], Literal[1]]) and instead infers it as tuple[Literal[1], ...]?

x = (1, 1)
y = tuple(x)
trim tangle
rough sluiceBOT
#

stdlib/builtins.pyi line 1001

def __new__(cls, iterable: Iterable[_T_co] = ..., /) -> Self: ...```
vague oar
#

would be cool if there existed something like this

def f(a: T, x: int) -> tuple[T, ...] of_length x: ...
trim tangle
#

you can always make overloads 🙂

vague oar
#

well yeah, that's how itertools does it, for example

class combinations(Generic[_T_co]):
    @overload
    def __new__(cls, iterable: Iterable[_T], r: Literal[2]) -> combinations[tuple[_T, _T]]: ...
    @overload
    def __new__(cls, iterable: Iterable[_T], r: Literal[3]) -> combinations[tuple[_T, _T, _T]]: ...
    @overload
    def __new__(cls, iterable: Iterable[_T], r: Literal[4]) -> combinations[tuple[_T, _T, _T, _T]]: ...
    @overload
    def __new__(cls, iterable: Iterable[_T], r: Literal[5]) -> combinations[tuple[_T, _T, _T, _T, _T]]: ...
    @overload
    def __new__(cls, iterable: Iterable[_T], r: int) -> combinations[tuple[_T, ...]]: ...

but that's a bit limited

stiff acorn
# 
from typing import Literal, reveal_type

def parse_segments(
    segments: dict[Literal["segment"], list[dict[str, str]] | dict[str, str]] | None,
) -> None:
    try:
        segment = segments["segment"]
    except (KeyError, TypeError):
        raise Exception
    
    reveal_type(segment)

pyright:

Object of type "None" is not subscriptable  (reportOptionalSubscript)
Type of "segment" is "list[dict[str, str]] | dict[str, str] | Unknown"

mypy:

main.py:7: error: Value of type "dict[Literal['segment'], list[dict[str, str]] | dict[str, str]] | None" is not indexable  [index]
main.py:11: note: Revealed type is "Union[builtins.list[builtins.dict[builtins.str, builtins.str]], builtins.dict[builtins.str, builtins.str], Any]"

pyrefly:

ERROR 7:19-38: Can't apply arguments to non-class, got None [bad-specialization]
INFO 11:5-25: revealed type: dict[str, str] | list[dict[str, str]] | Unknown [reveal-type]

ty:

Method `__getitem__` of type `dict[Literal["segment"], list[dict[str, str]] | dict[str, str]] | None` is possibly unbound (possibly-unbound-implicit-call) [Ln 7, Col 19]
Revealed type: `list[dict[str, str]] | dict[str, str]` (revealed-type) [Ln 11, Col 17]
#

Anything I can do to make mypy happy?

#

ty seems to be the only one that matches runtime here

restive rapids
#

why.. do you have a dict statically typed to have 1 key? and i dont think typecheckers check the except to "allow" something in the try

oblique urchin
stiff acorn
#

That works, thank you

stiff acorn
vague oar
#

would be cool if it could infer in this example that the return type is Literal[1, 2, 3] | None and not int | None, though i'm not sure if it would actually be useful

def f(n: int):
  if 1 <= n <= 3:
    return n
  return None

or Literal[3] | None in this example

def f(n: Literal[3, 4, 5, 6]):
  if 1 <= n <= 3:
    return n
  return None
trim tangle
#

Consider this: you have an x: Foo | Bar and only Foo has a quack (int, str) method. Is this good?

def handle(x: Foo | Bar) -> None:
    try:
        x.quack(42, "foo")
    except AttributeError:
        pass
    else:
        # surely we know that `x` is a Foo here
        x.foomigate()  # Another Foo-specific method
``` even if it's statically known that `Bar` does not define a `quack` method, this function could accept an instance of a subclass of `Bar` that has a `quack` method, which would call it with potentially wrong arguments. It could even succeed at runtime -- and then we'll access the `foomigate` method which is only present on `Foo`
#

None happens to not suffer from this problem, but that's extra knowledge that isn't incorporated into type checkers

#

Elixir actually seems to have a concept that's similar to this: https://elixir-lang.org/blog/2023/09/20/strong-arrows-gradual-typing/
It has the concept of "strong arrows"

The idea goes as follows: a strong arrow is a function that can be statically proven that, when evaluated on values outside of its input types (i.e. its domain), it will error.
but this would be quite tricky to add to Python

stiff acorn
indigo halo
#

When I want to make a factory for a class, can I somehow Unpack the existing init of the class to type the factory, or do I have to maintain a TypedDict with redundant information about the constructor arguments and types?

feral wharf
#

Latter unfortunately

vague oar
#

What about something like this?

from typing import TypedDict

class MyClassParams(TypedDict):
    name: str
    age: int

class MyClass:
    def __init__(self, params: MyClassParams):
        self.name = params["name"]
        self.age = params["age"]

def factory(params: MyClassParams) -> MyClass:
    return MyClass(params)

||nvm, this is wrong||

brazen jolt
# vague oar What about something like this? ```py from typing import TypedDict class MyCla...

at that point, this would probably be a lot better, the only issue is that it's kw only (no positional args):

from typing import TypedDict, Unpack

class MyClassParams(TypedDict):
    name: str
    age: int

class MyClass:
    def __init__(self, **kw: Unpack[MyClassParams]):
        self.name = kw["name"]
        self.age = kw["age"]

def factory(params: MyClassParams) -> MyClass:
    return MyClass(**params)
#

if you can't accept kw only, afraid you're gonna have to just re-type it

feral wharf
#

Or take a namedtuple / specific object for the parameters instead of shrug

mint inlet
#

You can use a decorator to update the signature to look like whatever __init__

brazen jolt
# mint inlet You can use a decorator to update the signature to look like whatever `__init__`

type checkers still won't understand it though, and with the current type-system, it's just not possible to express this (without repetition)

but yes, you could technically make it work on runtime just from the typedict class, but even then it has issues, first, you'd need to access the defined attributes for the typeddict, you could do this through __annotations__, but it's not a great approach, not to mention that you're relying on the order of the __annotations__ dict, since you're doing positional stuff, which just seems like a bad idea

mint inlet
#

Sorry I mean something like def decorator(constructor: Callable[P, Any]) -> Callable[[Callable[…, R]], Callable[P, R]]

#

Or extract whatever necessary via a protocol

#

Unfortunately not possible to do it for kw-only args though

brazen jolt
mint inlet
#

It would let you do:

@decorator(TheClass)
def factory(*args: Any, **kwargs: Any) -> TheClass:
  return TheClass(*args, **kwargs)
brazen jolt
#

oh I see, like to avoid the need for the typed dict entirely, that's interesting

#

not sure I'd put something like that into an actual codebase though

btw, you could improve the deco signature by making the constructor be Callable[P, R], instead of just Callable[P, Any], since you're not actually passing __init__, rather the class itself, and calling that gives you back an instance, that way, you get a little more type safety when using that deco

terse sky
brazen jolt
terse sky
#

that's a dataclass?

#

oh it's not

#

hah

#

that trick doesn't work with a dataclass?

brazen jolt
#

dataclasses can't be unpacked

terse sky
#

that's extremely unfortunate

brazen jolt
#

there's no supported way to do this

#

yeah

terse sky
#

personally I would probably still stick with the dataclass version

brazen jolt
terse sky
#

it's not ideal have to repeat the dataclass name but at least it's super simple and reusable

brazen jolt
#

but it's not easy

terse sky
#

yeah, definitely not

brazen jolt
#

since it's hard to design a nice API

terse sky
#

I had a wrapper around some subprocess stuff, which threw a much nicer exception on error

#

because the default exceptions suck a little

#

well, a lot, actually

brazen jolt
# brazen jolt since it's hard to design a nice API

like, think of this:

from typing import overload, Unpack, TypedDict

class PersonDict(TypedDict):  # Mirror for kwargs
    name: str
    age: int

@overload
def foo(*args: Person) -> None: ...
@overload
def foo(**kwargs: Unpack[PersonDict]) -> None: ...

def foo(*args, **kwargs) -> None:
    if args and not kwargs:
        name, age = args  # Positional
    elif kwargs and not args:
        name, age = kwargs["name"], kwargs["age"]  # Keyword
    else:
        raise TypeError("Use either positional or keyword args, not both")

foo("Alice", 30)          # Positional (type-checked as Person)
foo(name="Alice", age=30) # Keyword (type-checked as PersonDict)
terse sky
#

and yeah that was kind of miserable. I don't remember if I just copy pasted all the arguments, just the ones I cared about...

brazen jolt
#

you need to somehow annotate both *args and **kwargs if you wish to have support for both

#

and that means you need to handle that, since it's not the same as just def foo(name, age)

terse sky
brazen jolt
#

what I mean is that: {"name": "Jake", "age": 19} is PersonDict just as much as {"age": 19, "name": "Jake"} is

terse sky
#

I mean in general, yes, but in this specific case we're calling Unpack and the client is passing kwargs to us, not constructing a typed dict that we later need to unpack

brazen jolt
#

even if you could trust the order, you still have the issue with the API being wrong

#

since you're adding that annotation to **kw

#

that only takes keyword args

brazen jolt
terse sky
#

right, the syntax here is more the issue here than ordering

#

but I'm sure there's all kinds of miserable edge cases too when you get into the weeds

brazen jolt
#

definitely, like I mentioned, the ordering could be better enforced with say a namedtuple

brazen jolt
terse sky
#

i don't really see the issue with ordering as is, is what I'm saying, there's no actual ambiguity

brazen jolt
terse sky
#

hmm, that could well be. I don't have a lot of experience with TypedDict

#

for a dataclass I would seriously expects that fields always gives you the fields in declared order

#

it would be really strange and potentially sometimes even problematic if it didn't

#

the order of the fields matters for many of the methods that dataclasses generate already

#

so it would basically be going out of its way to say "screw you" to the user if fields didn't follow that

brazen jolt
#

yeah, but dataclasses seem like a massive overkill for this

terse sky
#

I don't really see how it's overkill personally

brazen jolt
#

that's why I mentioned that namedtuples would be more likely to see here

terse sky
#

I mean for starters they can support positional arguments and defaults

#

oh, you mean for Unpack specifically

#

yes, could well be

#

I thought you meant it was overkill for the original problem

brazen jolt
# 
from typing import OrderedTypedDict, Unpack, PositionalUnpack  # Hypothetical!

class Person(OrderedTypedDict):
    name: str  # Position 0
    age: int   # Position 1

def foo(*args: PositionalUnpack[Person], **kwargs: Unpack[Person]) -> None:
    if args and not kwargs:
        name, age = args  # Positional unpack
    elif kwargs and not args:
        name, age = kwargs["name"], kwargs["age"]  # Keyword unpack
    else:
        # you could have a better handling to this which can combine
        # the args & kwargs, but it's not trivial, also, you'd need
        # to prevent duplicate args
        raise TypeError("Use either positional or keyword args, not both")
    

foo("Alice", 30) # works
foo(name="Alice", age=30) # maybe works, if you ignore the issue of not all pos args being specified
foo("Alice", age=30) # could work with proper handling, type-wise, it'd work (with same assumption as above)
foo("Alice", name="Bob")  # problem, since this works type-wise according to the signature!
#

here's some hypothetical example of why this is really tricky @terse sky ^

#

maybe with something like def foo(***full_args: Unpack[Person]) -> None it'd be easier, but that'd no longer be a typing-only feature

rare scarab
#

What if you used a NamedTuple?

brazen jolt
rare scarab
#

combine it with something similar to inspect.signature(...).bind(*args, **kwargs)

brazen jolt
#

yeah, I know you can do that to handle that logic on runtime, but type-wise, foo("Alice", name="Alice") still works, which is bad

rare scarab
#

Ok, so we need a type that can work as an inline ParamSpec

#

What about ParamSpec? ```py
class Person(ParamSpec):
name: str
age: int

def foo(*args: Person.args, **kwargs: Person.kwargs): ...

brazen jolt
#

hmm, that's actually interesting

rare scarab
#

Add some annotations to specify keyword-only or positional-only

#

Add a subclass argument for extras.

brazen jolt
#

yeah, you could add types for that, like with NotRequired that we have with TypeDicts

rare scarab
# 
class Person(ParamSpec, extra_args=Any, extra_kwargs=Any, total=False):
  x: str
brazen jolt
#

realistically, it'd be something different, like TypedArgs, not ParamSpec, since paramspec is already used and it'd be really odd to extend it in this way, especially when it's use is more related to generics than defining the specific args, but this is a really smart way to do it, and I haven't actually seen that suggestion anywhere before

feral wharf
#

@ Jelle 🙏

brazen jolt
#

here's a bit better example of it maybe:

from typing import TypedArgs, PosOnly, KeywordOnly  # Hypothetical!
import inspect

class Person(TypedArgs):
    name: PosOnly[str]  # Position 0
    age: int   # Position 1
    # (note - PosOnly can't be used below "age" line)
    height: int = 170  # Position 3
    # (note - only args with a default value can be used below "occupation" line
    occupation: KeywordOnly[str] = "miner" # Any position (kw only)
    # (note - only KeywordOnly can be used below "occupation" line)


def foo(*args: Person.args, **kwargs: Person.kwargs) -> None:
    bound_args = inspect.signature(Person).bind(*args, **kwargs)
    # or perhaps just Person.signature.bind(*args, **kwargs)

foo("Alice", 21)  # works
foo("Alice", age=21)  # works
foo("Alice", 21, occupation="builder")  # works
foo(name="Alice", age=21, height=150)  # works
foo("Alice", name="Alice")  # type checker should be able to flag this, on runtime, the bind function would fail - even though the call itself would technically work
fleet ember
terse sky
brazen jolt
terse sky
#

in the end i'm not sure it's critical that this stuff have a solution in the end

#

just passing around dataclasses when you want to reuse type signatures is pretty fine and at least in python you can pretty easily convert between dataclass and kwargs

brazen jolt
#

I mean yeah, but the idea above that unalivejoy came up with does seem very interesting

terse sky
#

worth remembering that basically all statically typed languages lack this, have to statically type everything, and nobody is really losing sleep over it

#

you can definitely do stuff like that, but there will always be "one more" thing. Okay, now I want these positional args but with another argument inserted in the middle. Or with another argument removed

#

e.g. my use case with wrapping subprocess.run - in order to support giving better exceptions, I did that by extracting stdout and stderr as text and packing it in the exception

#

so obviously there's now a bunch of keyword arguments to subprocess.run that my wrapper no longer wants

#

just to show a very really world example; this "just one more feature" phenomena for forwarding these type signatures is very real

vague oar
#

Why in this code, it infers gender as Any | None, instead of just None? Am I missing something?

from typing import TypedDict

class Person(TypedDict):
    name: str
    age: int

def function(person: Person):
    gender = person.get("gender")
    print(gender)  # Outputs None
trim tangle
#

Closed typeddicts are supposed by pyright (from PEP728 which is in draft form) and require typine-extensions

terse sky
#

(or anything remotely like this, I mean)

#

You should have a specific reason to use typeddict over a dataclass (usually the reason is adding annotations to existing code that uses dicts)

vague oar
terse sky
#

Yeah, for something like that I would just use a dataclass

brazen jolt
terse sky
#

immediately unpack the response into the dataclass, verifying/converting data immediately

#

yeah, I mean pydantic is ultimately a dataclass flavor

#

I don't really like it but it does sort of have this exact usecase nailed out of the box

#

if I could go back in time I might have actually just introduced attrs to our codebase instead of dataclass. I feel like it took so, so long to get some things in the stdlib dataclasses

#

and iirc attrs still has some nice stuff

brazen jolt
#

yeah, attrs is great

#

even now it still has some features over dataclasses, and esp. if you still need backwards compat for older versions, like when you maintain libs

terse sky
#

I remember waiting on kw_only forever because we had some dataclasses that inherited from each other, and then without kw_only that puts really strong constraints on defaults

#

I remember there's some weird divergence between dataclasses and attrs though I was pretty mixed on

#

like, attrs not having init only variables... or something like that?

brazen jolt
novel notch
#

pylsp seems to misunderstand the type of self._item at the end as LSP completion suggest .get_a even though that's not a member of A.

class A:
    pass


class HasAnA:
    def __init__(self, a):
        self._a = a

    def get_a(self):
        return self._a


class Thing:
    def __init__(self, item: A | HasAnA):
        self._item: A
        if isinstance(item, A):
            self._item = item
        else:
            self._item = item.get_a()
        assert isinstance(item, HasAnA)

    def method(self):
        # self._item.  --> tab completes into:
        self._item.get_a()

Would you say this is a bug in pylsp? Can I typehint this in another way? Adding the commented out assert did not help.

#

And declaring def get_a(self) -> A: doesn't help either.

brazen jolt
#

The most likely reason I see for something like this happening is that pylsp didn't narrow the type down in the isinstance check, so when you assigned self._item = item, it probably still thought that item was of type A | HasAnA, I would consider this as a bug.

I will say though that annotating get_a(self) -> A: is certainly a good idea, you should also annotate your init though, to make sure the a parameter that you're passing will always be of type A too.

#

however, I'm not too familiar with pylsp or it's goals, while type narrowing in the isinstance check is generally what you'd want, if pylsp doesn't attempt to be a full type-checker, it might just be that it's out of scope for them to do such narrowing

vague oar
brazen jolt
#

for example this just works: ```python
from pydantic import BaseModel

class Address(BaseModel):
street: str
city: str

class User(BaseModel):
id: int
name: str
address: Address

data = {
"id": 1,
"name": "Alice",
"address": {
"street": "123 Main St",
"city": "Wonderland"
}
}

user = User(**data)

#

but many people don't like to introduce pydantic if it's just for a few / single response, since it's a fairly big lib and has quite a lot of overhead, so it might be an overkill, depending on how many nested responses you're handling. What's nice though is that it gives you a lot of power

#

If you want something with a bit less overhead, I think there's cattrs which you can use in combination with attrsfor serialization support

feral wharf
#

msgspec >>>

terse sky
#

like cattrs

novel notch
novel notch
silver vine
#

I see that typing.Iterator is marked as deprecated. Does that mean that there is a better way to indicate function return type than something like -> Iterator[int]?

https://docs.python.org/3/library/typing.html#typing.Iterator

Python documentation
typing — Support for type hints

Source code: Lib/typing.py This module provides runtime support for type hints. Consider the function below: The function surface_area_of_cube takes an argument expected to be an instance of float,...

#

Is doing it the same but importing the Iterator from collections.abc better?

oblique urchin
silver vine
#

I know that much, I'm thinking best practices, avoiding deprecation etc.

rare scarab
#

You should prefer collections.abc.Iterator, but typing.Iterator isn't going away any time soon.

#

It exists for pre-3.9 when collections.abc was not generic

terse sky
#

I really wish IDEs would get their act together on suggesting collections.abc exclusively or even first

feral wharf
#

The Python or pyright ext in vscode has a setting for it

terse sky
#

Oh really?

terse sky
#

When I googled I just found this

#

Which suggests the opposite

feral wharf
#

Can't check rn but I remember there being a setting to warn on deprecated typing stuff

vague oar
feral wharf
#

oh it's on the python extension, python.analysis.typeEvaluation.deprecateTypingAliases

stray summit
#

anyone aware of a easy way to treat modules as implementation of a protocol - when i pass modules that implement a protocol to a sequence for it, mypy claims im pasing modules instead

trim tangle
#

can you show the code and the error?

#

Now, if you do ```py
class Generator(Protocol):
def randommm(self) -> float: ...

def generate_two(gen: Generator) -> tuple[float, float]:
return gen.randommm(), gen.randommm()

generate_two(random)
you get
main.py:10: error: Argument 1 to "generate_two" has incompatible type Module; expected "Generator" [arg-type]

 which is an unhelpful error message
#

so perhaps your module doesn't implement some of the items in the protocol

#

(maybe run pyright on that part of the code? it should give a better error)

stray summit
#

i'm using cast now

evne the match statement for the runtime checkable protocols wors - just not the protocol match for a module

trim tangle
#

can you show the protocol and the module?

stray summit
#

hmm - it seems the issue is that im using attributes in the protocol and should have used methods

small eagle
#

I found this here https://typing.python.org/en/latest/spec/concepts.html:

"We can say that a type B is “assignable to” a type A if B is a consistent subtype of A. In this case we can also say that A is “assignable from” B. <...>
For example, Any is assignable to int, because int is a materialization of Any, and int is a subtype of int. The same materialization also shows that int is assignable to Any."

Am I misunderstanding the concept of "assignable to"? Shouldnt Any not be assignable to int because Any is not a consistent subtype?

Nevermind, Any is a gradual type, so it is consistent with int

wild valley
#

I'm having trouble with my pyright runs failing due to /home/alex/workspace/exceptiongroup/.tox/typing/lib/python3.13/site-packages/pyright/dist/dist/typeshed-fallback/stdlib/_typeshed/_type_checker_internals.pyi:52:91 - error: Unnecessary "# type: ignore" comment (reportUnnecessaryTypeIgnoreComment)

#

is this a common issue, and how do I fix it?

#

the error only seems to occur with the latest pyright – v1.1.400 doesn't exhibit this issue

tranquil ledge
#

Not sure there’s a workaround beyond locking the pyright version used *until the fixed version is released.

wild valley
broken isle
#

Hey guy, I've been programming in python for over two decades and I just experienced my first employer who's coding style guide require all variables to be type hinted, including in the body of functions that will never be called nor auto documented (think pytest test functions). Most of the code is of relatively low complexity and type hints can end up being more that 50% of the code written. So far I've noticed many of the developers just put any(this somehow passes code reviews)/Any everywhere and where more precise type hints are used they often end up getting violated later in the code. I have a few questions: 1) is this a new "thing" in the python community, to type everything despite not using jit/compilers? 2) am I crazy for thinking this is over the top and likely enacted by someone who isn't experienced in python? 3) if it's over the top, what argument would you use to convince them?

#

I'm thinking of pointing them to pep 8 and pep 20 regarding consistency/readability.

grave fjord
grave fjord
broken isle
#

Not even... I ran mypy locally and realized how badly things were getting violated. Not to mention having relied on type hint from a function only to realize later it also returned other things at times...

grave fjord
#

You should not use type hints if you don't check them

broken isle
broken isle
spiral fjord
#

in this specific case path can easily be inferred without a typehint by type checkers

grave fjord
#

I'd get the type checker running first

mint inlet
# grave fjord https://github.com/python/mypy/issues/18540

Though I don’t know about the actual viability of this because a) it’s rerunning inference logic for every type hint assignment and b) the type hints are used for some type checkers as hints towards whether a variable can be reassigned to a different type (pyright and ty have this for instance), so just because it’s obvious doesn’t mean it’s unnecessary

terse sky
#

It feels like maybe at your work folks are confused and see documentation as the primary purpos of type annotations? When it's actually checking

broken isle
terse sky
#

documentation is definitely another major benefit, don't get me wrong.
but if the goal were to slap annotations on every single thing then type checkers wouldn't support inference.
And not only do they support trivial type inference like that, but they support much less trivial inference too.
The main "stop" on inference is usually that it's not possible to infer well, or that the code becomes too fragile in the face of minor changes.
As opposed to "this is bad because the primary purpose of type hints is documentation"

#

the general rule of thumb or language rule in python and many newer statically typed languages as well is to annotate function/method signatures and class fields entirely
Local variables are expected to mostly be inferred, with occasional annotations when inference cannot be done, or helps clarity

oblique urchin
#

especially if you don't even run a type checker

stable fjord
#

I'm a big proponent of typing tests, but writing type hints without running a type checker in CI is almost worse than not having any

#

I've been bitten by outdated/misleading type hints, and that really isn't fun

hoary pagoda
#

Is there a way to split an @overload across multiple files?

trim tangle
broken isle
stable fjord
#

with Generator having defaults for sendtype and returntype now it shouldn't be too ugly to type them

trim tangle
terse sky
#

yeah I rarely use Generator directly

#

I probably usually annotate Iterable, not Iterator

#

i think in general I almost never use Iterator

broken isle
#

They might accept Iterator/Iterable.

jolly cipher
#

so the actual impl is in .py and overloads in a separate .pyi

rare scarab
#

What would be the point?

jolly cipher
#

while still having them in place

rare scarab
#

Sounds like they actually want functools.single_dispatch

jolly cipher
#

maybe. i've only vaguely understood they have some overloads and wanted to have them in different files.

#

not that they have some call-based-on-signature functionality idea

hoary pagoda
#

no, I'll be automatically generating overloads, and expect to get thousands, and would like to have them separate so that updates don't have to overwrite the full thousand or so.

terse sky
#

that's pretty terrifying

#

I'm pretty curious how this comes about

feral wharf
#

You could shove them in a (sub)class in some file

hoary pagoda
hoary pagoda
brazen jolt
#

why not just make annotated helper crud functions and use those instead of manual queries? generating overloads for sql literal strings sounds kinda crazy lol, do you just collect all calls and generate an overload for each variant? how do you even know what is the right type during the codegen?

hoary pagoda
trim tangle
#

(it calls into the database inside a procedural macro to figure out the parameter and return types of a query)

hoary pagoda
#

yea, that

#

I was going to not do it this way and make a DSL, but the Python type system is not good enough to make that possible without codegen, and if I'm code genning, may as well just use the database directly.

brazen jolt
# hoary pagoda > why not just make annotated helper crud functions and use those instead of man...

joins and other more complex things SQL queries can do that isn't CRUD.
yeah fair, do you use those that much though? I personally like mostly relying on helper functions + some queries with manual cast/type-hint for the return type for those (that is unless I just go with an ORM). But this definitely sounds interesting
the database provides this information.
huh, so you feed it the query and it gives you back the response structure? if so, this isn't necessarily just a specific table though, it can sometimes be custom things, do you like generate classes/typed-dicts for this too based on this?

hoary pagoda
#

things, do you like generate classes/typed-dicts for this too based on this?
yeah, that's half the point of this silly thing, also allows you to not query for every field in a table if you don't need all of them.

brazen jolt
#

yeah, that actually sounds pretty neat

rare scarab
#

Like ```ts
type T = unknown // use your own type
declare function overloaded<K extends keyof T>(
name: K, options: Parameters<T[K]>[0]
): ReturnType<T[K]>

#

What would be nice in python that would accomplish this is an Overload type that accepts Callable types. ```py
OverloadType = Overload[
Callable[[], tuple[*()]],
Callable[[str], tuple[int]],
Callable[[str, str], tuple[int, int]],
Callable[[str, str, str], tuple[int, int, int]],
Callable[[str, str, str, str], tuple[int, int, int, int]],
Callable[[str, str, str, str, str], tuple[int, int, int, int, int]],
]

trim tangle
#

One day we might get intersections...

#

well, an intersection will be slightly different because of overload priority

hoary pagoda
#

intersection would work well here, at least.

terse sky
#

like, if people want to do arbitrary queries that's fine, but obviously arbitrary queries cannot be strongly typed

#

if you have a more narrow set of queries that supports strong typing, then those narrower queries can just be named

#

get_user_data or what not

hoary pagoda
#

SQL queries are statically typed to at least a useful extent.

terse sky
#

is the idea here, or one of the representative ideas, that you can query e.g. users for some subset of their fields, and then get back a dataclass or what not with only those exact fields?

hoary pagoda
#

yea

#

as well as do joins and get a dataclass with all the fields you get out.

terse sky
#

I mean yeah, this is pretty overboard, imho

#

Even in a statically typed language, nobody would do this

hoary pagoda
#

It exists in at least haskell and rust

terse sky
#

for N fields you can have something like 2^N possible dataclasses

#

I seriously doubt that exact thing exists

#

what's a lot more reasonable is to pass the type in, and to use reflection to determine the desired fields

#

and fill that in

hoary pagoda
#

the plan is to use imperfect static analysis to identify all the queries across the codebase and only generate the relevant dataclasses.

#

Haskell and Rust have the advantage of doing perfect static analysis

terse sky
#

where does this exist in Rust, out of curiosity?

#

And why not have users write the dataclass instead, and pass it in?

hoary pagoda
# 
let countries = sqlx::query!(
        "
SELECT country, COUNT(*) as count
FROM users
GROUP BY country
WHERE organization = ?
        ",
        organization
    )
    .fetch_all(&pool) // -> Vec<{ country: String, count: i64 }>
    .await?;

// countries[0].country
// countries[0].count
hoary pagoda
terse sky
#

but users only write the dataclass they need?

hoary pagoda
#

Yes, even then it is a lot of dataclasses (and also, you have to manually keep them in sync and only get runtime errors if they go out of sync).

terse sky
#

keep them in sync with what?

hoary pagoda
#

with the underlying database

terse sky
#

okay, you're talking about the specific case where you have *

#

because when it comes to specific fields, either the query needs to be kept in sync with the DB, or the dataclass

#

it's no different

hoary pagoda
#

well, it's both the query and the dataclass in the manual case.

terse sky
#

no, because the way it would work is that the query is entirely or partly automatically generated from the dataclass

#

which is how more typical ORMs work (and, tbh, for good reason)

#

i sort of think that overall the costs of this approach are going to easily eat up any benefits from the static typing.
users should, IMHO, anyhow be trying to isolate their program logic from these queries by unpacking the database data into something they can consume - i.e. the dataclasses.

#

like, I dont' want a type generated automatically by the database framework all over my codebase - then if the database changes I have no isolation from it

hoary pagoda
#

You should be mapping it onto a domain model, yes

#

but you mostly do that anyway

terse sky
#

right, but if you're doing that then there's typically a very limited amount of places to benefit from static typing - you immediately map the database data onto the domain model, and then use the model thereafter

hoary pagoda
#

yea, and that's fine, just having all queries statically checked against the schema and their types figured out is good enough

terse sky
#

right, I'm saying the upside is limited. And the cost is kind of high - when you're talking about programmatically generating thousands of overloads, etc.

#

thus, I don't think the idea makes sense in the end. that's all.

hoary pagoda
#

I mean, I can just make the thousand overloads

#

that's not fundamentally an issue, the type checker should just figure it out.

terse sky
#

nobody's denying whether you can, just whether one should 😉

hoary pagoda
#

I don't consider having to make an ugly autogenerated file or twenty a cost

#

The cost is in the fact your code can stop type checking due to reasons not in the versioning and needing a database up during dev.

terse sky
#

I am confident that many other engineers will disagree with you on that, but I guess you'll see for yourself. Good luck!

lunar dune
#

That's 10,000 lines or so of generated overloads for a single function

trim tangle
#

average google library 🥴

soft matrix
#

how you know youre in for a treat

image.png
trim tangle
#

I remember using one of their libraries for google drive, it felt like it was just a useless pass-through and you could just use requests/urllib3/httpx instead

#

(well, if you are able to figure out how to authenticate)

#

like, it didn't even abstract away pagination

terse sky
#

Damn

wild valley
brazen jolt
oblique urchin
upper fox
#

I'm struggling to get the following properly annotated 🙄 Can anyone help? The code and errors are below:

from typing import TypeVar

from django.db.models import Model
from factory.django import DjangoModelFactory

from myapp.models import Contact, ContactList


ModelType = TypeVar('ModelType', bound=Model)


def get_factory(
  model: Type[ModelType],
  include_related: bool = False,
) -> Type[DjangoModelFactory[ModelType]]:

  T = TypeVar('T', bound=Model)
  factories: dict[Type[T], DjangoModelFactory[T]]

  if include_related:
    factories = {
      Contact: ContactWithRelatedObjsFactory,
      ContactList: ContactListWithRelatedObjsFactory,
      [...]
    }
  else:
    factories = {
      Contact: ContactFactory,
      ContactList: ContactListFactory,
      [...]
    }
  return factories.get(model)

class ContactFactory(DjangoModelFactory[Contact]))
  class Meta:
    model = Contact

class ContactWithRelatedObjsFactory(ContactFactory):
  notes = factory.RelatedFactoryList(factory=ContactNoteFactory, [...])

[...etc..]

Errors:

factories.py:27: error: Type variable "T" is unbound  [valid-type]
factories.py:27: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
factories.py:27: note: (Hint: Use "T" in function signature to bind "T" inside a function)
factories.py:31: error: Dict entry 0 has incompatible type "type[Contact]": "type[ContactWithRelatedObjsFactory]"; expected "type[T?]": "DjangoModelFactory[T?]"  [dict-item]
[...]
wild valley
#

why is your typevar inside the function?

upper fox
#

🤔 Not really sure, I'm still trying to teach myself type annotations

wild valley
#

how would T get bound anyway

#

the only bound typevar here is ModelType

upper fox
#

I think I need to learn exactly what it means to bind TypeVar 🤷

The issue with the dict is that the type of the value depends on the type of the key

wild valley
#

why are you even building up that dict on the spot

#

and not on the module level

upper fox
#

I guess I'm trying to avoid a bunch of if/then statements.

#

I suppose I could build it on the module level. Or two of them, bc of the if include_related

wild valley
#

yeah because the dicts look like they're always built the same way regardless of your arguments

upper fox
#

But if I build it at the module level, don't I still need a TypeVar bc the type of the value is dependent on the type of the key?

E.g., the keys are all subclasses of django.db.models.Model, but the values are subclasses of DjangoModelFactory[ParticularModel]

wild valley
#

I don't think there is any way to type a mapping like that

#

I could be wrong ofc

upper fox
#

Gotcha, hmm

#

It's not necessary, just trying to teach myself 🙂

vague oar
#

Would a change for mypy like this be useful?

image.png
# 
    @overload
    def __new__(cls, /) -> Literal[0]: ...
oblique urchin
winter lintel
#

mapping specifc arg types to specific return types? seems kinda cursed to overload return types

vague oar
hearty shell
#

My memory in kinda of foggy on this one, but was it ever possible to propagate args and kwargs forward in Python? Say you wanna subclass a class with a massive __init__ and you don't want to copy paste that in the client code or don't necessary want to have to follow the superclass since it is all sent to super anyway

#

Sorry, I mean in the case where you are pealing some kwargs or adding more args, otherwise that is just not changing __init__ 😅

terse sky
#

you just write *args, **kwargs in the init, and then forward them with the same syntax

#

!e

def bar(x, y, blub=None):
    pass

def foo(*args, **kwargs):
    bar(*args, **kwargs)

foo(1, 2, blub=10)
rough sluiceBOT
terse sky
#

the downside is that to the "outside", it looks like foo accepts everything. you won't get good type checking, or editor hints, on calls to foo

hearty shell
#

By propagate I meant the typing information

terse sky
#

ah, okay, that was not clear from your original post 🙂

hearty shell
#

Ah, apologies

oblique urchin
terse sky
#

this topic comes up and people always mention something but it feels like there's always an issue with it, and even though I have this issue in my own codebase, I've never come out of any of those conversations thinking I should apply this approach back to my codebase

terse sky
hearty shell
#

But in my case I can't really do that since the superclass lives in a library

terse sky
#

if you control the superclass and this is really a priority for you

#

then I would just use a Dataclass rather than a typeddict, personally

#

in essence, a Dataclass is basically like creating a reusable block of data that can easily be type annotated, forwarded, etc, between multiple places

terse sky
oblique urchin
terse sky
#

but again, I feel like this conversation has been had here many times, and each time I learn there's some kind of issue with ParamSpec that makes it not work for me - maybe this doesn't apply to you though

#

yeah, see 😛

#

I can never keep straight what it's really for - the description always makes it sound like it's exactly this use case

hearty shell
oblique urchin
#

however, you can use paramspec to write your own decorator forwarding a signature

terse sky
#

What would that look like for the foo/bar example above?

hearty shell
#

I gave it a try ealier before asking here, the class method didn't really pan out, and the latter for foo bar I hit a problem where you cant concatenate two param specs

trim tangle
# terse sky What would that look like for the foo/bar example above? 
def pinky_promise_has_the_same_signature_as[**P, R](original: Callable[P, R]) -> Callable[[Callable[..., R]], Callable[P, R]]:
    return lambda f: f  # type: ignore

def bar(x: int, y: str, blub: float | None = None) -> None:
    pass

@pinky_promise_has_the_same_signature_as(bar)
def foo(*args, **kwargs) -> None:
    bar(*args, **kwargs)
terse sky
oblique urchin
#

So I guess by most definitions of "working" the answer is no

tranquil turtle
trim tangle
rough sluiceBOT
#

stdlib/functools.pyi lines 116 to 120

def wraps(
    wrapped: Callable[_PWrapped, _RWrapped],
    assigned: Iterable[str] = ("__module__", "__name__", "__qualname__", "__doc__", "__annotate__", "__type_params__"),
    updated: Iterable[str] = ("__dict__",),
) -> _Wrapper[_PWrapped, _RWrapped]: ...```
visual bone
#

If I have an abstraction over reading something, I would use EOFError when there's nothing more that can be read from like a socket or file underneath, what would be a good exception to raise for when the same is true of being unable to send/write anymore?

restive rapids
feral wharf
#

What does that have to do with type hinting thonkG

visual bone
visual bone
hasty phoenix
#

Is there a way to typevar T annotate __getattribute__() without implementing it? I'm making a dynamic enum of type T, but I'm not getting the type hints working:

T = TypeVar('T')

class DynEnum(Generic[T]):
    @classmethod
    def create(cls, data: dict[str, T]) -> DynEnum:
        self = cls()
        for k, v in data.items():
            setattr(self, k, v)
        return self

data = {
    'READ': 0x01,
    'WRITE': 0x02,
}

TASKS: DynEnum[int] = DynEnum.create(data)
v = TASKS.READ
reveal_type(v)  # Type of "v" is "Unknown"
assert v == 0x01
#

Found it:

class DynEnum(General[T]):
    if TYPE_CHECKING:
        def __getattr__(self: item: str) -> T: ...
stiff acorn
#

Is there a way for me to type this:

class MarkdownTable:
    def __init__(self, fieldnames: Any, rows: Iterable[Any]) -> None:
        self.fieldnames = fieldnames
        self.rows = rows

where

fieldnames = ["firstname", "lastname", "age"]
rows = [
    ["Alice", "Smith", "30"],
    ["Bob", "Johnson", "24"],
    ["Charlie", "Brown", "45"]
]

each list in row must be the same length as fieldnames?

restive rapids