type-hinting

[trim tangle]

yay, even more ways to do the same thing

aka the (new) python way

[soft matrix]

well id hope they get rid of the old way

cause its confusing where TypeVars bind for noobs

[vapid quarry]

Python 3.9 supports list[int] that can replace List[int]

Wondering, if we can also do tuple[str, str]to replace Tuple[str, str]?

[void panther]

Yes, all the generics were replaced

you can see in the docs that they are marked as deprecated

[terse sky]

after upgrading, mypy is giving me tons of "error: Library stubs not installed for pytz" even though I'm running it with --ignore-missing-imports

Note: Even if you are using --ignore-missing-imports, mypy will ask to install stubs for packages that had bundled stubs in previous mypy releases. This helps you avoid losing type checking precision from missing stubs when upgrading to mypy 0.900.

🤦‍♂️

[soft matrix]

Why can't you install them?

[rustic gull]

from typing import Any, Callable, TypeVar, cast

F = TypeVar('F', bound=Callable[..., Any])

# A decorator that preserves the signature.
def my_decorator(func: F) -> F:
    def wrapper(*args, **kwds):
        print("Calling", func)
        return func(*args, **kwds)
    return cast(F, wrapper)

I'm not sure if I completely understand from a theory view why a decorator takes a function that is upper bounded by Callable[..., Any] (which is any possible function).

Is a function like def foo() -> int: return 1 a subtype of Callable[..., Any]? Why exactly does that make sense? I have a hard time thinking why it makes sense because I usually think in terms of class hierarchies and class methods. I guess it's saying def foo() -> int: return 1 is one possible function that is defined by the type signature Callable[..., Any], is that right?

[terse sky]

i can, it's just kind of aggressive behavior from mypy that there's no way at all to silence this warning

i was upgrading my dependencies for some other purpose, and in the process it upgraded my mypy version, and then I started getting a pile of stuff coming out.
So I went back to my dependencies file and locked down the version of mypy to the older one for now; will come back to it later

[languid leaf]

Is this possible with Python's type system?

I want to pass a set of keys into a function, and return a class/tuple/dict/typed with those keys, or none.

f([a,b]) -> {a: str, b: str}|None where we know the value is going to be a str

[trim tangle]

That's not possible

You mean, like, f(["foo", "bar"]) being inferred as a class with fields foo and bar (or None)?

[languid leaf]

yeah, basically

from typing import Literal, TypeVar, List, Dict
T = TypeVar("T")
def gen(keys: List[T]) -> Dict[T, str]:
    return {k: None for k in keys}
KEYS = Literal["a", "b", "c"]
seq: List[KEYS] = ["a", "b", "c"]
res = gen(seq)

well, here's the desired effect

[trim tangle]

Ah, I forgot that you can do that

yeah, you can do that, sorry

[languid leaf]

I feel like I could do something more generic in TS

but that's TS and not Python

[trim tangle]

yeah, in TS you can do stuff like ts const isPoint = schema({ x: isNumber, y: isNumber, }) and have it infer isPoint to be Schema<{x: number, y: number}>

Well, we can't have nice things, can we 😔

[hallow flint]

well, since we're on the subject of type comments, callable types like that are used in type comments. since python 2 doesn't have function annotations, you'd use # type: (str, int) -> None to add a type to a function

[hallow flint]

yeah, i find this unignorable warning pretty unfortunate too. the fix isn't too bad though, just add types-pytz to your deps / to whatever env you type check in

[pastel egret]

The reason is that there's two kinds of type checks - nominal typing, and structural typing. Nominal typing is what you'd do with regular classes, where you're checking that the class name matches. Structural typing on the other hand doesn't care about the name, instead checking the structure (methods and types) match.

Callable accepts anything, if calling that object with the given parameters would give the right result. So it accepts matching functions, objects with __call__, classes, etc.

[wary pivot]

Hi, i have to module files which contain both a class (Class A and Class B). Class A uses Class B in its constructor and Class B returns Class A in a method. This works on runtime until i add type annotations because that needs to add a circular dependency (both classes are using references of each other). What can i do?

[soft matrix]

!d typing.TYPE_CHECKING

[rough sluice]

typing.TYPE\_CHECKING

typing.TYPE_CHECKING```
A special constant that is assumed to be `True` by 3rd party static type checkers. It is `False` at runtime. Usage:

```py
if TYPE_CHECKING:
    import expensive_mod

def fun(arg: 'expensive_mod.SomeType') -> None:
    local_var: expensive_mod.AnotherType = other_fun()
```  The first type annotation must be enclosed in quotes, making it a “forward reference”, to hide the `expensive_mod` reference from the interpreter runtime. Type annotations for local variables are not evaluated, so the second annotation does not need to be enclosed in quotes.

[soft matrix]

use this

and see the example they have

[grave fjord]

if you can refactor to use one module file

if the classes are so tightly coupled is it really worth having them in different files?

[trim tangle]

Yep, if you have mutually recursive stuff, put it in one module

[faint dust]

I'm either tired or confused, why is mypy complaining on:

error: Incompatible types in assignment (expression has type "Union[str, List[str], Any]", variable has type "str")

The type can be either str , List[str] or even Any.. so why doesn't it just gobble up str?
I should mention that I'm pretty new to mypy and I've created a project just to learn to code with this in mind

[soft matrix]

list[str] and str arent compatible

so you cant reassign one to the other

[grave fjord]

but it's Union[str]

also that looks like an old mypy output

[grave fjord]

can you make a https://mypy-play.net?

[faint dust]

It's mypy 0.930 so I hope not

[faint dust]

I'll give it a go : )

Looks like pydantic is not included in mypy-play so that might be tricky. It's part of the variable that's causing the issue

Here's the code tho: https://gist.github.com/mypy-play/544ab8cefb364aa72f3162392eae3bf1 / https://mypy-play.net/?mypy=latest&python=3.10&gist=544ab8cefb364aa72f3162392eae3bf1

[soft matrix]

fyi

!pypi multidict

[rough sluice]

multidict v5.2.0

multidict implementation

[soft matrix]

exists

[faint dust]

Nice! That will come in handy for sure, thanks!

[cloud rune]

!e
print("hello world")

[rough sluice]

@cloud rune :white_check_mark: Your eval job has completed with return code 0.

hello world

@cloud rune :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 1, in <module>
003 | NameError: name 'socket' is not defined

@cloud rune :x: Your eval job has completed with return code 1.

001 | Traceback (most recent call last):
002 |   File "<string>", line 3, in <module>
003 | OSError: [Errno 101] Network is unreachable

[sour plinth]

Hello all
I am new to the group I am a beginner Python programmer. I have started a project to monitor the temperature using a BLE sensor. The following code allows me the retrieve the advertising data of the device. Probably going to post to mush information but I figure the more you have the better help I can get.
import asyncio
from bleak import BleakScanner
import binascii
import struct
import time

def detection_callback(device, advertisement_data):
if device.address == 'DD:34:02:06:D0:60':
print("Address: %s RSSI: %d" % (device.address, device.rssi))
print("Service data: ", advertisement_data.service_data)
for data_section in device.details.advertisement.data_sections:
print("Data type: ", data_section.data_type)
print("Data length: ", data_section.data.length)

async def run():
scanner = BleakScanner(filters={"scanning_mode":"passive"})
scanner.register_detection_callback(detection_callback)
await scanner.start()
await asyncio.sleep(5)
await scanner.stop()

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

This is the output I receive.
Address: DD:34:02:06:D0:60 RSSI: -53
Service data: {'0000feaa-0000-1000-8000-00805f9b34fb': b'!\x01\x0f\x0e\x07\x19\xff"\xb8\x00\x0e\xff\xf8\x04\x00'}
Data type: 1
Data length: 1
Data type: 3
Data length: 2
Data type: 22
Data length: 17

In the string for the service data is the temperature. Trying to correct the following code to retrieve the temp and humidity but I am failing. Pretty lost at what to do to decipher this information. I have a very low understanding of converting hex to ascii. Below is the mapping of the data from the manufacture.

Any help would be very appreciated. If I posted this to the wrong, I apologize in advance. Please let me know where I should post this. And again thanks for the help.

Forgot the translation code
import binascii
from pprint import pprint
from struct import unpack

bte = b'!\x01\x0f\x0e\x07\x19\xff"\xb8\x00\x0e\xff\xf8\x04\x00'

pckt = binascii.unhexlify(binascii.hexlify(bte))
print(pckt)
data = {}

data["AdvFlag"] = unpack(">B", pckt[1:2])[0]
data["ID"] = unpack(">B", pckt[2:5])[0]
data["Length"] = unpack(">H", pckt[5:6])[0]
data["Adv Type"] = unpack(">H", pckt[6:7])[0]
data["UUID"] = unpack(">h", pckt[7:9])[0] / 10
data["dew_point"] = unpack(">h", pckt[9:10])[0] / 10
data["temperature"] = unpack(">h", pckt[10:11])[0] / 10
pprint(data)

I do have a data map from the manufacture but can't seem to attach an image.

[rough sluice]

Pasting large amounts of code

If your code is too long to fit in a codeblock in discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the floppy disk icon in the top right, or by typing ctrl + S. After doing that, the URL should change. Copy the URL and post it here so others can see it.

[grave fjord]

can you use

your question doesn't seem related to type hinting - try opening a help channel #❓｜how-to-get-help

[rustic gull]

ohhh I see thank you. Thank you for explaining the nominal versus structural typing part, I wasn't thinking in those terms but it's more obvious now.

[rotund flax]

Use the built-in method .fromkeys() instead.

keys = ["a", "b", ... ]

result = dict.fromkeys(keys)
# or
result = dict.fromkeys(keys, "DEFAULT")

!docs dict.fromkeys

[rough sluice]

dict.fromkeys

classmethod fromkeys(iterable[, value])```
Create a new dictionary with keys from *iterable* and values set to *value*.

[`fromkeys()`](https://docs.python.org/3/library/stdtypes.html#dict.fromkeys "dict.fromkeys") is a class method that returns a new dictionary. *value* defaults to `None`. All of the values refer to just a single instance, so it generally doesn’t make sense for *value* to be a mutable object such as an empty list. To get distinct values, use a [dict comprehension](https://docs.python.org/3/reference/expressions.html#dict) instead.

[hallow flint]

you might find this docs page useful 🙂 https://mypy.readthedocs.io/en/stable/runtime_troubles.html#class-name-forward-references

[brazen jolt]

How would I type-hint that I've added x which is an optional keyword-only argument to the paramspec?

P = ParamSpec("P")
R = TypeVar("R")

def foo(f: Callable[P, R]) -> Callable[P, List[R]]:  # Here, the parameters are actually P + optional kw only `x`
    def inner(*args: P.args, x: int = 3, **kwargs: P.kwargs) -> List[R]:
        results = []
        for _ in range(x):
          ret = f(*args, **kwargs)
          results.append(ret)
        return results
    return inner

@foo
def bar(p: int):
    return p + 2

bar(5)  # Acceptable
bar(5, x=8)  # Should also be Acceptable

[trim tangle]

seems like you can't

[brazen jolt]

oh...

yeah, I was going through the PEP but this just wasn't there

[trim tangle]

I've been pretty disappointed by ParamSpec to be honest...

it is barely useful

[brazen jolt]

I mean, I think it's nice to have in decorators like these where you're changing the parameters, that said, it really should've had support for this, I know there is a way to add positional argument like this easily, or to get rid of one, but yeah, feels like it's just incomplete

[trim tangle]

There's also no way to add a parameter to the left. Or to the right, I don't remember.

[brazen jolt]

if I wanted to point out this issue, where should I submit it? Mypy?

[soft matrix]

Imo they just took the easy way out on this for fear of making things to hard to implement

[soft matrix]

python/typing

[brazen jolt]

ah, alright

[trim tangle]

@brazen jolt TypeVarTuple is actually useful in situations like this ```py
from typing import Callable, List, TypeVar
from typing_extensions import TypeVarTuple, Unpack

P = TypeVarTuple("P")
R = TypeVar("R")

def foo(f: Callable[[Unpack[P]], R]):
def inner(*args: Unpack[P], x: int = 3) -> List[R]:
results = []
for _ in range(x):
ret = f(*args)
results.append(ret)
return results
return inner

@foo
def bar(p: int):
return p + 2

bar(5) # Acceptable
bar(5, x=8) # Should also be Acceptable

if you want to provide a return type annotation, I think you'll have to write out a protocol...

😬

[brazen jolt]

huh, never heard of TypeVarTuple

I don't suppose there's TypeVarNamedTuple for kwargs

[trim tangle]

It's from a draft PEP 646 https://www.python.org/dev/peps/pep-0646/, but pyright supports it

[brazen jolt]

it would be so neat if you could just do Callable[[P.args, x: int = 3, P.kwargs], R]

[trim tangle]

*ten PEPs later...*

[little hare]

*13

[brazen jolt]

yeah, I'll at least submit the issue with this as proposed solution, hopefully it'll get there

[trim tangle]

I know that I only know how to complain... but I am pretty spoiled by TypeScript being good

[little hare]

Explain why promise has to be typed on an async function defintion

[brazen jolt]

glad I don't know anything about TS then, lol

[trim tangle]

I actually don't know

Maybe it makes some things possible, like how ...args needs to be an array/tuple type

(Python didn't do that, so now we need tons of new syntax to enable manipulation on *args/**kwargs, and it doesn't really work yet)

[little hare]

wdym

[brazen jolt]

btw, could you add links to mypy and pyright playgrounds to the description of this channel?

[trim tangle]

they are somewhere in the pins

I think the set of people who read the description but not the pins is very small

[brazen jolt]

yeah, but still, it'd be a lot more convenient

[trim tangle]

also... I can't edit channel descriptions lol

[trim tangle]

In TS you have to do: (...args: string[]) => string instead of (...args: string) => string, unlike in Python.
This seems extraneous, but it allows for manipulation of variadic arguments, like ```ts
type CommonArgs = [{x: number, y: number}, {dx: number, dy: number}]
const move = (...args: CommonArgs): void => {/.../}
const moveAfter = (...args: [...CommonArgs, ms: number]): void => {/.../}

So maybe there's something similar with Promises. But I don't see it

[brazen jolt]

alright, it's there https://github.com/python/typing/issues/1009

[trim tangle]

@brazen jolt What should happen if you decorate these functions?

def f(x: int) -> str: ...
def f(x: int, /) -> str: ...
def f(*, x: int) -> str: ...

if you want a temporary solution, I suggest making a curried function, like (x: int) -> (*P.args, **P.kwargs) -> R

[brazen jolt]

problem is x should really be optional and it's only used very rarely, I feel like this would make things way too messy for something that's usually not going to be utilized anyway

[trim tangle]

true

[brazen jolt]

let's just call those undefined behaviors

[trim tangle]

lol

segmentation fault at type check time 👌

wouldn't be surprised if this will be added in some PEP, to be quite honest

after the new UB in match-case

[brazen jolt]

interesting, there was already a proposal in 612 but it was rejected (I didn't scroll down that far): https://www.python.org/dev/peps/pep-0612/#id4
Their syntax suggestion was py Callable[Concatenate[("x", int), P], R] and their way of bypassing the problematic things was this: ```py
P = ParamSpec("P", banned_names=["x"])

[indigo locust]

for a class method that return the object's own type

def gettype(self: T_Self) -> Type[T_Self]:
  return type(self)

Like that?

[brazen jolt]

do you really mean class method? the first argument should be named cls then, and this wouldn't be what you'd want since the type of cls is Type[YourClass], not YourClass

[indigo locust]

Derp

Derrrrrrrrp

There

XD I'm jumping around in a lot of contexts right now

[brazen jolt]

In that case yeah, this would do it. Then again, why would you ever want such a method?

[cedar sundial]

For the type of kwargs, could you not do something like:

K = ParamSpec(‘K’)
def foo(bar: str, **kwargs: K.kwargs):
…

[indigo locust]

I know its silly. I'm working on a wrapper around another API which does implement this. As a matter of consistency I'm including this

[indigo locust]

Assuming K was bound to some callable, sure I think so

[trim tangle]

You can only use args and kwargs together IIRC

[cedar sundial]

Should be able to use kwargs without args

[indigo locust]

At this stage in the game, I'd say its going to depend on the checker

ParamSpec is barely out of its infancy

[brazen jolt]

pyright will complain if you do, it requires you to use both actually

[hallow flint]

PEP 612 specifies that you need to use both (so if it depends on the checker, the checker is buggy)

[granite plover]

shouldn't this be picked up as pathlib.Path automatically?

its pathlib.Path() / str /str / str

[soft matrix]

Is it typed all the way down the truediv chain?

[pastel egret]

It should be inferred yes if everything there is.

Some reveal_type() sprinkling might allow debugging.

[granite plover]

hm, maybe im missing some types in that chain

would it matter?

[soft matrix]

Maybe

[granite plover]

i would expect Path / Any -> Path

[soft matrix]

It depends if it's an Unknown I think

[pastel egret]

No, because the right hand side could be anything, and therefore return anything else.

[granite plover]

i got

Path / str / str / Optional[str]
``` im guessing thats the issue

i'll just manually put a type hint there then

[soft matrix]

Does Path implement rtruediv?

I feel like it doesn't

[granite plover]

https://github.com/python/cpython/blob/da8be157f4e275c4c32b9199f1466ed7e52f62cf/Lib/pathlib.py#L857-L861

[rough sluice]

Lib/pathlib.py lines 857 to 861

def __rtruediv__(self, key):
    try:
        return self._from_parts([key] + self._parts)
    except TypeError:
        return NotImplemented```

[soft matrix]

oh huh

[tranquil turtle]

You can dopy assert third_optional_str is not None or```py
Path('...') / str1 / str2 / cast(str, third_optional_str)

https://github.com/python/typeshed/blob/master/stdlib/pathlib.pyi#L37-L38

[rough sluice]

stdlib/pathlib.pyi lines 37 to 38

def __truediv__(self: _P, key: StrPath) -> _P: ...
def __rtruediv__(self: _P, key: StrPath) -> _P: ...```

[green marsh]

what is type hinting?

[soft matrix]

giving explicit types to variables

[twilit badge]

How should I make a protocol that basically says that one of n methods needs to be implemented?

it doesn't matter which of them, but there needs to be at least one, there can be more though

[soft matrix]

you need to use a Union of a bunch of protocols

[twilit badge]

urm

so if I have 4 methods, I need to go through all permutations?

that's insane

[trim tangle]

No, just make 4 protocols

And then have Union[Protocol1, Protocol2, Protocol3, Protocol4]

[twilit badge]

ah yeah

still though, it's annoying

[trim tangle]

That's a pretty narrow use case tbh

[twilit badge]

yeah, ig

[trim tangle]

It's good that you can compose smaller elements of the type system instead of having some special thing

[green marsh]

?

[twilit badge]

so much code for one type hint, damn, I might just do x: object and use hasattr checks

[trim tangle]

If you're totally new to type annotations, you can read one of these tutorials:
https://realpython.com/python-type-checking
https://dev.to/decorator_factory/type-hints-in-python-tutorial-3pel

[trim tangle]

Well, the contract that you require on the argument is pretty complex

you can't really describe it more concisely

[twilit badge]

yeah, I get that, it's just I'm not sure it's worth even doing for something that will just be used once and it will be an internal library function that will probably never get used directly by the user

[trim tangle]

that's up to you 🙂

people do read library source code, though

And people damaged by type hints, such as myself, have a hard time figuring stuff out without the types

[twilit badge]

I mean, I like to have type-hints too, but I'm just not sure if I want to do something like that won't really have an impact on end-users of the library and will just add a lot of cluttered code purely to handle single type annotation of one attribute to an internal function, for now I'll just leave it be and keep the issue in the backlog

[granite plover]

I have a custom logging function passed to various other classes/functions to be used
is there a way to have intellisense pick up that its a function and show me signature and docstring when i hover over it?

[blazing nest]

Then you can comment "this is currently only used once for a typehint* 😅

[blazing nest]

Docstring no (in the former case) because according to your signature (the parameter) any function will do there. You can type it as a Callable[[int, str, etc], returns] or (with docstrings):

class TheCallableThing:
    def __call__(...) -> returns:
        ...

[granite plover]

typing it up as callable still shows it as any other parameter and its bothering me

[blazing nest]

You should get a docstring for the type if you use the second option (you can't specify that it's always equal to a specific callable, at that point why is it a parameter)

[granite plover]

eh, idk how i feel about having to make a class for this

[trim tangle]

If you use typing.runtime_checkable, you'll also be able to use isinstance with the protocols

[twilit badge]

yeah, I know about that and that's really nice, but I don't need it here for my use-case.

[rustic gull]

So if I have a list that will only contain 1 object (Item), how would I typehint it?
I don't think it's l = [Item], so how is it done?

I googled it and it looks like it might be l = list[Item], but I'm not sure.

[brazen jolt]

type-hints for lists don't specify how many objects are in that list

they just talk about what type the items in that list have

[rustic gull]

It's gonna be a list of N Items

Every object in it will be an instance of Item lol

[brazen jolt]

l: List[ItemType] = [item]

[rustic gull]

Alright, and to start it with an empty list it'd be l: List[Item] = []?

[brazen jolt]

depending on the type checker and it's config, this may be assumed by default just by doing l = [item], this is often called a "strict" setting, without it, it may just assume l: list without actually checking if it's elements are all of ItemType, but if you specify it explicitly, no matter this setting, it will always be a list of purely ItemType elements

[rustic gull]

I uh, I don't have a typechecker

[brazen jolt]

yeah, if it only has 1 object obviously every object in it will be of that type, what I wanted to say by that was that maybe you shouldn't be using a list but a tuple instead

[brazen jolt]

I mean, you don't need one, but if you're going through the extra work of adding type-hints, you may as well use one. It will make your life so much easier when debugging things since you'll immediately know that some type shouldn't have been passed into some function for example

I'd recommend pyright

[rustic gull]

Typecheckers can check types and stuff without actually executing the code, right?

[brazen jolt]

yes, they're somewhat similar to linters

[rustic gull]

Ah, perfect.

[trim tangle]

@rustic gull if you don't want to go through the trouble of setting a type checker locally, you can use an online playground
mypy: https://mypy-play.net/
pyright (beta): <https://pyright-playground.decorator-factory.su/ >

[rustic gull]

I'd much rather setup a type checker locally

I've got hundreds of kb of code spread across a lot of files

and with it local it could probably integrate into vs code and throw warnings at me

[brazen jolt]

the python extension actually has a built-in type-checker

it uses pylance which uses pyright

it's just a matter of enabling it

{
  "python.analysis.typeCheckingMode": "basic",
}

this is where you could also put strict for the behavior I've shown above, though that's pretty rare, I'd suggest going with basic and only when you really want it, set strict on per-project basis

(by default, this is set to off)

[trim tangle]

ah, I meant checking out some hypotheses, like "will this type-check"

[rustic gull]

Ah

[trim tangle]

for a big project with imports yes, you will need it locally

[brazen jolt]

yeah, ideally, you could also add CI workflow to auto-run it for every PR/commit

[rustic gull]

So I enabled this and now I have 213 problems in my bot
https://i.skystuff.games/PceqFMYRo2mj.png

[brazen jolt]

there's even an integration to pre-commit which could run it locally before every commit you make on your machine

[trim tangle]

You can also provide more fine-grained configuration with pyrightconfig.json. For example, it's pretty inconvenient that in basic mode, incompatible overrides in subclasses are not checked

[brazen jolt]

I'm not surprised discord.py isn't great for typing

[rustic gull]

yeah lol

[brazen jolt]

especially with things like converters

[rustic gull]

Ah jeez, this is no fun.
https://i.skystuff.games/zOgO3pL6Pfol.png

[trim tangle]

I once added wemake-python-styleguide into a project... I got more errors than there are lines of code

about 2k

[brazen jolt]

def my_cmd(self, ctx: Context, x: MyConverter):
    ...
``` except `x` isn't actually an instance of the converter, it's the result after running `convert` on the input

etc.

[trim tangle]

yeah, it's not a great interface

[rustic gull]

However it is pointing out mistakes that I've made

[brazen jolt]

didn't it check libraries?

[trim tangle]

hm?

well, it's just flake8 with lots of additional opinionated rules

[brazen jolt]

I mean, wasn't it checking contents of .venv, which is why you could've gotten that many erros

[trim tangle]

I ran it specifically on my project

nope, I double-checked

[brazen jolt]

ah, lol, I've never used it but sounds pretty strict

[trim tangle]

Yeah, I don't agree with all of its rules

[soft matrix]

I almost fixed this

It was gonna be in v2

[rustic gull]

Holy jeez, one single file has 86 type check failures

im gonna cry

[soft matrix]

But then Danny killed it

[rustic gull]

another file has 105

[rustic gull]

im using discord.py v2 just fine?

[brazen jolt]

what about some forks? such as pycord?

[soft matrix]

Fuck that

[soft matrix]

The code is fine it just isn't being updated anymore

[brazen jolt]

yeah, I mean I get that, I'm not using those either, but at some point, it will be necessary unless you want to maintain d.py yourself

[brazen jolt]

how?

[trim tangle]

I want to write my own library one day... but I'm so lazy, and the API docs are so horrible

and then, I won't even use it because I don't really want to make discord bots

[soft matrix]

Originally I had made Converters generic so they could be sub scripted and the thing that it was generic over would be the type hint you'd use

[brazen jolt]

I was thinking about this too, but it'd take so much time I'm not sure it'd be worth while since I'm kinda busy and also quite lazy

[brazen jolt]

lol

[soft matrix]

Danny wasn't a fan cause you had to use weak refs and other not nice things to make reloads work

[void panther]

I just want people to use Annotated

and libs to understand it

[soft matrix]

But then I was gonna work on making a parameter function that would just store the metadata about that parameter ie it's converter and when type checking return Any so that you could annotate it however you want

I would be way more happy to use annotated if getitem kwargs were a thing

[void panther]

positional only makes more sense for me there

[trim tangle]

I wish there wasn't this whole situation with deferred annotations to be honest... makes the future pretty unstable

[void panther]

you provide some object that you check for in your lib when reading Annotated, and ignore others, if every lib does that then they can work together

adding kwargs and conflicts through that would introduce a bigger mess

[void panther]

was there any progress on that?

[trim tangle]

and of all solutions, store annotations as strings? if someone told me that I'd think this is a joke!

[trim tangle]

I don't know, but PEP 563 is accepted

[soft matrix]

Idk what this means

[void panther]

I (as a library) expect to read something from annotated, so I provide a class SomeLibMetadata that you pass instances of Annotated to give it values to be read

then an another library can provide SomeOtherLibMetadata without causing any conflicts because the libraries can just filter what they read from it

[soft matrix]

Oh I see

You want Annotated[int, SomeMetaData(), some_other_lib_metadata()]

[void panther]

Yeah

plain objects are ambiguous with who they are for so it's not really as extensible, and kwargs would cause a similar issue

[soft matrix]

I don't really see how this usecase actually works

Like when are you annotating something for one library and using the same type alias for another library

[trim tangle]

I would do something like this: ```py
@register(int, parse_english_number)
@dispatcher.command
async def foo(x: int, y: int, z: str): ...

@register(int, parse_english_number, only=['x'])
@dispatcher.command
async def foo(x: int, y: int, z: str): ...

- with `Annotated`, the annotations get really chunky
- with this approach, you can also define some converters common for sub-commands

[brazen jolt]

yeah, I'd also like something like this

[soft matrix]

My original version of the Converter pr supported this

[trim tangle]

Also, won't this whole annotation inspection break with Python 3.11?

[brazen jolt]

why?

[trim tangle]

it will work for simple cases where you can use typing.get_type_hints, but if you're using annotations from class or function scope, it will not work.

[soft matrix]

I thought that's what co annotations was meant to solve

[void panther]

depends on how the postponed evaluation ends up

the descriptor pep was still just a draft last time I checked

[trim tangle]

yeah, but 3.11 will supposedly have from __future__ import annotations by default, won't it?

Anyway... I am leaning more and more towards thinking that inspecting annotations might not be the best idea to do right now. You don't know if your library will break in a future release.

[soft matrix]

I'd just use inspect.get_annotations and accept that they aren't going to horrifically break everything

[void panther]

I don't think that's set yet, it was supposed to be in 3.10, got pushed back and now it's mostly between it and 649

[trim tangle]

right

so the future is pretty unclear

[soft matrix]

I don't think they will change it for 3.11

[trim tangle]

so I would personally avoid using runtime introspection of annotations for anything remotely long term... just like I'm not going to plan a vacation a few thousand kilometres away during the pandemic

!e
In this regard, would an inline definition of a TypedDict be somewhat future-proof?

from typing import TypedDict

Foo = TypedDict("Foo", {"x": int, "y": str})
print(Foo.__annotations__)

[rough sluice]

@trim tangle :white_check_mark: Your eval job has completed with return code 0.

{'x': <class 'int'>, 'y': <class 'str'>}

[trim tangle]

you can't really turn the annotations into strings here, can you?

[soft matrix]

If they break TypedDict we have big big problems

Unless you've put it in a local scope

Str annotations breaks this right?

[trim tangle]

hm?

[soft matrix]

!e
In this regard, would an inline definition of a TypedDict be somewhat future-proof?

from typing import TypedDict

Foo = TypedDict("Foo", {"x": "int", "y": "str"})
print(Foo.__annotations__)

[rough sluice]

@soft matrix :white_check_mark: Your eval job has completed with return code 0.

{'x': ForwardRef('int'), 'y': ForwardRef('str')}

[soft matrix]

Oh huh

I don't want to think about str annotations anymore

[trim tangle]

🥲

indeed

There is a way to specify command types, actually. Let me send it rq

@soft matrix @void panther @brazen jolt
https://pyright-playground.decorator-factory.su/?gist_id=20f2f28722c419e18af87adbbe7c31fe&filename=example.py
TL;DR: py command = ( Args.empty() .add(integer) .add(string) .add(opt(string)) .add(list_of(integer)) .build() ) type of command is inferred as ```
CommandSignature[tuple[int, str, str | None, list[int]]]

[indigo locust]

What is the right way to type either one of of two classes as a hint

Union[Type[A], Type[B]]??

Type[A | B]?

[soft matrix]

the two are equivalent for mypy and pyright

although they both change the second to the first in a reveal_type

[indigo locust]

Type[A] | Type[B]?

I guess that'd be exactly the same

😛

[soft matrix]

yeah

[faint dust]

What's the go-to way of dealing with dictionaries that have multiple values.
And you want to do a function call on one of them? https://mypy-play.net/?mypy=latest&python=3.10&gist=bd4ee74e5f72ff39a6acc20014b4ce02

storage = {
    'PROFILE_PATH': [
        './profiles',
        '~/.config/profiles',
    ],
    'PROFILE_DB': None
}
storage['PROFILE_PATH'].pop()

It gives Item "None" of "Union[str, None, int, bool, List[str]]" has no attribute "pop".
But I assumed that by defining the value of one of the keys mypy would understand what value we're working with?

[soft matrix]

!d typing.TypedDict

[rough sluice]

typing.TypedDict

class typing.TypedDict(dict)```
Special construct to add type hints to a dictionary. At runtime it is a plain [`dict`](https://docs.python.org/3/library/stdtypes.html#dict "dict").

`TypedDict` declares a dictionary type that expects all of its instances to have a certain set of keys, where each key is associated with a value of a consistent type. This expectation is not checked at runtime but is only enforced by type checkers. Usage...

[brazen jolt]

typeddict

[soft matrix]

use this if its not dynamic

if it is youd need some AnyOf special form (an unsafe union)

[brazen jolt]

class MyDict(TypedDict):
    PROFILE_PATH: List[str]
    PROFILE_DB: None
    ...

my_dct: MyDict = {...}

[faint dust]

Sadly it's dynamic, it's where anyone can store anything 🙂

[soft matrix]

youre out of luck then

for now™️

[faint dust]

Hehe. That sucks a bit ^^ really trying my best to follow best practices and learning this ^^
I guess I could get around it by doing if type(storage['PROFILE_PATH']) is list: before the pop.

[brazen jolt]

you could also do a cast, if you really want to

[soft matrix]

well youd need to do assert isinstance(storage["PROFILE_PATH"], list)

[brazen jolt]

or isinstance check yeah

[faint dust]

a cast? : )

[soft matrix]

its basically telling the type checker you know what something is

[brazen jolt]

from typing import cast

x = "hi"
cast(int, x)  # now type checker things x is an int

[soft matrix]

but i dont think it would be useful here

[faint dust]

ah.. humm

[brazen jolt]

cast(list, storage["PROFILE_PATH"]).pop()

this would probably be enough for most type-checkers, isinstance may be better though and you can add some error handling in case it wouldn't be a list for some reason

[faint dust]

True, probably safest to do the check even tho I'm 100% sure that particular key will never change heh.

Thanks for the help! Out of curiosity tho with the cast, how would you avoid Missing type parameters for generic type "list"?

Because I like that approach for some other things

Would you have to define a variable with List[str] and use that variable instead of list?

[soft matrix]

do list[typing.Any] or just turn off that option in mypy

[faint dust]

Check! : )

You guys have been a great help!

[soft matrix]

thats assuming you dont care whats inside the list

[brazen jolt]

yeah, and if you want it to be a list of strings, you can just do cast(List[str], x)

you don't need any intermediate variables holding List[str] type

[faint dust]

I'm beginning to like some of this typing business more and more ^^

[brazen jolt]

I'd actually suggest doing list[object] instead of Any since Any would turn off the type-checking completely while object will mean you can access the shared properties each object will have

if you really don't know what types are in your list, doing Any is unsafe since it wouldn't complain if you did my_list[2].my_arg, whereas with list[object] it would complain that object doesn't have my_arg attribute, meaning you'd need an isinstance check there. This is a good thing since it can catch a lot of errors before you even get to run the code

I try to avoid Any as much as I can

[soft matrix]

oh yeah thats totally fair

[silver birch]

Not sure if this is possible - I have a sort of container class called "Result" that will eventually contain a specific value once it gets calculated, something like this:

    def __init__(self):
        self._result = NO_RESULT

    def is_ready(self):
        return self._result != NO_RESULT

    def get(self):
        return self._result

    def set(self, result):
        self._result = result```

Is it possible to type hint the result subtype when the class is created? Similar to how List[] and Dict[] work. Something like `future_result: Result[int] = Result()`?

[brazen jolt]

look into generics

!d typing.Generic

[rough sluice]

typing.Generic

class typing.Generic```
Abstract base class for generic types.

A generic type is typically declared by inheriting from an instantiation of this class with one or more type variables. For example, a generic mapping type might be defined as:

```py
class Mapping(Generic[KT, VT]):
    def __getitem__(self, key: KT) -> VT:
        ...
        # Etc.
```  This class can then be used as follows...

[brazen jolt]

T = TypeVar("T")

class Result(Generic[T]):
    def __init__(self):
        self._result: Optional[T] = None

    def is_ready(self) -> bool:
        return self._result is not None

    def get(self) -> Optional[T]:
        return self._result

    def set(self, result: T) -> None:
        self._result = result

[silver birch]

Ah, thanks. This is exactly what I was looking for

[trim tangle]

proof of concept
https://github.com/decorator-factory/py-bot-command-parser

[silver birch]

I might be pushing it here, but does anybody know if it's possible to type hint a generic in a abstract base class with the output of a subclass method?

from typing import Generic, TypeVar, List
from abc import ABC, abstractmethod
T = TypeVar("T")

class AbstractClass(ABC, Generic[T]):
    def get_list(self) -> List[T]:
        return [self.value()]

    @abstractmethod
    def value(self) -> T:
        pass

class ConcreteClass(AbstractClass):
    def value(self) -> int:
        return 1

instance = ConcreteClass()
value = instance.get_list() # I want this to automtically be List[int]```

In this, `value` currently is type List[Unknown] but I'm wondering if it's possible to set this up so it automatically becomes List[int] based on the implementation of value()

[terse sky]

Did you try to inherit from Abstract class[int]

@silver birch

Not sure if I've tried it but that's how it would work in any statically typed language

[silver birch]

Yeah, that works fine. I was just hoping there was some way to make it automatically do something similar based on the return of that method, so I wouldn't need to define it in the class line.

[pastel egret]

Well, the type checker should error if you don't match, since it'd mean the abstract method is being overridden in a bad way.

[dawn glacier]

Does anyone know how to set copy return type correctly?

from typing import TypeVar, Generic
LF = TypeVar("LF")
T = TypeVar("T")


class Smth(Generic[LF]):
  ...
  def copy(self: T) -> T:
    ...

(I mean with bound etc)

[tranquil turtle]

your way is good enough

but you can use TS = TypeVar('TS', bound='Smth')
or def copy(self) -> Self: if your type-checker supports Self

[burnt moth]

guys i have a doubt actually iam kinda new to python
the doubt is that i want a code that like takes int from the input max is 20 for eg the input user gave is 5 then i want my written code to run 5 times or for eg the input given is 7 i want the written code to run 5 times
can u help me with this guys pls

[dawn glacier]

Bound with generic causes some errors with pyright.
But thx for Self

[twilit badge]

How should I define a protocol with a method where I don't care about how it's attributes are named, just that there are 3 of them and first is self?

class MyProto(Protocol):
    def test_method(self, arg1: int, arg2: str):
        ...

class MyClass:
    def test_method(self, x: int, y: str):
        return y * x

x: MyProto = MyClass()  # should pass

[soft matrix]

class MyProto(Protocol):
    def test_method(self, __arg1: int, __arg2: str):
        ...

although if you dont care about self being called self (which from your example shouldnt matter)

class MyProto(Protocol):
    def test_method(self, arg1: int, arg2: str, /):
        ...

[twilit badge]

what's /?

also, one more unrelated question: How could I mark an object as being of both protocol type A and type B? By this, I don't mean Union (being one of the types), I actually mean an object that passes both protocols at once. I'd rather avoid doing ```py
class MyType(A, B):
pass

[soft matrix]

its the positional only marker syntax its 3.8+

[soft matrix]

also you are looking for an intersection type which isnt currently supported so thats the best thing you can currently do

[twilit badge]

wait, so using that would mean I wouldn't be able to do something like MyClass.test_method(a=5) at all?

[twilit badge]

oh ..., are there some plans to add it?

[soft matrix]

yes

[twilit badge]

any ETA?

[soft matrix]

nop

[twilit badge]

is there at least some PEP ready?

[soft matrix]

no

its still being drafted last time i heard

[soft matrix]

yeah so theres no way to make that work at runtime and typecheck time

[twilit badge]

hm, I'll look into this later, for now prefixing the arguments with __ works fine

thanks

[median tangle]

Why am I getting Type variable "R" used in generic protocol should be covariant? I don't know what covariant variable means, this is my code: ```py
from typing import TypeVar, Protocol

R = TypeVar("R")

class Req(Protocol[R]):
def get(self, name: str) -> R:
...
``` I just want to mark that classes with this type have get method that returns the optionally passed generic type

weird, the problem disappeared after I added another function: ```py
class Req(Protocol[R]):
def get(self, name: str) -> R:
...

def insert(self, name: str, val: R) -> None:
    ...

[median tangle]

any ideas?

[trim tangle]

I've been planning to make an article on variance for a long time but I'm unbelievably lazy.
In the meantime, there's this article in the mypy docs: https://mypy.readthedocs.io/en/stable/generics.html#variance-of-generic-types

Extreme TL;DR:

• if R is covariant and A is a subtype of B, then Req[A] is a subtype of Req[B]
• if R is contravariant and A is a subtype of B, then Req[B] is a subtype of Req[A]
• if R is invariant (the default) and A is a subtype of B, then Req[A] and Req[B] don't have any subtype relation

Extraordinarily brief examples:

• tuple[T, ...] is covariant in T
• Callable[[T], int] is contravariant in T
• list[T] is invariant in T

So the type checker is telling you that the first version of Req should be covariant in R, while the second version should be invariant in R (think about why)

[median tangle]

I think I understand what's convariance and invariance, but I'm not so sure about contravariance, why would that be useful? I can't think of any reason I'd want to have a generic which would be a subtype to other generic in which the typevar is a supertype of the original one. (Not sure this sentence actually makes sense). Also, why should attributes to callable be contravariant?

Also, why aren't lists covariant? I'd assume that with something like list[Animal] I could do list.append(dog) which is of Dog class (being subtype of Animal).

[trim tangle]

Re lists:
You can do animals.append(Dog) because this is legal (Dog is assignable to Animal): py dog: Dog = # get dog somehow animal: Animal = dog animals.append(animal) However, this is not allowed (list[Dog] is not assignable to list[Animal]): ```py
def add_cat(animals: list[Animal]):
cat: Cat = # ...
animals.append(cat)

dogs = [dog1, dog2, dog3] # list[Dog]
animals: list[Animal] = dogs
add_cat(animals) # oops!

[proud brook]

So what's the point of animals: list[Animal] = dogs

[trim tangle]

This line will not pass the type checker. It's to demonstrate that list[Dog] is not assignable to list[Animal]

[median tangle]

I think that's where the typing error happens, once you assigned a list of dogs to list of animals, it isn't a copy, it's the actual list and now you can add non-dog there (Unless I missunderstood)

[proud brook]

Oh, I thought the call line won't pass the type checker but the variable assignment will pass the type checker

[trim tangle]

Exactly. But you could assign tuple[Dog, ...] to tuple[Animal, ...].
You can also assign list[Dog] to Sequence[Animal], because list[T] implements Sequence[T], and Sequence is covariant.

[proud brook]

Okay, so list[Dog] to list[Animal] is literally type upcasting?

[median tangle]

ah that's because tuples and sequences are immutable, right?

[trim tangle]

what is "type upcasting"?

[proud brook]

For example list to Sequence

Wait

Okay let me explain

What programming languages do you know other than Python?

[trim tangle]

Rust, Haskell, TypeScript, a bit of C

[proud brook]

It's basically type casting an object of type B to an object of type A where B subclasses/inherits A

[trim tangle]

ah

[upbeat wadi]

Thanks for this, I've been wondering for a while

[trim tangle]

I guess. If you have a variable of type T, you can assign a subtype of T into that variable.

I suppose that's the definition of subtyping 🙂

list[Dog] is not a subtype of list[Animal], so you can't upcast list[Dog] to list[Animal]

[median tangle]

this may be worth pinning

[trim tangle]

Re contravariance:
The simplest example of contravariance is a function. Suppose that you have an EventProducer, which produces AnimalEvents. This is the hierarchy of AnimalEvent:

               AnimalEvent    
              /          \  
             /            \
  DinosaurEvent           MammalEvent
  |           |              |      \
  |           |              |        \
ChickenEvent  TRexEvent   DogEvent  CatEvent

Suppose that you have this interface to the event producer:

def on_dog_event(event: DogEvent) -> None:
    ...

def on_mammal_event(event: MammalEvent) -> None:
    ...

def on_dino_event(event: DinosaurEvent) -> None:
    ...

event_producer.subscribe(DogEvent, on_dog_event)
event_producer.subscribe(DinosaurEvent, on_dinosaur_event)

Think about this:

• can you do event_producer.subscribe(MammalEvent, on_dog_event)? No, because there are mammal events that don't have the dog-specific stuff. For example, CatEvent.
• can you do event_producer.subscribe(DogEvent, on_mammal_event)? Yes, you can! on_mammal_event knows how to handle any mammal event, including DogEvents.
• can you do event_producer.subscribe(DogEvent, on_dino_event)? No, DinosaurEvent and DogEvent don't have any subtype relation

In other words, Callable[[MammalEvent], None] is a subtype of Callable[[DogEvent], None], not the other way around.
For this reason, Callable[[T], None] is contravariant in T.

[proud brook]

How do you make it so that it is?

[trim tangle]

make what?

[proud brook]

dogs: list[Dog] = [dog1, dog2, dog3]
animals: list[Animal] = dogs

[trim tangle]

You just can't do this, because it would lead to problems like the one I showed above

Well, you can do this

dogs: list[Dog] = [dog1, dog2, dog3]
animals: list[Animal] = [dog for dog in dogs]

[proud brook]

So this creates a new list, and thus it is able to hint it a different type than list[Dog]?

[trim tangle]

Yep, you can think of it as doing ```py
dogs: list[Dog] = [dog1, dog2, dog3]
animals: list[Animal] = [upcast(dog, Dog -> Animal) for dog in dogs]

[proud brook]

Upcasting in the languages I know is generally used in polymorphism only, otherwise you will have memory around the upcasted object that's not going to be handled

[trim tangle]

I guess the more general term would be "assignability"

which is also the term used in pyright at least

[median tangle]

took a while to process this, but I get it now, thanks a lot, this was very helpful!

[proud brook]

For example, being able to write an abstract base class Animal and a subclass Dog which has implemented one of its methods, write something like:

from abc import ABC

class Animal(ABC):
    @abstractmethod
    def call(self) -> None:
        pass

class Dog(Animal):
    def call(self) -> None:
        print("Woof woof!!!")

def animal_call(animal: Animal) -> None
    animal.call()

Now, you can pass a Dog object into animal_call and it will work.

This is done by doing something like

dog: Dog = Dog()
animal_call(dog: Animal)

But in Python you don't have to convert it to an Animal object manually somehow, do you?

[trim tangle]

You can actually build the whole system out of this rule: the function type A -> B is covariant in B and contravariant in A.

Then you can think of variance of a class type like this: the variance of a class as a whole must be compatible with the variance of each method.

Here Foo can be covariant in T because bar is covariant in T and baz is covariant in T ```py
class Foo(Generic[T]):
def bar(self) -> T:
...

def baz(self) -> tuple[T, T]:
    ...

Here `Foo` can be contravariant in `T` because `bar` is contravariant in  `T` and `baz` is contravariant in `T`  ```py
class Foo(Generic[T]):
    def bar(self, t: T) -> None:
        ...

    def baz(self, ts: tuple[T, T]) -> None:
        ...

Here Foo can only be invariant in T because bar is invariant in T. ```py
class Foo(Generic[T]):
def bar(self, t: T) -> T:
...

def baz(self, ts: tuple[T, T]) -> None:
    ...

Here `Foo` can only be invariant in `T` because `bar` is covariant in  `T` but `baz` is contravariant in `T`  ```py
class Foo(Generic[T]):
    def bar(self) -> T:
        ...

    def baz(self, t: T) -> None:
        ...

So that's what your IDE was telling you about your protocol initially

However, __init__ is kinda special: ```py
class Foo(Generic[T_co]): # ok
def init(self, value: T):
self._value

def get(self) -> T:
    return self._value

excuse the giant message

[proud brook]

In C++ it will literally be

class Animal {
    public:
        virtual void call() const = 0;
}

class Dog: public Animal {
    public:
        override void call() const {
            std::cout << "Woof woof!!!" << std::endl;
        }
}

void animalCall(Animal *animal) {
    animal->call();
}

Dog *dog = new Dog();
animalcall(dog);
delete dog;

[trim tangle]

that makes sense

The same works with typeclasses in Haskell and with traits in Rust.

[median tangle]

yeah, I figured that, thanks again though

[acoustic thicket]

@trim tangle what are some of the best things of typescript's type system which python doesn't have? and how come it has them, even though js is roughly as dynamic as python?

or is being dynamic irrelevant

[soft matrix]

Intersection is probably the nicest thing ts has typeof/keyof is cool conditional types are also very cool

[trim tangle]

• intersection types
• conditional types
• manipulation of tuple/object types (e.g. transform [Promise<A>, Promise<B>, ..., Promise<Z>] into Promise<[A, B, ..., Z]>)

how come it has them
My speculation:

• lots of money (Microsoft)
• JS is more popular
• It was designed by Anders Hejlsberg who is responsible for two other successful static languages: Delphi and C#

TypeScript also had/has competition in the form of Flow, Elm and some other alternatives of adding static types to JS

[acoustic thicket]

hmm

[blazing nest]

Also, since TypeScript is a completely different language that transpiles to JavaScript, the development process is extremely quickly.

Right now, for a typing improvement to be added to Python, it takes another small PEP for each feature being added, which has to be considered by the steering council who may not be very experienced with static typing (I do believe they try their best to get experience and knowledge about what they are making a decision of though. Still, they're developers of a so far dynamic language).

If you exclude typing_extensions it takes years for some improvements to make it into Python code

[trim tangle]

Yep, also that

I takes a lot of time to add typing features.

Another issue is that type checkers aren't totally compatible with each other.

So more globally, I would say:

• TypeScript is designed strictly be the people who implement it
• TypeScript is a separate language: JavaScript stays the same for everyone
• I think the most important point: TypeScript was good at capturing already existing JavaScript patterns. Most stuff you do in untyped JS can be typed more or less precisely, whereas common Python patterns are not typable

such as:

• pass-through **kwargs are just not typable
• variadic generics: stuff like asyncio.gather is not really typable even with the TypeVarTuple PEP
• we still haven't figured out what to do with methods. Functions act differently inside a class and outside of it.

[paper cape]

Hi there,

I have question about annotation which a not type hints, hope this is a right place to ask.

I'm developed a library for writing probabilistic models in python, which uses notation similar to probabilistic programming languages:

    class MyModel(Model):
        N: 'constant' = 100
        x: 'free parameter' = 0
        y: 'free parameter' = 3

I.e. annotation specify the role of the attribute in the model, not its type (types are always known anyway in my context).

I'm also experimenting with an alternative notations, where callable annotation are used to represent parameter's prior distributions:

    class MyModel(Model):
        N: 'constant' = 100
        x: lambda z: -z**2 = 0      # log of normal prior, centered at 0
        y: lambda z: -(z-3)**2 = 3  # log of normal prior, centered at 3  

This library works super well for me in my context, I'm using it to run official COVID model in Finland. The questions is: do you feel this is intended use of annotation? Do you know if this use of annotation would be supported in the future, with the slow pivot to typing-exclusive annotation?

[soft matrix]

guido has mentioned that he wants to make type hints are the only thing he wants in __annotations__ (not entirely sure this is a good idea currently) but you could definitely achieve something like what you appear to be aiming for with type hints currently

!d typing.Annotated

[rough sluice]

typing.Annotated

typing.Annotated```

typing.ClassVar

typing.ClassVar```
Special type construct to mark class variables.

As introduced in [**PEP 526**](https://www.python.org/dev/peps/pep-0526), a variable annotation wrapped in ClassVar indicates that a given attribute is intended to be used as a class variable and should not be set on instances of that class. Usage:

```py
class Starship:
    stats: ClassVar[dict[str, int]] = {} # class variable
    damage: int = 10                     # instance variable
```  [`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") accepts only types and cannot be further subscribed.

[`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") is not a class itself, and should not be used with [`isinstance()`](https://docs.python.org/3/library/functions.html#isinstance "isinstance") or [`issubclass()`](https://docs.python.org/3/library/functions.html#issubclass "issubclass"). [`ClassVar`](https://docs.python.org/3/library/typing.html#typing.ClassVar "typing.ClassVar") does not change Python runtime behavior, but it can be used by third-party type checkers. For example, a type checker might flag the following code as an error:

[soft matrix]

!d typing.Final

[rough sluice]

typing.Final

typing.Final```
A special typing construct to indicate to type checkers that a name cannot be re-assigned or overridden in a subclass. For example:

```py
MAX_SIZE: Final = 9000
MAX_SIZE += 1  # Error reported by type checker

class Connection:
    TIMEOUT: Final[int] = 10

class FastConnector(Connection):
    TIMEOUT = 1  # Error reported by type checker
```...

[soft matrix]

i meant this one

[acoustic thicket]

in your opinion, if someone competent were to fork cpython and work on the typing without being held back by PEPs and other formalities, they could make it as good as typescript's? there's not much in the language itself preventing it?

[soft matrix]

FreeParameter = NewType("FreeParameter", int)
class MyModel(Model):
    N: Final[int] = 100
    x: FreeParameter = 0
    y: FreeParameter = 3

class MyModel(Model):
    N: Final[int] = 100
    x: Annotated[int, lambda z: -z**2] = 0
    y: Annotated[int, lambda z: -(z - 3)**2] = 3

[blazing nest]

I think you could get very close yeah.

The reason it won't be 1:1 is because TypeScript can do whatever it wants and then compile that away which you don't have the liberty of doing in Python, and that a lot of Python stuff is extremely complicated in its dynamicness.

The fact that all typings in Python needs runtime fakes with metaclasses and similar probably is quite a big hassle.

[soft matrix]

i think if you stopped caring about backwards compatible stuff you could get rid of all the metaclasses in typing

[trim tangle]

I would rather have a language that transpiler to pure typeless Python.

(like TypeScript compiles to JS, kinda)

TypeScript was able to get (relatively) nice syntax because of that

[void panther]

would the correct typehint for a contextmgr be AbstractContextManager from contextlib?

[trim tangle]

typing.ContextManager

[void panther]

ah that redirect me to AbstractContextManager so I guess that's correct

[trim tangle]

huh?

ah, well

maybe you're right then

[acoustic thicket]

sounds interesting
any major-ish project which attempts that?

[trim tangle]

🤷

[trim tangle]

Well, there's https://github.com/hylang/hy and https://github.com/evhub/coconut

neither adds types, but the idea is there

[acoustic thicket]

hm

[median tangle]

I have one more question about this, how can any type be covariant, I mean, for a type to be covariant, it can only be used with other covariant generics, but since function arguments are contravariant, that type can never be in those arguments. But this poses an issue, if this type can't be in those arguments, how come we only do something with things of that type?

What I mean is: py class Foo(Generic[T]): def bar(self) -> T: # How do we even know what T is here, we never obtained it ... since if it was obtained in __init__ for example, it would already become invariant, since again, function arguments are contravariant.

Could you give me some actual example of a generic class that's useful and covariant , and how would it work, since I can't wrap my head around this. Even for tuple, which I get that it's covariant, on ititialization, it takes elements of that type T, but that immediately make it invariant, so how come it's actually covariant?

[buoyant swift]

i think you're misunderstanding function arguments' contravarianceness (is that a word?). function arguments are only contravariant when you're passing functions to other functions

[trim tangle]

I'm not totally sure, but I think __init__ is somehow treated specially

[median tangle]

huh, that's just weird, I mean I get that it could still be useful in protcols ig, but for full generics, unless __init__ does have some special treatment, I just can't see how it makes any sense

[trim tangle]

Not really, if you have a class like this

class Foo(Generic[T]):
    def bar(self, t: T) -> None:
        ...
``` it can only be contravariant in `T`

[buoyant swift]

o, 🤔

[trim tangle]

which does make sense

[median tangle]

if this was the case though, it would prevent covariance whenever you'd have custom constructor classmethods though, which could be really annoying

[trim tangle]

hmm, seems like class methods in general are also like this

[median tangle]

ig it could technically also be invariant, but there's no reason for that and making T contravariant will allow for assigning your generic to it's subtype, while making it invariant wouldn't. I mean, it is only a type warning, not an actual full error.

[trim tangle]

yep

[trim tangle]

This is ok with pyright ```py
from typing import Generic, TypeVar

T = TypeVar("T", covariant=True)

class Foo(Generic[T]):
def init(self, bar: T):
self.bar = bar

@classmethod
def from_first(cls, bar: tuple[T, T]):
    return cls(bar[0])

[median tangle]

hmm, that's interesting

though not all classmethods are constructors, wouldn't that be incorrect then?

[trim tangle]

I think I'll ask on the pyright repo

[median tangle]

alright, lmk when you have an answer

[median tangle]

mypy is also fine with this btw

though if you had ```py
class Foo(Generic[T]):
...

@classmethod
def bar(cls, x: T):
    return cls(x)

``` both mypy and pyright do complain, even though it's also a constructor. I suppose the reason you didn't get an error in your example was because you used a tuple of T, T and tuple itself is covariant.

[trim tangle]

well... T is covariant in T

[median tangle]

which is fine when it was used in a tuple that takes that covariant typevar, but not fine when used in a parameter, which doesn't expect a covariant typevar there

though it's apparently fine with __init__

[trim tangle]

it seems like a bug to me

[median tangle]

the fact that it isn't reported in __init__?

or that the classmethod constructor doesn't work?

I'm not so sure this would be a bug if both mypy and pyright act in exactly the same way

[trim tangle]

https://github.com/microsoft/pyright/discussions/2837

[median tangle]

looking through PEP 483, the example on covariance does also have a covariant typevar in __init__, though there's no mention anywhere about it having to be excluded from these checks, but it is obvious that it is intended for __init__ to be excluded

[terse sky]

Methods that do type inference have to be treated specially

Maybe I missed something from the convo above

But basically init and any functions that return an instance of the class, they are inferring a type, not acting under the constraint of one, so the issue of covariance and contravariance aren't really present

[trim tangle]

Actually, seems like __init__ is extra special

class Foo(Generic[T]):
    @classmethod
    def make1(cls, bar: T) -> "Foo[T]":  # error: Covariant type variable cannot be used in parameter type
        return cls(bar)
        
    @classmethod
    def make1(cls, bar: tuple[T]) -> "Foo[T]":  # ok
        return cls(bar)

Ughh, who thought this was a good colour for a comment. Fix your contrast, discord

[soft matrix]

highlight.js

being able to turn off covariance checking in parameters would be useful

[terse sky]

Well, even in statically typed languages you have to do this sometimes

[trim tangle]

Is there any chance that Discord employs someone who knows how to configure the colours in highlight.js? 😄

[terse sky]

At least depending on the language

I remember looking at Kotlin immutable and it was littered with unsafe variance annotation

@trim tangle I'm not really sure but it seems to me incorrect to reuse T there

at least, this would not be correct in many static languages

well. i guess class methods are going to be super confusing here basically

But basically you shouldn't use T for non-instance methods at all, it seems to me. To talk about variance you already have ot have an instance with a particular type parameter.
A function that is constructing an instance of a class and nothing else doesn't have any variance associated with it

You really want to do something like:

    @classmethod
    def make1(cls: Type[U], bar: V) -> U: 
        return cls(bar)

and also say that U has the upper bound Foo[V]

something like that? But honestly it starts to get really messy at this point, where you mix python's rather unique notion of classmethod, with generics.

possibly a better idea to just go for free/static methods

[trim tangle]

Can someone explain this to me?
https://github.com/microsoft/pyright/discussions/2837

I might be too stupid to understand this

I never understand Eric's replies

I don't know how to describe this, but this whole typing thing irritates me way too often.

Sometimes I want to just stop using type hints. I know this will hurt maintainability and autocompletion, but it is such a big time consumer. Sometimes you need to redesign a thing to make it play nicely with types. Other times, you realize that a simple thing that you want to express is simply impossible to express... Or possible, but requires a major reorganization. And if you get confused, you ask on github and get an answer which makes completely no sense.

[soft matrix]

Why don't you just ask Eric to explain it more simply

[trim tangle]

maybe

[soft matrix]

I feel like if you don't get it he hasn't done a good job explaining the reasoning

[soft matrix]

Also I know this feeling but I must just be a masochist :^)

I normally just suck it and tell myself it's worth it for end users

But I've never done anything particularly huge for just myself

[terse sky]

I find most of the pain comes with generics

When I'm not using generics it's generally easy and high value

Generics I find makes a mess because it really kicks up the divide between the static and dynamic type systems

That example above with class method is a perfect example

Class method is already doing something very dynamic that's not possible in static type systems really and the interaction with the generics is a mess

[soft matrix]

This is a very interesting opinion

[trim tangle]

Oh, I remember another thing

If you have a private attribute foo: T_co which some other code accesses, it's ok.

But if you replace it with a property with a getter and a setter, you can't have it be T_co because the typevar has to be invariant

I really want to start using some statically typed language. Hopefully I will get to try this on my job. Or at least my next job

[terse sky]

They can be very nice, especially the newer ones with decent ergonomics, i.e. type inference

I have fewer type annotations in Kotlin than python

[trim tangle]

lol yes

you can build a useful Haskell program without any type hints at all

and it's not a new language at all

[terse sky]

Hah yeah although people in industry don't generally do that

[trim tangle]

(whether you can build a useful Haskell program is a separate question 🙂 )

[terse sky]

Lol

[trim tangle]

yeah, Haskell's idiomatic way is to put type declaration for all the top-level stuff

[terse sky]

Yeah it's funny because pythons static type system is pretty expressive

More than many static languages including for example Kotlin I think

But there's so many more warts

[trim tangle]

yeah, it allows for a lot of stuff because it has a dynamic basis

[terse sky]

Then could ever be tolerated if the language were actually statically rypes

[trim tangle]

well, it bleaks in comparison to TypeScript, but that's a different song 🙂

[terse sky]

Yeah havent done much with TS but the type system also seems fairly expressive

It has variadics doesn't it

That's a very very rare and useful feature

[void panther]

Same here, trying pretty much any non trivial typing usually just ends up with me being frustrated and spending twice the time on typing than I did on the implementation. Doing it properly everywhere would most probably help catch some errors but I'd be surprised if the overall dev time was lower. I'd really only put more time into it for some public lib

[trim tangle]

Yep. And you can actually express stuff like asyncio.gather, which the TypeVarTuple pep doesn't address

[terse sky]

I think though that part of spending too much time is a trap though

Again using the example above

[trim tangle]

asyncio.gather being (Awaitable[A], Awaitable[B], ..., Awaitable[Z]) -> Awaitable[(A, B, ..., Z)]

[terse sky]

It doesn't work as well as it should but in the end you can just annotate the function correctly and use a cast in the implementation and move on

[soft matrix]

It originally had a Map special form

[trim tangle]

huh?

[soft matrix]

But it was removed cause they thought it would be too much to implement in one go

[trim tangle]

classic

[blazing nest]

@acoustic thicket an example from our discussion about things taking to long ^^ 😔

[acoustic thicket]

F

[eager vessel]

Is there a way to type hint dictionary like this? 🤔
dict[Type[T], T]

[acoustic thicket]

what's wrong with this?

[brazen jolt]

I'd assume he wanted to define a dict like this generally, outside of binding T to something from args/generic types/..., i.e.: ```py
from typing import TypeVar, Dict, Type

T = TypeVar("T")

a: Dict[Type[T], T] = {int: 0, float: 1.2, complex: 2j+8}
``` which would give you an error that T has no meaning in this context

[acoustic thicket]

oh

[brisk hedge]

One workaround is to define generic functions for interfacing with the dict, and type-ignore the internal code (since the dict would probably have Any in there somewhere)

[brazen jolt]

dict is generic by default, so it already has those definitions

but I was also thinking of something like that: ```py
class TypeDict(Generic[T], Dict[Type[T], T):
pass

issue is, this won't detect dict as it's subtype

you could make it a protocol, but you'll need to define everything that is needed for it to be a dict manually

[brisk hedge]

yes but the T are not heterogenous

[brazen jolt]

the ... what?

[brisk hedge]

so that example wouldn't work

[brazen jolt]

I'm pretty sure doing: py x: TypeDict[complex] = TypeDict({int: 0, float: 1.2, complex: 2j+8}) would work fine (since float and int are treated as subtypes of complex)

[brisk hedge]

It also erases the other type information

and in the general case (storing arbitrary type-instance pairs) you'd need to have [object]

[brazen jolt]

whereas this would cause an error (at least with pyright): ```py
y: TypeDict[complex] = TypeDict({"x": 1})

[brazen jolt]

obviously, but that's not really relevant

[eager vessel]

Exactly 🤔

[brazen jolt]

where?

this just extends dict class, it inherits the typing definitions from dict, which is generic and has proper definitions for each function.

I'm not sure what's the issue you're talking about @brisk hedge, could you give me some examples of where a typing information is ereased?

[brisk hedge]

Yeah it's fine

[brazen jolt]

though again, this has a major problem, being the fact that if you did y: TypeDict[SomeClass] = {SomeClass: SomeClass()} you'd get a type-error since dict isn't a subtype of TypedDict

depending on what you're doing @eager vessel, this may or may not be an issue for you ^^

[eager vessel]

I need to store type-instances pairs in my that dictionary, i guess creating functions that would interact with it could work 🤔

[brazen jolt]

yeah, you could just use TypeDict wherever you used dict before

sadly, you can't just do py class TypeDict(Protocol[T], Dict[Type[T], T): ... since that doesn't produce consistent MRO, as I said, if you actually wanted this to work with dict, you'd need to define all necessary functionalities that a dict has to support

you could probably do it with mapping though

[soft matrix]

In [8]: class TypeDict(collections.abc.Mapping[type[typing.T], typing.T], typing.Protocol[typing.T]):
   ...:     ...
   ...: ```works for me ;)

just ignore the fact this is line 8

[brazen jolt]

interesting, typing.Mapping didn't for me

[soft matrix]

In [7]: typing._PROTO_ALLOWLIST["collections.abc"].append("Mapping")

[brazen jolt]

that's interesting, though pyright doesn't even recognize that import (_PROTO_ALLOWLIST is not a known member of module)

[brazen jolt]

surprisingly, mypy is fine with this, but for some reason pyright isn't and reports that x can't be assigned to that type, yet doesn't report anything wrong with y, mypy correctly reports that y can't be assigned to that type while x is correct: ```py
import typing
from typing import TypeVar, Type, Protocol, Mapping

T = TypeVar("T", covariant=True)
typing._PROTO_ALLOWLIST["collections.abc"].append("Mapping") # type: ignore # _PROTO_ALLOWLIST is not recognized, but it's there

class TypeDict(Mapping[Type[T], T], Protocol[T]): # type: ignore # Mapping was allowed to be mixed with protocol
pass

x: TypeDict[complex] = {int: 0, float: 1.2, complex: 2j+8}
y: TypeDict = {"x": 1}

pyright doesn't mind this though: x: TypeDict[complex] = {complex: 2j+8} (neither does mypy)

probably the first time I see mypy supporting something more advanced and pyright failing

[tranquil turtle]

there are trick:py class TypeDict(Generic[T], Dict[Type[T], T): __class__ = dict[type[T], T] # type: ignore[assignment] some type-chekers accepts it and can cast dict to TypeDict

[brazen jolt]

you probably wanted : not = for assignment

but yeah, this could probably work

[tranquil turtle]

no

=

no, it doesnt work for mypy(

[brazen jolt]

it does for me

[tranquil turtle]

can you give example?

[trim tangle]

class TypeMap(Protocol):
    def __getitem__(self, type_: type[T], /) -> T:
        ...

[tranquil turtle]

should you also provide __setitem__, __delitem__, ... ?

.keys(), .items(), .values(), ...

[brazen jolt]

yeah, that's what I suggested initially, if generics wouldn't do

[trim tangle]

depends on what you need 🤷

[tranquil turtle]

class TypeMap(dict[type[T], T], Protocol[T]):
    ...```will this work?

[brazen jolt]

wait, actually it doesn't, but it doesn't for pyright either:

[brazen jolt]

no

dict isn't compatible with Protocol

[brazen jolt]

you could only do mapping like I did in the example above

and even then you need to use hacky ways to allow it to actually be created

[trim tangle]

dict[type[T], T] means a fundamentally different thing: for a given, concrete T it has keys which are type[T] and values which are T. For example, when T is int it can be {int: True, bool: -7}.

[brisk hedge]

Yeah you basically need to use a non-generic Protocol that wraps all the dict methods (so the typevars are only bound within each method)

[brazen jolt]

I'd assume that's desired with an approach like this, you only require keys to be a subtype of the values type, not necessarily the class of value type

[brisk hedge]

Because the issue is with dict[k, v] being homogenous (not what is wanted unless you want type[object] everywhere)

[brazen jolt]

I don't understand what you mean by this

[brisk hedge]

a dict containing arbitrary type-instance pairs is only type safe as dict[type[object], object]

[brazen jolt]

yeah, that's the default

[brisk hedge]

which of course doesn't prevent the pairs from being different types

so you need typevars to ensure {bool: "foo"} isn't valid

but the typevars need to be at the method level as opposed to class level

[brazen jolt]

oh, yeah, that's true

[eager vessel]

I would try that rn 🤔
type[T] couldn't be used rn afaik

[brazen jolt]

yeah, so you'll have to use a protocol @eager vessel

and depending on what support you need, you can only add some methods to the protocol. i.e. if you won't need the support for something like get, you don't need to specify it in the protocol, if you only want read-only mapping, just getitem may even be enough

[eager vessel]

I guess this works for me:

_CacheKey = tuple[Type[_T], Optional[Any]]


class _InstanceCache(Protocol):
    def __setitem__(self, key: _CacheKey, value: _T) -> None:
        ...

    def __getitem__(self, key: _CacheKey) -> _T:
        ...

    def __contains__(self, key: _CacheKey) -> bool:
        ...


class _BaseContext:
    def __init__(self, container: Container):
        self.container = container
        self.cache: _InstanceCache = {}
        self._token: contextvars.Token[_AnyCtx]

[brazen jolt]

yeah, that looks fine, though you'll probably want __getitem__(self, key: _CacheKey, /) to signify that these arguments are positional only and it's not actually viable to use _InstanceCache.__getitem__(key=x)

[eager vessel]

Yeah

setitem wouldn't be called directly though 🤔

[brazen jolt]

you can't be sure of that

[eager vessel]

"I will not do that"

"I swear"

[brazen jolt]

what if someone subclasses your class

or just uses it with it

[eager vessel]

Well, it's private for a reason

[brazen jolt]

context = _BaseContext(...)
context.cache._setitem__()

still though, why not do it properly

it's just a matter of adding / to the end of those methods anyway

[eager vessel]

I will, just saying that it's private to that module

[brazen jolt]

people sometimes use private classes to

[brisk hedge]

That is not the responsibility of type hints to consider

[brazen jolt]

if the protocol doesn't use /, it does enforce the need of the subtypes to support that specific keyword argument. i.e. this wouldn't match the protocol: ```py
class Foo:
def setitem(self, other_name_than_key):
...

.. # other methods

x: _InstanceCache = Foo()
``` wouldn't be valid because Foo doesn't support __setitem__ with key kwarg

[brisk hedge]

If it's a private protocol it won't matter

If you use private APIs that's your problem basically

[trim tangle]

yeah, if you mess with someone else's private thing, there's no guarantee that anything will work at all

[brazen jolt]

sure, but would it even allow dicts then? I don't think dicts support dict.__setitem__(key=x, value=y)

[restive kelp]

Most languages would narrow the concrete type of T at the if statement, so this would be allowed. Python gives an error. Can I do anything about it?

T = TypeVar('T')

def foo(t: type[T]) -> T:
    if t == bool:
        return True
    else:
        raise NotImplementedError()

pyright gives

Expression of type "Literal[True]" cannot be assigned to return type "T@foo"
Type "Literal[True]" cannot be assigned to type "T@foo"

[tranquil turtle]

use overload

[restive kelp]

like function overloading? I didn't even know python had that

[tranquil turtle]

from typing import overload, NoReturn
@overload
def foo(t: type[bool]) -> bool: ...
@overload
def foo(t: type[T]) -> NoReturn: ...

# actual implementation:
def foo(t: type[T]) -> T:
    if t == bool:
        return True
    else:
        raise NotImplementedError()

[restive kelp]

oh I see

[tranquil turtle]

no, typing overload
there are no built-in function overloading in python

[restive kelp]

it's slightly less convenient bc the actual type is this, but ig if that's what I have to do

    def getInput(self, inputType: type[T], default: T, ident: str, prompt: str) -> T:

I actually have this situation going on. Where would I put the overloads?

class InputSource(ABC):
    @abstractmethod
    def getInput(self, inputType: type[T], default: T, ident: str, prompt: str) -> T:
        pass

class ConsoleInput(InputSource):
    def getInput(self, inputType: type[T], default: T, _: str, prompt: str) -> T:
      # ...

[eager vessel]

Wait, does type[] actually work in py 3.10?

[restive kelp]

since 3.9

https://docs.python.org/3/library/typing.html#typing.Type

Deprecated since version 3.9: builtins.type now supports []. See PEP 585 and Generic Alias Type.

[oblique urchin]

mypy has been slow in adding support for it but I think it's fully supported in the latest release

[eager vessel]

Hm, it didn't work for me for some reason 😅, ah, thanks pycharm

[soft matrix]

!d functools.singledispatch

[rough sluice]

functools.singledispatch

@functools.singledispatch```
Transform a function into a [single-dispatch](https://docs.python.org/3/glossary.html#term-single-dispatch) [generic function](https://docs.python.org/3/glossary.html#term-generic-function).

To define a generic function, decorate it with the `@singledispatch` decorator. When defining a function using `@singledispatch`, note that the dispatch happens on the type of the first argument:

```py
>>> from functools import singledispatch
>>> @singledispatch
... def fun(arg, verbose=False):
...     if verbose:
...         print("Let me just say,", end=" ")
...     print(arg)
```...

[brazen jolt]

amazing how this docstring doesn't tell you anything about what the decorator actually does and how to use it, at least not without using those attached links. the pydoc's docstring is so much better: ```
Help on function singledispatch in functools:

functools.singledispatch = singledispatch(func)
Single-dispatch generic function decorator.

Transforms a function into a generic function, which can have different
behaviours depending upon the type of its first argument. The decorated
function acts as the default implementation, and additional
implementations can be registered using the register() attribute of the
generic function.

[brazen jolt]

yeah, but it's not impossible to make one yourself (for exmple: https://gist.github.com/ItsDrike/afc168f5fa592cb50c82a8ade7218c7f), basically every linter and type-checker will scream at you, but it's possible

[soft matrix]

You could make the overload function when type checking be typing.overload

Although bodies being present might confuse them

[brazen jolt]

interesting idea, that'd probably do the trick with type-checkers, though linters would probably not like overriding already defined functions

[rare scarab]

I hope we get a PEP soon for inline declaration of generics.

But I have no idea how it would be declared.

[void panther]

Well sometimes running a proper typechecker would be good, after a month I've noticed I accidentally used t.Optional[None] as a typehint

[trim tangle]

That didn't work out very well 😄

[granite plover]

im using importlib and its a pain trying to type hint things, currently im hacking together module types with types.ModuleType and nested classes

is there an easy way to type up something that comes from importlib.util.module_from_spec

[oblique urchin]

typeshed just has it return ModuleType: https://github.com/python/typeshed/blob/master/stdlib/importlib/util.pyi#L32

[rough sluice]

stdlib/importlib/util.pyi line 32

def module_from_spec(spec: importlib.machinery.ModuleSpec) -> types.ModuleType: ...```

[granite plover]

alright, i guess that would work, right now im using nested classes and it kinda works

import types
import BaseProcessor # class in module

class Module(types.Module):
  class Processor(BaseProcessor):
    pass

but this looks weird imho

[blazing nest]

Does Python have something like this? https://www.typescriptlang.org/docs/handbook/declaration-merging.html#module-augmentation

I want to patch some of the 3.10 additions to dataclasses into older code.

[oblique urchin]

Not exactly. You could point your type checker to a custom typeshed

[blazing nest]

Ah, that could work 🤔

[trim tangle]

Today I found this module... https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/analyzer/typeEvaluator.ts
My life will never be the same

[oblique urchin]

22k lines 😱

[trim tangle]

yeah, and the nesting is sometimes very deep

I am very scared

well at least they didn't contract "analyze"

[trim tangle]

@oblique urchin do you know any decent guide on making a mypy plugin?

[oblique urchin]

no, never tried it myself

[urban root]

Hi. I have a simple condition like but pylance warns me with a possible unbound warning

if len(l) == 0:
    # code
elif len(l) == 1:
    # code
elif len(l) > 1:
    a = 1
else:
    return

if len(l) > 1:
    print(a) # a is possible unbound

What am I missing here

I feel like I am missing something very simple here and being a derp

[tranquil crag]

why not just put the print statement in the original if statement

[upbeat wadi]

The first two conditions don't define a

Oh nevermind

[urban root]

well the print is there as dummy code

[oblique urchin]

Type checkers aren't going to understand that the condition is the same in both cases

[urban root]

I just wanted to demonstrate lol

oh

so uh, what should I do now

[terse sky]

that's harder to answer without seeing the code in broader context

generally it's not a great thing to have possibly unbound variables though, IMHO

[oblique urchin]

Right. Normally what I'd do is set a = None before the ifs, then check if a is not None below

[terse sky]

yeah pretty much

[urban root]

I see

[terse sky]

that way you always have something bound for a of type Optional[int]

instead of a maybe being bound to int and maybe being unbound

the static type system can deal with the former much better than the latter. And people as well; conditionally having variables is just annoying

[urban root]

yeah that makes sense

Thank you for the advice guys 😁

[grave fjord]

You could handle the len(l) > 1 first

[foggy prairie]

Hey yall, I'm currently testing out Python typing stubs for the project I work in. However, I'm not being able to make use of stubs in separate files.
The following code works as expected:

from typing import NoReturn, Optional, Union, overload

class Missing:
    pass

missing = Missing()

@overload
def fetch(id: str, default: int) -> int: ...
@overload
def fetch(id: str, default: Optional[int]) -> Optional[int]: ...
@overload
def fetch(id: str, default: Missing = missing) -> Union[int, NoReturn]: ...

def fetch(
    id: str, default: Union[Missing, Optional[int], NoReturn] = missing
) -> Union[Optional[int], NoReturn]:
    pass

reveal_type(fetch("", 1))    # int
reveal_type(fetch("", None)) # int | None
reveal_type(fetch(""))       # int | NoReturn

However, as soon as I move the overloads to a .pyi file, the types shown by reveal_type all become Union[int, None, NoReturn]

Could someone provide me with some insight as to what is happening?

[soft matrix]

its probably cause you have an "actual implementation" for fetch

!d typing.overload

[rough sluice]

typing.overload

@typing.overload```
The `@overload` decorator allows describing functions and methods that support multiple different combinations of argument types. A series of `@overload`-decorated definitions must be followed by exactly one non-`@overload`-decorated definition (for the same function/method). The `@overload`-decorated definitions are for the benefit of the type checker only, since they will be overwritten by the non-`@overload`-decorated definition, while the latter is used at runtime but should be ignored by a type checker. At runtime, calling a `@overload`-decorated function directly will raise [`NotImplementedError`](https://docs.python.org/3/library/exceptions.html#NotImplementedError "NotImplementedError"). An example of overload that gives a more precise type than can be expressed using a union or a type variable:

[tranquil turtle]

add py.typed file

[foggy prairie]

proj structure

[foggy prairie]

But I need one (?)

[soft matrix]

as mentioned here you dont need an actual implementation of the function for overload in stubs

oh are you only moving the overloads?

why are you using stubs in separate files anyways?

imo its bad practice unless you absolutely have to

[foggy prairie]

I have a file with 3k+ loc, adding stubs to every function (they need multiple overloads) will make the file very hard to navigate

if you can point me to a pep stating that it is bad practice that would help convincing some colleagues

[trim tangle]

I would rather have it inline than jump to a different file to see what type it is tbh

[soft matrix]

having 3k loc sounds like the problem not adding overloads ;)

[trim tangle]

🙂

[soft matrix]

thats for speed or something

[trim tangle]

yeah, there's a comment at the top

[soft matrix]

(idg the speed comment but idk how that would be compiled)

oh

wait i see

its a load of functions in a function

[foggy prairie]

people in python/typing gitter explained to me that the typechecker is kinda dumb and will only check one of the two, that explains why it doesnt work

i wish it worked tho because it would actually make sense, but oh well

ty!

[soft matrix]

what

it only checks one of the two?

of the files

ic

[foggy prairie]

if pyi exists, it wont actually check the py

[terse sky]

I've always assumed that stubs are just a non-intrusive way of adding type hints

so you can add type annotations to code you don't control

for code you control, I'd just assume it's a given to use type annotations inline

I mean maybe I'm wrong but I'd be surprised if this were in a PEP explicitly because it's just taken as a given?

another use case is probably for code that is mostly implemented in C, I haven't worked on such stuff in a while but I suppose in some cased you'd have functions whose definition never appears in python at all

so you'd need separate stubs there too

[foggy prairie]

you got it right

[terse sky]

not authoritatitve but e.g. this SO gives actual use cases for keeping types separate in stubs: https://stackoverflow.com/questions/59051631/what-is-the-use-of-stub-files-pyi-in-python

i guess the closest to something that could be considered justification is "minimizing code churn in existing files"

I would actually accept this for code that was, for example, heavily used in prod but wasn't very actively worked on; e.g. the original author left, anybody else woudl have to spend a day or two to understand the code well, and the code "just works" so nobody messes it. Not pretty but these things do happen in the real world 🙂

For code you're working on actively I'd just do it. Either all at once, or function/class by function/class. Each time you modify something in a given function, you also annotate it, some kind of rule like that.

[oblique urchin]

PEP 484 is pretty precise about what stubs are for: https://www.python.org/dev/peps/pep-0484/#stub-files

[languid leaf]

I don't like using Optional on return type signatures. Dunno why.

[soft matrix]

Use T | None then ;)

[languid leaf]

well, yeah

I'm fine with it in arguments. Just a stylistic thing.

Even though there's no difference.

[brazen jolt]

I usually don't like even making functions that return None or a value, at least not if you're doing to to mark that something didn't work out with returning the value, so you return None instead. Just raise an exception, that's way more explicit and you can just catch it later. Though if there is another reason or something, I don't really mind the type-hint, just dislike the idea of even doing it

[terse sky]

indeed

[pastel egret]

Another use of stubs could be for modules that are heavily dynamic and so opaque to type checkers. Then you could write static declarations, but keep the implementation more flexible.

[brazen jolt]

you could probably achieve a lot of that with if TYPE_CHECKING

[indigo locust]

If I wanted to put some work into improving PyCharm's static checker

Where would I start?

[soft matrix]

Do you know kotlin?

[oblique urchin]

Is it even open source?

[indigo locust]

I'm sure I could learn

[indigo locust]

Goodness, I don't know. I'd imagine so 😐 I mean, its not like they have anything worth stealing

The rest of the application sure, but the checker needs all the help it can get

[pastel egret]

The community edition is.

[soft matrix]

I'm pretty sure it's in pycharm community

(Personally I wouldn't bother I'd just write a plug-in to pipe pyright into it)

[pastel egret]

https://github.com/JetBrains/intellij-community/tree/master/python

[indigo locust]

Sure, that's an option!

PyCharm can't handle custom descriptors, and that seems like a major miss to me

I just thought I might take a look and see if there's potential for a solution

[soft matrix]

I'm pretty sure someone's already working on it

Although I'm pretty sure they have for like 3 years

[pastel egret]

Some issues here:
https://youtrack.jetbrains.com/issue/PY-47698
https://youtrack.jetbrains.com/issue/PY-44181

[indigo locust]

The most recent related issue I found was from just a weeks ago

[soft matrix]

https://youtrack.jetbrains.com/issue/PY-26184 this was the one I was thinking of

[terse sky]

I'm not sure why that couldn't be done in the file though

[fierce ridge]

Because then you'd get 2 definitions in the file

Is there a tool that converts Union to | and converts the now-unnecessary typing. aliases back to their "plain" versions?

Or would i have to roll my own / do it by hand?

[tranquil turtle]

you can do it by regex replace
(sublime has it, for exapmle)
typing.(\W+) (all typing.*) -> \1 (chars after .)

Union[(\W+), (\W+)] -> \1 | \2

[void panther]

That assumes no aliases etc.

[tranquil turtle]

yes, also you should import all by hand

[void panther]

I think I saw it under one refactoring project but can't remember the name now

[blazing nest]

You should be able to do it somewhat trivially through whatever editor you have

[hallow flint]

could try https://github.com/python/typeshed/pull/5564#issuecomment-896713040 (modify the glob to py)

[blazing nest]

The Union stuff may be a bit too complicated with changing the commas

[soft matrix]

You could just use pyupgrade

[terse sky]

I still don't follow

Can you give me an example of a piece of code that is "heavily dynamic", and can only be annotated in a separate stubs file?

[fierce ridge]

the Motor library is a good example

a lot of it (iirc) is dynamically generated from pymongo

also c extensions need stub files for obvious reasons

[fierce ridge]

i was thinking it'd be doable with ast but that would destroy all the formatting and comments

maybe i can use tree sitter to kind of identify a "smallest enclosing expression" and then use ast to do the actual manipulation

[terse sky]

Sure C extensions were already discussed

[fierce ridge]

also i think some library devs just prefer to keep the stubs in a separate file, e.g. when retrofitting an existing large library

[terse sky]

Yeah, that was discussed as well "avoiding code churn"

Not a great reason but I can picture cases

[fierce ridge]

yeah. i feel like i've seen other libraries that do a lot of dynamic code gen, but motor is the only one that comes to mind. it's not a common use case (and arguably not a good one)

[terse sky]

But re super dynamic stuff, if it's truly very dynamic then it's not going to be able to have static type annotations

If it's code gen but it doesn't affect the types then it can just be an implementation and you can have an annotated wrapper

[fierce ridge]

after messing with statically-typed languages, i see the value in preferring ahead-of-time code generation instead of run-time dynamism

[blazing nest]

The Cython typings are a mess to stub

[fierce ridge]

that is another option, but then you have a big "hole" around your interface w/ the generated stuff

[terse sky]

I don't think I follow

[fierce ridge]

plus, if you have a separate type stub file, you basically have a big test suite for your code gen

[terse sky]

Err how?

[fierce ridge]

yeah wait... nvm

i feel like you should be able to leverage that somehow

[blazing nest]

Technically namedtuple and dataclasses would be a mess if they weren't 🌠 special-cased 🌠

[fierce ridge]

but might be hard / require hacking deep into the cpython runtime. like checking the values in locals()

[terse sky]

Type annotations are annotations regardless whether they are in the same file

And yes, dataclass was going to be my example, if you're doing dynamic code gen then you can't annotate

[blazing nest]

No, if you use a stub file than that won't appear in __annotations__

[terse sky]

That's why it needs to be special cased

That's not what I mean

[fierce ridge]

indeed. this kind of thing makes me wonder how ruby is doing with its static types

i wonder if python is more complicated in some respects, even if python code tends to be less dynamic

[terse sky]

Anyhow basically if the types themselves are dynamic you can't do it either way

[fierce ridge]

ruby has no first-class functions and nothing (afaik) like a dataclass

[terse sky]

If the types are not dynamic you should be able to do it in the same file

I still don't see what stubs are enabling re "very dynamic" modules

[blazing nest]

Can it create types on the fly? Or has exec/eval-like functions?

[fierce ridge]

i'm not sure actually

[fierce ridge]

that's a good point. fwiw i don't see much downside in keeping them separate

if that's what the team prefers then 🤷‍♂️

[oblique urchin]

It has a lot of exec/eval

[fierce ridge]

i know ruby frameworks tend to do a lot of dynamic attribute lookup

and uses un-googleable method names

[terse sky]

i mean you're keeping things in sync across two files, its inherently a lot more fraught

there's a reason why, going back to the start of this convo, the PEP lists very specific use cases for this type stubs

and not "hey and it's nice in general to keep the typing information a whole file away"

[fierce ridge]

sure

either way i know at least 1 library that dynamically generates a lot of itself, so there was your answer

[terse sky]

that's not the answer though; I've asked for examples up and down this thread of why dynamic code gen would force using the stubs

[hasty hull]

Ahead of time code generation is really powerful, it's the only reason https://github.com/RobertCraigie/prisma-client-py can be statically type checked

[terse sky]

and there hasn't been any concrete explanation

[fierce ridge]

you answered it yourself: if you are generating whole classes

https://github.com/mongodb/motor/blob/master/motor/motor_asyncio.py

[terse sky]

No; I said if it was an implementation detail you could simply wrap the implementation details in a class that's properly annotated

[fierce ridge]

well there is 1 example in the wild where the entire class is dynamically generated, and not the internals, but the actual top-level class

[terse sky]

yes, but that's a choice

unless it's the actual API that's dynamic

[fierce ridge]

i personally would not want a whole extra layer of indirection just so i can keep the interface definition in the same .py file

[terse sky]

if it's a fixed API, then you could always make the top level class just a simple, type annotated thing

"whole layer of indirection" - what does this mean? performance concern?

[fierce ridge]

sure, but they're likely to be split over 2 separate files anyway at that point, so why not use the .pyi?

[blazing nest]

Can you send the file where this is done?

[terse sky]

I'd much rather users only "see" a very simple class with regular annotations and docstrings

[fierce ridge]

and yeah why add another layer to the call stack if you don't need it?

[terse sky]

and keep the complexity internal

[fierce ridge]

well you asked for a use case and here it is 🤷‍♂️

[terse sky]

Okay, it's not much of one but I guess it's something

[fierce ridge]

at this point it's a matter of taste and not a clearly preferable solution

[hasty hull]

The core types are generated from this Jinja template https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/generator/templates/types.py.jinja

[fierce ridge]

i'd be really interested in this if i needed to use graphql from python applications

is the code generator itself part of the prisma client api? or is it internal?

i assume if the former there is some cli tool that generates a module from a graphql schema

[blazing nest]

Ah interesting, so is this used by the user then from looking at that prisma file for structure?

[fierce ridge]

huh i actually don't think i understand what prisma client is! i thought it was a graphql client specifically for prisma-backed graphql servers

[hasty hull]

The code generator is built into Prisma internally, so the client API is generated whenever the user runs prisma generate

[fierce ridge]

prisma can generate client-side python code??

[terse sky]

I guess the thing that I don't understand is if you're only using one language, why do you prefer writing a prisma schema

to a dataclass

[hasty hull]

prisma generate looks into the schema file and executes the defined generators

[hasty hull]

Yes, Prisma is technically language agnostic

[terse sky]

e.g.

model Post {
  id        Int     @id @default(autoincrement())
  title     String
  content   String?
  views     Int     @default(0)
  published Boolean @default(false)
  author    User?   @relation(fields: [author_id], references: [id])
  author_id Int?
}

[hasty hull]

The CLI is just written in Node

[fierce ridge]

wow i had no idea

[terse sky]

obviously, we can write @dataclass class Post: ....

[fierce ridge]

i genuinely thought prisma was just a graphql server

so this is an entirely different schema from graphql

[hasty hull]

So you can do whatever you want with your schema file

[fierce ridge]

i see, that's completely different from and a lot more interesting than what i originally envisioned

[hasty hull]

https://prisma-client-py.readthedocs.io/en/stable/reference/custom-generators/

[fierce ridge]

because you do actually use more than one language

[hasty hull]

Yeah they originally were completely focused on graphql

[terse sky]

okay, sure, if you're using more than just python it makes sense

this is pretty much the same concept as protobuff

[fierce ridge]

it looks like their schema syntax isn't a superset of graphql though, was it originally using graphql schema files and they changed it?

i was going to say, how does this compare to protobuf and friends?

[hasty hull]

Because of the ORM, there's no other Python ORM that's fully type safe

[terse sky]

If all you care about is type safety in python, you could achieve that with dataclasses

[hasty hull]

Not with the interface to the database though

[terse sky]

why not?

[fierce ridge]

@terse sky this looks like it's a static description of the data models itself, from which you then generate both DDL and python classes

[terse sky]

what's DDL

[fierce ridge]

CREATE TABLE and such

[terse sky]

what do those functions look like?

[hasty hull]

Because the actual interface to the database looks like this: https://github.com/RobertCraigie/prisma-client-py#query-examples

Which you can't represent statically without code generation

[fierce ridge]

are you asking about prisma or in general? most python ORMs generate DDL from python classes directly: django, sqlalchemy, peewee, pony, et alia

[terse sky]

so e.g.

users = await client.user.find_many(
    include={
        'posts': True,
    },
)

[hasty hull]

You can inspect the generated DDL using prisma migrate dev

[terse sky]

you're saying that the posts': True part is statically checked and that's not possible without AOT code gen?

[hasty hull]

Yes that's correct

[terse sky]

gotcha

[hasty hull]

It also means you get autocomplete which is incredibly useful

[terse sky]

I mean, it is definitely "possible" in the sense that the dataclass has all the information, but it might only be possible by extending the dataclass plugins and such, which is messy

In principle one ought to be able to get the TypedDict corresponding to a Dataclass for example, but I'm not sure if that's actually easily possible

[hasty hull]

Yes that is true but that would be difficult and would be less compatible with the existing type checking ecosystem

[terse sky]

right, no question

makes sense

[hasty hull]

You could do that it just isn't possible to represent that with static type checkers yet

[terse sky]

right

sufficiently powerful static reflection is a hard problem

[fierce ridge]

does the prisma schema language itself have a spec?

[hasty hull]

There's a documentation reference https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference

But no spec per-se

[fierce ridge]

i see

that's too bad. although i know writing a proper spec can be hard

[hasty hull]

And here's the python API for the AST representation https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/generator/models.py

[fierce ridge]

i like that graphql has a spec

[hasty hull]

However that's not stable

They do have a spec but it's outdated :(

https://github.com/prisma/specs

[fierce ridge]

alas

it'd be pretty cool if you could integrate this with openapi somehow

[trim tangle]

OpenAPI doesn't even have generics/templates 😩

[fierce ridge]

true (and sadly)

maybe you could use prisma to generate an openapi spec

or at least generate the skeleton thereof

i guess they have somewhat incompatible design elements

[trim tangle]

Could someone help me with a mypy plugin?
<#help-ramen message>

mypy's documentation is pretty much nonexistent

[hasty hull]

That could be possible, I don't know enough about OpenAPI to comment

[fierce ridge]

you'd have to translate each model to a JSONschema

then you can either plop that schema right into the openapi file, or host it at another url and refer to it

[terse sky]

Yeah, I wanted the most trivial plugin ever: I literally just wanted to be able to define a decorator that is a pass-through to dataclass with different defaults

Sadly in python right now if you just write that as code it's useless

[hasty hull]

Ah I see that should definitely be possible then

[terse sky]

After like an hour of looking for nonexistent docs and examples I gave up

[trim tangle]

I spent about 15 minutes figuring out how to express bytes (the built in type)

It also seems like lots of the states are represented by setting a bunch of boolean flags on giant objects

which is just ?????

[soft matrix]

Don't you just use api.named_type("builtins.bytes")?

[trim tangle]

yep

[soft matrix]

I mean what other design would you suggest?

Idk how else you'd do it unless you did like api.modules["builtins"].bytes or something

[trim tangle]

I'm not suggesting a different design

I'm commenting on how there aren't really any docs

well, there are docstrings and such

but there are no tutorials, and no guides that answer questions like "how do I do X?"

[soft matrix]

Yeah that aspect definitely is annoying and the plug-in tutorial needs some work

But ig most if the mypy team probably wouldn't consider it important cause so few people make plugins

[trim tangle]

And so few people make plugins because... ?

right, because it's extremely hard to even get started

[soft matrix]

Cause most people don't need to extend current python types?

Idk I don't think the people here are representative of most people who use mypy

[trim tangle]

I agree that most people don't need to write a mypy plugin

but I think most libraries that do something dynamic would benefit from a mypy plugin

[terse sky]

idk, being able to change the defaults on @dataclass or @attrs seems like a very basic use case

just to pick an example

[hasty hull]

This already exists for node https://github.com/valentinpalkovic/prisma-json-schema-generator

[fierce ridge]

nice

[terse sky]

If your team has e.g. settled on kwarg_only for your dataclasses for example, for various reasons, it's really annoying that people have to remember to always pass kwarg_only=True, every single time.

IIRC, in attrs, the inability of users to easily write their own decorators with different defaults was part of what led to several different decorators in the library

[fierce ridge]

yeah, at least there's kind of a protocol for it now

although i'm not sure how mypy determines when to use that protocol

maybe it still is hard-coded by the decorator name

[trim tangle]

How do I debug mypy failures? I'm developing a plugin, and I get this:

test.py:17: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.931
Traceback (most recent call last):
  File "/home/df/.pyenv/versions/3.9.8/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/df/.pyenv/versions/3.9.8/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "mypy/checkexpr.py", line 3962, in accept
  File "mypy/nodes.py", line 1633, in accept
  File "mypy/checkexpr.py", line 2078, in visit_member_expr
  File "mypy/checkexpr.py", line 2096, in analyze_ordinary_member_access
  File "mypy/checkmember.py", line 129, in analyze_member_access
  File "mypy/checkmember.py", line 146, in _analyze_member_access
  File "mypy/checkmember.py", line 230, in analyze_instance_member_access
  File "mypy/checkmember.py", line 385, in analyze_member_var_access
  File "mypy/checkmember.py", line 559, in analyze_var
  File "mypy/maptype.py", line 20, in map_instance_to_supertype
AttributeError: attribute 'type_vars' of 'TypeInfo' undefined
test.py:17: : note: use --pdb to drop into pdb

[terse sky]

if there is a protocol that you know how to use to achieve this allegedly simple thing I want to do, I'd love to know 🙂

[trim tangle]

dropping into pdb doesn't work - I can't seem to access locals in the map_instance_to_subtype function

[oblique urchin]

it's because it's mypyc-compiled. maybe try using the pure-python version

[trim tangle]

Is there some flag?

or otherwise, where do I get the pure python version?

[oblique urchin]

not directly. you could just use an editable install, or something like pip install --no-binary

[trim tangle]

ooohhh thanks, it works!

Heyyyyyyyyy yes, it worked! thank you @oblique urchin

from attempt.things import struct

UserInfo = struct(
    ("name", "string"),
    ("city", "string?"),
)

def f(u: UserInfo):
    reveal_type(u.my_attr)  # note: Revealed type is "builtins.bytes"

[steel nimbus]

async def get_pre(bot, message) -> str:
    """
    A corotinue where it gets the message and returns a string
      ---
    Arguments -> bot : discord.Object
    message - > discord.Contenxt
    """
    prefix = await bot.db2.fetch(
        "SELECT prefix from prefix_table WHERE id = $1", message.guild.id
    )
    if prefix:
        return prefix[0]
    return "$"

``` how would bot be type hinted ?

[trim tangle]

what will bot be at runtime?

[oblique urchin]

you already have it! bot: discord.Object

[steel nimbus]

a discord.commands object

[steel nimbus]

sure

[trim tangle]

well, discord.Commands then?

a class is a valid type hint

it you define a custom class, use it as the type hint instead