#πŸ”’ Proper Typing for Decorators

60 messages Β· Page 1 of 1 (latest)

woeful zinc
#

Hey everyone! I'm trying to add type annotations for my python decorator, but its proving to be really difficult to get right

Decorator:

RT = TypeVar('RT')
P = ParamSpec('P')

def meraki_dashboard_setup(function: Callable[P, RT]):
    """
    Adds both a "meraki_key" and a "dashboard" argument to a decorated function. If the
    "meraki_key" argument is provided, the function will setup the dashboard object using the
    key. If the "dashboard" argument is provided, the function will use it directly.
    """
    @wraps(function)
    def wrapper(
        dashboard: Optional[DashboardAPI] = None,
        meraki_key: Optional[MerakiKey] = None,
        *args: P.args, **kwargs: P.kwargs
    ):
        if not dashboard and meraki_key:
            dashboard = setup_meraki_dashboard(meraki_key)
        if not dashboard and not meraki_key:
            raise ValueError(
                "Neither a dashboard object nor a Meraki key were provided."
            )
        if not dashboard:
            raise ValueError("Unable to setup the Meraki dashboard object")

        kwargs["dashboard"] = dashboard
        return function(*args, **kwargs)

    return wrapper

Decorated function:

@meraki_dashboard_setup
def get_organization_networks(
    organization_id: str,
    **kwargs: Any
):
    """
    Fetch all networks from a specific organization.
    """
    dashboard = cast(DashboardAPI, kwargs.get("dashboard"))
    return cast(
        List[MerakiOrgNetwork],
       dashboard.organizations.getOrganizationNetworks(organization_id)
    )

My problem is that when I use the get_organization_networks function, the function hints will show as the following:

(function) get_organization_networks: _Wrapped[(organization_id: str, **kwargs: Any), List[MerakiOrgNetwork], (dashboard: DashboardAPI | None = None, meraki_key: MerakiKey | None = None, organization_id: str, **kwargs: Any), List[MerakiOrgNetwork]]
crisp arrowBOT
#

@woeful zinc

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

woeful zinc
#

This hint basically consists of _Wrapped[P, RT, Mix of wrapper arguments + P, RT]. This is almost perfect, but I would love to get just this (like in a regular function):

(function) get_organization_networks(dashboard: DashboardAPI | None = None, meraki_key: MerakiKey | None = None, organization_id: str, **kwargs: Any) -> List[MerakiOrgNetwork]
Fetch all networks from a specific organization.

Seems like functools's wraps decorator is not able to finish copying the metadata for the wrapper into the function, causing no docstring from the decorated function to be copied, as well as making the function signature appear all wonky. One way I found of forcing the wrapper to be typed properly is by adding an output type to the meraki_dashboard_setup function like so

def meraki_dashboard_setup(function: Callable[P, RT]) -> Callable[P, RT]:
   ...

This returns the following hint

(function) def get_organization_networks(organization_id: str, **kwargs: Any) -> List[MerakiOrgNetwork]
Fetch all organizations from the Meraki Dashboard API

However, as you can see, the arguments passed down by the decorator are completely nuked by this approach, leaving only the arguments from the decorated function. This will make this function very susceptible to human error, which is what I want to avoid

I've investigated how to get this working, but I cannot figure it out for the life of me. Maybe something to do with ParamSpec? I could technically leave it with my initial approach and it will work, but I really would like to have docstrings and a proper function signature. Its going to drive me insane if it doesn't. What could I do?

floral escarp
#

This is a really well typed out question, and I feel bad that I don’t really have an answer for you, but decorators confuse the hell out of me too.

I will link you to this:
https://youtu.be/_QXlbwRmqgI
It spends a couple minutes going over the typing in not very much detail, and then it goes through another way of making decorators that waaaay simpler

ParamSpec / TypeVar / Callable oh my! this skips all of that and gives a simple easy-to-reuse decorator (and more!)

playlist: https://www.youtube.com/playlist?list=PLWBKAf81pmOaP9naRiNAqug6EBnkPakvY

==========

twitch: http...

β–Ά Play video
fringe thorn
#

It looks like meraki_dashboard_setup's return type should be Callable[Concatenate[Optional[DashboardAPI], Optional[MerakiKey], P], RT], as you're returning wrapper, which is essentially Callable[P, RT] with the added Optional fields

woeful zinc
scenic stone
#

Well, the title is clickbaity

#

he just replaces the decorator with a context manager

#

I don't think the signature you want is possible to express in python's type system. You can't require a dashboard kwarg and also arbitrary kwargs

woeful zinc
# fringe thorn It looks like `meraki_dashboard_setup`'s return type should be `Callable[Concate...

Thank you! But unfortunately I have tried this... It technically works once more. The docstring is back, but the thing that I dont like is that it makes appear as if the function takes two unamed positional arguments (which would be dashboard and meraki_key respectively).

(function) def get_organizations(
    DashboardAPI | None,
    MerakiKey | None,
    **kwargs: Any
) -> List[MerakiOrg]
Fetch all organizations from the Meraki Dashboard API

As you can see from the decorator, you should provide one or the other, and if you pass both you would get an error. I know you could technically pass it like this dashboard, None but I feel like its a bit more of "magic" that the developer needs to be aware of when using the function when it should be just a matter of using the function normally

scenic stone
#

But yes, if you want to add a positional argument, you can use Concatenate

fringe thorn
#

Ah, woops πŸ˜…

fringe thorn
woeful zinc
scenic stone
#

I meant that you cannot type an argument to be a function that has any arguments but which must include a dashboard: DashboardAPI

woeful zinc
#

Seems like such a simple problem to solve. So protocols is my only way to go?

scenic stone
#

You can't do this with a protocol either

#

Actually, why not just make a function like ```py
def setup_dashboard(dashboard: DashboardAPI | None, meraki_key: MerakiKey | None) -> DashboardAPI

Seems much simpler than a decorator
#

Actually, what's the point of this decorator? You already have setup_meraki_dashboard which you can call

woeful zinc
#

In this case I just made it so that I didnt have to add "dashboard" and "meraki_key" to like 30 different functions. In this specific case it can be considered arbitrary, but I do have a larger function where the output of the function is post processed and also caching is added. Adding that specific piece of logic (which is basically just copy and pasting across all functions) does seem a bit excessive to me

fringe thorn
#
class Setup(Protocol[P, RT]):
  def __call__(self, dashboard: DashboardAPI, *args: P.args, **kwargs: P.kwargs) -> RT:
    ...

Plus a couple of overloads should work, no? Unless I'm doing something wrong...

scenic stone
#

oh wait, you're allowed to mix paramspec with extra args?

scenic stone
fringe thorn
#

Apprently so... at least Pyright isn't complainig about it and can actually narrow the signature down Β―_(ツ)_/Β―

keen cedar
#

(because contextlib)

scenic stone
#

the resulting thing is not a decorator

#

oh wait

keen cedar
#

Watch the video. :P

#

I learned something about contextmanager from the video.

scenic stone
#

ah yeah, I missed that part about contexlib.contextmanager

#

though it still cannot replace an arbitrary decorator (e.g. this one changes up some arguments)

woeful zinc
#

Uh. Let me check the video once more. Didnt even go past the first minute because you told me it was not worth it lol

#

But what do you think, not worth it being a decorator then? Sorry, I did add a bit of context to one of my previous comments

scenic stone
#

Why do you want the 30 functions to accept both meraki_key and dashboard, instead of just accepting an already created dashboard?

woeful zinc
#

Cause in some cases you want to share the dashboard instance that its going to be used since its an object that calls a couple of endpoints whenever you first instantiate it

#

Other times, you may want to have the possibility to just pass your meraki key and get the dashboard instantiated for you from those credentials

#

I thought centralizing that logic to the decorator might avoid repetition, but you might be right

scenic stone
#

Are these 30 functions from the public interface of some library?

#

I would suggest just accepting a single dashboard argument and having the client create the dashboard if they don't have it already. It's a much simpler interface, you have 1 way of calling it instead of 2, and it doesn't add that much code

woeful zinc
woeful zinc
#

Still the question remains, as I still have another decorator that might need this πŸ˜…

#

But thank you! Sometimes I just get locked into my initial idea and dont realize its a bit stupid lol

scenic stone
woeful zinc
#

Thats true. I do have a couple of coworkers that might fall for this trap lol

scenic stone
# woeful zinc Still the question remains, as I still have another decorator that might need th...

Actually what Lee suggested seems to work

from typing import Protocol, ParamSpec, TypeVar


T = TypeVar("T", covariant=True)
P = ParamSpec("P")


class Dashboard:
    pass


class Dest(Protocol[P, T]):
    def __call__(
        self,
        dashboard: Dashboard | None = None,
        key: str | None = None,
        *args: P.args,
        **kwargs: P.kwargs,
    ) -> T:
        ...

class Src(Protocol[P, T]):
    def __call__(self, dashboard: Dashboard, *args: P.args, **kwargs: P.kwargs) -> T:
        ...


def deco(src: Src[P, T]) -> Dest[P, T]:
    ...


@deco
def foo(dashboard: Dashboard, something: str) -> int:
    ...


reveal_type(foo)  # Dest[(something: str), int]
foo(dashboard=Dashboard(), something="a")  #ok
woeful zinc
#

Ohhhh

#

Let me try it out. Might I quickly ask, what does the "covariant=True" accomplish for the TypeVar?

#

Thank you @scenic stone and @fringe thorn !

woeful zinc
#

For the protocol, shouldn't the first argument be some type of Callable? Or is that what the Src is trying to replicate?

scenic stone
crisp arrowBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.