How do I make tanjun.AnyHooks work with tanjun.dependencies.HotReloader()? | hikari | Page 1

spark oasis Jul 7, 2024, 5:07 PM

#

I've got hot reloading already set up ant working for other things.

The hooks are global, and currently added with tanjun_client.set_hooks(hooks) from a normal import.

The hooks.py file looks like this

hooks = tanjun.AnyHooks()

@hooks.with_pre_execution
async def pre_execution(ctx: tanjun.abc.SlashContext) -> None:
  # stuff
  pass

# more hooks

The usage guide at https://tanjun.cursed.solutions/usage/#execution-hooks mention that hooks can be in a component, but I can't quite figure out how to set it up.

I tried this in the hooks.py file. It loads and works, but it fails to reload.

component = tanjun.Component()

@tanjun.as_loader
def load(client: tanjun.Client) -> None:
    client.set_hooks(hooks)
    client.add_component(component)

@tanjun.as_unloader
def unload(client: tanjun.Client) -> None:
    client.set_hooks(None)
    client.remove_component(component)

Feels like I'm missing something.

#

It would be nice to be able to use load_from_scope() like in my commands.

#

I didn't find any @component.with_hook either (if that makes sense)

#

component.set_hooks(hooks) led to this error:

ERROR:hikari.tanjun.reloader:Failed to unload module `bot.hooks`; hot reloading is now disabled for this module
Traceback (most recent call last):
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 2701, in _reload_module
    self._call_unloaders(module_path, old_loaders)
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 2571, in _call_unloaders
    if loader.unload(self):
       ^^^^^^^^^^^^^^^^^^^
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 234, in unload
    self._callback(client)
  File "/Users/alvar/Code/scoreboarder/bot/src/bot/hooks.py", line 263, in unload
    client.remove_component(component)
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 1997, in remove_component
    raise ValueError(f"The component {component!r} is not registered.")
ValueError: The component Component(self.checks=[], self.hooks=<tanjun.hooks.Hooks object at 0x107982ca0>, self.slash_hooks=None, self.message_hooks=None) is not registered.

uneven dew Jul 7, 2024, 5:57 PM

#

I have a feeling it should follow something like this here: https://tanjun.cursed.solutions/usage/#concurrency-limiter

#

But for the hot reload one

#

@spark oasis

spark oasis Jul 7, 2024, 5:59 PM

#

Are you referring to with_concurrency_limit ?

uneven dew Jul 7, 2024, 5:59 PM

#

Oh, you got the hot reloaded to work

#

Nvm

#

One sec

#

Let me reread

spark oasis Jul 7, 2024, 5:59 PM

#

I got it attempting to reload, but failing to

uneven dew Jul 7, 2024, 6:00 PM

#

@spark oasis https://tanjun.cursed.solutions/reference/components/#tanjun.components.Component.set_hooks

#

You should be able to set the hooks onto the component

#

And then reloading the component will add/remove the hooks when being reloaded

spark oasis Jul 7, 2024, 6:01 PM

#

Do you mean like this

component.set_hooks(hooks)


@tanjun.as_loader
def load(client: tanjun.Client) -> None:
    client.add_component(component)


@tanjun.as_unloader
def unload(client: tanjun.Client) -> None:
    client.remove_component(component)

uneven dew Jul 7, 2024, 6:03 PM

#

Yep

spark oasis Jul 7, 2024, 6:03 PM

#

uneven dew Yep

Fails like this #1259556535638556712 message

#

It would be nice to be able to hot reload set_type_dependency as well but it seems like that might run into the same kind of problem.

uneven dew Jul 7, 2024, 6:06 PM

#

spark oasis It would be nice to be able to hot reload `set_type_dependency` as well but it s...

All this should work out of the box

#

It's confusing me why it isn't 🤔

spark oasis Jul 7, 2024, 6:07 PM

#

uneven dew All this should work out of the box

set_type_dependency is for tanjun.clients.Client not a component though right?

uneven dew Jul 7, 2024, 6:08 PM

#

I mean, you could make extensions that setup the dependencies

#

And in the loaders and unloaders do it

#

Ie

#

class MyDependency:
    ...


@tanjun.as_loader
def load(client: tanjun.Client) -> None:
    client.set_type_dependency(...)


@tanjun.as_unloader
def unload(client: tanjun.Client) -> None:
    client.remove_type_dependency(...)

#

No need for components

uneven dew Jul 7, 2024, 6:09 PM

#

spark oasis `component.set_hooks(hooks)` led to this error: ``` ERROR:hikari.tanjun.reloade...

What's confusing me is why it thinks the component is not loaded

#

Maybe try doing it without the component?

#

And just set and unset the hooks?

spark oasis Jul 7, 2024, 6:10 PM

#

uneven dew Maybe try doing it without the component?

Testing

uneven dew Jul 7, 2024, 6:11 PM

#

:)

spark oasis Jul 7, 2024, 6:12 PM

#

uneven dew And just set and unset the hooks?

Worked!

uneven dew Jul 7, 2024, 6:12 PM

#

:))

spark oasis Jul 7, 2024, 6:12 PM

#

uneven dew :))

I'm going to try real quick with the type dep too

#

@dataclass
class Config:
  theme_color = "#ffffff"

@tanjun.as_loader
def load(client: tanjun.Client) -> None:
    client.set_type_dependency(Config, Config())


@tanjun.as_unloader
def unload(client: tanjun.Client) -> None:
    client.remove_type_dependency(Config)

Does this look wrong @uneven dew ?

It crashes like this when reloading

ERROR:hikari.tanjun.reloader:Failed to unload module `bot.config`; hot reloading is now disabled for this module
Traceback (most recent call last):
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 2701, in _reload_module
    self._call_unloaders(module_path, old_loaders)
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 2571, in _call_unloaders
    if loader.unload(self):
       ^^^^^^^^^^^^^^^^^^^
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 234, in unload
    self._callback(client)
  File "/Users/alvar/Code/scoreboarder/bot/src/bot/config.py", line 48, in unload
    client.remove_type_dependency(Config)
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/tanjun/clients.py", line 2775, in remove_type_dependency
    self._injector.remove_type_dependency(type_)
  File "/Users/alvar/Code/scoreboarder/bot/.venv/lib/python3.12/site-packages/alluka/_client.py", line 300, in remove_type_dependency
    del self._type_dependencies[type_]
        ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: <class 'bot.config.Config'>

uneven dew Jul 7, 2024, 6:26 PM

#

Try something for me

#

Add print(hash(Config)) before loading and unloading

#

And tell me if they match

spark oasis Jul 7, 2024, 6:28 PM

#

uneven dew And tell me if they match

Good thinking. They do not match

#

Can I fake it?

uneven dew Jul 7, 2024, 6:28 PM

#

@lethal cape do you reload the module before you unload?

spark oasis Jul 7, 2024, 6:37 PM

#

I tried setting __hash__ to retun 0 but it didn't make a difference.

uneven dew Jul 7, 2024, 6:37 PM

#

Ok, quick look through the code and this is confusing

uneven dew Jul 7, 2024, 6:37 PM

#

spark oasis I tried setting `__hash__` to retun 0 but it didn't make a difference.

Yeah, that will only affect instances

#

Not the class itself

#

Ok, think this might be a Tanjun bug

#

.remind 4h

#

I'll have a look

uneven dew Jul 7, 2024, 7:34 PM

#

Actually, looks like I'll be more like in a couple of days

#

.remind 3D

spark oasis Jul 7, 2024, 8:19 PM

#

spark oasis Jul 7, 2024, 9:38 PM

#

I made it work like this for now

zenith islandBOT Jul 7, 2024, 10:51 PM

#

@uneven dew: Reminder!

No Message.

Original Message

[Jump](#1259556535638556712 message)

spark oasis Jul 9, 2024, 9:48 PM

#

uneven dew <@115590097100865541> do you reload the module before you unload?

By now I'm pretty sure that it reloads the file before it calls the unload function. If it did call the unload function before reloading the file, it would be possible to remove the type dependency before it has a new hash.

uneven dew Jul 9, 2024, 9:48 PM

#

Yeah, that's 100% the issue

#

I'll have some time to have a look at it tomorrow

#

Just haven't been able to these days

spark oasis Jul 9, 2024, 9:49 PM

#

uneven dew Yeah, that's 100% the issue

I've made this hack which also proves it

spark oasis Jul 9, 2024, 9:49 PM

#

uneven dew I'll have some time to have a look at it tomorrow

Cool, ty

uneven dew Jul 10, 2024, 9:24 AM

#

@spark oasis just had some time to have a look at it, and I can't reproduce the error 🤔

uneven dew Jul 10, 2024, 9:24 AM

#

spark oasis ```python @dataclass class Config: theme_color = "#ffffff" @tanjun.as_loader ...

This code works for me

#

If this is a windows thing, I can't have a look at it until way later in the afternoon

#

No matter what I change, it reloads properly

#

Can I see how you setup the hot reload?

#

I can get it to happen if you setup the hot reloader incorrectly

#

IE: you add both the directory and the module as what it needs to reload

#

In which case it will reload twice

#

The second one being the one that raises the error

#

I'm like 99% sure that's what is happening here, lol

spark oasis Jul 10, 2024, 9:55 AM

#

uneven dew If this is a windows thing, I can't have a look at it until way later in the aft...

I'm on macos

uneven dew Jul 10, 2024, 9:56 AM

#

Then it's then other thing 99%

uneven dew Jul 10, 2024, 9:58 AM

#

uneven dew Can I see how you setup the hot reload?

@spark oasis

spark oasis Jul 10, 2024, 10:00 AM

#

uneven dew IE: you add both the directory and the module as what it needs to reload

My bad, I was responding to something else. Forgot to mention this important detail. I add the modules normally to Tanjun first, then I add them to the hot reloader. I get how that could mess things up, but it was the only way I could listen to the started event for example.

uneven dew Jul 10, 2024, 10:01 AM

#

The hot reloader is the one that will take care of everything, so that double load is the one causing issues

#

So removing that will fix it

uneven dew Jul 10, 2024, 10:02 AM

#

spark oasis My bad, I was responding to something else. Forgot to mention this important det...

And you should be able to do the started event using hot reloads

spark oasis Jul 10, 2024, 10:02 AM

#

uneven dew And you should be able to do the started event using hot reloads

The module was always loading after the started event

#

So I could do it, but it didn't trigger ever

uneven dew Jul 10, 2024, 10:03 AM

#

And it also doesn't really make sense to hot reload starting listeners

#

What's the listener doing? @spark oasis

spark oasis Jul 10, 2024, 10:09 AM

#

uneven dew And it also doesn't really make sense to hot reload starting listeners

Right but it's gotta start somewhere tho. I'm using the module loading functionality to move code out into self-contained files.

spark oasis Jul 10, 2024, 10:10 AM

#

uneven dew What's the listener doing? <@121234615359766528>

Connecting to a DB for example. An async function

uneven dew Jul 10, 2024, 10:10 AM

#

Yeah, what I meant is that you can just not hot reload that specific file, so any updates there would require a full restart

uneven dew Jul 10, 2024, 10:11 AM

#

spark oasis Connecting to a DB for example. An async function

That could be done in a load, no?

#

If you really want to allow reloading everything

#

But maybe that one you could keep as a "requirement" and not allow hot reloading

spark oasis Jul 10, 2024, 10:15 AM

#

uneven dew If you really want to allow reloading everything

Right. I have some other ones that start an asyncio.createTask that loops and updates the discord precense once in a while. That one made use of the hikari started event specifically to get access to bot.

I tried to just start things directly in load(), but I had trouble getting the event loop working.

uneven dew Jul 10, 2024, 10:15 AM

#

Ah true, forgot it's not async

#

Yeah, I have a feeling this could all be stuffed into a module that doesn't get hotloaded

spark oasis Jul 10, 2024, 10:20 AM

#

uneven dew Yeah, I have a feeling this could all be stuffed into a module that doesn't get ...

Yeah for sure. That would definitely work. It's kind of what I have now. Commands and the hooks do hot-reload, these files do not.

Could keep it like that, but I'm used to React HMR which works for everything so I'm not entirely satisfied. Restarting is no fun.

uneven dew Jul 10, 2024, 10:21 AM

#

The annoying this would be shutting everything down and restarting when the module is loaded

#

Specially because stuff depends on it

spark oasis Jul 10, 2024, 10:21 AM

#

uneven dew Ah true, forgot it's not async

I tried to make some hacks to work around the event loop either being there or not.

spark oasis Jul 10, 2024, 10:21 AM

#

uneven dew The annoying this would be shutting everything down and restarting when the modu...

That's what I'm expecting

#

I want to use loader and unloader like useEffect

uneven dew Jul 10, 2024, 10:22 AM

#

In react, you can completely shutdown the server and restart it seamlessly, but thats kinda hard with hikari, since discord doesn't allow you to infinitely reconnect

#

At some point you hit the limit

#

So Lucy decided to go down the per file reload

spark oasis Jul 10, 2024, 10:22 AM

#

uneven dew In react, you can completely shutdown the server and restart it seamlessly, but ...

It's more like I want to reload a component and re-trigger the useEffect

uneven dew Jul 10, 2024, 10:22 AM

#

So anything that is depended on, needs to be done before

#

It's a different way of thinking

#

And impl stuff

spark oasis Jul 10, 2024, 10:23 AM

#

I could ignore the "started" event if I could init my db connection inline in load()

uneven dew Jul 10, 2024, 10:23 AM

#

I mean, if what you do in those listeners is start up and close stuff, you could create tasks for it and do it asyncly

#

And I believe Tanjun has a tasks extension

#

And for the db, you can have an object that will create the connection the first time you attempt to do something with the db and then keep that for the rest of the execution

uneven dew Jul 10, 2024, 10:24 AM

#

spark oasis I could ignore the "started" event if I could init my db connection inline in lo...

You could

uneven dew Jul 10, 2024, 10:24 AM

#

uneven dew And for the db, you can have an object that will create the connection the first...

If you do this

#

Then you don't need an async context

#

Could be a bit more expensive, but yeah

#

For development purposes, it could work

#

There is also component_added listener that you could hack your way into

#

And do the db instantiate there

#

But honestly, making the db create the initial connection blocking seems like the way to go

spark oasis Jul 10, 2024, 10:27 AM

#

uneven dew You could

If I do set_type_dependecy in load and then remove_type_dependency in unload, would other files using that type dependency (that didn't reload) be able to use the latest instance when reloaded?

uneven dew Jul 10, 2024, 10:27 AM

#

Yes

spark oasis Jul 10, 2024, 10:28 AM

#

uneven dew But honestly, making the db create the initial connection blocking seems like th...

I have aiomysql which is blocking and it's not going anywhere soon I think

spark oasis Jul 10, 2024, 10:28 AM

#

uneven dew Yes

But then as long as I'm able to create an async task in load() I should be able to fully init and clean up anything, as long as all I need is available via tanjun.Client.

uneven dew Jul 10, 2024, 10:34 AM

#

spark oasis But then as long as I'm able to create an async task in load() I should be able ...

No need to create a task

#

Just do it blocklingly

spark oasis Jul 10, 2024, 10:35 AM

#

uneven dew Just do it blocklingly

It's an async function. It doesn't let me just run it in load() without throwing a hiss.

spark oasis Jul 10, 2024, 10:35 AM

#

uneven dew Just do it blocklingly

And load() couldn't be async either I think

uneven dew Jul 10, 2024, 10:39 AM

#

spark oasis It's an async function. It doesn't let me just run it in load() without throwing...

You can create a loop, run it, and then discard it

#

Hacky, but it should work

spark oasis Jul 10, 2024, 10:39 AM

#

uneven dew You can create a loop, run it, and then discard it

Ok I'll look in to that, ty

uneven dew Jul 10, 2024, 11:47 AM

#

It just hit me

#

That won't work

#

Because you can't send the object through loops

#

I would just make it not hot reloadable

#

Or make it so that it starts only when the first commit is executed

spark oasis Jul 10, 2024, 1:01 PM

#

@uneven dew I just tried printing the hash of a class in load and unload with this exact hot reloading code:

tanjun_client = tanjun.Client.from_gateway_bot(...)

for module in static_modules:
    tanjun_client.load_modules(module)

hot_reloader = tanjun.dependencies.HotReloader()
for module in hot_reloadable_modules:
    hot_reloader.add_modules(module)
hot_reloader.add_to_client(tanjun_client)

bot.run()

(I have two lists of modules indicating wether they can be hot reloaded or not)

#

So now I'm no longer adding the static modules before also adding them to the hot reloader like I did before. I'm still getting different hash() es in unload compared to load()

uneven dew Jul 10, 2024, 1:04 PM

#

Couple of questions, would it be possible for me to have a look at your bot code (I can sign an NDA if needed)?

#

Because I can't even reproduce it

spark oasis Jul 10, 2024, 1:15 PM

#

@uneven dew Minimal reproduction code

create_bot.py

def create_bot():
    bot = hikari.GatewayBot(
        os.environ["DISCORD_TOKEN"],
    )

    tanjun_client = tanjun.Client.from_gateway_bot(
        bot,
        declare_global_commands=99999999,
    )

    hot_reloader = tanjun.dependencies.HotReloader()
    hot_reloader.add_modules("bot.database")
    hot_reloader.add_to_client(tanjun_client)

    bot.run()

database.py

import tanjun


class Database:
    def __init__(self) -> None:
        pass

    def connect(self) -> None:
        pass

    def disconnect(self) -> None:
        pass


@tanjun.as_loader
def load(client: tanjun.Client) -> None:
    print("load database.py")
    print(hash(Database))
    database = Database()
    database.connect()
    client.set_type_dependency(Database, database)
    client.add_client_callback(tanjun.ClientCallbackNames.CLOSING, database.disconnect)


@tanjun.as_unloader
def unload(client: tanjun.Client) -> None:
    print("unload database.py")
    print(hash(Database))
    database = client.get_type_dependency(Database)
    database.disconnect()

#

Log (starting the bot and saving database.py once)

📎 message.txt

#

I'm on a mac

uneven dew Jul 10, 2024, 1:56 PM

#

Ok

#

Found the bug

#

Module is reloaded on line 2691 through the yield (logic for reloading is done elsewhere) and then the unloading is purposely done using the new module a bit further down

#

@lethal cape would you be fine with me opening a pull request to call the unloader in the old module and the load on the new one?

#

or is there a reason its done that way?

#

(im assuming old_loaders get replaced at some point, but havent been able to figure out exactly where, that might be another way to fix this)

#

im assuming that happens due to some weird cpython thing when it comes to pointing to stuff in a module

#

Funnily enough, this doesn't happen if you hot reload a path instead of the module

#

Maybe changing what import lib calls might help (currently it's importlib.reload())

#

But importing the new one along the old one might have even more weird behavior, like python not removing it, and you having two copies of the same module

spark oasis Jul 10, 2024, 2:03 PM

#

side note: It'd be nice if unload() could be async so that unloading doesn't happen until it's done potentially awaiting suff. Edit: But I guess I can block the main thread as well.

lethal cape Jul 10, 2024, 2:51 PM

#

uneven dew <@115590097100865541> would you be fine with me opening a pull request to call t...

does this match what you're thinking of https://github.com/FasterSpeeding/Tanjun/pull/813

GitHub

Fix hot reloading module state bug by FasterSpeeding · Pull Request...

Summary

Checklist

I have run nox and all the pipelines have passed.
I have made unittests according to the code I have added/modified/deleted.

Related issues

zenith islandBOT Jul 10, 2024, 7:34 PM

#

@uneven dew: Reminder!

No Message.

Original Message

[Jump](#1259556535638556712 message)

uneven dew Jul 10, 2024, 7:36 PM

#

lethal cape does this match what you're thinking of https://github.com/FasterSpeeding/Tanjun...

Yep

#

That's exactly it

uneven dew Jul 10, 2024, 8:20 PM

#

@lethal cape since it's been open since 2023, was wondering it would get merged 🙃

lethal cape Jul 10, 2024, 8:59 PM

#

maybe

uneven dew Jul 10, 2024, 9:07 PM

#

anyways, @spark oasis you can bother snab about it now, that pr they sent fixes it

#

but its from august 15 2023, so might not be up to date with the latest additions

#

last update was a year ago

uneven dew Jul 10, 2024, 9:07 PM

#

lethal cape maybe

pretty please :)

spark oasis Jul 10, 2024, 9:43 PM

#

lethal cape maybe

Would be so great! Speaking from experience, good hot reload is a differentiator and speeds up development a lot.

#How do I make tanjun.AnyHooks work with tanjun.dependencies.HotReloader()?