I'm surprised you didn't mention AnyIO
#async-and-concurrency
stiff shale
What approach would you want to use with async if what you described doesn't work for you? (and why?)
dull ocean
Me too
tame saffron
What approach would you want to use with async if what you described doesn't wor...
Looks to me like I'm better off with Node.js, where the language is inherently async. I don't have to manually create and micromanage tasks that should run async. You simply write async and await and that's it in most cases. Unless I explicitly want to use Promise.any() or Promise.all().
stiff shale
Looks to me like I'm better off with Node.js, where the language is inherently a...
the async model in ECMAScript is among the worst I've seen. No cancellation, no structured concurrency.
the only worse kind is callback-based (e.g. Twisted, JS before async/await)
Kotlin and Rust, on the other hand, are among the best designs out there
dull ocean
Twisted has async/await
stiff shale
sure, but that's a relatively recent addition
not sure why someone would start using Twisted these days
dull ocean
not sure why someone would start using Twisted these days
If they want to make an LDAP server
stiff shale
If they want to make an LDAP server
doesn't sound like a very good reason to me, to jump on the Twisted bandwagon just for that
waxen fiber
it's better
ocean root
how?
devout harness
You may use it cause it's part of the standard library so no external dependencies 🙂
ocean root
that's no reason to use urllib over requests or http.server over flask/Django
serene badge
sweet dew
placing tactical . here because this convo is intersting
devout harness
I've heard people talk about working in situations where they are not allowed to use third party dependencies
Not experience it myself though
waxen fiber
I just find asyncio nicer, the docs are better, I find making tasks easer
floral atlas
Hi
ashen shale
Love this.
storm peak
And here, I am still not sure if running an event loop in another thread is a good idea
waxen fiber
threading and async shouldn't mix very offen
hollow orchid
cause technically you are mixing two schedulers, idk if that's a good idea
ocean root
asyncio isn't inherently threadsafe
storm peak
well yep
waxen fiber
what do you want to put in another thread
storm peak
Some listeners, basically looped functions doing requests
storm peak
yep
hollow orchid
can't you just use like some async web request library?
heck, even requests-html supports that
waxen fiber
then why need threads
storm peak
Well, I don’t wanna block the main thread
waxen fiber
async doesn't do that
hollow orchid
it doesn't though?
waxen fiber
that's the entire point of async
storm peak
Anyway yeah, it always comes to this question
hollow orchid
it keeps the async task asleep at a checkpoint until network activity, afaik
waxen fiber
and what type of listeners
hollow orchid
once you use await
storm peak
You need to run a loop if you want some looped task
The listeners are just coroutines being run, they request some stuff on the web, and call methods of one class
waxen fiber
from what you have said I see no reason to use threads
modest viper
Oh an async chat :0
hollow orchid
yep
modest viper
Hai Alexa o/
waxen fiber
storm peak
okay lol, guess I am just not knowing how to do stuff
modest viper
Async is a rather unfamiliar library, so it's alright
hollow orchid
yeah
storm peak
Oh, haha, Aeros?
modest viper
Look at me I'm suffering a lot and my bot makes pull requests from a database
trail forum
Heyooo
autumn haven
Hello, feel free to @ me for any asyncio specific questions!
hollow orchid
LOL
storm peak
LOL
hollow orchid
"accidental" ping
trail forum
Epic new channel 👌
storm peak
Very epic channel, true
autumn haven
Man I guess there's so many names on the server there's someone named "me"
hollow orchid
dw in this other server I'm part of this mod keeps pinging eve instead of everyone
trail forum
Bro @ me
hollow orchid
no u
storm peak
I have written a method for a coroutine class which created a new event loop and runs a coroutine in it
How terrible am I, hahaha
hollow orchid
why a new event loop?
waxen fiber
create a event loop or get the event loop
storm peak
haha, that question is pretty common
red wigeon
First
waxen fiber
a new built in method?
balmy slate
Ahhh my peoples
autumn haven
FYI in 3.7, it's no longer explicit required to aquire or create a new event loop, asyncio.run(coro) takes care of that. See (https://github.com/python/cpython/blob/b40442ba88fec5e64ab1802ce37df0afec59fa53/Lib/asyncio/runners.py#L8) for the implementation details.
ocean root
when are we getting await anext()
when are we getting an __await__ that isn't implemented by everyone as return some_coro.__await__()
storm peak
Eh, just to me
balmy slate
Eh, asyncio.run is only for your main entry point, imo
autumn haven
You can always implement your own version of it which doesn't
autumn haven
The code is pretty straightforward
autumn haven
@ocean root Not sure, I'll have to ask Yury. I recently started a core dev mentorship under him, so I'll see if he knows if anext() is coming anytime soon.
storm peak
@new_method(coroutine) # custom decorator using ctypes, getting coroutine class is a bit of difficulty though
def run(self):
loop = asyncio.new_event_loop()
try:
return loop.run_until_complete(self)
finally:
cancel_all_tasks(loop) # custom thing
loop.close()
``` and now, haha:
```py
async def coro():
print('Hello World!')
coro().run()
``` this is the weirdest thing I have ever createddo you really want to make a new event loop every time
doesn't seem the best thing to do
autumn haven
yeah it's usually better to check if you already have one first
storm peak
oh okay
balmy slate
^^
storm peak
So I like, create a one and just cancel the tasks in it after execution, without closing?
hollow orchid
wait
why would you cancel everything
couldn't there be something important there?
balmy slate
That should only be done at application exit
storm peak
oh, yep xd
waxen fiber
there will only be one thing in that loop anyway
hollow orchid
idk this is a contrived example
modest viper
Btw, how much does discord botting teach you in terms of async?
waxen fiber
a bit
storm peak
ok, then, don’t cancel anything 
hollow orchid
lol
waxen fiber
I had to read up a bit more after bot making @modest viper
autumn haven
Anything network-related can teach you about async if you choose to implement it in a non-blocking manner
waxen fiber
but it gets the basics in
storm peak
yep
balmy slate
I highly recommend that series
hollow orchid
I did look through the trio docs and it's pretty transparent
it talks about how the scheduler can only switch on awaits
and things like that
balmy slate
Even after working with asyncio everyday for close to a year I found I was doing a few things wrong
hollow orchid
hmm
modest viper
thanks @waxen fiber !
balmy slate
My biggest miss was knowing when to use create_task
autumn haven
I'll have to check that out at some point, it's useful to get detailed user feedback and some ideas for more detailed examples in the docs.
It's a difficult balance though, because while we want the examples to be useful, it's very easy to overuse them. The docs are supposed to be a technical manual with generalized use cases.
hollow orchid
hmm
how about footnotes then?
trio for example talked about checkpoints for like a sentence or two then linked to a page about checkpoints / scheduling iirc
ocean root
the asyncio docs no longer link to the source at the top like every other doc page and it's very annoying can you add it back
autumn haven
I was actually just about to bring that up to Yury, thanks for the reminder. (:
modest viper
Btw seeing your role Aeros, if I want to learn python along with C, should I be learning cpython or cython?
ocean root
👍
modest viper
surprise out of topic question
hollow orchid
came out of left field
storm peak
very out of topic
autumn haven
Cython. CPython really doesn't require usage of C unless you're writing extensions with the C-API.
Feel free to PM me about that, but I'll admit the C-API is not my specialty
modest viper
Hmm I'm already very thankful for your answer, danke~
storm peak
*speciality
autumn haven
Haha that's okay, it's a confusing language at times
storm peak
hehe yeah sometimes
old star
Hey @autumn haven, nice to see you here.
autumn haven
Hey @old star, thanks to the owners and admins for setting things up. I'll be in here periodically to answer questions and look into user feedback regarding asyncio.
old star
Excellent. 👌
storm peak
So uhm, could I interrupt a bit?
autumn haven
@old star Also, feel free to pin my earlier message "Hello, feel free to @ me for any async or asyncio specific questions!"
storm peak
@new_method(coroutine)
def run(self, *, loop=None):
"""Run the Coroutine in an event loop.
This docstring is long enough so it is cool.
"""
loop = loop or asyncio.get_event_loop()
return loop.run_until_complete(self)
async def coro():
print('Hello, World!')
coro().run()
``` So this one is better than the previous though? (sorry if I have interrupted)@autumn haven Done and dusted. Thanks again for hanging around here, that's super cool.
sweet dew
Aeros MVP
old star
we've come a long way when we have actual cpython asyncio devs hanging in our async channel
just sayin
storm peak
Haha, True
AGAIN, why is autocorrection capitalizing? Anyway, it causes funny situations in python talks
autumn haven
Haha, yeah I've been lurking around on this server before I started becoming involved with CPython development. This server has a great community behind it. (:
storm peak
@autumn haven uhm so yeah, is the new snippet better in a way that the one I sent here earlier?
autumn haven
@storm peak There's no need for run() to be a coroutine, and I would recommend changing the loop = runner_loop to loop = events.new_event_loop(). events.new_event_loop() is not only the correct syntax, but you should also create the new event loop within the method itself rather than outside of it.
storm peak
So I should create a new event loop each time in the method?
there is no need for run() to be a coroutine
Huh?
Also I changed the snippet a bit
autumn haven
def run(cor, *, loop=None):
if loop is None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return loop.run_until_complete(cor)
async def coro():
print('Hello, World!')
run(coro())
mhm, gotcha
asyncio.set_event_loop(loop), I suppose
(yeah, they are the same thing, just it is used from events in asyncio lib)
autumn haven
there is no need for run() to be a coroutine
I see now that you simply wanted to use the annotation to utilize it as a method for coroutine objects, I misread it as the older coroutine annotation (beforeasync defwas a feature).
minor plinth
woah
autumn haven
@storm peak Is there a particular reason as to why you wanted to use coro().run() as opposed to just run(coro)?
storm peak
As I have previously said, it is the weirdest idea that came to my mind
autumn haven
Also, corrected the example to use asyncio.set_event_loop(). I'm used to working internally and importing from events.py https://github.com/python/cpython/tree/master/Lib/asyncio
Either could technically work though depending on how it's set up, but asyncio.set_event_loop() is the proper way of interacting with the API.
storm peak
Okay, got it
autumn haven
@ocean root Just got a response back from Yury, he gave me the okay to add the source code headers at the top of the asyncio documentation pages. Some of it may be a bit nuanced (with the asyncio docs, it's not always one page = one python source code file), but at the very least I'll be adding a link to Lib/asyncio/ on the homepage for the module. The changes should be present for the 3.7, 3.8, and 3.9 docs within the next week or two. I appreciate the feedback and reminder to bring it up. (:
south crag
@Aeros#0717 when to use run_until_complete vs call_soon vs run vs create_task? 🤪
I feel like this is one thing i don't get from the separately laid out docs
autumn haven
@south crag So, essentially loop.run_until_complete() is the more simplified and high level API (of the ones you mentioned) for executing a future or wrapping a coroutine in a task. For most general use cases loop.run_until_complete() does the trick and has the highest degree of thread safety. Going down the list from highest to lowest level of the asyncio public API, it’s pretty much (assuming Python 3.7+) asyncio.run() > loop.run_until_complete() > asyncio.ensure_future() > asyncio.create_task() > asyncio.create_future().
If you want an awaitable object where you can manually handle when it returns (future.result()), when it’s marked as done (normal completion with future.set_result() and exception with future.set_exception()) use asyncio.ensure_future(). Typically, this is used in combination with loop.call_soon_threadsafe() (or loop.call_soon() when dealing with a single thread). I.E. loop.call_soon_threadsafe(future.set_result, result) or loop.call_soon_threadsafe(future.set_exception, exception). This essentially queues the event loop call `arg1(arg2) on the next iteration.
The lower levels would take a bit more time to explain, but it’s generally not needed for 99% of use cases if you’re not trying to write your own library.
It goes lower than asyncio.create_future() as well, but that's beyond your question.
south crag
I thought create_task would be pretty common as well
autumn haven
Yeah it's not entirely uncommon, but I personally don't use it directly. This is mostly because asyncio.ensure_future() will do that for you. See the source code for the function: https://github.com/python/cpython/blob/9e71917e0290972f65711f75510078f799cf0b59/Lib/asyncio/tasks.py#L653
I haven't seen many use cases outside of asyncio or third party asynchronous libraries where it's necessary to use create_task() directly.
Within the internals of asyncio, I recently made use of create_future() and call_soon_threadsafe() within a coroutine method that I added for 3.9, loop.shutdown_default_executor(): https://github.com/python/cpython/blob/9e71917e0290972f65711f75510078f799cf0b59/Lib/asyncio/base_events.py#L552
I added it to asyncio.run(), so I imagine that the majority of users will not end up calling it directly
red wigeon
Does anyone or anything use Twisted anymore
waxen fiber
is it possible to use asyncio to run a Coro and get the return of it in one line?
i do need to run it a certain loop I have that's in a variable
ocean root
result = await coro 
red wigeon
How would you get a return value if you didn't block until it was ready?
Unless blocking isn't an issue in which case what's wrong with await
autumn haven
Unless blocking isn't an issue in which case what's wrong with await
If blocking isn't an issue at all, asynchronous programming isn't needed.
red wigeon
I meant blocking as in lines running sequentially
Is there a term for that?
But yeah that was poorly worded
autumn haven
@waxen fiber To run the coro and get the return of it in a single line withhout specifying the loop, you should use ```python
return asyncio.run(coro)
If you want to specify the loop, you can use ```python
return loop.run_until_complete(coro)
keep in mind that asyncio.run() will terminate the event loop, so if you want to keep the event loop open after returning the result of the coroutine, you should use loop.run_until_complete()
ocean root
I assume the event loop is running which is the part that gives difficulties
autumn haven
yeah so you can get the currently running event loop and close it in one line ```python
asyncio.get_running_loop().run_until_complete(coro)
But that's a bit ugly, so I would recommend ```python
loop = asyncio.get_running_loop()
loop.run_until_complete(coro)
either can be directly returned
ocean root
but if it's already running you can't call run_until_complete on it
lean plume
This channel is going to be really helpful for users of #discord-bots
autumn haven
@ocean root I'm not sure where you got that impression from, you can use loop.run_until_complete() repeatedly on different coroutines from the same running event loop 😉 ```python
import asyncio
async def coro1():
await asyncio.sleep(1)
print("coro1")
async def coro2():
await asyncio.sleep(10)
print("coro2")
loop = asyncio.new_event_loop()
loop.run_until_complete(coro2())
loop.run_until_complete(coro1())
as in
autumn haven
A code example would help
ocean root
import asyncio
async def coro(message):
await asyncio.sleep(1)
print(message)
async def main():
loop = asyncio.get_event_loop() # not necessarily the loop form the main thread but one running regardless
loop.run_until_complete(coro('inside main'))
# pretend a 3rd party library called this
asyncio.run(main())
How I interpreted the original question was "how to get the result of a scheduled task consistently"
sweet vine
Like how you add another coroutine inside an already running loop.run_until_complete()
autumn haven
Oh, I think I see what was meant. I was interpreting it as "add another coroutine to the event loop", which can be queued in order of loop.run_until_complete(). Are you referring to the task that the coroutine is being wrapped in?
(I'm specifically referring to "add another coroutine inside of already running loop.run_until_complete", I understand @ocean root's original answer now).
Not trying to nitpick, but I think there may be some confusion here on differentiating between the different layers. Each time you run loop.run_until_complete(), it's using the same exact event loop (assuming loop wasn't changed), the difference is that it's creating a different task to manage the coroutine.
autumn haven
Also, I know it was a toy example so it was probably not intentional, but there's no reason for that main function to be a coroutine, and passing a function to any variation of asyncio.run() which already has loop.run_until_complete() is basically like using asyncio.run(asyncio.run()), since asyncio.run() already calls loop.run_until_complete()on the passed coroutine.
I'd like to see an example from @waxen fiber that shows their exact problem, it sounds like a code structuring issue more than anything
(For the "i do need to run it a certain loop I have that's in a variable", not "is it possible to use asyncio to run a Coro and get the return of it in one line?")
Unless I've just been awake for too long and am looking way too far into this question 😉
waxen fiber
I'm making a one line async program
stray kite
@ocean root if u think it might be usefull, i can implement that at my async library 
ocean root
implement what
autumn haven
why does it specifically have to be a single line?
ocean root
probably an esoteric challenge
autumn haven
Ah, I see
stray kite
Ava, i mean, your run until complete, hat clsoes the loop if there are no active tasks
ocean root
I don't think I posted that
stray kite
then some1 else did
also elaxa, i already solved that
>>> loop=EventThread()
>>> async def main():
await sleep(1.0,loop)
raise KeyError('dunno')
>>> result=loop.create_task_threadsafe(main()).syncwrap().wait()
Traceback (most recent call last):
File "<pyshell#32>", line 1, in <module>
result=loop.create_task_threadsafe(main()).syncwrap().wait()
File "D:\Python\discordclnt\hata\futures.py", line 447, in wait
return self.result()
File "D:\Python\discordclnt\hata\futures.py", line 349, in result
raise self._exception
File "D:\Python\discordclnt\hata\futures.py", line 1209, in __step
result=coro.send(None)
File "<pyshell#31>", line 3, in main
raise KeyError('dunno')
KeyError: 'dunno'
ocean root
await works the same way yield from does with the generator API which AFAIK can't be one-lined due to yields
stray kite
or dont u mean get the result like this?
autumn haven
That is probably the most painful single line of asyncio I've ever seen haha result=loop.create_task_threadsafe(main()).syncwrap().wait()
waxen fiber
the result as in what it returns
stray kite
thats not asyncio
thats my own library
then i think i nailed, i can retrieve result from an eventloop in a sync envirnment or in async as well
autumn haven
oh, that would make sense actually, create_task_threadsafe() and syncwrap() aren't functions in asyncio. Still equally painful 😉
stray kite
if u mean like i did above, then yes
>>> async def main():
return 'success'
>>> result=loop.create_task_threadsafe(main()).syncwrap().wait()
>>> result
'success'
or not exception case
autumn haven
Honestly, I have no idea. That's a bit too esoteric for me, I pretty much only focus on the practical usages these days.
waxen fiber
hmm ok
stray kite
i use this only at 1 place from 50k line code as well
autumn haven
I used to spend a lot of time on Codewars a while ago solving ridiculous JS challenges, but I don't have much interest in it anymore
stray kite
It is all right Aeros, we all have stuffs to do -> looks at the TODO list -> 
also noticed a bug at my library, calling EventThread.stop() does not wakes up the loop if called from an another thread. An another entry on the TODO list
autumn haven
Actually I just thought of something @waxen fiber. Try using https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.result
ocean root
It'll raise the invalid state one
lavish forum
@autumn haven what are async fuction?
autumn haven
Do you mean async def? Those are called coroutines.
lavish forum
ya
ocean root
@waxen fiber if you wanna know how yield from and await *roughly* work read https://www.python.org/dev/peps/pep-0380/#formal-semantics
autumn haven
ocean root
hmm
autumn haven
oops
ocean root
yeah
autumn haven
But a coroutine can specifically be a coroutine method or coroutine function, similar to subroutines. Coroutine function probably doesn't need to be it's own term though.
ocean root
confusing terminology all around considering how the generator API can also be considered coroutines as they're 2-way
I think they tried to push the term "native coroutine" at one point
I like the Trio documentation for this as it doesn't confuse you with terms; it uses "async function" for an async def which makes sense, and then only later on introduces that a coroutine is the return type of the innards of an async function
https://trio.readthedocs.io/en/stable/tutorial.html#async-functions
autumn haven
I completely agree with it being worded in a confusing manner. The main difference has to do with the blocking vs non-blocking rather than having mutliple points of exit and entry
Nathaniel is great at explaining things. I've interacted with him a few times, mostly on discuss.python.org
ocean root
He takes naming really seriously
oh I didn't even know discuss was a thing
I only knew about the mailing list
which one is more active
autumn haven
I'll admit that I'm not particularly well versed with the 3rd party async libraries for Python, I've mostly used asyncio. But it wouldn't be a bad idea for getting ideas, Yury is quite familiar with them
Yeah we use both, Discuss was just created towards the end of 2018
I personally prefer Discuss and find it much easier to follow conversations on there. But we've had a long history of using the MLs so it's going to take some time
ocean root
makes sense
autumn haven
It's taken as a serious alternative though considering all of the core dev nominations and polls are done through Discuss
stray kite
Aeros sama, i wanted to ask some1 before, why is asyncio using a _log_traceback attribute at Futures, meanwhile it could have just use a 4th state, like:
PENDING = 'PENDING'
CANCELLED = 'CANCELLED'
FINISHED = 'FINISHED'
RETRIEVED = 'RETRIEVED'
if debug mode is off, ofc RETRIEVED would not be handled / used anymore.
When i implemented my async library, i noticed, that some error messages are missing which were at asyncio, so i came up with this idea to solve it easily
viral goblet
I have a function timeConverter that I am trying to call from another file
@commands.command()
@commands.has_permissions(manage_messages=True)
async def mute(self, ctx, member: discord.Member, time='1 hour', *, reason='None given'):
await timeConverter(ctx, time)
print(timeSeconds)
async def timeConverter(ctx, time):
times = time.lower().split(' ')
try:
....
elif times[1] == 'hours' or times[1] == 'hour':
timeSeconds = int(times[0]) * 3600
print(timeSeconds)
return timeSeconds
But when I try to return the value of the function it doesn't return anything
ocean root
You don't assign the result to anything
viral goblet
so timeSeconds = await timeConverter(ctx, time)
ocean root
This is more fit for #discord-bots as it has a whole system for type hints and converters
yes
storm peak
Does someone have a link to some kind of tutorial or just an article about making threading and asyncio work together nicely?
red wigeon
I don't know of any but why do you need to do such?
storm peak
That's the question I am always being asked - well, you can say, out of curiosity
cursive karma
I'm going to ask you another thing
@storm peak what do you want to achieve?
For example, when using asyncio with a bot that generates images, you may want to move the processing of the image to another thread
Is that want you want?
storm peak
Nah, not really
cursive karma
You want to run two coroutines simultaneously in two threads?
storm peak
I have, say, some looped task that I want to run in the background without blocking the main thread. And in that task I want to call some functions in the main thread, if that makes sense.
cursive karma
Is the task that's blocking a coroutine?
storm peak
well, the point is that I need to do loop.run_forever() to make looped tasks, so that's why I need another thread
The task is a coroutine yeah
ocean root
just create a new event loop in the new thread
as long as you pass the event loop around (to your own and to 3rd party libraries) it'll work fine
cursive karma
So, then what you want is run two coroutine concurrently
ocean root
ah the good old "how to kill a thread" issue
storm peak
Hahaha
ocean root
does making them daemons work? if not then you're SoL
storm peak
Yes
ocean root
you'd have to implement some form of cross-thread event system
which I guess isn't hard
but it's not like thread.kill()
killing threads on demand is a horrible idea
storm peak
@cursive karma no, that is completely not what I am trying to achieve
thread.destroy()
cursive karma
Then, can you show an example of what you want to achieve?
storm peak
Well, I am just not sure if you will get me right. Let me write it down
So, there is a looped coroutine function that is performing some HTTP requests. There is a Client object, methods of which I wish to call. So, in the (another thread) I wish to do those requests and call the methods. But I am not sure how to accurately stop the thread
Simplified but that’s pretty much it
cursive karma
Then, why not just have a flag in the loop that's making requests and when it's set, stop it?
storm peak
Well yeah but like, not sure if it will really work in my case
The goal is basically to make the events, like in discord.py, for example, but without blocking the thread I would say
cursive karma
Because that loop is s coroutine, you don't need to use multiple threads
storm peak
actually what
cursive karma
Unless it's actually doing stuff like sync io or sleeping the thread
ocean root
>>> import asyncio, threading
>>>
>>> async def printer_task(message, delay):
... while True:
... print(message)
... await asyncio.sleep(delay)
...
>>> def start_asyncio(message, delay1, delay2):
... loop = asyncio.new_event_loop()
... loop.create_task(printer_task(message, delay1))
... loop.create_task(printer_task(message, delay2))
... loop.run_forever()
...
>>>
>>> t1 = threading.Thread(target=start_asyncio, args=('thread 1', 5, 9))
>>> t2 = threading.Thread(target=start_asyncio, args=('thread 2', 3, 10))
>>> t1.start()
>>> thread 1
thread 1
t2.start()
>>> thread 2
thread 2
thread 2
thread 1
thread 2
as long as you don't do cross-thread loop passing it will work fine
storm peak
Oh, haha
And another thing then
So, you can schedule some tasks into a loop, right?
ocean root
depends on what "schedule" means, yes
storm peak
Yeah, that loop.create_task, y'know
ocean root
yes
cursive karma
There's a thread safe wrapper around it
storm peak
Is there a way to run this loop, again, in another thread, with saving ability to schedule more tasks?
Getting crazy about not blocking the main thread here lol, guess I should stop
ocean root
I mean in my head
it sounds like it might work
but it'll probably break horribly
and you'll at least have fun breaking things
cursive karma
Btw, isn't the whole point of asyncio not blocking the thread?
storm peak
The point is, you need to block to not block, I think?
ocean root
asyncio and threading are different approaches
cursive karma
What I mean is that if you block the thread, then you can't use it as long as it blocked
storm peak
yep
cursive karma
So, blocking while you run an event loop will freeze the whole loop
Then, why blocking in the first place
limber sand
Can someone explain asyncio.Semaphore
Would i be right to say that it manages the amount of actions i can do at once
quasi flame
mostly right yes
its like a hall pass in school
semaphore = asyncio.Semaphore(20) means there are only 20 hall passes
you write your code in such a way that you can only access a resource if you grab a hall pass
def do_special_thing():
async with semaphore:
await _special_thing()
so if there are no more hall passes, you have to wait until one is free
(i never had hall passes in school, for what it's worth)
limber sand
me neither, but i understand. So what if there are no hall passes
quasi flame
then you wait
you wait until a hall pass becomes available
import asyncio
async def worker(semaphore, task_num):
async with semaphore:
print(f'Starting task number: {task_num}')
await asyncio.sleep(2)
print(f'Finishing task number: {task_num}')
async def async_main():
semaphore = asyncio.Semaphore(20)
tasks = [worker(semaphore, n) for n in range(200)]
await asyncio.gather(*tasks)
asyncio.run(async_main())
@limber sand
you'll see that only 20 tasks ever run simultaneously even though you started 200 tasks
cursive karma
The semaphore is a generalization of a lock?
quasi flame
you could say that
autumn haven
@storm peak Here's a pretty decent practical example of using threading and asyncio in combination with each other: ```python
async def shutdown_default_executor(self):
"""Schedule the shutdown of the default executor."""
self._executor_shutdown_called = True
if self._default_executor is None:
return
future = self.create_future()
thread = threading.Thread(target=self._do_shutdown, args=(future,))
thread.start()
try:
await future
finally:
thread.join()
def _do_shutdown(self, future):
try:
self._default_executor.shutdown(wait=True)
self.call_soon_threadsafe(future.set_result, None)
except Exception as ex:
self.call_soon_threadsafe(future.set_exception, ex)
For context, here's ThreadPoolExecutor's shutdown method (which is being called within _do_shutdown()):
def shutdown(self, wait=True):
with self._shutdown_lock:
self._shutdown = True
self._work_queue.put(None)
if wait:
for t in self._threads:
t.join()
I worked on that with some design assistance from Yury and Andrew (primary maintainers of asyncio)
autumn haven
@cursive karma I would recommend looking into https://docs.python.org/3/library/threading.html?highlight=semaphore#semaphore-objects. Each language has slight implementation differences with semaphores. Typically, any type of semaphore will use an internal counter, and each time it is acquired, the counter increments. If the counter is greater than or equal to the maximum, the semaphore is locked. It is unlocked once the counter is less than the maximum.
Also, the above link is the synchronous semaphore from the threading module. For the asycnio one, see https://docs.python.org/3/library/asyncio-sync.html?highlight=asyncio semaphore#asyncio.Semaphore. The API for both is very similar.
storm peak
@autumn haven thank you :) So I see, thread.join() might be what I was looking for actually
autumn haven
Np
@south crag For clarification on an earlier message, create_task() is considered to be a part of our high-level API (https://docs.python.org/3/library/asyncio-api-index.html). I had forgotten that it was originally just ensure_future() prior to 3.7. The somewhat confusing part is that ensure_future() actually calls create_task() if a coroutine is passed, so in a way ensure_future() could technically be considered a higher-level API after the changes made in 3.7. I've personally leaned towards using ensure_future() since it works for coroutines, futures, and any other awaitables. Specifically, coroutines are wrapped in a task via create_task(), futures are directly returned, and other awaitables are wrapped in a future. If you're 100% certain that you're only passing a coroutine to be wrapped in a task and not any other awaitable, you can use create_task() instead.
Also, keep in mind that the function asyncio.create_task() is not the same as the method loop.create_task(). asyncio.create_task() is the higher level version, to be used for the majority of cases. loop.create_task() is primarily for those who want to implement their own customized Task.
cursive karma
I thought it was the loop (library) job to implement Task and asyncio.create_task was just a wrapper around loop.create_task
bright geode
So I have a thread that updates a variable and another that reads it, do I need to use a lock?
The reader doesn't care if gets the older/newer version (it will get the newer version next time)
ocean root
if you don't care about getting the older version then there's no reason to use a lock
unless it's a resource that can end up in an invalid state
bright geode
ok good
minor plinth
What's the proper way to use multiprocessing and async together
I am using the snippet given in the answer to the question there https://stackoverflow.com/questions/45603116/async-multiprocessing-python
but I am not sure if creating async tasks in a worker would guarantee that they are executed in different processes
red wigeon
I know Trio has a good set of functions for sending tasks to other threads and processes
So if that's an option for you try it out
But no clue on the asyncio end. Wouldn't be surprised if it didn't exist at all
ocean root
Wouldn't it be to use EventLoop.run_in_executor(ProcessPoolExecutorInstance, cpu_bound_task, *args)
red wigeon
Something something CPU tasks better in other processes something something starving IO threads
minor plinth
data = await asyncio.gather(*(loop.run_in_executor(executor, cpu_worker, num)
for num in range(4)))```so that starts 4 different cpu bound workers
that execute cpu_worker async function
right
but will the subsequent calls that i make inside cpu_worker be executed in parallel
I'd like to start 20 threads inside of each process via
asyncio.create_task(another_coro())```but i am afraid those will be just put on a loop, and executed all in the same process concurrently
storm peak
https://github.com/NeKitDS/gd.py/blob/master/gd/events/scanner.py#L21-L129 I have done something terrifying
Plus it could be better in so many ways I think
minor plinth
🙏
viral goblet
Anyone have any clue what the best way to run a gevent task inside an async task would be?
sorta also d.py related
im currently running the two processes concurrently in 2 separate threads, although there is an issue, as the steam client part needs to be logged in to send messages the bot will never send messages
if you want to have a look at the abomonation
cursive karma
It's a trade bot for tf2?
viral goblet
no
it sorta forwards messages from a trade bot
and the steam thing is the gevent thing
i tried talking to the module maker and he hasnt responded yet so i was just interested if any of you would have a clue
cursive karma
🤷
balmy slate
gevent works by monkey patching the stdlib. I’m really not sure if it’s compatible with asyncio to begin with.
minor plinth
how would i make an event def that only runs once
for example
in sanic
@fallow plover.route("/")
async def DoOnce:
Oh frick
Hey dude im sorry for the ping
vague talon
Use caching?
dusk lava
Use a global variable
storm peak
def run_once(func):
def wrapper(*args, **kwargs):
if not hasattr(func, '_res'):
func._res = func(*args, **kwargs)
return func._res
return wrapper
>>> s = [1,2,3]
>>> @run_once
... def f(s):
... return s[-1]
...
>>> f(s)
3
>>> s.append(4)
>>> f(s)
3
``` 🤔
verbal aurora
I'm trying to write a script to poll some web resources and take actions on new responses. I want to make my application mostly async, especially the action part I'm writing myself entirely, but the library I need to use for the poll requests and response parsing is synchronous, using urllib under the hood. I made a fork and am trying to port it to aiohttp, but that takes more time and effort than expected.
Therefore I'd like to realize my project with a workaround for now, by making blocking calls to this synchronous library from my async context. Would it make sense to do each of these calls in a separate Thread so that my async event loop keeps running and can process the actions or keep triggering more requests in parallel? And if so, how might I go about that? Are there any established patterns for asynchronously controlling synchronous worker threads?
ocean root
Using a ThreadPoolExecutor with loop.run_in_executor(executor, sync_func, *args)
verbal aurora
oh. I haven't read about run_in_executor yet. Will search that, thanks.
storm peak
# Me: hey I want to run two apps
# Heroku: You have limited dyno hours
# Also Me:
import threading
threading.Thread(target=main_1).start()
main_2()
How can I schedule a coroutine to run concurrently in an already running loop from a synchronous context?
verbal aurora
I think with asyncio.create_task(coro()) (3.7+) or asyncio.ensure_future(coro()) (old) https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task
storm peak
asyncio.ensure_future(coro(), loop=loop) I think
formal snow
soo, asyncio is worth when you want to do other things in the meantime that you made a web request and the server that hosts it takes some time before answering, so it prevent this kind of blocking which ain't due to local performances.
Anywhere else?
don't tell me where there is blocking in general cuz that ain't true, asyncio is far from the key to prevent all blocking
formal snow
no it isn't
it is in some cases, when your script depends from other web sources,
but not in general
red wigeon
It does in every case, that's it's job. You may be doing actions that cause it to stop functioning properly but that's not fair to say it's not the solution
Async may not always be the best solution, sure
formal snow
you're wrong, for example computer operations will make it block anyway
formal snow
everythign blocks but some blocking can be avoided
especially the ones web related
and my question was which one else?
red wigeon
All can be avoided practically
formal snow
hm no
red wigeon
But if there's no asynchronous library that doesn't block and you can't be bothered to write one yourself, you use am executor
formal snow
blocking due to computer operations, using asynchronousy exclusivly, can't be avoided
red wigeon
This is what am executor is for
formal snow
i'm working with asyncio only to remind u
on one thread, one processor, etc
so don't get me wrong when i say that computer operations blocking can't be avoided with what i'm working with
red wigeon
You could always write it yourself at the most primitive level
But often an executor is good enough
formal snow
well you don't answer to my question you know, i'm not asking for solutions it's just curiosity
https://discordapp.com/channels/267624335836053506/630504881542791169/631750628972036096
but thanks anyway for trying to help :3
red wigeon
Your question was based on false assumptions
formal snow
nop
red wigeon
I really don't understand why you're being so stubborn. This isn't how a question-answer interaction goes
If you're really unable to avoid blocking then there's not much reason to use asyncio in the first place no? As most of the time it will only work as a normal synchronous program would?
formal snow
my question is just about what asyncio can do
curiosity
and i know that its not the key to ALL blockings, so then which kind of blocking except web related
this was my question
red wigeon
Anything given you can get around blocking in any case. Just that it's not always worth the effort and there's not much gain
Async is perfect for long IO bound tasks
formal snow
what are IO tasks?
red wigeon
Any time where you're waiting for something to complete when you could be doing other things
In/out. Web requests, file ops, etc.
red wigeon
Web requests and file operations are a massive category. "Any time where you're waiting for something to complete when you could be doing other things" is generally the criteria for a use case. Apply that to where you want
It's especially for great for anything that requires concurrent connections
Like with a connection pool to a Postgres server for example
formal snow
i see
by file operations, does this also include local file operations?
i mean on local server
cuz i thought that this kind of action were just like computer operations
red wigeon
Yes
formal snow
so like the file writing for example,between what is there concurrency? doesn't it run on a same thread than the rest of the script?
i mean, how does sth get written to a local file
i don't understand why this could be avoided with asyncio whereas big loops of data couldn't
@red wigeon
red wigeon
Asynchronous file operations are OS specific. I know asyncio doesn't support this. I think Curio and Trio do. But there's an executor wrapper implemented by aiofiles that will do what you want without blocking
formal snow
could we say that asyncio is effective only when at least 2 threads are concerned, both being on a same computer, or not
storm peak
It is very effective on its own if it is used correctly
With two threads, if you accurately manage them, even though using asyncio+threading isn’t really good idea, it becomes maybe more effective, you can say
formal snow
i didn't say i was using threading in my scripts
for example when u make a request to a seerver, a thread is running for them on their server, and a thread is running for me (my single threaded script)
@storm peak
storm peak
Not sure how does that count as two threads but okay
formal snow
well do u see what i mean?
storm peak
I mean you said the obvious thing right above here and I don’t see how there are 2 threads except that 1+1 is 2
autumn haven
@formal snow I'm not certain where you're getting the impression that asyncio is only effective when two threads are being utilized. Also, for non-network use cases, see https://docs.python.org/3/library/asyncio-subprocess.html#asyncio-subprocess and for interprocess communcation via OS signals (https://en.wikipedia.org/wiki/Signal_(IPC)), see https://docs.python.org/3/library/asyncio-eventloop.html#unix-signals. In order to have any understanding of what asyncio can do (or any library for that matter) you need to read through the documentation to familiarize yourself with the API and then look at some other examples of it being utilized. It sounds like you're assuming a lot without having done a substantial amount of research.
Also, see https://docs.python.org/3.9/library/asyncio-dev.html#running-blocking-code for performing OS operations without blocking.
autumn haven
@cursive karma That's why I said it was for those who want to use their own implementations, but perhaps I worded that a bit poorly. For implementing customized task creation you can change loop._task_factory (which is used instead in loop.create_task() if one is configured) via loop.set_task_factory().
@south crag asyncio.create_task() is ran against whichever loop is currently running, not just the default loop.
If a task factory is not configured, loop.create_task() uses asyncio.Task (technically tasks.Task) to instantiate one using the coroutine passed to the method.
quasi flame
also consider Trio if you're new to async and want a somewhat higher-level, cleaner API and don't mind having less library support
or if you're not new to async and you're sick of asyncio's verbosity and impedance mismatches all over the place
(and its kinda-bad error handling)
formal snow
@autumn haven thanks a lot for yor explanations.
Now just to make sure i understood well what an I/O bound task is, - i tried to read several definitions - those are tasks which speed depend of a "subsystem" which is for example another thread, a subprocess, a distant web server, etc...
Right?
sorry if i may seem to be weird asking those questions, but concurrencies really interest me programming wide :p
and i feel like before undderstand them i need to understand other stuff before
ocean root
yes
But it can be something as simple as trying to open a file
Python's async/await is useful for "long" IO like HTTP requests
slate current
literally all IO is 'long'
And in a sense, you were right about the 'at least two threads involved' thing @formal snow - (It wouldnt be incorrect to say) async/await is a way of writing code that describes the points at which we are interested in data that might not exist yet - which, ultimately, a thread other than the active event loop will be in charge of. Its just that the syntax's purpose is to abstract that away, so describing it in such a way is somewhat redundant
ocean root
?
"At least two threads involved" is a really hazy concept
async/await runs on a single thread
if you count everything external as "another thread" then sure
quasi flame
@formal snow i/o is really "anything where your process would normally be asleep waiting for something else to finish"
sleep() is effectively i/o
ocean root
all IO is not created equally
quasi flame
no but if you need a baseline definition for it, "process is waiting and not processing something" seems to be general and correct enough
ocean root
nah I wasn't replying to you
formal snow
ah okay guys
i understand why my phrasing led to disagreements and can still
but i get fully what async and I/O is now^^
thanks everyone that contributed sending answers
helped a lot
i'm satisfied now of having understood sth xd
autumn haven
To elaborate on what an I/O task would be, it's effectively any form of commucation from the CPU (or other processing system) to separate device. A task is considered to be I/O bound if more time is spent on waiting for a response than on processing (if more time is spent on processing, it would be CPU bound). The reason why asynchronous programming is well suited to networking is because the amount of potential time where resources are idle is typically the longest and most influenced by external factors, so the amount of other tasks that can be acomplished while waiting on the result of an I/O bound task is substantial. But it can also be useful for more localized I/O, such as sending a seek to a storage device (output) and receiving a response (input). In a very broad sense, a task can be bound by CPU, memory, or I/O. The asyncio library (AsyncIO) is primarily targered at I/O bound tasks.
autumn haven
@ocean root Just finished adding some source code links to the asyncio docs (https://github.com/python/cpython/pull/16640), it will be included in both the 3.8 and 3.9 docs. The changes were omitted for 3.7 since there were merge conflicts and 3.8 will be the new default documentation version once it's released (October 14th, assuming no last minute surprises or issues).
minor plinth
question about async and flask
here in aiohttp there is a server example that seems to use async to handle incoming requests
I thought, assumed, that the model for flask was like this:
-
Request hits the http server e.g. httpd
-
Httpd looks at the request and figures out it's a route to be handled by a python script
-
Python script gets run and fires up an instance of flask
-
Your Flask code takes a while getting the response together, meanwhile...
-
Httpd looks at the request and figures out it's a route to be handled by a python script
-
Python script gets run and fires up a second instance of flask
-
Even though both instances of flask are running at the same time, they are in seperate proccesses and can only, potentially, communicate via the DB or some other IPC
But uh, this aiohttp feature makes me think I might be super wrong about this - is flask, or the python webserver under it, just kinda running all the time waiting for requests, and the requests could potentially communicate through variables in the code
autumn haven
@minor plinth Note that I'm not especially knowledgeable with flask or aiohttp, but I do know one of the maintainers of aiohttp (Andrew Svetlov) since he's also one of the two primary maintainers of asyncio. So if you aren't able to find a solid answer within the next week or so, I can potentially forward the question over to him. Also, not to nitpick, but there's a distinction between async and asyncio. Within Python, async refers to the async/wait syntax (async def, async for, async with, and await) while asyncio is an asynchronous IO library built on top of the async/wait syntax, aiohttp specifically uses asyncio.
here in aiohttp there is a server example
Could you link the specific example?
this aiohttp feature makes me think I might be super wrong about this
What feature are you referring to exactly?
I may not be able to provide an exact answer to your question due to my lack of personal experience with flask and aiohttp, but more details would certainly help.
minor plinth
@autumn haven I guess my question isn't really about aiohttp so much as whether my understanding how how python web applications generally work is off
The example I was talking about was simply the server example on their main page: https://aiohttp.readthedocs.io/en/stable/
A more concise version of my question is whether normally:
- In a webserver a new instance of python gets run to server every http request normally or
- In a webserver one instance of python gets started when the webserver starts and serves each http request sequentially from a queue unless you use asynchttp
the reason I wondered about that was the way asynchttp.web is used to handle requests in the example
this isn't a "help I'm stuck" question so much as a "help I think I'm wrong about something" type question
I would 'try it out' but not even sure how to test this
quasi flame
@minor plinth it would happen the same way more or less
The asyncio socket server would listen on localhost:8807 or whatever, and Apache would reverse proxy the request
Or it could listen on a unix socket
Right now, the problem with flask is that it does both I/O and other work
And the i/o is synchronous
Theres a whole different world of WSGI, uWSGI, and ASGI
But that basically does the same thing, except it imports your flask application and wraps it in a more sophisticated HTTP server instead of running the development HTTP server
Nginx for instance supports uWSGI which means bypassing the whole reverse proxy business, but that's a special case and I don't see it used very often
Also worth noting that the path is actually baked into the HTTP request
minor plinth
interesting, that makes me think it's option 2
maybe
I think I can test this by running a flask application and incrimenting some kind of global variable in a request handler. If I run 2 requests and the variable actually gets incrimented each time, then it's the same instance of flask. But what you are saying suggests it might depend on your deployment maybe
autumn haven
Right now, the problem with flask is that it does both I/O and other work
And the i/o is synchronous
From my understanding, the solution to that is for the package maintainers (flask in this case) to implement a asynchronous version of their API in addition to the current synchronous/blocking one. Without knowing the internals of flask, I have no idea what that would involve. Django is gradually adding asyncio support (see https://github.com/django/django/pull/11650) and it's on the roadmap for Django 3.0.
quasi flame
@autumn haven better still, sans-io refactor
This is werkzeug though
Im not sure if you can just toss a flask app behind an existing ASGI server
That said you have libraries claiming to be flask-equivalent that support ASGI out of the box
That said it looks like my understanding of how WSGI/ASGI works is somewhat inverted
autumn haven
It seems like that's Django's plan as well (https://docs.djangoproject.com/en/dev/releases/3.0/):
Django 3.0 begins our journey to making Django fully async-capable by providing support for running as an ASGI application.
This is in addition to our existing WSGI support. Django intends to support both for the foreseeable future. Async features will only be available to applications that run under ASGI, however.
quasi flame
Django has an extra can of worms in that it has to support an ORM too
autumn haven
I would think the ORM itself wouldn't be much of an issue compared to the database driver as far as asynchronous support goes. What part of Django's ORM would have to be asynchronous?
quasi flame
Saving and fetching query results
autumn haven
Ah, I was under the impression that Django's ORM simply generated the queries and delegated the fetching and receiving to the respective DB driver.
quasi flame
Thats true but you cant have save be async only sometimes
Technically you can and they are no stranger to the meta-programming magic required
But that would be wildly confusing and I hope they don't go that route
autumn haven
Yeah I have no idea, I have little rather understanding regarding the implementation details of Django. The only part that I'm specifically aware of is implementation of their models using metaclasses, but I'm only somewhat familiar with that because of a PyCon lecture done on metaclasses that used Django's as a reference.
stray kite
@autumn haven do u know if there is a better way to do asyncronous io (not sockets, but buffers, like files or so), than using executors? I did not find anything at asyncio (sad), so i looked at trio, what used executors, so i made async io-s based on that idea.
verbal aurora
aiofiles library? https://pypi.org/project/aiofiles/
stray kite
so it uses executors as well
Thanks.
at least i noticed, that i used async init and not async new, so if not every arg was passed, del could raise AttributeError
balmy slate
It's exciting news that Django's going all in for async support, but it seems we won't have an asynchronous option for the ORM layer anytime soon:
Note that as a side-effect of this change, Django is now aware of asynchronous event loops and will block you calling code marked as “async unsafe” - such as ORM operations - from an asynchronous context.
quasi flame
oh huh
stray kite
will block you calling code marked as “async unsafe” this seems like a lot of work as well
balmy slate
yeah i feel like the big selling point of django is the builtin ORM and database management system... if you want async and can't use that without blocking (or having to remember to use an executor, etc) it's not very ideal
quasi flame
that, or, its there to discourage you from using django with async 🤔
balmy slate
but for whyyyy
quasi flame
probably easier than building a whole async api and integrations with multiple new database backends
viral goblet
I've got what I believe to be the best way to solve my issue (chat messages would only be sent if the login was in the same file I think) but I don't know how to start my task
https://gist.github.com/Gobot1234/cbfcd75e768415ed42ef44e2d571343c#file-steam-py-L8
you can't await the task cause it's in the _init_
stray kite
your task will be started when the loop is started up
slate current
You dont do anything async in steamlogin?
viral goblet
no
slate current
Why not just replaceasync def with def
viral goblet
and then just call it?
slate current
Yea
minor plinth
Does anyone know if pyramid web framework is gonna do async io?
autumn haven
@stray kite Have you looked into the the streaming/streams API for asyncio? https://docs.python.org/3.8/library/asyncio-stream.html
stray kite
i mean not sockets,
like what Byte said aiofiles
but without executors
or do streams work on files too
i already have http client, websocket client, async ios (with executors) and a simple sqlalchemy async wrapper implemented as well
autumn haven
Hmm, I believe there were some plans to implement filesystem support, but I think the current workaround is indeed using aiofiles (or a customized implementation that's similar to aiofiles using threads).
stray kite
I use similar system to aiofiles. Till today i did not even those exsist 
. Wanted to make sure, if the only way was using executors.
autumn haven
Is there a particular reason why using an executor is an issue?
stray kite
Nope, but i am maximalist, so if there is a better way, i ll use that
stray kite
autumn haven
Although, you could potentially look into the Executor (https://github.com/python/cpython/blob/2b7dc40b2af6578181808ba73c1533fc114e55df/Lib/concurrent/futures/_base.py#L547) and ThreadPoolExecutor (https://github.com/python/cpython/blob/193366e25c4f84a58d2f6c6c577fd9f0143dc6e1/Lib/concurrent/futures/thread.py#L108) classes to implement your own customized Executor class. I'm not certain that this would result in something better, but it might be worthwhile to experiment with it.
storm peak
Little fact: you can escape a url with <> if you wish to hide an embed.
stray kite
i already have executors implemented
It is actually pretty easy, just need an async -> sync queue, a thread what polls from the queue, executes and sets result (or expcetion), and a wrapper ofc to make sure everything is foolproofly handled.
autumn haven
@storm peak Thanks, I'm primarily used to using markdown syntax on GitHub that uses name for url embedding which slightly differs from the markdown syntax on Discord.
minor plinth
is AIOHTTP better than flask ?
autumn haven
What do you mean "better"?
minor plinth
oh
autumn haven
I would recommend reading the documentation of each and seeing what better suits your purpose rather than asking which one is "better".
at least the intro pages and examples
storm peak
it's like
I think you can do anything with aiohttp though
In terms of what Flask can do
storm peak
Well iirc aiohttp web app is wsgi
covert token
In the documentation for asyncio.Future.add_done_callback it says that it
should only be used in low-level callback-based code.
What exactly would be an appropriate low-level use case for it then?
stray kite
For example if u implement sleep, timeout, when u chain futures (shield for example), when u collect future results (gather for example). It is also used at the implementation of Task at many places, because callbaks make the generators to continue.
@covert token
Other frequently used low level methods are: loop.call_later and loop.call_soon, if u are interested.
covert token
I feel like I'm using it incorrectly but I don't know how else to write some stuff.
Like I wrote this for example:
devout basin
I have a dumb question: when should one use asyncio vs concurrent.futures executor.submit() type stuff? Is it simply that the former gives you more control?
covert token
Depends if you want parallel processing or single threaded concurrency.
asyncio has its own thread pool executor if you want to do blocking operations within an async context.
For stuff that's highly I/O and network bound use asyncio. If you want parallel processing capabilities using independent OS threads that's an entirely different thing.
stray kite
thats a solution as well pupper. U can also create tasks, which do the query and then print out the result without callback. asyncio.gather has a return value as well, and u can check the results of it and print out them there.
covert token
Yeah I think I thought to do that originally and then decided to avoid wrapping the queries with the printing to make it cleaner.
stray kite
also
if not query_task.exception(): -> u should check like if query_task.exception() is None:, but future.exception() can raise CancelledError or InvalidStateError still, so i should have use try-except and .result()
covert token
Okay, thanks
stray kite
well, u wont get that 2 exception at this case, just checked if the method can raise or not
covert token
Yeah, I omitted it for brevity, but it's better to use a try, except, else normally.
stray kite
thats a long try except block 
devout basin
Hmm okay I'll definitely need to read up on asyncio then. Only used concurrent.futures and multiproc myself.
Thanks!
terse ether
How can you use await outside of an async function so you can run an async function?
covert token
You can only use await from inside an async context
terse ether
So how would I run the async function then?
ocean root
You can't
You can schedule it to the event loop
This sounds like strange design, do you have a concrete example
terse ether
I tried asyncio.run but I just get; AttributeError: module 'asyncio' has no attribute 'run'
covert token
What Python version are you on?
terse ether
3.6.9
covert token
New in version 3.7: Important: this function has been added to asyncio in Python 3.7 on a provisional basis.
terse ether
fuck
ocean root
In < 3.7 you do
loop = asyncio.get_event_loop()
loop.run_until_complete(async_func())```but async_func is like a main entry point
you can only do this once, you can't try to run the same event loop twice
Here's a good guide on async functions even though it's for a separate library https://trio.readthedocs.io/en/stable/tutorial.html#async-functions
formal snow
Okay guys so I need help to understand this: http://prntscr.com/pifq82
PS: I'm aware that there is a subprocess thinggie included in asyncio module, which i guess i should use for it to work asynchrnously.
And I assume that if it didn't work asynchrnously is because subprocess module isn't designed for asynchronous by default and needs more implementations for it to be. If what I assume is right, it leads to these following questions:
- What would be those implementations? (what does a module <of a system in which async usage is relevant; e.g if I/O bound> need to implement to be asynchronous compatible?)
- How to know if a module is async compatible or will just do some syncronous blocking like subprocess module does?
ocean root
The "TLDR" of asynchronous python is that the scheduler can only schedule other things when you explicitly give it control back. This happens with the await statement
usually lack of await means it's not compatible and will need to be ran in an excecutor
formal snow
@ocean root so asyncio.subprocess is written totally different right?
i mean in this version of subprocess, awaits are mostly present and it uses sth like aiofile to communicate with stdin, stdout and stderr
ocean root
yeah
formal snow
okok makes sense
and what would be the 2 most basic modules allowing to make other asynchronous modules:
aiofile (communicate asynchrnously to local files)
and aiohttp (communicate asynchronously to APIs)
right?
minor plinth
How can I await a statement on class instance creation?
so essentially an async init function
ocean root
the preferred way is creating a classmethod
if you want to be really really hacky thought you can use this metaclass py class AsyncMeta(type): async def __call__(self, *args, **kwargs): obb = object.__new__(self) fn = obb.__init__(*args, **kwargs) if inspect.isawaitable(fn): await fn return obb
minor plinth
@ocean root the async function needs to use self variables though
wait I could use parameters
covert token
Use async def __call__, or an async context manager using __aenter__ & __aexit__: https://docs.python.org/3/reference/datamodel.html?highlight=__aenter__#asynchronous-context-managers
autumn haven
@terse ether For running coroutines, see https://docs.python.org/3.9/library/asyncio-task.html#coroutines. If those aren't suitable and you want asyncio.run(), something like this might work. Here's a rough and very minimalistic example (doesn't include errors or Task cancellation): ```python
def run(coro):
loop = asyncio.new_event_loop()
try:
asyncio.set_event_loop(loop)
return loop.run_until_complete(coro)
finally:
loop.close()
The main thing to do is setting up the event loop with `new_event_loop()` (or checking if one exists with and then running the coroutine using `loop.run_until_complete(coro)`. I would recommend looking into the event loop API (https://docs.python.org/3/library/asyncio-eventloop.html) if you can't use 3.7+. Is there a particular reason why you can't upgrade to 3.7+?@minor plinth Out of curiosity, what is your specific use case for needing to use await within class instantiation?
scarlet raven
Yeah you shouldn’t need to actually await anything on class instantiation. Nothing does this and with fairly good reasons, so if you feel like you do, describe the use-case because almost guaranteed there’s a better way to do it
unborn geyser
I'm trying to check url mime type (headers) that equals to these: "image/png", "image/pjpeg", "image/jpeg", "image/x-icon". The only thing i receiving only this text/html;charset=utf-8.
Here is the code of is_image:
IMAGE_MIMES = (
"image/png",
"image/pjpeg",
"image/jpeg",
"image/x-icon",
)
async def is_image(url):
timeout = aiohttp.ClientTimeout(total=600)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(url) as response:
mime = response.headers.get("Content-type", "").lower()
if any([mime == x for x in IMAGE_MIMES]):
return True
else:
return False
Just decode it then
unborn geyser
I'm trying to get it from imgur (https://imgur.com/gallery/YqLTsAG), @stray kite
@storm peak how?
storm peak
If it is in base64, decoding is simple
unborn geyser
I don't really think so
Hm. Did you change something? @stray kite
Can you show how it works?
:0
I thinks it's just a page
But i need the image
I'm just trying to get images from the chat and it can be the url
minor plinth
@autumn haven when I instance that class, I want it to create a discord embed, myui = await CustomUI(ctx, embed)
Something like that
unborn geyser
I don't think it's gonna work with other sources
:|
Well, it's time to find better solution to get content-type headers
quasi flame
is this for discord embeds?
quasi flame
oof. i had a similar problem
i was hoping that there'd be some kind "embed type" i could get
but nope just the URL and the raw HTML
unborn geyser
i was hoping that there'd be some kind "embed type" i could get
Same
So, did you found solution?
quasi flame
guess based on URL
i kind of stalled development on my thing because i was too lazy to start hacking around w/ heuristics
ill get around to it eventually
quasi flame
its fine i got other stuff to do in the meantime and nobody else has had my idea yet 😛
i mean you can get the <img> tag out of the HTML
right?
unborn geyser
Yes, but you need to write parser for each
quasi flame
no thats what im saying
if the embed is "mostly just an <img> tag" then you're good
maybe with special cases for imgur
ocean root
request the first few bytes
check them to figure out the file type
if they are good then request the rest
terse ether
Is it possible to run an async function from a normal function?
ocean root
depends on what this means
do you want to start the event loop? then use the run method of whichever library you're using
if you want to schedule something then that can usually be done too (e.g. asyncio.ensure_future)
if you want to run an async function from a regular function directly, that's not going to happen
unborn geyser
@stray kite can you show the source code?
I'm really curious
That will help to get those images
Well, do you have a github page?
My wrapper can only get images from embeds
Because, eh, it's easy
So, um, can you send me the link?
All
If i can
Yep
can i show mine? :3
oh no
i forget to change this pic
Where?
And yes, where can find that i want?
That image parser
I mean, url parser
Ah, yes
quasi flame
huh that library isnt related to discord.py at all?
quasi flame
cool
why did you write that
instead of using d.py
(just wondering)
because of the voice encoding differences?
round warren
@stray kite So could you run a Twitch Bot and a Discord Bot in the same project?
That would be awesome
Flask aswell?
Then I am very intersted in using it
Is there a documentation?
worn solstice
How can I manage multiple WebSocket connection in a script?
I mean listening and processing the data I get from it.
Separately.
Why not more?
🤔
I think I didn't get the question then.
At the moment I am using a doing a list comprehension to with await _.ws_connect(...) to establish a connection.
Could it be asyncio.gather()?
Oh, okay.
Thank you very much.
Cool. 😄
quasi flame
gather is basically a wrapper around wait
ocean root
not really?
cursive karma
Not exactly
cursive karma
Isn't it just multiple Future running concurrently?
quasi flame
Holy crap, you wrote your own discord client and async library
ocean root
how's it work without an executor system
cursive karma
uvloop :thonk:
quasi flame
@stray kite have you seen trio?
slate current
I dont understand. Why not just ```py
import asyncio
from functools import partial
gather = partial(asyncio.gather, return_exceptions = True)
gather_till_exc = asyncio.gather
gather_till_first = partial(asyncio.wait, return_when = asyncio.FIRST_COMPLETED)```Is passing the loop manually useful somehow?
Well I certainly agree that asyncio's api is bloated to hell, but the solution isnt just half-implementing its convenience functions. Those gather functions look like theyre going to swallow up a bunch of errors that the library's function handles
Why dont you like trio?
slate current
sounds like trio a very useful feature 😛
minor plinth
how to execute async function?
My code is
```python
import os
import asyncio
import dhooks
from dhooks import Webhook
from termcolor import colored
loop = asyncio.get_event_loop()
ihook = input('[*] Webhook: ')
hook = Webhook(ihook)
async def bye():
await hook.close
exit()
menu=True
while menu:
print("""
0. Exit\n
1. Spam webhook\n
2. Show webhook information\n
3. Delete the webhook\n""")
selection= input("> ")
if selection =="0":
bye()```And if i select 0
I get this error
If i run the code and select the 0 then i get this error
main.py:49: RuntimeWarning: coroutine 'bye' was never awaited
bye()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback```to run an async function you must await it
like
await bye()
but to do that, you need an asyncio event loop
minor plinth
It wont let me await outside a function
🤦🏽♂️
File "main.py", line 49
await bye()
^
SyntaxError: 'await' outside function```Id just get this error
vague talon
in python 3.7 you can just go asyncio.run(main())
so wrap your while loop contents in a function called main for example
vague talon
or since you've got your loop already you can go
loop.run_until_complete(main())
K
vague talon
yeah pretty annoying tbh; I think newest pycharm has a feature where the python console allows you to await, I should try that out soon
minor plinth
Asked this before, got somewhere but still trying to piece together the pieces.
I want to create a class that holds methods working with discord.py, some methods for working with custom discord embed uis.
I want an instance of this class: myui = await CustomUI(ctx, embed)
to ctx.send(embed=embed)
This rules out the use of class methods
I am unsure whether to use __new__ or __call__
and how that works
minor plinth
@minor plinth someone posted something where you override the new method
modest viper
are there any newbie guides for learning how run_in_executor and creating loops work?
sweet vine
I found the python docs to be most useful, did you check that out yet?
modest viper
Hmm not yet, is it structured so that you can learn everything little by little instead of throwing you every single method in a class?
modest viper
Thanks!
minor plinth
I just don't understand what is going on
minor plinth
Ok, so I want to use __call__ to await a statement using an argument passed in when creating a class
is there a way I can do that without defining extra classes?
Thanks for the explanation btw, really helped alot
minor plinth
yeah but that means I need to call async_create to create
I want to do ui = await CustomUI(ctx, embed)
and it does async commands
minor plinth
Why can't you just call the super().__new__
IDK, or return self at the end of the function manually
verbal aurora
Is there any way to examine a ThreadPoolExecutor's status, like busy/total thread count and queue length?
minor plinth
So could I do
class CommandUI:
def __init__(self, arg, arg2):
await self.__new__(arg, arg2)
async def __new__(cls, arg, arg2):
await arg.function(arg2)
self=super().__new__(cls)
return self
how would I pass self variables to __new__?
wait I get it
class ComandUI:
async def __new__(cls, arg, arg2):
self = super().__new__(cls)
self.value = await arg.function(arg2)
return self
right?
Thank you so much for the help!
umm, pycharm says __new can't be async tho
minor plinth
@stray kite even though pycharm says new isnt async, should it still work?
minor plinth
@stray kite Is there a way to do client.wait_for passively, without holding up the command?
Discordpy btw
minor plinth
is there a way to send data from synchronous to asynchronous
or to call async funct from sync
craggy fjord
Calling async function from sync environment does not make much sense as it will basically block for the whole execution (if you wait for the results that is).
Otherwise you could try to interact with the event loop directly.
What eactly do you try to achieve @minor plinth ?
minor plinth
it's kind of intense and a lot to explain
AlertsFile = 'Alerts.db'
connection =sqlite3.connect(AlertsFile)
connection.row_factory=sqlite3.Row
cursor = connection.cursor()
client = discord.Client()
def price(stock):
data = requests.get('https://query2.finance.yahoo.com/v7/finance/quote?formatted=true&lang=en-US®ion=US&symbols='+stock)
todos = json.loads(data.text)
rows = (todos['quoteResponse']['result'][0]['regularMarketPrice'])
rows = str(rows)[7:-1]
rows = rows.split("'")
result = (rows[3])
result = "301" ##MADE UP PRICE FOR TETT
return result
def del_and_update(ids):
cursor.execute('SELECT * FROM alerts')
cursor.execute("DELETE FROM alerts WHERE id = "+str(ids))
connection.commit()
#del_and_update()
def chkprice(ids, user, stock, d, price1):
if str(d) == ">":
if price(stock) > price1:
print (stock+' is greater than ' +price1)
del_and_update(ids)
if str(d) == "<":
if price(stock) > price1:
print (stock+' is less than ' +price1)
del_and_update(ids)
asyncio.sleep(2)
async def my_background_task():
await client.wait_until_ready()
while not client.is_closed():
cursor.execute('SELECT * FROM alerts')
row = cursor.fetchone() #GETS FIRST ROW IN ALERTS.db
while row is not None:
ids = row[0]
user = row[1]
stock = row[2]
d = row[3]
price1 = row[4]
Thread(target = chkprice(ids, user,stock,d,price1)).start() ##STARTS LOOP THREAD
row = cursor.fetchone()## GETS NEXT ROW IN Alerts.DB
connection.commit()
@client.event
async def on_ready():
print ('Running')
client.loop.create_task(my_background_task())
Ok, then just in short - calling async stuff from sync stuff may be possible in some cases, but is generally not a good idea.
minor plinth
so I have the Thread(target = chkprice(ids, user,stock,d,price1)).start()
then in the chkprice funct... I needed to post to discord
can't do that because async
if that makes sense
craggy fjord
Oh, so you try to use both threading and async.
minor plinth
i tried to just do async route alone but it's so much slower when I searching through tons of stuff in the sqlite3 db
or running the price checks on each thing in the sqlite3 db
should I say
are you thinking I should just make it all async and that would be better?
craggy fjord
That's probably due to the fact your sqlite lib is synchronious.
minor plinth
ah yes
craggy fjord
Well, yeah, I'd recommend to stay away from trying to mix different paradigms like async+threading unless you very specifically know why you need that and what the pitfalls are.
craggy fjord
It depends on the client implementation. I'm not sure if there is an async variant available out there.
minor plinth
what do you think of this?
by the way thank you very much for you insight and help!
craggy fjord
You are very welcome. I see that for every row you run chkprice. Is that correct?
minor plinth
yes
craggy fjord
And for every chkprice, you run price, which in turn tries to retrieve data from query2.finance.yahoo.com.
craggy fjord
Well first thing I would recommend is use asynchronious http request instead of requests library, which is synchronious.
minor plinth
ahhh great advice I did not know that
craggy fjord
Second thing is you may want to fetch that yahoo data only once and then pass it to all the checks.
minor plinth
I honestly thought it was the other way around
craggy fjord
So that you don't do a request every time for every line. Do it once for all lines.
HTTP requests are inherently slow and that might be your biggest performance issue.
minor plinth
ahh
great advice!~
so the yahoo link changes with each iteration
it's getting stock prices for each ticker symbol
craggy fjord
Oh, I see.
minor plinth
in the list
craggy fjord
Well then your best bet is using async library instead of sync for http requests.
minor plinth
that does give me another idea though
craggy fjord
minor plinth
sometimes there are same tickers in the database I could check if there are multiple then only grab the data once for those instead of each time
ah awesome!
craggy fjord
I'd go for async http first.
It will reduce your processing time to the longest http request, instead of the sum of all http requests times.
If that's not enough - then it may make sense to look for other optimization approaches.
craggy fjord
No worries, it's why we are here. =]
g2g now. Good luck and feel free to ask if there will be any further questions. Someone else will help if I won't be here.
craggy fjord
I'd say so, yes.
minor plinth
thank you so much again have a good night! 🙂 🙂
stray kite
@minor plinth I dont know what that is.
minor plinth
It's a coroutine that waits for a user reply. My issue is, it holds up the command, and I want it wait for a reply passively, while the user continues to interacts with the command, and has a chance to trigger the wait_for at any time
minor plinth
what do you mean by creating a task
whats a task
sorry, new to async
ah, gotcha, so what would LOOP be in this case?
so asyncio.create_task(client.wait_for...)
a bit confused, sorry
minor plinth
client.wait_for is a coro that waits for the user to reply to a thing, before returning a result
I want it to do it passively
meaning its waiting for a responce
but the rest of the command can still continue
minor plinth
I see,
why can I not just do asyncio
ok
how can I stop the task
thanks
so what do you mean when you say you do not use asyncio?
wait so like task = await asyncio.create_task(self.wait_cancel_task())
then task.cancel
oh ok
wait so if I dont use await, do I need to do task.start or smth
oh ok
and this way, when I do create_task, it will continue on to the rest of the function, while the task is in the background
thank you!
alright
why the light theme
dark topaz
ocean root
this sounds like a hot mess
created with
__new__to avoid calling__del__
what does this mean
ocean root
why is del being ran an issue
I feel like code depending on whether an instance was created or not is extremely fragile to the point where it should probably be all deleted and created from scratch
autumn haven
@stray kite ```python
class test():
... def init(self,value=None):
... self.value = value
... def del(self):
... self.value
test()
If you want to be able to instantiate an object from the class without passing the arg for instance attribute value, you have to assign a default for the parameter value.
autumn haven
The same applies if you want to utilize __new__ (although this is not very conventional usage of it):```python
class test():
... def new(cls,value=None):
... self = object.new(cls)
... self.value = value
... return self
... def del(self):
... self.value
Or if you don't want users to be able to explicitly pass value to the constructor but want to utilize it later for something, simply set it to None in the __init__:```python
class test():
def init(self):
self.value = None
icy aspen
autumn haven
@icy aspen Probably the most effective way of calling synchronous functions asynchronously is through the usage of the method loop.run_in_executor(). See https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor for more details and a usage example. From the example you posted, the synchronous function would be io bound, so the default executor (ThreadPoolExecutor) would work.
@stray kite Oh I see, I'll have to read the context later and see if I can find an answer then. I just glanced over it (mostly the example) and that was the first thing that came to mind. (:
Wouldn't you normally want an exception to occur if the constructor wasn't passed all of the required arguments? It was designed that way intentionally (IIRC).
So it's specifically the __del__ exception you're trying to address?
Ah, I see. I suppose I've ever had a need to specifically address the __del__ exception. Delving too far into that would probably be off-topic for this channel though, so I'll leave it at that.
Oh okay, i'll look it over when I get the chance. I'm currently working on the "3.8 What's New" document. There were some significant changes to asyncio that didn't make it in that I'm adding entries for.
formal snow
I know the following has not really much purpose with python and async or any other form of concurrencies, but i think here is the best place to ask about tho;
I understood most things of a quantum computers like a bit can have 3 states instead of 2, including one that goes with quantum property of superposition where it's neither 1 nor 0 / both 0 and 1.
Some people say that calculations with qbits can happen in parallel. I don't get that. Can someone explain me why we can talk about parallelism with quantum computers?
and what - sort of - parallelism it is
ocean root
Quantum computers operate with so called "qubits" which can exist in a superposition of 0 and 1 (they are coefficients for the probabilities for the bit collapsing into 0 or 1) this state can be represented with a sphere, and as each qubit requires the coefficients the amount of classical bits in n qubits is 2^n
you can actually think of that fairly intuitively by for example rolling three dies without looking at the result; before you look at them to "collapse" the result they have a certain probability for their state
this exact property of probabilistic output is the commonly referred to parallelism in quantum computers, it is this property that allows us to create probabilistic algorithms for hard problems for classical computing (e.g.) factoring a number
formal snow
@ocean root thanks a lot for helping me out understanding this designation here. So it has really nothing to do with concurrencies as I could expect, and even less to do with python of course. As this'd be totally off topic to this place, would you mind if I ping you on an off topic channel because I still have some questions going in my head. (I already watched a lot of videos / readed a lot about the subject but honestly they seem all to say different stuff and that confuses me a lot, thats why i'm asking for human help)
That'd be very nice from you ofc and i'd be very thankful, but if you got no time/desire i'd totally get it :3
autumn haven
Yeah definitely a different topic from this channel since it's not related to asynchronous programming. Quantum computing is certainly an interesting field though. I've not looked into it enough to provide much more of an explanation than @ocean root already did.
minor plinth
Is it a wise idea if for example I'm making a reminder system for a bot if I grab every minute the reminders that are about to expire within that minute and create a task for each of them and asyncio.sleep() in them until they're done? Example:
1) user creates reminder
2) reminder gets added to database
3) every minute the script grabs all reminders that will expire within a minute from the database
4) the script create_task() and awaits inside until that minute is done and then deal with sending the reminder,e tc
now that i think about it if the timer is < 1 minute its better to wait for it directly since it would have th chance to be skipped
Hmm I see, I'll take a look at that thanks
I see, what I was thinking was something similar but using discord.py dispatch with a custom event (should work the same way)
minor plinth
So uhh, @stray kite when I try to use a method I get the error:
reply = await ui.get_reply()
AttributeError: 'coroutine' object has no attribute 'get_reply'
This is the async __new__ class
that you helped me on earlier
nvm, got it to work, but I have an new problem
the passive asyncio task doesn't work
I do task = asyncio.create_task(self.wait_cancel_task()) But it doesn't trigger
sweet vine
Until you await it, it will not trigger
Because the program does not know when to yield control to it
autumn haven
@stray kite
What do u think about a method, what returns contextmanager, what allows u to run syncronized code between an async thread and a sync thread ?
Hmm, that sounds interesting, I'm curious to see an implementation. You could probably do something like this usingloop.run_in_executor()or the lower levelexecutor.submit()(withexecutorbeing an instance ofconcurrent.futures.ThreadPoolExecutor).
So when the contextmanager is entered, the sync thread waits for the async to pause.
Unless I'm misunderstanding what you mean by "waits for the async to pause", this might not be reasonably possible within sychronous code. With that ability, it seems like it would be effectively asynchronous (regardless of how it's labeled).
Also, I'm not certain that I understand what you're referring to when say "a sync thread" or "an async thread". Within Python, the threads themselves (instances of threading.Thread) can't be asynchronous, as they have a strict flow without multiple points of entry and exit. It's the ThreadPoolExecutor that can asynchronously schedule calls using a group of threads, but it's not the threads themselves that are asynchronous. I think you're referring to something else other than threads. An example might help.
@minor plinth Are you certain that you actually want/need to use a task (specifically an instance of asyncio.Task, not the abstract term) for what you're trying to acomplish? If you want it to instantly trigger upon creation, a task doesn't seem like the right tool for the job. Can you post the code implementation for the coroutine wait_cancel_task()?
minor plinth
Sorry, not at my computer right now, it's essentially a discord wait_for that waits for the user to cancel an interaction. I want the option to cancel to be always present, but wait_for blocks the command until timeout or they respond, so in order to make it run passively, I wanted to use a task
@autumn haven