🔒 Python AsyncIO hangs at 100% CPU usage. Stack trace shows some clue but don't know how to interpre

[real coral]

Hi,
our application is suddenly hanging after 10-20 minutes of use, which I'm trying identify the root cause for. Stack trace shows the application is stuck at some CPython call. Is there anyone familiar with Cpython internal that can explain me what this part of the code in Cpython do?

https://github.com/python/cpython/blob/v3.12.3/Modules/_asynciomodule.c#L3594 , the loop in line 3594.

[crisp rock]

@real coral

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

Modules/_asynciomodule.c line 3594

while (next != NULL) {```

[craggy stump]

Showing your code would be more useful.

usually 100% cpu usage means an indefinite/infinite loop somewhere with no sleep

[covert crow]

How many tasks and how many futures do you have instantiated?

This is the sort of thing that will happen if you have a hard loop around asyncio.create_task I think

[real coral]

I can't show code because it's a big repository and it's propieterary code. I don't know where the problematic code is, and I can't even reproduce the isuse on my computer. All the info I have is that stack trace 😦

[craggy stump]

It's probably as graingert said

[real coral]

I don't know either. (It's multiprocess and multi threaded) But I don't think it's because of a hard loop somewhere because:

• It was running fine before. We recently migrate from python 3.7 with a bunch of dependeny update.
• If it were stuck somewhere in our code, the stack trace would have shown, wouldn't it? But as it stands every time I rerun it's stuck at different part of our code base, and eventually the last call is down to that C function in CPython.

[covert crow]

Or I'm thinking you created a whole bunch of tasks then the GC runs

[real coral]

I'm more incined to think this is some kind of bug in python 3.12. Because it looks like it's trying to traverse some linked list, and if the linked list is circular then it would never end.

[covert crow]

Try using PyTask and PyFuture

[real coral]

thanks I will look into it

at this point I'm desperate for any hint

[covert crow]

        asyncio.Task = asyncio.tasks._CTask = asyncio.tasks.Task = \
            asyncio.tasks._PyTask
        asyncio.Future = asyncio.futures._CFuture = asyncio.futures.Future = \
            asyncio.futures._PyFuture

Before using any asyncio code

[real coral]

what does that do ?

I thought you were mentioning some library

ok

[covert crow]

It downgrades Task and Future to a pure python version, should avoid the code path that's failing

I think you can do sys.modules["_asyncio"] = None before importing asyncio

And completely prevent that code from even importing

[real coral]

I don't understand the difference between the 2 but I will try it anyway to see if it fixes the isuse. But the whole cycle of redployment and test would take take 1 or 2 hours. that's why debugging this sucks.

[covert crow]

it works because _asyncio is optional https://github.com/python/cpython/blob/3.12/Lib/asyncio/tasks.py#L405

[crisp rock]

Lib/asyncio/tasks.py line 405

except ImportError:```

[covert crow]

Well remember to ping the channel every hour to keep it open

[real coral]

@covert crow does it make a difference because we actually use Cython to compile the code first and then run it ?

[covert crow]

Depends are you directly calling CTask and CFuture?

[real coral]

no

[covert crow]

Probably not

[real coral]

we dont use any cython feature. just use it for a bit of obfuscation

[covert crow]

Lol

[real coral]

LOL i know xD

[real coral]

adding that doesn't work. Tornado, the library we are using, added some assertion around the task and future types. Can I just do sys.modules['_asyncio'] = None ? Cause that seems to work.

[covert crow]

Oh silly tornado, well make sure to do the sys.modules patch before importing tornado which imports asyncio

Out of interest what assertion is that?

[real coral]

it gives exception at :
Future = asyncio.Future

FUTURES = (futures.Future, Future)

def is_future(x: Any) -> bool:
return isinstance(x, FUTURES)

so is_future return false

[covert crow]

Ah you need to apply the patch before importing tornado or anything else that uses Future or Task

[real coral]

ah you are right silly me. ok I will test it now. finger crossed

[real coral]

@covert crow so apparently there are 2 implementations of asyncio module, one in C and one in Python. Why do we have it like this and isn't the Python implementation slower? How do i read more about it because there is no docs on this.

[covert crow]

Yep well sort of

[real coral]

what's the side effect of downgrading to pure python one ?

[covert crow]

There's a c implementation of some of the often called bits

Oh it's like dead slow when pure python

[real coral]

so if it runs on the pure python version then this means there is a bug in Cpython right ?

[covert crow]

Yeah maybe

[real coral]

well it's still stuck at the same code path.

I added the patch as early as possible

[covert crow]

Try the sys.modules trick then?

[real coral]

tried both

[covert crow]

Oh you're not importing it early enough then

You can also try deleting the .so file

[real coral]

it's the first statement that runs actually. But does it make a different when the first process spawns then forks another process ? it's fork so I think it doesn't matter. I also tried print sys.modules[_asyncio] in the forked process and it indeeds returns none

[covert crow]

Try deleting the _asynciomodule .so file

You can find the file by importing the _asyncio module and printing it out

[real coral]

i deleted it

now testing it again

finger crossed

[crisp rock]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.