#๐Ÿ”’ Attempt at using asyncio and aiohttp to make fast request

112 messages ยท Page 1 of 1 (latest)

barren summit
#

My issue is that it continues to be slow. I removed the links and cookies and some other stuff for privacy reasons but I want to know if the structure is the issue.

import asyncio, aiohttp, time, os

Results = None
Path  = None
validPath = False
# Request Results
Valid   = 0
Invalid = 0
# Request Variables
URL     = ""
COOKIES = {
    
}

# Visuals
async def Reset():
    os.system('cls' )

async def Error(ErrorMessage:str):
    print(f"[-] {ErrorMessage}")

async def Path2List(Path:str):
    global validPath, Results
    try:
        with open(Path, 'r') as file:
            result = [line.strip() for line in file if line.strip()]
        validPath = True
        Results   = result
        Path      = Path

    except FileNotFoundError as error:
        await Error(f"No such file or directory: '{Path}'")
        time.sleep(5)

async def IndexResult(result:str):
    with open("Test.txt", 'a') as file:
        file.write(f"{result}\n")

async def UpdateCurrent():
    Text = f"[+] {Valid} | [-] {Invalid}"
    print(f"{Text:^126}", end="\r")

async def Request(Session, result:str):
    global Valid, Invalid
    async with Session.post(URL, data={"value": result}, cookies=COOKIES) as Request:
        ResponseText = await Request.text()
    
    if ResponseText == "true":
        await IndexResult(result)
        Valid += 1
    elif ResponseText:
        Invalid += 1

    await UpdateCurrent()

async def main():
    while not validPath:
        await Reset()
        print(f"{"Path":^91}")
        Path = input(f"   >>> ")
        await Path2List(Path)

    await Reset()
    await UpdateCurrent()
    async with aiohttp.ClientSession() as Session:
        for result in Results:
            if len(result) >= 4:
                await Request(Session, result)

    print(Results)


if __name__ == "__main__":
    asyncio.run(main())```

This is my first attempt at switching from synchronous code and using requests to asyncio and aiohttp. It maintains a speed fairly similar to my original code being non-async.
round nymphBOT
#

@barren summit

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

patent seal
#

You're still doing things in series, with no concurrency. It will take the same time. Results is a list of things to do. This code:

        for result in Results:
            if len(result) >= 4:
                await Request(Session, result)

performs each request in series.

#

You may want to dispatch an async task for each Request(Session,result) call so that they can run concurrently.

prisma orchid
barren summit
barren summit
patent seal
#

Yes. That way you're not waiting for each to complete before dispatching the next one.

#

Then you can wait for all the tasks to complete (in whatever order they complete).

barren summit
# patent seal Yes. That way you're not waiting for each to complete before dispatching the nex...

Gave me this error

Task exception was never retrieved
future: <Task finished name='Task-8811' coro=<Request() done, defined at c:\Users\Teracotta Pie\Documents\Code\test.py:95> exception=RuntimeError('Session is closed')>
Traceback (most recent call last):
  File "c:\Users\Teracotta Pie\Documents\Code\test.py", line 97, in Request
    async with Session.post(URL, data={"value": word}, cookies=COOKIES) as Request:
               ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Teracotta Pie\AppData\Local\Programs\Python\Python313\Lib\site-packages\aiohttp\client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
                           ^^^^^^^^^^^^^^^^
  File "C:\Users\Teracotta Pie\AppData\Local\Programs\Python\Python313\Lib\site-packages\aiohttp\client.py", line 512, in _request
    raise RuntimeError("Session is closed")
RuntimeError: Session is closed```
prisma orchid
#

that should solve the core problem here, switching to async file io might be a good thing but probably wouldn't make that much of a difference here

patent seal
#

You might need to collect all the tasks before leaving the with ... as Session: clause. Otherwise you may be going:

async with ... as Session:
    for loop to dispatch tasks ...
... here the Session is closed, _before_ the tasks have completed ...
barren summit
#

Would I create the tasks and stick them in a list

#

then

#

make a loop that goes through them all?

naive carbon
#

aiometer is good for doing things concurrently

#

Also your file IO is blocking, I'd recommend anyio.Path for that

patent seal
#

I don't know how convenient aiohttp or httpx are. I'm still using requests. Sometimes in an async program, eg:

async def get_urls(urls : List[str]):
    """ Fetch `urls` in parallel.
        Yield `(url,response)` 2-tuples.
    """
    async for i, response in amap(
        requests.get, urls,
        concurrent=True, unordered=True, indexed=True,
    ):
        yield urls[i], response

which just does a bunch of requests.get calls in parallel.

patent seal
barren summit
barren summit
patent seal
#

Sure. But look at all that ... lather of async/await stuff ๐Ÿ™‚

barren summit
patent seal
naive carbon
#

Ic

barren summit
patent seal
#

I just wanted to not care so much about whether I was using sync or async stuff, so I wrote some shims.

naive carbon
patent seal
patent seal
barren summit
naive carbon
barren summit
#

@patent seal Changed my code to this py async with aiohttp.ClientSession() as Session: async with asyncio.TaskGroup() as Group: for word in Words: if len(word) >= 4: Group.create_task(Request(Session, word))

#

It kinda works?

#

It's showing the request are updating faster via my UpdateCurrent function

#

like insanely fast

#

the issue is the positive results are significantly less

naive carbon
#

So you're starting all your tasks off at once, and your file IO is not concurrent

#

You can't just stick async on a function to make it concurrent

#

You need to have an await that yields for performing IO

barren summit
#

I was getting about 700 positives with my un-async code now it's giving me 35

barren summit
naive carbon
#

You see you have calls to with open( ?

#

It's blocking file IO

#

Nothing else can run while you're writing the file

barren summit
#

ohhh

naive carbon
#

Use anyio.Path instead

barren summit
#

I see now that makes sense

naive carbon
#

And don't use time.sleep

#

And your async def Error should just be def Error(

#

You shouldn't make a function async if it didn't await

barren summit
barren summit
barren summit
#

Should I just be doing py Path = "Test.txt" await Path.write_bytes(b'hello, world')

naive carbon
naive carbon
#

You want await anyio.Path("Test.txt").write_bytes(b"hello world")

barren summit
#

then just do

#

wait idk

naive carbon
#

You don't need to put it in a variable you can pass it directly

barren summit
#

will it make a new line by default?

naive carbon
#

No

barren summit
#

also does it automatically append or is it going to rewrite it

naive carbon
#

Oh it's doing an append anyways

#

So you'll need a lock

#

Let me get to a computer

barren summit
#

Alright thanks

patent seal
naive carbon
patent seal
barren summit
#

would this just be easier to do if

#

I stored the results in a list

#

then at the end

patent seal
#

Or of course, keep the file open during the run and stream results to something which write in series eg via an asyncio.Queue.

barren summit
#

added all the results into the file

patent seal
barren summit
#

the only reason I didn't do it that way is incase of program crash or something

#

the current results are still saved

naive carbon
#

amap would yield the results so you could save them as you go

barren summit
#

I also wanted to eventually make a db system that stores all the tested items then stored the invalid and valid ones so it knows not to test them again

patent seal
barren summit
#

btw

#

the only text being written

#

is a word

#

it's not a large sum of text or

#

anything

#

the api is responding with words

patent seal
barren summit
#

I'm familiar with it

patent seal
#

Go right ahead. An sqlite db has the convenience of just being a local file for the db, not needing a server.

barren summit
#

I also rent a couple dedis that I use as rdps/servers so it's fine

barren summit
naive carbon
#

Oh it looked like you were switching to sqlite

barren summit
barren summit
#

@naive carbon what should I do

#

just the anyio.Path thing

naive carbon
#

I'd personally use aiometer and anyio.Path

#

Then you open the file once where you iterate the results

#

If you have big results you might instead want to stream them to disk in separate files and then stitch them together at the end

barren summit
#

With anyio.Path in this case

#

do I need to put b infront of the string

#

or can I make it an fstring

patent seal
#

If you open the file in text mode, use strings. In binary mode, bytes.

#

An f-string is just an expression which produces a string.

round nymphBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.