🔒 Python concurrent requests high CPU usage

[opal hinge]

@plush lark

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[plush lark]

Im using requests library to send cuncurrent get requests with concurrent.futures and threading library
the problem is that when i switch from urllib.request to requests library the cpu usage went crazy with 25 threads i got 90% and i got 4.8Ghz cpu so its weird
i used to use even 400 threads with urllib.request without problem

here is the code:

def b_search(query, tld='com', lang='', num=1, start=0, stop=None, pause=2.0, proxy=None, user_agent=None, verify_ssl=True, include_bing_links=False, form_code=''):
    
    page_offset = 50  
    base_url = f"{url}"

    for page in range(num):
        _page = page * page_offset + 1
        url = base_url.format(_page)  
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
            'Accept-Language': 'en-US,en;q=0.9',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'Connection': 'keep-alive'
        }

        time.sleep(pause)
        session = requests.Session()

        # Make the GET request
        req = session.get(url, headers=headers)
        # print("Status Code:", req.status_code)

        html = req.text
        soup = BeautifulSoup(html, 'html.parser')
        anchors = soup.find_all('a')
        
        page_links = 0
        for a in anchors:
            try:
                link = a['href']
            except KeyError:
                continue

            link = filter_result(link)
            if not link:
                continue
            
            page_links += 1
            yield link

        if page_links == 0:  # No more links found
            print("No more links found. Ending the loop.")
            break```

rest of the code:
https://paste.ofcode.org/RGibvrBKj3UkYsCEiKnM32

[opal hinge]

@plush lark

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.