🔒 Help solving pylint `too-many-branches`

[dusky pagoda]

    def crawl(self, url: str) -> None:
        """
        Crawls a given URL, extracts links, and adds them to the crawl results.

        Args:
            url (str): The URL to crawl.
        """
        if not is_valid_url(url):
            logger.debug("Invalid url to crawl: %s", url)
            return

        if url in self.crawl_result:
            logger.debug("URL already crawled: %s", url)
            return

        if self.respect_robots_txt:
            user_agent = requests.utils.default_user_agent()
            robots_url = get_robots_txt_url(url)

            # Use previously created RobotFileParser
            if robots_url in self.robots:
                robot_parser = self.robots[robots_url]

            else:
                robot_parser = setup_robots_txt_parser(robots_url)

                self.robots[robots_url] = robot_parser

            if not is_robots_txt_allowed(url, robot_parser):
                logger.debug("Skipped: Url doesn't allow crawling: %s", url)

                return

            crawl_delay = float(robot_parser.crawl_delay(user_agent))

            if crawl_delay is not None:
                time.sleep(crawl_delay)

This piece of code is causing pylint to raise too-many-branches
What can I do to fix it?

[languid kestrel]

@dusky pagoda

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[wraith parrot]

don't use so many if statements

use more functions

maybe something like this ```py
def attempt_crawl(self, url: str) -> None:
if not self._is_url_crawlable(url):
return

if self.respect_robots_txt and not self._is_allowed_by_robots(url):
    return

self.crawl(url)

def crawl(self, url: str) -> None:
# actually parse the website

or you could name it crawl and _perform_crawl or something if you really wanted crawl() to always check the robots.txt

or def crawl and def parse_site or whatever

the point is to separate the logic out into separate methods, here I've described four methods:

1. attempt_crawl checks if the url should be crawled then calls crawl
2. crawl actually parses the site
3. _is_url_crawlable checks if the url is valid and hasn't already been crawled
4. _is_allowed_by_robots reads the robots.txt and checks if the url is allowed to be accessed

method (2) should probably be broken up into many other methods as well (if not another class altogether), but you haven't shared that

[spring kestrel]

Do you have to use pylint?

wait, this is a PR to somewhere, right

Is this the whole method?

[languid kestrel]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.