🔒 How to differentiate a http-request?

[lusty raptor]

If I send a http-request to a specific domain (shown in the picture), which actually exists on the internet (The requested page does not exist), then I don't want to get 200 as an response. How can I filter those sites with the actual sites? 🤔

[echo monolith]

@lusty raptor

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[austere swallow]

Are you sure it gives 200? Such pages almost always return 404, even though user is given nice html - because any error page can still be renderable html

[lusty raptor]

Yes, it returns 200 indeed

[austere swallow]

Please show code and url to test

[lusty raptor]

URL: https://tweedback.de/thisisjustatest

Code:

def checks_url(url):
    response = requests.head(url, allow_redirects=True)
    print(response.status_code)

    if response.status_code == 200:
        print(f"Domain exists: {url}")
    else:
        print(f"Domain does not exist: {url}")


if __name__ == "__main__":
    url = "https://tweedback.de/thisisjustatest"
    checks_url(url)

[austere swallow]

Robots.txt of that website doesn't allow any automated scraping for anyone, so they probably don't care for handling errors correctly

[lusty raptor]

How to find that Robots.txt file? 🤔

[zinc saffron]

https://tweedback.de/robots.txt

[austere swallow]

Anyways, because their robots.txt doesn't allow automatic access for anyone (user-agent: *) to anything other than index.html and assets, I don't think we can help you

[austere swallow]

Uh, robots.txt file is by standard directly in the root

[lusty raptor]

What does user-agent: * even mean? 😕

[zinc saffron]

* Means anything and everything.

[austere swallow]

https://en.m.wikipedia.org/wiki/Robots.txt

[lusty raptor]

And what does Disallow: /secret/ mean?

[austere swallow]

Open the whole wiki page, not just the image

[lusty raptor]

But is it technically possible to search for these sites /secret/? 🤔

[zinc saffron]

@lusty raptor Looking at your past chat-history it looks like you are just trying this on different websites that have it against scraping/crawling in their TOS/robots.txt.

If you want help concerning this project of yours please find a suitable website that is designed and explicitly allows for scraping/crawling. This allows us to help you as what you would be doing wont be against our servers TOS.

[lusty raptor]

I definitely see the underlying concerns about this case. However, it is not in my intention to breach the ToS. I'm just asking from an educational perspective, whether this is possible or not. What do encounter as a website owner and so on?

[zinc saffron]

Its good thats its not your intention to break a websites TOS. However Breaking a sites TOS if you meant it or not is besides the point.
Here are some sites i found that may be of use to you. These are made to be scraped.

https://books.toscrape.com/
https://quotes.toscrape.com/
https://crawler-test.com/
https://the-internet.herokuapp.com/
https://realpython.github.io/fake-jobs/
https://webscraper.io/test-sites

[lusty raptor]

Thanks, but is it technically possible to scrape these websites? I'm curious, you know? 🤔

[zinc saffron]

Yes its possible? That's what they were made to do... To be scraped

[lusty raptor]

No, I talk about the sites, which are not made to be scraped. Is it possible to handle it differently, so that you can scrape them? Again: due to curiosity

[zinc saffron]

Every-site can be scarped in the same way. The difference between the sites i listed and the ones you have tried is their TOS (If they want to be scraped).

[lusty raptor]

Is the robots.txt supposed to be read by humans or bots? Is it a information for humans or a system that prevents bots, spiders to crawl on their pages

[lapis solstice]

sites that don't want for you to scrape them will actively block you from scraping them, with a series of measures from obfuscation to captchas to banning your IP address
We will not assist you to scrape such sites

[lapis solstice]

Both

it serves both as a reference for humans thinking about scraping it, but most importantly is well supported by many web crawlers

[zinc saffron]

@lusty raptor Please read this wiki page in its entirety.

[echo monolith]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.