#🔒 Can someone explain how the robots.txt file works and what is means

9 messages · Page 1 of 1 (latest)

radiant bridge
#

Can someone explain how the robots.txt file works and what is means, like what each key means

crude oakBOT
#

@radiant bridge

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
toxic canopy
# radiant bridge Can someone explain how the robots.txt file works and what is means, like what e...

The Wikipedia page goes into quite a lot of detail: https://en.wikipedia.org/wiki/Robots.txt#Examples

Robots.txt

robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pag...

#

I'd read that then narrow down the question.

Also, this question has nothing to do with Python, so it's not on topic here. A web development server would be a better fit.

radiant bridge
#

oh yeah forgot, sorry

#

thanks though

#

!close

crude oakBOT
#
Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.