#🔒 Scraping newspaper websites - advice needed

22 messages · Page 1 of 1 (latest)

winged cave
#

Hi all, python newbie here. I want to write code to scrape several newspapers for articles on a specific subject, output the results of the scraping on a .csv file and then analyse said .csv file. Figuring out the scraping part has been quite a headache, and as such I would like to get some direction from the Python Discord hivemind if that is possible. My initial approach was to use 'requests' to scrape a tag page HTML (e.g. National), and that worked, but then I realised I hit a dead end as this approach only took me so far (website would only display say latest 20 articles in page, with no pagination mechanism or 'show more' buttons; if I asked requests to return more pages of articles, it returned multiple copies of the same articles). My current thinking, based on this previous realisation and further research, is that I'm better off scraping search results pages, but requests does not seem to work there. How could I go on about tackling this problem of mine? What steps should I take? And how could I make sure that each scrape from each newspaper archive feeds into the same dataframe? All and any help is greatly appreciated.

long muskBOT
#

@winged cave

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
flat geode
#

!rule 5

long muskBOT
#
Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

flat geode
#

aka dont help with scraping if u cant show u are allowed but if u are allowed u should be haveing an api and useing that aint scraping

winged cave
#

Thanks for responding @flat geode and apologies for triggering rule 5.

#

I do not intend to scrape websites that do not allow it but understand I need to demonstrate this so I can ask this question publicly.

flat geode
#

can u give me a link to the site?

winged cave
#

I take it I would be triggering in this specific case this one --> Disallow: /search/*?

flat geode
winged cave
#

I know - I am just asking if this is an indication of their policy on scraping sorry

#

Appreciate you looking into this!

hardy abyss
#

according to the robots.txt /news/ is allowed

winged cave
#

Thanks 0x - this is the bit I am trying to scrape, and not even a massive amount, a couple of months more or less. I intend to be very much guided by what each website allows.

hardy abyss
winged cave
hardy abyss
#

yes, you could do further testing to see if they are consistent on updating the rss

long muskBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.