#🔒 help with quote web scraping

33 messages · Page 1 of 1 (latest)

cedar wing
#

here is my code

import tls_client
import bs4
import random

class EasyQuote:
    def __init__(self,page=None,keyword=None,amount=1):
        if page != None and keyword != None:
            self.url = f'https://www.goodreads.com/quotes/search?commit=Search&page={{}}&q={{}}&utf8=%E2%9C%93'.format(str(page), keyword)
        elif keyword != None and page == None:
            self.url = f'https://www.goodreads.com/quotes/search?commit=Search&page=1&q={{}}&utf8=%E2%9C%93'.format(keyword)
        elif keyword == None and page != None:
            self.url = f'https://www.goodreads.com/quotes?page={{}}'.format(str(page))
        elif page == None and keyword == None:
            self.url = f'https://www.goodreads.com/quotes?page={{}}'.format(str(random.randint(1,100)))
        else:
            return 'Missing required easyquote-init params'
        
        self.amount = amount
        
        self.headers = {
            'User-Agent':'Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1'
        }
        self.session = tls_client.Session(
            client_identifier="safari_15_6_1"
        )

    def generate(self):
        parsed=[

        ]
        html = self.session.get(self.url,headers=self.headers).text
        quotes = bs4.BeautifulSoup(html, 'html.parser').find_all('div',class_='quoteText')

        i=0
        for quote in quotes:
            q = quote.get_text(strip=True).split('―')[0].strip('“”')
            if ',' in quote.get_text(strip=True).split('―')[1]:
                a = quote.get_text(strip=True).split('―')[1].split(',')[0]
            else:
                a = quote.get_text(strip=True).split('―')[1]
            parsed.append({
                'quote':q,
                'author':a
            })
            i+=1
            if i == self.amount:
                break
        return parsed```

anyone know why EasyQuote().generate() returns an empty list instead of 1 quote from the random page
stone trailBOT
#

@cedar wing

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
next agate
cedar wing
next agate
#

They don't want you scraping their site and they're deprecating their API

cedar wing
#

it works fine with normal requests on my wifi

#

not home for a few weeks

#

i could use a dif site this is just the first i found nothing fraudluent went through my mind

junior vine
#

!rule 5 ?

stone trailBOT
#
Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

junior vine
#

dont hack

cedar wing
#

@junior vine is web scraping illegal?

junior vine
#

yes

cedar wing
#

!rule 5

stone trailBOT
#
Rules

5. Do not provide or request help on projects that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal.

cedar wing
#

I wasn’t aware

#

I was just doing a python course and an album was web scraping

#

Please mind that asking for help/assistance with tasks that may violate terms of service, or that may be deemed inappropriate, malicious, or illegal are not allowed under any circumstances.

leaden halo
#

Webscarping isnt illegal. However if the website you're scrapping has it in their tos that they dont allow it, then we cant help you.

cedar wing
#

which I’m scraping quotes

#

I assume that is public info

leaden halo
#

not exactly.
Subject to your compliance with this Agreement and your payment of any applicable fees, we grant you a limited, non-exclusive, non-transferable, non-sublicensable license to access and make personal and non-commercial use of the Service. This license does not include any resale or commercial use of any part of the Service, or its contents; any collection and use of any book listings, descriptions, reviews or other material included in the Service; any derivative use of any part of the Service or its contents; any downloading, copying, or other use of account information for the benefit of any third party; or any use of data mining, robots, or similar data gathering and extraction tools. All rights not expressly granted to you in this Agreement are reserved and retained by us or our licensors, suppliers, publishers, rightsholders, or other content providers.
https://www.goodreads.com/about/terms

cedar wing
next agate
#

If they're using cloudflare they don't want scrapers

rich copper
cedar wing
#

I’m only doing this to master web scraping

#

What other sites are good for practice

leaden halo
rich copper
# cedar wing What other sites are good for practice

I believe Wikipedia* is a fine target. You of course want to stay within the acceptable use policies but you'll get some HTML to work with (plus there's also the MediaWiki API which returns HTML as well)
-# *their terms seem to only prohibit disruptive automation which is understandable

leaden halo
stone trailBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.