🔒 How to scrape this specifically?

[worldly sequoia]

@oak willow

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[oak willow]

from bs4 import BeautifulSoup
import requests

gpuPerfPage = requests.get("https://www.tomshardware.com/reviews/gpu-hierarchy,4388-2.html")
soup1 = BeautifulSoup(gpuPerfPage.text, "html.parser")

graphicCards = soup1.findAll("tr", attrs={"class":"table__body__row"}) 

gpuList = []

for gpu in graphicCards[0:86]:
    tempList = []
    gpuMain = gpu.findAll("td", attrs={"table_body__data"})
    gpuName = gpuMain[0].text
    ultra1080p = gpuMain[2].text

    firstBrack = ultra1080p.index("(")
    lastF = ultra1080p.index("f")

    fps = ultra1080p[firstBrack+1:lastF]

    tempList.append(gpuName)
    tempList.append(fps)

    gpuList.append(tempList)

for gpu in gpuList:

    gpuName = gpu[0].split()
    gpuPerf = gpu[1]

    priceLink = "https://averagefinder.com/averageFinder?Search="

    for word in gpuName:
        priceLink = priceLink + word + "%20"
    
    gpuPricePage = requests.get(priceLink)
    soup2 = BeautifulSoup(gpuPricePage.text, "html.parser")
    
    gpuPrice = soup2.find_all("h2", attrs={"class":"text-center"}) 

    print(gpuPrice[3].text)
    print(gpuName)

I was currentlu just manually picking the 4th one but its not always the 4th on that page so it doesn't work all the time

[velvet snow]

do you have permission from tom's hardware to scrape them?

[mellow canopy]

the robots.txt seems to allow it outside of some specific sections (/reviews seems fine?)

[velvet snow]

https://futureplc.com/terms-and-conditions-us/#links_to_and_on_our_services the TOS doesn't seem too happy with it (ground rules paragraph k)

[mellow canopy]

[oak willow]

is it not fine if its for personal use?

[velvet snow]

the tos doesn't mention anything about it being ok for personal use

[oak willow]

mm ok mb shouldve seen it before

[worldly sequoia]

Python help channel closed using Discord native close action

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.