🔒 Web scraping (book)

[celest marlin]

Apparently this is supposed to open 5 new tabs, but it didn't open anything for me...?

[cursive idol]

@celest marlin

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[celest marlin]

What did I do wrong here?

oh wait,

maybe it's because

the cd

is in a different file...?

hmmmm...

It's in the same folder.

I guess it's not because of the cd?

Now, I only tried this in the command prompt...

I'll try it in VSC.

Hmmmm...

It didn't work.

The book said something about...

"Note that if the PyPI website changes its layout, you may need to update this program with a new CSS selector string to pass to soup.select(). The rest of the program will still be up to date."

Is it because of that, I wonder?

Ugh...

I tried with another site's link, and it didn't open new tabs either...

Can someone help, please?

#! python3
# searchpypi.py  - Opens several search results.

import requests, sys, webbrowser, bs4
print('Searching...')    # display text while downloading the search result page
res = requests.get('https://google.com/search?q=' 'https://pypi.org/search/?q='
+ ' '.join(sys.argv[1:]))
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')
linkElems = soup.select('.package-snippet')

numOpen = min(5, len(linkElems))
for i in range(numOpen):
    urlToOpen = 'https://pypi.org' + linkElems[i].get('href')
    print('Opening', urlToOpen)
    print(webbrowser.open(urlToOpen))

# TODO: Retrieve top search result links.

# TODO: Open a browser tab for each result.```

Here is the code...

[celest marlin]

This is the only thing I can think of why it doesn't work...

Hmmmm...

"After doing a search for Beautiful Soup, you can open the browser’s developer tools and inspect some of the link elements on the page. They can look complicated, something like pages of this: <a class="package-snippet" href="/project/pyautogui/">."

...There isn't a package-snippet class in the a?

Weird...

Web scraping (book)

Well, ping me if you're going to help.

[stiff maple]

The loop didn't run. See how it didn't print anything?
You have numOpen set to be minimum of 5 and length of linkElems
So this means linkElems is empty. You can try printing variables after each step of the code, since you're making the code in REPL anyways

[celest marlin]

huh?

Yeah it didn't print anything.

linkElems is empty...? What does that mean?

[stiff maple]

And that's the reason it's empty, no class meaning it didn't select anything

[celest marlin]

Oh.

Ohhhhhhh.

So I didn't do something wrong, then...

[stiff maple]

Your variable linkElems is an empty list. Because no element has that class

[celest marlin]

But it was because

YESSSSSSSSSS

[celest marlin]

this right?

[stiff maple]

Yeah

[celest marlin]

So I think it USED to have the package-snippet class

But now it doesn't?

Like maybe it got changed a few years ago or something?

[stiff maple]

Yeah. That tutorial thing seems to be old. Google changes stuff all the time

[celest marlin]

I see, I see.

Damn, I hit the nail on the spot.

[stiff maple]

I think Automate The Boring Stuff has some info on basic scraping

[celest marlin]

Yeah.

Until this,

all of the stuff worked.

Like elements, id's, etc, I could scrape them.

[stiff maple]

Ah, so this is Automate the Boring Stuff? I have never went deep into this book tbh, I just know the table of contents to suggest to people XD

[celest marlin]

Hahahah.

Yep!

It's literally that book.

Alright, thanks for the help!

[stiff maple]

The background got me confused, but it seems you just have some setting to force dark mode.
Also got confused because you had it in repl and not a file, as this is quite late in the book

[celest marlin]

Ahhh, yeah, I have a dark mode extension, lol.

as for REPL,

Well,

I tried it with VSC as well.

It's in a file,

"searchpypi.py"

[stiff maple]

So the repl was for testing, ok.
But when testing in repl, remember what makes repl different than the file - you can always check the variables and experiment interactively. 😉 So as I said, you could've printed the variables at each step and see if they look like they should.

Tbh I often test in repl as well, instead of using debugger. Because sometimes I just need to experiment before writing something, so then repl is nice to see if I'm doing something right... So when later some edge case bug happens, I also open repl to play like that (instead of using debugger to inspect the values at each step).

[celest marlin]

Yes, the repl was just for testing.

And well, I did experiment with it a little bit.

Instead of writing the whole thing, I wrote one element, and it still gave the same result, for example.

Alright then, I'm closing the post. Thanks again!

!close

[cursive idol]

Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.