#🔒 please help with traceback error

95 messages · Page 1 of 1 (latest)

hexed torrent
ivory coveBOT
#

@hexed torrent

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
ivory coveBOT
opaque fulcrum
#

2067 is not the usualy comic. Go to the page, have a look.

hexed torrent
opaque fulcrum
#

Anyway, if you script aborts at 2067, there is likely a traceback.

#

!traceback

ivory coveBOT
#
Traceback

Please provide the full traceback for your exception in order to help us identify your issue.
While the last line of the error message tells us what kind of error you got,
the full traceback will tell us which line, and other critical information to solve your problem.
Please avoid screenshots so we can copy and paste parts of the message.

A full traceback could look like:

Traceback (most recent call last):
  File "my_file.py", line 5, in <module>
    add_three("6")
  File "my_file.py", line 2, in add_three
    a = num + 3
        ~~~~^~~
TypeError: can only concatenate str (not "int") to str

If the traceback is long, use our pastebin.

hexed torrent
#

please help with request error

#

yep @opaque fulcrum

# 
Traceback (most recent call last):
  File "/Users/m/Documents/Scripts/wscrape.py", line 20, in <module>
    res = requests.get(comUrl)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
    ~~~~~~~~~^
        method=request.method.upper(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<10 lines>...
        hooks=merge_hooks(request.hooks, self.hooks),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/models.py", line 367, in prepare
    self.prepare_url(url, params)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/models.py", line 444, in prepare_url
    raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
requests.exceptions.InvalidURL: Invalid URL 'https:/2067/asset/challengers_header.png':```
#

please help with traceback error

subtle fiber
#

It was because it use relative path and omit the domain, which you have to supply I believe

severe saffron
#

That page uses a different src link type for some reason.

image.png
subtle fiber
#

Oh also it wasn't just an image

#

Maybe that's why

#

.xhcd 2066

#

Wait I pretty sure it's a command for it

severe saffron
#

The CSS should still select the img tag fine. I think it's just that 2067 uses a link relative to the current domain, and other pages use links to a different subdomain.

hexed torrent
#

I see, thanks. It works otherwise just intresting that I got the error

subtle fiber
#

.xkcd 2067

hexed torrent
#

So I would need to implement code to account for the error

severe saffron
opaque fulcrum
subtle fiber
#

#sir-lancebot-playground message
Yeah this is just entirely different stuff to handle

#

As well

hexed torrent
#

And what if the error occurs elsewhere on other pages

subtle fiber
opaque fulcrum
#

If I'm at web page https://example.com/foo/bah.html and it has a image whose src is /foo.png, you're supposed to get the https://example.com/ from the page's URL. Making https://example.com/foo.png

hexed torrent
opaque fulcrum
severe saffron
#

That's honestly one reason why I don't think scraping is a good beginner project. You also need to know basic HTML and CSS, which complicates matters for new programmers.

opaque fulcrum
hexed torrent
hexed torrent
opaque fulcrum
#

Just means they all followed some pattern, and 2067 did not. Now you need to see why that page is special.

severe saffron
hexed torrent
#

Thanks

opaque fulcrum
severe saffron
#

I'm glad I have the Dev Tools. Reading HTML from the terminal is painful.

opaque fulcrum
ivory coveBOT
hexed torrent
#

I noticed that 2067 used canvas instead of img scr

opaque fulcrum
#

Thus the indented listing above. See what you think.

severe saffron
opaque fulcrum
#

Wot's pug.

severe saffron
#

A templating language that lets you write HTML using indents like that, and without <>.

hexed torrent
#

I haven't looked through all the pages but it looks like 2067 is the only page to use canvas

severe saffron
#

I think the canvas is a red herring.

opaque fulcrum
hexed torrent
hexed torrent
ruby forge
#

isn't this illegal (here) or something? you seem to be sending a lot of requests to a website

hexed torrent
#

It uses 2 images thats why - one at the top being img src and the larger being canvas

severe saffron
#

Although, as well, the images on this page aren't the main focus.

hexed torrent
opaque fulcrum
ruby forge
severe saffron
#

This is also a project in Automate the Boring Stuff iirc. With as common as that book is, I would expect them to have a TOS discouraging scraping if it bothered them.

opaque fulcrum
#

Anyway, this is a site specific terms of service thing. I consider scraping to be an accessibility thing anyway. The scraper's an aid to me reading their site.

severe saffron
#

And this is their robots.txt:

User-agent: *
Disallow: /personal/

User-agent: GPTBot
Disallow: /
hexed torrent
severe saffron
#

Which is a whitelist with two exceptions.

hexed torrent
#

Is the main issue here? comUrl = 'https:' + comElem[0].get('src') in retrieving the src in the img div when the image is in canvas for 2067

severe saffron
#

The issue is that comElem[0] is not in the form that you're expecting. Like Cameron mentioned, put a print before that line and print it out in the problem case and a good case.

#

"When in doubt, print it out"

hexed torrent
severe saffron
#

The length may be interesting in some cases, but for now, the important part is comElem[0] specifically. I'd do

print(f'Comic img link: {comElem[0]}')
#

I will partially agree with Sun though that this is potentially hitting their server far more than it needs to. I wouldn't run the entire script until you've worked out the bugs. Hitting up thousands of pages every time you want to do a test is needlessly hitting their servers. It probably doesn't matter since a ton of caching will be in place, but I think it's a good habit to tend to respect other people's servers.

hexed torrent
#

Maybe I should have a limiter to run it up to a certain point then end as this is ongoing it can be much more than 5,000

severe saffron
#

Yes, and if you're interested in page 2067 specifically right now, you might as well just focus on that page instead of fetching 2000+ pages just to get a link that you already know.

hexed torrent
hexed torrent
#

right?

severe saffron
#

I don't think fetching the canvas would help you. The canvas would be drawn at runtime by JS. Since you're using BS4, JS doesn't run, so the canvas will neverbe populated. If you were running JS while scraping, you could export the image from the canvas to get it, but that's not an option with your current set up. The best you can probably do is fetch the two images that are available to you. This special case shows that there may be multiple images per page, so always getting only the 0th img may mean that you're missing other images.

hexed torrent
severe saffron
#

The image isn't nested in the canvas:

image.png
#

It's just after the canvas

#

I haven't used BS4 in forever, but there should be a selectAll or something function that fetches all matches instead of the first.

#

Wait, nvm, I'm getting tired. select already returns all of them, and you're doing [0] to get the first. Just check all the elements that get found.

hexed torrent
#

This defines the start point - url =

severe saffron
#

It appears so, yes.

#

You might find it useful to create a separate "playground" script that's a stripped down version of this one for when you need to examine single pages at a time.

hexed torrent
hexed torrent
#

!close

ivory coveBOT
#
Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.