🔒 img src error | Python | Page 1

wary shadow Sep 28, 2025, 1:23 AM

#

img src redirecting to a forbidden page

📎 scrape_test.py

elder prismBOT Sep 28, 2025, 1:23 AM

#

wary shadow img src redirecting to a forbidden page

#

@wary shadow

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

elder prismBOT Sep 28, 2025, 1:23 AM

#

wary shadow img src redirecting to a forbidden page

~~Please react with ✅ to upload your file(s) to our paste bin, which is more accessible for some users.~~

wary shadow Sep 28, 2025, 1:23 AM

#

And with ```py
from http.client import InvalidURL
from pyexpat import features
from colorama import Fore

import requests, os, bs4

url = 'https://xkcd.com/2067/'
os.makedirs('xkcd', exist_ok=True)
while not url.endswith('#') :
print(Fore.BLACK + f'\nDownloading page {url}...')

res = requests.get(url)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, "html.parser")
comElem = soup.select('#comic img')

if not comElem:
    print(Fore.RED + 'Could not find comic image.')
else:
    comUrl = 'https:' + comElem[0].get('src')
    print(f'\nDownloading image {comUrl}...')
    res.raise_for_status()

    imageF = open(os.path.join('xkcd', os.path.basename(comUrl)), 'wb')
    for chunk in res.iter_content(10000):
        imageF.write(chunk)
    imageF.close()
    print('Done nga')
    exit()```

#

it prints the text without the hyperlink

wary shadow Sep 28, 2025, 1:46 AM

#

How do I stop - res = requests.get(comUrl) being ran if the image is not valid

#

I want it to skip this image and continue to the next

topaz cloak Sep 28, 2025, 1:55 AM

#

wary shadow How do I stop - res = requests.get(comUrl) being ran if the image is not valid

how do you plan to even know if it's not valid until you have even tried to download it?

wary shadow Sep 28, 2025, 1:55 AM

#

topaz cloak how do you plan to even know if it's not valid until you have even tried to down...

because it raises the invalidurl error

#

as the image uses centre instead of img src

#

on that page

topaz cloak Sep 28, 2025, 2:03 AM

#

wary shadow because it raises the invalidurl error

it raises the exception after you have tried to get it and it fails
you can't know that it will fail before you tried unless you have some other indicator that it wouldn’t be valid

wary shadow Sep 28, 2025, 2:04 AM

#

topaz cloak it raises the exception after you have tried to get it and it fails you can't kn...

?

#

Have you ran the code

topaz cloak Sep 28, 2025, 2:07 AM

#

wary shadow Have you ran the code

no, but i don't even understand the problem and what you want to do

wary shadow Sep 28, 2025, 2:08 AM

#

This is an image scraper, it fails to run 2067

#

I want the script to skip over pages that dont have images

#

Traceback (most recent call last):
  File "/Users/m/Documents/Scripts/scrape test.py", line 21, in <module>
    res = requests.get(comUrl)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
    ~~~~~~~~~^
        method=request.method.upper(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<10 lines>...
        hooks=merge_hooks(request.hooks, self.hooks),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/models.py", line 367, in prepare
    self.prepare_url(url, params)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/requests/models.py", line 444, in prepare_url
    raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
requests.exceptions.InvalidURL: Invalid URL 'https:/2067/asset/challengers_header.png': No host supplied```

topaz cloak Sep 28, 2025, 2:09 AM

#

wary shadow I want the script to skip over pages that dont have images

oh, after you gotten the page and searched the html

#

got it

wary shadow Sep 28, 2025, 2:10 AM

#

yes

topaz cloak Sep 28, 2025, 2:10 AM

#

the url is invalid

wary shadow Sep 28, 2025, 2:11 AM

#

no - https://xkcd.com/2067/

topaz cloak Sep 28, 2025, 2:11 AM

#

https:/2067 is not a correct url

#

it's this code that is wrong

comUrl = 'https:' + comElem[0].get('src')

wary shadow Sep 28, 2025, 2:12 AM

#

topaz cloak it's this code that is wrong ```py comUrl = 'https:' + comElem[0].get('src') ```

yep

#

its not src

topaz cloak Sep 28, 2025, 2:12 AM

#

not only that, you are just trying to append https: to whatever url you find there

#

that's the real problem

wary shadow Sep 28, 2025, 2:13 AM

#

nope

#

the problem is with the js in some of the pages

#

I have already tried that

#

changed to http same thing, if u compare 2067 with 2066 you can see the difference if u inspect the image in the middle

#

just wondering how to continue if its not src

topaz cloak Sep 28, 2025, 2:16 AM

#

oh, the partial url you got on 2067 was for the image above the main content

wary shadow Sep 28, 2025, 2:17 AM

#

yes but the js interferes with it I guess the module bs4 doesnt handle that

topaz cloak Sep 28, 2025, 2:18 AM

#

this is the one you were trying to get but didn't really construct the correct url for https://xkcd.com/2067/asset/challengers_header.png

#

correct, bs4 can't handle js by itself

wary shadow Sep 28, 2025, 2:18 AM

#

I think my url is fine, if you can suggest a code insert I can try it but I think its the js man

topaz cloak Sep 28, 2025, 2:19 AM

#

people tend to use something like playwright or selenium for such things

wary shadow Sep 28, 2025, 2:20 AM

#

topaz cloak people tend to use something like playwright or selenium for such things

so how can i skip over the js pages thats all i need I plan on learning selenium later

topaz cloak Sep 28, 2025, 2:21 AM

#

look at the difference between the urls below

https://xkcd.com/2067/asset/challengers_header.png
https:/2067/asset/challengers_header.png
```the first one is the real url and the second one is the one you are trying to get, it's missing the `//xkcd.com` part after `https:`

wary shadow Sep 28, 2025, 2:23 AM

#

url = 'https://xkcd.com/2067/'

#

This is the page of the image

topaz cloak Sep 28, 2025, 2:23 AM

#

the problem with the part under that image that is the main content is that it's not just an image that is loaded by js, it's a whole dynamic

wary shadow Sep 28, 2025, 2:23 AM

#

yep

topaz cloak Sep 28, 2025, 2:23 AM

#

wary shadow url = 'https://xkcd.com/2067/'

yes, i know

wary shadow Sep 28, 2025, 2:23 AM

#

how can I skip over it

topaz cloak Sep 28, 2025, 2:24 AM

#

you can catch the exception and move on

wary shadow Sep 28, 2025, 2:24 AM

#

topaz cloak look at the difference between the urls below ```txt https://xkcd.com/2067/asset...

the scraper pulls them automatically, it has it like that by default

wary shadow Sep 28, 2025, 2:24 AM

#

topaz cloak you can catch the exception and move on

i have tried that I dont know where to nest it

topaz cloak Sep 28, 2025, 2:30 AM

#

wary shadow ```Downloading image https:/2067/asset/challengers_header.png... Traceback (most...

the code you posted isn't the same as the one that produced this error

#

or it's just incomplete

#

this line is missing from the code you posted

res = requests.get(comUrl)
```and is the one the throws that error form line 21 in your file named `/Users/m/Documents/Scripts/scrape test.py`

wary shadow Sep 28, 2025, 2:31 AM

#

here ```py
from pyexpat import features
from colorama import Fore

import requests, os, bs4

url = 'https://xkcd.com/2067/'
os.makedirs('xkcd', exist_ok=True)
while not url.endswith('#') :
print(Fore.BLACK + f'\nDownloading page {url}...')

res = requests.get(url)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, "html.parser")
comElem = soup.select('#comic img')

if not comElem:
    print(Fore.RED + 'Could not find comic image.')
else:
    comUrl = 'https:' + comElem[0].get('src')
    print(f'\nDownloading image {comUrl}...')
    res = requests.get(comUrl)
    res.raise_for_status()

    imageF = open(os.path.join('xkcd', os.path.basename(comUrl)), 'wb')
    for chunk in res.iter_content(10000):
        imageF.write(chunk)
    imageF.close()
    print('Done nga')
    exit()```

topaz cloak Sep 28, 2025, 2:32 AM

#

there it is, it's much easier if it's the right code that we are looking at

wary shadow Sep 28, 2025, 2:32 AM

#

yeah my fault

#

I might look into selenium

topaz cloak Sep 28, 2025, 2:35 AM

#

instead of

        res = requests.get(comUrl)
```you want
```py
        try:
            res = requests.get(comUrl)
        except requests.exceptions.InvalidURL:
            print("Skipping invalid URL: {comUrl}")
            continue
```or you can use `break` if you want to end the loop instead of `continue`

#

continue runs the next iteration of the loop

#

so it depends on what you want to do

wary shadow Sep 28, 2025, 2:39 AM

#

topaz cloak instead of ```py res = requests.get(comUrl) ```you want ```py tr...

this doesnt work

topaz cloak Sep 28, 2025, 2:40 AM

#

oh, sorry, copied the wrong line

wary shadow Sep 28, 2025, 2:41 AM

#

you meant - res = requests.get(comUrl)

topaz cloak Sep 28, 2025, 2:41 AM

#

there, yeah

wary shadow Sep 28, 2025, 2:41 AM

#

actually worked thanks a lot bro

topaz cloak Sep 28, 2025, 2:41 AM

#

the text moved up while i was copying it and i didn't check

wary shadow Sep 28, 2025, 2:42 AM

#

yh, it works but it prints out the print("skipping etc") on every line - i had this issue before with except

#

like it stays on 2067

topaz cloak Sep 28, 2025, 2:43 AM

#

as i said, the url you construct is not correct

wary shadow Sep 28, 2025, 2:44 AM

#

so I can't even skip it?

wary shadow Sep 28, 2025, 2:44 AM

#

topaz cloak as i said, the url you construct is not correct

I have to amend this - comUrl = 'https:' + comElem[0].get('src') ?

topaz cloak Sep 28, 2025, 2:44 AM

#

this line

        comUrl = 'https:' + comElem[0].get('src')
```needs to be
```py
        comUrl = 'https://xkcd.com' + comElem[0].get('src')

wary shadow Sep 28, 2025, 2:44 AM

#

I see

topaz cloak Sep 28, 2025, 2:45 AM

#

your script will download this image https://xkcd.com/2067/asset/challengers_header.png

wary shadow Sep 28, 2025, 2:45 AM

#

actually works

topaz cloak Sep 28, 2025, 2:45 AM

#

which is part of the comic for this page, but the main part isn't an image and you can't just download it in the same way

topaz cloak Sep 28, 2025, 2:46 AM

#

wary shadow actually works

yeah, i told you the url you were building was wrong

wary shadow Sep 28, 2025, 2:49 AM

#

topaz cloak yeah, i told you the url you were building was wrong

With selenium I would be able to grab the js image?

topaz cloak Sep 28, 2025, 2:49 AM

#

wary shadow With selenium I would be able to grab the js image?

i'm not so sure as it's so much more then just an image

wary shadow Sep 28, 2025, 2:50 AM

#

topaz cloak i'm not so sure as it's so much more then just an image

I know

#

But the image can be downloaded by itself, js was used for the zooming im feature

#

but there are js games that wouldnt work

topaz cloak Sep 28, 2025, 2:58 AM

#

no, it's not an image, it's a json file with coordinates that gets rendered on a html5 canvas on the page, here is the json file used for the data of the map https://xkcd.com/2067/asset/map-data.json

wary shadow Sep 28, 2025, 3:01 AM

#

Right click and save image as

#

It saves as a png

topaz cloak Sep 28, 2025, 3:03 AM

#

it just looks like it, it's what the browser has rendered on the canvas that you are saving
if you zoom in on a little part at one side and then save it again you'll only get that as an image

#

this is because it's just the rendered canvas that you are saving as an image

#

it's the javascript and the browser that is doing all the heavy lifting to create that image in your browser

wary shadow Sep 28, 2025, 3:04 AM

#

I see

#

Thanks

topaz cloak Sep 28, 2025, 3:06 AM

#

no problem

wary shadow Sep 28, 2025, 3:08 AM

#

what did u use to learn s a begineer? not just python. Just curious

topaz cloak Sep 28, 2025, 3:09 AM

#

books 🙂

wary shadow Sep 28, 2025, 3:09 AM

#

can you recommend some?

#

any language

topaz cloak Sep 28, 2025, 3:12 AM

#

not the ones i read back then, i tried to start at age 8 but i couldn't find any good books in my native language, so i hade to wait until i could read at least a bit of english at age 10 when i started to program

wary shadow Sep 28, 2025, 3:15 AM

#

topaz cloak not the ones i read back then, i tried to start at age 8 but i couldn't find any...

I see

topaz cloak Sep 28, 2025, 3:23 AM

#

wary shadow I see

one free online book that is quite good is https://automatetheboringstuff.com/

Automate the Boring Stuff with Python - Automate the Boring Stuff w...

A Page in : Automate the Boring Stuff with Python

wary shadow Sep 28, 2025, 3:23 AM

#

!close

elder prismBOT Sep 28, 2025, 3:23 AM

#

Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 img src error