🔒 Script runs perfectly in Pycharm configuration and using pdb, fails when run from terminal

[trim crescent]

When running the script when run via Pycharm or in local terminal using pdb, the script works perfectly. When run in the terminal, however, whether it's the local interpreter or Pycharm's emulated terminal, one of the lines fails out. The issue here is, as stated above, when I run the script through pdb in my local terminal (WITHOUT being inside the project env), it works as expected. All variables end up as they do in Pycharm, so I don't know where exactly the problem is occurring.

The file being executed is a symlink to the original project file. The error is occuring somewhere in here:

    def find_image_link(self):
        try:
            request = urlopen(self.src_url)
        except HTTPError as e:
            print("HTTP Error:", e)
        except URLError as e:
            print("Page not found:", e)
        except ConnectionError as e:
            print("Connection error:", e)

        else:
            try:
                html = request.read().decode("utf-8")
                soup = BeautifulSoup(html, "html.parser")
                div = soup.find("link", {"rel": "image_src"})
                link = div.get("href")
            except AttributeError as e:
                print("Could not find necessary attribute:", e)
            else:
                self.img_url = link```

[near sluice]

@trim crescent

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[trim crescent]

The only error being thrown is bs4's AttributeError when it tries to search the empty string. Traceback is below:

Traceback (most recent call last):
  File "/Users/kyle/Desktop/download-thumbnail", line 65, in <module>
    dl.download_image()
  File "/Users/kyle/Desktop/download-thumbnail", line 43, in download_image
    with urlopen(self.img_url) as img_response:
         ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyle/.pyenv/versions/3.12.0/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyle/.pyenv/versions/3.12.0/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 499, in open
    req = Request(fullurl, data)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyle/.pyenv/versions/3.12.0/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 318, in __init__
    self.full_url = url
    ^^^^^^^^^^^^^
  File "/Users/kyle/.pyenv/versions/3.12.0/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 344, in full_url
    self._parse()
  File "/Users/kyle/.pyenv/versions/3.12.0/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 373, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: ''```

It seems to be something with BeautifulSoup. I've confirmed that the attribute is in the source pulled and decoded by urllib

[rapid vault]

Can you paste the entire code? It's hard to troubleshoot otherwise.

!paste

[near sluice]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[trim crescent]

import argparse
import os
import pdb
from mimetypes import guess_extension
from shutil import copyfileobj
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
import sys
from bs4 import BeautifulSoup


# TODO: make executable in terminal
class Downloader(object):

    img_url = ""

    def find_image_link(self):
        try:
            request = urlopen(self.src_url)
            print(request)
        except HTTPError as e:
            print("HTTP Error:", e)
        except URLError as e:
            print("Page not found:", e)
        except ConnectionError as e:
            print("Connection error:", e)

        else:
            try:
                html = request.read().decode("utf-8")
                print(html)
                soup = BeautifulSoup(html, "html.parser")
                div = soup.find("link", {"rel": "image_src"})
                link = div.get("href")
            except AttributeError as e:
                print("Could not find necessary attribute:", e)
            else:
                self.img_url = link

    def download_image(self):
        save_dir = os.getcwd()
        filename = "downloaded_image"
        try:
            with urlopen(self.img_url) as img_response:
                content_type = img_response.headers["Content-Type"]
                ext = guess_extension(content_type)
                with open(os.path.join(save_dir, filename + ext), "wb") as file:
                    copyfileobj(img_response, file)
        except OSError as e:
            print(
                "Image destination is not writable. Check your directory permissions:",
                e,
            )

    def __init__(self, source):
        self.src_url = source


if __name__ == "__main__":
    print(sys.version)
    parser = argparse.ArgumentParser()
    parser.add_argument("url")
    args = parser.parse_args()
    dl = Downloader(args.url)
    dl.find_image_link()
    dl.download_image()
    print("Download Complete")```

shebang is #!/usr/bin/env python

[obsidian marten]

probably not related to your problem but any reason you're not using requests instead of urllib?

[trim crescent]

Not really. I generally try to limit dependencies as much as possible. I've heard requests is quite good. My script just isn't really long enough for the differences between urllib and requests to matter.

So, seems to be something with the parser. For whatever reason, the html that soup is spitting out is different than it is in all the other situations. So rel="image_src"exists in html but not in soup. But, again, only when the script is called directly through zsh.

[obsidian marten]

silly question: are you testing this against a public url? if so what is it? so many sites use client-side scripting to generate the ui so bs4 won't pick those up (i assume you know that already)

and just to confirm my understanding of this script:

• you plug in a website url
• it looks for the first instance of an image (in the server-generated html)
• download said image
• done

not looping or scraping or anything, just simply looking for the first image and that's it.

[trim crescent]

It's specifically pulling the youtube thumbnail. So this bit is pulling the link to the thumbnail from a <link> element with rel="image_src".

The thing that gets me is that urlopen(url).read.decode() DOES contain the element in question. I popped into TextEdit and Cmd+F'd it to confirm. However, BeautifulSoup(html, "html.parse") does NOT.

[rapid vault]

It's really against their ToS to scrape their site.

Google has loads of APIs for stuff.

If you're accessing YouTube with this then we can't help you with it.

[trim crescent]

Does YouTube have an api? I didn’t think it did but I’m not at all against calling it if it’s available.

[near sluice]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.