#๐Ÿ”’ Script Assistance

167 messages ยท Page 1 of 1 (latest)

regal pond
#

I'm VERY much a Python beginner and still very much unfamiliar with good scripts. I am trying to scrape Yearbook pictures from a website and each picture is connected to the link of the page, where the picture changes when the link changes. So for example, the "0001" in the link is the first page of the yearbook, whereas "0175" would be the 175th page.

I want Python to scrape all 200 pictures and save them to a file on my computer by using the links and searching/saving manually all the pages of the books by changing the numbers in the link. I got a basic script online, but anytime I use it, I keep getting "indentationerror" or other such errors, and I think my script is just wrong. The script will be inserted below. Any help would be enormously appreciated.

sacred patioBOT
#

@regal pond

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

regal pond
#

import requests
import os

Updated base URL (as per your provided URL)

base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg?h=7fb49000fc99d751f0efab372937d731"
image_extension = ".jpg" # Assuming the images are in .jpg format

Folder to save the images

save_folder = "1980 Fordson" # Set the folder name to "1980 Fordson"
if not os.path.exists(save_folder):
os.makedirs(save_folder)

Define the page range (from 0001 to 0208)

start_page = 1
end_page = 208

Loop through each page number from 0001 to 0208

for page_num in range(start_page, end_page + 1):
# Format the page number with leading zeros (e.g., 1 becomes '0001', 50 becomes '0050')
page_str = str(page_num).zfill(4)

# Construct the image URL
image_url = f"{base_url}{page_str}{image_extension}"

try:
    # Send an HTTP GET request to fetch the image
    response = requests.get(image_url)

    # Check if the request was successful
    response.raise_for_status()

    # Define the local path where the image will be saved
    image_path = os.path.join(save_folder, f"page_{page_str}{image_extension}")

    # Save the image to the specified path
    with open(image_path, "wb") as file:
        file.write(response.content)

    print(f"Downloaded page {page_str} successfully.")

except requests.exceptions.RequestException as e:
    # Handle errors (e.g., page doesn't exist or network issues)
    print(f"Failed to download page {page_str}: {e}")

print("Download process complete!")

sacred patioBOT
#

Hey @regal pond!

Please edit your message to use a code block

```py
print('Hello, world!')
```

This will result in the following:

print('Hello, world!')```
agile plover
#

well it's definitely affecting the visibility of your code to my eyes (it's very hard to read)

#

follow what the embed says

regal pond
#
import requests
import os

# Updated base URL (as per your provided URL)
base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg?h=7fb49000fc99d751f0efab372937d731"
image_extension = ".jpg"  # Assuming the images are in .jpg format

# Folder to save the images
save_folder = "1980 Fordson"  # Set the folder name to "1980 Fordson"
if not os.path.exists(save_folder):
    os.makedirs(save_folder)

# Define the page range (from 0001 to 0208)
start_page = 1
end_page = 208

# Loop through each page number from 0001 to 0208
for page_num in range(start_page, end_page + 1):
    # Format the page number with leading zeros (e.g., 1 becomes '0001', 50 becomes '0050')
    page_str = str(page_num).zfill(4)

    # Construct the image URL
    image_url = f"{base_url}{page_str}{image_extension}"

    try:
        # Send an HTTP GET request to fetch the image
        response = requests.get(image_url)

        # Check if the request was successful
        response.raise_for_status()

        # Define the local path where the image will be saved
        image_path = os.path.join(save_folder, f"page_{page_str}{image_extension}")

        # Save the image to the specified path
        with open(image_path, "wb") as file:
            file.write(response.content)

        print(f"Downloaded page {page_str} successfully.")

    except requests.exceptions.RequestException as e:
        # Handle errors (e.g., page doesn't exist or network issues)
        print(f"Failed to download page {page_str}: {e}")

print("Download process complete!")
regal pond
brittle hatch
#

right

#

and you say you got an IndentationError? can you copy-paste the full error message please?

regal pond
#

The characters for it are too long

brittle hatch
#

!paste

sacred patioBOT
#
Pasting large amounts of code

So that everyone can easily read your code, you can paste it in this website:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

brittle hatch
#

^ you can use this

regal pond
sacred patioBOT
regal pond
agile plover
#

oh are you using repl?

brittle hatch
#

@regal pond how are you running this code? it looks like you're just pasting it into a Python shell (like IDLE)?

regal pond
#

Yeah I just kinda copy-pasted it

#

I have a feeling it's not right but I'm too novice to know how to fix it really

agile plover
#

you are pasting your code into a window like this?

brittle hatch
#

you should save the code to a .py file then run it

regal pond
regal pond
agile plover
#

did you install any code editor?

regal pond
#

All of this is so new to me

agile plover
#

also just curious, who gave you the code? they didnt explain how to use it?

brittle hatch
regal pond
#

Not familiar with all of this stuff, just trying to learn as I go

agile plover
regal pond
#

As ashamed as I am to admit

#

Desperate times called for desperate measures, I'm very much anti-AI but I don't know how to do this stuff myself

agile plover
#

well i was wondering why the code is wrong

regal pond
#

I was hoping it would give me a template script I could use but it's not good unfortunately

agile plover
#
base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg?h=7fb49000fc99d751f0efab372937d731"
# ...
page_str = str(page_num).zfill(4)
# Construct the image URL
image_url = f"{base_url}{page_str}{image_extension}"
```the URL this generates will be wrong
#

instead of generating
website.com/0001.jpg and
website.com/0002.jpg and so on, it will instead create
website.com/0001.jpg?h=some_code0001.jpg and
website.com/0001.jpg?h=some_code0002.jpg

#

even if you successfully run it, it wouldnt work

brittle hatch
#
base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/{num:04}.jpg"
# ...
image_url = base_url.format(num=page_num)
regal pond
agile plover
regal pond
#

I wonder why it generated the script like that

agile plover
#

well, with the query params too

brittle hatch
#

it works without the param

agile plover
#

ah you tested it

regal pond
brittle hatch
#

it's probably just some tracking thing ๐Ÿ˜†

agile plover
#

ah right

brittle hatch
regal pond
#

This is one of those things I really need a one-time template for so I can kinda just bulk save a buncha old local yearbooks for archival purposes. As much as I do want to learn Python in the future, I feel like it'd be a bit of a hassle for a newbie like me to take on just to learn one script

#

I'm so lost rn ๐Ÿ˜”

#

So was there anything wrong with the second half of my script? Aside from formatting

brittle hatch
#

even if you got perfect code from somewhere, you need to actually run it on your machine

regal pond
#

Wouldn't the script be the only thing I'd need at that point, then?

#

Just sorta copy and paste it for future uses?

#

Replacing the URL of course

agile plover
brittle hatch
agile plover
#

right now you already met a scenario where it would have been nice if you knew how to code

regal pond
#

Of course, that's true. I want to get into sports analytics in the future and Python could be a big help, I get that. I'm just very preoccupied with personal stuff atm so it's one of those things I don't have the time to really sit down and try to learn with full focus

regal pond
brittle hatch
agile plover
#

perhaps we should install notepad++?

#

actually just notepad is fine

regal pond
#

I have Notepad, it's what I've been using to edit the scripts

brittle hatch
regal pond
#

Yeah, I have it open in Notepad right now and saved on my PC, but it's that same bad script I sent earlier

agile plover
#

you want to make sure file name extensions is turned on

#

let's make sure that you are saving your script as script.py and not script.py.txt

brittle hatch
#

(for Windows 11)

regal pond
#

Yup just checked it

agile plover
#

double check your script's file name

brittle hatch
regal pond
sacred patioBOT
agile plover
#

is it something.py or something.py.txt

regal pond
#

It's just .txt

agile plover
#

you have to rename the file

regal pond
#

To .py?

agile plover
#

yes

regal pond
#

So Python Script.py would be good to rename it, right

#

or .py.txt

brittle hatch
#

.py, not .py.txt

#

the name before it can be whatever you want
personally I would go with something more descriptive than Python Script.py, hah

regal pond
#

Lol I'm not very original, as you guys can tell

agile plover
#

it's fine

brittle hatch
#

OK, so which folder is it saved in?

regal pond
#

Just in downloads, should I make a folder in like documents and save it in that?

brittle hatch
#

it's fine wherever

agile plover
#

so long as you know where it is saved

brittle hatch
#

now please open Terminal

regal pond
#

Alright, it's open

brittle hatch
#

do cd Downloads then ls, and post a screenshot (censor any personal info if you want)

regal pond
#

Sorry I don't know if I did it right but I hope I did

brittle hatch
#

that Script.py is the one we're working on, yes?

regal pond
#

Yes

brittle hatch
#

ok. so, you can run it with py Script.py
I expect there will be more errors, but let's deal with them one by one

regal pond
#

I keep trying to open Python but it's only opening Terminal but saying it's Python, and when I paste it it keeps saying:

 File "<python-input-0>", line 1
    py Script.py
       ^^^^^^
SyntaxError: invalid syntax
>>>
brittle hatch
#

don't open Python.

brittle hatch
regal pond
#
PS C:\Users\Moodo> py Script.py
C:\Users\Moodo\AppData\Local\Python\pythoncore-3.14-64\python.exe: can't open file 'C:\\Users\\Moodo\\Script.py': [Errno 2] No such file or directory
brittle hatch
#

did you close that window?

#

the cd Downloads was important. it placed you in the Downloads directory so you could directly access Script.py

regal pond
brittle hatch
#

the Terminal window

brittle hatch
regal pond
#

Yeah I did

dire flower
#

Navigate back with cd or just open the terminal from file explorer if it's in the script's folder

regal pond
#

I think I did it right this time, but it just keeps saying:

PS C:\Users\Moodo\Downloads> py Script.py
Failed to download page 0001: 403 Client Error: Forbidden for url: https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg?h=7fb49000fc99d751f0efab372937d7310001.jpg

and so on and so on up until page 208 (final page of the book)

#

I think I have to change the start to put your script change from before, it still had my old one I believe

dire flower
#

Seems like you must be authorized to use this website

agile plover
regal pond
agile plover
agile plover
regal pond
#

Just put that at the start with the rest and just go from there?

agile plover
#

you need to replace the problematic part

#

not just add on the code

#
# Updated base URL (as per your provided URL)
base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg?h=7fb49000fc99d751f0efab372937d731"
image_extension = ".jpg"  # Assuming the images are in .jpg format
```first replace this with
```py
base_url = "https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/{num:04}.jpg"
#

this replaces the incorrect template that chatgpt was using to construct the image's URL

#

next replace

# Construct the image URL
image_url = f"{base_url}{page_str}{image_extension}"
```with
```py
# Construct the image URL
image_url = base_url.format(num=page_num)
#

this line of code generates the image URL from the template URL that @โ€‹Bharadwaj Raju fixed

#

copy and paste the content of the file that you have now

#

let's double check

regal pond
sacred patioBOT
agile plover
#

do you know how to run the file?

regal pond
agile plover
#

are you in the correct folder?

regal pond
#

Like Bharadwaj Raju said earlier

regal pond
agile plover
#

right, now give it a try

#

let's see if we meet some error

regal pond
#

Same as before

agile plover
#

have you saved the file?

regal pond
brittle hatch
#

Can you paste the full error message again? Please copy and paste it again, even if it seems similar to the previous one

agile plover
#

copy the error and paste it here

regal pond
#
PS C:\Users\Moodo\Downloads> py Script.py
Failed to download page 0001: 403 Client Error: Forbidden for url: https://yb.cmcdn.com/yearbooks/5/0/9/2/509240e16947ef504c34164a58ddb4a8/1100/0001.jpg
#

It just does it so on and so on until page 208 again

brittle hatch
#

OK interesting

#

I can visit that page in a browser just fine

#

So maybe it's rejecting non-browser user agents?

regal pond
#

Is the link in my script messed up, maybe?

brittle hatch
#

Nah, the exact link in the error would work if you open it in a browser. Try it yourself

regal pond
# regal pond

When I copy and paste the link here it leads to an error, but idk if that's just how the link is supposed to look like for this script

agile plover
regal pond
#

The main site of it is a bit strict but I don't believe they'd be able to bypass something like Python simply scraping the pictures from the link automatically though

#

It's gotta be an issue with my script that I have wrong

agile plover
#

nope

brittle hatch
agile plover
#

unfortunately that does mean it's against the rules for us to bypass this

regal pond
#

So there's no possible way to do it?

#

๐Ÿ˜”

brittle hatch
#

It's possible, quite easily too

agile plover
#

but it would be against the rules

brittle hatch
#

But as a rule, we don't help with circumventing website protections on this server

brittle hatch
sacred patioBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.