🔒 Return status 404 when trying to download PDF's from links

[dawn basin]

I have a .csv file that contains links to some PDF files. When pressed, these links automatically start to download the file (one example of such link would be https://rei.gov.ro/teza-doctorat-document/82405e69f7a7a04f8-TEZA-DOCTORAT-AGNES-GAZDAC-semnat-.pdf).

My attempts of building a python script to download all of them return 404. I want to understand why this is happening and to see if I can find a solution to this problem

csv = 'links.csv'  
df = pd.read_csv(csv)

def download_pdf(url, file_name):
    try:
        response = requests.get(url)
        response.raise_for_status() 
        with open(file_name, 'wb') as file:
            file.write(response.content)
    except requests.exceptions.RequestException as e:
        print(e)

for i, row in df.iterrows():
    url = row['Link']

    file_name = f'file_{i + 1}.pdf'
    download_pdf(url, file_name)

[heavy light]

@dawn basin

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[woeful goblet]

can u send the csv

[weary musk]

When i try to access the file through a normal browser i can't access it so it may be just be unavailable?

hope its not an ip grabber lol

[dawn basin]

This is weird, for me it just starts a download

[dawn basin]

Nope, it's just a link to a phD thesis don't worry about it :D

[weary musk]

Looking at the result im seeing there might be a region lock or potential access restriction as the server might block requests from "Agents" aka bots

[woeful goblet]

no that's 403

[dawn basin]

here is a sample of the csv, it's pretty large but I can see the error from the beggining as well

[woeful goblet]

some of them have "No link available"

[dawn basin]

Yes I'm aware, but the issue is still present for the ones that have

[dawn basin]

yes I also thought about that. Do you think a selenium based approach would bypass that?

[woeful goblet]

try printing the links

to see if there is a problem there

[dawn basin]

I've also tried that, but it seems to return the same link that is returned in the error message, both of them working when pressed

[weary musk]

Are these public files that anyone with the link can access or is there authentication needed that you might already have stored in your browser session?

[dawn basin]

The files are all public. I've scraped them from here https://rei.gov.ro/teze-doctorat
(Don't know if this is relevant or not)

[dawn basin]

Ok I got some insight. It seems that using a selenium-based approach doesn't work directly, BUT, manually navigating to the site to set cookies and then try downloading the files starts the downloads. Is do you know any way to load the cookies works with the requests package?

[heavy light]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.