I am trying to scrape a https site and i cannot for the love of god figure out what headers to use. ( trying to scrape html)
Atm, i am always reaching a "site not found" version of my site
#scraping Headers
rigid oxide
crude latch
Why would the headers prevent you from accessing the site???
rigid oxide
Why would the headers prevent you from accessing the site???
I solved the problem.
The Problem was: the website was blocking anyone who did not have the correct headers ( to check if they were a user)
so i just made a browser request to the server and copied all cookies my browser sent and put them into my code.