🔒 Requests question | Python | Page 1

wispy pilot Jul 21, 2024, 12:04 PM

#

I am trying to perform a get request on a url and extract a string from the respective url but i qm kinda new and i get the response from the site with "you dont have premission to acces .... on this server" this is because the site is coded so it detects requests from a python script or it can be because i didnt use some data on the request?

vital hareBOT Jul 21, 2024, 12:04 PM

#

wispy pilot I am trying to perform a get request on a url and extract a string from the resp...

#

@wispy pilot

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

sharp remnant Jul 21, 2024, 12:13 PM

#

wispy pilot I am trying to perform a get request on a url and extract a string from the resp...

Possibly both?

wispy pilot Jul 21, 2024, 12:13 PM

#

sharp remnant Possibly both?

How can i find ou

#

Out

sharp remnant Jul 21, 2024, 12:13 PM

#

Well. What site is it?

wispy pilot Jul 21, 2024, 12:14 PM

#

Htpps://mall.industry.siemens.com/

severe cloak Jul 21, 2024, 12:16 PM

#

Can you access the website normally from browser?

wispy pilot Jul 21, 2024, 12:16 PM

#

severe cloak Can you access the website normally from browser?

Yes

severe cloak Jul 21, 2024, 12:17 PM

#

Can I see your script?

wispy pilot Jul 21, 2024, 12:17 PM

#

That is what i dont understand since is a get request

#

Ok

severe cloak Jul 21, 2024, 12:17 PM

#

I dont think "the site is coded so it detect requests from a python script" is the reason..

sharp remnant Jul 21, 2024, 12:18 PM

#

Well since Siemens is in Germany, they fall under EU law which gives users the right to scrape data. It might be possible they have an API and want you to use that though

wispy pilot Jul 21, 2024, 12:19 PM

#

I dont see anywhere on their site anything API related but maybe i missed it

severe cloak Jul 21, 2024, 12:21 PM

#

Check the url is correct
Check the request headers is the same with your request from browser

wispy pilot Jul 21, 2024, 12:42 PM

#

severe cloak * Check the url is correct * Check the request headers is the same with your re...

it seems like i dont have the right headers i have to research how to fix that

blazing mountain Jul 21, 2024, 12:42 PM

#

Please share your code as well so it can be reviewed

severe cloak Jul 21, 2024, 12:44 PM

#

requets.get(url, headers=...)

wispy pilot Jul 21, 2024, 12:45 PM

#

from bs4 import BeautifulSoup as bs
import requests


prod_ID = input("Product ID> ")
url=f'https://mall.industry.siemens.com/mall/en/WW/Catalog/Product/{prod_ID}'


r = requests.get(url)

print(r.text)

Response:

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;mall&#46;industry&#46;siemens&#46;com&#47;mall&#47;en&#47;WW&#47;Catalog&#47;Search&#63;" on this server.<P>
Reference&#32;&#35;18&#46;4d677b5c&#46;1721563757&#46;39122ec3
<P>https&#58;&#47;&#47;errors&#46;edgesuite&#46;net&#47;18&#46;4d677b5c&#46;1721563757&#46;39122ec3</P>
</BODY>
</HTML>

wispy pilot Jul 21, 2024, 12:45 PM

#

severe cloak `requets.get(url, headers=...)`

makes sense

blazing mountain Jul 21, 2024, 12:45 PM

#

Maybe the request requires certain headers that you are not passing?

#

have you analyzed the request headers in the dev tools?

wispy pilot Jul 21, 2024, 12:46 PM

#

yeah but since i am very new to this is kinda hard to understand

#

Host: mall.industry.siemens.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://mall.industry.siemens.com/mall/en/WW/Catalog/Search?searchTerm=5SU1354-7KK06&tab=Product
Connection: keep-alive
Cookie: ste_vi=vMIUBOplWLRSke93VBEcxOD3a6bNpkxBV9J4qrbQdJiwhyAEyZOnmrAUl28xTwodTETSDSVu1sHE99fr33uDAh/LXk8Ek+/bDlSZLzvLJ2X80T8qH3z0VFeU3b2tEXXkipJSGxNMDkmiRjXU0atd5QhteWS7lTMCjZhEBeBUGE9wzQvb44fwSBXk2YU+P9nFHYVFzW81B9G/b8QD5YlCqtZ2/iWn4jdiobV4xxZF69O6k5BzFjWMalgfIlC2+7oNF0lO83ojVqHJuL0SdQB3lYUV8empRv0z8JO/i02mw//wiKeQrupEzxokCZY3MVF4xllfNFw=; bm_sv=33E7B86E72E332C6F781C82082AC721A~YAAQT2d7XC9oO9CQAQAAxI8z1Rg4U/nTSl+qmEwQUmqJBL00DmdE/O+bMIOeY9pfuDc0+Nz/0+vU6PxinZ19HzRXm8fnJt4vZxVS/VezpCJd7dnfyiZVlKGpPCIbExzS0pYWxdREamqPbI2tuW0dUIWBndxyZU1K4bCrYB43mKEygd+iRwDkMEQyu655khxPt4hpgN+vG1fh0Xme3useXL9np2lfuKzApTyqLqV8XH1E7fNXp9cMLQLOvq0YqVPKj6nwjANPEy+xTjY=~1; ste_s=stc%3Aundefined%7Cisterm%3A5su1354-7kk06%7Clang%3Aen%7Cdbid%3Aisp%7Cdbind%3Aundefined%7Cdbsubind%3Aundefined; _pk_id.5.ecfe=3ee36527e8b6193fd8649aa69f7c8b4b.1721563844.; _pk_ses.5.ecfe=1
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1```

#

these are the request headers that should be used

severe cloak Jul 21, 2024, 12:47 PM

#

Most likely the cookie is what you need

#

Try add the cookie to your header

wispy pilot Jul 21, 2024, 12:48 PM

#

the cookie is the same always?

severe cloak Jul 21, 2024, 12:49 PM

#

You can try load the web twice, and check if the cookie always same or not from browser network tab

wispy pilot Jul 21, 2024, 12:50 PM

#

seems to be the same

wispy pilot Jul 21, 2024, 12:54 PM

#

severe cloak You can try load the web twice, and check if the cookie always same or not from ...

i tried now using the cookie but for some reason is not used in the headers

#


from bs4 import BeautifulSoup as bs
import requests



headers = {"Cookie": "ste_vi=vi_fv%3A1721500273204%7Cvi%3A3ee36527e8b6193fd8649aa69f7c8b4b; ste_p=fv%3A1721500273204%7Cmpl%3Apage-length%253D1445%257Cword-count%253D324%257Cpage-name%253Dxm%253Aww%253Aen%252Fcatalog%252Fproductfamily%252Forderingdatatab%252F10263940%253Arccbs%7Cvs%3A1721562861097%7Clv%3A1721563844833; AMCV_EFB35E09512D2A530A490D4D%40AdobeOrg=1585540135%7CMCMID%7C24123175491935686236703609935927699020%7CvVersion%7C4.4.0; _sfid_baec={%22anonymousId%22:%22a0f92ed82b059c5a%22%2C%22consents%22:[]}; _evga_7af4={%22uuid%22:%22a0f92ed82b059c5a%22}; _gcl_au=1.1.1656965973.1721500276; s_fid=63BF165DCAFEE4F0-0C2C3DA09D1AABE2; ASP.NET_SessionId=f3033u34coa50qu0zxayxlrx; RegionUrl=/; .ASPXAUTH=D09291D010F179E4BCD84BDF8A77354F774D47D853051A4B6DD48773CEDB13526DF7124DB2CBFF0B14E2320E913CFA9420E25901CB04F9B892DD9FA4F254619626950845516F36EAE9405897D4F437830420CFC3437A8F52AF485925ABA7E22664FC5A101C0AB8373AE7419C101AC1FC3399F2C42556FAF48F0B16B4D578A230FA871D86F4060A538CBCB7172EE4A5C512B9CAFD; mall-sd=6b2a6e8e-1364-4324-be5a-5af2208d06fd; MallCookiePermitions=fnct:1,perf:1,targ:1; mall-tn=FAAC445E4F0620D3F8502E0E12E1AAF2|CatalogTree; s_cc=true; mall-ci=FAAC445E4F0620D3FcMLQLOvq0YqVPKj6nwjANPEy+xTjY=~1; ste_s=stc%3Aundefined%7Cisterm%3A5su1354-7kk06%7Clang%3Aen%7Cdbid%3Aisp%7Cdbind%3Aundefined%7Cdbsubind%3Aundefined; _pk_id.5.ecfe=3ee36527e8b6193fd8649aa69f7c8b4b.1721563844.; _pk_ses.5.ecfe=1"}
prod_ID = input("Product ID> ")
url=f'https://mall.industry.siemens.com/mall/en/WW/Catalog/Product/{prod_ID}'


r = requests.get(url,headers = headers)

print(r.headers)

#

is this the wrong way to parse the headers?

severe cloak Jul 21, 2024, 12:55 PM

#

Your printing the response headers, not request headers

wispy pilot Jul 21, 2024, 12:56 PM

#

oh wait

#

true

severe cloak Jul 21, 2024, 12:56 PM

#

Response headers != Request headers

wispy pilot Jul 21, 2024, 1:03 PM

#

severe cloak Response headers != Request headers

but how do i get the request headers

blazing mountain Jul 21, 2024, 1:03 PM

#

Request headers?

wispy pilot Jul 21, 2024, 1:03 PM

#

yea

blazing mountain Jul 21, 2024, 1:03 PM

#

they are mentioned right underneath the response headers

wispy pilot Jul 21, 2024, 1:03 PM

#

no i mean in python

#

i want to print the request headers

blazing mountain Jul 21, 2024, 1:05 PM

#

just print the headers you are passing. those are the request headers

wispy pilot Jul 21, 2024, 1:07 PM

#

i tried that previously but apprently those are the response headers

severe cloak Jul 21, 2024, 1:24 PM

#

print(headers) ?

wispy pilot Jul 21, 2024, 1:39 PM

#

severe cloak `print(headers)` ?

well i am using the cookie but still doesent work

#

ok update i managed to do it

#

it rquiered the user agent

severe cloak Jul 21, 2024, 1:54 PM

#

👍

vital hareBOT Jul 21, 2024, 2:54 PM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 Requests question