#🔒 Python requests return different results than normal accesses

30 messages · Page 1 of 1 (latest)

dawn fog
#

I am trying to get data from a website, specifically tasks to be done on the website, however when I use requests to send a get command to the page, the returned html code is quite similar to normal access but Some data is missing, what's missing are the tasks on the website that I need to get. Help me please

earnest bearBOT
#

@dawn fog

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
woeful hound
#

Could it be that the data inside that you want is lazy loaded due to it being fetched?

dawn fog
merry geyser
#

most websites return different data to "requests" than to a web browser, for a couple of reasons

#

one reason is that the web browser sends different headers

woeful hound
#

A way to handle it would be to check the network inspector and see if you could just fetch it directly. If that is not possible, you'd need to wait for the JavaScript to load. There are several approaches to this.

merry geyser
#

the other reason is that the browser executes the javascript in the response, which causes it to make more requests; all those requests get merged into the web page you see

#

you should see if the web site has an API, and use it if it does; or else you'll have to use "selenium" (that's a library that lets you control an actual web browser to do the requests). Neither is easy.

woeful hound
#

Depending on if it's a one time thing or not, you can also use tools like Tampermonkey to inject your own JavaScript and mutate the entire page to do what you want and even make requests inside the webpage.

dawn fog
#

That website does not support API

#

Is it possible to wait for the website to load javascript without using selenium? :<<

dawn fog
woeful hound
#

But it's something you don't need to do automatically but just to speed up workflow easier to just use Tampermonkey.

dawn fog
woeful hound
#

Then using the package I sent above could be the solution.

dawn fog
#

I tried it and it still doesn't seem to work =(((

woeful hound
#

You used the render?

dawn fog
woeful hound
# 
session = HTMLSession()

url = 'https://example.com'
response = session.get(url)

response.html.render(timeout=5)
dawn fog
wet venture
#

One more thing I have noticed is.

#

Some websites store cookies.

#

when you visit their site for the first time

#

and when an API access is made. It checks if the cookie is still present.

earnest bearBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.