Insights on how ChatGPT ACTUALLY works.

[prime stirrup]

Hi everyone! 👋

I'm working in R&D for a company that's preparing for a future where chatbots like ChatGPT might replace traditional web browsers. To get a better idea of how ChatGPT processes information, I'm running some experiments, and I was hoping to get some insights from the community here!

The Question:
Is it possible to manually simulate ChatGPT's browsing process? Here’s what I mean:

Could I instruct ChatGPT to perform a search, have it share the exact search terms and selected results, and then manually perform the search myself (using the same tools)?
I've already tried a "human search" comparison, and I’d like to analyze if ChatGPT uses traditional SEO and ranking criteria, how many sites it initially considers, what filtering process it follows, and how quickly it makes these decisions.
One Experiment So Far:
In a test, I asked ChatGPT to pick a random topic and perform a search. It chose "Generative AI 2024 current state healthcare applications" and used Bing for the search. Here’s what it used to form its answer:

I did the same search manually. Interestingly, the chosen pages were on positions 1, 3, 5, 11, and 14 of Bing’s results. If ChatGPT uses the same result order as a human (which I highly doubt), it would mean it analyzed over 130 pages in seconds before choosing those five.

This seems unlikely. I’m curious if ChatGPT:

Just picks the first results it gets (but perhaps receives a different order than a human),
Or if it actually checks 100+ pages each time in seconds. If so, how?
Does anyone know how this process works, or have any advice on setting up a fair test to get more clarity? Thanks a ton for reading, and sorry for the long post! 😅

[graceful anvil]

I think you have your figures mixed up. The LLM portion of ChatGPT is fixed in terms of knowledge. Only when the model is updated does it actually change.

Regarding the "search" aspect, ChatGPT does not give proper citations for a lot of its results compared to services like Google Bard. In those cases where proper citation is missing, assume ChatGPT came up with the response on its own and not from the web.

Additionally, ChatGPT (the model) is not browsing the web. The service is passing your query to some search engine API (ie DuckDuckGo or Google Search) to get a response and the LLM uses that response to stitch together a response to the user. Unless you know the search engine that is being used, there is no simulating the search results.