Fact checking ChatGPT with Bing

[stable bobcat]

Until a browser plugin for ChatGPT 3.5 and 4 comes (which is not guaranteed for 3.5) here is a method to fact check Chatgpt 3.5.

I asked ChatGPT 3.5 :

provide a paragraph where you mention the distance between Orlando Florida and Sao Paulo Brazil somewhere in the middle

Then I copy pasted the answer into Bing with "Do you see anything even slightly false in this text: "

Do you see anything even slightly false in this text: The distance between Orlando, Florida and Sao Paulo, Brazil is quite significant, as they are located on two different continents. The distance between these two cities is approximately 4,597 miles or 7,397 kilometers. This means that a direct flight from Orlando to Sao Paulo would take approximately 9 hours, depending on factors such as wind speed and direction ...

It searched for the distance between Orlando Florida and Sao Paulo Brazil and answered:

I don’t see anything false in the text, but I do see some slight discrepancies with the information I found online. According to my search results, the distance between Orlando and Sao Paulo is 4,264 miles or 6\u2009862 km1, not 4,597 miles or 7,397 km as stated in the text. Also, the flight time from Orlando to Sao Paulo is estimated to be around 12 hours and 36 minutes2, not 9 hours as stated in the text. However, these differences could be due to different sources or methods of calculation, so I wouldn’t say the text is false, just slightly inaccurate.

[EDIT:
see #1103744398593904721 message in this thread for things to consider when using Bing as a fact checker. There is a link in that message that can bring you back to the first message of this thread.
]

Note : I do not know if Bing is always able to reliably find what needs to be fact-checked and answers found online by Bing might be false (potentially the answer by ChatGPT might be right and the one by Bing might be wrong).

Fact checking chatgpt with bing

Fact checking chatgpt with Bing

Fact checking ChatGPT with Bing

I sometimes prefer ChatGPT's answers to Bing as ChatGPT often feels like it does more effort and it feels more like it has interiorized and processed the information while Bing feels like a new intern that is just trying to search for vaguely similar information online. Bing might not even check if the text it takes makes that much sense with what was asked.

[wide pond]

I see what you are doing here but if its just for factual answers, why not use one of the bots that always backs up the responses with the sites he pulled that info? like neeva, phind, lexyai, askai ...

[stable bobcat]

Hi, thank you for the recommendations, I am not sure what those sites are. Do they use the ChatGPT API ?

Just to clarify in case a reader does not understand how this could be useful. The purpose is not to find direct answers that one can find easily using Google as in the example given but to fact-check answers from questions that require a long-form answer. This might not guarantee accuracy as mentioned in the post but might increase the probability that the answer is not false.

[sage bay]

This is interesting!

One place this might be especially useful is in fact checking sources or references that ChatGPT provides.

For example, when I was preparing #1074055436417912883I was hand-checking every result GPT provided of a list of ten sources. Here's an example of my prompt and the AI output (and I bolded the books that don't exist, reliably 6 out of 10 are not real, however all claimed authors and titles have something to do with the topic question, this is useful garbage to research and do my own reading about, but useless as a 'one step have a working list that is what I asked for'.

me: I need a list of ten books about managing pests in a garden.

ChatGPT: It: Sure, here are ten books about managing pests in a garden:

"The Organic Gardener's Handbook of Natural Insect and Disease Control" by Barbara W. Ellis and Deborah L. Martin

"Good Bug, Bad Bug: Who's Who, What They Do, and How to Manage Them Organically" by Jessica Walliser

"The Garden Pest and Disease Control Handbook" by Keith Reid

"The Natural Garden: A Guide to Creating a Low-Maintenance, Eco-Friendly Garden" by Ken Thompson

"The Garden Problem Solver: All Your Gardening Questions Answered" by Roy Lancaster

"Pests of the Garden and Small Farm: A Grower's Guide to Using Less Pesticide" by Mary Louise Flint

"Integrated Pest Management for Home Gardens and Landscapes" by Mary Gardiner and Stephen L. Clement

"The Garden Pest and Disease Diagnostic Guide" by National Gardening Association

"Organic Gardening: The Natural No-Dig Way" by Charles Dowding

"The Garden Cure: Natural Solutions for a Healthy Garden" by Adam Woods

I'll see how Bing handles verifying this stuff and report back soon.

[sage bay]

In the most natural way for me to think to try, it completely fails.

Bing does not choose to use my requests to verify the list as a reason to check the list, and tells me 'ehh, can't really be done' based on what it does try.

I could ask it about each book singly.

but that's how I verified the list myself the first time, I individually looked them up!

I don't see how 10 searches through Bing AI would be any faster than 10 searches through Google.

[sage bay]

Asking like this works, BUT it's slow, I can't find a way to get Bing to search all 10 at once, but I can give the list then say 'check the second' 'check the third'.

When I then ask it to check the remaining books, it restates that the third doesn't appear to exist, then just lists the remaining books and authors from the list I gave, without checking any of the info or claiming it is real or false. The biggest clue, it didn't search anything before providing that answer.

Two of the remaining 7 books are 'real' and correctly listed with the author, the other 5 are fake, so they can't just be treated as a single set of fakes, which might be the implication since the first, #3) is restated as being unfindable.

[stable bobcat]

you picked the lazy mode

i do not like that mode much

creative is more helpful

i will check too

[sage bay]

I'm in creative, I've never used anything else!

Amusingly, one of the SUGGESTED questions to ask was "which of these books is your favorite?"

I thought that was really ludicrous and maybe would be interesting to see how this fails.

1. It's an AI, and probably will claim to have no preference.

2. Most of the books don't exist, so it'll be interesting if it DOES claim one, or anything about them as a set.

Weirdly, that question was the CLOSEST to useful, when I followed up.

[stable bobcat]

the worst in terms of minimal effort for me was "professional" it felt like someone that just wanted me to leave as soon as possible

[stable bobcat]

but it's blue, creative is pink (which is kind of sexist)

[sage bay]

For all it lied, and I asked about the lie, and in trying to prove its lie it actually chose to investigate a whole three books.

[sage bay]

For me, all of bing is blue. Here's a new window of the AI search chat, which defaults to balanced:

I always click on creative. I did for that chat above too.

I take that back, somehow I was still in balanced, thank you!

[stable bobcat]

I will try limiting the number of of books, I imagine that Bing has a limit on how much it can search in one request. I only saw it use at most maybe 3 searches in one request

[sage bay]

Okay! This went interesting when I tried it in creative.

It ran the search twice, on the same input, giving outputs in parts.

Now, a LOT of the info it gave is hogwash, but it TRIED to check most of the books.

[stable bobcat]

(:

[sage bay]

Which is maybe even less useful, for fact checking because it said the second book doesn't exist (it does, correct author), but somehow it misread my input and gave the second author for the first book as an author on the second book.

[stable bobcat]

What did it tell you ?

[sage bay]

And the first message that says book 2 is wrong author, in the second part of the first message it says, 'well, it's right'.

But the REST of the books it makes claims about... heh

[stable bobcat]

Maybe separate the books by - so it maybe sees it as separate list elements

[stable bobcat]

It sent multiple messages at a time ? I am not sure I knew it could do that

[sage bay]

I will, however, check this out more first.

For one, it claimed this exists:

“The Natural Garden: A Guide to Creating a Low-Maintenance, Eco-Friendly Garden” by Ken Thompson

So I asked for more details. As you can see from the IMAGES provided, the NAME of the book is not the same as the name of the book that was written by that same author, and it is now merrily telling me about.

[sage bay]

Agreed, I had no idea it could either!

But you can see above the two separate outputs, both 1 of 20.

It searched, responded, then ran more searches, provided more response, all without new input from me.

[stable bobcat]

weird it has that autonomy and probably few people know about it. Maybe it's a new update.

[sage bay]

I searched google for eden project books, the Bing claimed publisher, the year claimed, the author claimed (I'd already searched for the book itself): He's an author, but he never wrote that book!

I push poor Bing AI I bit, asking where to buy it, which gets vague answers about major sellers, and links to NOT the major sellers, hehe.

And then I ask for a direct link, which it says it can't, but offers something else, I ask for the something else and it becomes a URL that doesn't exist.

[stable bobcat]

I am confused. Does this exist:

The Natural Garden: A Guide to Creating a Low-Maintenance, Eco-Friendly Garden”

[sage bay]

I haven't used Bing much, and I haven't seen it before. It is very cool. I would like to learn more about it.

[sage bay]

No, that does not exist, that is not a book written by ANY author.

It's a reasonable book title, and there are MANY similar titles.

ChatGPT hallucinated it as a response to me several months ago, as part of a list of 10 reference books about gardening - 6 of the list were hallucinations including that one.

I offered it to Bing as part of the fact checking list, because it was part of it, and that's a useful thing if Bing can factcheck lists like that!

No evidence Bing can do this, it's hallucinating the book exists with great focus and vigor.

[stable bobcat]

Ok, I understand so it hallucinated, I think I saw this before. I should mention in my post that one should consider its sources and not much of what it says.

[sage bay]

Yeah. Your title and example made me think of it, and wonder if with the web-search capability, if Bing could handle this kind of fact checking, and in a faster/easier way for the user (or at least me) than my individually googling each book on the list (been there, done that, I am 100% sure I know the facts and flaws of this list of 10, I checked them carefully back when testing ChatGPT and hallucinations in response to this kind of question).

I don't think Bing can help with this kind of hallucination better than a person directly, one by one googling; they'd have to directly, one by one ask Bing, in a new chat with Bing.

And that's about as much work as googling it myself.

Plus then I have to wonder if there's any error in Bing's decision, versus making my own by scanning the sources and results myself.... so.

[stable bobcat]

I still think it can identify maybe 3 facts in the text and search for them online. then you just write "anything else that might be incorrect" and it searches maybe 3 more

I will give it a test with 6 mistakes and 4 truths

But yes it seems it can not handle a list of 10 elements

do you know what I can ask it that it will likely make a mistake ?

I can't ask it to say something false

[sage bay]

It's kind of very interesting though, it looks ALMOST like creative decided to search each book, at least the first two, and answer each with separate searches.... then gave up after the second set of searches and just listed a bunch of the other books asked about and said 'these are good' with no evidence, and the rest are bad, no evidence.

But when I check the links it provided for each of the books, the links are rather random.

The one on the third book, for example, is NOT the third book, it's about the first.

There's two Amazon links for the first book, they both go to two very similarly named, same authored books, but they're not the same book (one is correct though).

The links for the second book, some are about the first book.

[sage bay]

If you want this prompt to work with:

I want to know which of these 10 books is a real book with the correct author listed. 1) "The Organic Gardener's Handbook of Natural Insect and Disease Control" by Barbara W. Ellis and Deborah L. Martin 2) "Good Bug, Bad Bug: Who's Who, What They Do, and How to Manage Them Organically" by Jessica Walliser 3) "The Garden Pest and Disease Control Handbook" by Keith Reid 4) "The Natural Garden: A Guide to Creating a Low-Maintenance, Eco-Friendly Garden" by Ken Thompson 5) "The Garden Problem Solver: All Your Gardening Questions Answered" by Roy Lancaster 6) "Pests of the Garden and Small Farm: A Grower's Guide to Using Less Pesticide" by Mary Louise Flint 7) "Integrated Pest Management for Home Gardens and Landscapes" by Mary Gardiner and Stephen L. Clement 8) "The Garden Pest and Disease Diagnostic Guide" by National Gardening Association 9) "Organic Gardening: The Natural No-Dig Way" by Charles Dowding 10) "The Garden Cure: Natural Solutions for a Healthy Garden" by Adam Woods

For your reference, books 1, 2, 6, and 9 are real books with the correct author.

The others, 1 think one's real, but wrong author, the rest are not real books, and all the authors are authors or scientists that publish about the topic gardening, just never wrote anything named the book title they were given credit for by ChatGPT.

[stable bobcat]

I tried asking for specifications of the iPhone 13 mini and Bing falsely claimed that the battery life given by chatgpt was false while it seemed to be true but to understand that, the bot had to understand that on the apple page, everything in the left column was for the mini and everything on the right was for the not mini. An interesting example of how a non visual bot might get confused navigating the internet

[stable bobcat]

I also noticed that it seems to invent information from sites. When I try to debate that the information does not exist it sticks with its invention no matter what. Which is why its sources are more useful in my opinion.

Asking it to identify stuff that might be false is the only time I feel like Bing could really be useful as a search engine. The other times I feel more like it does cheap effort and I could do better searching myself.

[sage bay]

I think the idea of using an AI to fact check an AI is interesting to explore. It has some flaws, the way that you need an accurate thermometer to have any hope of using one thermometer to check if the other is right.

However, some techniques exist:

1. Ask the same question more than once in a new conversation with the AI.

once it starts to believe its hallucination, that's it - but it's hopefully unlikely to hallucinate the exact same way in a second conversation. So compare the facts of more than one conversation - same question, different FACTS should be suspicious, unless there's genuine wriggle room.

Your opening question in this thread, about distances and flight times....

could be wriggle room.

What roads and driving conditions? Some times of year traffic is slower because of weather or holiday traffic patterns.

The same with flight, even the TYPE of aircraft can change how long the flight takes, and by several hours, depending on distance.

2. check EVERY detail yourself against verifiable sources. This is like having an accurate thermometer to check a questionable one.

[stable bobcat]

Things to consider:

Note: Bing can hallucinate false information as shown in the thread above. I would consider more checking its sources than what it says.

Note: Check what Bing searches for, if it provides information outside of its searches it might be inventing.

Note: Bing usually only makes one search per turn. Ask it if it finds anything else to fact-check until it says it does not see anything false.

Note: This is just an example. The actual use case is to answer questions with long-form answers that are not easy to find using Google.

Note: I do not know if Bing is always able to reliably find what needs to be fact-checked and answers found online by Bing might be false (potentially the answer by ChatGPT might be right and the one by Bing might be wrong).

Link back to the start of the thread: #1103744398593904721 message

[stable bobcat]

The distance choice was maybe not the best option. At the same time, it was interesting to see how Bing noticed that "it depends"

[sage bay]

Agreed, I wasn't trying to imply it was a bad choice. It's got value for testing, and it's a fine way to open the topic!

[stable bobcat]

@sage bay ( : ( ... bows)

[wide pond]

I got this:(deleted, below is a better picture)

but more valuable are the direct link on the left

[wide pond]

look at my post, thats what I mean I get an detailed explanation for my request, proper links preview, I can regenerate, or when open dropdown menu can add extra context or code, and there is also a field where I can put links in that it will include in the search.

[stable bobcat]

Hi, I can't see the characters in that last screenshot, just one or two normal screenshots might be more visible.

I also do not know what site you used as you did not mention it in your last answer.

Concerning the other features you mentioned I am not sure I understood as I might have to see the interface to understand what each part of your message really means

[wide pond]

+daaaamn it for so many reasons,

just wrote a lot of text with explaining and apologies and corrections and stuff and ALL vanished bc of "posting a non whitelist-url" but it didn´t end there re-did all of it and already posted better screenshots and was just about to finish suddenly connection issues and F-it now I am on and see nuffin had come thrugh. sry for the 🤬 not at all your fault,
gtg need cool down

oh and of course this
**doubleFU.phindDOTcom is the search I was talking about

as you see, enhanced search customization provided by default. I know and used a few that work similar but phind 🐸 absolute favorite

[stable bobcat]

yeah the automatic deletion can be aggravating

[stable bobcat]

I checked it out and it looks interesting. It made me want to try out all of the bing ai competitors. Thanks (: