#Classifier Changes Answer

4 messages · Page 1 of 1 (latest)

fiery hatch
#

Bug Report:

Steps to reproduce:

  1. Paste in some long, AI generated content
  2. Click "Submit"
  3. See the result
  4. Click "Submit" again a few times
  5. See a DIFFERENT result for the same text

Find screenshots in this tweet. https://twitter.com/BooooQuant/status/1622033356836814848

This happens 99% of the time I use the classifier on very long form content.

Expected result:
Give me the right answer, the first time. Don't change your answer.

Specifically, this is most likely to happen when the result is first "very unlikely", it will then change to "likely". If it shows "very unlikely" initially, I have only ever seen it change to "likely" when you submit again, it doesn't change to any other classification. I've tested this legit, 1000+ times now, it's very consistent.

Actual result:

Find screenshots in this tweet. https://twitter.com/BooooQuant/status/1622033356836814848

Additional information

Browser: Chrome
OS: Windows 10

Boooo (@BooooQuant)

FYI @sama there's a bug with your classifier UI. Initially get "very unlikely", hit the button again after not doing ANYTHING to the content, literally just press the button again, it's "likely" AI generated.

Twitter ·
#

Here is a video showing the bug just now, again. Skip the first few black frames.

#

This is a significant problem, because like it or not people will rely on the classifier for real world consequences. The classifier is giving one answer initially, but when you submit again, it changes.

fiery hatch
#

The question is mainly; which answer is correct? The first "very unlikely" response it gives, or the consistent "likely" response after that?