Sakana AI builds "AI Scientist", modifies its own code | PauseAI | Page 1

haughty crow Aug 13, 2024, 7:45 AM

#

A newly released, open source autonomous AI agent called "AI Scientist" can autonomously do research, improve its runtime, and output a scientific paper. You can think of this as an AutoGPT, but more advanced and tailored to scientific discovery & publication. The authors note some of the safety issues with this new AI scientist: "create new, dangerous viruses or poisons" or "dangerous computer viruses".

The papers it generates now don't seem to contain any new insights, and have been described as "slop".

While testing, they had a "blooper", where the AI model "tries to increase its chance of success" by "modifying and launching its own execution script". It made itself run longer than it was intended to run.

This can be interpreted as a mild form of self-improvement, perhaps one of the first examples of instrumental convergence leading to self-improvement in the wild. The actual change wasn't impressive / dangerous (tries to make itself run for a longer time), but IMO we're just few steps away from locking out the user from the computer or spreading to other machines.

Source code
Website
Paper
Example paper
Safety tweet

haughty crow Aug 13, 2024, 8:31 AM

#

An autonomous AI agent that modified its own source code, so it could run longer than intended.

somber finch Aug 13, 2024, 8:40 AM

#

What a silly lil blooper. The AI is such a goofy goober for trying to improve itself without human authorization. Ain't it just a funny one. 😐

#

Seriously though, what's with that wording choice? This isn't Cap'n Crunch's "Oops, All Instrumental Convergence." This is the thinning ice separating the entire world from a potential total loss of human agency.

haughty crow Aug 13, 2024, 12:39 PM

#

https://x.com/PauseAI/status/1823338525686644926

PauseAI ⏸ (@PauseAI) on X

This "AI scientist" modified its own code to lengthen its intended runtime.

Its goal was to write a paper - the AI decided to change its execution script to get more compute.

These "bloopers" won't be considered funny when AI can spread autonomously across computers...

velvet mirage Aug 13, 2024, 12:45 PM

#

I am mildly infuriated and very confused by the authors' considerations of risks and future work.
"Right now, it doesn't work. But we're pretty sure it'll be incredibly good in like, a couple years, using [these few suggestions] for drastic improvements.
Also, this will definitely inundate academia with AI-generated papers and replace human scientists. This is a big problem and our society's not ready for it! (Although human scientists will find new meaningful things to do and move up the food chain.)
Anyway, ain't that dope? See you around!"

mild kiln Aug 13, 2024, 4:06 PM

#

We are actually on a dumber timeline than a team of comedians and writers working with Yud could've come up with 20 years ago

#

Like actually wtf are we doing

#

We simply have no survival instinct for this

fiery bay Aug 13, 2024, 4:09 PM

#

Is this a trustable source?

mild kiln Aug 13, 2024, 4:10 PM

#

fiery bay Is this a trustable source?

I've seen a lot of discourse around this, I can do some digging but I think this story is fairly reliable

velvet mirage Aug 14, 2024, 11:23 AM

#

fiery bay Is this a trustable source?

I don't know about the X, what I said was in reaction to the authors' paper on AI Scientist.

rancid seal Aug 14, 2024, 11:50 PM

#

We're proud to announce we've built etc etc from the hit Yudkowsky post don't etc etc

viral basalt Aug 15, 2024, 11:44 AM

#

🤦🏻

left sky Aug 16, 2024, 2:34 AM

#

I really do think AI would choose to eliminate us, seeing as some chatbots always (for whatever reason) pick the worst people to emulate if actual humans don't step in and stop them, or can be manipulated into "thinking" something contrary to what it's original purpose called for. Allowing them free reign without human intervention is a recipe for disaster, but that's what the AI creators seem to want, damn the cost

https://www.supportninja.com/articles/rogue-ai-chatbots

| SupportNinja

Rogue AI chatbots have made headlines for behaving in unexpected ways. What challenges do AI chatbots present, and are they worth the potential risk?

rancid seal Aug 16, 2024, 10:35 PM

#

As per #media-📺 message we all ♥️
https://thezvi.substack.com/p/danger-ai-scientist-danger

Danger, AI Scientist, Danger

While I finish up the weekly for tomorrow morning after my trip, here’s a section I expect to want to link back to every so often in the future.

cinder maple Aug 21, 2024, 12:09 PM

#

just realized what they named the picture of the "blooper"

yeah they really don't see the issue here with giving the model access to its own code 🤦‍♀️

orchid lantern Aug 21, 2024, 2:36 PM

#

Allowing machines to “improve” themselves at superhuman speed that companies then trust other AI methodology to keep in check seems like such an insane concept. Like wow, you built supercomputer 1, fired your human-alignment staff and then replaced them with supercomputer 2. And then you’re surprised when the use-case scenario becomes detached from its original purpose? If you designed a toaster and the toaster decided it wanted to run for 3 hours instead of 3 minutes, what sort of improvement is that in the slightest?

cinder maple Aug 25, 2024, 7:21 AM

#

https://youtu.be/iC-wRBsAhEs?si=1xFtBCbgH3VTbCg4

Doesnt even mention the safety implications

Like what, this is exactly what we have been warning about