general

[olive mesa]

thinking budget wow

[balmy mist]

what tests you ran with flash?

[olive mesa]

2.5 pro still better but 2.5 flash is so good for developers

[humble sonnet]

How does AI score calculation work?

[drifting elk]

Thank you brother

Absolutely awesome

Absolutely right

[drifting elk]

Python libraries, bruh

[humble sonnet]

Okay

[balmy mist]

are you a developer?

[drifting elk]

Engineering student, 4th year

[humble sonnet]

btw, the new update with images, can we request images and modify them?

[tall summit]

...what

[drifting elk]

Hahahahahaha

[zinc ore]

What y'all think about o4 mini getting 100% on one of the math benchmarks?

[balmy mist]

full report by logan:
https://x.com/OfficialLoganK/status/1912966497213038686

Logan Kilpatrick (@OfficialLoganK) on X

Gemini 2.5 Flash is here, our first unified reasoning model with thinking budgets. 🔥

It’s on the perato frontier and punches above its price and size!!

https://t.co/uO1DsPXlE8

X · Apr 17, 2025, 8:28 PM

technically flash should be the best all around model in terms of efficiency

but i wish it was faster man

that would be make undisputed best

[humble sonnet]

You cant ...?

[balmy mist]

its almost on par with o4 mini, but cheaper

[tall summit]

this is a dumb question, do reasoning models even reason in lmarena?

[raven void]

o3 and o4 mini is so good it leads me to think we'll get AGI achieved internally by December

[balmy mist]

yeah

[balmy mist]

what tests have you ran on them?

[balmy mist]

i aint gonna lie, my convos with o3 been smooth

[zinc ore]

https://x.com/OfficialLoganK/status/1912966500794654855

Logan Kilpatrick (@OfficialLoganK) on X

This is an early version of 2.5 Flash, but it already shows huge gains over 2.0 Flash.

You can fully turn off thinking if needed and use this model as a drop in replacement for 2.0 Flash.

It’s available across the Gemini API, AI Studio, Vertex, and the Gemini app!

X · Apr 17, 2025, 8:28 PM

[cedar tide]

Put 2.5 flashes in thinking off on the arena

[zinc ore]

Actually good point, anyone know how early into the cycle o4 mini is?

[balmy mist]

bro i just posted this lol

[tall summit]

well obviously there's no thinking... box like there is on any client actually meant for running models

[zinc ore]

You didn't post that specific tweet tho

[tall summit]

i mean, for a very topical example, is the 2.5 flash on lmarena reasoning or not?

[balmy mist]

its in teh comments

[zinc ore]

Yes

[balmy mist]

like right there

as soon as you click link

check the relevant ai news

[zinc ore]

I'm asking a specific question about something stated in that specific tweet

[balmy mist]

channel

[zinc ore]

#general message

[tall summit]

because it's not like i have a way to tell

[balmy mist]

idk what that means

maybe someone else does

[zinc ore]

I'm literally sharing the tweet to ask how early into prod cycle o4 mini is, compared with flash 2.5

[balmy mist]

anybody know?

[cedar tide]

The only benchmark we have for 2.5 Flash in Thinking Off
Compared to 2.0 Flash, it goes from 60 to 74 on GPQA and from 34 to 41 on LiveCodeBench.

[balmy mist]

lmaoo im not gonna start with you

[tall summit]

what

[cedar tide]

Gemini 2.5 flash vs grok 3 mini thinking

[tall summit]

sorry i dont know the intricacies of lmarena's internals and i want to know

[zinc ore]

That's gem 2 not 2.5

[balmy mist]

hmm now this is interesting

[zinc ore]

Gemini 2 Flash bro

[balmy mist]

oh okay, im not sure then

[cedar tide]

Its 2.5

[balmy mist]

i think someone might know here

[cedar tide]

I made this

[tall summit]

i dont get how you took it as an insult but im still sorry

[tall summit]

most likely

[zinc ore]

Oh okay, was labelled as Gemini 2 Flash, didn't realize

[balmy mist]

no not as an insult just trying to disagree for the hell of it like last time lol

cause i said yeah and then you said obviously no, but i misread

[tall summit]

ohhhh yeah i get how you might have thought that

well sorry i do it a lot im an indecisive person

[keen beacon]

u can tell if its a thinking model in arena battle, if one of the models is definitively not a reasoning model and both completions were delayed

[tall summit]

im like ehhhhhh these are good but ehhh these are bad
devils advocate sorta guy

for literally every choice ive ever made in my life

[keen beacon]

it delays both completions as it waits for the thinking model to reason

[cedar tide]

Fix

[zinc ore]

Gg

[tall summit]

guess its time to side by side

[zinc ore]

Show if it is single or multiple attempts

[tall summit]

i knew it does that but thats still quite clever

i like when it uses latex

except it doesnt display

man i hate that

so annoying honestly

[balmy mist]

is this true?
https://x.com/DaveShapi/status/1912971940597514289

David Shapiro ⏩ (@DaveShapi) on X

If you're not using o3 full at least two hours per day you're falling behind.

It sharpens your mind as a diamond file might sharpen a blade.

X · Apr 17, 2025, 8:50 PM

[torn mantle]

imma try it first

i still havent tried it xd

[tall summit]

anyway it seems to be the thinking version on the arena

[tall summit]

lmao what

[keen beacon]

didnt that guy say that o4 mini solved math or something lol

[tall summit]

what does that tweet even mean

[keen beacon]

ahahaha

[balmy mist]

but 2.5 flash is free tho, wait is grok mini free?

[cedar tide]

This price for the paid tier

[balmy mist]

i still say o3 is better but in different ways

like in terms of agi yes

[cedar tide]

We will see grok 3 thinking too

[tall summit]

i think it's o3

[keen beacon]

lmao xai 🤣

[balmy mist]

bro grok is not on o3 and 2.5 pro level

[tall summit]

based on o4-mini

[balmy mist]

anybody using o3 every day? is it making you smarter?

[cedar tide]

the final version with reasoning has not been released and we have zero benchmarks you can't say that

[tall summit]

how about o4-mini vs 2.5 flash

[balmy mist]

you a grok fanbaoy?

[tall summit]

i technically used o3 for all the 2 days it released

[cedar tide]

Nope

[balmy mist]

r u sure

[cedar tide]

But we will see

[tall summit]

and i feel the same amount of intelligent

[cedar tide]

Im a price fanboy

[tall summit]

oh well i wonder.

[cedar tide]

So Im loving gemini

[keen fulcrum]

Gemini 3 when

[patent aspen]

Is "o3 full" just the o3 model with maximum thinking time and resource consumption?

[cedar tide]

But i see today that grok 3 mini thinking has too a good quality for this price

[patent aspen]

Or is it distinguishing o3 from o3 mini?

[cedar tide]

o3 full its o3, its not 2 models

[keen beacon]

yeah most people do that

otherwise u would call it o3 full high (reasoning effort) i think

[cedar tide]

There are o3 mini and o3 and o3-pro

[tall summit]

i assume o3 full meant o3 with all the tools so it can do its stupid chain of thought bs

[patent aspen]

Isn't o3 full still way more expensive than 2.5 pro?

[balmy mist]

@torn mantle https://liveweave.com/A9OGzH# i cleaned this up a lil

[tall summit]

which is limited on platforms like lmarena

[cedar tide]

Yes, before the release of the gemini beast maybe

[brittle tiger]

on api yes but with plus subscription you get 50 a week of o3.

[keen beacon]

probably though its cheaper than o1

it depends on the task i guess

im curious to see the numbers when artifical analysis runs their benchmark suite

[patent aspen]

IMO Gemini 2.5 Pro is the best model in the world not because of raw performance but because it chose the right set of product and engineering trade-offs

[brittle tiger]

i def wouldn't count out grok. they don't have infra like goog but the whole team is basically former gdm and they have tons of gpus

[patent aspen]

o3 depends on expensive trade offs to be good

which isn't sustainable

[calm sequoia]

Winsurf turned out to be so good. Have anyone tried anything better or is it sota?

[keen beacon]

they are working on efficiency though since it is more efficient than o1 i believe

[patent aspen]

OAI loses like $8B a year now

[ocean vortex]

very early, I think it's 4.1-mini

[keen beacon]

i dont think openai loses money on api pricing with the current model set rn tho

[brittle tiger]

I really want to o3 on this

https://x.com/DillonUzar/status/1912671944459034834/

Dillon Uzar (@DillonUzar) on X

I've updated the MRCR results with the full results for GPT-4.1, o3-mini, and o4-mini, I also added in some Anthropic models.

No tests beyond 200k tokens are included for o3-mini, o4-mini, and Anthropic due to limitations.

X · Apr 17, 2025, 12:58 AM

[patent aspen]

Their APIs aren't profitable enough to cover their other costs

[keen beacon]

ya prob cohere idk lol

that line of anon names seem familar

[cedar tide]

Amazon

[keen beacon]

yea

[cedar tide]

Maybe nova pro 1.5

[keen beacon]

i forgot

[worthy thunder]

It's uh, rather pricey. I'll see about running it though (I ran those)

Also looking into running graphwalk on all of those models (another long context benchmark)

[keen beacon]

even with a much more efficient/much more profitable line of models, i doubt theyd be able to recoup costs on api pricing tho

its somewhat inherent in their strategy rn

select in direct chat if u wanna use o3

fyi this should be easy

its a thinking model with a distinct cut off

if it returns its made by openai, answers specific knowledge cut off questions right, delays both model completions

4.1 mini/o4 mini despite the claimed june 2024 cut off lacks a lot of more recent knowledge

[cedar tide]

Today we was at 224 or 223 ?

[balmy mist]

yeah im over flash

[zinc ore]

https://x.com/lmarena_ai/status/1912979323734425948?t=TyWExZCG-cl8BdEcaDFCPA&s=19

lmarena.ai (formerly lmsys.org) (@lmarena_ai) on X

Gemini 2.5 Flash ⚡ has landed in the WebDev Arena, and we can see that its strength in coding has impressed the community enough to land at #7 - solidly in the top 10!

X · Apr 17, 2025, 9:19 PM

[balmy mist]

back to o3 and o4

[keen beacon]

im still maining 2.5 pro lol

[balmy mist]

idk bro im starting to like o4

its just openai trash webui that makes me mad

[keen beacon]

it seems fine for me

nothing to really complain about in my experience

weird - they're still there

the branching feature in ai studio is extremely questionable though

[balmy mist]

it always crashed for me

but o4 is cooking for coding

flash makes bad code, like always issues

but it has good designs and ideas

[wheat onyx]

started testing o3. It's REALLY good. I had it look at the internals of my broken TV - it first looked for obvious issues, then told me specifically what I should do to test, what results i should get, etc. That's the first time AI would have been able to get detailed advice for something like this. That's a game changer

no idea if it's great for coding, but probably way too expensive for it?

[wheat onyx]

being acquired by OAI now, right?

[balmy mist]

o4: https://liveweave.com/A9OGzH#

plus flash

[wheat onyx]

what is this?

[balmy mist]

you cant see it?

[keen beacon]

worked after refresh

[wheat onyx]

i do, but what is it for

[balmy mist]

try this instead:
https://www.vibeshare.ai/c/PTfr5FWJ_q

but let me update it one more time

i want to make the animations be the default thing you see

[wheat onyx]

i know it looks like i cut off my snapshot, but this is actually what it looks like

you can scroll down on the right side to see more of the green iconbox

[balmy mist]

https://www.vibeshare.ai/c/kBaroVM2ln

yeah thats it

ignore the warnings

click the link again

and press play button

flash and pro have issues with large code

but o4 handles them nicely

[tall summit]

you sent it a picture?

[torn mantle]

nice

[tall summit]

why

[balmy mist]

thi

this*

[tall summit]

good to know regarding o4

[balmy mist]

for some reason with code larger than 1000 lines 2.5 has trouble making edits

i have seen this happen a lot

[balmy mist]

omgg i knee it lol

[tall summit]

maybe google's repl platform could actually work

[hardy pecan]

went from 223 to 225 - 2 models added just a few hours ago

[balmy mist]

send twitter link please

this could be gpt lol

jk

[cedar tide]

Thanks for confirming what I thought, but what is the second model? I haven't found it.

[tall summit]

this but unironically

[hardy pecan]

https://x.com/OfficialLoganK/status/1912968696265343424

Logan Kilpatrick (@OfficialLoganK) on X

@theLance Time for Gemini code

X · Apr 17, 2025, 8:37 PM

[torn mantle]

AAAAAAAAAAAAAAAAAAAAAAAAAH

PLS

but wen

😖

could it be on 22th of this month?

[hardy pecan]

I think llama nemotron lol or another useless model, not sure

[cedar tide]

🥴

[balmy mist]

idk i think im an openai fanboy now

[tall summit]

has anyone played chess with o3 yet

[balmy mist]

yeah i cant beat it

it cooks me

whats its elo?

[tall summit]

for some reason i tried 2.5 pro and it couldnt interpret a FEN

[balmy mist]

sama's plan

join the openai fanboy club

like me

[tall summit]

lol

i want to test o3 on the same

[balmy mist]

do a chess game with o3 vs 2.5

[tall summit]

this was in the context of asking to analyze a specific move

[balmy mist]

hmm i might build an app for that for every model to go against each other

[hardy pecan]

nevermind it was this model.. that no one knows about

[tall summit]

GLM

ALL MY HOMIES LOVE GLM

[cedar tide]

Nope he was here before

[tall summit]

hmmmmmmmmmmm

mmmmmmmm

2.5 pro takes ages

[hardy pecan]

oh yeah, my screenshot is cutoff, I cant see any other new model added in the top 16 other than 2.5 flash, so again itsw another inconsequential model added that scores poorly lol

[tall summit]

2.5 pro cant do ascii diagrams

[keen beacon]

oh wait it's you

[balmy mist]

o4 is king now

i used to love 2.5 but i think i might cheat on it with o4

[tall summit]

o4 mini against 2.5 flash would probably take less time

lmao

[keen beacon]

3.5 turbo instruct would destroy 2.5 and o3 (in chess) probably lol

[tall summit]

how about o3

[keen beacon]

im not even joking

[balmy mist]

i love o3 to, but i only get certain amount of chats with it, so I gotta save those lol

and o4 is better at coding

[balmy mist]

no way

chill on my side piece o4

[tall summit]

this is already looking much better for o3 LMAO

even sends me the fen

[keen beacon]

https://dynomight.net/chess/
https://dynomight.net/more-chess/ another reminder of these articles, its an extremely cool read btw

[tall summit]

never seen

will read

from nov 2024

i mean sure

[keen beacon]

its still relevant lol

does anyone know the system prompt lmarena uses for models?

it depends on the model but some of them are in the fast chat repo

they dont include everything there though, you'd have to extract them manually from lmarena

also see this: a recent tournament with gpt 3.5 turbo instruct and more recent models
https://dubesor.de/chessbeta/tournament2

(just an unnecessary random note but that guy blocked me for no reason, still confused to this day lmao)

[wheat onyx]

Yeah I gave it photos of the tv internals to analyze

[keen beacon]

that's weird lol

i've had fairly loud disagreements with him and he hasn't blocked me

[keen beacon]

yeah

[blazing rune]

Maybe he thought you were someone else from looking at your profile picture

your profile picture is quite generic

[keen beacon]

same he suddenly blocked me one day randomly we havent even talked in a while lol

[keen beacon]

nah i confronted him about it in the openrouter server me and another guy were talking about something and im pretty sure he saw my message

he seemed a little upset that he was being ignored

[balmy mist]

nvm o4 and o3 is ehh at coding large codebase 🙂

back to gemin

[quiet pollen]

What is the difference between Gemini and Gemma?

Other then reasoning

Exactly my thoughts

[keen beacon]

the gemma line are the open weighted models

[balmy mist]

gemini is the dad, gemma is the mom

[keen beacon]

gemini are the frontier/closed source ones

[quiet pollen]

The question is o4 or Gemini

[keen beacon]

ig u mean o4 mini

prob worth testing all of the models on ur specific tasks and see which one u like

[quiet pollen]

Gemini seems to be sticking out for me

[tall summit]

what does a temperature of 2 mean

i remember when temperature couldnt go above 1!!!

what is 1.5 creativity

whys it max 2

[upper wolf]

you guys use a temp of <0.6 with gemini??

[tall summit]

yes

[blazing rune]

I usually use 0.5, it seems to work better than 1, at least for coding

for gemini 2.5 pro

gemini 2.0 flash I usually use at 0 or 0.5, since it hallucinates so much

[keen beacon]

i personally use 0 for mostly everything, 1 for everything else (poll was about creative but coherent tasks/etc)

1 because i dont bother changing the default ai studio value 🤣

[sweet tinsel]

I'm maybe late but 2.5 flash has dropped for me in the Gemini App replacing 2.0 Flash Extended Thinking which I had until now.

[tall summit]

great news its out etc etc

same limits in ai studio as 2.5 pro

[tawdry phoenix]

Literally going insane

[glass arch]

as a wise man once said:
"you must pick between autism and schizophrenia"

(this is about AI models btw)

it seems that openai does not tell the models what its name is

[tawdry phoenix]

Roblox injector lol

[tall summit]

roblox injector

can ai do that

well first of all would

probably not

[glass arch]

which AI can make the most soulless slop

[olive mesa]

2.5 flash hallucinates a lot

[olive mesa]

also gogle

google

i asked 2.5 flash what its name was via api

it says its just an ai

[balmy mist]

Hi Daniel

[glass arch]

it's so sad

like it enters the world and does not have its own identity

[late path]

is nightwhisper/dragontail 2.5 flash?

[hardy violet]

I gave it a quick try myself – not coding, of course – but my conclusion is the same as the friend above. It seems like that hypothetical '2.5flash-thinking-high' model doesn't really exist. Cranking the 'thinking budget' parameter to the max just seems to make things slower and the interface laggier, with no actual improvement. At least, not in my initial test scenario.

[keen beacon]

thinking budget seems to just cut off the thinking prematurely. openai reasoning efforts are tuned to produce different lengths of chain of thought i think. so the current 2.5 pro and 2.5 flash are basically always at the "high" setting no matter what

if you have thinking budget disabled

with claude's thinking budget its a little more involved than cutting off i think, they seemingly tuned several antml:max_thinking_length</antml:max_thinking_length>, so its closer to openai's

[ember rapids]

no

thats probably coming next week

[zinc ore]

That's what I think too. One of them is updated 2.5 pro, the other is a coding model.

[late path]

Yes, I've read this before. I think the final reason is quite obvious. As a completion model, gpt-3.5-turbo-Instruct formed some internal neural network structures because its training corpus contained chess game data. This structure represents a kind of "mechanical" learning, akin to training a specialized neural network dedicated to playing chess, rather than indicating the model has any awareness that it is "playing chess".
And as mentioned in the second blog post, when the model transitions from a pure completion model to a conversational one, this highly specialized completion ability becomes very fragile and fails to generalize effectively. The special tokens and separators inserted for role marking in conversational models might act as interference, potentially preventing the original "chess-playing circuit" from being reliably activated. Therefore, the decline in the conversational model's chess ability confirms that gpt-3.5-turbo-instruct's chess skill stems purely from its completion mode, rather than indicating it possesses any magical advantage over other models.

[hardy violet]

I just tested these models on a short text for comprehension. Like I said before, o3's approach is simply intolerable. The framework it builds is ridiculous – that way it sounds plausible but is actually wrong (specious), just does its own thing, grabs onto one word and elaborates endlessly, or makes random, awkward analogies everywhere... it's totally on par with DeepSeek R1.
For tasks like text interpretation or writing, honestly, anyone who likes o3 should probably just use DeepSeek – it's more cost-effective.
Gemini's comprehension felt spot-on. Seeing Flash's performance, I thought it was pretty much perfect. But then looking at Pro, there's still a noticeable gap.
This gap [between Flash and Pro] is clearly wider than the difference between o3 and o4mini high. Considering cost and speed, Flash might have its value, but from a practical and progress standpoint, my vote goes to o4mini-high.

[keen beacon]

yeah. if we had access to newer base models of frontier models theyll perform much better than gpt 3.5 turbo instruct/a similarly trained model that didn't lose its chess skills

the problem is we don't have access to any of them, so it is magical in a sense (since we have no comparison points) but i wasnt ever implying that gpt 3.5 turbo instruct is one of a kind if you consider other internal frontier base models ( og gpt 4 base model would be more than enough anyway) . we only have access to the reasoning tuned/instruct/etc. versions

[balmy mist]

so whats teh best setting, 1 thinking or auto?

[keen beacon]

disable thinking budget

[tall summit]

i can't believe there are people who think 100k+ token context is minimum useful baseline

[tall summit]

can you give more context regarding what exactly you... did?

[olive mesa]

kind of

but at the same time it's just an algorithm really good at repeatedly predicting the next token

i honestly think the closest thing we can get to consciousness is an ai with really good cot being aware of its own thoughts, aware of it being aware of its own thoughts, and hallucinating emotions

[keen beacon]

under an ai fiefdom you say that to your ai overlords, you're just predicting tokens 🤣 imagine lol

[olive mesa]

lmao

[drifting thorn]

It is for agentic purposes

[still jetty]

suggestings for an image to image model? a family member wants to take photos of their yard and generate landscape ideas

[keen beacon]

can't gpt 4o image gen do that?

[still jetty]

i'll take a look. ive never tried/had a need for image to image yet so i dont know which models do

[raven void]

it depends on the kind of task but if 100k is not a minimum to expect then what is

[leaden meteor]

Do we have any reliable ones yet? I tried one to give me ideas for a fence in backyard. 4o was also terrible. I think these things are still not good enough to edit existing images. They are only trained for creating new images from scratch....

or the vanilla 'professional passport photo' schtick...

Giving landscape ideas with realistic edits of your backyard will be gamechanging....

[keen beacon]

Iirc they did train in editing images at least with gpt 4o image gen

[keen beacon]

I guess this is too hard for now

It might also not be passing enough info to the gpt 4o image gen model

Since theyre different models for now

Chatgpt calls a tool to a different 4o model with image gen enabled

Too finicky to deploy one together for now I guess

[barren prairie]

Flash and have those limits ? 🙂🤨 I didn t hit it but really????!!!!
And on the app?? Do we have the same issue??

[zinc ore]

https://x.com/bongrandp/status/1913040244963766432

Pierre Bongrand (@bongrandp) on X

For the first time, Google is responding to OpenAI's announcement in < 24 hours

The WAR is officially ON, and Google wants the LLM market

Google is now dominating +90% of the price share

X · Apr 18, 2025, 1:21 AM

[leaden palm]

where grok 3 mini

oh, it's right there

(not to mention that the x axis isn't adjusted for thinking tokens)

actually the cons@64 is doing a lot of the heavy lifting, but even without, it's a better model than gemini 2.5 flash

[upper wolf]

have you guys tried 2.5 flash with thinking turned off? if so what is your experience

[zinc ore]

Grok 3 Reasoning Beta and Grok 3 mini Reasoning’s scores for AIME 2025 at “@1” — meaning the first score the models got on the benchmark — fall below o3-mini-high’s score. Grok 3 Reasoning Beta also trails ever so slightly behind OpenAI’s o1 model set to “medium” computing.

https://techcrunch.com/2025/02/22/did-xai-lie-about-grok-3s-benchmarks/

Cons64 boosts the scores quite a bit, it's not reliable to compare with @1 passthroughs.

[thorny drum]

december o3 biggest offender of this lol

but tbf they were pretty transparent of how hard they juiced that model for benchmarks

[zinc ore]

Yeah agreed, openAI did it and Grok also did it. I'm hoping in the future these companies cut it out.

[alpine coral]

i'm confused by the december o3 reference.. do you mean that arc-agi chart, with o3-high in the top right corner (costing >$1000/task)? seems kinda different to this ( @conensus )

[thorny drum]

i think their approach to codeforces was comparable?
idk

but my understanding was o3 high was effectively cons1000 or whatever

[plain zinc]

I WANT

[keen beacon]

pro is only like 300-400B

[plain zinc]

LETS GOOOOO

It's confirmed

[real totem]

Aint no way

Ultra

[keen beacon]

no way

maybe in the next couple months

[torn mantle]

next batch releases are gemini coders

[plain zinc]

Why?

[real totem]

Cuz

Its gonna be

To good

[ember rapids]

Honestly with the pace theyre moving at

i wouldnt be surprised if we got it say 2 months from now

[keen beacon]

The world knowledge 2.5 pro has is incredible, I would really like a 2.5 ultra if it's done well and pricing isn't ridiculous

[novel flame]

I tried running my usual mini benchmarks on 2.5 Flash and got the same perfect 5/5 score that all the top models get (from 4o and up). So far, so good.

Then I gave it my harder “zero-shot browser game” task and it did something unexpected; it didn’t do it. It didn’t fail, rather it replied that such a task was too big for a single prompt, explained why, gave a detailed plan for how you would do it in steps, and then implemented a working sandbox to build on.

So now my scoring system doesn’t work because it’s based on implemented features and Flash deliberately didn’t implement any features.

The thing is, no other model has done this. Many models will provide an incomplete game and list some improvements to make, but 2.5 Flash pushed back on the premise itself, which I don’t think I’ve seen before.

So I don’t know how to evaluate this for coding…… 🤨

At the same time, c’mon Flash, what the hell? The original Super Mario was a total of 20KB, which is maybe 16k tokens, so why not give it a whirl?

And on the subject of coding, has anyone else had the experience that o4-mini generates minified JS code?🥸🤪 It is the weirdest thing. I mean, if it’s trained on random JavaScript from the public web then obviously most of that is minified, but surely a bare minimum of human preference tuning would have taught it not to produce minified source code?!!???

[keen beacon]

O4 mini in particular can be very weird it might be an artifact of the rl

[keen fulcrum]

So gemini 3 will come during summer right?

[keen beacon]

for me o3 > 2.5 but very small margin

o3 strength is not in question answers, is in its ability to use tools, run python code , search etc , compliting multi step tasks

[torn mantle]

you can enable grounding ( search ) on gemini 2.5 pro?

you have it all

[cedar tide]

https://fixupx.com/ArtificialAnlys/status/1913057626117820438?t=eDD2-6fuPdTFRq7safK1DQ&s=19

Artificial Analysis (@ArtificialAnlys)

Grok 3 mini Reasoning’s recently launched API has highly compelling Intelligence vs. Price positioning
︀︀
︀︀@xai has recently launched APIs for Grok 3 and Grok 3 mini, after initially only making the models available via the Grok chat interface at launch. We have now completed benchmarking for both Grok 3 and Grok 3 mini.
︀︀
︀︀Grok 3 mini (high reasoning) stands out in the top left of our Intelligence vs Price chart, achieving one of the highest Artificial Analysis Intelligence Index scores ever, with pricing well below even DeepSeek R1.
︀︀
︀︀Grok 3 family overview: We’re initiating coverage of 6 versions of the Grok 3 family: Grok 3, Grok 3 Fast, Grok 3 mini (low reasoning), Grok 3 mini Fast (low reasoning), Grok 3 mini (high reasoning) and Grok 3 mini Fast (high reasoning). Today’s post focuses on our intelligence results; performance breakdown of all 6 versions will be available soon.
︀︀
︀︀Intelligence: As we highlighted at la…

FixupX · Apr 18, 2025, 2:30 AM

[keen beacon]

o3 does it better, seen it work 10-20min just for 1 answer

[keen fulcrum]

Gemini 2.5 Ultra lol

[fleet lintel]

that's actually quite smart to offer a plan if it can't do it, compared to just do something incomplete.

[fleet lintel]

too cryptic, it could mean anything.

[novel flame]

I agree, I mean that’s the answer I’d expect from a human if I gave them such a vague task with an unrealistic scope. But it makes it impossible to quantify its coding skill/usefulness against the others (including 2.5 Pro) on that test

[fleet lintel]

true

[plain zinc]

https://x.com/rakyll/status/1913099386793636162

Jaana Dogan ヤナ ドガン (@rakyll) on X

Time for Gemini Code 💎

X · Apr 18, 2025, 5:16 AM

👀

[hardy violet]

For some lyrics translated from Japanese to Chinese, we can directly tell if the Chinese version flows well and spot translations that are significantly off compared to human standards. Also, with essays I wrote myself – even though they aren't great – I know exactly what I was trying to say. This is where I noticed the issue with models like R1 and O3: they tend to misuse structures and over-elaborate. What's more, the writing style became noticeably 'R1-like,' and other users I talked to agreed. Since I'm not willing to pay for the API, and LM Arena can't handle testing slightly longer short stories, I didn't get around to testing how it interprets classic texts. (Plus, testing classics is easier to get away with; people rarely criticize over-interpreting classic works anyway）

[keen beacon]

today maybe?

[keen beacon]

another one!

[fleet lintel]

any good?

[fleet lintel]

thye might be talking about coding using gemini 2.5 flash

[cedar tide]

Livebench

[torn mantle]

woah

[cedar tide]

Its already sure that we will have coder model https://x.com/legit_api/status/1912888062138347773?t=sflUSnSaHTCa6DmVNGfZ5g&s=19

ʟᴇɢɪᴛ (@legit_api) on X

WE ALSO SEEM TO BE GETTING CODE MODELS!

WE MIGHT SEE THE REVEAL OF NIGHTWHISPER

X · Apr 17, 2025, 3:17 PM

[cedar tide]

Its new version of gemini 2.5 pro no ?

[keen beacon]

we don't know yet

it's just a hidden category on the model selector

[dusky aurora]

I have feedback abut beta interface.

In direct chat, there are no sampling options

[keen beacon]

#new-ui-feedback

[barren prairie]

Go to iu feedback

[keen beacon]

lol, 3.7 sonnet beat by a flash model

[cedar tide]

Anyone have hard prompt that 2.5 pro dont get ?

[keen fulcrum]

Its the stable
after preview -> stable

[keen beacon]

what

im talking about the category on AI studio's backend referred to as "1P CODE MODELS"

(first party code models)

[keen fulcrum]

Interesting to have special code models
Although I want them to have their own debug capabilities, I don't want tools

[novel flame]

Or… 1st place (on benchmarks) coding models?

[scarlet flint]

Hello

did google remove the image editing model from aistudio?

[ocean vortex]

Most of them don't even know why they would need it. If you aren't inputting videos, 99% of the time you don't need it nor it even provides a better experience. For an average convo that is already beyond the point of the model losing track of what exactly happened in the beginning of it regardless of the total context size

reasoning models are a different thing, but that only applies if you are saving reasoning into context (currently not possible with openai)

[torn mantle]

is this 2.5 pro?

[ocean vortex]

on that last point... I think this actually might be one of the reasons why openai has longer chains than anyone else, since they don't worry about this flooding the context lol

[torn mantle]

its kinda slow

[torn mantle]

seems like its a better version of gemini 2.5 pro thinking

recent checkpoint?

[cedar tide]

why 2.5 flash without Reasoning does not replace gemini 2.0 flash on gemini?

[keen beacon]

where do you find this

[tall summit]

i agree

[kind cloud]

random battle

[unborn ocean]

They are probably still working out the inference to some degree and its also supposed to be experimental for now

[willow grail]

someoneneeds to make a website which takes in all benchmarkes and creates a average

[sweet tinsel]

For me it replaced 2.0 Flash Thinking in the Gemini App

Atleast

[cedar tide]

Yes but im speaking about the non thinking version

[sweet tinsel]

But like Not so said it's most likely Multimodal feature and inference bount

[ocean vortex]

"experimental" is just a name that does not mean much lately

[barren prairie]

Most of google models are experimental... Rarely stable

We are mice and rabbits for google

[unborn ocean]

Well they use the time to optimize inference mainly. But I get your point about them basically being normal models

[calm sequoia]

Anyone checked what is claybrook?

[torn mantle]

google

i cant really tell if its better than gemini 2.5 pro thinking or not

[calm sequoia]

There's nothing else to cook for them, what else could this be 😄

Probably adapted to some niche application

Or flash with thinking at some intensity setting

[ocean vortex]

coding model?

[torn mantle]

no

it does give in-depth details

a bit more than 2.5 pro 03

[calm sequoia]

I thouth nightwhishperrer was already that

[torn mantle]

yea i think it may be better than gemini 2.5 pro 03

from my initial tests

[calm sequoia]

Full version, update or, maybe, Ultra 👀

Allgirht then, lets test it.

[torn mantle]

this model

is kinda different from other google releases

i like the formatting

it also goes into many points

[tall summit]

oh you were still talking about translation. thanks for the info even if you weren't talking to me specifically earlier i personally appreciate the results

[torn mantle]

its better at explaining things too

[calm sequoia]

Could this be the LearnLM

[torn mantle]

its really really good

at general knowledge

it doesnt give you the typical generic answers

[calm sequoia]

Okey then, it's update, full or ultra.

[torn mantle]

woah

its actually good

i cant believe this

[torn mantle]

its kinda slow

i wont be surprised if its ultra

i didnt try it on reasoning tasks tbh

[cedar tide]

Can anyone send the "claybrook" this prompt ?

creates a 99% copy of the discord front end, in a single html file, (without the backend)

[hardy pecan]

Claybrook confirmed quite good, defintely 2.5 pro, dragontail level for sure

[torn mantle]

ok

[hardy pecan]

It's funny, google probably reads this chat and we figure out what kinda model they JUST released super fast lol

[cedar tide]

Claybrook is on webdevarena too

[torn mantle]

since when

i just checked like 10 min ago

[cedar tide]

I just see it now

[torn mantle]

interesting

two new models added

[plain zinc]

Two?

[torn mantle]

yea

[plain zinc]

Which?

[torn mantle]

dayhush and claybrook

both with a high priority ( 12 ) so you should get them quite often

[plain zinc]

Who seems stronger?

[torn mantle]

i will try them first

google are going crazy

new model per week

[plain zinc]

Which model seems to be pro?

Well, sort of... which model is slow?

I read it here somewhere.

[hardy pecan]

claybook is pro level for sure

[torn mantle]

yea

[calm sequoia]

No

[torn mantle]

claybrook

[plain zinc]

Really good ✨

Interesting... Are they better than nightwhisper?

[torn mantle]

wait

dayhush

this model may be gemini code or smth

[calm sequoia]

This prompt can be solved by the latest generation models, including 2.5 PRO, o3 and o4-mini. It can't be solved by 2.5 Flash, 4o and others. It could not be solved by the claybrook; therefore, it's not general purpose model, or worse at physics than general 2.5 PRO. The prompt: "You take two carabiners and wrap a piece of high strength polyester webbing between them in 10 turns while there's as 30cm distance between carabiners. Webbing is conventional 25mm width strap used in industry. The on end is attached rigidly to the carabiner, the other is left free. How much time would such configuration hold until all the webbing would unroll itself if the applied tension from one carabiner to another is 100kg. Accidentally one side of the webbing was left hanging without attachment to anything - it is free to unroll. Answer only in how much time, no explanations."

[cedar tide]

What the response ?

[real totem]

Is dahush
In normal arena

[hardy pecan]

I haven't got dayhush yet

[plain zinc]

Bro, how many times have you given this promo for models? At least it should be given three times because of the temperature of 1

[torn mantle]

no

[hardy pecan]

in normal

[calm sequoia]

Correct response is "It will not unroll"

[calm sequoia]

Other - multiple times, the claybrook only once. But all other models did not make any mistakes even once.

[torn mantle]

whats claybrook response?

[calm sequoia]

Some seconds

The same as 4.1 or lower level models. Even the o3-mini can do it right.

[torn mantle]

yea i think dayhush may be gemini coding model

[hardy pecan]

https://alexchen.dev/

LOL

its a real guy

[torn mantle]

lmao

what

[hardy pecan]

hes been contaminated

[cedar tide]

I just see that 2.5 flash in the leaderboard

[torn mantle]

actually thats a thing i noticed about nightwhisper too

it gives you real references and doesnt hallucinates much

could be a thing in their coding models

[plain zinc]

And how does claybrook encode?

I mean, how does claybrook write the code

Better, worse or much better or much worse?

[torn mantle]

you mean writing-wise?

still didnt get it yet on webdev

[plain zinc]

Nah, coding

[torn mantle]

so far im only getting dayhush

seems kinda similar to nightwhisper

[calm sequoia]

Inferior even to the o3-mini 💩

[keen beacon]

is dayhush only on webdev

[torn mantle]

yea

[calm sequoia]

Yeah, I've concluded that the o3 is the king of the next generation (for my tests). Still, I use 2.5 Pro for coding.

[brittle tiger]

https://x.com/slow_developer/status/1913175292006580688?t=ED94PaxdnFtvRSlstNQgtA&s=19

Haider. (@slow_developer) on X

ok, now this is interesting

LiveBench puts gemini 2.5 flash above 2.5 pro in coding

how's your experience?

X · Apr 18, 2025, 10:18 AM

[brittle tiger]

Day night
Hush whisper

[real totem]

They're shet

[calm sequoia]

Need gigantic context length

[plain zinc]

Bruh💀. Where does skibidi come from?..

[calm sequoia]

It's for those who want to vote but dohnt know nothing

[cedar tide]

Which is other ?

[torn mantle]

yea they seem kinda similar

though i cant decide yet which one is better

[brittle tiger]

There's no way they're not related given names. Is nw still best outputs?

[keen ferry]

wtf

[keen fulcrum]

https://www.rxddit.com/r/Bard/comments/1k1v8vd/a_surprising_reason_why_gemini_25s_thinking/

[hardy pecan]

Variance

[torn mantle]

im trying this prompt now

holy

dayhush

the magnetic field needs a bit of adjustment tho

[keen beacon]

woah what

that's a cool design

[torn mantle]

gpt4.1

[keen beacon]

this is probably a new checkpoint for nightwhisper

[torn mantle]

kinda cool too but its not animated with particles

[keen beacon]

the names are similar

[torn mantle]

yea

could be

[torn mantle]

this is actually so cool

[plain zinc]

Wow, claybrook better gemini 2.5 pro in web design

[keen beacon]

dayhush is def better than claybrook

claybrook to me felt meh

it flopped on Qs 2.5 pro can get right

[plain zinc]

But it's enough that claybrook is better than 2.5 pro.

[keen beacon]

not in my experience

🤷‍♂️

[novel flame]

Looks like dayhush is nightwhisper, exciting times for coding models! Next cpt: dawnshush? No - DUSKMUMBLE!

[keen beacon]

it won't be nightwhisper

it'll be an updated ver

it's probably just 2.5 with a focus on code

[novel flame]

Yes of course but same model

[torn mantle]

it didnt feel solid at coding tbh

[tall summit]

hes going to the Flosstradamus rave tomorrow

[hardy pecan]

2.5 pro with 128 code experts!!! Or sumthin

[torn mantle]

i think you are right

its probably an even better version than nightwhisper

[tall summit]

i mean this is nice

[tall summit]

people are bashing them in the comments

[novel flame]

“Just 2.5 with a focus on code” though… that could finally dethrone 3.7 Sonnet for real world coding.

The current state of coding models (warning: possibly hot take): 3.7 Sonnet is still the king, 2.5 Pro is very close, and o3 is sometimes better but expensive and not reliable/consistent. o4 mini is sometimes on par with or slightly better than 2.5 Pro but again, too inconsistent (like generating minified code) to rely on for everything. I haven’t put Grok 3 Mini fully through its paces recently so I’m going to do that later today, it may be in there somewhere too.

But a coding focused Gemini 2.5 variant? I see no reason why it wouldn’t beat Sonnet.

[balmy mist]

https://3000-ii7v95x3mngfpizk6honw-65401bad.e2b-foxtrot.dev

[tall summit]

i keep forgetting webdev arena even exists

[balmy mist]

dayhush is phenomenal

i wonder how long it stays

[sage raptor]

dayhush is insane

[balmy mist]

and if they will actually release it

[torn mantle]

yep

[torn mantle]

dayhush?

what was the prompt

[novel flame]

I suspect part of the recent Gemini magic is the ‘Entropy’ logit search method, and part of the reason why they’re training models so rapidly is Titans. I’m on my phone so I don’t have the papers here but DeepMind has seriously cooked under the hood.

[torn mantle]

did you tell it how the characters should face eo?

[balmy mist]

create pokemon game

[torn mantle]

thats it?

[balmy mist]

yup

[torn mantle]

nah

thats a first

[tall summit]

i am disheartened that i've no idea what you are talking about, but never fear! research

[torn mantle]

you see how the characters are facing eo?

[balmy mist]

yeah man very clean

and the run feature, and animations

og nightwhisper didnt even get that without extra prompting

[hardy pecan]

https://3000-izzpt6s87uo0372wrmkxh-0b3529ea.e2b-foxtrot.dev

This was "create Pokemon game"

By day hush

So... Not quite the same result as above

[torn mantle]

doenst look good

[tall summit]

lmao ok dayhush is good

oh it competed with llama 4 maverick

[keen beacon]

0-shot

did a lot better than 2.5 pro

[balmy mist]

damn you sure? weird how we can have different results

[sage raptor]

nw was able to do that too

[balmy mist]

prompt: my friends birthday was yesterday and i missed it, make me something to give Andrew
https://3000-igg96j3ix44l5qgzr6uwq-ec17a5a5.e2b-foxtrot.dev

[keen beacon]

well yes I never said it couldn't

but imo these outputs have been slightly better than nw

[sage raptor]

yep

[tall summit]

i was so confused for about 2 seconds

[balmy mist]

this was the exact prompt for pokemon, i added a ' by mistake lol, maybe that makes a difference?

[hardy pecan]

Yes very sure lol, not much to get wrong

[brittle tiger]

https://3000-ixz0h6ax00b2ls83bhvlo-d3b1dbce.e2b-foxtrot.dev/

Dayhush
prompt: Create a visually stunning, interactive web visualization of bioluminescent deep-sea organisms responding to subtle environmental cues (like simulated currents or light sources triggered by user interaction). Focus on ethereal beauty, fluid animation, and a sense of discovery. Amaze me with the organic feel.

[balmy mist]

trying doing this prompt exactly:
create a pokemon game'

[novel flame]

Entropy: https://timkellogg.me/blog/2024/10/10/entropix and ~~https://arxiv.org/html/2407.01082v2~~ https://arxiv.org/pdf/2402.10200

[tall summit]

oh thanks so much

[novel flame]

Titans: https://www.llmwatch.com/p/google-releases-transformer-20

[hardy pecan]

I'll try that

[torn mantle]

claybrook isnt that good at coding

thats for sure

[novel flame]

Also for coding agents, I recommend checking out the paper on ‘Moatless Tree Search’ gaining 20 percentage points on SWE-Bench through search: https://arxiv.org/abs/2410.20285 — showing how you can replicate the performance of much bigger models by taking a really small model and letting it cook with search. This is why I suspect a lot of the top Reasoning models are actually using far smaller models than the community seems to think.

[hardy pecan]

Dayhush vs Gemini 2.5 pro

[torn mantle]

nice

[ocean vortex]

benchmark issue

[balmy mist]

i hope the release it soon

[ocean vortex]

this would be normal with openai since mini does indeed perform better on some tasks

but not for gemini lol

it's literally behind in their every single measured metric

[tall summit]

funny prompt

[balmy mist]

dayhush pinball game: https://3000-iroe1qqj1b1pwvwlb0ny7-9c4a2736.e2b-foxtrot.dev

[brittle tiger]

yeah for sure. flash getting better than 2.5 pro basically confirms their coding category isn't great

[cloud meadow]

Google is secretly using atom of thoughts (trust!)

[balmy mist]

almost, but a very hard prompt

[blazing coyote]

I fell like Nightwhisper was better in design than Dayhush

[torn mantle]

didnt seem much different from gemini 2.5 pro

03

this is so confusing

[balmy mist]

this is nice

[torn mantle]

maybe

[balmy mist]

this is o3 or 2.5 pro?

[torn mantle]

im lost xd

[torn mantle]

2.5 pro

[balmy mist]

what prompt?

same exact? 0-shot?

[torn mantle]

yes

create a beautiful looking pokemon game battle mode with pokeapi, with an extremely pleasing design, wow me.

[balmy mist]

not exact lol

[ocean vortex]

try web design or svg, should be less confusing then

[balmy mist]

let me try that

right now its generating a super mario game, lets see how it does on that then i will try that updated prompt, but im still shocked that 2.5 pro could do that

it was not able to do that before

you think they updated it?

[ocean vortex]

2.5 pro is a fairly big model, it's gonna do well on visuals

[balmy mist]

https://3000-iihadi240nvhk3fjc9zoe-ed96e18a.e2b-foxtrot.dev

[ocean vortex]

unlike all the minis and stuff

[real coral]

I'm quite liking the current 4o. In the prompts I care about it's pretty much on par with 2.5. Has a nice style to it.

[balmy mist]

idk bro it seems diff, before it didnt do well visuals like that when testing with pokemon, i had to really prompt it and guide it

[balmy mist]

okay mario game, let me tell it to make it guide it a lil

[torn mantle]

@balmy mist https://3000-iibslowa8wbqm3106z5gm-337d29f2.e2b-foxtrot.dev

can you try this?

[balmy mist]

wow

thats nice

prompt? and 0-shot?

[torn mantle]

yes

[brittle tiger]

all this dayhush nightwhisper talk but i'm going to morning scream if i get webdev arena bug where a side doesnt load onemore time

[calm sequoia]

People who voted Gemini or "Other" - what prompts are you using?

[blazing coyote]

Claybrook also appers in the webarena

[balmy mist]

wow they fixed webdev arena

[keen beacon]

yeah the webdev arena implementation kinda sucks..

[balmy mist]

now your creations are persistent?

[keen beacon]

sometimes it just breaks and ends the round if you try and follow-up

[torn mantle]

no

still the same issues

[balmy mist]

i just reloaded the page and i still can load the creation

like this one:
https://3000-itjougubgpyvisl9zdlw0-2266d380.e2b-foxtrot.dev/

but i started a new round and currently generating something

yall can access it?

[torn mantle]

yea

[balmy mist]

hmm i wonder how long it lasts

better than before when it would timeout fast lol

i guess they slowly working on the website

still gotta fix the glitching tho, i wonder if the results are actually fair in the leaderboard

cause multiple times things would not load for me or even gen any code

and it was the better model but i picked the other one since it generated code

[torn mantle]

yea i had the same issue

they still didnt fix it

dayhush makes some questionable choices

nightwhisper felt more reliable tbh

it will rarely miss

[alpine coral]

there are no anon models in the beta arena right?

[cedar tide]

Yes

[alpine coral]

i phrased that poorly lol

i haven't encountered anon models in the beta arena before, is that still the case (they're not in there?)

[tall summit]

the ones in beta arena are only the ones also in beta direct chat

[brittle tiger]

jinxed myself so hard here. have gotten this bug maybe 6 times in a row. anyone have best practices for avoiding it in webdev arena?

[tall summit]

it happened to me so many times today

i hope there is a way to avoid it

[brittle tiger]

is there a way to at least figure out the name of model that had an output without voting?

[calm sequoia]

no 🙂

[narrow elbow]

whats dayhush? google new model?

[worthy thunder]

That's a pretty cool output

[narrow elbow]

https://3000-ief2zmf8bfinbq4n4yrsz-884333c4.e2b-foxtrot.dev/

[barren prairie]

Is it as good as NW?

[narrow elbow]

i dont know..

[novel flame]

Night/Day, Whisper/Hush. It’s got to be a newer checkpoint of the same model

[balmy mist]

day and night

[worthy thunder]

I ran the OpenAI-MRCR benchmark for Gemini 2.5 Flash (Auto Thinking and Non-Thinking). Interesting curve for Gemini 2.5 Flash Non-Thinking. Meanwhile Gemini 2.5 Flash Thinking (Auto) matches Gemini 2.5 Pro. Wanted to be able to compare what the perf was like with thinking on vs off, which there does appear to be a difference, with some odd curve. That curve is consistent over all of the runs of the full benchmark for Flash Non-thinking.

For other results, you can check my thread: https://x.com/DillonUzar/status/1913208873206362271

I'm still working on o3 access and will update when I have access. Some error on OpenAI's side, they're looking into it for my org account. If anyone has access in the meantime and is willing to coordinate, let me know. If anyone wants other models, just let me know which and I'll see what I can budget and run.
Still working on setting up a website to allow everyone to view individual run results (like on matharena.ai), and also setting up Graphwalks (another long context benchmark from OpenAI)

[worthy thunder]

Was the sandbox destroyed?

[narrow elbow]

yea

[narrow elbow]

Maybe webdev was automatically destroyed within a certain period of time, and I didn't close it.

[torn mantle]

yea

[calm sequoia]

Oh man your graphs are always very interesting, but you must make the color lines thicker and please, rank the table according to results 😄

[torn mantle]

Its on the same level

[worthy thunder]

Will do, thanks for the suggestions 🙂

[plain zinc]

... And indeed... If we compare the sky with the earth, then we know that the sky is better than the earth, but the day and night...

Which is better?

[torn mantle]

1. NW = DH
2. Dragontail
3. Gemini 2.5 pro 03
4. Claybrook
5. Gemini 2.5 flash thinking

[brittle tiger]

pretty good dayhush output for complex task
https://3000-ilb9lm2u2urwebu9sl55g-a7c98fd5.e2b-foxtrot.dev/

prompt: Create a captivating, interactive web demo visualizing fluid dynamics, such as laminar flow breaking into turbulence or convection currents. Use particle systems or shader-based techniques (like reaction-diffusion) to create organic, flowing patterns. Allow users to introduce 'heat' sources, obstacles, or change flow velocity to see the beautiful complexity emerge. Focus on the mesmerizing, almost alive patterns.

[plain zinc]

But no._.. Nightwhisper was much better

[glass arch]

can you do this in the beta UI?

[torn mantle]

Sometimes it gave me better results

Im not sure tbh

But its close to nw

[real totem]

Dayhush is nightwhisepr

[plain zinc]

By the way, please. Give dayhush this prompt:

I'll send the file.

[real totem]

Bro its an udpate4d

Nightwhisper

Thats why it got the similar naem

[real totem]

Wydym

Wdym

The other ais

Got diffferent name

Yes

Ye

IUts an updated

Nightwhisper

[plain zinc]

And give me the result for sure.

[alpine coral]

hes trolling lol

[real totem]

Whisper hush night day

Ye

[torn mantle]

ok

[narrow elbow]

how?https://beta.lmarena.ai/?

here?

[real totem]

nah

Beta is for

Normla text

Thewebdev is for code

Thats the only one

With dayhush

[torn mantle]

https://3000-iic8yfm1u12c8wyynb21b-0ce36864.e2b-foxtrot.dev

[plain zinc]

Thanks

[torn mantle]

how is it

[plain zinc]

Much better than 2.5 pro and claybrook

[torn mantle]

We need nightwhisper comparison

[plain zinc]

I think and feel! What's coming out tonight nightwhisper

Because the catalog is already ready

for coding models

[narrow elbow]

I feel like Google needs some pressure, or they won't be squeeze out good things🤣

[plain zinc]

I didn't understand what you meant. Can you explain to me, as a non-native English speaker?

[narrow elbow]

haha

[ocean vortex]

this is what I meant though. o4-mini-high vs 2.5 pro, same prompt for svg drawing

neither is perfect but 2.5 pro output is leagues ahead in this instance

[willow grail]

can u do poll

[ocean vortex]

A well dressed professor standing on the street with eiffel tower in the background, in front of basketball court, holding an umbrella, and observing people playing. It is CRUCIAL you make this in SVG only with the corresponding code!

the last sentence I formed this way to stop openai models from using imagen lmao

[keen beacon]

biased result xD

[cedar tide]

Who want to Remade this with claybrook and dayhush ?

[keen beacon]

im trying the prompt with o3 vs 2.5

blind test guys

pic 1

or pic 2

[willow grail]

both are trash

liiek wtf

[cedar tide]

Dragontail made this 👇
So dayhush better

[keen beacon]

yeah .. not the best of prompts
i actually like to make them make small games to compare

[willow grail]

make real python games... not small tetris trash

[keen beacon]

wtf am i looking at

[keen beacon]

oh its a discord clone

i thought this was like a discord theme i was about to say how can you like this

[keen beacon]

yep, my prompt was to make a 2d game with the following rules

1. if ball is shot it splits into two , unless radius is too small in which cases it pops
2. player can move left and right and shoot up
3. balls continue falling and moving at an angle as if thrown.
4. game starts with big ball falling and has a timer. player wins if he destroys all balls.
5. if time runs out or player gets touched by ball -> game over

[willow grail]

o3, o4mini, 2.5 who was best

[keen beacon]

o3 > 2.5 by a bit

[willow grail]

and what about o4 mini

[keen beacon]

didnt try

[thorny drum]

do you also have views on the canadian election?

[keen beacon]

hard to think we dont tbh. graphics are getting there and ai is getting at us but whatever

[keen beacon]

cmon we are in lmarena not poly

[thorny drum]

or the likelihood of jerome powell being removed this year

[keen beacon]

yep

[sour spindle]

What’s everyone’s honest real life assessment of 2.5, o3, o4 mini. Personally I am really like the outputs of o3.

[keen beacon]

as you said there are likely god knows how many matrixes within matrix and statistically we are probably at the very bottom of them but whatever

o3 with tools is amazing]

[sour spindle]

I haven’t run into that problem at all really with A/B testing

I’ve been very happy back checking sources used

[keen beacon]

from what ive heard so far people think o3 is better slightly or miles ahead, im on group 1

[balmy mist]

u good bro?

[sour spindle]

Better slightly is a good way to put it

[keen beacon]

take it ez you just learned you are in a simulation and everything is fake, no big deal

[narrow elbow]

whats the answer to life, the universe, and everything? 42?

[brittle tiger]

i think o3 is great. if there was a problem with it, it's that it does too much tool calling

[keen beacon]

r/im14andthisisdeep

we can become gods and make universes of our own.

but only Elon Jeff and their friends will

[balmy mist]

lol

[keen beacon]

why would they enslave us

[balmy mist]

bro been watching to much movies

[keen beacon]

nah they dont need slaves

once ai becomes smart enough theres no need at all for human slaves

[tall summit]

HAHAHAHA

[keen beacon]

ai + robots are smarter, faster, 24/7 work, no pay, no demands for high standard of living

[glass arch]

I wouldn't mind tbh

slop forever!

[keen beacon]

certainly get us out of the way more than eradicate, both are very bad

[balmy mist]

the point for normal humans is diversity

[glass arch]

counter argument: AI does not have hormones

[keen beacon]

but its not evil / bad in their eyes, more than it was evil/bad for us to use cars over horses .. which did lead to the "genocide" of horses, with their population drastically decreasing after the invention of cars

[glass arch]

I think AI should replace unskilled labor first over artists, programmers, and writers

[keen beacon]

should ?

[balmy mist]

why dont we eradicate the other lifeforms on earth? also ai would probably be just as curious about us as we are about it and other lifeforms

[keen beacon]

ai will replace everything, first low hanging fruit like coding math, then intellecutal jobs, then at the very end physical jobs via robots. and all the while every job will be devalued cause ai will continuesly lower the entry barrier.

[cloud meadow]

This statement alone is why midwits shouldn't be allowed to use AI. What we have compared to AGI is like a lobotomised cat being compared to a lion.

[tall summit]

when i'm always impressed with something on web arena, it's dayhush

[glass arch]

if all the unskilled labor goes away, then all we have left is the skilled labor

[keen beacon]

if there is a solution , if i heard any, is to brainchip all of humanity, combine it with ai and go forward as one mind colony

[tall summit]

i'm quite sure it can't be controlled. so there is no "should"

[cloud meadow]

💔

AGI is like 2-10 decades away

[balmy mist]

bro if you think about it the people with power already control the world lol

[keen beacon]

ai will take both skilled and unskilled away
theres nothing any human can do better compared to a smarter ai + robot

[balmy mist]

with ai it will just be more in our faces

but its the same thing just scaled

[cloud meadow]

Maybe a brain organoid can be made just for you so you realise that humans won't manufacture their own demise so quickly.

[tall summit]

i can juggle better

[glass arch]

guys, all you need to stop the AI uprising is a glass of water

[narrow elbow]

https://tenor.com/view/explosion-mushroom-cloud-atomic-bomb-bomb-boom-gif-4464831

[tall summit]

good one

[cloud meadow]

https://tenor.com/view/terminator-terminator-robot-looking-flex-cool-robot-gif-16625083

We are not getting skynet.

[glass arch]

it'll screw up everything if you pour water on their circuits

[keen beacon]

why dont you become billionaire and join him

[cloud meadow]

Bro, are you like 12?

[glass arch]

I am suspecting this too

[cloud meadow]

He is either 12 or a teenager from a third world country.

[balmy mist]

fr i was going to ask, but i didnt wanna insult him lol

[glass arch]

I think we should just democratize AI

[cloud meadow]

Brotha log off.

[glass arch]

if anyone can run it, then it's a much more balanced playing field

[cloud meadow]

AI will be weaponised.

[glass arch]

in 100 years, we will be able to run the current top-of-the-line AIs on our phones

[balmy mist]

if its the truth, what are you going to do about it?

[cloud meadow]

We
You need to grow up

Do not threaten real world action like this

This is illegal and will get the server taken down

This will never happen you moron.

[tall summit]

worked well with stablehorde

[cloud meadow]

We do not live in a sci-fi novel.

[brittle tiger]

using phones 100 yrs from now

Also it's probably closer to 5-10 yrs

[tall summit]

2025 trolling sucks

[cloud meadow]

Is this guy being serious?

[glass arch]

the FASTEST speed we can ever hope to achieve is light

[balmy mist]

he just joined the server two days ago

[cloud meadow]

@alpine pasture hey, I think you should moderate this user.

I'm ceasing communication with you

[willow grail]

are you a elon fanling?

[glass arch]

anyways, has anyone noticed that o4-mini-high doesn't speak with emojis anymore?

[cloud meadow]

No, he is just another puppet billionaire.

[willow grail]

the richest person is just a puppet?

or is he just middle manning the money for the non-puppets? @cloud meadow

[glass arch]

when o4 came outon wednesday, it wrote with emojis in every header

but now it fixed that and doesn't write with emojis

[cloud meadow]

• This is not a political server
• There are bankers who are richer than any of these public figures

[tall summit]

@alpine pasture help

[balmy mist]

you think they are actively updating it?

[glass arch]

I believe so

[balmy mist]

they are doing that with 4o

[willow grail]

huh? like who

[glass arch]

I can't think of a reason it would just switch to not using emojis hours after release

[tall summit]

maybe it does and your sample size is small

[brittle tiger]

Are you talking about arena, app or api?

[cloud meadow]

Refer to the first point

[glass arch]

https://chatgpt.com/share/6802556d-5578-8010-a4bc-bfeb15b4c6c2

[glass arch]

app

I have a link to a new conversation

[willow grail]

just answer the question who has more money than musk

[balmy mist]

ask grok

[willow grail]

wtf is grok

[balmy mist]

wait are their discord bots for ai models yet?

[glass arch]

grok doesn't defend elon surprisingly

[balmy mist]

really? why are you in this server if you are clueless about ai models?

[cloud meadow]

https://en.m.wikipedia.org/wiki/List_of_banking_families

[balmy mist]

sorry if you are trying to learn

but grok is xai model

on twitter

[willow grail]

why u saying twitter

[alpine coral]

oh ffs no more musk

let's talk about llms

[willow grail]

..... shows me dead people... with low money

[glass arch]

remember when grok 3 released?

[balmy mist]

because i was born pre 2023 and saying X can be confusing

[glass arch]

grok 3 is actually very useful for searching for information

[alpine coral]

grok-3-mini is legit impressive

[cloud meadow]

Families
Not gonna waste my time with you.

[willow grail]

thats not musk-fan enough

[glass arch]

have they announced plans for grok 3.5 or anything new?

[earnest parcel]

X is objectively just a bad name. and Grok-3 mini is great

[willow grail]

they are very poor

[balmy mist]

im just a curious guy, i dont fanboy over ppl

[willow grail]

so, u still saying musk is a puppet despite having most cashlol

[balmy mist]

are you a fanboy?

[cloud meadow]

Do you not know how money works?

[native shore]

you.wish has been warned.

[willow grail]

?

[cloud meadow]

Again, it's not my place to educate you. Since we are in an AI server, maybe you can ask gemini 2.5, it has a lot of knowledge (january 2025 cutoff!)

[willow grail]

@alpine pasture what did i say wrong

[alpine coral]

it's already been posted but whatever
via artificialanalysis

[alpine pasture]

Please stay on topic - this is an AI server for the LMArena community. Plenty of servers for other topics. Thanks all!

[cloud meadow]

I think this dude should be dealt with btw, Cherry.

[glass arch]

wow

this is a very neat graph

[cloud meadow]

That's interesting

[alpine coral]

i was looking into how they calculate 'price'

Price is a blend of Input & Output token prices (3:1 ratio).
It's ofc going to be blunt / imperfect, but 3:1 (input-output) i don't think is a suitable ratio to use for thinking models

they inherently use more tokens in their outputs (whether we see the reasoning token or not, they're output)

[balmy mist]

is this real?

how is that possible

grok 3 mini

[glass arch]

interesting to see that gemini says it doesn't know who's AI will be the best by the end of 2025

and then chatgpt says that GPT-5 will be the smartest

[balmy mist]

i need to test it out

its cheaper than flash?

[cloud meadow]

Why would you ask an LLM such a question?

I mean in terms of predicting events

It will just say whatever

[glass arch]

I wanted to see its response and if it would prioritize itself

[cloud meadow]

Right

Lmao, Sam Altman training pro-OpenAI propaganda into the model sounds funny

[glass arch]

grok says to keep an eye on xAI and deepseek

it sort of prioritizes itself

[cloud meadow]

Can't wait for R2

[glass arch]

the sad part is that the model doesn't even know its own dang name

[balmy mist]

how fast is grok mini fast vs grok mini?

[cedar tide]

and besides that, the price here is only the price per token but the reasoning models output many many more tokens so it comes out much more expensive

[native shore]

ontologicity has been warned.

[earnest parcel]

OpenRouter stats put -fast 40% faster. (187 vs 132 t/s)

[balmy mist]

wait so grok 3 mini is better than grok 3?

[balmy mist]

wow, i have been sleeping on grok 3 mini, need to try it out

[earnest parcel]

i don't know if +40% speed justifies +800% price, but eh, different use cases I guess.

[balmy mist]

yupp

[cedar tide]

Ozone ?

Yep

Deep comprehension on long context buts its on fiction not on code

o3 has better score

[tall summit]

fiction is harder

[balmy mist]

did anyone actually test this?

[tall summit]

??

besides the benchmark, you mean?

[keen beacon]

lol we were wondering why the beach water looked so brown. turns out sewage is being discharged right into it

won't be going for a swim any time soon 🤣

[keen beacon]

o3 surprised me me w this

[tall summit]

HAHAHAHAHAHA

[balmy mist]

yeah, like did anyone here test that

[narrow elbow]

dayhush, really good

[tall summit]

i can always start on a fanfic and we'll see

[narrow elbow]

https://3000-ijgbqghtlcyt6dub4lhh5-d3b1dbce.e2b-foxtrot.dev/

[alpine coral]

grok3-mini is a reasoning model

[balmy mist]

you been using it?