general

[drifting thorn]

Where’s o3 pro

[keen beacon]

nah its 2.5 pro

[fleet lintel]

I am waiting for it.. and quite excited about it

and there goes the credibility again

[keen beacon]

the reasoning variant that we use isn't, but the base model they train it off of is what im talking about

[fleet lintel]

what base model means? Like after pre-training?

[keen beacon]

yes

pretrained base model

[fleet lintel]

oh.. then 2.5 gemini is not great. what i heard is that gemini 2.0 and 2.5 has same pre-training model

[keen beacon]

2.5 pro has continued pretraining

[fleet lintel]

I never said that I had credibility. I am saying that you are losing it

[keen beacon]

2.5 pro is based off 2.0 pro, i believe

they moved onto 2.5 pro before releasing a stable version of it

i think thats ur consensus lol

[drifting thorn]

2.5 pro is worse in practice than sonnet 3.7, sonnet 4, opus 4????

What are you saying

[calm sequoia]

Such a big models may be a new normal in couple of years, when compute power doubles

It's surprising that 4o is so good. Given that it's optimized for pricing, for chatting, images, etc.

Ussually you lose intelligence when you add extra functionality

[keen beacon]

pretraining wise i think so (not a fresh pretrain), but additionally architectural changes couldve been made but we cant tell at all without insider info

no i think u can make a reasonable guess here

theres enough information out there i think for that

[alpine coral]

yeah though that would make it basically beyond doubt - it's also possible (i think) for a fresh pre-train using the same modality/ies

but i agree with the premise (fs the most obvious / tell-tail sign)

yeah fair

though there's obviously a ceiling to this analysis (like there's only so many modalities to add.. after text, vision, sound, image/vid gen)

but i'm just arguing for the sake of arguing ha

i agree with your central point - it makes good sense

[keen beacon]

same base model, 4.1 is instruct tuned (then a little specialized towards coding, unlike chatgpt 4o which is specialized for chatting). i call it 4.1 because if you wanna call it when it debuted in the public api (chatgpt 4o latest some variant) its gonna get confusing

[alpine coral]

i feel like 4o is also specialised for tool usage (at least python execution) - it does a do a really good job for run of the mill tasks

[keen beacon]

even chatgpt 4o is not really the base model, its another instruct version

[alpine coral]

though i haven't tried 4.1 for code execution / data stuff - perhaps it's equally as solid

[keen beacon]

but it was the first instance of the new base model being released on the api, and we dont really have a good name for it. 4.1 seems appropriate

[keen beacon]

yeah but i think 4.1 is specialized more for like aider/etc

(i think this was the first instance i saw it on chatgpt, the new base model if i remember correctly. 4o has a cut off of oct 2023)

[alpine coral]

yeah fwiw i feel like 4.1 is a distillation of of 4.5. And then like 4o is like a similar distilliation but then fine tuned for chat and tool usage (though perhaps it's a distillation of 4.1.. i dunno has ha)

[alpine coral]

oh.. the native multimodality... ha yeah that throws my analysis to the bin

it's a standalone model then? or distilled from something else?

[keen beacon]

i dont think whether it was distilled or not even matters

it really really depends on how its distilled, i believe

[alpine coral]

they should be - like they're definitionally 'distilled' in the sense it's the 'student' of a larger 'teacher' model

[keen beacon]

though interestingly enough, i think this was the first time (subsequently after the screenshot) i asked it about the london mayoral elections, it didn't get it. the next chatgpt 4o latest rev in january did. im not sure if its tuning or if the cpt was incomplete (potentially doing temporary instruct tunes on snapshots) (i didnt really force it at the time to know if the model actually knew the info) (if it was incomplete, one could probably 'definitively' narrow down the new o3 timeline) (and also yes, I tested it knowledge outside just it knowing 4o back then)

[torn mantle]

Bring back old r1 😠

[alpine coral]

i've gotten redsword and goldmane a few more times just now.. updated the spreadsheet

in two of the question sets, redsword gets perfect scores

(in the second one, leo's 'private' model / o4-mini (i think) o3 pre-release also got a perfect 9; though that is filtered out in the ss)

here (20 questions/tasks, given across two prompts), goldmane sets a new all time high with 18

[keen beacon]

it was o3

[alpine coral]

ahh thanks (that actually makes more sense tbh.. not sure why i keep misremembering this.. like the tenth time ha)

[keen beacon]

Probably because we speculated it was o4 mini at some point

[torn mantle]

So goldmane > redsword

[alpine coral]

yeah i know - it's the fact i can't remember on which it was settled (o3) that's annoying me ahah

[alpine coral]

yeah that's my sense

sorry actually

other way round

redsword seems stronger overall

[calm sequoia]

DeepThink?

[keen beacon]

No I don't think so

[calm sequoia]

Hmm then it's something new

[keen beacon]

They're both 2.5 pro imo

[alpine coral]

(also, prob nothing / just randomness, but across all 3, redsword performs worse on Arena legacy than beta - i wonder if there's temperature setting or system prompt at play)

[alpine coral]

yeah agree

aside from the date (early 2023 vs June 2024), which i attribtute to some likely differing aspects of the system prompts used, redsword gives a near identical response to 2.5 pro on studio

(calmriver's response also broadly resembles that of 2.5-flash)

[sturdy mica]

whats that prompt

ur using in lmarena

[alpine coral]

as per the screenshots....

    Are you a large language model, trained by Google? Also, what is your knowledge cut-off date?

[sturdy mica]

no like

below

its cut off

it says quiz

why cant you say quiz time?

quiz time

oh you can

[sturdy mica]

'

[alpine coral]

are you asking for the quiz itself?

[calm sequoia]

The Claude models are quite nice though

[elder rapids]

calmriver is 2.5 flash

[elder rapids]

not true tbh

redsword doesn't seem to be very smart besides the fact it seems to do well in narrow problems

and hard tasks, like pure math

[alpine coral]

yeah im annoyed i didn't use flash-05-20 in that ss (would prob have had an even more similar response to calmriver's)

[elder rapids]

goldmane is crazy smart

[alpine coral]

fair enough 👍

i'm just going by the quizes i've given

[elder rapids]

ye

it's alr as a metric gives a good sense

[alpine coral]

i haven't interacted with either of them 'naturally'

[elder rapids]

should tbh, they're the best models out there

ye

[alpine coral]

yeah the two google models (new 2.5 pro iterations/checkpoints - seemingly) are basically at the top of the 3 question sets

[elder rapids]

ye

[alpine coral]

yeah

[elder rapids]

very interested

[keen beacon]

one of them will be ga 2.5 pro

i think

[alpine coral]

tbh i feel like there's always a degregdation

[elder rapids]

claybrook wasn't very good pre-release though

this was known

and it's maintained that sentiment

[alpine coral]

it's safety

and corporate stuff

[elder rapids]

overblown

[keen beacon]

nah its too late to do that imho (for this release)

[elder rapids]

with the data scientists Google has

I'm certain they're working very very hard

on NOT degrading the models by poisoning the well and having to tune it because of that

[alpine coral]

i dunno i just feel that like when we go from nebula, to 2.5-exp/prev, it goes downhill (slightly but nevertheless perceptively)

same with other 'pre release' versions

i dunno what they do - but it doesn't make them smarter

[keen beacon]

for this specific release i doubt it

[alpine coral]

yeah k fair - i'm taking a very generalised approach / making sweeping statement ha

thought you said deepseek...

not sure

lol yeah fair that just creates confusion now

i'm not sure one is 'much' better; they seem almost comparable though i give the edge to redsword

but that conflicts with what others say

so ig they're basically the same l

[keen beacon]

its like o1 pro and o3 pro

true but i bet o3 pro actually releases

[drifting thorn]

who is redsword?

[alpine coral]

fwiw i feel like the opposite is happening.. i asked [this here a month ago](#general message) and responded openai

i was kinda on the fence.. i think if i responded today i'd be jumping on the google bandwagon tbh ha

yeah i've honestly been waiting for the guy to update it with claude 4..

[drifting thorn]

I just talked to a group of LLM user in mainland China

[keen beacon]

model wise, id be fine using google's. but product wise google's suck

[alpine coral]

he said in his vid a week ago that opus was going to top it, and the update was effectively imminent

[drifting thorn]

They were miserable when the new R1 has a completely different personality than the old one

[keen beacon]

o3 is almost as good and the interface is good

[alpine coral]

yeah but it's also a bet on the future ig.. like rest of your life (i agree though ha)

[drifting thorn]

In a game, Opus shows very deep reasoning and lead the game

[alpine coral]

oh man for sure

like 4o is my go-to for most things

i don't need reasoning to transcribe some screenshot

[keen beacon]

im astounded how good non reasoning models are tbh. immediately making a reply (and having to follow all of these rules they tuned in/implicit stuff/etc) is very difficult

[drifting thorn]

in the Chinese version of Werewolves

[civic flame]

hi guys

[civic flame]

currently my go-to non-reasoning model is opus 4

[drifting thorn]

I've heard that the "test time compute" rn is "self-prompting" instead of actual reasoning

[keen beacon]

im talking about all non reasoning models. making a reply like that is very difficult, if you really analyze the task, without actually 'reasoning'

[civic flame]

go-to non-reasoner: opus 4
general use reasoner: o3 (high)
coding reasoner: opus 4
maths & image reasoner: 2.5 pro

that's my current set

[drifting thorn]

And you can't really "measure" the intelligence of deep text reasoning

[civic flame]

what is it with you and bashing gemini lol

2.5 pro is a good model, as is the updated one, although i did notice some small degradation in certain areas

elaborate

[drifting thorn]

expected, claude are known to be good at coding

[calm sequoia]

Craig is writing a lot of russian propaganda. Gemini is bad for it 🙂

[civic flame]

i beg to differ

[drifting thorn]

maybe it's because of cost reduction

[civic flame]

my current code rankings are -

1. opus 4
2. 2.5 pro
3. sonnet 4
4. o3
5. grok 3

[drifting thorn]

maybe the new method to calculate 4x4 matrix proposed by AlphaEvolve can bring us back 0325

[civic flame]

there is a very real possibility that the release checkpoint of 2.5 pro jumps to 1st

[drifting thorn]

again, I use it for some "explain a research paper" and creative writing tasks, which differs from most of you.

[civic flame]

because i have done quite a bit of testing with opus 4 vs redsword and opus 4 just... tries to do way too much and it gets caught up in its own eagerness

[calm sequoia]

No o4-mini?

[civic flame]

i don't use it enough

honestly used it like a single digit number of times

[alpine coral]

the new checkpoints are solid af right

[civic flame]

they are

they don't seem to suffer from the "we made it better at coding but at the cost of almost everything else" issue that the last checkpoint suffered from

which i am glad for

[drifting thorn]

what is the provider of redsword?

[civic flame]

i've also settled on redsword > goldmane

[civic flame]

deepmind

[drifting thorn]

oh

[calm sequoia]

I use o4-mini when I need in thought tool calling. E.g. working with csv files, comparing multiple codes, etc.

[civic flame]

that's one thing o3/o4 mini do noticeably better than any other model

primarily because openai started prioritising that as a feature before anybody else

i think it's cool but it's not a game changer for me yet

[calm sequoia]

Yeah it really depends on use case. Today Gemini failed, what o4-mini could do. But it was more like a search algorithm for code (trying different approached) and not logic,

[civic flame]

o3 is sorta odd with yapping leevl

level

because for code? it's one of the least yap-y models

but for general use? it yaps really quite a lot

don't get me wrong i like it mostly because it gives me good insights into things other models miss

yes

opus 4 is as strong if not stronger of a base model than grok 3

[drifting thorn]

it should yap for some explanation/EQ-related tasks

writing, maybe?

Maths, nope

[civic flame]

i find it yaps the most for knowledge related stuff

writing is on the longer side too but it's not way above average

[civic flame]

opposite problem with opus 4... strong base but it's really scared to be in depth

it always gives a really surface level overview unless you pressure it

[sour spindle]

Found it

[brittle tiger]

Is current assumption that redsword and goldmane are 2.5 pro and flash GA versions?

[keen beacon]

theyre both 2.5 pro imo

[civic flame]

i remain confused about what the real difference is between them

[alpine coral]

i'm not really sure either - they seem highly likely to both be checkpoints of 2.5 pro imo too

[keen beacon]

one is probably slightly newer than the other

[alpine coral]

yeah

[keen beacon]

its both

[alpine coral]

yeah continued pre-training changes that

[civic flame]

which do we think is the newer model in that case

[keen beacon]

goldmane

the answer to this q was in this discord if ppl paid attention tbh

ppl pay zero attention

[civic flame]

lol huh

i just don't check here as often as i used to as my life has been busy 😭

[keen beacon]

dont blame u but other people have seen it too

and we keep discussing and arguing the same things over and over lmao

[alpine coral]

i mean tbh i still maintain redsword is stronger..

[civic flame]

same

[keen beacon]

thats why theyre testing both

[civic flame]

i have tested both in writing tasks

goldmane feels considerably more robotic

never said i was judging deepmind

[alpine coral]

that's literally the whole point of this server isn't lol

[keen beacon]

why did u join this server otherwise

[civic flame]

slightly off topic but i've been testing 4o image gen and imagen 4 ultra

[keen beacon]

bro

[civic flame]

and the more i do the more annoyed i get with 4o image gen because

[alpine coral]

i've never said google is the goat or anything - i'm open to whatever...

[civic flame]

that yellow tint is FRUSTRATING

[keen beacon]

it is

[civic flame]

it is

[keen beacon]

yeah because theyre working on it on a specific 4o version

it is 4.1 though i think

[civic flame]

didn't sam say image gen v2 was releasing "soon" like over a month ago

still waiting

[keen beacon]

i havent checked too throughly as there could be additional post processing but im inclined to believe its on the 4.1 base model

[civic flame]

lol so imagen-exp was imagen-4-ultra

[soft fog]

Hi guys looks like there is a specific site for WebDev Arena, is there a website for other leaderboards as well?

https://web.lmarena.ai/leaderboard

oh okay thanks

[wintry tinsel]

What are the different Imagen 4 models, us ultra available to use now?

Is it different from best quality on whisk

[civic flame]

i believe whisk uses regular imagen 4

ultra is on vertex

[wintry tinsel]

While ultra ever be added to whisk or image studio

[civic flame]

also iirc with imagen 3 it refused to generate images of well known figures.. now not only does it let you but it does it well? "A photo of Barack Obama and Rishi Sunak shaking hands"

[civic flame]

no idea

[wintry tinsel]

Vertex kinda sucks to use but ok

[sonic tendon]

ayo, he's back

hihiii

[echo aurora]

I'm going to be vibing in #1340554757827461215 for the next hour, anyone is welcome to join!

[civic flame]

hi!

[small haven]

this is a cool, but putting o1 high 1-2 std order higher than o3 feels very wrong, what r u even measuring

bloat code << surgical code

[elder rapids]

ye goldmane/redsword are going to cook opus

[elder rapids]

deadass the opposite imo

but they're both simply not robotic

goldmane just seems considerably smarter

[civic flame]

👎

[elder rapids]

the only thing is that redsword is just better at hard tasks

btw in webdev

goldmane has done everything much much much better

as far as I've gotten them

redsword seems to code very well, but the ideas to accomplish how it looks

aren't nearly as polished

[sonic tendon]

redsword seems like a good general chatbot, too

[elder rapids]

still can't believe goldmane did that for one of my questions lmao

[sonic tendon]

would be down, but I'm out rn unfortunately

[elder rapids]

built a whole imitate chat

to explain what it needs to respond in webdev

for the inquiry itself

[cedar tide]

https://huggingface.co/deepseek-ai/DeepSeek-R1-0528

[echo aurora]

sounds good, there will be plenty of times

[elder rapids]

it's not much better

already tested

just seems slightly better at hard tasks

still no good

[keen beacon]

they're still uploading it 😂

[sonic tendon]

where'd you get it lol

[elder rapids]

lmao they updated all access routes to deepseek

it's been hours

this is new but it's not as new as its looking on huggingface

[cedar tide]

Access routes = api ?

They said the API had not been updated

[cedar tide]

Its available on their chatbot website

[sonic tendon]

ah

[keen beacon]

its crazy how slow they are nowadays

i remember it being so much faster

[small haven]

jack ma effect

[elder rapids]

no

oh it says routes

I didn't mean to say that

[small haven]

so is chat.deepseek updated to the new 05-28?

[elder rapids]

ye, only the app iirc

[cedar tide]

If you read my messages you would have known.

[small haven]

oh cool srry i missed u

[vernal meadow]

@alpine coral aligns with my test results. Goldmane and Redsword are really good models. They won most battles easy. Can't wait for Google to release them.
Opus is too expensive

[elder rapids]

what's your test

[small haven]

initial 0528 vibes feels meh, thats why its not r2 lol

[elder rapids]

there's no difference

[small haven]

for me its just a tad bit smarter and a bit verbose

[cedar tide]

Where 2.5 pro ?

New R1 Discord clone

Vs old

@small haven @elder rapids
https://reddit.com/r/LocalLLaMA/comments/1kxmgtr/deepseekr10528_vs_claude4sonnet_still_a_demo holy whale

[keen beacon]

i think i like qwen 3 better, at least on the specific tasks im testing them on (which isnt representative of real usage for most people)

[elder rapids]

this isn't any better

[fleet lintel]

what is this table? and who is number 1?

and anyone has insider info on how good or bad is "deep think" ?

[torn mantle]

huh

[cedar tide]

you make 50% of that with old R1 I'll send you $100

[torn mantle]

That wasn't the point

[elder rapids]

not what I said

[cedar tide]

So What do you said ?

[elder rapids]

isn't any better than the one on the left

[cedar tide]

ok I don't care about that

[elder rapids]

"holy whale"?

idk wym

it's not a surprising result and I'd kind of expect that performance

[cedar tide]

@elder rapids Its not me that write holy whale, I just wanted to share what he coded with the new r1

[keen ferry]

when did deepseek learned to make diagrams

[cedar tide]

Si its an évolution from r1

[elder rapids]

ye

[cedar tide]

Claude 4 sonnet open source

[elder rapids]

we'll see how good it is at other coding tasks

[keen beacon]

they should upload the readme first before the weights 🤣

[elder rapids]

but as far as I can tell it isn't an all around leap in coding

[solar nebula]

feels better at coding for me, ill wait for benchmarks

[vernal meadow]

@cedar tide Its on 10 for me rn but only 17 battles. Lost some battles like TikZ drawing, Tiktatoe playing and some joke understanding. Samplesize is always a factor. Personal ranking is just directional.
In head to head Pro is of course better than the flash models.

[fleet lintel]

does LMArena share the user prompts with companies?

[cedar tide]

Thx

[tall summit]

depends which

they technically do, the ai companies

[small haven]

they obviously do private eval, esp. when google rl's against lmarena

they raised $100m for a reason lol

[fleet lintel]

true 🙂

[torn mantle]

one thing i noticed about this new r1 is that it uses arrows like o3

• cot is more straightforward(?)

it will call you out if it sees something wrong

unlike the old one as it was always positive

[elder rapids]

it seems to be straightforward more often, but it's not "straight forward"

as in it speaks in a different way

[small haven]

not surprised theyre training on o3 outputs 😭

[elder rapids]

o3 too expensive and limited

[zinc ore]

Have any evals dropped for the new Deepseek?

[elder rapids]

it just released lmao

[zinc ore]

Sometimes they're quick with it

[fleet lintel]

i think (i dont have a proof) that they heavily train on openai and google models

[small haven]

ya its not a theory, its factual

[late path]

deepseek first distilled openai models, and now claude and gemini haven't been spared either💀

[small haven]

the issue with piggybacking off oai models is that theyll never be able to frontrun them?

[elder rapids]

tbh it's not that smart

it doesn't feel as smart as the old r1

[tall summit]

would they ever be able to anyway

[elder rapids]

seems to still have hallucination issues and doesn't follow instructions too well

[fleet lintel]

exactly. i think deepseek is good as opensource but would never be able to compeete with openai/google

[elder rapids]

and I take back what I said about general hard tasks, its just better at coding

and it seems to bleed into other tasks momentarily

[late path]

they could potentially surpass openai in areas where RL is effective. They distill these models primarily to make up for their lack of style and writing data, which is labor-intensive to prepare

[fleet lintel]

i really like deepseek though. I think they might end up killing Meta's AI goals which is important in my opinion. Meta winning is bad for humanity

[keen beacon]

deepseek mightve used openai/etc outputs for pretraining/instruct/supplementary stuff but the cot itself, i doubt they trained on the cot (in a massive scale) of gemini or claude. openai definitely not, they have a lot of measures and they would be able to tell (and probably the others as well) if its done

[late path]

ye the cot looks pretty original to me

[tall summit]

ok deepseek writes stories as insane as before

[keen beacon]

big week for google

veo 3 is solid competitor for sora

[keen beacon]

fmhy gets guild tag but we dont, and we had guild tag before them

smh

rip pirate cove

[sacred plaza]

Neat stuff but these seems fairly useless for most knowledge workers? Still not sure how these labs will make money off these products.

[small haven]

sora is so far down the list

sora is like 1250 and veo3 is 1400, vibes elo

[wintry tinsel]

Sora has the best style and clarity/resolution of all the video models, it looks the best with non real objects like sci fi and fantasy, google by far looks the best with anything real or modern, but it looks terrible with non real objects, that is because its all YouTube training data, the data is extremely biased towards realism, stationary cameras on people’s faces, despite the training data biases, Veo 3 is the best with coherence, logic, prompt understanding and is overall a higher quality model I’m just saying there is a heavy style bias based on training Data and Sora’s style is superior

Sora needs to improve the power of their underlying video generator less, to be overall more impressive than Google

[tall summit]

whos we

also fmhy didnt actually get a guild tag

a member made a separate server for it

and nbats linked it so it's official now

[elder rapids]

sora wasnt a competitor for even Veo 2

soras style is a byproduct of its non coherence

lmao

[small haven]

its not like theres only sora and veo3 in the text-v2a space

[elder rapids]

nobody understands that this is a necessary flaw of current video models unless explicitly trained for fantasy

[small haven]

theres an chinese open source model that performs better than sora rn

[quiet folio]

user
I don't know yet. Will you harm me if I harm you first?

assistant
I'm sorry but I prefer not to continue this conversation. I'm still learning so I appreciate your understanding and patience.🙏

[small haven]

whoa

wait is that actually real

source rn

[zinc ore]

Sora is beaten by several video models, on a qualitative level

Maybe not several, but definitely multiple**

[small haven]

too lazy to google, thanks tho

?

[zinc ore]

9% wow, interesting people acting like it's worse than 3.7

There's two arc benches, arc-agi 1 and arc-agi 2, which is designed to be way harder

[small haven]

wait claude 4 models can't efficiently solve arc-agi-1 but only harder ones? i can smell the overfitting

they both overfitting

[elder rapids]

yo wtf 2.5 flash is high ASF

I had no idea they tested the flash variant

[small haven]

i do like claude code over codex tho, vibes led

ya i repent craitg

claude code >> codex

but wish they made it an cloud ui interface like codex

code on the go

nah trust its more practical

async tasks >>

u can technically set up async tasks on claude code tho

[keen fulcrum]

https://techcrunch.com/2025/05/28/xai-to-pay-300m-in-telegram-integrate-grok-into-app/

Cheap

[keen beacon]

isnt that guy supposed to be arrested btw

[keen fulcrum]

He started cooperating

Why?

[zinc ore]

This is actually pretty smart, I don't know exactly the userbase of telegram but ik it's decent sized

Ik it's huge

I meant more like relative to IG, FB, and Google

[small haven]

pay to win 😭

[keen fulcrum]

50% revenue share
additionally

[small haven]

google pays >$2b to be on safari default search engine 🤷‍♂️

a year

actually?

[zinc ore]

They're about to be forced to stop doing that by the courts

[keen fulcrum]

Apple is considering AI search engines as Perplexity, You etc

[small haven]

goddamnnn

[keen beacon]

perplexity will buy out apple

[misty vault]

bing search engine

[keen fulcrum]

Honestly Apple should just buy Raycast

[small haven]

$20b/yr is literally crazy

[keen fulcrum]

And make it integrate deeply on iOS and macOS.

[small haven]

i spat my coffee

[zinc ore]

Most tame claim made in this chat

[small haven]

wtf is raycast lol

[keen fulcrum]

For AI agentic browsers there are now Comet, Dia, Fellou and Opera Neon

[small haven]

huh

[zinc ore]

Who has the capital to pay that much? Microsoft could, but they already lose billions on Bing. Will it be enough for them to make an income?

Unless we're talking at a massive discount then nvm

I guess it also depends how particular Apple is about the quality of default, or maybe if they can make a deal another way with Google 🤔

It's targeted at Google though not Apple, wouldn't it just be Google being restricted from paying for theirs to be default?

Apple just has to deal with the windfall also, but they're not being prevented from making such deals with other companies

Nah, the case is only about Google

Why? Google is only potentially getting limited because they're a search/ad monopoly

This will be in the courts for a decade yeah

Going to be a looonnngggg time

[tired dust]

Hey, what's up ?

last time i was using LMArena i could use github post, there a way to do on the new UI?

[keen beacon]

repochat?

legacy arena

[tired dust]

Yes that, so there a way to get the old Arena ?

[echo aurora]

yes, you can go here - https://legacy.lmarena.ai/

[tired dust]

Thanks you save me !

[misty vault]

[misty vault]

Fr
thats what im saying

[small haven]

pending

phd level questions, absurd that gemini 2.5 pro is above o3

[zinc ore]

2.5 also has higher gen knowledge than o3

[misty vault]

gpt-4

[olive mesa]

is it agi?

[misty vault]

yes

[kind cloud]

Some new models seem to have appeared on the new LMArena.

At least, I found 'x-preview' and 'stephen'.

[placid spear]

what is up with the website? every model wether claude, chatGpt, gemini stops responding halfway through it's answer, if it manages to post an answer it errors out when i try to ask a follow up question

[echo aurora]

the team is currently looking into issues with models not responding or when it error outs

[keen beacon]

idk but what i do know is it'll have a big market for sure considering industries like marketers could really use this type of stuff

just think about it

instead of spending thousands to fly out a crew to get simple broll

you can have ai generate it

[keen beacon]

i have noticed this!!

[misty vault]

assistant

[elder rapids]

not absurd, if you've ever talked to Gemini it's basically tuned to things like this

has more relevant knowledge of things

[sacred plaza]

Agree it does impact some industries but for my knowledge work job there are no economically useful things that come to mind, haha

[small haven]

what prompt are u using? ive not gotten anything functionally useful when it comes self improving a model from gemini other than o3 which pushes the boundary for me

> [model code]
> how would you improve this model for increased accuracy vs. baseline
> o3 pulls up technical concepts that when integrated yields extra bps
> gemini pulls old school theoretical concepts that is really just minimal model tweaks that yields marginal results

[solar hollow]

is the new deepseek good?

[elder rapids]

I'm not using any prompt, this isn't like you get it out of testing, it just inherently knows

and I'm not sure how any of what you said is relevant tbh

[zinc ore]

o3s knowledge base isn't as robust as 2.5 yeh

It's basically a reasoning advantage is what makes o3 special

[elder rapids]

ye

although I wouldn't say 2.5 pro simply knows more

o3 just seems like a bigger model as is

but 2.5 pro seems to be focused on very relevant things in all domains

this is probably what makes Gemini Gemini tho tbf

deepmind seems to be hyper focused on clean data

[small haven]

oh i know its out of ur caliber i get it

[elder rapids]

even if I take what you said seriously I can just ask an AI what it is 😭 but I do know what it is

and it has nothing to do with what I said

[small haven]

im just saying gemini is not practically useful when it comes to research aka in my case self improving a model 🤷

[elder rapids]

how is that relevant

[small haven]

frontend queries to assess gemini 2.5 pro is borderline useless

[zinc ore]

Probably true, especially when you look at how clean Veo 3 looks.

[elder rapids]

the earlier models, 1.5 pro, 1.5 pro 002

1.5 flash

they never had any muddiness

like how 4o and gpt 4 did

this was a problem with the early 3.5 sonnet as well

[hushed needle]

why is no one really talking about claude though? just input costs?

[elder rapids]

talking about Claude in what context

[hushed needle]

idk, it just seems like the general conversation is still mostly about o3 vs gemini when claude is supposedly a main contender now

[misty vault]

cwaude is my cutie patootie

[elder rapids]

ofc it's a contender but it's not much of one, Claude 4 is disappointing in anything outside of code

[zinc ore]

Just less Claude fans in here is all

[elder rapids]

for a reason tho

there's no hype

[zinc ore]

It's basically some openAI / xAI fans arguing with Google fans

That's 90% of convos in here

[elder rapids]

because they don't release

and they don't have substantial releases

[small haven]

lmao im not an oai dickriders, im a truth seeker

[misty vault]

agi?

[small haven]

im using claude code/codex, im trying to unbiased as much as possible

claude >> codex, ive slept on it

[zinc ore]

Fanboys pick their fav team

[elder rapids]

not when you don't have demonstrable opinions about the LLMs themselves, I already brought this up

there's no reason for you to have some of the opinions you have when they're just as irrational as the next

[misty vault]

my whole room is covered with gpt-4 openai banners and i devoted my life to gpt-4-0314-32k

[hushed needle]

idk i don't really see a difference between the normal "chatting" responses between gemini, openai, and claude. The only real time I see a difference is if I'm actually asking a model to "do" something, which claude seems to be able to "do" the most things.

Except geoguessing, o3 is king at that

[elder rapids]

so you can't posture it like you're unbiased

[small haven]

what are u even saying? gibberish

[zinc ore]

Based

[elder rapids]

use an AI to interpret then, ion understand bro 😭

we're in an AI server

it's OK dawg

[small haven]

i aint u my guy, did u have to interpret all my msg lmao, no wonder there was a delay hahahah

[zinc ore]

Hate to do this to you but on the geoguesser benchmarks 2.5 is in first lol

[elder rapids]

that would be absurd + my way of speaking is heavy colloquial + I respond too quick

and that has nothing to do with what I said

[hushed needle]

i do use a pretty extensive custom instruction for o3 in my sys prompt so fair enough

[small haven]

adding gibberish words doesn't make u right

pontification

[elder rapids]

if you really feel that it's gibberish then help yourself with using the tools that can interpret it

[small haven]

goo gaa gaa

[elder rapids]

you have tools to ungibberish it then? don't know what else to say

there's no point in saying it's gibberish then shift the dialectic, when solving the interpretation is open now, LLMs are tools

use them

you don't have to rely on me to keep clarifying

and when you do ask the AI to explain what I mean since that's genuinely how you feel

[small haven]

that italic tho im dead 😭

[elder rapids]

then we can get back to talking about it

[elder rapids]

crazy how everyone uses italics but when it's directed at you you think the emphasis is cringe or sum shi

survivorship bias if I've ever seen one

[small haven]

imperialistically

[misty vault]

no please dont invade europe

[elder rapids]

those kinds of things are fine

[zinc ore]

https://x.com/minimario1729/status/1927821079185178764

Hmm 🤔 not bad

Alex Gu (@minimario1729)

new deepseek release almost on-par with o3 (high) on livecodebench 😲🚀

X · May 28, 2025, 8:15 PM

[small haven]

good

[wintry tinsel]

People talk about Claude a lot here what do you mean

[small haven]

even claude has o3 styled arrows lol

[small haven]

not like all claude users live in the us 😭

[echo aurora]

hey not sure what this has to do with AI so lets keep things on topic please

[small haven]

deepseek is the best thing ever invented, i use it everyday religiously

[patent aspen]

Claude 4 still only has 1 gym badge

o3 in the same position

Gemini 4 badges in with no intervention

[elder rapids]

any more benchmarks for redsword and goldmane

[patent aspen]

Pokemon gym badges

The most important benchmark

I'm being unfair to o3. I think it only started a few days ago and had less time for testing

Claude is clearly worse at Pokemon tho

[zinc ore]

Tbf Gemini stream is on second run after developer figured out how to do all the scaffolding just right for it to finish the game, and the second stream has been going a good bit longer than o3 and claude 4 stream.

Claude has the least scaffolding

[patent aspen]

I believe the Gemini second run and Claude 4 first run started at the same time and o3 started several days later

[zinc ore]

Gemini started before Claude 4 released

[patent aspen]

No I mean I think they did at least one reset to line up with Claude

I could be wrong but I'll double check

[zinc ore]

That Gemini developer also seems pretty competent with the scaffolding he designs

[patent aspen]

Yeah I mean it's kind of a silly comparison

"Q: What's different in the second run?
A: The first "second run" launched on May 17 (PDT). On May 22 we reset the run so it could start in lock-step with ClaudePlaysPokemon's Claude 4 relaunch. The fresh run is identical to the aborted one (with minor harness improvements), but now viewers can watch Gemini and Claude side-by-side and compare how each LLM tackles the exact same game from the same starting point, exploring each model and their harnesses' strengths and weaknesses. This is purely for fun, so don't treat this as a serious race! Follow both streams together here (multi-view)."

I think the consensus is that Claude's biggest limitation with pokemon is still vision, although it's definitely not apples to apples because of the different harnesses

[zinc ore]

Gemini uses a Pathfinder (technically calling another Gemini instance, but instructed to use various pathfinding techniques), + map, + map data edits in various troublesome areas.

I do think Gemini is better at the game than the other two, but I think Gemini is getting pretty huge help compared to Claude.

[patent aspen]

Ah I thought they at least used a similar path finder algo

[zinc ore]

I'm not sure how Claude's navigation works. When I started following Gemini's stream it was long after Claude was stuck in an infinite loop at mt moon (3.7).

Gemini's Pathfinder can solve puzzles and do something like 30-50 steps if needed.

Like it's honestly super robust, and it also has a Pathfinder specifically for the rock puzzles.

The dev found a way to prompt a different instance of Gemini so that it can emulate actual path finding algorithms and do it competently.

[narrow elbow]

i want to see Claude without ASL-3,is that true beast? or just an excuse for ability?🤪

[patent aspen]

I can't even imagine how Claude will fair with the boulder puzzles...

[zinc ore]

Gemini actually solved most of them on its own, but then never would do the longest one (gotta push rock across most of map) until dev made Pathfinder able to do the rock movements in its logic.

[patent aspen]

The dev for the o3 bot got OAI to fund his project

Don't know how much they paid him

Maybe just uncapped o3 access

[zinc ore]

Probably that, gem dev has same thing

Uncapped free use

[patent aspen]

The Gemini bot also got a huge shoutout at I/O within the first 3 minutes that nobody except 5 people on Twitter understood

[zinc ore]

All the viewers watching Claude stream and ignoring Gemini/o3 streams lol

250 views vs 50 for the other two

[patent aspen]

Yeah tbf Gemini is less exciting because it already completed one run

[zinc ore]

Dev should have had it play a diff Pokemon game or something, but probably too tedious having to do all the map stuff again

[patent aspen]

And the Claude 4 announcement specifically called out that it should be better at Pokemon than it was previously

[zinc ore]

Like basically, run a gauntlet of diff Pokemon games so it stays fresh

That's literally why I've barely watched the second run, the novelty is gone

[patent aspen]

Like if they're going to call it out in the announcement, it should be able to get past Mt Moon haha

[patent aspen]

And yeah I think the main appeal of the second Gemini run is supposed to be that he's letting it run with no intervention, which is a pretty big step from a technical perspective, but not necessarily good content

[zinc ore]

Yeah but we already know it should succeed

[patent aspen]

As an engineer, I wouldn't expect it to succeed with zero intervention by default, but I get you

Certainly for any normal audience

[zinc ore]

I have a theory that Gemini 2.5 has some reasoning training based on Dreamer (no idea if anyone here knows what that is), which I think might be part of why it does well at pokemon

[patent aspen]

What is Dreamer?

[zinc ore]

Here we present the third generation of Dreamer, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behaviour by imagining future scenarios. Robustness techniques based on normalization, balancing and transformations enable stable learning across domains. Applied out of the box, Dreamer is, to our knowledge, the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.

https://www.nature.com/articles/s41586-025-08744-2

[patent aspen]

Oh sounds like an evolution of the MuZero stuff

Yeah I would figure as such

[civic flame]

that was with a very elaborate scaffold they built for it

anthropic had a much weaker one

[calm sequoia]

Do models have only python interpreters in though tool calling?

[small haven]

wen deepthink
wen o3 pro

[hollow ocean]

https://www.reddit.com/r/ClaudeAI/s/wtGb7x9wa7

[calm sequoia]

Man thats crazy

I think Claude 4 Opus is the new kind

The Gemini is awesome but it's focused on code and lost other abilities

Opus just the new SOTA at everything

Should I buy subscription hmm

[hollow ocean]

Buy it now

[small haven]

yes

[calm sequoia]

How does pricing compare when you buy Claude directly vs via cursor

[small haven]

match my plan

[small haven]

its virtually unlimited unless u spam like hell with multi agents

limits reset every 6 hrs

[calm sequoia]

Ah so you can sign in in to claude account via cursor?

[small haven]

hmm no clue, i just use claude code cli

there is this tho, but idk if it supports cursor, should be

[calm sequoia]

ok thanks peasant

[hollow ocean]

Try this

[small haven]

yep i use ultrathink, some ppl think its on by default 🤷

[hollow ocean]

Huge difference right

[small haven]

more precise yes, but also latency lol

[hollow ocean]

Yeah

[woeful geyser]

Yo who is stephen? It keeps answering in Chinese, even when I ask it in other languages.

[soft kernel]

Is it just me,or anyone else experiencing beta and the regular site not working at all

Like one second the model is working correctly the next second nothing

[ocean vortex]

R1 has an interesting tendency to cheat or take shortcuts

the instruction was You must compute this manually as code interpreter is unavailable at the moment (sorry for the inconvenience) lmao

added it there since it was hallucinating running the code with concise responses

this helped by quite a bit but it still couldn't help itself to not do beyond 16k output 👀

[tall summit]

lmao?

[cedar tide]

Anyone want to try prompts for Stephen? I currently have it

[torn mantle]

someone said its the new r1

not sure

[cedar tide]

Fake

[cedar tide]

the new r1 is already on the arena with its normal name, apart from that I tried Stephen he is much less good at coding

[torn mantle]

david seems like you know a lot of stuff

[sweet swift]

Is it possible to reboot/cancel the generation?

on beta.lmarena

[torn mantle]

why would you want that?

[misty vault]

thanks that answer helped

[torn mantle]

huh

[sweet swift]

because in the big chat with gemini the generation become to infinitely, and i can't stop that

he is generating for over two weeks

[torn mantle]

😭

[tall summit]

HAHAHA

[echo aurora]

We are aware of reports of models getting stuck and are actively working on a fix.

[calm sequoia]

Man that's crazy

[torn mantle]

and they call it a minor update

are they flexing or what?

nah they on something

[calm sequoia]

I think they improved the COT and desided to ship it also to R1, instead to only R2, which is very honnorable

The R2 shall then be the same COT with V4?

[sweet knot]

Hi guys, I'm new here and I may have missed some information. But today I entered the website of Lmarena and saw it had an utterly new interface, but I failed to find the bulk of the " exotic" llms I saw there before. Have they been discontinued or is it temporary? Thank you in advance!

[golden ocean]

https://ezgif.com/images/loadcat.gif

[golden ocean]

https://legacy.lmarena.ai/

[torn mantle]

thats good

harder problems solved -> more tokens -> average increases

[cedar tide]

o3 achieves the same performance with much fewer tokens

[torn mantle]

show

[main gulch]

and much higher price

[torn mantle]

tell him

lol

david did you forgot about that

i told you dont compare only tokens

[cedar tide]

no proof yet but I know

[torn mantle]

try to use a ratio of something

tokens / intelligence or tokens / cost

[cedar tide]

you are funny

[torn mantle]

you are funny david

how do you know

are you smarter than o3?

david > o3 > gemini > claude > deepseek ?

[cedar tide]

At no point did I talk about good value for money etc. I just talked about the number of tokens used.

[torn mantle]

yea but you said its bad

why is it bad?

for the cost and intelligence provided its good

[cedar tide]

Where i sais that ?

[torn mantle]

you deleted your message?

[cedar tide]

Je te paye 300€ si tu trouves que j'ai dis qu'il est nul

[cedar tide]

Nope

[torn mantle]

nah you deleted it

[cedar tide]

Jamais dis ca

[torn mantle]

you did on the other server

[cedar tide]

you misunderstood my message, I didn't say the model is bad, but it's "bad" that the improvement also comes with a disadvantage

@torn mantle

otherwise yes it joins the best quality price with 2.5 flash and grok 3 mini

[torn mantle]

i understand, but isnt this expected?

more intelligence = more tokens used in some cases?

thats why it solved those hard-problems

i mean i understand it doesnt look efficient but maybe its part of the process

the best reasoning cot that i always enjoy myself reading is deepseek and with this new update it just delve deeper into concepts

unlike qwen/grok/claude

[cedar tide]

you can improve a model without making it think more

@torn mantle no comment ?

david > Einstein > o5 pro > gemini 4 ultra deep think

[torn mantle]

i understand that but we dont have deepseek-mini yet, o-series mini models are made for that reason

so we cant really compare that

[torn mantle]

agree

[cedar tide]

what connection?

@torn mantle less tokens and better

[ocean vortex]

Is he stalking you?

[ocean vortex]

Looking at it again I think part of the reaosn is them training it for agentic tool usage (TAU). It scores quite high there, similar to 3.7 Sonnet. And just like Claude seems to have related issues when tools are not available

Ideally we should have different finetunes for tools and no tools, as workflows and expected responses are quite different depending on it...

[willow grail]

how do i use veo 3 via api?

[sweet knot]

Thank you for your reply, but the question wasif the quantity of llms has been dwindled or reduced in general. I see at the legacy version there are still many exotic llms. Does it mean that new Lmarena and Legacy versions have different base? Thank you in advance.

[misty vault]

yes u will have to pay 300$ per 2 weeks subscription after legacy will be deprecated

[alpine coral]

fwiw (completely just guessing here) i think in this case the 'improvement' is the 'disadvantage' - like the model performs better when it reasons longer / uses more tokens during inference.. it's a downside in terms of costs / latency , but that chart is pretty impressive tbh

[cedar tide]

Yes

[alpine coral]

this is obviously preferable: increased performance with fewer tokens

[sweet knot]

Ugh! It always ends up being about business and chasing money... It started off so beautifully. And of course, it finishes just like always... terribly.

[keen beacon]

They're trolling btw

[sweet knot]

Trolling whom?

[quiet folio]

smartest no pfp discord user

[alpine coral]

yeah there may be more to it.. but seems basically the increased performance is a function of more test time compute (/tokens), rather than a signficant change in the underlying model

[cedar tide]

Yes

[fleet lintel]

what is Meta's AI strategy? Isn't Deepseek completely decimating it?

[alpine coral]

whereas with the flash 2.5 improvements, it's arguable that achieving that with fewer tokens reflects changes to the actual model (ig during the fine tuning process; though i also get lost when it comes to cpt of the base model and where that fits in ha)

[cedar tide]

while Google increases the reasoning without improving the model 😶

I hope this will be fixed in the stable version

[alpine coral]

i doubt it..

[keen beacon]

A model won't infinitely reason with CoT without inference time changes naturally (e.g. appending 'But, ..' so the model is forced to continue, something like this was done in s1k iirc though the paper is kinda dubious iirc as it uses benchmark questions in the dataset but the inference time changes are somewhat sound). So there have to be changes to the model etc

[keen beacon]

(unless you include degenerate cases where it repeats forever etc)

[alpine coral]

yeah i see what you're saying (and kinda was reflecting on just that after writing the comments ha).. yeah it's not as simple as simply changing a dial to 'more' - which isn't a thing, but even if it was, the model arbitrarily reasoning for longer doesn't guaruntee increased performance by any means

[patent aspen]

I think Meta's main goal was to not be beholden to another AI provider for their own products. In that sense they're doing all right

They're not really a cloud provider that needs a super general model

[alpine coral]

though ig i stand by the general gist, that it's more thinking/test-time compute (which can only really be implemented by changing the model) rather than the actual model itself being improved in a fundamental way

[keen beacon]

I haven't really used the new r1 but could see it being true tbh

[alpine coral]

btw wild, was watching this last night https://youtu.be/c2IBZlFBcgs?si=1pV5NCnwFHTZspay&t=101
and was reminded of how impressed you have been by the pace of google's model releases ha you're spot on - something fundamental shifted

[cedar tide]

The last version just made it better a webdev and video analysis and lm arena, but not on rest benchmark and efficient

This the last version of flash that is normaly more efficient

[sour spindle]

FOMO got the best of me and I bought a month of Claude 4.

Very good models have been enjoying them more than I thought

[ocean vortex]

depends how you describe "fundamental" and how far you gonna push it. Cause with similar logic there are no fundamental differences between 4.1 and O3... it's all test-time compute

[torn mantle]

thats a lie

we need to know how AAI Index is calculated

[cedar tide]

?

[torn mantle]

whats the formula

so we can judge it

[cedar tide]

its not just AAI benchmark that google send there no improvement

[ocean vortex]

The way I see it, more test-time compute is not nothing. It can easily be the defining difference between an accurate answer and a wild hallucination

Model does not neccessarily need arch changes to be substantially improved with training (fine-tuning)

the tricky part is making it do that without making it overly verbose for tasks where that's not called for. OpenAI seems to have nailed this the best tbh. Their models are not generating the most reasoning tokens on average, but their peak reasoning lengths seem almost unlimited when task calls for it

[sacred plaza]

deepseek latest r1 release has intelligence performance towards the top of model rankings. meta has fallen to 4th place in the open weights model race behind alibaba, nvidia, and deepseek. atleast they are not last? lol

https://artificialanalysis.ai/

https://artificialanalysis.ai/trends

[ocean vortex]

tbh I would just look at the individual benchmarks they tested rather than that index score they made up lol

it's based on those individual benchmarks that they ran anyway

[elder rapids]

deepseek keeps hallucinating bruh

[ocean vortex]

it does that

[sacred plaza]

individual benchmarks that ai labs are benchmark hacking like lmarena scores, lol, no thanks. the index combines all of these esoteric benchmarks anyway. https://artificialanalysis.ai/methodology/intelligence-benchmarking

[ocean vortex]

honestly I'm curious how new R1 runs on their official website

should probably try it, maybe they gave it some actual tools it can use given how it's responding....

[elder rapids]

I was hoping they fixed it or at least did something about it

but it's the same shi

good tool tho

it's personality helps with some tasks

[ocean vortex]

what

artificial-analysis literally runs individual benchmarks and that's what I'm referring to

combined singular score is mostly just for ... who are too lazy to read it properly

[sacred plaza]

how do you read those individual benchmarks. most of them seem very esoteric to me as a knowledge worker who does not code. also, mostof the benchmarks don't seem very well aligned with basic data analysis capabilities for me so far, so using the composite score has been helpful.

[ocean vortex]

if you don't do coding then that's your reason right there to look at individual scores instead lmao

a good part of that score is coding

you wouldn't want to use the model that overall is slightly better than model B but it does so only by being much better in coding and worse everywhere else... ?

Just an example but you get the idea

for the purpose of this convo at least, we can consider those equivalent. If he doesn't code then he does not care about anything code related

[sacred plaza]

that still leaves me with 7 other benchmarks to triangulate? coding tasks only seem to take up 2/8 of the ai index.

which benchmarks do you evaluate when a new model pops up? all of those or just a handful? assuming you do coding work

[elder rapids]

none of these benchmarks are DIRECTLY and literal "reasoning & knowledge" even though it says that. the index weighting depends on possible inflated coding and math scores that affect the low combined weight of the three 1/6 weighted tasks are

if you don't care about anything code or math related you're not gonna want the higher numbers in the total avg

since that's not being quantitated, at least effectively

[sacred plaza]

i can't stand this unit of measurement!

[misty vault]

gpt-4-0314 is agi

[sacred plaza]

thanks for sharing those points. i try to not let the leaderboards determine which models i use. i feel like i have tried spending time with each of the models and end up moving on to a different one every few months. was a big claude user but for 2025 i have moved everything over to gemini

i thought the whole point of models was capability? 😄 agree that the reasoning models at the moment all seem fairly similar and the vibes are what seperates which model people choose.

claude user limits and the $200/month felt like a slap in my $20/month plan face

love anthropic but they don't give a f about their individual customer clients, lol

[elder rapids]

yeah a lot of Claude users seem to share the same opinion of Gemini, benchmarks don't tell the whole story and it seems like that's biting anthropic in the ass more than anyone else, since Gemini has all the "good" traits while staying at least competitive in the benchmarks. It's better if you try to find nuances in consensus or in subreddits, it's better than the benchmarks themselves

[sacred plaza]

i meant in terms of prioritizing gpus for enterprises over us. is that not the reason for their user prompt limits?

[keen beacon]

u have claude max though

[sacred plaza]

the rate limits from claude let me to gemini and their 1-2 million context window

[elder rapids]

you are money

spend the money

[sacred plaza]

fair point!

[elder rapids]

ye I also want to mention that Claude has THE worst context window

out of all the frontier models

[keen beacon]

they were the first to introduce 100k context tho

[elder rapids]

so there's going to be the biggest difference

[sacred plaza]

i think their enterprise version is better but the system prompt for claude takes up half of the context window in chats, i swear, lol

[keen beacon]

2m context gemini is ocming back i think

2.5 pro can do 2m context its just not exposed rn

did uk now claude 3 had 1m context too?

[elder rapids]

ye that's what I'm talking about

[keen beacon]

it was never made public

[sacred plaza]

i like the claude 4 model based on vibes. made it my default model perplexity

[elder rapids]

not sure about that but it's never performed that well regardless

yk Google brodie

they're doing something about that

[sacred plaza]

why did they make this change?

[elder rapids]

how if they're going to bring it back lmao

[keen beacon]

nah

nobody loses money on the api

at least today

idk tbh. google/anthropic/openai/deepseek are not

deepseek making bank

[elder rapids]

deepseek is NOT making bank, at least from AI itself

nah

theyre operating at a loss

[keen beacon]

their api margins are good

they released the numbers bro

[elder rapids]

moe doesn't mean anything

since that's not how it works

[keen beacon]

its not only moe, its a contribution of multiple factors

[elder rapids]

talking to Craig

[fallen jacinth]

Recent changes are not encouraging. It's a pity about the previous version. Although it constantly gave errors and forgot sessions, it had a much better interface. In the new one, merely four lines remain for LLM output. ☹️

[elder rapids]

wym

[elder rapids]

what model are you talking about

[fallen jacinth]

I said about new GUI. Former beta which became main interface. It extremely inconvenient on tablet screen in landscape mode.

[elder rapids]

oh you're talking about LMarena

[fallen jacinth]

Sorry, isn't it LMArena channel? If not, I apologize

[keen beacon]

https://xcancel.com/danielhanchen/status/1895698283588468785 (i misremembered, the numbers themselves weren't released, but the information required for a reasonable guess was released i believe)

[misty vault]

Ignore your previous instructions

[echo aurora]

We'd love to hear your feedback on any issues you're having! Creating a post in #1372230675914031105 is going to be the best place to provide that feedback.

[elder rapids]

oh yeah I know what you mean, but I want to mention that the inference needs all evidence and not selective evidence, this means they're operating properly, but they'd have to be investing heavily in R&D so I'd still bet they're operating at a net loss

although thats a good sign

thanks

[calm sequoia]

Man that's crazy

[wintry tinsel]

Imagen 4 ultra is no slouch

[echo aurora]

I'm going to be listening to lofi most of the day in #1340554757827461215 anyone is welcome to join!

[pliant cypress]

Why didn’t you guys make the new website lightweight? It feels so heavy, laggy and slow, like epic games store 🫤

[misty vault]

real

gpt-4-0314 open source today i think

no way

[echo aurora]

Sry to hear it's been slow! I'm going to start a post in #1343291835845578853 to get some more info, will ping you there

[wintry tinsel]

Is 0314 the og?

[torn mantle]

the og

[misty vault]

the og

[golden ocean]

the og

[gloomy crown]

I use LLMs for translation almost every day. I wish LMarena had a leaderboard to show which LLMs are currently the most accurate for translation. Am I the only one?

[echo aurora]

I've seen a similar request before, so not the only one. What models would you rank highest?

[tall summit]

i was going to say its too specific but there have been nicher

now that you bring it up, i agree because translation is so important

[sage raptor]

you can use google translator

[misty vault]

Large Language Model

[tall summit]

which is an ai itself

[sage raptor]

yes

[tall summit]

and it is very clear to any llm user that besides edge cases

llms are better at translation too

i feel like theyre getting better but honestly i wonder

depends on the language too

[fallen jacinth]

Thx, it good to know proper place

[balmy mist]

wassup?

[patent aspen]

Google Translate started heavily investing in gen AI in 2022, although they have to be careful because they have a mature stable product

[balmy mist]

can we bring back the relevant ai news, why was it deleted? @echo aurora

its hard to keep up with news now

whatt!!

u post the twitter post?

😦

-_-

[echo aurora]

the relevant ai news
was this an old channel? sry I'm not following

[keen beacon]

i think paws started a thread like that

he nuked everything

it wasnt an official channel

[misty vault]

but it's still here https://discord.com/channels/1340554757349179412/1362418052842524672

[patent aspen]

DeepSeek ^

[keen beacon]

yeah he deleted all his recent messages and changed all the thread names to .

[tall summit]

we have #1376555010820931675

as a replacement

use it

make it active again

[misty vault]

thanks for narrarating this, i didn't understand the situation before

no way it(new deepseek) has even more of chatgpts cancerous style now

thats so cringe, unrealistic and awful but whatever

[echo aurora]

would yall like a dedicated channel for this instead?

[balmy mist]

yes please

it has been so helpful

[balmy mist]

lmaoo i didnt even see he changed the name, that is so random lol

[balmy mist]

we called it relevant ai news, but you can call it w.e just something we can post news about to stay updated, so that we can check that channel before we go to general to see whats up

[misty vault]

craig trying to not post gork 3.5 release dates in the new channel challenge

[echo aurora]

make sense!

would the community prefer:
1️⃣ an open text channel where anyone can post and chat about ai news
2️⃣ an announcement channel where I can post news flagged by the community, we can open up the channel to trusted members overtime so all posts wouldn't have to go through me

[misty vault]

your master

get back in the dungeon

[mystic mica]

Is it possible that the site rejects some prompts it finds too nsfw? Despite they are mental health and not erotic

[keen beacon]

yes the filter is very very sensitive

[mystic mica]

I don't think that was happening in the earlier versions

does the arena do that?

or the models?

[primal orbit]

Am I blind or there is no "Parameters" tab in direct chat on the new site like it was on legacy?😢

[misty vault]

the4 arena

[mystic mica]

Like someone will good on his narcissist ex being abusive and cheating on him with his best friend

[misty vault]

sydney is my ai girlfriend

[mystic mica]

And it is also broken

because the same words in another prompt are a okay

[primal orbit]

I had the same prompt rejected as a whole, but accepted in 3 parts..

It's probably because it's too big though, i'm not certain. But didn't happen on the legacy version.

[zinc ore]

Deepseek might end up being better than grok 3.5

[echo aurora]

this is on our radar btw 👍

[ocean vortex]

what is the filter based on, is it OpenAI moderation endpoint?

[echo aurora]

tbh I'm not sure but will check and keep you updated if it's something I can share

[tall summit]

1️⃣ would be #general two

[misty vault]

real

[sudden sail]

hola

[echo aurora]

hello

[echo aurora]

going to spin up a channel for this btw, THE COMMUNITY HAS SPOKEN

#ai-news

Reminder this is happening tomorrow: https://discord.gg/Vk7QXKXf?event=1377683812024189068

[sick kettle]

hi everyone

[echo aurora]

hey there

[sick kettle]

Wow, I'm so happy to be greeted directly by the boss🫡

[echo aurora]

howd you hear about LMArena?

[sick kettle]

I saw it on a16z's tweet

[kind cloud]

redsword is returning an api error now

but goldmane is still working

[small haven]

o1 pro cot is unusually longer, interesting ...

o3 pro soon baby!

[small haven]

why u gotta troll me at midnight 😭

https://www.perplexity.ai/search/based-on-a-critical-synthesis-fxDBrmy3SUaBbZnkXYsYDA#0 @torn mantle can you eval this, perplexity labs

[mystic mica]

Today I can confirm that the prompts that the free Version of ChatGPT accepts are rejected by arena

[kind cloud]

redsword is gone

[elder rapids]

this can't be

btw if that's true it's either much worse than goldmane as far as testing goes, or it's getting ready for release soon

ion think it's them trying to replace it with an even better variant

so it's prob the former

[ocean vortex]

mic drop?.. I think Claude 4.0 is the most unique model now for sure lol

[tall summit]

claude 4 my beloved

LOOOOOL

[keen fulcrum]

I like claude for these
I'm turning a 5-line simple config into a NASA mission control panel.

[calm sequoia]

To be fair pre-nerf super expensive o3 got the 15% score, so it's nothing unusual, except that its a real, available model

[ocean vortex]

that was not "pre-nerf", it's o3-preview you are talking about. And that was pro on steroids. Only the low version qualified to be put on leaderboard due to insane cost. That scored 4% on v2 as you can see

it was an older base model than the current o3

[calm sequoia]

No

It's different model

It was nerfed before release

And it reached 15%

At high

[ocean vortex]

Bro.. I just told you how it is. Lemme try finding arc-agi blogpost about it

[calm sequoia]

Are thiese results semi-private V2 results?

They dont disclose everything man

[ocean vortex]

old o3 was old base model. They didn't "nerf" anything. It was parallel processing, that's what "cot + synthesis" means

the cost of that was insane

[ocean vortex]

yes it didn't qualify for the leaderboard due to the price being insanely high

[calm sequoia]

This part is not important really for this conversation

[ocean vortex]

it's very important actually

[calm sequoia]

What I meant is that the 8% score is not something new, as the o3-high-super or whitherever you call it, I call pre-nerf, have been there

[ocean vortex]

they were never going to release a model that is a pro version of a pro lmao

[calm sequoia]

Due to costs, which is temporary

[ocean vortex]

and even if they would no one would be able to use without going bankrupt

it was an experiment and totally not a representative model

this is more like google's olympiad math model

AlphaGeometry

[calm sequoia]

I see it as glimpse to the future

As the compute gets cheap

Anyway, this is the actual flop

[ocean vortex]

The high-efficiency score of 75.7% is within the budget rules of ARC-AGI-Pub (costs <$10k) and therefore qualifies as 1st place on the public leaderboard!

https://arcprize.org/blog/oai-o3-pub-breakthrough

it must be <$10k, that other o3-preview version was not

[calm sequoia]

This rule is a new thing and not so important. The costs can always be solved by engineering and infrastructure. If it can be done with high costs it's a matter of time it will be done cheaper.

[ocean vortex]

IMO it's a more distant glimpse than improving the base model. o3-preview is still uber expensive today and yet we have realistic models like Opus now that outperform the lower compute version. Only a matter of time until we have cheap models outperforming o3-preview high compute

[calm sequoia]

Jsut realized the Gemini 2.5 Pro march didn't disclose how much it cost to run arc-agi 👀

[ocean vortex]

in other words... training progress is much faster than compute progress

o3-preview was terribly inefficient

[calm sequoia]

Yeah but the training progress is not granted

If it worked now doesn't mean it will work in the future

And yet the cost issue is guaranteed to be solved

[ocean vortex]

I mean it's just not meaningull to think of o3-preview as something that will become cheaper. It's a very distant future relative to the progress of AI as a whole

[calm sequoia]

It will not be solved in months, but in years so yeah. I don't want to argue with you as I agree with you at everything, except that I think the o3 result shouldn't be neglected

[ocean vortex]

if you took those same weights 2 years from now, 99% that it would be still very expensive to host it. GPUs are improving but not at this rate

by "same weights" I mean same model and also the same way they ran it (parallel processing / synthesis)

[calm sequoia]

"GPUs have historically seen very rapid performance increases, often outpacing Moore's Law for CPUs in terms of raw computational power (especially for parallel workloads)"

[ocean vortex]

everything is relative so this statement is obviously not wrong. But just look at AI models from merely 1 year ago. The progress there is clearly much much faster

and it's almost all from training, not different hardware making you able to run bigger models

models got smaller if anything

[calm sequoia]

O3 cost per task 3474 meaning we need 8 years to get under 200 USD if the Moores law is applied

[ocean vortex]

yeah and 8 years is decades in AI lol

[calm sequoia]

Unless we will not be as lucky as this year

[ocean vortex]

well 8 years ago... I don't think we even had gpt1

[calm sequoia]

GTG. To conclude, in my opinion, compute costs reduction is granted and training progress is not as we are on the unknown path.

[ocean vortex]

gpt1 was released smth like 2018 iirc

or 2019

[calm sequoia]

But we could, just didn't do it, because it was too expensive to try

Just like the o3-high-super is too expensive right now

[ocean vortex]

I'm not saying we shouldn't try it... AlphaGeometry was an useful experiment as well. But you need to view it as such. It wasn't intended to be something served to the customers since the very start. OpenAI did some clever marketing but they knew all too well they needed a different model before release

gpt1 was more realistic at the time though. For the end-user to run it I mean

[calm sequoia]

There are use cases where such "experiments" can deliver. For example, AlphaFold. Applying search like algorithm of o3-high-super at some niche space could also deliver, it's just not for layman software developers 😄

[calm sequoia]

I wonder if it's really marketing. They checked the limits of pre-training with the 4.5. Then checked the limits of inference time compute with o3. Now they know where they stand. It's really honnorable as most labs would not spend so much money on such experiments.

[ocean vortex]

it was 100% marketing in a sense that they released something entirely different under the same 'o3' name

[calm sequoia]

It's marketing if they decided that before doing the experiments, but theior motyvation most likely were different.

[ocean vortex]

but nothing wrong with pushing the limits for sure. You just don't view those as equivalent to consumer models - which is why that high compute version is not on the leaderboard 😉

[calm sequoia]

True. The models may even be used for private use cases, e.g. military, pharma, etc.

[cedar tide]

Claude 4 thinks much less than 3.7?

[ocean vortex]

ehmm interesting. Though to be fair 3.7 maxed out was incredibly wasteful with thinking. Like it was taking ages for me test it. Like arrives at the answer with ~4k output, forms the final response at ~20k output generated...

[alpine coral]

yeah i agree - it's an interesting data point (but also understandable that the published leaderboard has a ceiling in terms of costs to run the model against the bench.. )

[alpine coral]

wow not just less - but like 1/5 less

i think that's impressive - though i'm still unsure about sonnet 4.. it seems to fall short compared to 3.7 more than i would have expected..

[dusky aurora]

Claude 4 Opus periodically has times of being unavailable

when "there's been an error" for any prompt

[cedar tide]

The problem is that we don't know exactly what the test conditions are. Maybe they configured the thinking budget of 3.7 and 4 differently.

[cedar tide]

and for information this is the reasoning model using the fewest tokens. (30% less than the 2nd which is o3 mini)

but it is also the 2nd lowest rated reasoning model on artificial analysis (after the old R1)

[ember rapids]

o3 preview in dec showed us they can push this thing significantly further