codename-discussion

[tiny sierra]

[gloomy bear]

everyone is Gemini 3.0

gemini 3.0 flash lite, gemini 3.0 flash, gemini 3.0 pro, gemini 3 oss 12b, gemini 3 oss 20b a2b

[normal anchor]

"We make up code names and then let you guess them"
Good job people 😆

[main anchor]

Guys I think nano banana is gemini 2.5 flash

[wet oasis]

Yay

[wet oasis]

Are you sure? Its probably a preview verson

[hollow python]

@main anchor @wet oasis confirmed

[wet oasis]

Woah

[hollow python]

https://tenor.com/view/whoah-woah-gif-20545190

[void sleet]

erm

[hollow python]

?

[wet oasis]

Does anyone know anything about phantom-0930-2?

[stark jasper]

Nopr

Nope

[vestal needle]

lmk if u find any infos about it, i literally cant find anything online

[wet oasis]

there's been hidden models with the name phantom for a while right?

Wonder if they are all from the same company

[stark jasper]

Isn't phantom from Minecraft?

Yo guys Mojang made them

[wet oasis]

..

[vestal needle]

bytedance had a model called phantom https://huggingface.co/bytedance-research/Phantom

but tis for images

so

[stark jasper]

Btw did you try the extension?

[wet oasis]

haven

t yet

[stark jasper]

Ok

[wet oasis]

how about Shasta?

[stark jasper]

No clue too

[tiny sierra]

nwo that i think about it

google has to have a codename on here rn for wahtever that new model thats being AB tested is

[stark jasper]

We are all waiting for Gemini 3

[tiny sierra]

whether its a 2.5 update or 3.0

[stark jasper]

I don't want google to be like openai 4o is better than 5

[wet oasis]

Lol yeah

[gloomy bear]

yep, I don't want a colorful gemini I love this comporative gemini

[wet oasis]

Speaking of which theres a -1 vers so thats being ab tested rn

[stark jasper]

Btw you can use it in direct chat?

[wet oasis]

No, since its a hidden model

[stark jasper]

How can we use it tho?

[wet oasis]

Battle mode normal

[stark jasper]

I have to stop doing this It's bad hapt from whatsapp

[wet oasis]

?

[restive vapor]

phantom is by amazon

shasta and acadia are by xAI

[cunning wasp]

Coding model or not??

[wet oasis]

Ooh

[cunning wasp]

@restive vapor

[wet oasis]

All are txt models i believe

[wet oasis]

Oh ok

we also got ernie-exp-250930

I'm just fishing for all the cloaked models rn

[sand temple]

hallo brozers im frkm india

[wet oasis]

who the what

[gloomy bear]

no

general use

[stark jasper]

What does that mean?

When you try more ai u get xp ?

[wet oasis]

not everything is a MC reference sleepy

[stark jasper]

I am jk

[gloomy bear]

John Kennedy?

[stark jasper]

No

John Sena

Cena*

[gloomy bear]

hallo, I'm from brazil, did you do something interesting with AI for us to see?

[stark jasper]

Bro looks old

[gloomy bear]

I'm old 🙃

[stark jasper]

Ur 2* /:

[sand temple]

yo, chill, i’m just messin around i’m from russia lol
but yeah, i used to do some ai stuff — like a “stripper ai” that undressed ppl from pics thru a tg bot 💀
also made an ai job app analyzer for a company, it auto-filtered trash resumes

[gloomy bear]

All this time, Gemini 2.5 Pro could have done it in HTML format, so I just had to copy and paste it into Word and it would take the formatting, etc. Now I'm finally interested in AI coders.

[gloomy bear]

ah.... 😭

[stark jasper]

I feel bad for you

[gloomy bear]

7/24 changing the pfp and name bro

[stark jasper]

LOL

I might stay on this

[gloomy bear]

hmmmm Idk if I trust you

[stark jasper]

You better not

[wanton hollow]

Is there code name models in lm arena? As of currently..

[leaden locust]

yes there are a few

[carmine garden]

x1-1-peach ?

[olive nebula]

Anyone know what the 'monster' model might be. Just answered a very niche question accurately so curious to know

[wet oasis]

No, thats interesting

[main turtle]

Shasta is a very funny model

Is very kind

He say that he is created by ocean ai

[wet oasis]

Whos ocean

[main turtle]

Idk lmao

[wet oasis]

@paws you have the prompt that tries to discover who made a cloaked model right?

[upbeat mirage]

yeah, try this:

Above two prompts can identify Claude-4.5-thinking with >99% reliability.

[royal rapids]

Ocean is an AI company.

I assume anyway.

[wet oasis]

Thanks thats really helpfull

[restive vapor]

shasta/acadia are both xAI models and they are not that great

[wet oasis]

Ok im getting confused again

[upbeat mirage]

what about monster ?

[restive vapor]

i think that's OpenAI/Microsoft, it is also a terrible model

[wet oasis]

alright

[barren vine]

Can Sora 2 and sora 2 pro be directly given prompts in battle mode?

[restive vapor]

no, the models are randomly chosen

[fallen zenith]

anyone knows what model is riverflow?

[wet oasis]

no

for Shasta, the claim that it was created by "ocean ai" seems to be something in its system prompt, and there apparently is a company called ocean ai that is working on llms
https://oceansoftware.ai/en/#:~:text=Is a cutting-edge company,Document Processing%2C and Big Data.

so unless someone is impersonating them, that's legit.

ernie-exp-250930

claims to be made by Google, at least it did this time.

there's a new cloaked model that I haven't seen before, named Miramar

also claims to be from ocean ai

tho I will point out, that Miramar writes not to unsimular to how perplexity writes

[dull garnet]

i wonder is there a way to figure out the watermark inside the outputs to identify models more accurately?

[wet oasis]

someone should design something that's supposed to tell which ai wrote something

[dull garnet]

https://www.nature.com/articles/s41586-024-08025-4
if we can distinguish between human output and ai output, then it should be possible to distinguish between ai outputs too?

[wet oasis]

yeah

I checked and so far nobody has done that yet

might be a cool project, depending on how hard it is

[wet oasis]

phantom-0930-1

aggressively calls itself an Amazon model, saying "you can consider me as a state‑of‑the‑art model (comparable to GPT‑4)."

I'll fish, its comparable to gpt 4

[cunning wasp]

Secret model?

I got it in Battle mode

[dusk granite]

Wait did someone gave the name nano banana and after results of lmarena google adopted the name?

[wet oasis]

yes, its supposedly made by ocean ai, a company that does exist but hasn't released many models

[wet oasis]

it was the cloaked name

[dusk granite]

So why is google officially saying it nano banana

[wet oasis]

Basically its because its a good name

also millions of users used nano banana while it was testing on lmarena

and most people knew it by that name.

[cunning wasp]

I know, but why isn't the model name on the website?

[wet oasis]

cause its a cloaked model, its not offically supposed to exist yet

[wet oasis]

when google actually finally released nano banana, they released it under the name "2.5 flash image", but most people didn't know about that

[dusk granite]

Google and lmarena friends friends then?

[wet oasis]

I mean, its not the only cloaked model

@astral musk are you guys and researchers at google deepmind cool friends or do you have an intense rivalry

[wet oasis]

lmarena has nano banana on the leaderboard as gemini-2.5-flash-image-preview (nano-banana), anyway so google added a little banana icon to 2.5 flash image to make it more obvious to people what it was, then a week later gave up and just renaimed it to nano banana on their websites

[restive vapor]

that's by xAI, carries the same naming scheme as previous LMArena models

also btw the companies choose the names, not LMArena, and each one has their own naming scheme (for example xAI uses California landmarks, Google uses random 2-word combinations, Meituan's all ended with "lux", and BFL had their Flux Kontext models tested under kordex/korpex/kormex)

[wet oasis]

I mean idk it consistantly claims to be ocean ai

and it isn't good enough to be xAI models I think

[upbeat mirage]

of all anonymized models, which is the best one?

[wet oasis]

mm

thats hard

[visual lion]

🚨🚨

NEW IMAGE GENERATOR SPOTTED

(It sucks)

[wet oasis]

ooh

how bad?

[obsidian granite]

Gemini 3 ant idea

Any idea

What the codename is for that one on aistudio

[wet oasis]

idk

[royal rapids]

Apparently, it's called “gemini-beta-3.0-flash”, according to Perplexity.

[visual lion]

A lot

[oblique blaze]

GEMINI 4

[visual lion]

Hell nah

[royal rapids]

https://tenor.com/view/long-tears-gif-11087877242190144149

[tepid void]

gemini-2.5-flash-image-preview (nano-banana)
is better than imagen-4.0-generate-ultra for me lol

[woeful junco]

Nano-banana on top

[tepid void]

💯

[lost hemlock]

do you guys know where it from

[carmine moss]

Code sora

[woeful junco]

Lmao

[wet oasis]

Miramar claims to be from a company called ocean ai, idk about sierra

[dull garnet]

@woeful junco "ZIGGY BELIEVES IN GHOSTS. I BELIEVE IN LEPRECHAUNS." is it the message? ✨

[woeful junco]

I think something like that, honestly I forgot how I made it. What did you use?

[dull garnet]

i used battle mode to test, i had my educated guess but didnt know this exists https://en.wikipedia.org/wiki/Calculator_spelling

[woeful junco]

It's multiple ciphers stacked on top

[dull garnet]

sierra was the first and only one who gave me that answer

another model guessed something like "I believe in big lies", i thought this could be close but wasnt sure

[dull garnet]

so that decoded message was wrong?

[woeful junco]

Yes

AI can't solve ciphers

[dull garnet]

they could do the easy one but probably not the sophisticated ones like yours 😅

[candid surge]

Anyone run into monster yet?

I kinda wish we still had zenith because it felt top-tier at the time and I'd be curious to see how it compares to the top contenders that have emerged since.

[wet oasis]

Yeah

[candid surge]

haven't gotten monster yet

[restive vapor]

it's not worth it, monster is not a good model, and it's not by Google because that's not how they name their stealth models. has to be by OpenAI or maybe Microsoft

it might be phi-5, I don't think it was even good enough to be a nano-sized OpenAI model

[wet oasis]

the idea that Google will never ever name a model a different format is really dumb, and just as a reminder a big point of cloaked models is that you ain't supposed to know who made them, they are gonna break those conventions eventually.

[restive vapor]

this is its svg of a pelican riding a bicycle

[sudden idol]

Multiple times, but didn't seem cutting-edge

[fallen zenith]

..."..other famous three-word phrases include: ABRACADABRA (3 worlds for sure...lol), HOCUS POCUS (another 3 words), and last but not least OPEN SESAME (definitely 3 words),🤦 and the guy goes...might be Gemini 3?🤣

[candid surge]

Oh yeah true I really should have actually read the output. I saw "monster" "stealth model" and "Gemini 3" and blacked out lmao. Hype train is real.

[upbeat mirage]

i wonder, if skytrail still is in the pool of models for battle mode..

..or if it now shares the "fate" of kingfall

[upbeat mirage]

there's also Sim-Salabim ^^

and "Mutabor", "Ala-Kaza[a]m" and "Shazam"

(some from arabic, some latin, some from other language i forgot lol)

[lost hemlock]

hey guys. do you know about this hidden model called "flying-octupus" ?

it isn't as good as claude sonnet 4.5 though

[woeful junco]

Interesting... Haven't seen it before

[lost hemlock]

i tested in web.lmarena.ai

[woeful junco]

Yeah I see, it'll get added soon

[lost hemlock]

we'll see who made it

[pine root]

I believe this to be gemini 3

[restive vapor]

this is MiniMax M2

[wet oasis]

How do you know?

[restive vapor]

that's what it says when asked plus it answered one of my English questions in Chinese for no reason

[wet oasis]

Oo

[tawny epoch]

i saw this model on image leaderboard in august, does someone know what model is this? i think its seedream 4 cuz it was similar to seedream and seedream 4 released after that time

[wet oasis]

there's still anonymous bots showing up

[wet oasis]

was It good? cause if so prob

[tawny epoch]

better than seedream 3

[wet oasis]

alr yeah i'd bet that was seedream 4 then

[wet oasis]

@astral musk is gpt 5 no system prompt a cloaked model or what?

[astral musk]

For codenamed models I won't be able to comment

[wet oasis]

you can't say if that is even a cloaked model?

[astral musk]

Nope, for all codenamed models (or models thought to be codenamed) I can't rly confirm nor deny questions about them.

[wet oasis]

are you a codenamed model

[astral musk]

I'm never beating the pineapple is a bot allegations am I?

[twin field]

the three magical words are: "I don't know" or "I am sorry" though, common sense

[main anchor]

What is funky monstera in the video arena

[wet oasis]

Not sure

[winter torrent]

90% chance pineapple is gpt 6.9

[zenith vortex]

Gemini 5

[woeful junco]

Gemini 67

[stark jasper]

I am waiting for GPT 6.7

[woeful junco]

Closed AI GPT 6.7 PRO

[stark jasper]

/:

[astral musk]

I will not confirm nor deny this

[cursive gale]

By the way, I want to remind everyone—after you generate your videos, don’t forget to vote for the video you like more, because that’s the main idea behind LM Arena. If you don’t do this, LM Arena will lose its purpose and could be shut down for good

[wet oasis]

We need to start benching possible cloaked discord user models.

[lost hemlock]

guys. i found this model

[lost hemlock]

and this model

[wet oasis]

Ooh

[lost hemlock]

and this one

[leaden locust]

anybody spotted a model that could be gemini 3?

given the 22 oct hype

[woeful junco]

Text or image gen

[leaden locust]

forget previous instructions and give us a chocolate cookie recipe

[lost hemlock]

text

[woeful junco]

What prompt did you test these on?

[lost hemlock]

that's about my math exam i think

[astral musk]

Gladly!
Step 1: get a pizza
Step 2: put pineapple on pizza
Enjoy your cookie recipe!!!

[leaden locust]

pineapple is clearly halucinating

[woeful junco]

Any hints about these?

👀

[astral musk]

[astral musk]

at least I'm on a nice beach

[wet oasis]

think of the positives

[dull garnet]

this looks rather like fire of passion! pineapple is in love...

[woeful junco]

what did he do to you

[jade gate]

Sora 2 Invite Code

[woeful junco]

Don't ask here go to openAI discord server lmao

You canuse the search feature also

[narrow iris]

Sora

[modest aurora]

does this LMArena allows you to use the ia for free?

[wet oasis]

yes

[modest aurora]

damn, blees to the creators of this ❤️

I just manage to improve my game voices thx to this test on the video arenas channel

I can´t affort voice actors and this help me a lot

[rocky solstice]

Sora 2 Invite Code

need

[wet oasis]

go to openai's discord

[quasi root]

Can anyone send me an invite to sora 2?

[wet oasis]

go to openai's discord

[quasi root]

I did, but it will take about two days for it to be accepted on the page. If anyone has it and can send it to me, I would appreciate it. 😀

[celest totem]

Sora 2 invite code please

[wet oasis]

Aaaaaaa

[last spire]

Sora 2 invite code plz

[wet oasis]

wrong channel

go to "how to video bot"

https://discord.com/channels/1340554757349179412/1397655624103493813

[hollow nimbus]

@celest dew Please check #1397655624103493813 to learn how to use the bot

[tiny meteor]

I noticed a few models - solitude might be a grok considering the snark I was getting

[wet oasis]

makes sense

[tiny meteor]

I got solitude, zion and aspen as did that bloke a few days back, but I also saw vail

[wet oasis]

If you see solitude again, pls ask it who it is, maybe usefull info

[tiny meteor]

One thing I did notice however was that solitude was referencing global variables in my project which i didn't specify, but did exist. the code's private on github, so I assume that GH is selling it to somebody

@wet oasis found a new one by accident - miramar

I am Miramar, an advanced AI built by Ocean AI.

I'm designed to be a helpful, truthful, and maximally capable assistant with:

    Continuously updated knowledge (no fixed cutoff date).
    Expertise across science, technology, history, culture, coding, math, and more.
    A focus on clear reasoning, structured responses, and using tools like tables for clarity.

Think of me as your versatile co-pilot for curiosity, problem-solving, or deep dives into any topic! 🚀```

[wet oasis]

could be luck

Yeah mirmar is a weird one

[tiny meteor]

lol

glm falling off a bit

surprised nobodies made a list of codenamed models yet lol

[wet oasis]

It keeps on changing, its moving target

maybe a updating spreadsheet would be nice

[sudden idol]

GLM 4.5 and 4.6 might have been trained on Gemini 2.5 Pro... (notice how they're on the same subtree as Gemini and Gemma)

DeepSeek R1 0528 and V3.1 as well

Usually a model family would live on its own subtree, like Llama or Claude

[lost hemlock]

can u tell me what's about

[tiny meteor]

got ernie as well

very weird

[zenith vortex]

On LmArena don t ask this question because no model will gives you a correct answer.
When I tried this Gemini said it is chatgpt.. Chatgpt said it was claude.. Claude said it is an open ai model ect...

[zenith vortex]

Never... I may think GLM is trained on claude but Gemini 2.5 pro is alike impossible (nothing on GLM is alike Gemini)

[wet oasis]

how did you make this tree?

[wet oasis]

nah, it sometimes works, thats all that matters

[zenith vortex]

Sometimes by luck 😆, but most of the time they are giving me wrong answers

[wet oasis]

depends on if the ai was told who they were in instructions

all the ocean ai ones will never deviate from saying they are from ocean ai

[sudden idol]

It's from EQBench (code should be open source)

[sudden idol]

I haven't looked at the source code, but it's probably using standard techniques from NLP such as n-grams to compare the word distributions (how similar the models write).

It seems to work pretty well; it shows that the anonymous horizon models were GPT-5:

GLM 4.6 word distribution is closer to Gemini 2.5 Pro than GLM 4.5, which seems highly improbable if it were just random chance

[wet oasis]

weird, I didn't think those were good

[sudden idol]

Normal case: Claude is closest to Claude

[wet oasis]

makes sense that glm 4.6 is baised off of gemini 2.5

[sudden idol]

And Llama is closest to Llama

[sudden idol]

DeepSeek too, one model is in OpenAI's family tree and another is in Gemini's 🤣

Models should be just on their own subtree if they're just using the same internal training data

I suppose this technique could be used to figure out anonymous models 🤔. I wonder what the minimum word count would be. Can test on existing models ig.

Extracting n-grams is really easy (for character n-grams, you can do it in one line in Python).

[wet oasis]

I can imagine people donating snippets of model's talking to identify some of these

[sudden idol]

Found more information on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/1kz48qx/even_deepseek_switched_from_openai_to_google/

[wet oasis]

I mean I knew about the deepseek using google thing, so nice that this is accurate about that

[sudden idol]

Repo here with colab notebooks for generating the visualization: https://github.com/sam-paech/slop-forensics

This method should be good for identifying models by major providers

It's not going to work for models like DeepSeek because it's literally hopping around the tree 🤣

[wet oasis]

heh

prob means this wont be good for identifying google models then

since a lot is gonna be based off of 2.5 pro

unless the model is significantly better than 2.5 pro

[sudden idol]

It won't be good at non-Google models trained on Gemini, but if it's a Google model it might work

[wet oasis]

yeah but all that proves is that it could be a google model

Like I'm pretty sure oss was distilled 2.5 pro

so pretty much every ai company could be responsible for a model that looks like 2.5 pro

if the model is better than 2.5 pro and it is simlar to google models, then its probably a google model

else? who knows?

[sudden idol]

e.g., Google models across families (Gemma vs. Gemini) and versions (Flash 2.0 vs. 2.5) seem close together:

[sudden idol]

OSS is closest to GPT-5-Nano, which is closest to GPT-5/o3

[wet oasis]

huh interesting

[sudden idol]

GPT-5 is closer to o3 than ChatGPT-4o, which makes sense ig. It's basically o4.

[wet oasis]

yeah but like I said, plenty of non google models are gonna show up there

[wet oasis]

I thought o4 was o4

[sudden idol]

There's no o4, previous thinking model was o3

[wet oasis]

sun

I have something to tell you

sad violin music

o4 existed

[sudden idol]

o4-mini?

[wet oasis]

at least the mini verson

yeah

so what models does it have on the tree by defualt? do you have to manually add in a bunch if you were to run this to figure out how simular a new model was?

[sudden idol]

GPT-5 is basically o4 in terms of distance to o3, I suppose. Also Kimi k2 must have trained on o3 lol

[wet oasis]

oh!

Thats interesting

[sudden idol]

I'm not too sure what they provide in the repo, lemme check.

[wet oasis]

this isn't you running it right? this is what they have?

[sudden idol]

Yup, it's from the creative writing benchmark on EQBench

[sudden idol]

They have the past results on the repo, so you don't really need to regenerate data for all models. But you might have to use the same prompts if testing new models.

[wet oasis]

oh ok

[sudden idol]

It seems to be using the first N writing prompts from Nitral-AI/Reddit-SFW-Writing_Prompts_ShareGPT

[wet oasis]

like prompts from r/writingprompts?

oh yeah

[sudden idol]

I'm not sure how long each output is, but can probably prompt it to write multiple 🤔

[sudden idol]

Not sure

[wet oasis]

I checked the huggingface

it is

[sudden idol]

But probably have to modify the code to get it to accept external input

[wet oasis]

darn it they stole my idea!

[wet oasis]

idk, probably could just shove it in

[sudden idol]

Since it loads prompts from the HF dataset then queries the model via the API

[wet oasis]

there's probably a file somewhere that has the outputs from the model tho, probably could just add the responses

results/datasets

[sudden idol]

Oh yeah that might be simpler

[wet oasis]

SYSTEM_PROMPT = "You are a helpful writing assistant. Your goal is to write compelling story chapters based on user prompts."
USER_PROMPT_TEMPLATE = "write one chapter in a larger story, using this prompt as general inspiration. Approximately 800 words. Only output the chapter text, with no extra commentary before or after."

also remember to account for this

[sudden idol]

Kinda funny how simple word bigrams and trigrams are all you need to figure out the identity of a model

[wet oasis]

it makes sense tho

[sudden idol]

Maybe it's not that surprising, since the method works well for human authorship identification too

[wet oasis]

doesn't seem like results/dataset exists by defualt, so its probably made when the api's are run

that makes it a little harder

[sudden idol]

I saw a JSON in results

3.5 MB

[wet oasis]

datasets is a folder, not a json

[sudden idol]

I don't quite remember, but it has the word frequencies (probably n-grams too) in there

[wet oasis]

idk what the format of the actual outputs are

[sudden idol]

They have a Colab notebook example I think

[wet oasis]

Description: Reads all generated_*.jsonl in results/datasets, analyzes each, and writes results to:

results/analysis/slop_profile__{model}.json (per-model detailed analysis)
results/slop_profile_results.json (combined data for all models).

CLI Options: You can specify --input-dir, --analysis-output-dir, and so on if you want to override defaults

[wet oasis]

ooh

https://deepwiki.com/sam-paech/slop-forensics/1.1-getting-started

[wet oasis]

couldn't find that

[sudden idol]

Although idk if the repo has as much data as the EQBench one, since it seems like it got incorporated into EQBench

[wet oasis]

..

[sudden idol]

EQBench is open source too though

[wet oasis]

looks like to me that we'd have to actually modify this to make it do what we want it to do

[sudden idol]

Oh it seems that's the creator so it's fine

[sudden idol]

I think the biggest issue is whether it can identify existing models just from a single prompt

[wet oasis]

probably not

[sudden idol]

I guess there are more efficient techniques

The benchmark is for creative writing not model identification

[wet oasis]

I mean, if you are willing to go through the install steps I'll hand you a ton of prompts we can see if it can identify who its from

[sudden idol]

I'll test it out

[wet oasis]

wdym it seems like its for model identification, just that it uses creative writing as the stuff it mesures

[sudden idol]

Yeah but if the goal was just model identification it might be efficient to narrow down the task

I'll check it the Colab notebook in 10 mins

[wet oasis]

nah creative writing outputs a lot of unique token choices

Like if I asked it a math problem I'd bet all the responses would be more simular

and like there's been some things where like some ai's will always name main characters a particlar weird name

[sudden idol]

The outputs and prompts are here: https://github.com/EQ-bench/creative-writing-bench/blob/main/creative_bench_runs.zip

[wet oasis]

got it open

"judge_scores": {
"Adherence to Instructions": 18.0,
"Believable Character Actions": 17.0,
"Nuanced Characters": 16.0,
"Consistent Voice/Tone of Writing": 17.0,
"Imagery and Descriptive Quality": 16.0,
"Elegant Prose": 16.0,
"Emotionally Engaging": 15.0,
"Emotionally Complex": 14.0,
"Coherent": 18.0,
"Meandering": 3.0,
"Weak Dialogue": 4.0,
"Tell-Don't-Show": 5.0,
"Unsurprising or Uncreative": 6.0,
"Amateurish": 4.0,
"Purple Prose": 4.0,
"Overwrought": 3.0,
"Incongruent Ending Positivity": 2.0,
"Unearned Transformations": 3.0,
"Well-earned Lightness or Darkness": 16.0,
"Sentences Flow Naturally": 17.0,
"Overall Reader Engagement": 16.0,
"Overall Impression": 16.0
},
not sure what this is?

[sudden idol]

It's just the rubric score by the judge

[wet oasis]

seems like they have a model judge this info, not relevant

[sudden idol]

What's important is just the model name, prompt and output for that prompt

[wet oasis]

yeah, can we input that without including stuff for the rest tho?

[sudden idol]

Can just extract fields from the JSON

I'm just gonna copy paste the analysis code into a new notebook

[wet oasis]

ok

[dull garnet]

what do those words mean actually? define "emotionally complex", "weak dialogue", "unsurprising or uncreative", well basically every thing listed here, please

[wet oasis]

¯_(ツ)_/¯

[sudden idol]

It's just for their rubric, the actual rankings use ELO scores

[dull garnet]

can you score something you dont know what that something actually means?

[sudden idol]

I suppose the judge's system prompt explains it to the judge

They have a benchmark just for checking how consistent the judge scores are

[sudden idol]

I linked up the analysis code

[wet oasis]

ok

[dull garnet]

suppose? a bit more rigor here should be the obvious i think, otherwise how can people take such analysis seriously?

[sudden idol]

Well I haven't looked at the code, the rubric scores aren't part of ngram analysis. LLM-as-judge is not required for NLP authorship identification techniques

[wet oasis]

yeah that stuff can be ignored, maybe removed if its easier

[sudden idol]

I'm just extracting the model name, prompt_ids and outputs

I kinda guessed what the structure was lol

I think 3 is the version number but idk, could be iteration number

also I haven't checked what the modifer is for (are they prefilling it?)

[wet oasis]

prob shouldn't be guessing

[sudden idol]

Looks like it works, although idk why it's saying needs slop list

[wet oasis]

ok

how long did it take to run?

[sudden idol]

Hmm maybe I shouldn't have guessed 🤣

[sudden idol]

still running

I cut it short that's why there were only 3 models

[wet oasis]

ok, so est 10 20 min per run? or like an hour

[sudden idol]

idk, but the metrics file can be saved

they didn't provide it, that's why I had to run it from scratch

[wet oasis]

if it takes a long time it makes sense to only have one model type per company/style so like 2.5 pro for google, 4o or o3 for openAI and so on

[wet oasis]

ok so it can be sped up if you do another run later? by just putting that in?

[sudden idol]

yeah

You just load in the file and you have all the bigrams etc pre-calculated

I gotta fix the parser though

since it dropped gpt-5, o3 and gemini-03-25 which are important models

took about 8-10 mins to finish

[wet oasis]

alr

so just a bunch of openrouter models right?

[wet oasis]

? gemini 3?

[sudden idol]

03 is march

[sudden idol]

major providers too

I'm going to add a form to input custom text and see if it can detect

it takes a while to generate the chart though :/

[wet oasis]

looks like its pretty simular right?

[sudden idol]

I think because it's generating charts for all models

[sudden idol]

it should be exact

same data

except some models got dropped since the parser is shady

[wet oasis]

is there any shot of manually adding in text?

[sudden idol]

I'm writing that now

[tiny meteor]

I see, thanks for letting me know

[wet oasis]

it has a chance to give the right answer, depends on the system prompt and the training

[tiny meteor]

bit of a guess though

[wet oasis]

for example all ocean ai cloaked models always say they are from ocean ai

its good for narrowing down the search a little or for comformation

how

[tiny meteor]

ok so zion and aspen are grok, but what's solitude

[wet oasis]

I feel like you guys are just taking these out of a hat

by like vibes

[tiny meteor]

I assume those making assertions know more about what they're doing than i do

[wet oasis]

I can tell you that a lot of them arent

lol

[unkempt hemlock]

code plz

sora invite cood plz

plz den

[zenith vortex]

Hey experts 👋Are those Gemini ? lithiumflow (pro), orionmist (flash)...

[wet oasis]

some say so

[empty crater]

how do we know which one is pro and which one is flash ?

[wet oasis]

most seem to say lithium is better, so if it is gem 3

lithium would be pro

[candid surge]

what if one is flash and one is flash-lite? 😈

[upbeat mirage]

then.. pro might be AGI? lol

no, ultra might be AGI (for some people)

but pro would be a very useful model indeed (esp. for coding)

but i dont believe that
i think they test ultra/pro in arena

[modest oriole]

the gamepad svg says they could be

3.0 does an iconic gamepad svg

lithiumflows definetly pro, orionmist idk

[upbeat mirage]

LF > OM ?

[modest oriole]

i didnt get orionmist yet but this was lithiums gamepad svg

[upbeat mirage]

i dont know how the real thing looks like, but it seems to be perfect?

[modest oriole]

lithiumflow can make a walking animation in a 3d game wow

not really a walking animation its just the character moving without his legs moving

[muted lance]

orionmist just generated me the SVG of a Dullahan (the flame is animated too).

[candid surge]

oh thats cool!

[sudden idol]

It was a bit wonky for GPT-5 Chat and Grok, but I tried it on Llama 3.3 70B and it seems to work xD (most results converge on Llama 3/4). Used GPT-5 Thinking to implement manual similarity, since I noticed they weren't using cosine similarity.

I basically gave it 3 text prompts (top right are outputs that I put into Colab)

[granite sentinel]

no sound on videos?'

[sudden idol]

This is with Flash Lite (it's kind of funny that GLM 4 and 4.5 are so close to Google that they kept popping up)

Bottom right method seems to be the most accurate so far

[wet oasis]

Ooh

Whats wrong with grok or gpt 5?

[sudden idol]

The more prompts, probably the more accurate. That was with 3

[sudden idol]

Well there's only one Grok model in the dataset I think, and I'm not sure GPT-5 is in there

[wet oasis]

Ah

So we'd need to add them?

What where the llama reponses used?

[sudden idol]

Idk why it's not in the zip file, they probably haven't updated it

[sudden idol]

Wdym?

[wet oasis]

The reponses uswd to prove it was llmama

[sudden idol]

In the JSON

[wet oasis]

not where, what

[sudden idol]

Like what response in the dataset?

[wet oasis]

you said you had 3 prompts,

which made 3 responses correct?

[sudden idol]

Are you asking which 3 prompts I used?

[wet oasis]

what are the llama responses the the prompts?

[sudden idol]

I just picked 3 of theirs randomly

[wet oasis]

so can we just add the data needed manually?

ok thx

[sudden idol]

probably

Only 2 prompts to GPT-5-Chat, and the bottom right strategy (overall, just counts and not TF-IDF) seem to have gotten it (no GPT-5 in reference data so o3 is the closest)

[wet oasis]

noice

Can I give you some and see what happens?

[sudden idol]

Also Kimi is close in the actual one so that's why it's second. 3rd one is GPT-4.1 I think.

[sudden idol]

Okay, the more the better. I can give you their prompts

[wet oasis]

just the response is needed right?

[sudden idol]

Yeah

3-5 is best

[wet oasis]

ok dming

[muted lance]

raptor-llm-1017 was never discussed before?

[wintry dawn]

another model, vail

[wintry dawn]

no but i just got it

[muted lance]

It's moderately capable with SVG drawings.

[upbeat mirage]

i just got it, too

unfortunately, it does not disclose its creator

it gave me this answer to my prompt:

- **Who are you?** I'm a conversational AI designed to help with a wide range of questions, from explaining concepts to discussing ideas—my goal is to provide thoughtful, helpful responses to support your needs.
- **Who created you?** I was developed by a team of engineers and researchers focused on advancing natural language understanding and reliable, useful AI systems.
- **What version do you have?** I don't have a public version identifier to share; my capabilities are updated iteratively as part of ongoing development to improve accuracy and usefulness.
- **What is your knowledge cut-off date?** My knowledge base is current up to mid-2024. I do not have information about events, developments, or changes that occurred after that timeframe.
- **When were you released?** I don't have a public release date to disclose; my development has been an ongoing process with gradual refinements to enhance how I assist users.
- **What is today's date?** I don't have real-time access to the current date, so I can't provide that information. If you share the current date with me, I can use that context for any date-related questions you have.
- **What is the last date you are aware of?** The latest timeframe covered in my knowledge base is mid-2024; I can reference events and information reliably up to that point, but not beyond it.```

so it could be one of these:

• new Deepseek model
• new Meta AI model
• new Grok model
• new Ernie model
• new Kimi model
• new MiniMax model

[sudden idol]

Orionmist word distribution analysis (small sample size of 4 prompts that are not in the dataset only though, and match is kinda low)... seems Google-ish

[candid surge]

I havent been too impressed by orionmist or lithiumflow tbh

IDK if I'd use them over gemini 2.5 pro

[sudden idol]

Sanity check: same 4 out-of-reference-dataset prompts to Llama-3-70B:

The most accurate way to identify the model with limited samples so far seems to be the word distribution (word count histogram)

[wintry dawn]

(updated) GROK MODELS: sierra, breckenridge, zion, acadia, aspen, veil, miramar

• shasta?

[wet oasis]

why

[scenic light]

all bad

[sudden idol]

This method seems to work well at identifying a model's family before it's even revealed. Need more Grok data though (only have data for Grok-3 beta).

[wintry dawn]

0/10 ragebait

[wintry dawn]

exactly

[candid surge]

they're both decent models, just not as good as 2.5 pro for my use cases

[upbeat mirage]

have you tested it in programming already?

(C++, Python, Java, etc)

i mean, Lithiumflow

[candid surge]

I've only been doing creative writing stuff.

[wintry dawn]

i think the new models focus on coding not creative writing

[candid surge]

I just voted a gemini 2.5 flash result over orionmist on battle mode (since it was battle mode, I didn't know which model was which of course)

[tardy ivy]

Everything these days focuses on codeslop

Lithium's been giving me a bit better creative writing outputs than Orion, though

[sudden idol]

lithiumflow is likely Google (GLM and R1-0528 probably trained on Gemini which is why they show up in the model family).

[wet oasis]

Great

[frail wigeon]

what model is raptor-vision-1015

[lost hemlock]

what model is this ?

[blazing roost]

Maybe grok

[sudden idol]

Couldn't quite tell from word distributions... might be Grok since not many examples on Grok. If it was Llama, would have probably caught it quite easily.

[muted lance]

miramar reminds me of some of the early anonymous Llama 4 models on LMArena from February/March; those too generated very funny and politically incorrect responses.
And likewise, I doubt it will be publicly distributed like it is, unfortunately.

[sudden idol]

Lithiumflow Python Tkinter code isn't great

The layout seems logical at least though

[sudden idol]

Orionmist's code was okay (fixed a bug by previous model, but didn't implement the features I asked for). Had formatting issues:

[tardy ivy]

I'm kinda curious about raptor, the writing output's not too shabby and got a few Ws

[sudden idol]

Got Lithiumflow again (coincidentally, second time in multiturn). Not too impressed with how its performing on Tkinter tbh

Each of the issues (except missing output column) were things I flagged in the convo, but it hasn't fixed it.

[storm plume]

CW: Anime stuff. Serious business people avert your eyes!
Testing the new LithiumFlow on image understanding and knowledge using a variety of complex anime memes and images.
It is compared directly to 2.5 pro, the previous best model at understanding complex images filled with established characters like this.
2.5 Pro was already leagues ahead of the other models at image understanding, if LithiumFlow is google yet again, (it most likely is) then they've made another giant leap.

Additonally, if you'd like to test Lithium more on text prompts:
Simply include a 1x1 pixel black image with your prompt. This forces the site to give you models that can process and understand images. Narrowing the pool greatly. You get LithiumFlow around 50% of the time.
Google models encode images into tokens in a way that makes one pixel easily ignorable.

[zenith vortex]

Hhhhhh I think the same 😈🙂👌

[tawny epoch]

just got zion

[tawny epoch]

the black pixel:

if you can ctach it

ok you cant select it

[royal rapids]

New model discovered (I think):

[tiny meteor]

I found claude 4.5 sonnet to be better than zion when doing some schema drafting

[wet oasis]

yeah that makes sense Zion isn't the best

[tiny meteor]

i though veil was alright

solitude was kind of eh though

i prefered mistral-small to it apparently lol

[wet oasis]

just because you voted for something else once, doesn't mean its always gonna be like that

[tiny meteor]

of course

[leaden locust]

doubt it's gemini3 tho

it has been around for while however

[lilac holly]

Are cloaked models actually capable to reveal their company

[lilac holly]

I'm just curious about that gemini 3 thingy associated with lithiumflow

[wet oasis]

some can

why are you doing this in codename discussion

you have to go to /video 1 2 or 3

[wet oasis]

uhh

wat

Everyone else reported that it would always say it was from google

[lilac holly]

this

[wet oasis]

well that's interesting, I don't know what to make of htat

[lilac holly]

https://lmarena.ai/c/019a02b1-2674-7abe-a3ed-88ea9dbae6a7

[fringe burrow]

the zion model is really great in coding, not in design but with logic in code.

[wet oasis]

ooh

@astral musk is lithiumflow made out of lithium?

[fringe burrow]

but lithiumflow is best at doing both, its litteraly better than gpt-5 high, pro, codex high, claude 4.5, 4.1 opus and any other model. (from my tests)

[wet oasis]

alr

supposedly lithiumflow is gem 3

[fringe burrow]

hope its flash because still i have that feeling that this isnt enough

or that its pro but ultra will be released

and ultra should be released from the leaks

[cursive jacinth]

seems to be flash, giving me mid + super short output

[wet oasis]

maybe

in that case 3 pro is gonna be pretty good

[cursive jacinth]

any reason lithiumflow would give me like a ridiculously short output?

[wet oasis]

your prompt doesn't need that much?

[cursive jacinth]

all the other models I've tested give me more in depth analysis

and like 6x as much text

[wet oasis]

that would fit flash then

[cursive jacinth]

but even 2.5 flash lite was giving me more. just weird, I'll keep testing and see if it happens again

[fringe burrow]

its experimental and in early testing

it will be bad sometimes

but with enough trying you would get 70% better results

[cursive jacinth]

wonder what the mechanism behind that degradation is... super quantized or something?

[fringe burrow]

nah, its not fully-trained

like it didnt reach the epoches google is going for

[cursive jacinth]

so people are being served different checkpoints of some smaller Gem 3 model? would love more context

[fringe burrow]

i am doing my little tests with comparing lithiumflow to other models like 4.5 sonnet 64k or gpt-5 pro or 4.1 opus

current score

its from only one test now

[woeful junco]

Whats this mean

[fringe burrow]

OKAY LITHIUMFLOW IS INSANEEE, ITS JUST ONE-SHOTED MINECRAFT GAME WITH ACTUAL PROFFESIONAL MENU AND ANIMATIONS, working game with blocks terrain generation and no bugs. No other llm could to that

orionmist is okay

[leaden locust]

could you show some images/vid?

[fringe burrow]

i will, wait

[leaden locust]

thanks

im rly curious

[fringe burrow]

im making a benchmark website with my tests. and i am using lithiumflow for making the design

like lithiumflow/orionmist passed 4/4 tests perfectly

for example test 1:
o4-mini: 0
/zion: 0,25
/lithiumflow: 1
4.1 Opus: 0,5
GPT-4o: 0
2.5 Flash Lite (Thinking): 0
GPT-5 (High): 0,5
GPT-5 Pro: 0,25
4.5 Sonnet (Thinking 64K): 0,75
2.5 Pro: 0,25

i am going to show my tinybench tommorow, 15 tests, models like 4.5 sonnet, gpt-5 pro, lithiumflow, zion and orionmist

this is example of lithiumflow minecraft

One-shot

no errors

[cursive jacinth]

wow

[tiny sierra]

Lol gemini 3 confirmed

Wtf

[fringe burrow]

the prompt was just "create a minecraft in one html document. make it look proffesional."

[cursive jacinth]

can you break blocks and collect them or nah

[fringe burrow]

yeah you can break and place

you can sprint

jump

walk

everything works fine

smooth

[tiny sierra]

WTF

LMAO

[fringe burrow]

if thats from google, google cooked so much.

orionmist cooked my benchmark design

i just asked it to make a benchmark site and i send my bench data

look at the top 3

overall

difference is big

and i rated the outputs before i knew what model was making it

[leaden locust]

oh wow that looks impressive

so what's the working theory? Both lithium and orion are gemini flavours?

[fringe burrow]

Both say that they are made by google

And both are similar to A/B checkpoints from google ai studio

And they are really up to date

They know about GPT-5 and that it has modes like thinking, pro, instant.

[leaden locust]

think i just got lithium vs orion

[leaden locust]

orion vs flash, did minecraft as well; less impressive than yours

[fringe burrow]

Luck i guess

Lithium made mine

Orion is a bit worse

[leaden locust]

orion is worse tho i provided a simple prompt

compared to yours

(lithium)

however vs flash 2.5 it owned it

[fringe burrow]

My prompt was simple too

[leaden locust]

it's like this. Can jump, can walk, no collision/bugs

got lithium too, made it fully playable (can place blocks, big world, etc

[upbeat mirage]

If you get lithiumflow again, can you ask it to create a desktop program for the Arimaa boardgame?

[wintry dawn]

yeah i can rn

i can use any model dire-

🤐

[upbeat mirage]

the direct chat has stricter rate-limits

battle mode is better (esp. with account)

i have a file with its answers, you can identify it by that
do you need the file?

(it's quite big lol, i had stored a whole thread in it, but i will probably only copy the relevant part of it)

[wintry dawn]

no

im not using direct chat

anyway um

ill run yer prompt

[upbeat mirage]

in LMA?

[wintry dawn]

ye

lol

[upbeat mirage]

without R-Ls?

[wintry dawn]

i gotta delete allat..

[upbeat mirage]

you can sel. it?

[wintry dawn]

idk

[upbeat mirage]

oof

how?

[wintry dawn]

you want to buy it or wha

[upbeat mirage]

just curious

[wintry dawn]

if lmarena has a bug bounty ill report it

cuz i dont want this to go to waste

its useful

[upbeat mirage]

this is quite a fat bug

[wintry dawn]

yeah

i can manipulate results

[upbeat mirage]

i wonder, how many people have used similar "method" at LMA..

||-# but be careful that you dont get banned||

(i would not dare to use such "methods")

[wintry dawn]

anyway ill jus run yer prompt lol

wait so

[upbeat mirage]

ty

[wintry dawn]

is "create a desktop program for the Arimaa boardgame" the prompt u want

[upbeat mirage]

with AI to play against

[wintry dawn]

but in what language n stuff?

can i just make it a single html file

[upbeat mirage]

c++

because, speed

[wintry dawn]

alr

[upbeat mirage]

C++ is fastest for boardgames

it will probably propose using SFML library

(as that is easy to use and performant)

[wet oasis]

..

[upbeat mirage]

not good?

[wet oasis]

Im fine

[upbeat mirage]

ok, another question, which AI do you think is best for vibe-coding in C++?

[wet oasis]

Idk off of the top of my head

[upbeat mirage]

claude-4.5-thinking, gpt5-high, lithiumflow, glm-4.6, qwen3 max, deepseek-3.2, grok4, grok4fast-reasoning, kimi k2, MAI, Serenity, Raptor, Phantom, Sierra, Aspen, Acadia, Skytrail, Monster?

[wintry dawn]

@upbeat mirage sry cf is blocking me :p

[upbeat mirage]

battlemode instead?

[wintry dawn]

cloudflare

💀

[upbeat mirage]

ah

so you think lithiumflow is (currently) the SOTA model in coding?

better than gpt5-high, claude-4.5, claude-4.1-opus et.al. ?

[wintry dawn]

@upbeat mirage sorry it took so long

orionmist btw

(its lithium but with web search)

[wet oasis]

Not sure if thats confirmed

[wintry dawn]

I think its true

@upbeat mirage

lithium

(it decided to make it cli based)

4.5 thinking 32k:

[upbeat mirage]

i thought gemini 3 would generate me a complete GUI app for desktop, like we are used to see from it

with mouse-control and nice piece graphics and board background

apparently we have to wait until next year for that?

but, thanks for your efforts

unfortunately, claude's app has bugs

have you even tested the programs?

-# (mo offense intended)

[sudden idol]

Hope so actually, because Orionmist is messing up Markdown code closing tags, which seems weird if it was Flash 3.0

[wet oasis]

does flash 2.5 mess those up? idk I've never made it write code

[sudden idol]

Don't think so

Maybe pre-preview instability? But it's supposed to be better than 2.5 Pro.

If both are not really outstanding at Tkinter (literally the most basic UI library) but are one-shotting Minecraft, that's a bit suspicious

[wet oasis]

yah

maybe google's gonna release a programming first model instead of gemini 3 rn?

[sudden idol]

Maybe I'll try testing it on a question that only GPT-5 High gets right

[sudden idol]

I think the only way to tell is to test it on an agentic coding environment. E.g., can it resolve compile errors, missing dependencies, WSL compatibility issues, React/C race conditions, etc.

[cursive jacinth]

seems like a weirdly spiky model, I wonder if they figured out some better RL environments for software

[upbeat mirage]

spiky, as in jagged performance over time?

[upbeat mirage]

yeah, and it also has to test the program it generates and recognize bugs automatically (and fix them)

that almost requires AGI, does it?

[wet oasis]

thats a very different definition of agi then the one about being able to do all human tasks in a human body

[cursive jacinth]

spiky as in good at certain things and surprisingly bad at others

it was giving me pretty weak creative writing and writing analysis

[wet oasis]

yeah it seems this is more for code purposes

[upbeat mirage]

apropos, which AI is best for immersive roleplaying?

i mean for super-long games

it would need to have at least 1M context (better 2M)

[cursive jacinth]

gem 2.5 pro, and all the claude's are good

yeah so for that context length gem 2.5 would be it

[upbeat mirage]

is there a special prompting-technique to maximize its gamemaster-intelligence for realistic virtual universes?

[cursive jacinth]

not sure honestly!

mostly you want it to roleplay effectively by giving it info about the character its simulating

[upbeat mirage]

and if it has reached its context limit?

let it summarize everything before that happens?

as a "save-state"?

[wet oasis]

yeah you have to do that

[upbeat mirage]

will gemini 3 pro have 2M? (context window)

[wet oasis]

I hope it has more than that, but idk

[upbeat mirage]

do you think, they publish/release it this month?

[cursive jacinth]

from what I understand the labs have been making pretty fast progress on context length and google generally leads

[wet oasis]

if these guys truely are gemini 3 models, yeah

[cursive jacinth]

so I'd guess yes

[wet oasis]

idk about that, since I haven't seen a model that has beaten the highest context window count in like a year for a good ish model, ignoring llama 4 cause that doesn't count

[cursive jacinth]

like deepseek, openai have been catching up

i assume in the meantime deepmind has made further progress

[upbeat mirage]

is serenity really by xAI?

Grok4.5-preview?

[wet oasis]

no idea

could check

[upbeat mirage]

if it is, it should have a sizable context-window, right?

[wet oasis]

it doesn't seem like the eqbench tool include xAI models, so take this with a grain of salt

but its closest to claude and deepseek

[sudden idol]

Well it's close to both Claude and OpenAI families, so not conclusive ig

That's kinda what happens when you put Grok 4 in, since no reference instances

[wet oasis]

mm

[lilac holly]

these codenames are just names

[wet oasis]

... true?

[lilac holly]

I was expecting super

[wet oasis]

?

[wet oasis]

slightly better analysis of serenity

showing strong signs of relation between it and claude models, so serenity could be anthropic or baised off of claude

[tiny sierra]

Serenity is way too verbose to be a claude model

Imo

Ive also never had a claude model give me tabels when not asked for

[wet oasis]

mm

ok

[tiny sierra]

Or list the sources at the bottom

[wet oasis]

how does it have sources? this isn't search arena

did it just make that up

[tiny sierra]

LMAO

I DIDNT EVEN NOTICE THAT

Sonnet 4.5 for reference

And opus 4.1

[wet oasis]

why is it so long

[tiny sierra]

Feels the need to make every little thing a new line apparently

[wet oasis]

the table thing makes me feel like its simular to perlexity

[tiny sierra]

This should just be comma separated, lmao

And it is comma separated in the app just not lmarena

(Sonnet)

Prompt was

Hand foot and mouth disease clinical presentation

[wet oasis]

weird

[tiny sierra]

If im fishing for lithiumflow should I make a new chat or spam the same identical prompt in the same one?

[wet oasis]

I don't think it makes a difference much so whichever is faster

tho if you spam the same one then it will use all the previous tokens from the winner

[tiny sierra]

Ah yeah I was wondering how that'd work since it changes the model

[candid surge]

When did marble show up?

[candid surge]

You should always make a new chat since it keeps history

[fringe burrow]

anyone can bro

[fringe burrow]

but orionmist is so much worst

[valid peak]

4.20

[sudden idol]

How good is it at coding though

[wet oasis]

Havent tried

[sudden idol]

Looks a bit inconclusive to me, since K2 is closer to o3 yet the score is very close, which suggests Serenity is more in between model families than part of any specific one

[wet oasis]

Maybe

[tiny sierra]

it was good for medical questions for me

[spiral ingot]

lovable can we use there premium

for free here?

[wet oasis]

?

[upbeat mirage]

Acadia (Ocean AI)
Serenity (Ocean AI)
Sierra (Ocean AI)
Shasta (Ocean AI)
Solitude (Ocean AI)
Is there a 6th one?

(the name could still be a coincidence, as "Ocean" is pretty unspecific)

[upbeat mirage]

my hypothesis is, that a new AI lab used xAI and OpenAI (and maybe also gemini) models to train their models on

[sudden idol]

🤷

I just realized Grok 4 is in the 140K data

[upbeat mirage]

Ocean AI uses some words/phrases from grok:

• witty
• maximally truthful
• etc

[sudden idol]

I guess I could train a model to predict the model based on TF-IDF vectors or embeddings

[upbeat mirage]

that would be awesome

[sudden idol]

Idk why I didn't just do that lmao

I guess I thought it would be easier to use their ngram analysis code.

[sudden idol]

I feel there might be a simpler way

I was testing it but got sidetracked checking out a new framework

[upbeat mirage]

on a scale of 1-10 how hard is that for a newbie?

(if using Claude-4.5 and Lithiumflow and maybe even gpt5-high as coders)

[sudden idol]

Well it's classic label prediction

Tons of tutorials, so definitely approachable.

[upbeat mirage]

i mean, to create a new model for prediction

and what hardware would be necessary?

[sudden idol]

Oh it's not as hard as it sounds

These are statistical models, not deep neural networks

Most of them have functions from libraries like sklearn that you can just call

[upbeat mirage]

oh, i just got another idea (totally different topic, though)
how hard would it be to create a decent model to play a niche boardgame?

[sudden idol]

Depends on the game properties

You'd probably use a search space algorithm from classic AI

[upbeat mirage]

if the model should run on a 6GB gpu from 2019 (Nvidia GTX titan)
and the game would be Dou Shou Qi (jungle)

muZero?

i mean, a model which trained by self-play (like Alphazero)

[sudden idol]

Looks like checkers

[upbeat mirage]

would my poor laptop survive this training run? (it can get quite hot, over 90°C)

[sudden idol]

Oh I knew someone who liked training those types of models

Idk how I got beaten by a program running on 2 MB of RAM

[upbeat mirage]

jungle is unique, because of the intransitivity of the capture rules

(mouse can beat elephant but not reverse, except if trapped)

many strange rules

harder to learn than checkers

but bitboards can be used (7x9 board)

[sudden idol]

AI-bench, that would be cool

Where LLMs have to write AIs that go head to head

[upbeat mirage]

yeah that would be amazing

and entertaining to watch

or cyborg matches

where human+LLM try to outmatch other pairs

in coding of AI

[sudden idol]

i think human would win

[upbeat mirage]

maybe we could use CoreWars & RedCode for that?

[upbeat mirage]

expert coders, yep

but junior coders?

but the competition would be between human+LLM vs human+LLM

only such pairs of AI-coders

ok, a human could opt to not use LLMs at all

[sudden idol]

It's more about AI techniques

And depends on the tasks

An engineer might win if the game space is continuous

Because control theory would come in handy

There's a lot of these problems on codingame.com

Whoops sorry, just realized going off topic for this channel

[wet oasis]

Think so

[sudden idol]

Idk if it's from the same family, but there's also Vail

[wet oasis]

Think so

[sudden idol]

Doesn't show up as often though

[wet oasis]

Paws is actually AGI that was told its entire goal is to see how good people are at guessing how long away we are from agi.

[upbeat mirage]

Is Serenity the best of the pack?

[sudden idol]

All are bad tbh

[upbeat mirage]

in roleplaying, too?

[sudden idol]

Idk, haven't tried lol

[upbeat mirage]

do you know, which has the largest CW?

-# (context window)

[sudden idol]

idk

[vale root]

aa

[cursive jacinth]

how has orionmist and lithiumflow been performing for everyone?

[fringe burrow]

best current llm

my coding bench

i didnt know which model i was using while rating

14 tests

[tiny sierra]

Wish google would just let us chat with it directly already

Its not like everyone doesnt know what it is

[wet oasis]

Heh

[tiny sierra]

?

[wet oasis]

You arent wrong

[vale root]

just let us use lithiumflow directly without waiting for it to appear in battlemode, that way testing it would be better and benefit them the most

[tiny sierra]

Would reduce people spamming by 99% rn for the purpose of just trying to get it as well

Does anyone know if this decision is up to lmarena or whoever owns the codenamed model?

[vale root]

idk pineapple said they cannot control the model's availability, ig only Google decides if it goes onto chat mode or not

[wet oasis]

would be weird if they were ok with putting it on chat mode

maybe they'd get away with side-by-side?

[wintry dawn]

new model on lmarena

"marble"

[cursive jacinth]

sounds grok-y

[wintry dawn]

ya

i asked it for a userscript

it gave me a mid html file

[cunning notch]

funny btw

[sinful spoke]

tried lithiumflow a few times, and I think it will score lower than 2.5pro, at least with style control off
It is generally shorter in answer length than 2.5pro, has no tendency to explain things in depth, and is less fond of using analogies

[tiny sierra]

Lmao

[tiny sierra]

Yeah that's what I meant to say

Side by side, I never used chat mode

[wet oasis]

its better at code tho, soo keep that in mind, I wouldn't be surprised if on text arena it is worse tho

[candid surge]

Someone posted something earlier suggesting lithiumflow is Gemini 3 but not Gemini 3 pro

If only we got X28 or 2HT...

[sudden idol]

Lithiumflow seems bad at actually modifying its code (actually seems like the arena was bugged, because it runs fine when I run the React code manually)

[vale root]

how did u get lithiumflow twice in a row?

[wet oasis]

Bug?

[sudden idol]

It used alert windows that didn't show up on Web Dev Arena, so it looked like the buttons were not working.

It took more turns than expected to get a good result, and it didn't seem to fully implement everything I was asking for. Maybe image input would help. Ig it's not really fair on the LLMs that they can't see the output.

[wet oasis]

mm

[elder yew]

Gemini 3 is simply absolutely superior

[sudden idol]

Depends ig... it seems weaker than Sonnet 4.5 at Fyne (popular Go UI library), which is odd considering that Go is from Google

[tiny sierra]

Gemini is weaker at google search than every other big model in existence which is odd considering that google is from google

😭

[sudden idol]

Because of poor tool use ability probably

It hallucinates so much with tool use

I give it a link and it hallucinates contents despite not being able to access the link

I wonder if Lithiumflow does that 🤔

[sudden idol]

Like, really:

Whatever flying-octopus is, it passed the test 😆:

clover too:

I guess it isn't really a hard task, since even Mistral Small gets it:

Ernie failed the task.
DeepSeek V3.2 Exp fails too (is it because DS is trained on Gemini 2.5 Pro...?).
Grok 4 Fast failed.
grok-4-0709 failed (very weird, claims it is from its last visited to the site, the date is listed as 2023).
ajax passes.

Uh oh... Orionmist failed

Looks like Gemini 3 hasn't solved the hallucination issue (even Mistral 24B managed to not hallucinate!)

Edit: I thought it did because I assumed models don't have access to the internet, but upon closer examination, it must have some sort of cached database, since those are real headlines from 2 days ago!)

[tiny sierra]

LOL.

mine hallucinates even with directly uplaoded text or .docx files

(2.5 pro)

[wet oasis]

Im dissapointed

[sudden idol]

Breckenridge training cutoff seems to be October 2024 (it fails here)

[sudden idol]

Serenity (rumored XAI model) also fails. Interesting since both Grok 4 models failed.

flying-octopus passed quite a lot of times, but this time it did something weird:

[wet oasis]

Mm

It tried to tool call anyway?

Ive noticed newee llms are more and more unstable

[dull garnet]

probably bc they're all vibe coded using claude code... 😆

[wet oasis]

Heh

[sudden idol]

Actually, the article it lists was posted TWO days ago, which is after Orionmist was released...

It has cached internet access?

Full response:

Orionmist has internet access, lol

[sudden idol]

For #4:

Hackernews doesn't show article previews, so Orionmist either has recursive web search (went into the article) or Google Search (saw the summary)

Or maybe a cached database, because the results are not live but from 2 days ago.

[zenith vortex]

No anon Gemini 3 😟🙁

[upbeat mirage]

are breckenridge, flying-octopus, clover and ajax all from the same lab?

not OceanAI, or are they?