#Sonoma Sky Alpha
alpine carbon
Adam Holter's benchmark: both performed significantly worse in his code tasks than Grok 4, both slightly under GPT-5 Nano level
autumn solstice
Checks out in my experience
valid loom
managed to find a way to get the reasoning of the model without getting filtered
sinful marten
I feel like this model is better than v3 0324 in Roocode
v3 0324 doesn't read the files surrounding the file you want it to edit for context unless you explicitly tell it to, but this model does
verbal silo
so i asked sonoma sky to build a fishtank simulation on a website and it made this. For some reason, the fish food is flying upwards and the fish do too. It also lookes like a sunset? weirdest fishtank i ever got
verbal silo
cline is giving up on sky alpha. I wonder what this model is supposed to be good for? its not coding
real lily
cline is giving up on sky alpha. I wonder what this model is supposed to be good...
That message is always there whenever it gives invalid result
kind pike
yeah its settled by this point
sinful marten
IF breaks after 40k tokens
Even regenerations won't work
Source: my testing with Roocode
Nevermind. It works at 51k tokens
I definetly degrades though
pastel jewel
I think its grok.
I managed to get it to say it was Mistral. And others got it to say Claude!
spring pulsar
it's grok
fossil tundra
I would be shocked if it was not grok
clever swan
what model is Sonoma Sky Alpha ?
That's the funny thing about the stealth model, nobody knows for sure. But there are some indications that it is Grok.
viscid tree
what model is Sonoma Sky Alpha ?
almost everyone is pretty sure its grok (myself included)
upbeat veldt
I thought it was Gemini 3
grand gull
That's the funny thing about the stealth model, nobody knows for sure. But there...
Grok hybrid reasoning looks like. Are they trying to implement the steps from gpt5 model card? LOL
It's reasoning but not always. When it does it responds largely like grok4
thinks for ages looking like it's stuck and then gives you the most concise response possible. But if it's easier task it will be very verbose and start responding immediately
north dawn
I managed to get it to say it was Mistral. And others got it to say Claude!
Great? How?
Can you share your prompts?
I'm not saying it is really anything, mind you. But always wonder at people's prompts
delicate agate
I managed to get it to say it was Mistral. And others got it to say Claude!
It's too uncensored for either of them
grand gull
it wasn't fine-tuned on the real identity of itself. Attempts to make it leak that are gonna be kinda useless
dark lotus
maybe another oss model
terse cloak
It's too uncensored for either of them
Facts, I got it to make a camsoda clone
This model can literally make a Phub clone and not give you issues about it being adult content.
dark lotus
would this be the 1st time Grok has done a stealth model? and it's doing it with 2M context and 2 different versions? You think so?
dull wind
Its xai
Its blatantly obvious to everyone that's not coping hard about it not being xai.
The description language = xai
The benchmark performance similar to grok model
Multi different kind of jailbreak show xai
Thinking tags show xai
Model emoji usage and personality is similar to grok.
dark lotus
why not just say Yes this is the first time Grok has done a stealth model, and is doing two different 2M context models.
hushed storm
does anyone know why it gives this error when used in qwen cli?
valid loom
does anyone know why it gives this error when used in qwen cli?
your prompt is being filtered
valid loom
no clue then, but for me when i was using the api to jailbreak the model's thinking i got this "ERROR" error
it might be the system prompt from the cli itself causing it to get this
someone at openrouter would need to answer
hushed storm
like i tried it with claude code router earlier today and it worked fine
the model wasn't good at doing stuff though
calm birch
getting null responses from api / sdk as well
playground is ok, even for stuff that normally would be censored
@dark turtle do we know anything about this?
dark turtle
just looks like intermittent issue
terse cloak
Could def be an OpenAI model that is heavily helped by Oak AI which is an partner with OpenAI.
calm birch
API still returns blank for me
anyone with similar issue?
even when not 429'ing it's blank
quaint mortar
just looks like intermittent issue
Cloudflare?
static cobalt
Have they been updated since the 5th? Current opinion: absolute garbage
dark lotus
not worth it
dark lotus
west flower
Its blatantly obvious to everyone that's not coping hard about it not being xai....
buddy hates elon so much that he doesn’t care that the knowledge cutoff is grok 3 levels
😹
dull wind
buddy hates elon so much that he doesn’t care that the knowledge cutoff is grok ...
I hate xai but I hate bad information more.
dark lotus
Okay sky is REALLY good at long context
dark lotus
because it has 2 million
context length doesnt matter if its not good at utiliszing it
you are new to this it seems
west flower
im trolling
autumn solstice
so then why do you spread it
It is likely xAI
dark lotus
It is likely xAI
#attachments message
most importantly this proves that geminis hold of long context is GONE , first OAI and now XAI
and claude is unbeatable for long context coding, and its not even close.
clever swan
Only temporarily. Gemini will probably be a big improvement with 3.0.
dark lotus
Only temporarily. Gemini will probably be a big improvement with 3.0.
cat and mouse chase will continue , point remains there is no moat and this is a race to the bottom.
dark lotus
@dark turtle Could you give a day/timeline/probability when this model will be taken down?
echo juniper
whole merlin
This fails terribly at my long-context japanese to english translation task, outputs very unnatural and broken English.
long-context in this case being 100K tokens because I cba to wait for the massive latency at higher token counts.
dark lotus
This fails terribly at my long-context japanese to english translation task, out...
similar results testing, this 2M model is terrible with context even up to 100k. #1413616210314133594 message
maybe there are use-cases where 2M works for something else? but then, is it just charging you for more tokens that it won't use anyway? /conspiracy
dark lotus
Btw this model doesn't reason , so account for that
terse cloak
This could be Gemini 3.0
valid loom
if it is then thats a downgrade from 2.5 pro
whole merlin
Btw this model doesn't reason , so account for that
it does, it's just not exposed.
dark lotus
it does, it's just not exposed.
ahh quite fast if thats the case
slow nova
im very certain its grok
my ST preset behaves the exact
same way with grok 4 and sonoma
hybrid monolith
One message removed from a suspended account.
alpine carbon
It's confirmed this is Grok
dark lotus
It's confirmed this is Grok
post source
alpine carbon
This message has the details
dark lotus
i'm unconvinced. to me, confirmation means an official source or overwhelming consensus.
alpine carbon
The level of evidence we have is beyond reasonable doubt, but ok
kind pike
i'm unconvinced. to me, confirmation means an official source or overwhelming co...
i'm almost sure it's Amazon again
kind pike
there are two
alpine carbon
Pretty likely it'll be Dusk
clever swan
Sky is the thinking model
limber yarrow
most importantly this proves that geminis hold of long context is GONE , first O...
mej
meh
1m and 2m isnt too much of a difference
dark lotus
Pretty likely it'll be Dusk
yup , dusk is the dumber one
valid loom
exists on the grok website in the chunks
theres also a non thinking & concise non thinking version
so that might be dusk
leaden shell
Would make more sense given the mid performance of the model that both Sky and Dusk are Mini, just thinking/non-thinking.
Would explain the fast inference as well.
valid loom
and grok 4 mini thinking tahoe?
dark lotus
okay they are taking this fucker down
valid loom
sudden uptime dip
dark lotus
what kind of context size do those grok models have?
the sky alpha model did seem to be more like a gemini model in my testing with context #1413616210314133594 message
valid loom
what kind of context size do those grok models have?
doesnt seem to be listed in the grok website's code, its probably only handled / stored server-side
dark lotus
could be Grok has improved its context reasoning with these new models
valid loom
theres also a non thinking & concise non thinking version
the concise version is no longer existant in the code
and the original grok 4 mini thinking, now its only the tahoe version
sinful marten
Instruction following in RooCode is hit or miss
It always starts to falter around 50k tokens though
zenith dust
what kind of context size do those grok models have?
the apparently tried up to 1m context on the grok 4 release, but only delivered 250k
anyways it is basically confirmed that this is a grok model
wise bear
Do you guys also suffer from internal tokens leaks on tool calls? I see some weird things every few completions like xai:function_call
dark lotus
crossed 1Billie on one account
3 days btw
sky-alpha has been really great for summarizing. i'm a fan. i hope it's cheap.
true
sinful marten
crossed 1Billie on one account
Gawd damn
viscid tree
crossed 1Billie on one account
i think that cost them between $50 and $5000 of electricity. you're welcome for the very precise and useful information.
viscid tree
ok maybe $10-$1000
dark lotus
i think that cost them between $50 and $5000 of electricity. you're welcome for ...
I am at almost 2.5 billion
real lily
I am at almost 2.5 billion
what needs that many tokens?
dark lotus
what needs that many tokens?
Data
feral jackal
maiden thicket
i can talk about my (REDACTED) with this model, which I know I cannot do with ChatGPT. I can even get it to write (REDACTED) that ChatGPT never could.
You could fit a series of a little over ten novels in that context window.
zealous tusk
I only get empty responses from this model.
Same code I use to query every other model
Anyone else run into the same issue?
zealous tusk
Yeah, if I literally just replace the model slug with "openai/gpt-5-mini" the code runs fine
dark lotus
sorry about that
zealous tusk
It's okay, I don't blame you
languid moth
It always starts to falter around 50k tokens though
using MCP seems to alleviate this tendency
blissful crypt
so literally what is the model? grok or gemini?>
cerulean elk
don't know why people would think gemini. I'd be surprised if it wasn't Grok tbh
dark lotus
don't know why people would think gemini. I'd be surprised if it wasn't Grok tbh
the 2M context might be a reason
cerulean elk
the quality of responses doesn't scream Gemini (neither does the general style of responses). Just my two cents.
And xAI probably has the money to ship 2M context
But who knows
dark lotus
https://developers.googleblog.com/en/new-features-for-the-gemini-api-and-google-ai-studio/ looks like by June 2024 gemini 1.5 pro had 2M context. Has xAI delivered 1M anywhere yet?
i'm genuinely curious who can put out multiple 2M context models, and I have no preference who it is. I think way too many people in here are too confident it is xAI. I don't see the smoking gun myself.
cerulean elk
Fair enough. I just don't think Google would ship something that feels worse (and messes up world knowledge that 2.5 got right) than the previous iteration of the Gemini series.
dark lotus
Idk how it isn't clear to y'all this is a xai model
zealous tusk
Can tell it's Elon just by the description
No other human would say "Maximally intelligent".
alpine carbon
There's overwhelming amount of evidence, not really room for doubt anymore
A few of which are proof by themselves (like the downtime graphs lining up perfectly)
sinful marten
A few of which are proof by themselves (like the downtime graphs lining up perfe...
That's by far the strongest piece of evidence
If you presented that graph to me, I'd be convinced instantly even if there wasn't any other evidence
viscid tree
That's by far the strongest piece of evidence
If you presented that graph to me...
but in order to be really convincing, would probably need to be compared with other providers graphs at the same time, to show the correlation was only with x.ai models
sinful marten
but in order to be really convincing, would probably need to be compared with ot...
Ah, yeah. Yes, that'd make it stronger
zealous tusk
Still can't get it to work =(
dark lotus
Still can't get it to work =(
spent 2.5 billion tokens
might hit 2.8-3 and call it quits , lets see
zealous tusk
Congrats
zealous tusk
Oh, interesting, it's just the only model so far that requires a user message instead of a system message for the first turn
Or, to phrase it better, it will only respond if the last message was from the user role
sinful marten
Bad thing I noticed with Sonoma Sky
It will try to add things that aren't directly needed when asked to make a change (in my case, it tried to add a caption key with a type of string to the TypeScript type when all I wanted it to do was remove the id key)
maiden thicket
I wonder how much this model will cost when it’s out of testing
dark turtle
dark lotus
maiden thicket
dark turtle
dark turtle
brave estuary
so alpha was a model with reasoning and dusk was without?
lyric gazelle
so alpha was a model with reasoning and dusk was without?
@dark turtle
Does dusk and sky are the same model
full geyser
Where is this model gone? anyone know where we need to go to use this model again
dapper leaf
I guess these are offline now?
real lily
I guess these are offline now?
correct
real lily
I guess these are offline now?
dapper leaf
got it, thanks!
real lily
but remember since its a free model the free rate limit now applies
dapper leaf
but remember since its a free model the free rate limit now applies
Oh cheers, is this accurate? got it from the docs chat
General limit: Up to 3 requests per minute
Daily limits depend on your credit purchases:
If you've purchased less than 5 credits: 50 free model requests per day
If you've purchased 5 or more credits: 500 free model requests per day
Additionally, note that:
Creating multiple accounts or API keys won't bypass these limits as they're governed globally
If your account has a negative credit balance, you'll get 402 Payment Required errors even for free models
Cloudflare DDoS protection will block requests that significantly exceed reasonable usage
1
real lily
Oh cheers, is this accurate? got it from the docs chat
General limit: Up to 3 ...
its now 10 credits bumps to 1000 requests
dapper leaf
its now 10 credits bumps to 1000 requests
sweet
dapper leaf
hmmmmm.. i think there is a hidden rate limit. i can barely get one per minute done
admittedly my prompts are 1.3M+ tokens
real lily
Could just be the model takes that long to process all those tokens lol
“Fast” only goes so far lol
dapper leaf
No it returns with just nothing and I have have to submit again
dark turtle
but remember since its a free model the free rate limit now applies
this one doesn't have limits