questions to ask
#general
whole wagon
It sometimes randomly has nazi outbursts it's not just political questions
wind moth
a livestreme
hollow ocean
guys did it start yet
wind moth
like what many companies do
empty stump
guys did it start yet
No
whole wagon
Like the other day it was flaming literally random Jewish ppl on X. Since they had Jewish surnames
zenith saffron
yea agree they should just go full hackathon mode and just replay things
not actually use it
whole wagon
Without any political prompt
whole sundial
supergrok heavy is not sold out anymore
(don't worry, i don't have $3,000 to spend. also, "SuperGrokPro"? I thought this was SuperGrok Heavy!)
they must have changed the name in the last second
whole wagon
Bets on it starting on the hour?
north vale
which hour of the night can it start at to minimize viewership
that's where my money is
whole wagon
You can just downgrade it and it'll refund the diff scaled by days
whole sundial
wind moth
X
north vale
ggs
whole wagon
Maybe they really did want it to start at nein
jade egret
FINALLY
hallow pelican
it started eventually
empty stump
We were all wrong
hallow pelican
🥳
whole sundial
whole wagon
That's a lot of viewers damn
zenith saffron
as predicted, one hour and one minute late
whole wagon
It's climbing rapidly
jade egret
or is that vies
whole wagon
At 60k rn
jade egret
It's climbing rapidly
how do you see how much people are watching
zenith saffron
now we're being baited
keen beacon
Loading screen for another hour
jade egret
oh this?
whole wagon
80k
echo aurora
we have cool music now tho
jade egret
we have cool music now tho
: )
leaden meteor
"worlds most powerful ai model" hmm....
jade egret
What the hell
LOL
small haven
over/under elon musk happy or sad on stream
echo aurora
This is going to be the most watched livestream on X by far
do you know what the current record is?
elder rapids
This is going to be the most watched livestream on X by far
wym? the most watched is well into the millions
whole wagon
I mean watched live ofc
elder rapids
yeah, live
jade egret
146k for me rn
jade egret
152 now
zinc ore
140k live will end up being millions of viewers
elder rapids
do you mean the most watched AI related thing
jade egret
yooo
zenith saffron
i wonder who's the voice
elder rapids
it's AI generated
torn mantle
Who's talking
elder rapids
it's deadass AI
empty stump
When is it gonna be on lmarena leaderboard
zenith saffron
lol "AI is advancing faster than any human"
small haven
so this is going to better than kingfall?
jade egret
it's deadass AI
fr?
zenith saffron
oh wait what
elder rapids
lol "AI is advancing faster than any human"
progression wise yeah ofc
jade egret
238k
whole wagon
college exams were solved long ago man we are already on phd questions
get to the good stuff
zinc ore
Elon is so low energy rn lol
whole sundial
"Grok 4 is smarter than almost all graduate students in all disciplines simultaneously."
hardy pecan
blud needs to sleep
whole wagon
blud needs to sleep
bullish
hardy pecan
order of magnitude is big
zenith saffron
is elon trying to emulate jensen
whole wagon
is that good
yes
zenith saffron
compute used for RL step
hardy pecan
gwak 4, based
zenith saffron
they didn't confirm that explicitly
hardy pecan
oh nvm there is sneaky orange colours to cause confusion
elder rapids
oh nvm there is sneaky orange colours to cause confusion
how is that confusing
?
they're highlighting the RL difference
whole wagon
400k viewers sheesh
jade egret
zenith saffron
yeah it's not explicit but yeah i see what you mean
it's likely it's basically the same base
zenith saffron
i wonder how it does in math research
rly want to see my math phd friend try this
zinc ore
Elon doesn't even believe himself
whole sundial
26.9%
whole sundial
no tool
whole sundial
41% with tool
whole sundial
"we put the tools in training"
elder rapids
lmao
jade egret
41% with tool
is that good?
whole sundial
i think so?
jade egret
ooo
jade egret
holy it double
zinc ore
Elon is so out of it lol
whole wagon
he seems depressed
jade egret
can this dude talk faster
fr..
jade egret
he seems depressed
why
elder rapids
did he just say "idk"
jade egret
705k is crazy.......
wind moth
he needs to stop yapping
echo aurora
I'm sure there will be
jade egret
is it gonna be on arena?
elder rapids
ye
jade egret
:0
echo aurora
🍊
whole wagon
they trained it on puzzle type things i bet
zinc ore
Ik openAI and Google feeling pretty good rn
zenith saffron
"closing the loop on reality" makes me worried about paperclips
zenith saffron
wait, text-only subset?
whole wagon
wait, text-only subset?
the image stuff isnt ready yet
zenith saffron
wdym by your question?
whole sundial
maybe they didn't finish the multimodal part yet?
zenith saffron
the image stuff isnt ready yet
o i see.
how do other models do on text-only subset?
whole sundial
oh, test-time compute is Grok 4 Heavy
zinc ore
Consc@1024
hardy pecan
lol]
proper prawn
They seem to be hiding reasoning
somber hatch
test time compute is just using more compute when answering a question. Reasoning tokens are an example of TTC, also running multiple in parallel and voting or colaborating is TTC. It's just use GPUs a lot while answering the question, not training or updating the model at all
elder rapids
they tryna hypnotize us lmao
zinc ore
the odds plunged back down
Like I said, Google and openAI feeling pretty good rn
somber hatch
their graph was shoing how adding more TTC increases the performance. OpenAI has shown similar charts when they released O1 and O3
elder rapids
their graph was shoing how adding more TTC increases the performance. OpenAI has...
ye
storm needle
what are the chances of everyone there getting fired
elder rapids
they're not showing anything impressive
what's going on lol
they're showing what makes o3 cool
zenith saffron
"wen grok 5"
proper prawn
they're not showing anything impressive
The HLE numbers are impressive. Just the presentation and likely drugged Elon couldn't show it
elder rapids
they're saying grok 4 is doing all this research to get the simulation correct
whole wagon
latex was wrong lel
elder rapids
The HLE numbers are impressive. Just the presentation and likely drugged Elon co...
we haven't seen other labs do what they've done with HLE tho, Google could have even higher numbers and we'd never know
keen beacon
I wonder if they switched off the qwq cold start lmfao
whole sundial
so grok 4 is currrently able to look at images
proper prawn
Training on tool use could be big
whole wagon
its not that wild man
keen beacon
They have so many resources. They should make their own XD and actually innovate
elder rapids
no
zenith saffron
imo there's a lot they're not revealing
whole sundial
38.6% with tools, 44.4% heavy
empty stump
Is the gemini 2.5 hle score w/o tools
whole sundial
25.4% without tools
zenith saffron
imo there's a lot they're not revealing
all they say is "RL", the devil is in the details
elder rapids
38.6% with tools, 44.4% heavy
bs why is the discrepancy so low
empty stump
Wonder grok 4 superheavy vs o3 pro vs gemini 2.5 pro deepthink
zenith saffron
just because they're still using RL doesn't mean there isn't any innovation lol
clever estuary
reich4 looks itneresting
whole wagon
bro what in a few weeks
zenith saffron
that's like saying GPT-3 was not innovative because it was just scaling the same method up
zenith saffron
(whereas in this case we don't even know what methods they used)
zinc ore
Gemini no tools does better than grok 4 no tools on HLE
jade egret
back
whole sundial
proper prawn
61.9% in USAMO
jade egret
what did i miss
echo aurora
what did i miss
some demos
jade egret
some demos
is grok 4 very very good?
zinc ore
Okay finally more benchmarks
keen beacon
For 10 minutes 😂
keen beacon
Maybe not this month
zinc ore
I completely didn't retain how the reasoning works with grok heavy
jade egret
Maybe not this month
isnt gpt 5 this month or nah
elder rapids
I completely didn't retain how the reasoning works with grok heavy
just parallel
whole wagon
I completely didn't retain how the reasoning works with grok heavy
its simple man
they just do the parallel thing
same as o3 pro
holy cringe
proper prawn
lol they know their audience
whole wagon
bruh
jade egret
diet coke
hardy pecan
my stomach hurts
whole wagon
how are they not cringing
zinc ore
😐
zenith saffron
this is so ex machina coded
torn mantle
Oh it failed
hardy pecan
that was weary weary cringe
torn mantle
Ye
elder rapids
nice voice tho
echo aurora
nice voice tho
agreed
whole wagon
it didnt fail he didnt realise it retained the context
empty stump
now advertising openai
whole wagon
lol
torn mantle
Openai?
whole wagon
RIP
jade egret
did openai fail
empty stump
no
jade egret
openai die : (
hardy pecan
do they have a hitler voice, based
empty stump
oh thats what they were showing the speed
elder rapids
oh nice api
whole wagon
👀
zinc ore
Yeah previous sota was 8%
jade egret
holy 2x
empty stump
I wonder about SWE bench score
torn mantle
Now we are getting to the real stuff
elder rapids
it's arc agi 2 score isn't proportional
torn mantle
That's benchmarks looks crazy ngl
torn mantle
i thought it cant see images
Just text
torn mantle
how did it do do arc
Text
torn mantle
translated to text?
Yea
whole wagon
1.5M viewers damn
whole sundial
grok 4 0709 is the current version
whole wagon
what the hell is vending bench
jade egret
what the hell is vending bench
money : )
elder rapids
nobody knows
empty stump
Running a small shop or something
zinc ore
Grok downloads about to skyrocket
jade egret
when do yall think google and openai will take to catch up
whole wagon
how is bro gonna add it
jade egret
cuz gemin 3 and gpt 5 is comming
torn mantle
Arent you one of lmarena staff team?
elder rapids
when do yall think google and openai will take to catch up
probably by their next releases
hollow ocean
@deep adder deepthink already dead
jade egret
probably by their next releases
oo
torn mantle
Okay~
empty stump
when do yall think google and openai will take to catch up
few weeks
small haven
1.5M viewers damn
views not live viewers
jade egret
rlly fast ig
empty stump
how does it do in IMO
empty stump
so dumb they dont show live count
whole sundial
whole wagon
not much coming tbh
frigid phoenix
Is grok that good? lol
Didnt watch the stream
zinc ore
Last 20 mins was way better than the beginning
empty stump
not trying it out until leaderboard results come out
proper prawn
Is grok 4 in arena?
empty stump
no
tidal schooner
thats usamo
ah damn
empty stump
hmm no o3 pro
tidal schooner
don’t think they tested it for imo then
still extremely impressive
better than i could ever do
zenith saffron
don’t think they tested it for imo then
doesn't imo literally start today? lol
zenith saffron
i am lowkey a little worried
empty stump
july 10 yes
wow that is crazy
zenith saffron
i am lowkey a little worried
i hope we don't get paperclipped
within the next 5 years
whole sundial
grok 4 rate limit: same as grok 3 free, 20 per 2 hours for supergrok subscribers (not available for free rn)
empty stump
someone try out grok 4 and tell me how good it really is
zenith saffron
my p(doom) has been climbing
small haven
Deepthink dead before even releasing
which deepthink?
dawn wharf
which deepthink?
Gemini
dawn wharf
@small haven middle column
zinc ore
tidal schooner
grok 4 rate limit: same as grok 3 free, 20 per 2 hours for supergrok subscribers...
grok 4 is free?
elder rapids
it's like 3$ input 15$ output
tidal schooner
huh
whole sundial
no
whole sundial
elder rapids
wait where are we getting this from, I'm only seeing 3 in 15 out
if it's actually that expensive then all the hype for me is squashed
keen beacon
It's a typo
elder rapids
id thought so
keen beacon
The pricing below that is correct
zinc ore
elder rapids
ye see
keen beacon
Do they still give you reasoning 'summaries'?
Can u show if you don't mind I can't check rn
Oh I mean regular grok 4
Oh lol they actually summarize now
Thanks
keen fulcrum
Grok 4 achieves SOTA
keen beacon
Did they update the cut off or is it literally just rl on grok 3 lol
Yeah I didn't really watch it lmao
keen fulcrum
@echo aurora when can we expect grok 4 to be added to lmarena? ArtificialAnalysis received early access
keen beacon
Ultra should beat it though
keen fulcrum
It would be great if grok 4 heavy can be tested by us as well
hollow ocean
grok 4 heavy is the truth
fleet pine
grok 4 heavy is the truth
so when grok 4 is coming in arena?
echo aurora
<@283397944160550928> when can we expect grok 4 to be added to lmarena? Artifici...
sorry to say I don't have any details to share atm 😦
hollow ocean
so when grok 4 is coming in arena?
keen fulcrum
looks like its available
elder rapids
yeah
@leaden palm gave me a story it wrote
and I don't need any other examples
ts BUNS at writing
💔😢
empty stump
what llm is best at writing so far
empty stump
So gpt 4.5 is not best at anything
elder rapids
So gpt 4.5 is not best at anything
if you give it an environment that communicates with the other models it'll do really well
but it's not that good by itself
dapper storm
Why do you think that?
tidal schooner
looks like its available
@echo aurora yay
tidal schooner
and it’s gone again?
elder burrow
i only trust scicode for coding
look at 4 opus on livecodebench..
its behind nemotron 💀
this is accurate
empty stump
grok 4 gone from arena bruh
elder burrow
grok 4 gone from arena bruh
IT WAS THERE?
empty stump
for a few mins
elder burrow
bruh
indigo hazel
for a few mins
lmao
whole wagon
keen fulcrum
grok 4 gone from arena bruh
why lol
empty stump
no idea
primal orbit
I managed to get 1 prompt to grok 4 in lmarena then it dissapeared and chat switched to chatgpt
zinc ore
Yep
fleet lintel
I tried scrolling 100 comments to understand grok4 is SOTA or not... I am still not sure.
Is it SOTA or not?
elder burrow
💀
zinc ore
empty stump
i can do better on ms paint
elder burrow
@old garden
elder burrow
2 Treys
old garden
lol
civic flame
i've noticed currently frontend dev seems really poor. i am giving it the benefit of the doubt because i am only able to use it through X where it has a system prompt and all of that applied, but still..
zinc ore
Why u tag me twice
civic flame
zenith saffron
you think it's contaminated for HLE, USAMO, etc.?
fleet lintel
SuperGrok Heavy: $300/month (Multi Agent Version)
Dayum! Is it really that good?
elder burrow
wtf is stonebloom
fleet lintel
@deep adder time for you to buy $300/month version and tell us the truth
whole wagon
Just tell it not to search
whole wagon
Tell it in the prompt
whole wagon
The API will not randomly call tools
Without it being enabled
There are benchmark numbers without tools
stuck orchid
Waiting for Grok 4 on LMArena to vote 👍
hardy pecan
seems to be bad at coding, cant edit my code without bugging
small haven
it thinks a lot
dapper storm
No I hope
stuck orchid
I think Grok 4 should be something like o3 in terms of reasoning and cognition.
o3 is lazy and also worse at coding than Gemini, especially in web coding, but according to my tests, o3 is the most powerful model for researchers.
Grok 4 should reason "from first principles."
abstract leaf
I think Grok 4 should be something like o3 in terms of reasoning and cognition.
...
Grok 4 should reason "from first principles."
they hide reasoning
winged wing
same base model as grok 3?
abstract leaf
whn will you guys update the leaderboard ?
winged wing
really i totally missed that
abstract leaf
grok-4 should be around 1500.
-# my assumption.
winged wing
gd damn good thing i got out break even
stuck orchid
whn will you guys update the leaderboard ?
When Grok 4 appears in the arena
abstract leaf
yes
winged wing
u manage to catch any of the swings on HLE?
indigo hazel
basically like a student who cheats during the test at school
civic flame
<@1338136168344064040>
yes i've gathered
but right now i've only tried via Grok on X
elder rapids
grok on X is different
elder rapids
alr
civic flame
so i'm waiting for lmarena to add grok 4
winged wing
It might have some sauce with the scaffolding. Idk I wouldnt be so quick to judge. Its def gonna get blown out of the water by the end of the month tho
civic flame
lol we're still waiting on R2
elder rapids
yep, but nice benchmarks
civic flame
seems to also suffer from being too succinct when writing code
whole sundial
trained off of same base model as grok 3, they are just now training the new base model
civic flame
no wonder it's mid
blazing bison
Bro stop yapping, for god sake
zinc ore
They should have kept the 3.5 naming scheme
rare python
no wonder it's mid
wolfstride vs stonebloom, which one is better at web design?
whole sundial
so i guess there will be a Grok 4.5 with image gen? Gemini 3/GPT-5 image gen will be better
blazing bison
Because you're poor, don't even have access to the model and is yapping around
Yes you're
civic flame
wolfstride vs stonebloom, which one is better at web design?
can barely tell the difference tbh
blazing bison
So stop yapping
hardy pecan
children, stop arguing
blazing bison
This is the mass user model, the true model is api only
small haven
are all your tools set to OFF
search was off
hardy pecan
im thinking 0.00025km long is very small
blazing bison
If you use o3, sonnet, everything on their frontend is dumber than api
hardy pecan
the bridge is 25cm
small haven
check prompt
civic flame
civic flame
check prompt
okay maybe it isn't too stupid
hardy pecan
https://grok.com/share/c2hhcmQtMg%3D%3D_14297bc2-398f-427e-9195-687e89e05e81
is the original question, having the bridge 25cm long?
elder rapids
okay maybe it isn't too stupid
nah it's not stupid but it's not smart like 2.5 pro, or even o3
blazing bison
It's not even secret, just search on X and you gonna see openai team talking about it. You're a yapping machine bro
elder rapids
doesn't catch anything im saying
lmao
it just got crushed in a debate with 2.5 pro too
thorny bane
what makes you think it's very contaminated?
small haven
oh yea terminator svg benchmark, i forgot
whole sundial
maybe grok would have even higher bench scores if it didn't use brave search, which has their own index so it kinda sucks. a sample search/deep research across gemini 2.5pro, grok deepsearch, duckduckgo, and perplexity shows that grok is the clear loser. brave's index is so bad that when i asked the question a few days ago, it finally found a relevant source while all previous attempts were just... bad. idk why they can't just license from duckduckgo lol
whole sundial
august
small haven
when are they dropping the code model 💀
august
civic flame
yeah this is disappointing
whole sundial
september for "multimodal agent" a.k.a. new image gen
indigo hazel
so i was waiting for the best model ever, but it's worse than o3 and 2.5 pro?
whole sundial
to be fair grok was the first major ai that i know of that had native image gen. all the others did it this spring. they had it last winter.
zenith saffron
it's just jimmy ba and tony wu the co-founders lol
indigo hazel
so what
zenith saffron
just researchers
whole sundial
i think grok 4 was rushed, it's all hype
elder rapids
yo
undone hull
Can't wait to see MechaHitler 4 on the charts
winged wing
ok lets see it
wintry tinsel
i think grok 4 was rushed, it's all hype
It was definitely rushed but is it better than 2.5 pro is the question
whole sundial
wanted to try my "basic" knowledge question that only OpenAI models (o1, o3, GPT-4.5, maybe 4o/4.1?) get right. Claude Opus fails it. Gemini 2.5 Pro, stonebloom (Ultra) gets it wrong. Grok 3 got kinda close, but I don't think it will get it right if it uses same base model. Might get it right with reasoning.
civic flame
what's the question?
whole sundial
I'll wait for it to be on arena, thank you though
thorny bane
grok 5 will invent new physics
civic flame
I'll wait for it to be on arena, thank you though
alright
have you tried it with wolfstride?
indigo hazel
I'll wait for it to be on arena, thank you though
then let you know if it answers correctly or not pls
elder rapids
It was definitely rushed but is it better than 2.5 pro is the question
nah in my testing
if you find different results lmk
whole sundial
i will
elder rapids
this model has the same issue as grok 3 with Improving through a context
dawn wharf
direct chat
civic flame
lol ok
indigo hazel
so it got that high result in arc agi 2 by using tools?
small haven
stuck orchid
Is Grok 4 similar in communication style to o3?
civic flame
not in direct chat and i haven't seen it in battle in the 10 rounds i've gone thru
whole sundial
grok 4 from what i heard was available on the arena for a few minutes earlier
civic flame
grok 4 from what i heard was available on the arena for a few minutes earlier
yeah but it seems to have disappeared
cc @echo aurora
echo aurora
yeah but it seems to have disappeared
it's in Battle Mode
civic flame
i've gone through about 15 battle rounds now and not seen it 😔
either i'm unlucky or something's up
echo aurora
there are issues with putting it in Direct & side-by-side, but we're working on it
stuck orchid
it's in Battle Mode
Is he still in Battle mode?
whole sundial
it's really just grok 3 but the very smoky Colossus is pumping more power into it to make it reason better
echo aurora
Is he still in Battle mode?
yeah should be
small haven
grok 4 sucks
echo aurora
I haven't heard otherwise but will flag if more folks are saying they're not getting it
civic flame
on what platform
whole sundial
i'm an idiot for doubting Dork 4
this was the question lol. Grok 4's response is fully correct. First non-OpenAI model to get it correct!
civic flame
memphis in general is being screwed over by a bunch of ai-related developments that aren't adequately planned (imo)
stuck orchid
Why then did Elon say that Grok 4 could rewrite the entire current dataset for AI training?
I think Grok 4 has hidden potential. If it's not very good in other areas, it must be good for some specific type of task. We need to try to find that area.
whole sundial
grok 3
stuck orchid
Esl
whole sundial
well i guess all of that pumped up energy did something to Dork 4. Not paying $300 a month to see what "Heavy" is like, though.
dawn wharf
oh i have a more niche one of these knowledge Qs
like?
eager mica
Why then did Elon say that Grok 4 could rewrite the entire current dataset for A...
Current pretraining datasets are crappola. Other AI companies are already thinking of this, but I don't think it's being done at scale yet.
fleet lintel
this was the question lol. Grok 4's response is fully correct. First non-OpenAI ...
AI overview handles it better.. AI overview probably uses like gemini super small model
whole sundial
this benchmark is for offline models only, no cheating here!
eager mica
Current pretraining datasets are crappola. Other AI companies are already thinki...
https://arxiv.org/abs/2506.04689
From Meta.
elder rapids
after even more testing I can say that it's not smart but it's knowledgeable ASF and can pinpoint things through a context well + good tool usage + brute forces puzzles really well
2.5 pro and o3 >>>
fleet lintel
after even more testing I can say that it's not smart but it's knowledgeable ASF...
how are benchmark number so high then? Are they pulling some shady stuff like Meta/Llama?
elder rapids
how are benchmark number so high then? Are they pulling some shady stuff like M...
grok 3 mini benchmarks
should tell you everything
lmao
whole sundial
more fails from Claude 4 Opus Thinking and Gemini 2.5 Pro (the Ultra anon models fail here)
fleet lintel
somehow these CEOs characters are reflected in these models... shady CEOs == shade models
whole sundial
the tools they use bump up the numbers, but grok 4 offline is better than grok 3 offline
elder rapids
yep
blazing bison
the tools they use bump up the numbers, but grok 4 offline is better than grok 3...
this
elder rapids
it ends there tho tbh
it got crushed in a debate with o3 too
that's crazy
one thing Ive noticed tho
blazing bison
that's crazy
it's a math model, they are using a lot of spacex and tesla math stuff
dawn wharf
weren't you glazing elon before the announcement?
elder rapids
is that it's not as dogmatic
as other models
I have to ask it not to hold back and be efficient etc etc
blazing bison
weren't you glazing elon before the announcement?
he is just a professional yapper
keen fulcrum
grok 4 is such a meme and a legend
haughty siren
Is the Grok 4 model in the arena with thinking and is it super heavy
keen fulcrum
Is the Grok 4 model in the arena with thinking and is it super heavy
got shut down after 5min
blazing bison
in my math tests it actually crushed them
keen fulcrum
believe due to rate limit
winged wing
he is just a professional yapper
professional yapper and good at navigating c suite in startups
wind moth
hows grok 4 doing for those who have tested it
elder rapids
it's a math model, they are using a lot of spacex and tesla math stuff
yeah but I don't care about that stuff
haughty siren
got shut down after 5min
Interesting, I just was able to message it a minute ago.
elder rapids
that's redundant overall
indigo hazel
hows grok 4 doing for those who have tested it
it uses a lot web search. it doesnt only use X.
whole sundial
aka the arena!
blazing bison
hows grok 4 doing for those who have tested it
it's good at math and programming, on par with O3 and 2.5. However, O3 might be better when it comes to actual logic usage
indigo hazel
it's a math model, they are using a lot of spacex and tesla math stuff
a paper i saw a few days ago said that llms which are trained and better at maths are also better in the other areas
blazing bison
a paper i saw a few days ago said that llms which are trained and better at math...
I don't see it in Grok 4. The writing is terrible
pure anvil
a paper i saw a few days ago said that llms which are trained and better at math...
It doesn't work in practice because a 32b qwen fine-tune could be better in math than 3.7 sonnet but it will be worlds worse in everything else
blazing bison
it's an example
I know it's a good example because they're bad at math but good at other things
elder burrow
in battle mode?
stuck orchid
Current pretraining datasets are crappola. Other AI companies are already thinki...
Yes, it seems that's the issue. Models nowadays are often fixated on the specific text of the question they are asked. They might provide a good answer to task X, but get confused when the same task X appears as part of task Y.
Some tasks require broader reasoning, for instance, in agent modes, to even understand the environment they are operating in. This is likely due to training on unformatted web data.
However, the o3 model really stands out in this regard; while it can be somewhat 'lazy', it sometimes understands the task context better, though it also hallucinates quite significantly
whole sundial
in battle mode?
i think it's a joke
elder burrow
scicode is the best
blazing bison
I would use Grok 4 for math-related problems, but I wouldn't switch from ChatGPT to Grok. Sorry, Musk, but you need to do more
Maybe heavy grok is good, but $300?
indigo hazel
basically it's a cycle. like a student who uses ai to cheat, or like he uses calculator in maths tests, grok uses tools to respond correctly
keen fulcrum
I would use Grok 4 for math-related problems, but I wouldn't switch from ChatGPT...
Get outta here
dawn wharf
basically it's a cycle. like a student who uses ai to cheat, or like he uses cal...
not a good comparison
if it's truly smart, it shouldn't need tools
keen fulcrum
Maybe heavy grok is good, but $300?
$300 for code, multimodal and video model is very cheap
- full context 256k + grok 4 heavy
blazing bison
$300 for code, multimodal and video model is very cheap
I know. I pay about $700 every month with APIS
pure anvil
$300 for code, multimodal and video model is very cheap
In that case the gemini ultra sub would also be competitive
elder burrow
have yall seen grok 3 mini reasoning high price to performance
pure anvil
2.5 pro + veo3
indigo hazel
if it's truly smart, it shouldn't need tools
in fact it's the right comparison. a student who can do 1+1 using his own mind doesnt need to use calculator to do it. it's the same. is there a benchmark where llms cant use tools?
elder burrow
have yall seen grok 3 mini reasoning high price to performance
20x cheaper than 4, 2nd fastest model, beats 4 in a benchmark or 2
keen fulcrum
claude max is cheap if you do code
whole sundial
coming soon: Dork 4.5Vo SuperHeavyDuty Ultra DeepThink - super duper early preview available in SuperDuperDork Pro Max+ for $1,000,000 a month and $10,000,000 a year
may get 35% on arc agi 2
elder burrow
coming soon: Dork 4.5Vo SuperHeavyDuty Ultra DeepThink - super duper early previ...
😭
sour spindle
Is grok 4 heavy only available on the super premium $300
elder burrow
btw yall seen the last few secs of grok stream? they revealed timeline, coding model in august
sour spindle
Don’t know if grok 4 then is worth it compared to o3
blazing bison
GPT-5 is still months away
elder burrow
Is grok 4 heavy only available on the super premium $300
yes
not even available in api
sour spindle
Does seem like all the impressive benchmarks are grok 4 heavy
elder burrow
GPT-5 is still months away
sam said "in a few months" in feb/march im pretty sure
violet adder
@echo aurora Where did Grok 4 disappear to in Direct Chat mode?
elder burrow
not even available in api
@deep adder mr betterknower am i right
mr moreknowing
oh damn
blazing bison
There's no point in discussing this, but GPT-5 is not a July thing
whole sundial
try the arena battle mode
echo aurora
<@283397944160550928> Where did Grok 4 disappear to in Direct Chat mode?
we're working on a fix
whole sundial
i'm surprised i got grok 4 on the first try in the battle mode
it didn't even reason, it just spat out the correct response
echo aurora
i thought its intentional
nope, just was giving us troubles in the other modes. no ETA on when it'll be available in direct/side-by-side but is something we're working on
blazing bison
i'm surprised i got grok 4 on the first try in the battle mode
I could be wrong, but I think you can't disable Grok 4 reasoning
indigo hazel
it didn't even reason, it just spat out the correct response
bro looked for it the on web, it found the respond in the first link and then gave it to you lmao
patent bane
is grok 4 better than o3 or 2.5 pro?
blazing bison
is grok 4 better than o3 or 2.5 pro?
short answer: no
patent bane
is it just a hype?
blazing bison
also no
pure anvil
Did yall see the grok 4 demo livestream? elon is ket'd out
whole sundial
if it was doing that, we would see Llama 4 Maverick: the sequel
blazing bison
is it just a hype?
It's good, but there's no reason to switch
whole sundial
the response began like 1 to 2 seconds after i hit enter, it did not even have time to search and reason
blazing bison
Llama will be fine now. They hired very experienced people
whole sundial
but tools are likely disabled on the arena
whole sundial
Llama will be fine now. They hired very experienced people
ik, they got OpenAI's best minds now and some others as well
i was just saying that if xAI did use tools on the model in the arena, something like that would happen again
elder burrow
nope, just was giving us troubles in the other modes. no ETA on when it'll be av...
👍
whole sundial
but i don't think they are using tools, it's just model output
there was a search arena on the old site, likely still usable
but this is just the standard arena on the new site
i wonder why the Step-1X edit and SeedEdit 3.0 models only appear in Arena mode for Image editing. I assume this is something the model owner set in place? but you can access Seedream 3.0 just fine, so idk
pure anvil
he's so jittery
rare python
i wonder why the Step-1X edit and SeedEdit 3.0 models only appear in Arena mode ...
Bagel
whole sundial
and bagel, forgot about that one
whole sundial
bagel is a separate open source model they made based off of Qwen-2.5 VL 7B (I think)
hollow ocean
misclick $3000/yr
cedar tide
We want grok 4 on direct chat
echo aurora
We want grok 4 on direct chat
so do we! just need to work out a few issues first
winged wing
its not being ran in battle mode either, no?
echo aurora
its not being ran in battle mode either, no?
its in battle mode
lilac nimbus
Grok4 seems great I try
cedar tide
@echo aurora the site is all bugged
echo aurora
<@283397944160550928> the site is all bugged
Thank you for flagging
indigo hazel
Sorry everyone
dont worry, it's only 9 am. take your time
whole sundial
is it related to grok 4 or is it something that just... happened?
echo aurora
is it related to grok 4 or is it something that just... happened?
Looks unrelated
echo aurora
indigo hazel
grok 4 hacked lmarena
using tools lmao
rapid merlin
wasn't grok in a controversy like a tiny bit ago?
where it "shared its thoughts" on some twitter replies
blazing bison
the mecha thing
rapid merlin
yeah
keen fulcrum
wasn't grok in a controversy like a tiny bit ago?
It got instructed to not shy away from political incorrect statements if it can provide sources that claim his position
winged wing
@echo aurora does basically every model provider ask for the 1 day heads up thingy?
keen fulcrum
Its now the second incident where grok ran crazy 🤪
rapid merlin
Solid cover!
winged wing
if the llm doesnt seek the truth that you think is correct, you make it. At least this is what elon thinks.
blazing bison
They said it could go crazy. It's an experiment
ornate stump
Just woke up and saw all the bench—I'm a little hyped, not gonna lie. But where's Grok 4? Is it just an announcement?
echo aurora
Site should be working again btw 👍
blazing bison
@ornate stump battle mode only
keen fulcrum
if the llm doesnt seek the truth that you think is correct, you make it. At leas...
Elon uses the same mentality as with spaceX (fail until succeed with rapid testing). Releasing such instructions without testing internally is irresponsible
echo aurora
<@283397944160550928> does basically every model provider ask for the 1 day head...
I’m not sure what that is tbh 
keen fulcrum
I’m not sure what that is tbh <:pikaconfused:398202117493620740>
Do you have a rate limit for grok 4?
echo aurora
Do you have a rate limit for grok 4?
Yeah, I’m not sure the limit, can get more info tomorrow
winged wing
I’m not sure what that is tbh <:pikaconfused:398202117493620740>
i found it highlighted here
echo aurora
i found it highlighted here
Oh gotcha, I’m not actually sure
whole sundial
now the site is just... slow
winged wing
mhm i see
whole sundial
and that thing happens where your chat history gets wiped and have to accept the terms of use again... except you can't!
echo aurora
Site is struggling again
Apologies for the issues, I spoke too soon about it being fixed
whole wagon
All that grok 4 induced demand 
cedar tide
Anyone have webdev example from grok 4 ?
burnt pulsar
Is grok 4 heavy also planned to be available?
blazing bison
Is grok 4 heavy also planned to be available?
If the price is reasonable
blazing bison
Is grok 4 heavy also planned to be available?
But 99% chance no
burnt pulsar
Understandable, it's the priciest of the new Grok 4 models.
main gulch
grok-4-heavy is not available even in the official API yet
sullen finch
why did they remove the ability to attach files in direct chat
burnt pulsar
That did never work when I tried to attach files. 🙂
mystic mica
I can't even make it past the Cloudflare verification
sullen finch
why did they remove the ability to attach files in direct chat
for some ai models
sullen finch
That did never work when I tried to attach files. 🙂
everything worked fine for me with claude, but now there is simply no possibility
burnt pulsar
Maybe there is no disk space left? 😉
sullen finch
Maybe there is no disk space left? 😉
but for some models this option is still available
soft kernel
If the price is reasonable
No way it'll be too costly
Why they didn't release an image gen?
burnt pulsar
but for some models this option is still available
The AI gods give it, the AI gods take it... 🤖
whole sundial
Why they didn't release an image gen?
because it wasn't ready yet. grok 4 is just 3 with a lot of RL. they are training a new multimodal base model (Grok 4.5?) that will have image gen.
burnt pulsar
Even more impressive if they achieved these kinds of improvements just with a lot of RL with grok-3 as base model....
hardy oriole
Grok 4 not generating answers on webdev arena? Got it two times in a row and only the opposite model generated code
verbal nimbus
Why is Gemini 2.5 Pro so dumb on Github Copilot
It can plan an entire 2 week schedule on AI Studio, but can't even swap two time slots within 2 hrs of each other on Copilot. I ask it to swap slots A and B and it forgets to re-add one of them.
Tried it in AIStudio and it runs fine
keen ferry
I just woke up, where can I see the grok 4 demonstration?
blazing bison
Is grok4 really on lmarena?
On battle mode only
torn mantle
On battle mode only
Yea ik
blazing bison
Why is Gemini 2.5 Pro so dumb on Github Copilot
Because github copilot doesn't use full model context but rag
I think only cursor with max mode offer full model context
soft kernel
because it wasn't ready yet. grok 4 is just 3 with a lot of RL. they are trainin...
Yeah but they are far behind of image gen race
languid crescent
Can't find grok 4 in direct chat?
civic flame
I couldn't get it once
honestly
i've had enough
literally had 60 battles
it's come up 0 times
keen ferry
Can't find grok 4 in direct chat?
only on battle arena
torn mantle
literally had 60 battles
Sigh
keen ferry
what chances of getting it
opaque adder
I just bought grok 4 heavy give me some prompts
keen ferry
we need grok 4 on direct arena
keen ferry
I just bought grok 4 heavy give me some prompts
how much does the monthly subscription costs I might get one
opaque adder
300 usd
keen ferry
wtf
indigo hazel
rare python
lmarena is extremely laggy when both model generating code
indigo hazel
300 per month
.
opaque adder
.
how much does the monthly subscription costs I might get one
Runo — 09:56
I just bought grok 4 heavy give me some prompts
swen — 09:57
how much does the monthly subscription costs I might get one
indigo hazel
but both of them are per year.
indigo hazel
Runo — 09:56
I just bought grok 4 heavy give me some prompts
swen — 09:57
how m...
he asked monthly, but you said the "per year" cost
opaque adder
he asked monthly, but you said the "per year" cost
no
grok 4 heavy is 300 per month
are u braindead
indigo hazel
are u braindead
yes because i didnt see on the top the fact that it also has the "month" section xD
sorry
opaque adder
yeah you are
keen ferry
@echo aurora can we get grok 4 on direct chat it's just impossible to get it in battle
torn mantle
Don't make @opaque adder mad
opaque adder
asura
torn mantle
Runo
opaque adder
i see you here every time
i come into this discord
and i come here once per month
i never miss your username
i havent checked your message count but i assume you have over 40k
civic flame
<@283397944160550928> can we get grok 4 on direct chat it's just impossible to g...
at this rate I have doubts about it even being in battle
torn mantle
Your eyes playing tricks on you
civic flame
or they've deliberately made it ridiculously hard to get
opaque adder
ok thats surprising
civic flame
in which case, why even bother
torn mantle
5k???
opaque adder
grok 4 is not going into lmarena
torn mantle
That's a lot
keen ferry
at this rate I have doubts about it even being in battle
I only got it once
civic flame
HOW HAVE I GOT IT 0 TIMES AFTER 80 BATTLES ☠️
keen ferry
HOW HAVE I GOT IT 0 TIMES AFTER 80 BATTLES ☠️
I got it only once in 20 battles lmao
civic flame
still better than me
stuck orchid
I got it only once in 20 battles lmao
How do you know that you are communicating with Grok4 and not Grok3?
rare python
How do you know that you are communicating with Grok4 and not Grok3?
After voting
They reveal the name
keen ferry
How do you know that you are communicating with Grok4 and not Grok3?
I'm always voting tie and saw grok 4
stuck orchid
After voting
Hmm. I wish I could chat with Grok4 and be sure that I was chatting with Grok4
rare python
Hmm. I wish I could chat with Grok4 and be sure that I was chatting with Grok4
Direct Chat comming soon
stuck orchid
Sometimes Deepseek-r1 says that it is Grok4
rare python
DeepSeek is the mastermind
keen ferry
Direct Chat comming soon
if we beg enough it will appear on direct chat lmao
rare python
It was made from multiple models
torn mantle
HOW HAVE I GOT IT 0 TIMES AFTER 80 BATTLES ☠️
Me too
stuck orchid
if we beg enough it will appear on direct chat lmao
Pineapple says it solves one problem that makes Grok4 difficult to add to DirectChat. Apparently, this is related to the use of tools
keen ferry
Pineapple says it solves one problem that makes Grok4 difficult to add to Direct...
he's online
offline*
rare python
Why did Gork drop?
tidal schooner
Why did Gork drop?
gemini 3 is probably gonna cook hard
also some people said the demos were meh
indigo hazel
why is o3 so low if it's comparable with 2.5 pro?
tidal schooner
why is o3 so low if it's comparable with 2.5 pro?
it’s a tight race tbh
grok 4 seemed kinda rushed to me for some reason idk why
elon was stuttering half the time during the announcement
rare python
elon was stuttering half the time during the announcement
awkward silence
burnt pulsar
As it is a highly competitive landscape, he possibly wanted to get the attention in the summer days.
rare python
nervous laugh
tidal schooner
awkward silence
he had to pass on the actual talk in regards to the model onto his engineers tbh
ngl bro was just yapping about the stuff in his tweet prior to the livestream
tidal schooner
After GPT 5
hope it doesn’t flop like llama 4
rare python
hope it doesn’t flop like llama 4
better than r1 0528 won't be a flop
r1 0528 is a decent model
tidal schooner
better than r1 0528 won't be a flop
basically r1.5
rare python
basically r1.5
But like we need v4 first
The base for RL training
Or they will merge into one model
tidal schooner
Or they will merge into one model
i wouldn’t mind that ngl 🥶
would prob be slower for a lot of tasks tho tbf
fleet pine
does anyone tried deepseek r1 with 600+ billions parameter?
indigo hazel
does anyone tried deepseek r1 with 600+ billions parameter?
the january version had 671b, the 0528 has 684
keen beacon
Hey, do you guys know if Grok 4 is ever coming to direct chat?
cedar tide
@echo aurora grok 4 not working
keen ferry
<@283397944160550928> grok 4 not working
what the hell how did you get it on battle
cedar tide
Anyone see a new mystery model that he dont want to say his name and his good ?
cedar tide
what the hell how did you get it on battle
I will dm you
torn mantle
I will dm you
dm me as well
i want to try it
also just reading some grok 4 outputs i think it wont top lmarena
torn mantle
Why did Gork drop?
because people have tried it and concluded that its still behind
keen beacon
I will dm you
dm me i want to try it
keen fulcrum
dm me as well
Isn’t it available in arena
cedar tide
I dont have a real way to use it, just I increase my chances of getting it faster
@keen beacon
@torn mantle
sacred plaza
What will be higher the amount of companies that will overpay for a Nazi aligned AI model or the amount of trade deals trump signs with other nations (3 so far)
🤡🤡🤡🤡🤡🤡
keen fulcrum
What will be higher the amount of companies that will overpay for a Nazi aligned...
Can you stop promoting fascism
Any model has hiccups of misinformation and wrong beliefs
Its common in the industry
cedar tide
Pikachu by grok 4
stuck orchid
Elon is focused on training the 256K token model and does not want to increase the context window yet, probably because he wants to first achieve something groundbreaking at this context window length before expanding it to a longer context
ocean vortex
So Dork4 AGI? 
torn mantle
So Dork4 AGI? <a:PikaOMG:1149518663288889385>
its good on benchmarks
ocean vortex
its good on benchmarks
yeah and this time there doesn't seem to be much manipulation. Everything is surprisingly clear
even confirmed by AA
rare python
yeah and this time there doesn't seem to be much manipulation. Everything is sur...
getting mixed vibe check from people
ocean vortex
the only way they could have cheated if they are serving different model publicly (with safety alignment and whatnot) than the one which was tested, but that's a reach...
probably even for Elon
torn mantle
yeah and this time there doesn't seem to be much manipulation. Everything is sur...
did you try it yet?
bruh
my brain is fried
i messed up my sleep with the release
ocean vortex
did you try it yet?
only 2 prompts yet. The way they are hiding reasoning is... interesting.
alpine coral
it thinks for so fkn long
torn mantle
only 2 prompts yet. The way they are hiding reasoning is... interesting.
what was this service called again?
alpine coral
at least on OR
ocean vortex
what was this service called again?
openrouter
alpine coral
weirdly underwhleming so far.. like compared to the screenshots of the evals here - was expecting way more
but only just started playing around
ocean vortex
weirdly underwhleming so far.. like compared to the screenshots of the evals her...
it did pass this. The only models able to solve this were either insanely dumb which didn't know the original riddle (Amazon), or Opus4 doing it properly. 2.5Pro and o3-pro fail. It knows the original version since it did mention it:
rare python
it did pass this. The only models able to solve this were either insanely dumb w...
test it on simple bench question 10
alpine coral
yeah it's definitely solid - but i suspect it excels at single prompt, exam/eval-style questions.. i've tried a few questions (non-riddles) that require multiple steps of knowledge recall, which only 2.5 pro and o3 get right, and it fails kinda terribly
alpine coral
its not working for me
i'm using it atm on OR - does it throw an error or like?
hazy quest
On the Wes Roth live testing, he asked how many times a basketball dropped from 100m would bounce if no air friction, and Grok 4 Heavy, after almost 5min of thinking, answered "infinitely many times", which is an absurdly wrong answer to a simple question
Worrying that it gets it that wrong
alpine coral
it kina feels like o1/3-pro
how long it thinks
which can lead to both brilliant and ridiculous respones
torn mantle
i'm using it atm on OR - does it throw an error or like?
error
torn mantle
i'm using it atm on OR - does it throw an error or like?
This request requires more credits, or fewer max_tokens. You requested up to 230367 tokens, but can only afford 2349. To increase, visit https://openrouter.ai/settings/credits and upgrade to a paid account
is it telling me im poor?
keen beacon
hardy oriole
Grok has lost every single "battle" so far in webdev arena for me, probably 6 times total... There's something wrong with the model or idk what's going on lmao
It's producing worse results than even 2.5 flash
torn mantle
nobody knows
keen beacon
OpenAI better step up their game
torn mantle
Grok has lost every single "battle" so far in webdev arena for me, probably 6 ti...
yea its bad at coding
seems like they wanted to focus on other things and make a seperate coding model
keen beacon
Yeah they have a separate coding model for that reason. Interesting choice.
hazy quest
The unhiged voice mode is pretty cool
But every company could do that, they just chose not to for now
keen beacon
Im excited for the open source o3-mini like model next week
hardy oriole
seems like they wanted to focus on other things and make a seperate coding model
Makes sense
burnt pulsar
I'd love to see Gemini 3 getting rid of some character encoding issue that are very annoying in my coding tasks.
cedar tide
discord clone by grok 4
alpine coral
is it telling me im poor?
ahah what shows up on Credits ? I think you need at least like $1 balance (or dial back the max tokens a bunch or something)
ocean vortex
This request requires more credits, or fewer max_tokens. You requested up to 230...
yeah it's not very cheap. To run arc-agi it cost the same as Opus4
hardy oriole
discord clone by grok 4
It's doing much better than in my tries, are you using on the grok website?
ocean vortex
Slightly more even
rare python
cedar tide
r1 0528 in comparaison
keen ferry
discord clone by grok 4
he even added you're pfp into it
cedar tide
It's doing much better than in my tries, are you using on the grok website?
lm arena
keen ferry
why does everyone keep testing it in html and css and not like python or c++
cedar tide
discord clone by grok 4
grok 4 its very similar to grok 3 0224
hardy pecan
grok4's geoguessing abilties aren't so great