#@vito regarding Gemini vs GPT, right now
copper egret
Yeah Gemini takes it like a champ. I run 20 evals against it in parallel and it doesn't break a sweat
OpenAI and Claude routinely fail with one
brave parrot
It's just our API limits right? Weird that we can't get those bumped up
@fervent sequoia for comparison, we have 500rpm and 30,000tpm limits with openai. Gemini has 2000rpm and no token limit
You can try 4o-mini or 3.5-turbo for 200k tpm, but I don't know how they perform
copper egret
It's just our API limits right? Weird that we can't get those bumped up
yeah token limits in particular are probably what bites us atm
until we can get some sort of auto function masking to work
brave parrot
There's some issue we haven't been able to debug yet with the repo Jeremy linked where the test function in the workspace seems to add like 20k tokens each call even though the output is like 80 lines of simple logs
copper egret
does it return a Container?
brave parrot
No just string
copper egret
