stax・🥞

[oblique pewter]

Introducing Stax: a new tool for AI evaluation. 🥞

Stop guessing if your AI is working. Stax gives you the hard data and flexible tools to see what’s really working in your AI, so you can build breakthrough products with confidence 🔥

Try the product for free at stax.withgoogle.com.
💡 Learn more about Stax at https://developers.google.com/stax/overview 🥞
💬 The Stax team would love to hear your thoughts right here in this channel! Whether you’ve found a bug, want to share what you’re evaling, or learn about best practices, this is the place to be look out for folks with the next to their names!

[little peak]

when I try to login/signup I get this

[rare egret]

@little peak Thanks for your feedback. Are you trying to sign in with a workspace account or gmail? Also note currently Stax is only available in US

[dense python]

first

[patent sun]

Yes unfortunately this is the error when you aren't in the US, but we do plan to expand to other countries soon! Let us know if there are particular regions you'd like to see Stax in

[stable gale]

Third

[vapid dagger]

Same error with Gmail

[crimson star]

Explain to me like im 5 , this AI tool is for what?

[dawn lily]

sad

[oak path]

or where can we see the documentation to ask nicely NotebooksLM to anwser with videos

[rare egret]

Thanks for your interest. Stax is a LLM evaluation tool. Check out https://developers.googleblog.com/en/streamline-llm-evaluation-with-stax/ to know more

[halcyon sphinx]

whoaaaa

[rare egret]

You can find Stax documentation here https://developers.google.com/stax/overview

[stark quarry]

Only US

🙁

[restive canyon]

Yooo

[oak wren]

hmmm

[woven spear]

Getting the XML error, be kind make an HTML page that says only available in the US

[weary hazel]

but still doesn't work on firefox

[limber summit]

Hmmm

[final meadow]

Works with VPN or no?

[echo palm]

Hey guys vpn please oursufe of US please

[limber summit]

If it's geoblocked then VPN also doesn't help cos it needs login and the region is set in my Google account

[wary latch]

Is it possible to use it with best answers as reference?

[limber summit]

Was/Is the same with other geoblocked Services/Labs Projects.

[patent sun]

We're on it right now!

[patent sun]

In terms of expected output for the model? Yes!

[shell valve]

Cool

[shut phoenix]

Same as all the other non us sign ups! Looking out for an update

[wary latch]

Great! Untill now I have used my own vibecoded evaluation app. Will try out Stax. 😀

[shut phoenix]

Is this intentionally US only or an error?

Have been using langtail so excited to try this

[little peak]

Thanks for your answer. I'm in Portugal.

[patent sun]

Intentionally US since early experiment, but we do have plans to expand if folks share where they'd like to see Stax

[patent sun]

We'd love to hear how it works for you!

[glacial vortex]

Is it possible to add custom models via OpenRouter API key?

[patent sun]

Not yet, but we can add that to the backlog

[shut phoenix]

That's gutting, any non us workaround or exceptions. I consider myself a citizen of the world if it helps!

[glacial vortex]

Thanks! Would really open up the options for testing widely without having to get individual keys for loads of models.

[weak bloom]

could this be used to perhaps automatically tune a system prompt for example to fit my own criteria?

[patent sun]

Due to legal regulations, it's just takes some time, but will plan to do it soon

[wary latch]

GitHub models is also good for evaluations. Easy way to test many models easily.

[wraith turret]

Taiwan, please.

[runic sigil]

Germany pls

[patent sun]

Not automatically, but you can test how prompts compare.
Do you already have golden eval sets and you want a system to create good prompts for you?

[restive canyon]

Uk pls

[quick musk]

on release, broken

"POST" is not supported

"Unexpected Error: Something went wrong. Details: Request method 'POST' is not supported"

[limber summit]

I'm Ireland

[patent sun]

Could you provide a bit more context? What were you trying to do? If you have a screenshot that would be great, or this button in the product makes it really easy to provide feedback.

[quick musk]

Here is a screenshot. Just trying to compare DeepSeek and Gemini

[patent sun]

We're on it!

[amber cedar]

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
</Error>

[patent sun]

Unfortunately Stax is only avaliable in the US, but we're working on making this error more informative!

[amber cedar]

ohhh thank you!!!

will there be any update on when it'll reach other countries (👀 new zealand)

[patent sun]

Not yet, but the more feedback we hear, the higher we'll prioritize it!

For all folks who are blocked because of the country restrictions, I'd love to hear what you're looking to use Stax for!

[amber cedar]

i would be keen in seeing how it works just generally and especially am interested in comparing models at different types of tasks (especially general knowledge comparisons, and data manipulation)

[sonic nexus]

@patent sun can not access it how can access outside of us

[patent sun]

Yes unfortunately only avaliable in the US for this initial launch

[cosmic mulch]

I can't go in.

Ok, I got it

[sonic nexus]

@patent sun hmmm any plans labs and real agent mode for pro users or do you have any suggestions? Cli updates?

Maybe it out of your knowledge base.

Is out*😅

[patent sun]

I am only here to talk about Stax! 😄 😉

[patent sun]

Let me know what countries though!

[sonic nexus]

How can access stax without living in us 🫣 heard it should be available beta. Sweden please. Is possible ultra subscription think model when I am in pro sub. Any plans for merchants place?

[patent sun]

Will work on getting Stax to you in Sweden! For the other feedback I'll let the Ultra team know you want the thinking model

[quick musk]

Status?

[cosmic mulch]

Kazakhstan

And what else could be the problem? If anything, I'm comparing two models (Google and openai) And there are API keys for both

[quick musk]

Same here!

Supposed to be fixed coming out

[cosmic mulch]

I'd like to look at AI coding from an engineering perspective, not vibe coding. After all, real coding and objective evaluation are more relevant and safer than code written using fluids.

[patent sun]

We're working on this right now!

[brittle parcel]

I clicked that pancake emoji yet getting access denied error on web

Is that unavailable in India?

[vestal linden]

It sounds so interesting, it would be great to have access to it and try a few features,
I have access denied now.

[lusty holly]

wow

[lusty holly]

simply perfect

If anyone needs help, they already know where to find it.

[stone turtle]

Would love to know when support in Canada is coming out

[patent sun]

Not available in India or Canada yet, but we're talking to legal!

[lusty holly]

I'll show you two prompts, which were refined on my opal, and were tested on stak

[lucid estuary]

woo

[lusty holly]

These two prompts basically have nearly 16 processes, not even counting the OPaL refinement steps.

[lucid estuary]

ok?

[lusty holly]

👀

[noble talon]

this has been fixed

[deep dragon]

Please open for South Korea too!

[lusty holly]

of 81 refinement flows in one step, all failed

result of 31 prompts, refined with 6

It didn't work for me, I lost it

[last pivot]

@patent sun I'm from India, and I am unable to access Stax. Any update?

[quick musk]

It's rolling out to different countries eventually

also, I still got nothing

[patent sun]

Working on it right now!

[last pivot]

Ahh okk. I got it. Thank you

[last pivot]

Ok. Thanks.

[lusty holly]

almost 81 tests

there are 187 tests, and I didn't get past 32.6

[patent sun]

Can you show us the error? Also, at the moment Stax only works for text, so if you were trying to output images, that might be the reason (although we're working on image support!)

[lusty holly]

no, they are test refinement tests

[lusty holly]

here of 31 only 6 gave a result

I didn't think it would be so difficult to refine text

[brisk heart]

@patent sun Hello! I'm in Korea, but I can't use a stax. Could you please do something to make my account available?

[patent sun]

It's currently only avaliable in the US, but working on opening up access!

[woven spear]

Thanks Prashant!

[junior coral]

I would like to access stax from India tho

[patent sun]

working on it!

[junior coral]

Well thank you, I would look forward to its release in India.

[light flume]

Is it possible to add images to the input prompts to be evaluated?

[patent sun]

Not yet, but we are working on it!

[patent sun]

What are you trying to evaluate?

[light flume]

I use Gemini to extract information from construction blueprints, however, because these have very complex and diverse layouts, I am trying to test which prompts provide the most effective extraction from many drawings.

some examples @patent sun

[patent sun]

That's very cool. Images is going to take a few weeks, but we're working on it! Are they PDFs?

[light flume]

Originally, yes; but not when I shared them with Gemini. I converted PDFs to PNGs, then I share PNGs only.

[brazen shoal]

Hello. Thanks for Stax, it's an amazing tool ! I think you really nailed the workflow for evaluating Gen AI applications 🙂

Do you plan to support Vertex AI in the coming months (e.g., with Service Account key file) or propose Stax from GCP ? This would help us use Stax in our organization.

[mint laurel]

is it possible to evaluate generative images also in stax ? would stax be going in the image segment also or would it be more into the text only evaluation segment ?

[royal lance]

@patent sun Hello Swilt, is it possible to add another AI provider to the list? If yes, could you explain how? Thanks.

[cosmic mulch]

What does it mean?

[cosmic mulch]

And it keeps me away from the site forever.

[livid summit]

Pls can someone answer my question

I'm in Nigeria and when I tap to use stax in my chrome browser, it said it's not available in ur country. But still it redirect me to this site

And said "if u want u use stax u must tell us here"

Pls explain

[weak zinc]

I'm hoping to get a chance to try it from Italy.
Do you have a release date for people outside of the US ?

[lusty holly]

looking like your Gmail

[cosmic mulch]

Yes, it's mine

[lusty holly]

session is leaving on its own, and excluding the training

?

[patent sun]

Not yet, but in discussions with legal

[patent sun]

Are you able to sign in?

[patent sun]

Which provider?

[patent sun]

We are working on images! What specifically do you want to evaluate?

[patent sun]

We'll put this in the backlog!

[cosmic mulch]

Thanks a lot! It was fixed

[sweet wigeon]

how to use?

[sweet wigeon]

hello <@&1366831844502274180> could you please make this available for me

I am from India

[cerulean ruin]

hello, this is not a moderation issue. please only ping mods if there is an active issue that goes against server rules. Thank you!

[sweet wigeon]

ok

[royal lance]

Minimax ai wants to join the list, if I can link them up with you, that’d be awesome.

[patent sun]

Stax is only avaliable in the US, but we're working on opening up access!

[sweet wigeon]

there are other tools as well which are not available, is there any other way I can access?

[royal lance]

@patent sun Than you. how would you prefer to move this forward? would it be alright if they contact you directly to discuss what’s needed, or is there a specific channel you’d like them to use?

[patent sun]

What country are they in?

[lusty holly]

not yet

mine basically ends the session, and even deletes the process done, everything simply disappears

This is the third time I've been doing refinement and the session simply ends and everything disappears.

then you need to clear the entire browser cache

I did these two with training to refine, then I refreshed the page and basically everything disappeared

the result of this refinement simply disappeared 😩

These two refinement processes I worked on were basically 187 prompts that generated 60 results.

basically the session ended and took the whole process ☹️

@patent sun

[weary hazel]

I couldn't understand the resultant of Stax. Would you mind telling me how it's related to Veo 3 video?

[lusty holly]

Basically, it was 187 prompts in the refinement flow. The results came from 60 of those refined ones. In that process, I ended up losing the refinement path.

[weary hazel]

ohh

thanks for explaining 😅

[weary hazel]

thats a lot of stuff lol

[royal lance]

Shanghai, China, they develop and operate their own llm models across text, video, and multimodal. you might already be familiar with their image and video models under Hailuo ai.
https://www.minimax.io

[weary hazel]

thanks!

[rare egret]

Perhaps your session was idle for a long time and it timed out. Did you try to sign in again?

[cosmic mulch]

Yes, I tried and everything went well. If someone from the team fixed this problem, then special thanks to him.

[dense folio]

Pls add Germany for this

[lusty holly]

вы поняли?

[dense folio]

i can't speak russian

[quick musk]

so...

stax seems to be still broken

[patent sun]

Oh no! Can you show me?

[quick musk]

same post error

yeah, just "Unexpected Error: Something went wrong. Details: Request method 'POST' is not supported"

[patent sun]

Yes, we're still debugging that, but working on it!

[quick musk]

ah okay :)
cheers! just got curious!
if you need my help lmk. i'm available today

[lusty holly]

speak what language

[dense folio]

english/german/spanish

NO russian

[patent sun]

We'll let you know when the post issue is resolved!

[crystal vortex]

Can anyone explain to me like a 5 year old what Stax actually is and how it works?

[patent sun]

Made this with the help of Gemini 🙂

Imagine you have a super smart robot that can write stories for you. But how do you know if the stories are any good? Are they exciting? Do they make sense?

Stax is like a special helper that reads the robot's stories and gives them a grade.

Here's how it works:

• You give Stax the robot's story. This is like turning in your homework.
• You tell Stax what makes a good story. Maybe a good story needs to be funny, or have a happy ending. These are your rules.
• Stax reads the story and checks your rules. It's like a teacher with a checklist, making sure the story has everything it needs to be great.
• Stax gives the story a score. This helps the person who built the robot understand if it's a good storyteller, or if it needs more practice.

So, Stax is a tool that helps people check if their smart computer programs (AI) are doing a good job. It makes sure the AI is working the way it's supposed to.

[quick musk]

"Vibe-benchmarking" is no longer a thing. Actually analyze

[lusty holly]

no problem

[patent sun]

I love this!

[patent sun]

Do you vibe benchmark?

[quick musk]

nah not really

i do most of the work myself

[royal lance]

@patent sun Any updates on this matter? Thank you

[quick musk]

it's happening we swear

[royal lance]

What is happening?

[patent sun]

Very cool! Okay unfortunately we probably will not roll out to china for a while

[quick musk]

the fix pretty sure

[patent sun]

i do most of the work myself

[lusty holly]

I'm Brazilian, is there a problem if I respond with my cloned voice in other languages?

[royal lance]

Ok, thanks for the feedback.

[royal lance]

the thing is, i noticed deepseek is already integrated, which makes me a bit confused.

[lusty holly]

?

[silver glacier]

How can I get access to Stax?

[quick musk]

what country are you in?

it's US only (currently! expanding soon...)

[silver glacier]

Turkey

[quick musk]

nice! @patent sun ^^

[quick musk]

also, i noticed something kinda funky here;
curling Gemini does use POST. in the literal code that it gives you

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent" \
  -H 'Content-Type: application/json' \
  -H 'X-goog-api-key: GEMINI_API_KEY' \
  -X **POST** \
  -d '{
    "contents": [
      {
        "parts": [
          {
            "text": "Explain how AI works in a few words"
          }
        ]
      }
    ]
  }'

the bolded. POST. that seems... funky.

i'm not gonna question it, but i'm just more or so directly pointing it out for y'alls

[lusty holly]

👀

[ionic condor]

I can't access stax in Nigeria

please, resolve this asap

[subtle vector]

Lovely new product, i'd love to try stax, we are using langfuse and it's quite confusing!

[ionic condor]

Alright
thanks

[patent sun]

Have you tried it yet?

[fading arrow]

Yes, I tried and everything went well.

[subtle vector]

I can't sadly, i'm in 🇨🇦

[patent sun]

Okay yes, sorry, working on opening up access!

[subtle vector]

🤞 can't wait to try it

[patent sun]

the POST error is fixed!

[snow flicker]

Can we use stax in UK

[lusty holly]

one more try

[patent sun]

US only atm

[lusty holly]

I made two variations of this

What were those two seeds

That I took these two seeds, that I generated this

I went to change tabs

@patent sun ?

[patent sun]

is it okay when you sign back in?

[lusty holly]

this is happening all the time

[patent sun]

Can you clear your cache and try again? If possible if you can send me a video that would be great!

[lusty holly]

I even do that, and for a while it works. But then it happens again, and I have to clean it up all over. That way I lose the entire process. If I run a long process and this happens, I lose it completely.😅

[patent sun]

This is happening right now?

[lusty holly]

all the time

[patent sun]

Would it be possible to DM me a video?

[empty ocean]

Hi I’m in Taiwan

Would love to try Stax. I’ve been using braintrust

[patent sun]

Would love to try Stax. I’ve been using

[quick musk]

YEESSS!
gonna test rn!

[patent sun]

Let us know what you think!

[quick musk]

Did the classic "What is bigger? 9.9 or 9.11?" and Gemma 27b and 2.5 pro both answered right surprisingly!

Crying laughing 😂
Perplexity can't even get it right

Gemma is a local beast, bro
That's gonna be the AI that's controlling my HA setup soon

[small hollow]

@gritty domethanks ill check it out👍

[mint laurel]

something like the reality score of the generated image kind of ... as in how fake the generate image looks ....

and also say if the description given within the prompt is correctly depicted in the image or not

[finite schooner]

Heyy When global access?

I am from Taiwan

[spiral musk]

hey everyone

[pine crown]

hello to all

when will stax will be available in europe?

[spiral musk]

i was wondering whether i can actually use that eval dataset (or part of it) sitting in stax to automaticlly improve my original prompt with good/bad examples?

[pine crown]

seems a great product!

how can I access it from EU ?

[quick musk]

Not right now

Us only

[placid notch]

I would like to try Stax, but it seems it is not available in Japan yet

[pine crown]

we have to find a way to access it 😄

[frail herald]

For the automatic evaluator, can we change what model is doing the eval? Or is it a custom built model to test the output in a certain way ?

[patent sun]

i was wondering whether i can actually

Did the classic "What is bigger? 9.9 or

[patent sun]

You can change the models! The example prompts are calibrated to gemini, so I would just test to see if the autorater output aligns to your human ratings (which you can also give in Stax)

[carmine vale]

Hey, can i use stax for AGI- abstract reasoning benchmarking ? (Like solving abstract problems) or is it for LLM for specific uses only ?

(Ive never tried stax, but in considering it)

[patent sun]

I'm not sure what you mean? Can you give me some examples?

[sand trail]

South Africa

[opal raven]

Hi y'all! Is anyone exploring improving spatial reasoning for LLM-based evaluators? Super excited to test out Stax but I do wonder about the quality of LLM-as-a-judge for spatial correctness checks

[drowsy cosmos]

Hey! My free API key allows me to sign in momentarily and then I get the sign that my session has expired. Is there a certain amount of money I should be expecting to spend to mess around in the sandbox a little bit?

[patent sun]

Is this for images?

[patent sun]

Hey Sarah! We're having a sign in bug right now, but we're working on fixing it

[drowsy cosmos]

okay, I assumed it was me

glad to be helping identify stuff. Chat in 20 minutes

[patent sun]

If you could DM me a screenshot that would be great! I assume you are seeing "Session expired. Please sign in again"

[drowsy cosmos]

totally, it flashes pretty quick but I can do that

[carmine vale]

Nevermind i was confused about something 😅, im so so sorry for bothering you 😅😅😅

[opal raven]

yep images and videos! Either on the image itself or the code that compiles it

if LMs aren't performant I'll be thinking abt a way to do this with rule-based checks 🤔

[patent sun]

yep images and videos! Either on the

[patent sun]

@drowsy cosmos this should be working now!

[patent sun]

This is fixed!

[main grotto]

Hi stax team! I wanted to try stax, but unfortunately it is not yet allowed in my country. I just submitted the form, and join the discord to get to know more about the tool 🙂 thx!

[elder tapir]

Not available in Argentina neither Peru 😩 . When could i be able to try it? Thanks 🙂

[ruby flare]

Any way we could have native json importing?

[jaunty pagoda]

Very impressive and seems very user friendly. I look forward to learning more with Gemini.

[patent sun]

Sounds great, we're working on opening it up in more countries!

[patent sun]

Can you give me more info on what you want to input? If you choose "chat format" that will work with chats in JSON format

[ruby flare]

Correction I was just not sure what I was doing.

[ancient root]

Japan please

[smoky pelican]

india

[patent sun]

If you want to send me a DM of a sample of your data I can help make sure it works in Stax!

[ruby flare]

So I had initially grabbed a sample of the R1 reasoning dataset just to play around with it. Which was a bit of a process to move from their json to csv. I haven’t put a lot of time into it but the last checkpoint was the evaluations screen timing out trying to load the 10k examples.

My actual goal requires moving about 10ish long form reports I’ve made via a very defined system prompt that can produce 10k word reports into a usable csv. Then using Stax to improve my “deeper research” prompt itself using existing report creation planning and output as a dataset.

TLDR: stax doesn’t like the R1 reasoning dataset and I’m still in the process of coming up with the right way to make the dataset itself for the deeper research prompt refinement project.

I’ll still circle back on this once I’m back at my desk.

[patent sun]

If you are willing to DM me a sample of this data, we can try it on our side to make sure it works!

If you upload data with the chat format, it supports JSON chats in a CSV

[ruby flare]

How I missed the chat toggle is beyond me. Ha. That answers a lot of questions I had. Doing some digging this was the initial dataset I was trying to work with. Note: it was more a test to understand Stax. I have no real understanding if this is the right tool or workflow.

https://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M/blob/29d2baec029f630403382b1db2e3bd2172315c73/am_0.9M_sample_1k.jsonl

[ruby flare]

What I was initially confused on(which the chat toggle I assume addresses) is that there were more columns than options to tag them with. So most of the thinking aspects I tagged as metadata. Which seems incorrect thinking about it now.

[patent sun]

What I was initially confused on(which

[dawn lily]

hey i am non US

i would like to use this service

[dull nexus]

Hi Team, I am from India
Could you help me understand how can I get access to stax?

[shell coral]

Hello Team , I am from India

Could you help me undetstand how can I get access to stax?

[fringe bobcat]

I would love to use this from Singapore! I've had to use a US VPN and another US Google Account for now

This is shaping up nicely, I'm using Promptmetheus as my prompt engineering IDE and there's really not much competition. It's good to see Google getting into this critical space for us AI developers.

[patent sun]

Have you used Stax yet? What have you been using it for?

[fringe bobcat]

Using it to evaluate sentiment analysis of customer call logs! Comparing OpenAI vs Gemini models. This is for my clients in CX

[humble wave]

Which stax region is available

?

?

[patent sun]

Just the US for now

[lusty holly]

Hello my friends
anyone need help?

[humble wave]

Bro, I'm from Brazil using a VPN and it didn't work.

It worked with VPN

Thanks

[patent sun]

Are you able to log in now?

[patent sun]

Just US for now, sorry!

[lusty holly]

try

[lusty holly]

I came back in the best shape

I built a complete flow in Opal with 15 agents, where they refine ASMR prompts that are already prepared. After that process, the result is a single prompt reviewed by 16 agents. Then comes the STAx stage, where I can further refine this result.

So, I created two TERTs and refined them for other models, generating a sequence of 200 prompts. From those two that I first redefined in Opal, I used the evolution process and produced more than 200 prompts based on those initial two I made in Opal.

result of a refinement of 200 prompts, ChatGPT 04 mini model

this was the result of 30 model 5 nano ChatGPT

[lusty holly]

👀

[median mural]

Please open up to Australia

[obsidian ravine]

PLS OPEN UP FOR MALAYSIA

[chilly wing]

Please open for the Netherlands 🙂

[vestal patrol]

Please open up to Singapore🤗

[storm lotus]

How to get access to stax on this page it says you can apply here

[hearty dust]

Australia please

[lofty osprey]

Please make it available in BRAZIL

[brittle schooner]

is there a possibility to make stax available in germany ?

[oblique pewter]

if you're located in the U.S., you can try Stax at http://stax.withgoogle.com/landing

📣 Interested in gaining access to Stax outside of the U.S.?

Please reply in this thread with the country you'd like to be added! If you've previously shared, your response has been recorded.

[storm lotus]

I'm using vpn but how do you even use it

[patent sun]

Can you try again?

[wheat rampart]

Any idea when STAX will be available in Belgium?

[shut phoenix]

Or updates on UK release

[red roost]

how is stax different than other benchmarking/evals platforms out there?

[rare egret]

Thanks for your interest. Stax moves beyond generic benchmarks by letting you evaluate AI models using your own specific data and criteria. Its key feature is the ability to build custom "autoraters" (LLM-as-a-judge) to grade model outputs on your unique quality standards. Stax is free, although you have to bring API keys. We are constantly working to make this tool more useful and open to hear your thoughts for improvements.

[mint depot]

I'm trying to use a custom model that's hosted on Amazon Sagemaker (URL is like "https://xxxxx.execute-api.us-west-2.amazonaws.com/prod/models/test/invoke"). It uses the OpenAI response format. It works when I call it from my command line. However, in Stax it fails and I'm getting a "403 forbidden" error (see image). Is there a way to debug the request that Stax is making so that I can try to figure out what's wrong with the request?

[patent sun]

Sending to the eng team and will get back to you!

[analog ruin]

hello

[urban cloud]

hello everyone

[unique cave]

When will stax available in India?

[urban cloud]

2026 soon

[rare egret]

Hi Yashjit, Can you please confirm if its still the case?

[weary hazel]

Hi, thanks for considering the issue. I checked and can confirm that it's working

[patent sun]

<@&1410311843674849300> - exciting news!

Stax is now available in 10 new countries/territories:
🇧🇷 Brazil
🇨🇦 Canada
🇭🇰 Hong Kong
🇮🇳 India
🇯🇵 Japan
🇲🇽 Mexico
🇳🇿 New Zealand
🇵🇰 Pakistan
🇰🇷 South Korea
🇹🇼 Taiwan

We're working on getting access to more countries. So stay tuned!

[terse hornet]

Vietnam please

[neat ferry]

Hi! When would Stax be available in Singapore?

[mossy dragon]

We are working for availability in other regions as well, will do that soon

[neat ferry]

Ok! Thank you!

[queen monolith]

Hi! I am a developer working on a LLM-based chatbot for my company. Can I ask how Stax ensures consistency across multiple runs of the LLM-as-judge evaluation? Based on our experience using other LLM-as-judge tools, even if the dataset and LLM is the same, the evaluation ratings might differ because of the LLM judge.

[patent sun]

Hi! I am a developer working on a LLM-

[raw hamlet]

Is there any ETA on when Stax will be available in the UK?

[mossy dragon]

Currently there is no ETA available. Will share once we are ready.

[bronze rover]

Demo looks cool but still waiting for availability in Germany.

1. Is it possible to evaluate Vertex AI Search results? So instead of evaluating LLM's I wish to evaluate an app endpoint.
2. Would I be able to collect feedback via Stax API from my app frontend? Currently Vertex AI Search has this feature only via the widget.

[restive juniper]

Hi,
I have been trying out Stax a bit but have a few questions:

• Is it possible to change the thinking budget on models, especially flash 2.5?
• The sample csv for import is locked. Can you share it with me?
• Will it be possible to try more than 2 Prompts/ models at the same time?
• Will there be a way to share projects with other people in the future?
• Is this tool expected to be locked behind a subscription in the future or will there be a free tier?

[fading arrow]

we are fixing the CSV import issue, you can for now use this https://developers.google.com/stax/datasets#csv_upload.

[mossy dragon]

We are checking this with our internal team about evaluating the endpoints.We appreciate your patience.

[patent sun]

Demo looks cool but still waiting for

[night notch]

Is it possible to use stax to test my custom agent that uses Llama 3.18 b instant model via Groq api?

[mossy dragon]

Yes, Stax can be used to evaluate custom agents.

[covert cloud]

Hey, how can I share my projects with my team so that they can take a look?

[mossy dragon]

Which project sharing are you reffering to? Stax..?

[patent sun]

Hey, how can I share my projects with my

[covert cloud]

Yup, Stax projects

[zenith haven]

India

[mossy dragon]

Stax launched in India, Try it here https://stax.withgoogle.com/landing/index.html, and let us know how you feel and any queries you have. Thank you

[raw hamlet]

Hello there, I keep on getting the following error when I try to generate an output. Is anyone else experiencing this:
"inference_reason": "Error while sending request to Gemini API: Retry failed after 3 attempts" - Can someone help me troubleshoot?

[mossy dragon]

Check once with your api-key, might be a problem with your api-key. Try changing the key.

[patent sun]

Hello there, I keep on getting the

[raw hamlet]

Hey guys something really odd just happened, I was creating my own evaluator and after making the changes in the editor but before I could click on save ,all the content got replaced with content from another evaluator.

[mossy dragon]

1. Did you change/move to any other tab/page before clicking on save? In this case evaluator data is lost. 2. We will raise your concern with our eng team and will respond back for resolutions. Thanks for bringing up the issue to us.

[covert cloud]

Hey is there a maximum character limit for output column?

[patent sun]

Hey guys something really odd just

[patent sun]

Hey is there a maximum character limit

[minor thunder]

Hello Guys, I think Stax is now available in South Korea! Does anyone know how to add documents like PDF files as user input data? Does Stax only supports text-based data at the moment? 🧐

[mossy dragon]

Yes, Stax is available in South Korea.Currently you can upload only CSV files and accept text as user inputs. We are working for adding other formats as well.Thank you

[mossy dragon]

Can you provide more context/ what you are trying to achieve and any use-case focused on ? Whether you will upload a single pdf file for multiple user inputs or individual files as an input for each?

[minor thunder]

I'd like to upload multiple files for a single user input. It seems like only text input is available on the playground screen.

[mossy dragon]

Yeah, Currently you can work only for text input, our team will work on adding other options.Thank you for your information.

[opaque pivot]

Hi! The documentation page is not working: https://developers.google.com/stax

[mossy dragon]

Can you please share more info about the issue you're facing so we can answer better? Still getting the same error?

[spiral skiff]

Hi! Really looking forward to have access to Stax in Malaysia

[mossy dragon]

Thank you for your interest! We are working for launches and we'll be sure to update our community as new regions are added.

[wintry surge]

I'm hoping to try and add an antrophic model from vertex. Anyone done that?

[vague galleon]

I swear discord hides the good stuff until it’s too late 💀

[mossy dragon]

Hi, Have you tried this?

[plain hound]

Would love to start using stax in BE 🇧🇪

[mossy dragon]

Great! We're currently working on making the tool available in more regions and will share updates as soon as we can.

[little gorge]

amazing

[hushed furnace]

When will stax be available in Germany?

[mossy dragon]

We will update in this community , once we have an update.

[little gorge]

ok

what is that

[solemn totem]

we need stax in POLAND 🇵🇱

[mossy dragon]

We are working for availability and will update soon.

[oblique pewter]

Regions Thread 🗺️

Thanks everyone for sharing where you're interested in seeing Stax expand to. To help keep conversations organized, and to help keep track of these requests, please reply to this thread with the countries/regions you'd like to see Stax become available in.

If you've already made a request in this channel, you do not need to also post in the thread. Thank you!

[mint panther]

hello, Still there is limitation in Gimini

[rigid sun]

If wanting to add a Custom Model (which includes some further orchestration), and choosing the OpenAI Model Provider, does that call assume the OpenAI create "Chat Completions" api request and response body format? (in contrast to the new Responses api). If so, will it make the call with stream: true and accept server-sent events?

[sullen crown]

Hello,

I am new to stax. I am trying to upload a dataset via CSV file. When I do so, the system loads forever without showing a success or error message.

Even when I upload the sample csv provided in the documentation (pasted in next message for reference), the system loads forever in the same manner.

Any help is appreciated. Thank you

Sample csv from documentation:

input,system_instruction,output,expected_output,model_nickname,human_evaluation,tags,metadata
Write a poem about {{metadata.poemType}},Respond using iambic pentameter,"At dawn, the ducks glide smooth upon the pond, Their feathers glisten in the early light. They quack as if to greet the waking day, And splash with joy the calm of mirrored sky. Their life unfolds in graceful arcs and streams, A lively dance upon the water's stage. Each ripple speaks of nature's humble art, Bestowing peace upon the human heart.","Along the quiet pond, the ducks do glide, Their feathered coats as soft as morning dew. They quack a tune beneath the rising sun, While ripples dance in water’s gentle run.",Gemini 2.5 Flash,1,poetry,"{""poemType"":""ducks""}"
Write a poem about {{metadata.poemType}},Respond using iambic pentameter,"At dawn, the iron beast begins its run,
Its steadfast wheels embrace the morning sun.
Along the tracks it whispers ancient tales,
Through mountain mists and over verdant vales.
Each rhythmic clang recalls a bygone art,
A journey forged by both the pulse and heart.
The engines song resounds with strength and grace,
Uniting lands with steady, measured pace.
In heated nights or chill of autumn's rain,
The train endures, a bridge o'er time's domain.","The iron serpent glides along the track,
Its beating heart revives the steel and black.
It weaves through fields and towns at break of day,
A moving dream that never fades away. ",Gemini 2.5 Flash,-1,poetry,"{""poemType"":""trains""}"

[rigid sun]

I've been unable to login to Stax this afternoon. When I try, it reports "Your Stax account is pending deletion." before kicking me out; however, I did not request account deletion. Is there an issue right now?

[mossy dragon]

Hi, I tried to recreate this and it successfully worked for me. Could you please try again by closing all the tabs and confirm whether it is still the same. Thank you..

[mossy dragon]

Hi,Thank you for contacting us and immediately bringing this critical issue to our attention.But Stax working well for login & logouts for us. W'll raise your concern with our eng team and revert you back.Meanwhile try logging the account and update us. Thank you.

[rigid sun]

Thank you for responding. Can you explain what you mean by "revert you back"? I've tried again, same issue. I get through the "You’re signing back in to Stax" on the sign in with google page, end up on Stax briefly with the banner "Your Stax account is pending deletion" before it kicks back out to the sign in with google flow again. Tried clearing cookies/storage/etc.

[mossy dragon]

That's great efforts from your end. We raised your problem to our eng team who will look into the issue and resolve it.Once they have resolved it, things will work well for you and any others who are facing the same. If there is any issue from your end , we will respond it here. Thank you.

I've been unable to login to Stax this

[weary hazel]

Where can I find the introduction to Stax? I wanna know what it's about and try it. Last time I was really confused when I tried it

[mossy dragon]

Hi, You can find Stax here https://stax.withgoogle.com/landing/index.html Stax is a tool to evaluate how a model performs, compare 2 models and choose best model from them...

[weary hazel]

thanks a lot!

I see now. So it's like testing different models to find out which one has the best fit for my use case

and like creating our own benchmarks

the outputs kinda feel a bit unnatural but understand what the product is for

[mossy dragon]

Yeah that's a major part of Stax and also you can create your custom models and can compare them.

[weary hazel]

ohh create my own custom models? Sounds interesting!

[mossy dragon]

Like you need an endpoint of your custom model to upload not exaclty create custom model here.

[weary hazel]

yess ofc it's not creation. I understood that part but I still find it pretty interesting

I'll take a look on how to do it

and provide feedback

[mossy dragon]

That's great, looking to hear from you, Thanks.

[weary hazel]

i'll try it with a fine tuned gemini model

or maybe some open source model and call it from the api within stax

[queen dagger]

is stax is available in India?

[mossy dragon]

Yes , it is available.

[spring cliff]

#stax Can I please get approved to use Stax

[mossy dragon]

Hi, Could you please tell me from which country you are trying to access?

[meager siren]

Hello

[mossy dragon]

Hello

[bronze rover]

I would like to evaluate Vertex AI Search endpoint (discoveryengineapi) from Germany/Austria using Stax. Would this be possible?

[spring cliff]

From Serbia

[mossy dragon]

Yes , You can evaluate custom endpoints using Stax here https://stax.withgoogle.com/settings?tab=modelManager . But access to stax in Germany/Austria is not yet available.

[mossy dragon]

Access to Stax is not yet opened here and we are working for availability in these regions.

[muted bay]

Is there a rough timescale for a UK release? Keen to try it

[mossy dragon]

No, Currently there is no fixed timeline for release.

[bronze rover]

Thanks for the response and hoping to see the service available in EU.
I have a feeling this wont work as I wont be able to specify vertex ai search in place of models. The result will be that Stax won't be able to interpret the output.
Stax should have an option for me to provide a JSON example and learn the schema from it.

Model Provider: Select a provider (e.g., OpenAI, Anthropic) whose output format your model's responses match. This is required for compatibility.

[mossy dragon]

Thank you for your feedback and will raise an Feature Request for this.

[onyx cliff]

Would love to use stax in Bangladesh. 🇧🇩

[cobalt matrix]

Hey team! Love the concept of this product. Can't use it in Australia however. Any chance this could be rectified? Alternatively is there going to be any issues using this with a VPN?

[cerulean ruin]

please request a country to be added here: https://discord.com/channels/1172568727942860810/1428429587104137351

[deep dragon]

In the sample project, when I hit Generate Output,

I get Path not found: inference/projects/chat error. Is this known issue? (I googled, and searched in this Discord server for it but nothing found)

[mossy dragon]

Hi, this is due to some internal error and will be cleared soon. Thank you for reporting the issue.

[deep dragon]

Thx!

[real zephyr]

#stax・🥞 Why we can not use it from Bangladesh?

[cerulean ruin]

please submit a country here:https://discord.com/channels/1172568727942860810/1428429587104137351

[mossy dragon]

Thank you for your interest, But stax is not currently available or supported here.

[dire canyon]

@earnest smeltAs stated in #start-here・📍; currently, our team can support engagement in English; the discussion in this server will largely remain in English for now.

[long canyon]

👋 are there any plans to support more integrated methods of uploading data?

I'd love to be able to programmatically upload CSV data, pull in from BigQuery, etc. Something more along the lines of Opik or DataDog online evals where they pull in all the data from the traces

[mossy dragon]

Hi @long canyon , That is a critical area of focus, and while we cannot confirm specific dates for future roadmaps.We encourage you to monitor the official Stax documentation for more information.Thank you.

[mossy dragon]

Now is it working or issue is still there.. ?

[unkempt void]

when access from kazakhstan?

what is the point of gating access geographically?

especially for an eval service

[jagged wadi]

me and my students fellow really like to have it here in Morocco

[mossy dragon]

Currently we are working for availability and will update in the community once we back with an update.

[mossy dragon]

We are working for availability and will update in this community.

[sleek garden]

isnt stax a bit costly if it is an ai powered app builder? (arent the apps going to be costly to run I mean resource-wise)

[cerulean ruin]

https://discord.com/channels/1172568727942860810/1428429587104137351

[honest cypress]

why isnt stax available in germany/europe, or some other locations? like whats the reason?

[oblique pewter]

hey Elias! each market has a variety of considerations (legal, for example) that we need to address before we're able to launch any Labs tools or experiments - especially experiments, which, are experimental in nature 🙂 but in every case the ambition of the teams working on these projects include shipping to every market we can as quickly as possible. we'll ping the roles here in the server when we have updates in regards to expansion. appreciate the feedback!!

[honest cypress]

okay, i understand now. thanks for the reply! 🙂

[true thunder]

Hi Hi,

Stax in South Africa would be a great plus.

[vague kindle]

Hey team! Just discovering Stax, looks promising while addressing one of our core issues is to evaulate our results.
I run a finance workflow automation company and I am wondering how we could use stax to quicks judge our results

I am based in Sweden, thought this is important since certain solutions are geo locked

[mossy dragon]

Hi @vague kindle , That's great to hear. You can use Stax to compare results for models Side by Side, Create a your own data(often datasets), upload CSV files (large data) and evaluate ,..Also can create a custom evaluators to suit your own evaluator.
Can we know on what areas or use case you were focused on?
Thank you.

[real arch]

Were

[tribal carbon]

use a vpn

[cerulean ruin]

please use https://discord.com/channels/1172568727942860810/1413236517370990794 !!

[tepid barn]

Hi

[rigid frigate]

Hi bro

[stoic kraken]

Hello, Stax in Bulgaria would be great 🍀

[hardy verge]

Doesn't work either

[tribal carbon]

oh

[proven valley]

I wish I had a Stax-like interface to evaluate HuggingFace-downloadable models.

[frozen trellis]

Hi guys, I stumbled upon Stax today, and it would be great to have it working in Italy and CH.

[lucid isle]

That would be really nice

[mossy dragon]

Thank you for your feature request inputs. We will add this on our future roadmap of stax and will work on it.Thank you!.

[lucid isle]

Also, Can we make Stax work directly on Opal

[mossy dragon]

Could you please provide some more information/any use-case your are reffering to?

[lucid isle]

Stax would really nice to work with in Opal (another Googles Labs Experiment) if we can evaluate prompts for various steps within Google Opal using Stax would be amazing

Like, For Example. If i made a Evaluator for checking how does it identify and organize priorities. Since, My work with Opal requires that. Opal is an n8n tool

[mossy dragon]

Sure, will try to check on this .

[hoary vale]

forwarded to #1413236517370990794

[rain knot]

From what I understand, you can already do this manually with the existing tools. Opal gives you the pipeline/orchestration side, and you can design and refine system prompts in AI Studio or Gemini to enforce invariants and priorities step by step. Integrating evaluation just automates that decision layer, but the workflow itself is already possible.

Don't get me wrong, I totally see your point. My take is that while we can automate these steps, the real challenge is keeping it precise. General automation can easily lose its way without very sharp system prompts and output constraints to keep things on track. Doing it manually for now at least gives you that granular control over the logic

[lucid isle]

I am sorry, I want to access Stax's Interface from Opal itself. Not Automate it. So, I can test my prompts easily

[eternal thistle]

When is stax avaliable in aus and can a solid wire guard VPN bypass this

Very unethical to charge the same rate for a subscription for a lesser version of the product and features

Also. Why does Google not make these tools more innately intuitive, even their descriptions are so simple and useless.

Google labs has tools thag can make tutorial type videos. Why not make them for each of the google lab tolls with apeici use case video variants.

My team pages ultra because many tools are simply not well explained and have outdated infom

Is this by design? So people use the tools more frequently and for longer amounts if time to benefit training data?

[cold swift]

Get started with Stax at https://developers.google.com/stax/overview.

https://developers.google.com/stax/overview

[drowsy dune]

Sweden please 🙂 as rest of the tools

[surreal badge]

hello guys

[ivory carbon]

Hi Google Stacks team,

I’m currently conducting my Master’s thesis in Germany, focused on evaluating LLM responses and classifications on PKM datasets. I’d love to use Google Stacks, but it appears the Labs platform access is currently restricted for German users.

Would it be possible to unlock/enable Stacks for users in Germany? It would let me avoid a lot of manual preparation, coding, and evaluation setup by using capabilities Stacks already provides—making a major difference for my thesis work.

Thanks a lot for considering this, and I’m happy to share more details about my research needs if helpful.

Best regards,

[next fossil]

Have u tried vpn

[cedar zenith]

Hi bro

[ivory carbon]

yes... seems to relate to the google accounts origin or billing address :/

[sturdy quartz]

VPN works

Will work in most cases ig

[hushed python]

Hey Men.

[paper bluff]

Can we integrate Stax with any of the monitoring tool like Dynatrace, which is configured to capture Prompt input, Tool call response and final response [agentic AI], and use that as input to Stax and run evaluation (in Production), and generate metrics/score such as as Faithfulness or correctness and we gauge Hallucination?

[paper bluff]

anyone?

[faint nymph]

Any chance to have this product in more countries(Lithuania)?

[lapis willow]

Any chance to have stax in Hungary?

[cold hull]

what is stax

[polar hill]

Hey 👋 new updates on Stax Labs here?

[waxen onyx]

Stax in South Africa please.🇿🇦

[vernal cove]

Will STAX be available in APAC soon?

[celest fox]

Hey, can we have Stax in Ukraine? 🇺🇦

[outer turtle]

Hey, bring stax to Kenya please