Mistral Large 2 (Mistral-Large-2407-Instruct)

[fluid igloo]

New 123B param, 128K token context model

License: Mistral Research License MRL-0.1 (noncommercial use only)

HuggingFace weights (instruct, bf16) | Blog post + benchmarks

Hosted on Le Plateforme

• HumanEval+ and MBPP+ performance slightly below GPT-4o

Looks to be GPT-4o level by benchmarks

[valid fjord]

Does noncommercial means it won't appear on OpenRouter?

[fringe cloud]

I think it should be fine: #general message

nvm, I guess not

[glossy harness]

Mistral hosts this as mistral-large-2407, OpenRouter could route to that

[jagged reef]

It's hosted on the big cloud providers as well. On Vertex AI - https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/mistral - $3/$9 per 1M input/output tokens, and you agree to https://mistral.ai/terms/#terms-of-use for terms

Still waiting for AWS Bedrock/Azure AI to update their models

[vague sedge]

@mental moat the pricing didn't get updated?

[mental moat]

cc @eternal oyster

[mental moat]

Will be updated soon

https://mistral.ai/technology/#pricing

[eternal oyster]

its already fixed, someone pointed it out in #arc-requests

[mental moat]

Ah mb didn't check

[dusty edge]

Its almost same on price like gpt4o. Don't see point in this

[tall parrot]

Small correction for Large's description on site - it's not closed-source anymore, it's proprietary weights-available now

(I can't bring myself to call model licensed like that open)

[hollow sand]

FYI: The slug for mistral-large still says it's 32k context while the API reports 128k -> https://openrouter.ai/models/mistralai/mistral-large

[eternal oyster]

fixing both the above now, thx yall

[tall parrot]

nice!
also it would be nice to have to the link to it's weights
https://huggingface.co/mistralai/Mistral-Large-Instruct-2407
It's a weights-available model afterall

[eternal oyster]

yah that's getting added in a sec, when I add lepton as a second provider for half price

[tall parrot]

oh? Seems like it's not impossible to get a commercial license then
That's news to me

[eternal oyster]

thought they had one but they actually just didn't look at the license yet, so nah they won't be hosting

[pine hill]

is this any good? : O

seems very pricy ;_;

[fringe cloud]

cheaper than gpt-4o and claude 3.5 sonnet

[pine hill]

ya slightly cheaper

how's performance tho, if it's slightly worse, then it's kinda just priced as a stopgap model ig : D

[hollow sand]

Small observation: The 'weights available' Mistral Large v2 on Huggingface has -> "max_position_embeddings": 32768,, which seems to limit the context size to 32k (without further RoPE scaling configuration) if I understand this correctly? -> https://huggingface.co/mistralai/Mistral-Large-Instruct-2407/blob/main/config.json

[hollow sand]

Fixed -> https://huggingface.co/mistralai/Mistral-Large-Instruct-2407/commit/5c9ce5b5f7a7ad62d03e8c66c719b66d586de26b

[viscid fern]

yup it would be awesome

[exotic copper]

worse that gtp4o in my testing

but i only tested logic questions so far

[exotic copper]

but still better than llama3

[exotic copper]

an so much less censored

[hollow sand]

I do not think Mistral models get censored in any way, at least I have never seen them refuse any request like other popular LLMs.

[exotic copper]

haven't used base Mistral that much,
but I find Wizard22b censored in an insidious way, which is really annoying, would prefect if they just straight up refused.

I think it because other providers been offereing m22bx8 at the lower cost is why they released it non-commercial.
though it pretty terriable timing since l3-405b just got released.

[vague sedge]

I think 405b is much better at making responses good

[exotic copper]

ya, but censored and not in a good way.

[vague sedge]

I don't really have any use cases related to censoring so I can't really feel anything, I feel like multilang is still not the best but it's a significant step up

[primal steeple]

Anyone else having issues with regenerating message not working for this model at the moment?

[exotic copper]

no, but im currently using the direct api with $5 trial.

[molten field]

So is this already available at openrouter? mistralai/mistral-large <- is this it?

[hollow sand]

Yes, the old mistral-large got replaced with "v2" on the same endpoint (unless you have very specific needs you won't miss the old version).

See also the model card -> https://openrouter.ai/models/mistralai/mistral-large

[upper lava]

After messing with this model. I would say, it's quite creative. Almost better than wizard 8x22b.

[viscid fern]

love this model, will add it to my app within 2 days

very good one

[spring marten]

That explains why it feels diffrent from when I played with it months ago.

Personally I like it better than Sonnet

[exotic copper]

Been switching between this a sonnet 3.5. It's pretty good for the size but sonnet is still a lot better at creative writing. Though this is better than 405b.

[coral turret]

@mental moat Mistral's own API is horrendously slow for this. Maybe you guys could add Azure? We compared them and Azure was ~2.5x faster on average.

[mental moat]

Ohh, is it the exact same Mistral large model???

I thought the weight is private for that obe

[coral turret]

afaict it's the same? It was announced in Mistral's official blog for Mistral-Large-2

https://mistral.ai/news/mistral-large-2407/

AWS/Azure/GCP host Mistral non-OSS models due to their partnership

The pricing is the same as Mistral API, but wayyy faster

[bold pier]

I felt like it follow my instructions much better than WizardLM-2-8x22b which often miss simple instructions. Tho I wish it had a cheaper price.

[upper lava]

Yeah, i loving it this right now more than Wizard. I hope they can upgrade with mistral small and medium for cheaper price.

[mental moat]

Azure host has been added! Thanks for the flag 🙏

[iron steeple]

this model is suddenly refusing all "unsafe" requests, it wasnt doing that up until a few days ago :/
anyone else notice this?

[tall parrot]

well, only that changed is that Azure was added as a provider
try to exclude Azure

[iron steeple]

hmm looks like with different prompting it doesnt refuse anymore. (but with weird changes, for example appending "use a json string" made it refuse)
and changing provider didnt affect refusals, but thx for the idea.

[midnight rock]

It’s an Azure issue indeed.

Frustrating as Mistral’s own endpoint is really slow.

If anyone knows whether any of the other possible Mistral Large 2 endpoints are reasonably fast and uncensored, please let me know!

[tall parrot]

It's on AWS
Haven't tested it personally, but afaik AWS doesn't add any additional moderation by default

It's also on Vertex

[mental moat]

if changing provider didn't affect refusals, I think it'd be the same thing on other cloud. It seems they are just taking the engine and run it as-is on the cloud infra

[iron steeple]

one big thing, i think, is ambiguity of the request makes it more likely to refuse.
so like, ask write about <this crazy thing with X Y and Z>, isntead of write about <this crazy thing>
(i think that was the problem in my case)

[midnight rock]

Yeah, it definitely depends on the provider, as in Azure is definitely censoring it.

Do you have AWS on OpenRouter? Would be great to be able to add that to the list of providers.

[spring marten]

ML2 suddenly gotten really strict on me. Ignoring Azure is not helping.

[buoyant atlas]

ML2 (and small/medium) all don't seem to work in the chat playground right now with the Mistral provider and instead returns this error. Only requests through Azure seem to work.

Looks like all the Mistral models are throwing the same error only from the Mistral provider. I'm guessing any requests with unsupported samplers aren't ignored on their end and instead give an error now?

[mental moat]

cc @placid sonnet

Fix coming soon, thanks for the flag!

[mental moat]

It should be up now fyi

[spring marten]

It is working fine now, thanks!

What was going on?

[mental moat]

we were forwarding an anonymous user id to providers (this is useful to detect abuse so we're not de-platformed by the provider themselves). It appears Mistral itself does not accept the parameter

[placid sonnet]

Thanks for flagging @buoyant atlas and @spring marten - happy to give you guys some free credits for this issue if you dm me your openrouter emails. we started submitting anon user ids to a few openai-style providers to help them smooth out traffic from some very aggressive users, and our e2e test suite for Mistral wasn't included.

We're going to improve our e2e tests to prevent this issue going forward.

[hollow sand]

If an ID is anonymous, it can never be correlated to an actual user, but then abuse prevention would not make sense. So I think you are using pseudonymous IDs, which can be traced to a certain user, if necessary, correct? "anon IDs" is IMHO misleading in that case.

[spring marten]

I think its less about targeting individuals, and more about finding patterns between abusive users so they can better train the AI to ignore those requests.

[spring marten]

Nah, its ok. I'm not going to sweat over a few cents. Thank you for the offer though.

[buoyant atlas]

same, didn't lose anything really, just noticed it happening and switched the provider while it wasn't working

[placid sonnet]

for providers without moderation requirements in their terms, like Mistral, we hash the user ids before submitting to the provider

(though, for Mistral today, we aren't sending anything at all. we were just submitting user: null and they didn't like that)

[hollow sand]

Hashing the user id is still pseudonymous IMHO as this consistently produces the same ID/hash and can be traced back the second this hashed ID can be correlated to the real ID via a request log on both sides. The necessary metadata for this trace back gets logged officially, all that needs to be done is hashing all real user ids and see which pseudonymous ID from the other log fits to the log entry with the real ID, correlated by timestamp or other very unique request identifiers.
Anonymous would be if this was not possible.

(I'm an annoying nitpicker, I know)

[raven sorrel]

very good model but it is too slow...

[midnight rock]

@mental moat @placid sonnet would be great to add a faster & uncensored provider. Amazon Bedrock and IBM watsonx seem to be the available options.

[placid sonnet]

fair point!

[harsh portal]

I used it via OpenRouter chat and it followed the instructions fantastically,
I simply needed to extract information from several texts and format them in several different lines, I created special instructions to enter in the system prompt of the interface provided by OpenRouter.
I compared it to the 405B Lllama Sonar and it wins Mistral Large, at least in my tests.
Please note: I am only talking about strict adherence to the instructions!

[timber mantle]

Sorry to revive the thread, but does anyone know why Microsoft charges more for mistral-large than mistral-large-2407?
($4/M in, $12/M out VS $3/M in, $9/M out)

[vague sedge]

the pricing is determined by Mistral lol and after the release of the new large they didn't bother to change the pricing of the old large

[timber mantle]

https://mistral.ai/news/september-24-release/

Minus thirty three percent

(on the application programming interface 👀)

[jagged reef]

mistral taking the google route of harvesting free tiers for data