I use WizardLM-2 8x22B through Venus.ai and I'd like to set it to use the provider DeepInfra only. But I can't find out where to put select this info on openrouter. Am I not able to do it on OpenRouter's end?
Thanks
#Help selecting provider
polar horizon
flat jay
you can do it via our API, but it's up to Venus to implement support for provider selection
polar horizon
Yeah, I was playing around with that code from the page. I just didn't know where to put it.. Venus.ai doesn't have that feature that I can find. And google isn't really helping when I try to find out if any other character front ends do or don't.
I tried putting it in the system prompt when using chat on OpenRouter.ai... but that didn't seem to work even temporarily.
azure brook
Yeah, I was playing around with that code from the page. I just didn't know wher...
You can kinda do it with SillyTavern, but you'll have to modify it's code
example of provider injection in ST
polar horizon
example of provider injection in ST
That's good to know, thanks. I was thinking of trying Tavern, Risu, and Telegai in the future.
polar horizon
If anybody comes back here in the future, I am now using Risu. It's like SillyTavern but looks much nicer, and super easy to select your provider.
azure brook
Upd: SillyTavern will be adding provider selection in update 1.12.0
This functionality is already available in staging branch
frosty spoke
Upd: SillyTavern will be adding provider selection in update 1.12.0
This functio...
nice
ember crypt
Help! I'm using langchain's ChatOpenAI and don't know how to switch providers
flat jay
can you pass in custom JSON for the request body?
ember crypt
can you pass in custom JSON for the request body?
I have found a solution by modifying the source code of langchain and openai
In fact, the openai method adds a parameter, and langchain can be passed in.
@flat jay I want to know which provider is the most reliable and is there any way to measure it?
flat jay
We're planning a public dashboard soon to help people figure this out
@dapper pewter anything we could suggest here in the meantime?
dapper pewter
@ember crypt for one model in particular or just generally?
and mind expanding on what you mean by "reliable"? like thruput & uptime or "consistent quality responses with my chosen parameters"?
flat jay
I think the priority is probably uptime, followed by 0-token responses (eliminating those)
but curious about what @ember crypt says as well
ember crypt
<@760779461620400128> for one model in particular or just generally?
and mind ...
What I mean is that the provider used is always changing dynamically when calling the api. According to our user feedback, sometimes the chat works well and sometimes the chat messages are wrong. Therefore, I want to use only a certain provider to ensure that the chat works well
By customizing the order of setting the provider myself, I found that the provider is still dynamically changing when actually used.😂
I'm guessing that your current call strategy is to sort the providers first, and then prioritize which ones are in low use at the same time(Throughput and latency trade-offs). Right?