Mixing OpenAI & local models | Dagger | Page 1

simple cosmos Feb 19, 2025, 10:14 PM

#

not sure how to fix

eager mesa Feb 19, 2025, 10:14 PM

#

yeah, we'll have to have multiple clients at the same time. The routing idea is a good start

simple cosmos Feb 19, 2025, 10:16 PM

#

The routing as we discussed it is implemented now

#

But, we need a more expressive way to configure the router now

#

Adding OLLAMA_... env variables will help.. but only if you use ollama. Don't want to lock ourselves too tightly to infra tooling

eager mesa Feb 19, 2025, 10:17 PM

#

right it should be more generic as an "other" probably

#

are there ways to run gpt-* models outside of openai? We could be smart about the base_url for gpt-* models and send the rest to the base url. But then it still needs another name

simple cosmos Feb 19, 2025, 10:19 PM

#

Maybe this is where LLM_BASE_URL and LLM_KEY come in, as a generic fallback?

eager mesa Feb 19, 2025, 10:20 PM

#

yeah seems fine

#

we're basically assuming the fallback is an openai compatible api. Otherwise it'll need it's own client and router

#

safe bet

#Mixing OpenAI & local models