I'm trying to access my multiple local models running via litellm and ollama but I'm always getting "Incorrect API key provided: NULL" , here is example of the config:
{
"model": "phi",
"api_key": "NULL",
"base_url": "http://192.168.3.11:5001"
}
I tried to add api_type, to use it without api_key, to use sk api key etc. none of these was working. I can successfully open the LiteLLM api via the browser. Any suggestions would be greatly appreciated.
#Ollama with litellm incorrect api key error
minor pike
tawny sandal
set a fake api key (OLLAMA_API_KEY = xx-yyzz) . It might just be that the value needs to be a string, and "NULL" might be treated as NULL
hazy moth
set a fake api key (`OLLAMA_API_KEY = xx-yyzz`) . It might just be that the val...
I couldn't solve this problem? 🥲
even though I'm setting up the mistral ollama config properly, it gives me openai error.
I tried with and without model name "mistral"
hazy moth
I'm trying to access my multiple local models running via litellm and ollama but...
have you solved this issue?
I'm just trying to communicate Ollama and autogen, my project is not involved with litellm
azure solar
Should /api/generate be included in base_url?
hazy moth
Should /api/generate be included in base_url?
either way same error
I'm checking config_list examples right now
tawny sandal
404 means the URL is incorrect, and i see you're using a custom url..
hazy moth
I got it!
the reason it gives me 404 is because the ollama mistral api is not compatible with openai api
tomorrow I'm just gonna look out some way to utilize gguf files of models, and try with openai compatibility mode
tawny sandal
have you tried litellm proxy? it provides openai-compatiblemisral support
tawny sandal
okay, well i havent tried ollama directly, but i have used mistral, ollama, and huggingface behind litellm, and it works fine.
im sure there are other solutions, but this one is fairly straightforward and should get you started.
lament bloom
Had same problem, fixed it with litellm
hazy moth
my end goal is orchestrating open source llms on top of the kubernetes, does it possible with litellm?
im gonna research later today about it
minor pike
Finally found some time after the holidays. The issue was messy virtual/ python environment. Now it is working fine with litellm, but it is not very stable. Also tried it with lmstudio. All of these tests are with remote machines, locally everything works fine. These days will have more time to test it.
calm quartz
By me local LLM works with AutogenStudio as api key “no-needed”
hazy moth
By me local LLM works with AutogenStudio as api key “no-needed”
how you're running local llms? with lm-studio, ollama, or others like vllm?
hazy moth
I mean which one? 🤔
calm quartz
I mean which one? 🤔
Mistral. But I was do update and stopped to work NULL and “no-needed” both don’t work now.
Update ! Works agin ( api key “no-needed” ) after downgrade to autogenstudio==0.0.28a0