#Internal Server Error 500 - Issue when using Autogen Studio + Litellm + Ollama/mistral

5 messages · Page 1 of 1 (latest)

chrome zinc
#

I've been trying to get a 100% local LLM arrangement to work, but keep running into this "500 Internal Server Error" no matter how I configure the model in AG Studio.

  1. AG Studio
    A) I create a new model and populate only:
    i) "model name" field = 'mistralai'
    ii) "base URL" = "http://localhost:8000/v1" (confirmed same port as litellm reports)
    B) Try the "test model" button and attempt model+agent+playground sanity test, but receive HTTP 500 - Internal Server Error 100% of the time. - See text snippet
    -This seems like an AG Studio issue because:
    >litellm acks the request --> INFO: 127.0.0.1:59266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
    >Passes to ollama, ollama loads and serves the LLM request (see 2nd attached log snippet)
    >Ollama passes it back to litellm with HTTP 200 success message. --> [GIN] 2024/02/03 - 01:30:52 | 200 | 607.967781ms | 127.0.0.1 | POST "/api/generate"

I've tried a handful of LLM model config field permutations already (specifying "openai" as type, trying "ollama/mistral" as model name, etc.) with no change in results. Also tried using LM Studio server to try and debug/workaround but got the same result to litellm.

📎 Autogen_Studio_Error_Output 📎 Ollama_LLM_Request_Logs
raven lotus
#

@chrome zinc, here's my setup that works.

LiteLLM command:
litellm --model ollama/mistral:7b-instruct-q6_K --port 8801

Model setup in Autogen Studio (note I put text in the API key field but I'm not sure if it's needed any more):

Name: Local8801
API Key: NotRequired
URL: http://0.0.0.0:8801
Description: Local LLM through LiteLLM on port 8801

Test Model button:
"Model tested successfully"

Once the model is working please ensure that if it's attached to any Agent or Workflow that you go into them and remove the LLM selection and add it again so it has the latest version.

chrome zinc
#

I tried the following and no improvement:

  • @raven lotus 's configuration on port 8801, same HTTP 500 error
  • http://localhost:8000/v1, http://127.0.0.1:8000/v1, and http://0.0.0.0:8000/v1 with no change
  • http://localhost:8000, http://127.0.0.1:8000, and http://0.0.0.0:8000 with no change

I'm using windows subsystem for Linux (WSL) on Ubuntu with Python 3.10 so perhaps something odd is happening with that?

chrome zinc
#

✅✅✅ SOLVED ✅✅✅

Looks like AutoGen Studio can't route properly to litellm when all components are hosted on WSL. The fix for me was:

  1. Install Autogen Studio on my native windows environment through powershell/pip
  2. Keep Ollama hosted on WSL/Ubuntu (there's no windows client as of 2/3/24)
  3. Keep Litellm hosted on WSL/Ubuntu (it can't access the ollama model without being on the same virtual environment)

Not sure if a <@&1153754985344016546> can file (or help me file) either A) bug report and/or B) release note/FAQ entry.

queen echo