#Trouble connecting Ollama

1 messages · Page 1 of 1 (latest)

buoyant pelican
#

I've been trying to connect LibreChat and Ollama. I'm on Windows 11 using the Docker version of LibreChat and I'm getting this error message when trying to get Ollama to work:

Unexpected Application Error!
s.filter is not a function
TypeError: s.filter is not a function
    at oj (http://localhost:3080/assets/index-d-wIOvje.js:71:129123)
    at s6 (http://localhost:3080/assets/vendor-CssRXSeb.js:38:19539)
    at H_ (http://localhost:3080/assets/vendor-CssRXSeb.js:40:3139)
    at FO (http://localhost:3080/assets/vendor-CssRXSeb.js:40:44833)
    at DO (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39790)
    at yX (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39718)
    at Kg (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39570)
    at t3 (http://localhost:3080/assets/vendor-CssRXSeb.js:40:35934)
    at d8 (http://localhost:3080/assets/vendor-CssRXSeb.js:40:36738)
    at Ll (http://localhost:3080/assets/vendor-CssRXSeb.js:38:3279)

Any help is appreciated, please feel free to ask for more information.

warm mountain
buoyant pelican
#

And yeah I read it and tried my best but I might have broken something

#

Also, does LibreChat support Llama3.1 or just up to 3? Does it make a difference?

warm mountain
warm mountain
buoyant pelican
warm mountain
buoyant pelican
#

I just tried to follow the instructions as best I could

warm mountain
#

wsl2?

#

i have a branch where we include ollama as part of the compose file, which simplifies that part of it

#

but we can also try to make it work with how it's already installed

buoyant pelican
warm mountain
#

ok let me see

#

i have it installed on windows, too, so maybe i can just document this, too

buoyant pelican
# warm mountain ok let me see

If it makes a difference, here's the config in librechat.yaml:

custom:
    - name: "Ollama"
      apiKey: "ollama"
      # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
      baseURL: "http://host.docker.internal:11434/v1/" 
      models:
        default: [
          "llama3.1:latest"
        ]
        fetch: false
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"
      addParams:
        "stop": [
          "<|start_header_id|>",
          "<|end_header_id|>",
          "<|eot_id|>",
          "<|reserved_special_token|>"
        ]
warm mountain
#

the windows section, you need an env var to bind ollama to 0.0.0.0

#

you need to quit ollama, and then you can run this in a powershell terminal:

$env:OLLAMA_HOST = "0.0.0.0:11434"

#

then you can run ollama serve from the same terminal

#

im testing the next step

buoyant pelican
warm mountain
#

@buoyant pelican and then the exact config you shared should work (on docker)

image.png
warm mountain
#

on windows, you can see it in the taskbar, I quit it before running ollama serve

buoyant pelican
#

Oops, had ollama running in a terminal so I couldn't see it in the taskbar

warm mountain
#

you can check like this

                                                                                                                        PS C:\Users\danny> curl http://localhost:11434/

StatusCode        : 200
StatusDescription : OK
Content           : Ollama is running
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 17
                    Content-Type: text/plain; charset=utf-8
                    Date: Sun, 11 Aug 2024 22:00:48 GMT

                    Ollama is running
Forms             : {}
Headers           : {[Content-Length, 17], [Content-Type, text/plain; charset=utf-8], [Date, Sun, 11 Aug 2024 22:00:48
                    GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 17




PS C:\Users\danny>
#

curl http://localhost:11434/

buoyant pelican
#

It says Ollama is running, but I'm still getting the same error as in the beginning for LibreChat.

warm mountain
#

(for librechat) docker compose down

#

docker compose up -d

#

if you have docker desktop, you can try a simple test from the container terminal

image.png
buoyant pelican
#

Did that, I'm trying something else real quick to see if it fixes it

#

So Ollama is responding, but the application keeps giving an error

warm mountain
#

which is?

buoyant pelican
# 
Unexpected Application Error!
s.filter is not a function
TypeError: s.filter is not a function
    at oj (http://localhost:3080/assets/index-d-wIOvje.js:71:129123)
    at s6 (http://localhost:3080/assets/vendor-CssRXSeb.js:38:19539)
    at H_ (http://localhost:3080/assets/vendor-CssRXSeb.js:40:3139)
    at FO (http://localhost:3080/assets/vendor-CssRXSeb.js:40:44833)
    at DO (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39790)
    at yX (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39718)
    at Kg (http://localhost:3080/assets/vendor-CssRXSeb.js:40:39570)
    at t3 (http://localhost:3080/assets/vendor-CssRXSeb.js:40:35934)
    at d8 (http://localhost:3080/assets/vendor-CssRXSeb.js:40:36738)
    at Ll (http://localhost:3080/assets/vendor-CssRXSeb.js:38:3279)
warm mountain
buoyant pelican
#

Okay, I'm no longer getting an error, but I don't see Ollama as an option

warm mountain
#

really you just need to step 1 from that guide

#

since you already have ollama configured

#

in your librechat.yaml

#

and if you still have issues, it could be a misconfiguration/indentation error of the librechat.yaml

2024-08-11 18:14:40 
2024-08-11 18:14:40 > [email protected] backend
2024-08-11 18:14:40 > cross-env NODE_ENV=production node api/server/index.js
2024-08-11 18:14:40 
2024-08-11 18:14:40 2024-08-11T22:14:40.544Z info: [Optional] Redis not initialized. Note: Redis support is experimental.
2024-08-11 18:14:41 2024-08-11T22:14:41.345Z info: Connected to MongoDB
2024-08-11 18:14:41 2024-08-11T22:14:41.413Z debug: [indexSync] There are 374 messages and 374 indexed
2024-08-11 18:14:41 2024-08-11T22:14:41.413Z debug: [indexSync] There are 90 convos and 90 indexed
2024-08-11 18:14:41 2024-08-11T22:14:41.425Z info: Custom config file loaded:
2024-08-11 18:14:41 2024-08-11T22:14:41.425Z info: {
2024-08-11 18:14:41   "version": "1.0.5",
2024-08-11 18:14:41   "cache": true,
2024-08-11 18:14:41   "fileConfig": {
2024-08-11 18:14:41     "avatarSizeLimit": 3.11
2024-08-11 18:14:41   },
2024-08-11 18:14:41   "registration": {
2024-08-11 18:14:41     "socialLogins": [
2024-08-11 18:14:41       "github",
2024-08-11 18:14:41 ... [truncated]
2024-08-11 18:14:41 2024-08-11T22:14:41.427Z debug: Custom config:
2024-08-11 18:14:41 {
2024-08-11 18:14:41   version: "1.0.5",
2024-08-11 18:14:41   cache: true,
2024-08-11 18:14:41     fileConfig.avatarSizeLimit: 3.11,

you would see it in your container logs

#

if there's an issue with the config file

#

some other apps package their app with ollama so you dont have to jump through all these hoops

#

and i think we can have some examples on how to do that, it's mainly beneficial for users who have not setup ollama yet

buoyant pelican
#

[+] Running 7/7
✔ Network librechat_default Created 0.1s
✔ Container librechat-ollama-1 Started 0.9s
✔ Container chat-meilisearch Started 0.6s
✔ Container vectordb Started 0.6s
✔ Container chat-mongodb Started 0.8s
✔ Container rag_api Started 1.0s
✔ Container LibreChat Started

It appears to be running but I don't see the option, I checked the blob folder under models in the LibreChat source folder and it's empty, would that be a possible cause?

warm mountain
#

librechat-ollama-1... you're running ollama off the container?

buoyant pelican
#

I honestly have no idea, I have Ollama locally installed. Should I just post the uncommented parts of my yaml files?

warm mountain
#

if ollama is running on windows, and you prefer to use it from windows, you should not have ollama configured as a container

#

and your docker-compose.override.yml should only have this:

services:
  api:
    volumes:
      - type: bind
        source: ./librechat.yaml
        target: /app/librechat.yaml

nothing more nothing less, unless you got other custom stuff

buoyant pelican
# 
services:
  api:
    volumes:
    - type: bind
      source: ./librechat.yaml
      target: /app/librechat.yaml
    image: ghcr.io/danny-avila/librechat-dev:latest
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [compute, utility]
    ports:
      - "11434:11434"
    volumes:
      - ./ollama:/root/.ollama
warm mountain
#

your docker-compose.yml should be untouched, exactly what it is from the github repo

warm mountain
#

that is telling docker to download and run ollama in a linux subsystem

#

so you're gonna have errors with 2 instances of ollama running

#

also i recommend adding this to your ollama config (in librechat.yaml):
dropParams: ["user", "frequency_penalty", "presence_penalty", "temperature", "top_p"]

so

  custom:
    - name: "Ollama"
      apiKey: "ollama"
      # use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
      baseURL: "http://host.docker.internal:11434/v1/" 
      models:
        default: [
          "llama3.1:latest"
        ]
        fetch: true
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Ollama"
      dropParams: ["user", "frequency_penalty", "presence_penalty", "temperature", "top_p"]
      addParams:
        "stop": [
          "<|start_header_id|>",
          "<|end_header_id|>",
          "<|eot_id|>",
          "<|reserved_special_token|>"
        ]
#

unless you want to experiment with temprature settings

#

and the other dropped params

#

it gets me better results with llama3.1

image.png
#

thanks for experimenting here @buoyant pelican we need another blog post specifically for using Ollama from an already running instance on mac, windows, etc.

buoyant pelican
#

I'm really sorry, but it just doesn't appear to want to work, I tried stopping the ollama serve and restarting it but it just doesn't seem to work. I tried clicking on the "http://host.docker.internal:11434/v1/" and it says it can't be reached now for some reason.

warm mountain
buoyant pelican
warm mountain
#

do you have docker desktop, what does this look like?

#

you should not see ollama there

#

then on windows, you can repeat the steps from before

#

setting the env variable, running ollama serve (making sure its not running before you do via windows task manager)

buoyant pelican
warm mountain
#

then repeat this:
#1272302507623977011 message

#

quit ollama from here if you need to

image.png
#

after which you can try this again #1272302507623977011 message

#

I've gtg for now but hopefully you can get this working

buoyant pelican
#

I'll keep messing with it, if I get something to work, I'll post it here

warm mountain
buoyant pelican