Ollama on a Jetson Orin Nano / Docker | Home Assistant | Page 1

umbral trout May 24, 2025, 2:21 PM

#

I have been struggling to get Ollama up and running on my new Jetson Orin Nano. I can get it running using jetson-containers but really want to go the docker-compose route. I can't keep the Ollama container running. Any assistance would be greatly appreciated

`services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
network_mode: "host"
environment:
- OLLAMA_BASE_URL=http://127.0.0.1:11434
volumes:
- "/home/pi/docker/open-webui:/app/backend/data"
restart: unless-stopped

ollama:
image: dustynv/ollama:r36.4.0
container_name: ollama
runtime: nvidia
environment:
- OLLAMA_MODELS=/ollama dustynv/ollama:r36.2.0
network_mode: "host"
shm_size: "8g"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/home/pi/docker/ollama/data:/data"
- "/etc/localtime:/etc/localtime:ro"
- "/etc/timezone:/etc/timezone:ro"
- "/run/jtop.sock:/run/jtop.sock"
restart: unless-stopped`

#

open-webui | 2025-05-24 14:22:16.779 | INFO | uvicorn.protocols.http.httptools_impl:send:476 - 192.168.51.100:52429 - "GET /_app/version.json HTTP/1.1" 200 - {} ollama | ollama | Starting ollama server ollama | ollama | ollama | OLLAMA_HOST 0.0.0.0 ollama | OLLAMA_LOGS /data/logs/ollama.log ollama | OLLAMA_MODELS /ollama dustynv/ollama:r36.2.0 ollama | ollama | ollama | ollama server is now started, and you can run commands here like 'ollama run gemma3' ollama | ollama exited with code 0

sour apex May 24, 2025, 2:28 PM

#

umbral trout I have been struggling to get Ollama up and running on my new Jetson Orin Nano. ...

are you actually using the username pi on a jetson?

umbral trout May 24, 2025, 2:28 PM

#

yes

sour apex May 24, 2025, 2:30 PM

#

I can get it running using jetson-containers
what did you run to use this?

umbral trout May 24, 2025, 2:31 PM

#

jetson-containers run --name ollama $(autotag ollama)

#

but I prefer to use docker-compose because I don't understand what is going on under the hood with jetson-containers

sour apex May 24, 2025, 2:35 PM

#

this line
- OLLAMA_MODELS=/ollama dustynv/ollama:r36.2.0

#

should just be:
- OLLAMA_MODELS=/ollama
i think

umbral trout May 24, 2025, 2:37 PM

#

still the same

sour apex May 24, 2025, 2:40 PM

#

so on the documentation it suggests
docker run --runtime nvidia --rm --network=host -v ~/ollama:/ollama -e OLLAMA_MODELS=/ollama dustynv/ollama:r36.2.0
which in compose would look something like

services:
    ollama:
        runtime: nvidia
        network_mode: host
        volumes:
            - ~/ollama:/ollama
        environment:
            - OLLAMA_MODELS=/ollama
        image: dustynv/ollama:r36.2.0

this is 36.2.0 instead of 36.4.0 but perhaps worth stripping back some of your settings?

umbral trout May 24, 2025, 2:56 PM

#

same thing

#

keep crashing ollama

#

following this:

#

https://www.jetson-ai-lab.com/tutorial_openwebui.html

Open WebUI - NVIDIA Jetson AI Lab

Showcasing generative AI projects that run on Jetson

sour apex May 24, 2025, 3:13 PM

#

jetson-containers must know something we dont...

#

so if i look at https://www.jetson-ai-lab.com/tutorial_ollama.html

Ollama - NVIDIA Jetson AI Lab

Showcasing generative AI projects that run on Jetson

#

there is a gif that shows "what you should see"

#

#

looks like when you run jetson-containers it shows the docker command it ends up running

#

so you could run, then cancel it. copy and convert the docker run to compose

#

you could also just native install if your not that bothered about it running a container?

umbral trout May 24, 2025, 3:25 PM

#

not 100% sure what that would look lik

#

sudo docker run --runtime nvidia --env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/pi/Downloads/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-7 --device /dev/i2c-9 --name ollama dustynv/ollama:0.6.8-r36.4-cu126-22.04

sour apex May 24, 2025, 3:27 PM

#

services:
    ollama:
        runtime: nvidia
        environment:
            - NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics
            - PULSE_SERVER=unix:/run/user/1000/pulse/native
        stdin_open: true
        tty: true
        network_mode: host
        shm_size: 8g
        volumes:
            - /tmp/argus_socket:/tmp/argus_socket
            - /etc/enctune.conf:/etc/enctune.conf
            - /etc/nv_tegra_release:/etc/nv_tegra_release
            - /tmp/nv_jetson_model:/tmp/nv_jetson_model
            - /var/run/dbus:/var/run/dbus
            - /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket
            - /var/run/docker.sock:/var/run/docker.sock
            - /home/pi/Downloads/jetson-containers/data:/data
            - /etc/localtime:/etc/localtime:ro
            - /etc/timezone:/etc/timezone:ro
            - /run/user/1000/pulse:/run/user/1000/pulse
        devices:
            - /dev/snd
            - /dev/bus/usb
            - /dev/i2c-0
            - /dev/i2c-1
            - /dev/i2c-2
            - /dev/i2c-4
            - /dev/i2c-5
            - /dev/i2c-7
            - /dev/i2c-9
        container_name: ollama
        image: dustynv/ollama:0.6.8-r36.4-cu126-22.04

umbral trout May 24, 2025, 3:35 PM

#

hmmmm...I think it is up

sour apex May 24, 2025, 3:37 PM

#

I was thinking about getting a Orin Nano to play with but being able to buy it seems to be limiting facctor

umbral trout May 24, 2025, 3:40 PM

#

what do you mean?

#

it seems to be working

#

thank you

#

I even downloading models from webui

sour apex May 24, 2025, 3:41 PM

#

umbral trout what do you mean?

no stock currently unless you want to pay an insanely inflated scalping prices

umbral trout May 24, 2025, 3:42 PM

#

I got it for like 250 I think

#

trying to run something and i dont think it is workin

#

I had to wait for back order

sour apex May 24, 2025, 3:43 PM

#

😦

umbral trout May 24, 2025, 3:44 PM

#

bammm

#

now it is working

sour apex May 24, 2025, 3:44 PM

#

woo

#

maybe if they are available to buy and ship in a few months and i still have some "play around" ideas ill give it a go but dont really want to backorder it

umbral trout May 24, 2025, 3:47 PM

#

yea it took almost 4 months

sour apex May 24, 2025, 3:48 PM

#

my server running ollama has a 5060ti anyway. so its not like i "need it"

umbral trout May 24, 2025, 3:49 PM

#

Auh yea, I couldn't get ollama working on anything

#

none of my stuff has a beefy gpu

sour apex May 24, 2025, 3:50 PM

#

i wanted to mess with local llms and my server had a pretty old gpu for transcoding so i decided to just update it

#

that server probably needs an overhaul tbh

umbral trout May 24, 2025, 3:55 PM

#

bruh....it is working awesome...thank you

#Ollama on a Jetson Orin Nano / Docker