left mountain
So I just got HAOS running on hardware with an Intel APU and wanted to get oolama with a tiny LLM model running on device but I couldn't find any resources on how to do it. Could anyone point me in the right direction?
I should be fine since I have over 10 GB of Ram left.
I hope it is possible without proxmox, since I only have Wifi to work with.
sharp anchor
You can't run LLMs on HA OS. It doesn't meant for that.
left mountain
Can you run Docker on HAOS?
Are there any other options to host an ollama server and HA on the same device while using Wifi?
sharp anchor
I don't think so.
Well, you could, earlier: https://www.reddit.com/r/homeassistant/comments/x6c85k/portainer_addon_here_did_it_go/
But i think it's unsupported now.
Brief google search shows this also: https://github.com/SirUli/homeassistant-ollama-addon
left mountain
Brief google search shows this also: https://github.com/SirUli/homeassistant-oll...
I did take a look at it but it does not support GPU acceleration and it crashes some peoples setup according to the issues reported on GitHub
hoary lark
for GPU acceleration you'd have to run outside HAOS, HAOS does not use GPU nor does it expose it to any addons. Most people run their Ollama instance on a separate server/VM/docker container