I have tried setting up Ollama as a local AI to use with assist and I think that I have it set up correctly but it seems to be hallucinating. It is telling me that it has tried to turn off the lights that I told it to but it "couldn't find a way to do so. It seems like these devices aren't responding to my commands." I have the entities of the devices I'm trying to control exposed to it so it should be able to control the devices without an issue. I don't know what to do about this.
#Ollama wont control my devices
frigid crag
inland roost
You have the llm set to allow control right? Also how many entities/scripts are exposed? The more stuff exposed the larger the context window needed. Also depends on the model, some don't perform too well.
frigid crag
You have the llm set to allow control right? Also how many entities/scripts are ...
Yes I have it set to allow control and there are 144 entities exposed. I am using ollama 3.2 for my llm.
inland roost
At 144 entities you probably need at least a 16384 context window for the model to maintain everything and not hallucinate. Also not sure llama 3.2 performs super well
3.2 is a bit low on the leaderboard it seems
frigid crag
Ok Ill try to install llama3.3 and increase the context window and see if that fixes anything. Ill let you know how it goes. Thanks!
inland roost
Bear in mind llama 3.3 is a 70b model that requires 40gb vram at q4 quantization 😅
frigid crag
Well maybe I wont then, good to know.
frigid crag
Bear in mind llama 3.3 is a 70b model that requires 40gb vram at q4 quantization...
Is there one that is better that wont blow up my computer when running it?
inland roost
People seem to have luck with qwen 2.5 7b and 14b
frigid crag
Ok, Ill try the 7b one. Thanks