#Unable to interact with Web UI

8 messages · Page 1 of 1 (latest)

nimble folio
#

I've spent 20+ hours trying to get a local AI working, to no avail. I'm running Ubuntu Desktop, and the machine I'm trying to run it on has a GTX 1080. I chose a light model (For obvious reasons) and i got it setup, but I have not been able to actually interact with it in any way. It loads up fine, but when i try to connect to the web interface, 5 popups appear on the right of my screen, giving errors. In the command line, a bunch of errors also pop up. I am trying the model "Meta-Llama-3-8B-Instruct" which I have Symlinked to a larger drive. The Symlink works fine, as the model loads, but I have greatly reduced options in the Web UI, and anything I click gives more errors. I am a complete newbie to the local AI field, so I probably did something wrong. I have edited and created multiple files (can easily be undone) in trying to fix this issue, but nothing has worked. I believe it would be best if I could VC with someone who actually knows what they're doing, but continuing this thread will also work, albeit slower. Feel free to request any info that can help this get fixed, thank you.

Web-Errors.png 📎 message.txt
desert compass
#
  1. Try installing and using the app with start_linux.sh and not with python3 server.py
    This ensures the app uses an isolated python installation with exact library versions it needs and not your system python that's wildly different from what's intended to be used.
  2. I'd recommend using models that has a -GGUF suffix. It will use an engine that's better suited for your GPU and in general faster and more reliable.
#

Also if you look at GGUF models, you could see that they have multiple files in repo. But each file is a dedicated model version. One of the most optimal versions when VRAM is limited is using GGUF files that end with Q4_K_M.gguf as they have a decent quality/size ratio.

#

And there's no need to download them all, one file is sufficient.

nimble folio
#

Nvm, it worked with the argument, and the web UI is now functional, thank you so much, I would've never thought of that. You are actually a God send

#

I'm going to mess around with the model I have right now, but do you have any other info on GGUF models? Any suggestions would be appreciated, as I am completely new to this.

desert compass
#

Well, it's probably the most popular format for local users. Works faster than Transformers models. File size is usually 2-4x smaller.
Offers decent quality and flexibility.
Thanks to community, -GGUF models are widely available and their versions exist for most models.

If you need some specific model, you can go straight to:
https://huggingface.co/bartowski
https://huggingface.co/mradermacher
And they most likely have a GGUF version of the model you're looking for.