Hi, any suggestion about some tutorial or ready made pack to serve AI via browser on locally hosted solution? I got Llama3 in local environment, but I'm stuck at that point: how to access it via browser using some similar chatgpt ui.
#Use local Llama3 with webserver for UI via browser
gentle salmon
grave skiff
Hi, any suggestion about some tutorial or ready made pack to serve AI via browse...
I believe there already exists a solution with open webui, I have a link to a video which has a full tutorial on how to set it up using ollama and docker :)) here's the video : https://www.youtube.com/watch?v=Wjrdr0NU4Sk
gentle salmon
I believe there already exists a solution with open webui, I have a link to a vi...
Awesome, thank you!
grave skiff
Awesome, thank you!
you're most welcome :))
nocturne plaza
Awesome, thank you!
for just personal use that doesn't need to worry on privacy for some sensitive data, renting some cloud GPUs from vast.ai, AWS, etc is enough, since building a local rig for that is pretty costly, yea dual 4090s may work but a workstation rig & gpus is more preferrable
gentle salmon
for just personal use that doesn't need to worry on privacy for some sensitive d...
In trying to get things working on my pc just to study how it works and look at training part too, than im trying to get infos about this due to privacy concern with AI for a company, so it is useful to build what is needed to get something that works locally in the company only, with maybe some sensitive data
hushed caveBOT
Ayo? @gentle salmon level 1 !!! 
nocturne plaza
In trying to get things working on my pc just to study how it works and look at ...
depend on how much billion parameter LLM and the amount of dataset the company wants, if it requires more than 48 GB vram (dual 4090s), yeah you really need a sufficient workstation rig and GPUs
since I have no knowledge about that, I think you'd better discuss more with your client company
gentle salmon
depend on how much billion parameter LLM and the amount of dataset the company w...
Ye it is something we must figure out, how much it changes from 8B to 70B to 400B params, what may be worth.. if kore params need more time to load the output, it is not an issue at this stage, clearly in production it will be but I think the company can afford some high end workstation/gpus
neat cairn
With llama.cpp or ollama, almost any middle-end consumer GPU would work well with up to 12B param models (which are good enough for most cases). So you don't need high-end here specifically for local use
Though I'm not really aware about the UI side of things. llama.cpp provides a very basic web interface with the server