Hello everybody, I wanted to ask if anybody here is running a device dedicated for running LLMs to be used with HA. Currently thinking bout getting a NVIDIA JETSON Nano Orin Super. Recently I started using Mistral AI by feeding it my HA sensors. Sooner or later Mistral AI won’t be free anymore and I’m thinking bout running a LLM locally. I do own a gaming PC with a RTX 3080 with 12 GB of VRAM, but that seems kinda overkill and I don’t want to abuse it for LLM… so, did anybody build something and would like to share their thoughts? Thanks in advance
#Dedicated „PC“ for AI LLM?
odd crane
plush glen
Hello @odd crane,
I have a 4060 in the laptop that I use as daily driver, and just have the HA machine talk to that for LLM.
odd crane
Hello <@194892727479959552>,
I have a 4060 in the laptop that I use as daily dri...
Yeah, but a RTX 4060 draws a lot of power.
The Jetson Nano Orin Super has a max usage of 25 Watts. I really wouldn't want to use a gaming gpu due to the high power consumption. I don't require realtime conversation with the LLM
plush glen
This is a chip built into my laptop, and it's my daily driver. Also for video editing. I've never opened a game in my life. I'm saying mine will be on anyway, and being built in I doubt idle is any more power on standby that that thing. Plus if you didn't want to hear alternatives, why post the question?
You will need memory in the GPU at least 8gb, more if you can, to run the models.