I want to do many stuff with LLMs. Now with google AI. I want to make it local.
From wich jetson (i think thats the cheapest if you want a system for LLMs) do you get fast responces in home assistant?
#What for jetson to buy
bronze anchor
solid python
Well it depends on what kind of llms you want to run. For more serious/powerful stuff you're probably better off with a gpu, as they tend to have for TOPS than the Jetson devices and in many cases more vram
For instance the newest jetson orin nano super is only 8gb of unified ram (so probably only like 6-7gb usable). Most decent 7b models like qwen2.5 will use around 5gb if that just for the model, leaving only 2gb for context window. That probably is only gonna allow for a window size of around 8196 at best. If you have alot of devices and scripts exposed that will fill up fast. I'd say nowadays for a decent local llm experience you'd want at least a 16gb gpu.
native prism
so thats like 1000 euro for a whole system
It's more, if you use 4x series 🙂