I've been trying to set up my local LLM with use with HA and it's served via vLLM. Sadly, I'm not finding any applicable integrations. Am I missing something or is it time to put on my contributor hat and write one?
#vLLM support?
final mauve
gritty hazel
Pretty sure you can use this: https://docs.vllm.ai/en/latest/serving/openai_compatible_server/
And this: https://github.com/acon96/home-llm
It worked for me, but I did not find that VLLM was worth the hassle over Ollama with my 3090