#Issues configuring the LLM endpoints and models

1 messages · Page 1 of 1 (latest)

weary agate
#

Hello, I did some testing of the new LLM features over the weekend and had difficulties getting it to work. We host about six models internal to our company and do not use any of the public endpoints. I believe they are just ollama models, but they are hosted in a k8s cluster. Here are some of the issues I ran into:

  • I think prompt mode ignores my OPENAI_MODEL setting. When using this mode I get a 400 error bad message, but when I use the command line llm --model <model> | with-prompt "Test" it works fine.
  • Creating a dagger function using the LLM ignores my environment configuration and defaults to the OpenAI public URL.

Are there different ways to configure the LLM endpoint and model that I am missing? I followed the online docs and set OPENAI_BASE_URL and OPENAI_MODEL in a .env and in my terminal.

storm pelican
#

👋 When you are in the module directory and have a dagger shell, can you run .echo $OPENAI_BASE_URL to ensure its the correct address?

weary agate
#

I tried some suggestions in the other thread and it works now. Apparently using 'WithModel' was also changing the endpoint back to the public openapi endpoint