#Using scripts with openai - unpredicable results?

1 messages · Page 1 of 1 (latest)

lapis sage
#

Hi Guys,
I'm trying to build on the great work of ViewAssist (VA), to include better support for LLMs and in particular OpenAi. My plan was to use modified ViewAssist scripts to help the LLM set the correct modes for VA. However, I'm finding that even a simple weather request is being unreliable in so far as it sometimes calls my script but most of the time doesn't. I also find that sometimes I ask the LLM what script was used for the last (failed) response and it states that the script WAS used but HA doesn't show any trigger.

My method is to expose the script to the LLM and then write a description for the script like this:

This script is used with viewassist to switch the current display to the weather view.
The tool takes the following arguments: area. area is always required and must be supplied.
Use this tool whenever the user requests information about weather - such as the weather forecast
or outside temperature.

Is this the correct approach and the problem is unpredicable LLM responses, or, do I need specific elements in my script to make this work?

outer cave
#

Does the LLM know whaty viewassist is? It probably thinks it needs to call another tool when you say "used with viewassist"/ If viewassist is simply asking the LLM to do something and intends it to use a tool, then you wouldn't tell the LLM it has to use the tool with viewassist, you would just tell it it's a tool to get the weather and what arguments it takes. You probably also want to tell it how area is determined (Does it come from the user request? Do I assume it comes from the area in which a voice request is received?)

lapis sage
#

Good point on the viewassist comment. I also copied an old statement by mistake - it shouldn't have included the area requirement. The script description is now this;

Use this tool whenever the user requests information about weather - such as
the weather forecast or outside temperature.

and it doesn't seem to be any better at being used.

#

Also - there seems to be a pattern. If I save the script having just made a change to the description (which could be as simple as removing a full-stop from the sentence), the LLM will corretly use the script and reply with "weather not available at the moment" because I've simplified the script to only make the view change. On subsequent requests it responds with the weather forecast but doesn't run the script. Is it faiing for me because my script doesn't provide a weather response so it learns not to use it in the future?

outer cave
#

I think so, that is misleading for the LLM

#

when the LLM uses a tool it expects to get back some kind of data from it usually, so if the script isn't giving any kind of feedback the LLM probably thinks the tool failed or is broken

#

likewise once it calls a tool and gets that data, it remains in context so the LLM likely doesn't re-call the tool because it already knows the data from it.

lapis sage
#

Ok, thanks. I'll investigate that. What is considered a working script for the LLM ie what should I make the script output?

lapis sage
#

So I tried a script by TheFes from here: https://community.home-assistant.io/t/weather-forecast-access-for-llm-eg-chatgpt-or-gemini/803166 and that script has the same symptoms as my script i.e. it runs the first time but not on subsequent runs. It seems like the weather is cached somewhere? I tried to force the script to run with these instructions:

Use this script every time a weather request is made. Do not report the weather if this script has not been used. Do not use previous script results when a new request is made.

Home Assistant Community
🌦 Weather forecast access for LLM (eg ChatGPT or Gemini)

Introduction Since some time Large Language Models like OpenAI/ChatGPT or Gemini can be used for voice commands, they can even control your house now. 2024.12 will even bring an option to prefer local processing of the command first, so the LLM will act as a fallback. This opens a whole new world of possibilities. If you expose a script to Assi...

outer cave
#

What LLM are you using?

lapis sage
#

OpenAI

outer cave
#

I have scripts to retrieve weather and they work every time

#

are you using the core OpenAI integration?

lapis sage
#

Yes, with 4o-mini. I just asked the LLM how it was getting the weather and this was the reply - so I'm going to check and hide any weather sensors from assist...

image.png
outer cave
#

better to ask it what "tool" it used

#

to the LLM, scripts are Tools

#

and you are certain the script is exposed? And it returns weather data for the LLM to consume?

lapis sage
#

Yes, I usually use tool. The script is exposed and as far as I can tell, nothing else (the weather.home WAS exposed but now isn't).

#

Say's it used the script:

#

but it didn't fire:

outer cave
#

hm so if you go into the script settings, it shows exposed to assist?

#

strange that it wouldn't trigger if the LLM said it called it. There's not some other script or intent script for weather anywhere?

#

and also if it called the script once, it probably won't call it again

#

because it's result is in context

#

One thing you can try is to ask the LLM to call the tool again 😄

lapis sage
#

Well, that's interesting. "Call the script again" and it fires each and every time...

lapis sage
#

I've done nothing - except hide the home.weather entity some time ago, and now the script is firing every time. Could the hiding of the entity take time to ripple through?

outer cave
#

Yeah it's possible