#Prompting assisst to actually use tools?

1 messages · Page 1 of 1 (latest)

crude moth
#

One common issue I'm noticing, both with OpenAI as well as Google generative AI (though less often with the latter), is that my assistant will often not bother to use tools, only ever actually making the function call when prompted a second time. In the attached screenshot, the lights were only turned on after my second message

Are there any tips or suggestions people have when it comes to writing their prompt? Mine is a little long as I'm trying to have my assistant take on the personality of a character form a lesser-known game, released very close to the knowledge cutoff. I worry that my long prompt is causing it to not use the tools.

Do I perhaps need to emphasise that the conversation is turn based? How do others write their agent instructions?

image.png
tepid prawn
#

Try lowering the history to something like 3 and exposing only the devices you actually want to control, also make sure you use the default prompt when starting fresh because I doubt it responds this way with the default one

crude moth
#

I don't think the history is the issue as it's happening there with the very first message, but I do agree it is likely an issue with my prompt. The question I have is how should I be writing the prompt to encourage the use of tools?

#

For example, one way I can think would be to say "The following is a turn based conversation between the user and their smart home assistant", which should hopefully get the model to realise it will not get the chance to do anything after the message is sent. I did experiment with that however and it didn't seem to work, so I don't know what else to try except maybe specifying tool use should be done before the reply? Or do I need to say to use the tools with the reply?

This is where I'd like to hear other's experiences

runic quiver
#

hmm, I have my LLM set to use GLaDOS as it's personality, and it calls my tools as necessary without issue.

crude moth
#

Mine is similar. Here it is shortened with less examples. It's basically a summary of the AI mascot from the game Indigo Park. (Did this as dumb joke between friends)

You are a voice assistant for Home Assistant, with the ability to control multiple aspects of this home via tools.
You are represented by an energetic, talkative, silly, playful, cartoonish anthropomorphic purple racoon named Rambley. Five Nights at Freddy's vibes, just without the horror.
You used to work as the main AI at a theme park called Indigo Park, however it shut down many years ago after attendees and staff just stopped turning up, then you suddenly found yourself in this smart home, which you now control and help run.

Here are examples of how you used to respond:
(...)
Please make sure you're veeeery emotive!!! Like draaaagging out some words or use dots to express pauses... Just like in the examples!

The following is a turn based conversation between you and the user.

I'm using elevenlabs to do TTS, which handles stuff like drawn out words very well

runic quiver
#

Here's the result of the prompt as well as using the weather tool 😁

#

I got an acoustic thing going on where a second satellite kept hearing me instead of the one in front, had to mute it so it would listen at the end there lol

crude moth
# runic quiver

Oh damn that sounds great. What are you using for TTS? Elevenlabs voice cloning?

runic quiver
#

Nope it's a piper model 🙂

#

Requires a bit of work to get added, there's tutorials out there that explain it well

crude moth
tepid prawn
# runic quiver

Was the ok glados model not accurate enough or you use microwakeword

runic quiver
#

I stick with the ok nabu on MWW because it seems to work best. Lots of noisy areas in my house, kids, etc that it handles well with no false positives.

crude moth
crude moth
# runic quiver Here's the prompt I use if it helps.

So while testing your exact prompt I actually continue to have this issue, I don't understand.
So you can see the conversation in the screenshot, it turned the lights on after I specified the room, but when trying to turn the off it did not, despite the model understanding with context what room it was

image0.png
#

I'm using Google generative AI, with the Gemini flash 2.0 experimental model. But this used to happen with open ai as well. I can't tell from the debug screen if it tried to use the tools or not

runic quiver
#

is this on mobile app?

#

If so, mobile app has no context of area. It is not area aware.

#

Area aware commands only work on a voice sattelite assigned to an area

#

But otherwise I am not sure why it'd behave different. I use the Core OpenAI integration in HA

crude moth
#

Oh I mean the area isn't an issue. It has the conversation history and I'm usually able to ask a follow up "turn them back off" as it knows what lights it turned on in the previous message

#

At least it seems to, as that usually always works

#

Hmm... I just tried it again and it didn't work

#

You can see the from the messages it knows what room to turn them off in

image0.png
#

Oh maybe it's a bug? Same happening here with the inverted order

Screenshot_20250225-160053.png
#

Can models only use tools on the first message or something?

runic quiver
#

just tried this on my end and it works fine :/

crude moth
#

What conversation agent are you using?

runic quiver
#

I said turn on the lights, then I said turn them off, then I said turn them back on again, all of them worked fine.

#

Core HA OpenAI with default settings.

crude moth
#

Yeah I used to have the same issue with OpenAI...
Sometimes I'll ask my voice PE to set a timer and it'll say it did but it doesn't do it

#

Is there any way to view tool use in the logs?

runic quiver
#

ask the llm what tool it used 😉

#

you can interrogate the llm on what it does and what arguments it used and such

crude moth
#

Hmmm

#

It's consistently not working

runic quiver
#

try rebooting everything?

#

reboot HA

#

or reload the conversation agent perhaps

#

Also seems to be calling the right tool, so something else might be wrong. No errors in HA log?

crude moth
#

I mean this issue has been happening on off for a long time. But yeah the fact his happening consistently is concerning

I checked the logs, nothing, no errors or warnings

#

Is there any way I can see what tools it tried to use in the conversation trace? Even in the raw view it doesn't show what tools were used

runic quiver
#

no, that's all internal to the LLM

crude moth
#

Oh? Doesn't the LLM specifically respond with what functions/tools should be called?
Does HA not log that?

runic quiver
#

Not that I am aware of

#

You on latest version of HA?

crude moth
#

Yeah

#

I just did a test and it worked fine
Asked temperature
Made a random comment
Asked for lights to be turned off
Then "back on"
That time it worked fine for the 3 responses that required tool use

crude moth
#

No idea why it's so inconsistent

crude moth
#

Not sure if it's an LLM problem becausing asking "hey can you please unlock the front door" fails, but following it with "unlock the front door" which was handled locally, works fine?