Custom sentences not all being used by local LLM

[jovial wren]

I have created a list of sentences I would like my Ollama driven LLM to use. The following one is used correctly:

`alias: Sentence - Movie Time
description: >-
This changes the basement scene to "movie time" when Voice Assistant given
command
triggers:

• trigger: conversation
  command:
  • Movie time
  • Time to watch a movie
    conditions:
• condition: state
  entity_id: binary_sensor.basement_overhead_motion_group
  state: "on"
  actions:
• action: input_select.select_option
  metadata: {}
  data:
  option: Movie Time
  target:
  entity_id: input_select.basement_state
• set_conversation_response: Enjoy the show!
  mode: single`

However, the following sentence does not work at all...

`alias: Sentence - Wondering
description: >-
This changes the basement scene to "Wondering" when Voice Assistant given
command
triggers:

• trigger: conversation
  command:
  • Wondering
  • Doing normal activities
  • Just wondering around
    conditions:
• condition: state
  entity_id: binary_sensor.basement_overhead_motion_group
  state: "on"
  actions:
• action: input_select.select_option
  metadata: {}
  data:
  option: Wondering
  target:
  entity_id: input_select.basement_state
• set_conversation_response: |
  Do not get lost
  mode: single`

I can tell when they are being used by looking at the last triggered flag. Appreciate any help.

[cobalt hornet]

wrap code in triple ticks please. Like this:

Some code here

Makes it easier to read. 🙂

[jovial wren]

I used the wrong tag... Fixed.

[cobalt hornet]

So one thing to mention here, LLMs do not look at automation triggers. At best, an LLM may choose to use an automation if it is exposed to it, and the description or name of it give it enough information to understand how to use it, but I have found generally LLMs don't use automations much.

You'd want to create scripts, as LLMs use those as tools based on the contextual information they receive in the request made to them

[jovial wren]

Oh... Let me fix that

I had referred here: https://www.home-assistant.io/voice_control/custom_sentences when I made this originally.

Since the page is listing triggers, how do I do those in a script?

Am I reading the page incorrectly? I cna be dense at times...

[thin tree]

Custom sentences are made to work with HA inbuilt conversation agent. LLM has nothing to do with that 🙂

[thin tree]

Don't use triggers, there's no triggers in scripts. Just add comprehensive description to script, what it does.

If you need some info to be passed to the script, use script fields.

Settings -> Automations and scenes -> Scripts

Those scripts.

[jovial wren]

@thin tree thanks for that - totally was not tracking inbuilt vs LLM. Given that these automations were just setting a scene, would I just include those in the model instructions?

[thin tree]

Yup\

Just tell AI what to do 🙂

It's convenient to use script description (in "rename" section) to describe "this tool is used when user wants to set a scene" or similar.

[jovial wren]

A script with a single command - but an extensive description of when to use that command...

[cobalt hornet]

Yeah exactly

Here's a really basic one I have:

description: "Gets the daily weather forecasts. Use this when asked about the weather."
action:
  - action: nws.get_forecasts_extra
    target:
      entity_id: weather.kpvd
    data:
      type: twice_daily
    response_variable: result
  - stop: ""
    response_variable: result
speech:
  text: "{{ action_response }}"

granted this is an intent script so that speech part isn't relevant here, but the concept is the same for a regular script

the description helps the LLM understand when to use to tool. You would also want to add instructions in the description about what arguments the tool takes and what values are appropriate

Here's an example from a script I have to launch apps on my android tvs:

description: >-
  Used to launch an app on an android tv. Takes 2 arguments.
  The 'Name' argument, which is the name of the device to open the app on, 
  or the 'Area' argument which is the current Area the request came from  if name is not provided.
  The 'App' argument, which can be one of the following: moonlight, twitch, twitch streaming, plex, disney plus, 
  netflix, crunchyroll, youtube, youtube tv, hulu, prime video, paramount plus.

[thin tree]

Isn't that 3 arguments tho? You're trying to confuse that poor LLM Nick?

[cobalt hornet]

It says name or area as first argument, then app 🙂

It seems to understand that lol

[jovial wren]

🤣

[thin tree]

ah, there's OR. I see. 🙂

[jovial wren]

@cobalt hornet In your weather example, I see you mention 2x daily. Does the LLM pull the weather info down and store it locally in order to speed up response?

[cobalt hornet]

No the tool is called every time it is needed. That's just a property of my particular weather integration set there 🙂

[jovial wren]

Gotcha... Too bad.

[thin tree]

LLM cannot store info. It just holds some context of previous questions, depending on context window available...
Tool usage is the quickest part of your interaction with LLM though. 🙂

[jovial wren]

I have been wondering if RAG (right acronym?) would be something to look at...

[thin tree]

I would think so. I can't do it myself tho - lack of good hardware is restraining my strieves 🙂

[jovial wren]

Well... I just tried my new scripts - and boy am I not close.

Coming up with a good description is harder than it seems

[cobalt hornet]

Script is exposed right?

[jovial wren]

It is now... And with that things start to work. Man am I slow.

[cobalt hornet]

But you got there eventually! 🙂

[jovial wren]

Thanks to all of the help - which is much appreciated.