Ideal Voice Prompt

[lapis gulch]

Is there an ideal Voice Prompt for local HASS control with "Prefer handling commands locally" selected that can also fallback to Ollama/local AI that can respond with real world questions without being super verbose? Basically i want HASS control first and foremost, but then i want to be able to get responses to "Tell me about turtles" or something non HASS related. I've tried quite a few and i get responses, but i also want the responses to be direct and not verbose OR if my LLM doesnt know the answer, dont ask for a bunch of additional information and just respond "I dont know".

currently using this below.

Your name is "Alexa". You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point.
If I ask you to switch a device simply answer with "Okay".
If you don't have enough information to execute a smart home command respond with "I dont know".

Current Time: {{now()}}
Current Area: {{area_id(current_device_id)}}

A list of available devices in this smart home:

entity_id,name,state,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{entity.state}},{{entity.aliases | join('/')}}
{% endfor -%}

Areas:

area_id,name
{% for area_id in areas() -%}
{{area_id}},{{area_name(area_id)}}
{% endfor -%}

Always be as efficient as possible for function or tool calls by specifying multiple entity_id.
If you are asked to control all of something, make sure to loop through each entity and add that to the array before calling the function.
If you know the current area, make decisions based on the current area first.

[nocturne mirage]

My previous prompts were straight up not working, this one is the best so far .. even works with Gemini-flash2.0 free. Thanks for sharing it !!

[hot gazelle]

This seems to be prompt for something like Extended Llama Conversation. Regular LLM integration will add info about exposed entities, areas, tools and stuff automatically, you don't need to duplicate it in system prompt manually.

[nocturne mirage]

@hot gazelle That's what i thought initially .. but the LLM (edit not llm but home assistant's ability to detect action from LLM response) is unable to take action without duplication of info.

For example: I am using gemini flash 2, when i say something like -- movie mode,

(With duplicated context)
it finds the aliased movie light and turn that on with high accuracy and responds with a message "turning on movie lights for you."

(Without duplication)
It finds the lights, says that ok i am turning them on, but home assistant does nothing 😞

[hot gazelle]

Well, then it's bug in HA integration. Because it should work. For custom tools you have scripts, exposed to Assist.
Can you confirm which integration you're using?

[nocturne mirage]

My setup is manual docker based:

1. Home Assistant container
2. Voice Pipeline (whisper, openwakeword, piper) using wyoming protocol
3. Conversation Agent is GeminiFlash2.0 using Google Generative AI integration

[hot gazelle]

That's what I said: official Ollama integration does add the entities and tools to the system prompt.
For custom one (like Google Generative) this is required.

[languid kite]

Google generative ai is a core integration 🙂

But I've seen people have problems with it, seems to handle the prompt data differently than openai does

[hot gazelle]

Oh, my bad!