Fetching live data from the internet to complement gpt-4o info

[gleaming umbra]

HI. I'm just just getting my head around chatGPT and integration into HA. Like many, I'm trying to relicate Alexa but with more control. I'd like to be able to ask for news headlines and sports scores but clearly this is out of scope for chatGPT 4o.

Is there a recommended way? I guess I could have HA sensors that might expose some of this (in the same way that weather works) but is there a more appropriate and comprehensive solution to achieve this?

[simple sentinel]

If you use GPT as your main integration, sensors would be the easiest (but I guess very computation intensive) way.
I use Assist, with fallback to local LLM.

[glossy oasis]

You can create an intent script that calls out and gets the data back, and spits that out as the "speech". You then write a description for the intent script that helps the llm know what it does. That then let's the llm use that intent as a tool.

For example i have this intent script to get the daily weather

description: "Gets the daily weather forecast for the next several days."
action:
  - service: weather.get_forecasts
    data:
      type: twice_daily
    target:
      entity_id: weather.kpvd
    response_variable: result
  - stop: ""
    response_variable: result

speech:
  text: "{{ action_response }}"

Then whenever i ask how the weather is, the llm knows to use this tool to get the current weather.

[simple sentinel]

BTW does that work for LLM? Can it parse data from service successfully?

[glossy oasis]

Yeah I use that exclusively with llm

It returns the json from the weather service call, and the llm uses it to formulate a forecast

[glossy oasis]

It also lets you do open ended stuff, like "what's the weather like tomorrow? Tuesday? Tonight?" Etc, or "will it rain today? Tomorrow?" Basically it can answer anything as long as it's in the response payload

[gleaming umbra]

@simple sentinel don't you find the Assist too cumbersome when asking for music? With chatGPT I can be a lot less formal with the instructions and it still understands what to do.

@glossy oasis thanks for your insight. I'll give that a go.

[simple sentinel]

Well for that I use LLM to parse user request and give me track, album, artist. Then use with my own Music Assistant script.

I have 2 reasons to use Assist: i don't have powerful enough local LLM for this (and don't want to use ChatGPT with my data), and still don't trust LLM to run my house, if it can hallucinate and e.g. open front door.

[gleaming umbra]

how do you control whether Assist or your LLM processes the utterance?

[simple sentinel]

I use script, that's calling conversation.process action from custom Assist intent script.

[uneven quest]

This is blowing my mind a bit - I’d never thought to use LLM for this. Would you mind sharing the script?

[simple sentinel]

There will be several. Prepare for it 🙂

1. This is the main script that is called from intent:
   https://pastebin.com/5yn499XZ

2. This is actual script that is using LLM: https://pastebin.com/1i7LDvdK

3. This one is to find and play on MA: https://pastebin.com/75WdwAp1

@uneven quest do you need sentence/intent_script?

[uneven quest]

That's awesome, thank you! Sure, it'll help.

And on the conversation side, you're just using a bunch of sentences with wildcards, right?

[simple sentinel]

Here: https://pastebin.com/VU5ckNAb

[uneven quest]

Ah ok, scratch that I see what you're doing

[simple sentinel]

Yeah, check the last paste. It has regular sentence with area, and without area for using in satellites, assigned to the room.

[uneven quest]

Fantastic. Thanks again

[glossy oasis]

dunno if you saw, but music assistant recently made a whole bunch of intents: https://github.com/music-assistant/intents

they've also been refactoring their codebase to be more intelligent with these intents as well as the LLM stuff

https://music-assistant.io/integration/voice/

[uneven quest]

Also very cool. I haven't looked at MASS in a while - I had issues early on, and so wrote my own (really long-winded) Spotify Search->Sonos functionality. Basically API calls and a ton of entities based on a search string. It works fine for my needs, but I reckon I need to go back to MASS at some point, there are a few advantages I'm likely missing out on (including these intents).
Right now my intents are written to basically automate passing data to the Spotify Search thing which I configured, via a script.

[simple sentinel]

Yeah, MASS is actively preparing to be merged into core. IDK if it's good or bad, actually. On the one hand, stability and support. On the other hand, all things in core are SOOO SLOOW in upgrades, that i will miss MASS frequent releases....

[simple sentinel]

I don't want to rely on MASS Assist intents solely, to be honest... They will be very strict, and also my "play" thing works for video on Jellyfin too. 🙂
Also i don't want to rely solely on LLM, as it may work well here, but require too much access to be fully helpful.. So i decided something in the middle. 🙂

[little blade]

I'm a little late to this party, but how accurate is all of your setup? Additionally is the LLM searching your library or is it just seeing if what you've asked for exists overall and then you just hope you already have the media you want to play?

[simple sentinel]

It's pretty accurate, if i'm using some well-known bands or songs in my prompt. GPT is better with it (MUCH), but i don't want it to be in my chain 🙂
I just ask "play Queen", or "play In the End", and it returns JSON (with only "artist: Queen", and with "track: In the End, artist: Linkin Park"). Then i search it in MASS library, and if it's there it plays.

[little blade]

I've just learned about MASS today so I haven't got that up and running yet, do you know if it can control/play media from Roon Server/Players?

my endeavor is to get voice control of my roon players (whole home audio) to have the same features I can have on a google home and a local LLM seems like my best path forward for this at this time. All of my music is hosted locally. I was hoping my LLM was going to be able to find available media in my libraries and then play it, but seeing what you've put together, it looks like it needs a lot of hand holding.

[little blade]

did you set up a voice assistant specifically for processing music requests? Is that how it always parses your prompt into a media request or is there some other magic going on that it knows when to apply your Refine play request?

[simple sentinel]

Check the scripts i gave. Intent script is working on any "Play ...." sentence, taking everything to 1st script i pasted. Then that script sends the user input (raw message) to LLM via "refine" script, getting json with track/album/artist back. Then it performs search in MASS library, and plays it

[little blade]

It looks like your assist is looking for that sentence, then based on the sentence match it kicks off your script, your script then queries the LLM via another script waiting for the results back which it then takes those results and forwards on to one of two scripts a music and a video one. Hopefully I've followed the logic correctly.

[simple sentinel]

Yes

music OR video, based on LLM parsed response

[little blade]

would you mind sharing your video script as well? What you have here looks pretty close to drop and go for most of us if we have all the components in place. This is really awesome.

[simple sentinel]

Uh, that thing i'm afraid is pretty personal. It is turning my TV on if not turned on, then is using REST command to find ID of movie on Jellyfin, and another REST command to launch it. Also that part is pretty unstable (because of Jellyfin client instability on Google TV...)

[little blade]

ah okay, so not so drop and go for that part. The video would have just been a bonus to the rest anyhow. 🙂

[simple sentinel]

Yes, looks like it. Sorry for misleading 🙂

[little blade]

I've worked backward to get everything else working for me, except for the intents. I am getting a missing slot error. I thought maybe the variable named media was the problem so renamed to media_string but still is an issue. My error is below:

File "/usr/local/lib/python3.12/site-packages/hassil/recognize.py", line 1107, in match_expression
raise MissingListError(f"Missing slot list {{{list_ref.list_name}}}")
hassil.recognize.MissingListError: Missing slot list {media_string}

my intent: https://pastebin.com/x1RhpwFp

I have the intent script in my configuration.yml but I don't think it's getting that far.

Any help would be appreciated as I've struggled with this for the last day.

[simple sentinel]

My bad. You declared using slot (media_string), but didn't define slot itself.
On the very bottom of your sentence file, paste this:

lists:
  media_string:
    wildcard: true

That means, you're expecting any text in place where media_string will be used as {slot}

[little blade]

I was reading the documentation and it was referring to slots, but I was like, this seems different since I am not mapping to an entity or list, because it's the LLM request. So I was thinking you just had in there what was needed. I'll give it a go!

[little blade]

Do you have a buy me a coffee link? This is awesome.

[simple sentinel]

There is one here: https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration but it's completely unnecessary 🙂 Community should help each other!

[simple sentinel]

@little blade thanks a lot! ❤️

[little blade]

You're very welcome. I'm so grateful for all your efforts and sharing them.

[little blade]

Recently my prompt has been wonky. It's like it became stupid and doesn't even know the context of what it is supposed to be doing. I'm using the Mistral_Nemo LLM. Have you had any issues with yours recently? I've not taken any updates I'm aware of, unless the LLM's update on their own and that caused it.

[simple sentinel]

Hmm, no, i use GPT-4o. Everything works as before...

[little blade]

It started working again this evening so not sure what was going on earlier.

[stuck niche]

Can I directly create a script within the script section in Home Assistant and expose that script to the LLM? Is the ‘speech’ part still necessary?

[glossy oasis]

You can use regular scripts as well, but they need to output data to the llm so it can use that data for summarization

[turbid geode]

Just a tease for you want you can do with live data

Not i use google AI and its tought it does not know live data. If you dont discribe the script enough it wont run.

If you describe in detail it thinks its a simulaton or a stub.

[gleaming umbra]

I've just revisited this because of my struggles with getting consistent chatGPT output. I have to say @simple sentinel, your scripts are remarkably good! I am using online chatGPT for refinement and the results are great. Very responsive and much more control. I've extended it to include playing favourite radio stations in MA. Very happy. I'm sending you a coffee 🙂