Return script result to LLM answer.

[foggy geode]

I have this code:

sequence:
  - action: llmvision.image_analyzer
    data:
      include_filename: false
      target_width: 720
      max_tokens: 30
      temperature: 0.2
      provider: 01JKZVKXSB5ZD4KDXQ1JG81CXH
      model: maxi_llava-phi3:latest
      image_entity:
        - camera.front_door_camera
      message: >-
        Shortly describe the person or persons at the door. Also say how many
        they are.
    response_variable: answer
  - set_conversation_response: "{{answer}}"
alias: Generate front door person description
description: >-
  This script can be used by the LLM to describe the current situation at the
  front door. 

But unfortunately, the set_conversation_response: "{{answer}}" is not passing the data to my Voice assistant.

Any ideas?

The traces show me it should work

It doesn't seem to work neither when specifying which object:

End goal is to ask my ESP32 endpoing "What's going on at the front door" and have an answer

[scarlet orbit]

has to be structured kinda like this:

description: "Gets the daily weather forecasts. Use this when asked about the weather."
action:
  - action: nws.get_forecasts_extra
    target:
      entity_id: weather.kpvd
    data:
      type: twice_daily
    response_variable: result
  - stop: ""
    response_variable: result
speech:
  text: "{{ action_response }}"

you need that stop command in there, then you use action_response

though to be fair this is from an intent script. Now that I think about it, automation handles this a bit differently I think. If I remember correctly, the set_conversation_response does not work right when using LLM stuff, think that's a known bug in HA

[rapid birch]

set_conversation_response should be used in automation, triggered by sentence. Not in the script.
If you use LLM conversation agent in pipeline and exposing this script to it, then use stop block and just return your {{answer.response_text}} there. And in script description give instructions to your agent on how to interprete and sound this data.

[vague prism]

I'm trying to do this exact same thing as well. I have a script with a stop block and a dedicated response variable, which according to the traces has a value set, but my assistant says it was unable to see what was there?
Am I returning the response variable correctly? Here is my script

sequence:
  - action: camera.snapshot
    metadata: {}
    data:
      filename: /media/temp/doorbell_snapshot.png
    target:
      entity_id: camera.reolink_video_doorbell_poe_fluent
  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >-
        Describe in detail what you can see on my video doorbell. Be descriptive
        when it comes to people and cars. You don't need to describe the
        buildings unless they are relevant
      image_filename: /media/temp/doorbell_snapshot.png
    response_variable: generated_content
  - variables:
      description: "{{ generated_content.text }}"
  - stop: Complete
    response_variable: description
alias: Describe Doorbell Camera View
description: >-
  This script can be used to fetch a description of what can be seen on the
  doorbell camera

It's reassuring to see other people trying to do the same thing though it also doesn't work for me

Ah, after asking my LLM what the error was, supposedly it was this:
HomeAssistantError "Failed to process the returned action response data, expected a dictionary, but got <class 'str'>

[rapid birch]

Returned variable should be dictionary.

variables:
  result:
    description: .....

And then

stop:
  response_variable: result

[vague prism]

Eyy that's correct! That works now perfectly fine

Hopefully this helps you Maxi with your script

[foggy geode]

Oh shit, this never pinged me for the answers!

Let me try all of this 🙂

[foggy geode]

I am trying this atm! Will come back

that worked. Genial!

Final code for those wondering:

sequence:
  - action: llmvision.image_analyzer
    data:
      include_filename: false
      target_width: 720
      max_tokens: 100
      temperature: 0.2
      provider: 01JKZVKXSB5ZD4KDXQ1JG81CXH
      model: maxi_llava-phi3:latest
      image_entity:
        - camera.front_door_camera
      message: >-
        Shortly describe the person or persons at the door. Also say how many
        they are.
    response_variable: answer
  - stop: Finished
    response_variable: answer
alias: Generate front door camera description
description: >-
  This script can be used by the LLM to describe the current situation at the
  front door. ```

in a script

[rapid birch]

Worked with plain answer for LLM. Okay I guess. 🙂

[foggy geode]

It's working once out of two

Sometimes it just returns "started"

ok omg

I created one script per camera

and it seems like one call can run the three scripts?!?!

[scarlet orbit]

Yeah llm can call multiple tools if it needs

[foggy geode]

Man, this is one welcomed surprised! I need to get a 5090 to speed up processing times tho

my 3090 is slow