#Notify automation randomly works and randomly doesn't?

1 messages · Page 1 of 1 (latest)

dreamy seal
#

I don't know what's going on, I have an automation set up to grab alert images and videos from blue Iris 5, and sometimes when it updates the notification it doesn't resolve the image and video:

https://i.imgur.com/GkPgMOr.jpeg

I can't see any reason why this would happen, I run an mqtt topic which is fired directly from bi5, home assistant grabs the file path on the network share directly from BI and fires an initial notification with just the image. It then listens for a second topic to confirm the video clip is converted, does a quick shell command to copy it to a temporary directory in /www, then updates the notification with the video link while preserving the same image using variables in the automation. I'll post the script below.

The normal behavior ~75-80% of the time is it just works, but every so often the update notification action breaks the image and the video link leads to an error:

https://i.imgur.com/JgJeTWZ.jpeg

For example, this one came in with the image on the initial notification, then when it updated the image broke and I got the above error. When I check the automation itself there are no missing or incorrect parameters:

https://i.imgur.com/WUUnPDy.jpeg
https://i.imgur.com/Vw42iLF.jpeg

Furthermore, when I go check the file system from an ssh shell, the image exists where the automation is pointing to:

https://i.imgur.com/Y6nl7Xy.jpeg

So the image at the very least should not be disappearing.

The video on the other hand...it's there in the nas but when I go to the direct path on one of these broken notifications it returns a 404 and it seems the shell command didn't actually copy the file to the www dir even though it exists in the source path exactly as provided, and the shell command didn't error out in the logbook.

I'm not sure where to go from here? The shell command itself is dead simple, copies from the file path provided in the payload to the www folder, and it uses the same variables passed from the original automation.

#

(let me get to a PC to grab the scripts, phone isn't playing nice with the vs code plugin to select text)

elfin forge
#

You need to post the full trace, not images of the trace

#

Otherwise, we have to take your word

#

There is likely an error that's occurring in your logs and the trace would show that

dreamy seal
#

How exactly do I get text from a "full trace" when it splits it into 10 different pages?

elfin forge
#

3 dots, download trace

dreamy seal
#

OK, one moment

elfin forge
#

FYI, IMO you should be triggering your automation off the generation of the image.

#

right now you're hoping the image is copied over before you fire out the notification

dreamy seal
#

I am. I'm updating it when the video link is ready.

#

The BI5 software does not fire the MQTT topic until after the alert image has been generated.

elfin forge
#

right but your script copies it

#

anyways, please post the trace

dreamy seal
#

No, the file is grabbed directly from network storage

#

for the image

#

I can't post the trace, too long and discord won't let me attach as a file

elfin forge
#

you somehow messed that up

#

open the file, copy the contents, paste the contents on dpaste.org

#

open it with a text editor

dreamy seal
elfin forge
#

Yes

dreamy seal
#

So, specific behavior for the image. On initial notification, the image is there. It breaks on the update part, if the video link is missing from the www folder. But I see nothing in this trace that suggests the shell copy failed.

#

If the video file is successfully copied to the www dir, the image doesn't break on notify update.

#

Shell command, dead simple shouldn't have any problems:

  copy_video_to_www: 'cp "{{ file_path }}" "/config/www/New/{{ file_name }}"'
  clean_old_videos: 'find "/config/www/New/" -type f -mmin +1440 -delete'```
elfin forge
#

You're firing the notification directly after the copy without waiting for the copy to complete

#

that's going to be the only way this fails outside a bug

dreamy seal
#

Shouldn't the automation wait for the command to complete before moving on?

elfin forge
#

and it would be a bug in the notification service, however because you aren't waiting, I'd say that's where you should look.

elfin forge
#

fire and forget

#

It's possible that the shell command is blocking

dreamy seal
#

They're defined as "steps" in the automation though, that implies otherwise...so how do I know when the copy command finishes if it doesn't track it by itself?

elfin forge
#

that does not imply otherwise

#

there are many actions that are blocking, there are many that are not

#

e.g. script.turn_on is not blocking, same with homeassistant.turn_on

dreamy seal
#

Where can I see a list of all commands that are blocking and not blocking? I've never once seen a description of this behavior in my hours of pouring over the documentation to get this running so far.

elfin forge
#

There is no list

#

you can assert that the shell command blocks by using response_variable

#

however, if the shell command does not block, then neither will adding a response_variable

#

that's why I said it's probably best to trigger off the creation of the file

dreamy seal
#

OK, how would you do that?

elfin forge
#

with the folder watcher

ebon finchBOT
dreamy seal
#

OK. So, I can integrate that. Apart from it, none of this explains why the image in the notification, which is 100% there and accessible, suddenly breaks if the video file doesn't exist?

elfin forge
#

Well, if you make the change and the issue still exists, then it's not the automation

#

remember, you posted in automation, so you're getting help with your automation

dreamy seal
#

So you're saying it's with how the app handles a notification with a broken video link?

#

I guess that makes sense

elfin forge
#

I have no idea, I don't have android

dreamy seal
#

It's dumb, but it makes sense

elfin forge
#

I have iOS and when the video is missing, iOS presents the image properly but the video doesn't show

dreamy seal
#

OK, well I guess one final question on this. How/where can I find the output of these shell commands so I can see what, if any, errors are happening on the copy command?

#

Because if I go run a shell command myself it works every time. so IDK why it doesn't as part of the automation

elfin forge
#

I still don't think you understand what the problem is

dreamy seal
#

No I understand there are two problems

elfin forge
#

the video may not exist on the server by the time you send the notification

dreamy seal
#

One, I'm firing an action before the file may be available. Two, sometimes the file doesn't copy

elfin forge
#

there's a disclaimer with shell command that anything that takes over a minute to execute will be canceled

dreamy seal
#

OK. So, where can I see the log from that?

ebon finchBOT
elfin forge
dreamy seal
#

I see on the trace I posted it took 8 seconds to wait on the file to be finished writing, the shell command doesn't run until after the wait, and a copy of a ~2mb file does not take more than a second or two

#

So the only sane reason the copy command wouldn't work would be because it errored out.

elfin forge
#

all you can to is turn on debug for the integration, or get the response from the command

#

I doubt debug would list anything

#

anyways, all of that will be in your logs

#

the response_variable will be shown in your trace if you add it (but not use it)

#

oh, and it looks like the response will give you

dreamy seal
#

OK, yes I see that it returned with an error, so I need to figure out how to store the output as a response variable:

Logger: homeassistant.components.shell_command
Source: /usr/src/homeassistant/homeassistant/components/shell_command/init.py:129
integration: Shell Command (documentation, issues)
First occurred: 4:55:08 AM (8 occurrences)
Last logged: 7:46:42 AM

Error running command: cp "{{ file_path }}" "/config/www/New/{{ file_name }}", return code: 1
NoneType: None

#

I think I have enough to get started here. Thank you for your patience as you walk me through this. Home Assistant is very much unlike anything I've ever used before. I'm not used to all of this just yet.

dreamy seal
#

Hmm, so indeed there is a problem with the copy command. Some other process may be touching the file after it's created. Wonder if a live backup process isn't configured to use shadowcopy or something...

    Resource busy```

I will have to look into this further. In the meantime, implementing the folder watcher as a "wait_on_trigger" step did prevent the notification from updating with the missing video file, so at least the thumbnail stays while I debug this. Thanks again!
#

In the meantime I am just adding retry logic to try the shell copy command again up to 3 times total if it happens to fail.

dreamy seal
#

Hmm, so do I have this implementation of the folder watcher correct? It always times out even when I see file watcher events in the log book. I have yet to see how I can find the exact filename the folder watcher event trigger is responding to, it doesn't seem to be included in the logbook, it just includes the folder and the word "closed"

code:

        - platform: event
          event_type: folder_watcher
          event_data:
            event_type: closed
            path: /config/www/New/{{file_name}}```

(file_name defined in variables at the beginning of the action, 

```    - variables:
        file_name: "{{ trigger.payload_json.file }}"```

Also for some reason I'm getting like 6 duplicate entries for the folder watcher any time one file is copied???
#

Nor am I seeing any useful info from the folder watcher event in the logbook itself, though I am confused as to why it doesn't report the filename it triggered from:

https://i.imgur.com/Xg24OnM.png

dreamy seal
#

So, I found the dev tools for states, as far as I can tell this folder watcher is "working" even though it's triggering 6-10 times with a "closed" state instantaneously. The end result is the entity state reports the proper data well within the timeout period:

https://i.imgur.com/WYJ0hJd.png

But the automation is still timing out.

elfin forge
#

so just look for 'closed' and condition off the path

dreamy seal
#

Then doesn't that make it useless for positive identification in the first place? I could have more than one instance of the same automation running at once, so that would generate a false positive if it did.

elfin forge
#

trigger.event.data.path

#

Not if you make it a separate automation

#

and yes, you can have multiple automation running concurrently, make the mode: parallel instead of single

#

so if you want it in the same automation, then you'd make use of trigger id's and a choose action

dreamy seal
#

I feel like there should be a simpler solution to all of this. In fact it'd be much simpler if it just let me serve the file directly from the network storage in the first place, but that never worked either when I tried it earlier.

#

Which is funny because it does let me serve the notification image directly. Just not the video

elfin forge
#

you can blame browsers for that

#

This is how the websites work, they all have a www folder with their served assets exposed

dreamy seal
#

I was directed to use a "media-source" but it always just redirected to the home dashboard.

#

But yes the browser method would just return a 401

elfin forge
#

if you want to just host the video, not on an HA dashboard, this is what you do

dreamy seal
#

I don't mind having it on an HA dashboard, but is it even possible to do it so that you could view a file after the automation has run again and "replaced" it on a media entity?

elfin forge
#

I don't follow

#

if you make a second automation, you'll be OK

#

You can make a video notification handler

#

or, keep it in the automation with the mode parallel and use trigger id's with choose action

#

it's really not that hard to do

#

take your trigger, move it to the upper area. Put an id on it. Then add a choose statement that conditions off that trigger_id. Also, add a second condition that checks the path.

#

and you're done

#

just ID the other trigger to have it go the other path

#

This is actually how all the "famous" blueprints work

#

i.e. the frigate blueprint that does the exact same thing

dreamy seal
#

I can only assume I'll understand what you're saying once I look up all these technical terms you're referring to. I'll give it a try this weekend. But for now the bandaid fix is a 3 second delay between the recording complete MQTT trigger and the shell copy command, and that's working so it will stay that way until it isn't working or until I have another 8 hours to bash my head against the wall while trying to figure this all out.

vagrant seal
#

what petro is talking about with trigger ids is an absolute game changer for automations. It can be used endlessly.

dreamy seal
#

I'm sure there's plenty of elegant solutions to this that I could do if I understood home assistant as a whole. Unfortunately that's a long uphill battle from where I stand right now and I'm sort of in a "get it working now so the family stops complaining" mode.

vagrant seal
#

100%

dreamy seal
#

It does look like maybe I was trying to use this template backwards. The documentation seems to suggest you could grab the file that triggered the watch alert:


#Send notification for new image (including the image itself)
automation:
  alias: "New file alert"
  trigger:
    platform: state
    entity_id: event.created
  action:
    action: notify.notify
    data:
      title: New image captured!
      message: "Created {{ trigger.to_state.attributes.file }} in {{ trigger.to_state.attributes.folder }}"
      data:
        file: "{{ trigger.to_state.attributes.file }}"```

That would suggest you're not really meant to be using it for filename validation, rather just serving a file that causes a trigger. At least, I think that's how I'm understanding it?
#

Anyway I do appreciate the help. It can be very difficult when you're new to all this to know exactly what to look for to solve a specific problem. This should speed things up a lot when I do get the time to sit down and run through it all.

#

(and I will readily admit almost all of this is outside my comfort zone in general. My knowledge of programming pretty much stops at whatever is needed to hobble together single 50-100 line python scripts and whatnot, but the "easy" solutions that involve first party apps are either too expensive to justify or just suck in general)

dreamy seal
#

OK, so progress here.

Assuming this current code:

  alias: Notify on AI Object Motion with Video Update
  description: ""
  mode: parallel
  max: 100
  trigger:
    - platform: mqtt
      topic: BI/motion
  condition: []
  action:
    - variables:
        file_name: "{{ trigger.payload_json.file }}"
        cam: "{{ trigger.payload_json.cam }}"
        memo: "{{ trigger.payload_json.memo }}"
        object: "{{ trigger.payload_json.object }}"
        image_path: /media/local/DS720/Alerts/{{ trigger.payload_json.alert_path }}
    - delay: 00:00:01
    - service: notify.mobile_app_sm_s918u
      data:
        message: "Confidence Rating: {{ memo }}"
        title: "{{ cam }} detected {{ object }}"
        data:
          image: "{{ image_path }}"
          tag: ai_motion_{{ file_name }}
    - wait_for_trigger:
        - platform: mqtt
          topic: BI/recording_complete
          value_template: "{{ value_json.file == file_name }}"
      timeout: 00:01:00
      continue_on_timeout: false
    - service: shell_command.copy_video_to_www
      data:
        file_path: /media/DS720/HomeAssistant/{{ file_name }}
        file: "{{ file_name }}"
    - wait_for_trigger:
        - platform: state
          entity_id: event.folder_watcher_config_www_new
          attribute: event_type
          to: "closed"
      timeout: 00:01:00
      continue_on_timeout: false
    - condition: template
      value_template: "{{ trigger.to_state.attributes.file == file_name }}"
    - service: notify.mobile_app_sm_s918u
      data:
        title: "DEBUG: Updated notification"
        message: "Created {{ trigger.to_state.attributes.file }}"
        data:
          image: "{{ image_path }}"
          clickAction: "local/New/{{ trigger.to_state.attributes.file }}"
          ttl: 0
          priority: high
          tag: ai_motion_{{ file_name }}
          channel: ai_motion_alerts```

But it doesn't detect the value_template so it times out, and I'm not sure why?
#

I tried putting the value trigger inside the wait-for-trigger block but apparently that's not permitted

#

Oh and if a second file comes in that doesn't match, it ends and doesn't keep looking for another file that does match

#

Is wait-for-trigger not the correct way to do this?

elfin forge
#

Or you have to use choose with a trigger at the top

#

Not a wait for trigger

dreamy seal
#

OK sure. So, a trigger ID for a second automation that waits for the file watcher to give an entity state change on a specific file?

elfin forge
#

Yes, trigger id and the trigger condition

#

Inside a choose action

#

Anyways, why do you need to check the file if it’s created? Shouldn’t you be notifying on all new files?

#

Worse case, if you’re just looking for a specific format, parse the filename

dreamy seal
#

I need to match the file with the image alert generated previously

elfin forge
#

Then name the file with the image alert information

#

So there’s no question when to trigger in a separate automation

dreamy seal
#

I have both in the initial payload

elfin forge
#

Right but you won’t have that info in a separate automation

dreamy seal
#

Why is that? Are you not able to send conditions with the trigger ID?

#

I'm not sure we're on the same page here.

#

Let me back this way up for a second. Maybe I'm going about everything here the wrong way. I have two simple requirements for this process: I have NVR software, Blue Iris 5, which generates AI confirmed alerts and saves video clips if a specific event happens on a camera. BI5 handles all AI object detection, filtering, and delivering of payload information, so the MQTT trigger should only ever fire when I want to be notified of something.

The goal is to take this payload info, extract the alert image and video, and deliver it to the family's phones. But the video takes time to record, convert, and export, so we want an initial notification with the alert image, that then gets updated with the relevant recorded clip when it's ready to view.

My initial plan was to have a single automation handle any incoming MQTT triggers, send the alert notification with the image, then wait until it gets a confirmation the video is ready and udpate the notification with a clickAction. If we deliver the video path first and it doesn't exist, a user tapping on the notification will get an error and the notification will disappear, so that's why I'm doing the update notification in the first place.

elfin forge
#

Your second trigger is the file being generated in a separate automation, so how is it going to get the information from the other automation/trigger?

dreamy seal
#

I have a second MQTT trigger that fires with the same filename alerting the video is ready to view

elfin forge
#

But you’re using the image being generated as your trigger

dreamy seal
elfin forge
#

The only context it has is: this image was created, this is the filename.

dreamy seal
#

So I have several variables. The "file_name" variable is the name of the video clip generated, and the "image_path" is the name of the image file generated.

#

They are stupid names, I know. Blame the developers of blue iris for that

elfin forge
#

The wait for trigger won’t work if the filename that’s created is not matching, and it doesn’t continuously wait for the right file

#

Hence why I’m saying a second automation

dreamy seal
#

Right, and this can have several triggers going at once. A dog could run by 3 cameras in the span of 5 seconds

elfin forge
#

Hence why I’m saying: you need to pass that info via the filename

dreamy seal
#

I have the filename. Where do I need to pass it?

#

How do I pass it?

elfin forge
#

No, you have a condition that you need to check if it’s the correct file

#

I feel like you have a miss understanding on how the automation works with parallel

#

It will not have the information from the other trigger

#

It will not reside in the automation

dreamy seal
#

So you are telling me there is no way to pass a payload to a second automation?

elfin forge
#

Because it’s in a separate instance of the automation

elfin forge
dreamy seal
#
  • variables:
    file_name: "{{ trigger.payload_json.file }}"

this is what I need to pass

#

That is the name of the file I'm waiting to see in the folder

#

LIne 11

elfin forge
#

Do you really need to pass that? Or can you create a separate automation that just fires based on the filename being parsed

dreamy seal
#

You're telling me I do

elfin forge
#

Like, you want all this sophistication but there are easier ways to do this. Embedding the info into the filename, then parsing it out and acting on the parsed info is much easier

#

E.g does the filename have xyz format? If yes, notify. Else do nothing.

#

And you can parse the object, camera, etc out of the filename

#

I have to run, dentist

#

Bb later

dreamy seal
#

I can't control the filename. It comes in as it comes in from the software.

#

Sure

dreamy seal
#

Most of this stems from the fact that home assistant absolutely refuses to serve a clickaction containing a network video path, even though that path is accessible to home assistant itself. Almost all of the complexity revolves around moving the file around and adjusting the file permissions so it can serve it over /www

#

If it would just serve the video where it gets placed, I'd be able to send a single notification and be done with it.

#

I mean, if I want to simplify it I can just rip out the folder watcher entirely and just rely on the shell command to return with exit code 0. Then assume the folder is up to date. I will try that.

elfin forge
#

You control the name of the copied file, it’s in your script

#

All I’m saying is add to that filename, give me a second. I’ll write the automation.

#

Just got home, so it’s gonna be a few

dreamy seal
#

Yeah I'm sorry I'm just really not following you. I'm sure it'll make sense when I see it, but I just don't have the knowledge to understand how to execute on the concept.

#

In the meantime I was working on removing the folder watcher entirely. This is (as far as I can tell) a 100% working script, just with a 5 second hardcoded delay after the copy command (which tries to copy 5 times with a 1 second delay between tries) and a choose to cancel the update notification if the shell command returns any error:

https://dpaste.org/KGYOF

elfin forge
dreamy seal
# elfin forge https://dpaste.org/2EY6U

Double quote to single quote here on "image."? Doesn't seem to like it as is:

value_template: "{{ trigger.to_state.attributes.file.startswith("image.") }}"

elfin forge
#

change inside quotes to single, was a mistake

dreamy seal
#

OK just double checking I didn't paste it wrong.

#

So...I'm trying to understand this. Is it basically two automations in one id?

elfin forge
#

yep

dreamy seal
#

I didn't know that was possible. cool

elfin forge
#

2 automations in 1

#

well, I've been saying to use trigger id and choose from the beginning 🙂

dreamy seal
#

I was trying to figure out how the trigger ID gets passed to a second automation, not a second instance of the existing automation.

elfin forge
#

the id is just an id for the trigger that occured

dreamy seal
#

Although I suppose you could do it both ways?

elfin forge
#

yep

dreamy seal
#

Right, yeah. so I misunderstood the concept of the trigger ID. I thought it acted like a payload or something. But the actual trigger here is the folder watcher state change

elfin forge
#

Yep

dreamy seal
#

This makes a lot more sense now that I can see an example of it. Thank you!

elfin forge
#

np

dreamy seal
#

I feel like I need a full college course or something to understand this platform lol.

So, I'm stepping through and I do see pretty much everything here. We get the payload data, send the initial notification, wait for the recording trigger, and then send the copy command with the updated file naming pattern you're using for the second trigger. The entire action is parallel so if the copy command completes very quickly, it isn't going to fall out of sequence, I think. That was a problem before. Then when the copy command finishes and the new trigger fires, the choose will select the updated notification to work with. As far as I can tell, this should work with multiple triggers running in parallel with no crosstalk.

Let me give this a try and see. I think it'll work as you say out of the box.

#

Well, not quite out of the box lol

elfin forge
#

I'm sure I made some other clerical error

dreamy seal
#

It might be something I broke too, hold on...

#

I've broken a lot of things today

#

So, I think it's with the string split. Second notif comes in with missing/incomplete strings

#

It may actually be failing because some of the payload strings are null when I manually trigger the camera to record.

#

Let me try with some dummy payload data

#

Ah, no it does need some help.

Screenshot_20240917_162824_Nova_Launcher.jpg
#

(I'm trying to step through it myself before I ask you for more help, give me a few minutes)

#

There's a rather interesting interpretation of the string split here. I'm not sure where it's getting that from?

        {
          "path": "action/0/choose/1/sequence/0",
          "timestamp": "2024-09-17T20:27:42.805920+00:00",
          "changed_variables": {
            "file": "image.TEST OBJECT.0.Doorbell.20240917_162726.mp4",
            "items": "['image', 'TEST OBJECT', '0', 'Doorbell', '20240917_162726', 'mp4']\"",
            "object": "'",
            "confidence": "i",
            "cam": "m",
            "timestamp": "a",
            "ext": "g"
          }
        }
      ],```
#

String split resolved, stray quote character at the file.split line

#

Remaining: notification doesnt update, it sends a new one. Image not in updated notification, video path leads to a 403: forbidden error.

#

Can confirm the files are showing up in www folder:

image.png
#

Oh, we're not passing the image filename along as part of this, so the {{image_path}} variable is blank. That would explain no image

#

Updating code to send image path as prefix instead of the word "image", adding "image_path" as items[0]

#

This one seems tricky. Doesn't want to accept the image as a prefix on the file path

#

ah ok. So here's the problem. The image and video files are two different formats. There's a squence number in the images that seem to be random:

image.png
#

So I tried updating the file pattern to have the full image path as a prefix, but that doesn't work with the split command because the image path itself as the same separator . in it.

#

I'm going to assume the pattern remains the same in terms of the number of periods in the image filename. In that case, I think it just needs to be updated to concatenate back the first 4 or so parts from the split as one image filename.

#

Can I do an {{ items[1-4] }} or do I need ot have four individual item definitions?

# 
            - variables:
                file: "{{ trigger.to_state.attributes.file }}"
                items: >
                  {# update.<image>.<object>.<confidence>.<cam>.<datetime>.<ext> #}
                  {{ file.split('.') }}
                image: "{{ items[1-4]}}"
                object: "{{ items[5] }}"
                confidence: "{{ items[6] }}"
                cam: "{{ items[7] }}"
                timestamp: "{{ items[8] }}"
                ext: "{{ items[9] }}"```
#

Actually can't do that anyway, need to add the periods back in for image variable. Also it's 5 not 4. I think it's close now...

#

OK, very close, now properly parses notification text and updates the notification rather than send a new one. Also updated the image path to exclude quotes and it now works on the update notify as well.

Updated code:

https://dpaste.org/2Fepx

This is where I might need more help, as the video on clickAction now simply refuses to work. I wonder if the filename is now too long and it's breaking something in the app? That's the only explanation I can think of. File is there

image.png
#

Ah, no it's the space I had in the object. So...can't pass anything in the filename with a space in it, the clickAction parser will break. Fun

#

I'll have to go back and make sure there's never a space in the memos or anything...

#

But other than that I think it's fully working now! Thanks for the assistance, I greatly appreciate you being so patient with me here.

#

It can be very hard to follow along from a newcomer's perspective!

elfin forge
#

Sorry, was watching the live stream thingy

#

well, you got it working, that's all that matterse

#

So why did you add the filename of the image? Did you need that for something?

dreamy seal
#

Yes, for the notification. If you don't include it again it strips the image from the notification and then you can't see the detected object on screen when you're reviewing activity.

#

Just because the video is ready doesn't necessarily mean we want to watch every single one to see what was detected! Sometimes the image is enough to know it's not important.

#

In fact, that's probably the case for a good 75% of the alerts

#

Before we were using the stored variable from the initial notification in both places, but when it was changed to run in parallel with the trigger IDs that information was no longer available to use in the update step.

elfin forge
#

You can separate the filenames with an _ and split it that way

#

then you'd only need to split the 2nd half

dreamy seal
#

Ahh, ok. Makes sense, I might go back and do that later. For now I have it active and I'm making sure it runs as expected for at least a few dozen alerts.

dreamy seal
#

So it apparently broke itself again. But this time I don't know if it's the automation or the companion app that's misbehaving. I left my phone alone for an hour or so and when I returned every single notification I received was missing the update with the video path. But if my screen is on and I'm using my phone, I receive the updates and they work fine

#

I'm going to go back in and add a ttl 0 and high priority to the notification, as I wonder if omitting them in the first notify step is causing them to be received out of order.

#

But if that's not it, then idk

elfin forge
dreamy seal
#

Not currently but I'm keeping an eye on it. Caught it after a reboot

dreamy seal
#

It doesn't seem to be returning. I'm getting the occasional issue due to file locking, but that's a bug with blue iris not releasing the file locks in a timely fashion. As far as the notifications go it seems it was fixed by forcing the initial ones to ttl 0 and priority high.

dreamy seal
#

Hmm, so there is one weird issue going on here. It seems the MQTT wait_for_trigger step is not actually waiting for the value_template to evaluate as true. It is firing off immediately the moment it receives a topic. Why is this and how can I prevent it from happening? It's like it's just flat out ignoring the value template in its entirety. Here's a recent trace, you can see on line 100 the "file_name" variable is initialized to "Doorbell.20240920_104420.mp4" but the wait_for_trigger steps on lines 154-190 show it triggering on a payload "file" value of "Doorbell.20240920_104227.mp4", which are two different timestamps. In this case it's pulling from an older payload that took a while to post due to a copy permission delay, and the camera just so happened to trigger again right before that lock was released, causing a second instance of the automation to kick off.

https://dpaste.org/LEmSk

dreamy seal
#

i can make this reliably trigger improperly by setting any retain message on MQTT as well.

image.png
elfin forge
#

Hmm, so there is one weird issue going on here. It seems the MQTT wait_for_trigger step is not actually waiting for the value_template to evaluate as true. It is firing off immediately the moment it receives a topic.
Yes, this is how it works

#

which is why this shouldn't be a wait for trigger 😉

dreamy seal
#

What should it be?

elfin forge
#

it should be at the top, didn't I do that in my example?

dreamy seal
#

Don't think so. This was modified from your example but I didn't touch that part of the code except to extend the timeout

elfin forge
#

you'd have it at the top, with a condition and choose

dreamy seal
#

Well, there are two things going here. This should probably be a separate automation entirely I guess.

elfin forge
dreamy seal
#

We're waiting on a file lock to release in order to copy the file to a folder that home assistant can serve it from. Then we're waiting on that copy to finish before updating the notification

elfin forge
#

ah, I didn't mod that area IIRC

dreamy seal
#

But since the wait on file lock is triggered by an MQTT topic, it technically doesn't matter that it happens within the context of the automation, right? I could just have a separate automation on the MQTT topic to move any file once the topic triggers, and the file watcher/choose logic will take care of the context for notification purposes. At least I think that's how I understand it.

#

So basically I can move this entire bit to its own automation:

                - platform: mqtt
                  topic: "BI/recording_ready"
                  value_template: "{{ value_json.file == file_name }}"

            - service: shell_command.copy_to_www
              data:
                file_path: "/media/DS720/HomeAssistant/{{ file_name }}"
                file: "update.{{image}}.{{ object }}.{{ confidence }}.{{ file_name }}"
                response_value: err```

And if I understand this right, it shouldn't affect the functionality of the current automation. In fact, I don't even need a timeout then, it could take an hour for the recording to be ready since the trigger IDs take care of the context, not the flow of the automation 🤷
elfin forge
#

probably, but you'd need the other information

dreamy seal
#

oh, hmm...you're right.

#

That's why I had it in there in the first place then

#

I suppose if I really had to I could offload the variables I need into a json file named for the original filename, and extract them again later?

#

Unless there's a more elegant solution to this

#

Ah, or I could just include the original payload JSON in the recording_ready MQTT trigger and re-extract them.

elfin forge
#

If you can supply all the info in the payload then there's no reason for all the parsing along the way

#

you could even turn it into an entity with mqtt discovery and work off that

#

I have blue iris myself, but I let it handle all the notifications, works great

dreamy seal
#

Hmm, so the recording_ready mqtt topic is done by a custom python script that monitors the same BI/motion topic home assistant does, and uses a shadow copy method to get it off of the database drive and onto a shared network drive (I don't want my full NVR available to my home assistant OS). So technically it has all the same information HA does when it triggers the initial notification.

#

I could just have it rename the file when it does the copy

elfin forge
#

So, just a side question, is there a reason you're not using the blue iris app?

dreamy seal
#

Yes because the web UI sucks and their app requires a $10 per user license. Also I don't want to expose the blue iris software to the internet. I have it restricted to specific LAN IP addresses. With home assistant I can control exactly what camera feed clips are made available and for how long.

elfin forge
#

zerotier

#

it's a vpn, so it's not exposed

dreamy seal
#

I'd still rather not pay $70 for seven copies of the BI app 😉

elfin forge
#

fair enough

dreamy seal
#

And forget about trying to get my family to understand the concept of a VPN...lol

elfin forge
#

I never set up the MQTT side of BI, if you can control the information that goes in the topics, you can simply just create a single topic with a dictionary of information

#

even pointers to other topics

elfin forge
#

literally

#

I had the same reservations until I used it

#

and it just works

#

the vpn just silently runs in the background, auto connects, auto uses it

dreamy seal
#

Eh, zerotier is also not free, so I'd rather spend 30 minutes rewriting my python script.

elfin forge
#

it's 100% free

#

anyways, I'm trying to move apst that because you have no interest in it 😉

dreamy seal
#

Sure. Anyway, here's what I'm thinking: Remove MQTT recording_ready topic from the main automation along with the shell copy command. Create a new folder watcher for the network share where the file goes, attach shell copy to that automation with the .mp4 mask and allow it to copy anything delivered to that folder. My python script that generates that topic will rename the file to the same pattern as the notify script is expecting, so it'll just transparently process once it hits the destination.

#

I should have this modified in as little as 15 minutes...let me try it.

dreamy seal
#

This looks to be working:

    worker_name = threading.current_thread().name  # Get the worker's thread name
    try:
        data = json.loads(payload)
        file_name = data.get('file')
        cam = data.get('cam')
        memo = data.get('memo')
        obj = data.get('object')
        image = data.get('alert_path')
        
        # Extract the confidence value, removing the percentage symbol
        confidence_match = re.findall(r'(\d+)%', memo)
        confidence = confidence_match[0] if confidence_match else '0'
        
        timestamp = file_name.split('_')[-1].split('.')[0]  # Extract timestamp from the filename
        
        # Rename the file to match the format from Home Assistant
        new_file_name = f"update.{image}.{obj}.{confidence}.{cam}.{timestamp}{FILE_EXTENSION}"
        
        if file_name:
            file_path = os.path.join(RECORDINGS_DIR, file_name)
            logging.info(f'{worker_name} assigned to monitor file: {file_name}')
            monitor_file(file_path, new_file_name)
        else:
            logging.error(f'{worker_name}: No file name found in payload')
    except json.JSONDecodeError:
        logging.error(f'{worker_name}: Failed to decode JSON payload')```

Automation:
  • id: "1247824598"
    alias: Copy file to www
    description: "Copies recorded files from NAS to local www folder for serving via notification"
    mode: parallel
    trigger:
    platform: mqtt
    topic: BI/recording_ready
    action:
    service: shell_command.copy_to_www
    data:
    file_path: "/media/DS720/HomeAssistant/{{ trigger.payload_json.file }}"
    file: "{{ trigger.payload_json.file}}"
    response_value: "err"
#

And no sooner than the moment I enabled it, I got three simultaneous triggers from my cameras. All three handled perfectly. 😄

#

Yep, definitely working in parallel mode now. Thank you once again for your guidance! I wouldn't have thought to split that automation if you hadn't helped me through the initial setup here.

elfin forge
#

Are you using pyscript?

#

or something completely separate?

dreamy seal
#

completely separate, python flask app running on the same server the BI software is on. Keeps things more secure if the HA os doesn't have full access to my NVR system, and that NVR is completely walled off from the LAN save for the one network share.