#Pulling 'Last Active' from Z-Wave JS into HA

1 messages · Page 1 of 1 (latest)

proven fable
#

I'm trying to find a way to get the 'Last Active' metric from Z-Wave JS into Home Assistant so I can create automations to alert me of missing battery powered devices. I found this post, but I haven't ever done anything with MQTT before so I'm not 100% sure how to start (or if it's even still possible with current versions)
https://community.home-assistant.io/t/zwave-js-missing-last-received-attribute/307501/14

I did try toggling the "Disable MQTT Gateway" to 'off' and adding an MQTT integration using both hosts 'a0d7b954-mqtt' and 'a0d7b954-zwavejs2mqtt' but couldn't get either of them to add.

any tips/pointers to get me going in the right direction? I'm particularly interested in 'Last Active' and not 'Last Seen' because 'Last Seen' is subject to restarts of ZW, HA, the host, firmware updates of other devices, etc - where 'Last Active' is literally the last time the Z-Wave controller saw activity from that Node (much more reliable)

dull mirage
#

It's already there

#

That comes from the same stats ZUI uses

proven fable
#

"Last Seen" works fine until you need to restart HA, restart Z-Wave JS UI, or do many other tasks. it then goes to an "unknown" state until that device is seen again

dull mirage
#

that's a bug then

proven fable
#

(apologies, seems I don't have permissions to post images)

dull mirage
#

I'm not seeing the same behavior though, if I restart HA the sensor doesn't go unknown

#

Is it specific to battery?

proven fable
#

it's unkonwn until the next time the device is active, which is sort of fine for devices that haven't died yet - but for ones that are already dead they won;t come back from "unknown"

proven fable
dull mirage
#

I'm not seeing the same behavior, at least with an HA restart.

#

Does your device diagnostic file, or ZUI node debug, have a "last seen" value?

#

statistics.lastSeen

proven fable
dull mirage
#

As far as I can tell, the lastSeen entity value is populate when the integration loads, and updated when stats are received.

#

What version of ZUI are you using?

proven fable
#

zwave-js-ui: 9.9.0
zwave-js: 12.4.4
home id: 3457030142
home hex: 0xce0e17fe

#

Core 2024.2.1
Supervisor 2024.02.0
Operating System 11.5
Frontend 20240207.1

#

I did a restart of HA and as you described, the 'last seen' did persist. perhaps that got fixed in a recent dot release. however, the 'unknown' remains - and of course it's my one dead device (that I'm leaving dead for testing at the moment) 🙂

dull mirage
#

I've got at least one device like that

#

This is a problem then in the driver, not HA

proven fable
#

the hardware sensor driver?

dull mirage
#

No, Z-Wave JS

proven fable
#

oh sorry - that makes sense

#

Z-Wave JS is tracking it perfectly -- it shows the last time that device was seen and that timestamp has persisted restarts and reboots. seems that it's not reliably passing that information for dead devices back to HA

dull mirage
#

Yeah, I see the same. Well, driver or zwave-js-server.

#

Server just passes through the data though.

#

I'm not quite sure what's different with ZUI

proven fable
#

for that angle - whats the best place to ask?

#

I am still interested to see if I can expose the raw value using MQTT, even if only for educational purposes

dull mirage
#

there's an mqtt topic with the value

#

<mqtt_prefix>/<?node_location>/<node_name>/lastActive

proven fable
#

I have zero experience with MQTT, much less getting the MQTT server going within ZUI and having HA connect to it. i tried toggling MQTT 'on' in ZUI and then adding an integration using the hosts mentioned above, but no luck. any tips on how to get HA to talk to the ZUI MQTT server?

#

this is HAOS supervised on an rPi 3b+

dull mirage
#

you run an broker external to ZUI, ZUI is not an mqtt server (broker)

#

If you're using HAOS there's a mosquitto add-on

proven fable
#

ohh ok, that's probably what I was missing. so high level steps:

  1. install Mosquitto broker add-on
  2. in ZUI toggle "Dissable MQTT Gateway' to off (so it is enabled)
  3. Add an MQTT integration to HA
  4. pull in the topic you mention above
dull mirage
#

yeah. step 4 is an MQTT template sensor

#

Just curious, does this driver code work for your device (it does mine)?

const node = driver.controller.nodes.get(9);
console.log(node.lastSeen?.getTime() || "no lastSeen")
#

plugin 9 with your node id

proven fable
#

where would I type that?

dull mirage
#

Driver function in ZUI Advanced menu

#

Click "Run" and it will be viewable in the ZUI debug log window, which you would need to start first. Or in the add-on console logs.

#

result:

2024-02-26 16:23:50.194 INFO Z-WAVE: Calling api driverFunction with args: [
  '// replace `undefined` with the id of the node you want to clone\n' +
    'const node = driver.controller.nodes.get(9);\n' +
    'console.log(node.lastSeen?.getTime() || "no lastSeen")',
  [length]: 1
]
1707116695329
#

Ah, I think I know why

#

The key is that our devices don't have lastSeen in the node statistics

proven fable
#

lost me here: Driver function in ZUI Advanced menu

I have the debug menu open in a separate window to monitor that. in my main window I am at the control panel.

#

I found it 🙂

#

hamburger!

dull mirage
#

🍔 menu -> 🪄

#

ZUI requests the initial value in a different way than HA

proven fable
#

GOOD NODE:

  'const node = driver.controller.nodes.get(9);\n' +
    'console.log(node.lastSeen?.getTime() | | "no lastSeen")',
  [length]: 1
]
1708977163116
2024-02-26 19:29:30.065 INFO Z-WAVE: Success zwave api call driverFunction undefined```

DEAD NODE:
```2024-02-26 19:30:02.135 INFO Z-WAVE: Calling api driverFunction with args: [
  'const node = driver.controller.nodes.get(17);\n' +
    'console.log(node.lastSeen?.getTime() | | "no lastSeen")',
  [length]: 1
]
1707241647200
2024-02-26 19:30:02.146 INFO Z-WAVE: Success zwave api call driverFunction undefined```
dull mirage
#

Yeah, so for some reason the value is set in the node properties, but the node statistics don't have it. Two different places.

#

They are supposed to be in-sync

dull mirage
#

Then Al can tell us whether the statistics and node property should be in-sync.

proven fable
#

OK if I link to this thread?

dull mirage
#

Yeah, although I think the answer is going to be that HA is doing it wrong. It doesn't look like stats are persisted. So maybe HA is the place. 🤷

proven fable
#

seems like Last Seen is relatively new to HA so that's defintely possible

dull mirage
#

Yeah, it was recently added.

#

I wonder if restarting Z-Wave JS UI will cause all HA battery ones to be Unknown

#

or ZUI Restart -> HA Restart

proven fable
#

I just restarted HA and the 'last seen' persisted

#

and I did a "reload integration" on z-wave js

dull mirage
#

Restart HA / reload integration, same thing

proven fable
dull mirage
#

I think the node stats are kept in memory

#

Yeah, restarting ZUI loses the battery nodes

proven fable
#

loses them in HA

dull mirage
#

Anyways, good find. Thanks.

proven fable
#

I did some learning tonight -- followed your guidance, got the MQTT broker running and ZUI pushing values to it! that's the good news.

the bad news is -- ZUI is only pushing "lastActive" values for ..... active sensors. after a restart of ZUI, it will go and re-push just about everything it knows about all of the sensors live or otherwise -- EXCEPT the lastActive value. that only seems to push for sensors it's seen since startup (or very recently before)

https://imgur.com/a/4Ib3p7Z
in this pic the HVAC Water Sensor is battery powered and only checks in every 12 hours, and hasn't checked in since ZUI started a few min ago.
the Smoke Detector on the other hand is mains powered and checks in very frequently, and thus has lastActive.

I did check another battery powered device that happened to check in since ZUI started and it is now reporting LastActive via MQTT:
https://imgur.com/a/0kGxEIr
The Kitchen Fridge Temp Sensor reports temp & humidity frequently, and the front door sensor just happened to come up to its 12hr reporting intereval
The Stairway Door hasn't yet hit it's 12 hour reporting interval so ... no lastActive 😦

apologies if this is restating what you said another way above, but it helped me understand some of what's happening - and frustrated me that even the MQTT solution doesn't seem to get me what I want - the persistent lastActive value.

in the ZUI DEBUG log I do see all the MQTT publish reports, and I do see they are flagged as "retain=true" so it will be interesting to see if the MQTT broker will persist all the lastActive values through a restart once it has something to report for all the sensors?

DEBUG MQTT: Publishing to zwave/Dining_Room/Beer_Fridge_Temp_Sensor/lastActive: { time: 1709011348188, value: 1709011347936 } with options { qos: 1, retain: true }

dull mirage
#

that's what retain is for.

#

If it's not publishing anything on startup, maybe a bug, not sure. You can ask in their Discord (see pinned posts), or their GitHub project.

#

looking at the code, it may just be publishing on statistics updates, so same state as HA it seems

proven fable
#

yea I think you're right. all of my sensors now have "Last Active" published to MQTT except for the 1 dead sensor, which just gets me right back to where I am with "Last Seen" already in Home Assistant. I do think it's strange - even if intended - that ZUI is storing this timestamp for the dead sensor in some persistent way, but not pushing it to HA or MQTT when it pushes (almost) everything else on startup:
console.log(node.lastSeen?.getTime() | | "no lastSeen")', [length]: 1] **1707241647200**

dull mirage
#

just seems like an oversight is all

dull mirage
#

Have you checked the /nodeinfo topic for lastActive?

proven fable
#

sorry - not sure what that means. is that something I look at in the MQTT browser?

dull mirage
#

No clue, I just see there's a topic with that name.

#

there's a "publishNodeDetails" config option

#

Under Gateway:

#

For that I think you'd have to add two separate "last active" and "last seen" mqtt sensors, and then combine them in a single template sensor, returning the most recent time.

proven fable
#

nice! enabling Publish node details does in fact send the nodeinfo via MQTT, and it does include the accurate lastActive!

#

and a little ChatGPT for the win!

      state_topic: "zwave/Dining_Room/Slider_Door_Sensor/nodeinfo"
      device_class: "timestamp"
      value_template: "{{ (value_json.lastActive / 1000) | timestamp_local }}"```
#

and now the HA sensor is accurate:

February 6, 2024 at 12:47 PM```
dull mirage
#

it updates?

proven fable
#

after enabling "Publish node details" I changed the sensor YAML above from zwave/Dining_Room/Slider_Door_Sensor/lastActive to nodeinfo, chanced the value_template from value_json.time to value_json.lastActive, did a YAML reload, and it updated automagically

#

hmm

dull mirage
#

hmm, not sure how that could be, guess I'm missing something.

proven fable
#

yea it seems nodeinfo only gets published.... maybe on restart?

dull mirage
#

only for a handful of node events

proven fable
#

I switched one of my mains sensors over to use nodefino, and it is correct (or was) - but when I ping the device... ZUI updates lastActive correctly but it's not publishing nodefinfo with the updated lastActive

#

:/

dull mirage
#

node ready, node sleep / awake / alive / dead, etc.

proven fable
#

rather than setting up MQTT, the lastSeen and lastActive sensors, and a template to combine both --- would it be feasible to just create a RESTful sensor to poll value direct from the ZUI websocket?

dull mirage
#

nope

#

api requires a few messages to establish a connection, don't see how you'd do that in a restful sensor, maybe a command line sensor, assuming you can install something like websocat.

#

or write a custom python client

#

I think it can just be fixed in HA (or upstream lib)

proven fable
#

yea I still plan to follow through with the discussion (and bug if appropriate) - just looking to set something up in the interim, even if just for educational purposes. trying to learn more about REST/API/MQTT/ZUI/etc

dull mirage
#

not impossible, but not easy either.

$ cat << EOF | websocat -B 10485760 --text - $ZJS_HOST | tail -n1 | jq -r '.result.state.nodes[].lastSeen'
{ "command": "initialize", "messageId": "init", "schemaVersion": 34 }
{ "command": "start_listening", "messageId": "listening" }
EOF
null
2024-02-27T16:13:42.526Z
2024-02-05T07:04:55.329Z
2024-02-27T19:22:27.942Z
2024-02-27T19:05:34.322Z
2024-02-27T19:30:34.519Z
2024-02-27T19:21:45.194Z
2024-02-27T14:34:26.216Z
2024-02-27T01:16:02.044Z
2024-02-27T02:19:15.465Z
2024-02-27T19:20:17.651Z
2024-02-27T14:26:42.194Z
2024-02-27T07:52:32.234Z
2024-02-27T01:16:02.383Z
2024-02-27T04:19:03.560Z
2024-02-27T19:20:22.792Z
2024-02-27T16:51:04.503Z
2024-02-27T00:38:28.525Z
2024-02-27T16:56:04.947Z
2024-02-26T21:46:58.800Z
proven fable
#

thanks - yea the combined sensors is probably the lowest effort path for now while we await a code fix.
I have the helper combining states but it's reporting Unknown, I think because the state_class of the template is measurement and instead needs to be timestamp. I'm guessing I have to do that via YAML, there's no way to specify the state_class of a helper entity thru the UI (yet)

dull mirage
#

it would be device class

#

Timestamp

proven fable
dull mirage
#

I don't know what that helper does

#

does it work with timestamps? no idea

#

I understand a template though.

proven fable
#

my understanding is that helper "combines the state of several sensors" -- and if you use "most recently updated" it should report whichever of teh entites in the list is most recent

#

might have to do it the old fashion yaml way

dull mirage
#

which platform would that be then?

proven fable
#

not sure

#

trying to embrace doing things via the UI as much as possible

dull mirage
#

You can write template helpers in the UI

proven fable
#

bingo

#

for some reason i thought the combine helper was built to do the same thing without having to fiddle with hand building a template, but this works so i'll take it!

{% set timestamp1 = states('sensor.dining_motion_last_active_info') %}
{% set timestamp2 = states('sensor.dining_motion_last_active_time') %}
{% if timestamp1 > timestamp2 %}
  {{ timestamp1 }}
{% else %}
  {{ timestamp2 }}
{% endif %}```
dull mirage
#

Yeah, I think it only works on numbers. You can check your HA Logs , see if you see "Unable to store state. Only numerical states are supported"

#
{{ (timestamp1, timestamp2) | max }}
proven fable
#

I blame ChatGPT for that one 😉

dull mirage
#

technically correct

#

neither quite handle invalid states

proven fable
#

that's my next test -- I have to see what it does with my dead sensor since one of them is reporting 'Unknown'

dull mirage
#

can use has_value('sensor.my_sensor') and more complicated conditional like your original one

dull mirage
#

Guess I'd do this:

{% set ts_info = states("sensor.dining_motion_last_active_info") %}
{% set ts_time = states("sensor.dining_motion_last_active_time") %}
{% if has_value("sensor.dining_motion_last_active_info") and has_value("sensor.dining_motion_last_active_time") %}
  {{ max(ts_info, ts_time) }}
{% elif has_value("sensor.dining_motion_last_active_info") %}
  {{ ts_info }}
{% else %}
  {{ ts_time }}
{% endif %}
#

You only need the /nodeinfo topic, you can use the HA "last seen" entity as the second one

proven fable
#

oh yea good call -- that'd cut the number of mqtt sensors in half. appreciate that

proven fable
#

thanks again for all the help - got a quick/dirty dashboard and automation going that will hold me over until last_seen is sorted out
https://imgur.com/a/wowJgWN

proven fable
#

after updating HA Core Addon to 0.5.0, I noticed that the lastActive value of the nodeInfo MQTT topic is not updating any more - but only for my two mains powered sensors. all of my battery powered sensors continue to work as previous:
https://imgur.com/a/fyYn7Hy

dull mirage
#

Please tell me you are not running the core add-on

proven fable
#

oddly, there is a separate lastActive value now outside of the nodeInfo blob - can't recall if that was there before

proven fable
#

hmmm... I do most of my Z-wave management in Z-Wave JS (ZUI?) in the sidebar. v9.10.2. I built this instance on rPI in Jan and I think the steps looked a little different then - I still have the Settings > Devices & services > Z-Wave integration

dull mirage
#

integration != add-on

#

you should confirm which add-on the integration is talking to (URL). you should not be using the core add-on.

proven fable
#

ok great - that's what I thought but wanted to be srue

#

Z-Wave JS UI is running. Z-Wave JS is not

dull mirage
#

but you just said you update the core add-on to 0.5.0

proven fable
#

it looks like it's installed but not running

dull mirage
#

phew

proven fable
#

🤣

dull mirage
#

you don't need it installed, as long as your integration is using URL ws://a0d7b954-zwavejs2mqtt:3000

proven fable
#

ws://a0d7b954-zwavejs2mqtt:3000 👌

#

good to know - I will uninstall the defunct add-on

#

DONE.

#

i'm going to do a full system reboot, clear out the cob webs, give it a day or so ... see if the lastActive is still misbehaving

dull mirage
#

has the device actually been active? I have several mains-powered devices that can go days w/o activity.

proven fable
#

oh my yes ... it's a motion sensor in the middle of my house. it's by far my chattiest sensor

#

the other one is a zooz DC sensor for my smoke detector

#

I've still got that MQTT and lastSeen template and automation that you helped me setup (thanks!) so nothing is technically "broken" with it not updating ... but if lastSeen were to go stale it wouldn't fall back on lastActive via MQTT

dull mirage
#

well, hopefully the HA sensor is fixed next release

proven fable
proven fable
#

🤩

dull mirage
#

but shouldn't your mains-powered ones be working in HA then?

proven fable
#

they are - all the lastSeen values are working normally at the moment so no "missing device" alarms are tripping. but if lastSeen were to go wonky, we had set it up to fall back to the MQTT lastActive

#

since it looks like HA lastSeen may be fixed soon ... I can hopefully junk this whole MQTT setup and go with the built-in lastSeen

#

I was sifting thru changelogs to see if something changed with lastActive via MQTT but I didn't see anything

proven fable
#

i've been watching the pull request above and see it's been merged, and i've seen a few releases of ZUI, but my github-fu is not as strong as it should be, I can't tell if the PR is included in a release version yet, or if it's still queued up. any idea what version did/will include the fix?

dull mirage
#

Should be in 2024.5

#

That's a HA library, no relation to ZUI.

proven fable
#

interesting, thanks - I've got 2 instances running 2024.5.3 and I thought I saw the same issues but didn't dig too deep yet. I saw 2024.6 just dropped - i'll get upgraded to that once it's had time to marinate and then do some more testing to see if last_seen is still problematic.

related: somewhere not long after we requested the fix above, the MQTT reporting started misbehaving. the PR was in flight so I knew a fix was coming so I haven't looked into it much, but both of my powered sensors aren't reliably updating the MQTT last_active stat. battery sensors seem OK but I haven't done 100% checking on those either

dull mirage
#

I'm on 2024.5. I just restarted HA and all of the last_seen entities are times and none are "unknown".

proven fable
#

I just rebooted two of my instances, and the last_seen values reset back to 'unknown'. I have one instance at 12.3 / 2024.5.3 and another at 12.2 / 2024.5.4 -- both are resetting last_seen on a full host reboot. anything I might be missing, should I update the github issue to report these findings, or is a full reboot not covered by the scope of the PR above?
https://imgur.com/a/DgxVIx2
left half of the image is my primary instance where we setup the MQTT last_active sensor as a backup to last_seen - right side is a secondary instance I have at another location fresh built on 12.2 where I didn't setup MQTT as it seemed that last_seen was working properly
EDIT: restarting only HA but not the host does not suffer the same issue - the last_seen values seem to persist

dull mirage
#

Yeah, restarting HA works, but restarting ZJS does not.

dull mirage
#

looks like the fix was only partially implemented

proven fable
#

Anything I should do to keep things moving toward a complete fix, or is that already in motion? I saw my original git question was closed as resolved