Matter-over-Thread OTA Updates "Target node did process the update file" | Home Assistant | Page 1

signal monolith Dec 5, 2024, 10:38 AM

#

I've trying to get my Matter devices to update via Home Assistant. Several dives into posts have yielded no helpful nuggets to my situation, mostly they seemed to involve issues that could be resolved by rebooting the Matter server and/or HA itself to clear some improperly cached stuff. I've given up on trying to solve it by myself, it's been months, and I would just like some help. 🙂

I have a handful of Matter-over-Thread devices from Eve and Nanoleaf. Home Assistant happily tells me there are updates; great! But they always fail with "Target node did not process the update file." This happens no matter which device I try to update, nor how recently I've rebooted. The logs -- as far as I can tell -- have nothing interesting in them until the failure message and then it says it found the same update again.

My network topology is flat, and since we're talking Thread, indeed is totally regulated by HA. Setup is generic aarch64 HAOS in a VM with a passthrough to SkyConnect flashed to only Thread. Thread network is small (18 devices), and devices in question are both powered (Nanoleaf) and battery (Eve). I have no other Border Routers. There are no Matter devices not attached via Thread. Any help would be greatly appreciated! Thanks! 🙂

RESOLUTION: Avoid having devices on Apple Thread networks to update properly (steps on redress not present here), and additionally make sure HA's network (Settings -> System -> Network) has access to IPv6 networking to perform Matter OTA Updates, regardless of if the device is connected over Thread or WiFi/Ethernet.

exotic umbra Dec 5, 2024, 12:18 PM

#

So, you do not have any Apple border routers present ? Because this issue is known if one or more apple border routers are present.

signal monolith Dec 5, 2024, 12:38 PM

#

Correct, no Apple Border Routers present. Not directly nor even multiadmin, my devices are all single admin.

exotic umbra Dec 5, 2024, 2:14 PM

#

hmmm, interesting - I wonder if it broke in our recent update of OTBR then

signal monolith Dec 5, 2024, 2:20 PM

#

Unlikely, these devices have wanted these updates since the functionality was added back in... ~~June? July?~~ August. Same error. I've seen forum posts that have resulted in successful updating of these devices in particular (esp. the Nanoleaf bulbs), so I suspect a "me" issue rather than something general.

graceful ferry Dec 5, 2024, 2:20 PM

#

Can you share the log output of the Matter Server add-on when doing an update attempt?

signal monolith Dec 5, 2024, 2:20 PM

#

02:06:26.890 (MainThread) INFO [matter_server.server.ota.provider] Update file '3.6.196_r8.matter' downloaded to '/config/updates/6'
02:06:26.891 (MainThread) INFO [matter_server.server.device_controller] <Node:6> Starting update using OTA Provider.
02:06:26.891 (MainThread) INFO [matter_server.server.ota.provider] Starting OTA Provider
02:06:26.893 (MainThread) INFO [matter_server.server.ota.provider] Commission and initialize OTA Provider
02:06:26.963 (Dummy-2) INFO [chip.ChipDeviceCtrl] Established secure session with Device
02:06:27.286 (Dummy-2) INFO [chip.ChipDeviceCtrl] Commissioning complete
02:06:27.286 (MainThread) INFO [matter_server.server.ota.provider] OTA Provider App commissioned with node id 990006.
02:06:27.682 (MainThread) INFO [matter_server.server.ota.provider] Waiting for target node update state change
02:06:27.789 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kIdle: 1> to <UpdateStateEnum.kQuerying: 2>
02:07:12.673 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kQuerying: 2> to <UpdateStateEnum.kIdle: 1>
02:07:12.673 (MainThread) INFO [matter_server.server.ota.provider] Cleaning up OTA provider
02:07:12.674 (MainThread) INFO [matter_server.server.ota.provider] Terminating OTA Provider
02:07:12.689 (MainThread) ERROR [matter_server.server.client_handler] [281473235354304] Error while handling: update_node (node 6): Target node did not process the update file
02:07:22.912 (MainThread) INFO [matter_server.server.device_controller] <Node:6> New software update found: 3.6.196 on UpdateSource.MAIN_NET_DCL (current 3.6.173).```

#

This is for a Nanoleaf bulb, I've elided the dates so it can fit. I can do an Eve Motion too if desired. These are from today (after 2024.12 update) with default logging.

graceful ferry Dec 5, 2024, 2:25 PM

#

I actually never had a chance to test the feature with Nanoleaf. The bulbs I've updated using their app before we added update capability. And when I bought additional ones, they were shipped with a too old firmware wwhich wasn't update capable (and their App only offered to go to the lastest) 🙈

#

So testing with Eve would be interesting, that is what I did most updates with so far.

#

That said, both should work. From the log it seems the Thread device cannot find the Matter Server essentially (the time, 45s, when going from Querying -> Idle indicates a mDNS resolve timeout).

signal monolith Dec 5, 2024, 2:32 PM

#

I have a mix of the current and immediately prior firmware on my Nanoleaf bulbs, got them in two waves and the first wave the latest wasn't out yet for the updates through the app. As for the second wave, added them no problem, but the even older firmware was buggy (something something timeout limits) and so I had to update via the app prior to HA supporting that just to get stuff to connect consistently. Likewise, I have a mix of Eve firmwares, with only the older wave of Motions being not on the latest.

#

Would you like any different logging options before I try with one of the Eve Motions?

graceful ferry Dec 5, 2024, 2:33 PM

#

No same logging is fine

signal monolith Dec 5, 2024, 2:36 PM

#

06:34:41.795 (MainThread) INFO [matter_server.server.ota.provider] Update file 'eve_motion-3.2.1-matter-benc-enc-CF2853C8-8EA6-478C-87FA-3A5EAB5CFF30.bin' downloaded to '/config/updates/24'
06:34:41.796 (MainThread) INFO [matter_server.server.device_controller] <Node:24> Starting update using OTA Provider.
06:34:41.797 (MainThread) INFO [matter_server.server.ota.provider] Starting OTA Provider
06:34:41.800 (MainThread) INFO [matter_server.server.ota.provider] Commission and initialize OTA Provider
06:34:41.884 (Dummy-2) INFO [chip.ChipDeviceCtrl] Established secure session with Device
06:34:42.183 (Dummy-2) INFO [chip.ChipDeviceCtrl] Commissioning complete
06:34:42.183 (MainThread) INFO [matter_server.server.ota.provider] OTA Provider App commissioned with node id 990024.
06:34:43.738 (MainThread) INFO [matter_server.server.ota.provider] Waiting for target node update state change
06:34:43.956 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kIdle: 1> to <UpdateStateEnum.kQuerying: 2>
06:35:29.102 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kQuerying: 2> to <UpdateStateEnum.kIdle: 1>
06:35:29.102 (MainThread) INFO [matter_server.server.ota.provider] Cleaning up OTA provider
06:35:29.102 (MainThread) INFO [matter_server.server.ota.provider] Terminating OTA Provider
06:35:29.116 (MainThread) ERROR [matter_server.server.client_handler] [281473235354304] Error while handling: update_node (node 24): Target node did not process the update file
06:35:39.282 (MainThread) INFO [matter_server.server.device_controller] <Node:24> New software update found: 3.2.1 on UpdateSource.MAIN_NET_DCL (current 3.2.0).```

graceful ferry Dec 5, 2024, 2:48 PM

#

Hm, ok, let me try on my end. Which version of the Matter Server are you using?

signal monolith Dec 5, 2024, 2:48 PM

#

6.6.1

#

And, for thoroughness, OTBR 2.12.2

graceful ferry Dec 5, 2024, 3:13 PM

#

Bad news, I can reproduce it. But the good news is, I can reproduce it 😅 🙈

#

I'll have a look. I guess the latest OTBR bump broke it.

graceful ferry Dec 5, 2024, 3:30 PM

#

Hm, 2.11.1 also doesn't work here. Not sure what is happening, I need to dig deeper

#

Are you using beta or dev channel?

signal monolith Dec 5, 2024, 3:41 PM

#

Well, reproducibility is good. As far as I know, I'm on stable for everything.

#

The oldest combo I know it was broken for me is Matter 6.3.1 and OTBR 2.9.0, because that's what my backups from August have, when the OTA feature was added. It has had these updates available since HA could start checking for Matter OTA on stable releases, and it has always said this error for me.

silver grotto Dec 5, 2024, 8:11 PM

#

A bit of a side topic (hope you don't mind), but I have a Nanoleaf Essentials light bulb NL67 that I commissioned well over a year ago, and it is showing in HA having firmware version 3.5.10, (I noticed above mentioning 3.6.196 for what looks like an NL67 Nanoleaf device), but wanted to ask if there is an OTA firmware update available for this light bulb even though the Matter Server Web UI says there is not one available.

signal monolith Dec 6, 2024, 2:47 AM

#

Tommy, I'd assume it should, the update for 3.5.22 is on the OTA list, but it looks like they are missing some of the intermediary options for updates to help you reach the latest published OTA firmware (and Nanoleaf's 4.1.0 isn't actually published to the OTA site at all, yet). But I'm not sure how the inner workings of the OTA system work, so maybe since the latest isn't (supposedly) compatible with your bulb, there could be an issue there.

#

I looked up when HA added OTA support for Matter (it was August), and have updated my comments above to reflect that for future readers (nothing critical, no need to go back and re-read, everything important in this message too). This does mean that Matter 6.3.1 and OTBR 2.9.0 seem to be the stable versions when the OTA support was added.

graceful ferry Dec 6, 2024, 1:45 PM

#

silver grotto A bit of a side topic (hope you don't mind), but I have a Nanoleaf Essentials li...

The Nanoleaf firmware only added Matter firmware update capability later. I am not sure when exactly, but at least last time I bought Nanoleaf Essentials A10 bulbs they were pre-flashed with a firmware which was not Matter update capable. The Nanoleaf proprietary update worked though. But then I already had the latest version, so no updates from Matter side possible 🙈 Once the next Nanoleaf update is public on the DCL updates through Matter should then work.

#

Actually, I just realized that the Eve device I've tested was on the Apple Thread network. So that is why it did not work 🙈 Check the the Device page, expand the "Device info" section and double check if Network name is indeed the OTBR Thread network.

#

With the Eve connected to the OTBR for me updating started:

2024-12-06 14:45:43.475 (MainThread) INFO [matter_server.server.ota.provider] Starting OTA Provider
2024-12-06 14:45:43.478 (MainThread) INFO [matter_server.server.ota.provider] Commission and initialize OTA Provider
2024-12-06 14:45:44.279 (Dummy-2) INFO [chip.ChipDeviceCtrl] Established secure session with Device
...
2024-12-06 14:46:12.550 (Dummy-2) INFO [chip.ChipDeviceCtrl] Commissioning complete
2024-12-06 14:46:12.552 (MainThread) INFO [matter_server.server.ota.provider] OTA Provider App commissioned with node id 990014.
2024-12-06 14:46:12.927 (MainThread) INFO [matter_server.server.ota.provider] Waiting for target node update state change
2024-12-06 14:46:13.186 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kIdle: 1> to <UpdateStateEnum.kQuerying: 2>
2024-12-06 14:46:14.875 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kQuerying: 2> to <UpdateStateEnum.kDownloading: 4>
...

signal monolith Dec 6, 2024, 1:53 PM

#

Hah! Apple strikes again! Bad news for me, good news for you.
I have only one entry listed under my Thread network page "Home Assistant OpenThread Border Router"

graceful ferry Dec 6, 2024, 1:54 PM

#

Just double checked also with the latest OTBR, 2.12.2, works fine as well, download starts shortly after querying.

signal monolith Dec 6, 2024, 1:55 PM

#

I have exactly one powered-on Apple device in my home, the M1 Macbook Air on which the HA VM is running.

graceful ferry Dec 6, 2024, 1:56 PM

#

Hm, yeah I guess it is not that then. Just double check on the device info page if indeed the ha-thread... network is shown for your Eve device.

signal monolith Dec 6, 2024, 1:57 PM

#

Network name: ha-thread-####

graceful ferry Dec 6, 2024, 2:06 PM

#

@signal monolith there is probably a multicast issue on your network then. You write:

Setup is generic aarch64 HAOS in a VM with a passthrough to SkyConnect flashed to only Thread.
What VM are you using? Both, the Matter controlller and OTBR run on that same virtual machine? 🤔

signal monolith Dec 6, 2024, 2:10 PM

#

There probably is some sort of mDNS issue on my network, but I'm not sure what it is. I used a Bluetooth dongle for most of my commissioning rather than the Android app because it only worked 5 times and failed much more often than that. I expected it to be a non-issue once everything was local to the VM and the SkyConnect (and it worked flawlessly for commissioning at least).

#

I am running Add-Ons in the standard "click to install" way which I think spins up Docker stuff under the hood? But they are all the same VM, yeah. It's running using UTM.

graceful ferry Dec 6, 2024, 2:21 PM

#

They are on the same VM, and even on the same network namespace (we don't isolate the Matter server and the OTBR). I would expect that this stays local too, tbh...

#

So technically, Thread does not use mDNS, but a replacement protocol named SRP.

#

The way update works is that the Matter Server running in the Matter Server add-on (container) spinns up a OTA provider (C++ written application). This application announces itself on the network under the _matter._tcp service. Now, it does not directly announce itself on the Thread network (wpan0 interface). mDNS is considered to heavy for Thread. Instead, it "just" announces itself on the "network". The OTBR (otbr-agent user space program) has a SRP component which translates the picked up mDNS packets and forwards them onto the Thread network using the SRP protocol.

So it seems that in your case the otbr-agent does not pick up those messages 🤔 Now from what I understand the loopback (lo) interface is not Multicast capable. So the packets must travel through an actual network interface... as in, OTA provider multicasts on eth0, and otbr-agent listens on eth0. I guess that is where things fail.

#

Do you have maybe other add-ons running which might interfere with networking somehow? How many network interfaces do you have (e.g. is the correct one considered primary in ha network info?)

#

Btw, that OTA provider C++ application writes a log file which might be helpful in debugging this. Can you use the VS Code add-on to pull the log from the /addon_configs/core_matter_server directory? There should be a per node per update attempt log file. Just pull the last one from the Eve node.

signal monolith Dec 6, 2024, 2:54 PM

#

From the perspective of the VM, the outside network interface (enp0s1) is the only outbound network it has access to (though it has in the past had access to different interfaces). If you want info on the internal Docker network, I'm afraid I'll need directions for how to obtain that.

#

None of my add-ons should be messing with the network, but I'll go ahead and list the ones that are running (I can add the ones that aren't if there's some worry a config changed something). File Editor, Matter Server, Mosquitto broker (the official one, at 6.4.1), Music Assistant Server, OpenThread Border Router, openWakeWord, Piper, and Whisper.

graceful ferry Dec 6, 2024, 3:01 PM

#

signal monolith None of my add-ons _should_ be messing with the network, but I'll go ahead and l...

Yeah that sounds pretty standard set of add-on and non-invasive

#

I wonder if the aarch machine has some special kernel config which makes this behave differently 🤔 I see there is CONFIG_IP_ADVANCED_ROUTER not set, whereas it is set on other configs 🤔

signal monolith Dec 6, 2024, 3:06 PM

#

I have located the C++ OTA log file, I'm gonna scan through it myself and see if anything stands out.

signal monolith Dec 6, 2024, 3:23 PM

#

graceful ferry I wonder if the aarch machine has some special kernel config which makes this be...

I have no idea what that parameter is, but is there a way I can set it (what to?) and test if it helps?

graceful ferry Dec 6, 2024, 3:37 PM

#

signal monolith I have no idea what that parameter is, but is there a way I can set it (what to?...

That is a Linux kernel compile time configuration, you can't do anything to test this. It's just a wild guess on my end. I'll setup a aarch64 VM here, to see if that makes it reproducible on my end.

#

Hm, do you have IPv6 addresses assigned on the enp0s1 interface? 🤔

graceful ferry Dec 6, 2024, 5:13 PM

#

Tested now with everything setup on generic-aarch64, and update started as well:

2024-12-06 18:11:49.877 (MainThread) INFO [matter_server.server.ota.provider] Starting OTA Provider
2024-12-06 18:11:49.885 (MainThread) INFO [matter_server.server.ota.provider] Commission and initialize OTA Provider
2024-12-06 18:11:52.207 (Dummy-2) INFO [chip.ChipDeviceCtrl] Established secure session with Device
2024-12-06 18:11:54.460 (Dummy-2) INFO [chip.ChipDeviceCtrl] Commissioning complete
2024-12-06 18:11:54.467 (MainThread) INFO [matter_server.server.ota.provider] OTA Provider App commissioned with node id 990004.
2024-12-06 18:11:55.047 (MainThread) INFO [matter_server.server.ota.provider] Waiting for target node update state change
2024-12-06 18:11:55.228 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kIdle: 1> to <UpdateStateEnum.kQuerying: 2>
2024-12-06 18:11:56.965 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kQuerying: 2> to <UpdateStateEnum.kDownloading: 4>

#

@signal monolith with IPv6 set to disabled the same happens to me! So probably that's the case in your end? 🤔

2024-12-06 18:18:54.977 (MainThread) INFO [matter_server.server.ota.provider] Starting OTA Provider
2024-12-06 18:18:55.013 (MainThread) INFO [matter_server.server.ota.provider] Commission and initialize OTA Provider
2024-12-06 18:18:57.414 (Dummy-2) INFO [chip.ChipDeviceCtrl] Established secure session with Device
2024-12-06 18:18:59.038 (Dummy-2) INFO [chip.ChipDeviceCtrl] Commissioning complete
2024-12-06 18:18:59.044 (MainThread) INFO [matter_server.server.ota.provider] OTA Provider App commissioned with node id 990004.
2024-12-06 18:18:59.542 (MainThread) INFO [matter_server.server.ota.provider] Waiting for target node update state change
2024-12-06 18:18:59.716 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kIdle: 1> to <UpdateStateEnum.kQuerying: 2>
2024-12-06 18:19:44.797 (MainThread) INFO [matter_server.server.ota.provider] Update state changed from <UpdateStateEnum.kQuerying: 2> to <UpdateStateEnum.kIdle: 1>
2024-12-06 18:19:44.798 (MainThread) INFO [matter_server.server.ota.provider] Cleaning up OTA provider
2024-12-06 18:19:44.798 (MainThread) INFO [matter_server.server.ota.provider] Terminating OTA Provider
2024-12-06 18:19:44.881 (MainThread) ERROR [matter_server.server.client_handler] [281472897243552] Error while handling: update_node (node 4): Target node did not process the update file

signal monolith Dec 7, 2024, 2:09 AM

#

I have IPv6 disabled because it breaks HACS (the bridged network adapter on macOS apparently doesn't handle IPv6 correctly, it makes an address, but cannot reach Github over it). In theory though, this process shouldn't depend on that working correctly, so I shall try that and see what happens.

#

OMG IT'S WORKING

signal monolith Dec 7, 2024, 3:32 AM

#

Results confusing but promising thus far. Updated a Nanoleaf bulb, which then immediately claimed it needed the update again. Rebooted it and HA, still old version. Updated an Eve Motion, same thing. Updated the same Nanoleaf bulb again (maybe rebooting was a bad idea?), finishes update and it sticks. HUZZAH!

So the issue I was having initially was definitely HA cannot issue OTA Matter updates without IPv6 on the network interface, even if updating over Thread. My guess is this is a side effect of Matter running over Thread and WiFi/Ethernet and the update process at that point not caring which is needed for this particular update.

graceful ferry Dec 7, 2024, 8:33 AM

#

The Firmware version in the left "Device info" block unfortunately doesn't update currently, that is a bug I still need to look into 🙈 But the update entities firmware version should be updated..

graceful ferry Dec 7, 2024, 8:38 AM

#

signal monolith I have IPv6 disabled because it breaks HACS (the bridged network adapter on macO...

Uh, that sounds weird, a bridged network adapter should behave just like macOS itself. To me that would sound more like IPv6 not assigning correctly routable IPv6 addresses at all.

signal monolith Dec 7, 2024, 8:45 AM

#

I have now successfully updated all the Nanoleaf bulbs (when they did update, all the numbers seemed to update accordingly in the interface, but I rebooted a lot so I'm not 100% sure). And one of the Eve Motions.
Oddly, the "ritual" seemed to require the device to reboot before it could accept updates. Two of the bulbs were update (fail), reboot, update (accept), and the third was update (fail), update (fail), reboot, update (accept). I don't power off my devices too often, maybe they needed to reboot to clear some space/cache or something? IDK. The one Eve Motion I've updated just worked. The other two have rejected both before and after rebooting.
I think it's solved, but something is causing interruptions. Matter Server itself thinks the updates have happened because the device state changed back to Idle.

#

When I was looking into it, the rational I saw was basically "bridging WiFi hard" and technical networking stuff that I didn't understand. It does bridge correctly and provide IPv6 when I hook it up to Ethernet.

signal monolith Dec 7, 2024, 9:15 AM

#

Second Eve Motion updated. And I checked without rebooting HA: "Firmware: 3.2.1" (AKA the new version) listed just fine on the left panel on the device page once the device reconnected. Maybe it wouldn't have updated if I had the page open during the update (or maybe I misunderstood what you said didn't update).

signal monolith Dec 7, 2024, 10:00 PM

#

Final Eve Motion have relented. Not sure why it took so long, it kinda just kept stalling and tming out back to Idle. Multiple reboots. Finally left it off for about 7 hours, then it updated! That's all of them!

#Matter-over-Thread OTA Updates "Target node did process the update file"