All Matter devices are unavailable after a power outage | Home Assistant | Page 1

tranquil osprey Nov 27, 2024, 12:20 AM

#

I have a 26 Matter devices commissioned in the HA (25 lights and 1 switch).
Today we had a power outage due to an electrical accident when renovating one of the rooms.
When the HA came back online all Matter devices were unavailable - Matter server Web UI shows 0 nodes.
In the NanoLeaf app I can see that all light are still in the Thread network.

In the Matter server logs i noticed this:
2024-11-26 20:12:41.236 (MainThread) ERROR [chip.storage] Expecting ',' delimiter: line 29 column 111 (char 4111)
2024-11-26 20:12:41.236 (MainThread) CRITICAL [chip.storage] Could not load configuration from /data/chip.json - resetting configuration...
2024-11-26 20:12:41.236 (MainThread) WARNING [chip.storage] No valid SDK configuration present - clearing out configuration
2024-11-26 20:12:41.236 (MainThread) WARNING [chip.storage] No valid REPL configuration present - clearing out configuration

Is my configuration gone or can I restore it somehow?

bold kraken Nov 27, 2024, 12:24 AM

#

dang, it sounds like the disk on your home assistant machine has gotten corrupted.

#

your best bet might be to restore from a backup which includes the matter add-on

#

what sort of hardware do you have HA running on anyways?

tranquil osprey Nov 27, 2024, 12:44 AM

#

Thanks. I will try to restore it from a backup.
I am running it as a VM on the Linux machine.

#

I restored it from the latest backup and my configuration works again. Thank you for the hint.

bold kraken Nov 27, 2024, 12:53 AM

#

might be worth double-checking what disk cache mode is in use on the vm - make sure you haven't selected an unsafe option.

#

hmm, i just took a look at the home assistant matter server code, and it looks like it's not actually doing writes to the "chip.json" file in a way that's safe over power loss.

#

@umbral meadow is https://github.com/home-assistant-libs/python-matter-server/blob/main/matter_server/server/storage.py#L149 the relevant code? It looks like a power loss interruption can result in a partially written "chip.json" file - I would have expected that to write to a new (temp) file, fsync, then rename to replace the existing file.

bold kraken Nov 27, 2024, 3:53 PM

#

I've opened an issue on github to track this, and described what I think the issue is: https://github.com/home-assistant-libs/python-matter-server/issues/977

GitHub

Persistent data storage writes are not power-loss safe · Issue #977...

In this discord thread: https://discord.com/channels/330944238910963714/1311124568081174628 a user reported an issue where following a power loss, all of their matter devices disappeared. From the ...

bold kraken Nov 27, 2024, 4:40 PM

#

Actually, looks like the issue is in the Matter SDK itself, so I've filed https://github.com/project-chip/connectedhomeip/issues/36654

GitHub

[BUG] Persistent data storage in Python controller bindings is not ...

Reproduction steps A Home Assistant user reported an issue where following a power loss, all of their connected Matter devices had disappeared. From the user's provided logs: matter-logs.txt It...

umbral meadow Nov 27, 2024, 11:10 PM

#

Storage is a combination of a json file written by the sdk and a json file managed by matter server. Regardless of changing the code to duplicate writes, you should make sure to backup your HA server. You can not make this 100% fail proof, imagine a power loss at the exact time while copying backupfile. Imo there is one trick, which is reverting to a previous file at startup if corruption is detected. But still.... backup, backup, backup!

bold kraken Nov 28, 2024, 1:42 AM

#

Yeah - with the changes I recommend, there will never be a power loss situation that results in a missing or partially written config file (assuming your storage device works correctly; some sd cards can be problematic...), but there are still other possible bugs or problems that could result in configuration getting lost even without power loss.

#All Matter devices are unavailable after a power outage