I paired one of my Nanoleaf GU10 bulbs

[severe flint]

I think this is due to the same firmware issue so it get overloaded soon(er) from multi admin. Next time just pair it directly to HA and skip Apple because Apple is the platform that is currently overloading the devices with requests. You can always pair it to the Nanoleaf app because that does not use Matter but the custom nanoleaf protocol (and works even over bluetooth in case of thread trouble)

[sudden jetty]

Is it the same behavior for you?

[severe flint]

Yes, I've seen these issues as well but on all platforms. The devices are randomly restarting hence the instability. Just pair it to one controller at a time and it will be more stable.

[sudden jetty]

My one GU10 bulb is relatively stable at the moment. Only the short disconnect/reconnect issue. At least it seems to have no impact on my Thread network topology…

[elder cove]

I think there is a restart counter in the basic information cluster. Could be helpful to figure out if they actually “restart”

[severe flint]

I'm sure they are as that is confirmed to me. Don't know though if its the matter stack restarting on the device or the entire firmware

[elder cove]

I think that’s pretty hard to separate the way the matter sdk works.

[severe flint]

I think so as well but technically they could be running it on top of some base OS, you never know.

They have it running side by side with their normal infrastructure because the app is not using matter to control the device

[elder cove]

I really don’t think there is space to run it on some kind of base os on the chips. And freertos is deeply integrated into the sdk.

[severe flint]

Yeah but then explain how they are running two systems on one chip... Point is that we don't know nd can only guess. Maybe running both on one chip is part of the problem (the chip is just not powerful enough)

[elder cove]

But I have 12 bulbs on my way, so I can soon test it myself 😂

[severe flint]

haha, good luck, I hope that the firmware update is released by then 🙂

[elder cove]

Running multiple protocols at the same time is always trouble

Especially if the chip vender doesn’t support it by default

[severe flint]

Yeah, I have that feeling as well

[elder cove]

But for the price of their gu10 bulbs I’m happy to beta test 😂

[severe flint]

Yeah, the price is really good and so is the (hardware) build quailty

colors are good as well

so if they solve these fw issues they have a Hue killer

[elder cove]

For that price you probably have to save money somewhere

My Christmas lights are also getting hot when turned off, their support answered me in less than 30 minutes though

[severe flint]

its a feature of saving some heating costs

[elder cove]

😂😂 electrical heaters are kind of expensive

[sudden jetty]

Yes, the colors of the GU10 bulbs are fantastic. Imo it’s better than my GU10 Philips Hue.

[severe flint]

Well, I wouldn't go that far. Hue bulbs have the best colors (and stability) imo but they definitely come close, especially if you consider the price.

[sudden jetty]

I made some comparison. Yes, you are right, but it’s nearly the same color. The stability is the point for me. I don’t really care about the price. I want it all to be Matter over Thread, but it really needs to be stable.

The following procedure worked fine for two more GU10 bulbs:

1. pair the bulb to Apple Home
2. share the bulb from Apple Home to Home Assistant
3. pair the bulb with the Nanoleaf app and update it to 3.5.41

Now I need the new Nanoleaf firmware the solves the instabilities. 😃

[severe flint]

Why do you also pair to Apple Home ? You should be able to just pair to HA directly (using Apple Thread network) if you use the HA Companion app.

[elder cove]

Isn't apple home better intigrated into ios?

Also it doesn't suddenly "crash"

[severe flint]

The HA Companion app uses the Apple Home SDK under the hood.
Apple Home hides the issue because it has a 2 second reconnect interval and a debounce on the disconnect so it completely hides the issue from you, it is actually making the issue go worse because every thread device is required to do a full report of all attributes every 2 seconds. They have adjusted that to 10 minutes (and will go 1 hour later even) but I don't know when that change is going to be included in iOS. If it does, you will have some serious issues with your Nanoleaf devices.

For now I'd recommend NOT to join the Nanoleaf's to Apple Home to prevent them overloading the Thread network and the devices and that will also help HA and the restarts will occur less.

[sudden jetty]

Really good information. But I need the bulbs to show up in Apple Home, too. My wife and me often use the EVE app for manual interaction. We like the EVE apps user interface. I never went through the process of designing a HA user interface / dashboard. I use HA mainly for reliable automations. To design an appealing HA interface seems to take a lot of time, that I don’t have… That’s another story.

Alternatively I could bridge the devices from Home Assistant to Apple Home. But I want to have independent systems. So I can use Apple Home, when HA is down and vice versa.

At the moment I am waiting for the Nanoleaf beta firmware. If this will not solve the issue, I am going to remove them from Apple Home and bridge them from HA to Apple Home.

[elder cove]

Well its a subscribtion, so it just needs to send a “keepalive” every 2 seconds if nothing changes.

And in pretty sure there was something in the spec with specific attributes having a minimum report interval aside from the subscription to avoid flooding the network with fx updates from a package count in the threaddiagnostics cluster. Cant find it in the spec on the phone though.

But ofc this still adds load to nanoleaf bulbs.

I kind of Think its strange that they told you that they do a wildcard subscription since they dont know what app needs access to what. They could easily add this though like they do with their mdns services.

[severe flint]

No, no, that is not how it works. Setting the subscription ceiling to 2 seconds would mean that the entire subscription data is sent at that interval. In case of wildcard subscriptions this means every attribute of every cluster.

[severe flint]

Attribute subscriptions are (officially) limited to 3 paths per subscription and a max of 3 subscriptions per fabric so 9 paths to watch which is very easily reached so you hit the need for wilcard pretty soon

[severe flint]

Yes, there is indeed but so far we've noticed that devices just follow the controller blindly and they have no guard in place

[elder cove]

Im pretty sure that your understanding of this is wrong.

the purposes of receiving data reports from that publisher thereafter, for the duration of the subscription. This allows the subscriber to maintain a coherent snapshot, or twin, of the subscription
data as it currently exists on the publisher. The session itself is kept synchronized on both sides
through the receipt of timely data reports with the intervals defined by a negotiated maximum
interval subscription parameter.```
```Each Report transaction in a subscription reports changes to the subscription data.

To keep the subscription alive, a Report transaction is sent from the publisher every maximum
interval, or possibly more frequently```
And if there are no changes this is send.
```8.6.2. Report Transaction Empty```

I understand it as only updates being send, how did you come to your conclusion?

[severe flint]

No, I’m not wrong on this, trust me

[elder cove]

But isn't this more a bug in the implementation then?

[elder cove]

Okay i have just tested it. I have subscribed to the onoff cluster of an eve plug. I just get data in the initial transaction.

[severe flint]

How did you test that and be aware that some devices actually have a sane threshold

[elder cove]

https://pastebin.com/9gGtqNzZ

The first line i set the max treshhold to 20 seconds

and minimum to 0

you can see in the first status report that i get the data "false"

Afterwards the ReportDataMessages include no data.

[severe flint]

What I've heard is that some devices blindly trust the controller with the ceiling interval and do no sane limiting at all. So besides a keep alive at a too soon interval (which is already killing performance maybe) some devices send the actual data.

I assume Eve devices are some of the good devices then ?

Did you try a 2 seconds interval and see if it actually limited it to a more sane ceiling ?

[elder cove]

https://pastebin.com/LPcaxcD1
this is the data i get once i change the state of the plug.

[elder cove]

I can do that

[severe flint]

And what about wildcard ? Curious about that too.

[elder cove]

I need to see if i can do that with chip-tool

But these things should actually be part of the "core" sdk of matter and not within the vendor specific implementations.

So thats why im confused why some devices would do that.

[elder cove]

My data look the same for 2 seconds.

[elder cove]

https://pastebin.com/pev3N9PW

This is for a wildcard subscription

So there are basically comming no data

Unless i change the state ofc

[severe flint]

no but a message is a message wether it includes data or not

every 2 seconds a message for each subscription is still a lot of messages on a low bandwidth network

[elder cove]

I totally agree with that. At least if you have a lot of devices it can get a lot fast. And when this times out a subscription, all data will be send again.

[severe flint]

yeah, I created a testsetup with about 30 thread based devices and it got congested pretty fast, especially with only 1 border router

imagine what happens on really big networks - you get the same kind of issues we had to deal with in the zigbee world

[elder cove]

Yeah, this can get bad fast. Im with you on all of that. Just not that all data will be send with a 2 second interval, just some.

[severe flint]

Oh yes, sorry that was a bit misunderstanding from my side. messages with optional data

a message is data over the radio

but a message itself can have data included

[elder cove]

But i think the key difference with thread is that you can have more than 1 border router, which with the help of trel hopefully gives an easy fix for bad actors.

[severe flint]

YES, it is defintely a whole lot better than zigbee or z-wave in that regard

well...potentially.. because we;re not there yet 😉

[elder cove]

Almost 😄

But its just working sporadic

sometimes my nest hub joins my apple network, other times not

[severe flint]

I'm going to try out that idea with having a short-ish interval for just one cluster

[severe flint]

Yeah, that is not working at all yet. That should really be fixed/enforced by the thread alliance at one point

[elder cove]

Maybe this should actually be device specific? If a eve product is good at just reporting its state and doesn't send all diagnostics, then a wildcard subscription could be enough for it

And the bad devices get special treatment 😄

[severe flint]

I have a few different devices here, I'll play with it a bit to see what I can find out.

[severe flint]

Oh no... that smells like zigbee quirks all over

[elder cove]

I hope that this will be fixed by the core sdk being good at having good default values. Then it should be harder for vendors to fuck their products up

[severe flint]

Yes, there is now some discussion about this going on that there should be defaults

[elder cove]

I have had the wildcard subscription running on my eve plug for 10 minutes. I have only gotten empty status reports.

Gonna be interesting what other devices do.

[severe flint]

Yes, and what did you set as ceiling ? Imo, the sdk should protect setting insane short ceilings, or at least not allow setting a ceiling shorter than the wake up interval of sleepy devices

[elder cove]

I have set it to 20 seconds

[severe flint]

Like, it should not be possible at all to set a 2 second ceiling.

I think the minimum should be somewhere like 1 minute

[elder cove]

Depending on what device it is its maybe good to have short intervals. If i leave an office i would like to know if a window is closed or the sensor just is offline, so i can manually check. A false positive would be really bad in this usecase.

But 1 minute is probably a good amount.

[severe flint]

No, that is the floor, which you can even have at 0

[elder cove]

But if my device drops, it could take up to the max ceiling amount before i notice it.

[severe flint]

yes, but that is not so bad as a device does not drop that often

so it is perfectly acceptable to have a device go unavailable in the UI after being offline for 10 minutes even in most cases

maybe also depends on the kind of device

[severe flint]

1 minute is still on the low side if there are many devices.

For battery powered devices, even a 1 minute interval can drain the battery very fast

[elder cove]

I agree, but the point i was trying to make that there could be usecases where it could be beneficial to detect if a device is offline fast. Since a false positive can be more problematic than to know that a value cant be trusted.

Thats probably just an edge case though

[severe flint]

I'm curious to what usecases. Remember that the floor interval still determines that on value change, the data can be send immediately. This whole discussion is just about the "aliveness detection" of a node

[severe flint]

If there are edge cases we should cover those specially and do a more safe default for the standard cases

Like I said, having a 1 minute ceiling means a 1 minute keep alive message which is hurting a battery powered device but is safe for an ethernet connected bridge

[elder cove]

Yeah, there is probably just not one fits it all value

[severe flint]

Exactly, this whole thing needs discussion and some recommendations in the spec.
Is your company a CSA member joining the Geneva meeting ?

[elder cove]

Im sadly just an external consultant 😔

But the company i have been working for is.

[severe flint]

Ah bummer, we're meeting up with a few folks in Geneva to talk about this and settle on some common ground / best practices. We're hoping to also get some feedback from device vendors.

[elder cove]

Tbh, to me this sounds a bit like coming up with random numbers.

[severe flint]

well, we need something to start with and supported by thread chip vendors

some people call for 30 or even 60 minutes, some for 10 minutes

[elder cove]

Still a good idea to have something to start with, but im just imagining 20 different setups right now where 20 different numbers would be needed

[severe flint]

again, as ceiling. floor will be as low as possible for the controller

[severe flint]

I'm interested to hear that as I don't see it that way so far. Just a difference between (sleepy) thread devices and wifi/ethernet ones.

[elder cove]

I think a thread network should be able to handle a lot of "status report" messages.

[severe flint]

really the ONLY downside of having a high ceiling is that we have to wait that amount max to mark a device as unavailable

[elder cove]

Having a interval of just 2 minutes should in theory be enough for 100 devices to send empty updates every 2 minutes.

[severe flint]

I think you're wrong on that as has been proven already in large networks and it drains batteries of sleepy devices.

[severe flint]

Yeah, I'm also tempted to say that 2 minutes is good for powered devices

and battery powered devices we need something more careful

[elder cove]

But how does a small or large network make a difference for sleepy devices?

Because they will stay awake until they get a response?

[severe flint]

2 different things...
large network = more traffic
sleepy devices = need larger ceiling to not wake them up every few seconds or minutes

so I think that 2 minutes is really a solid compromis for net powered devices

you will need really A LOT devices before you congest the network with that interval

so what's left is the battery powered nodes where we can just learn from zigbee/z-wave best practices I guess

[elder cove]

I dont like the zigbee interval at all. Pr default it takes ages to detect that an motion sensor stops working

[severe flint]

yes and that is for a reason to not drain batteries

[elder cove]

In the mean time my bathroom lights turn off every 30 seconds when i sit on the toilet 😄

[severe flint]

Well, a device reports immediately on status change, that is regardless of the keepalive messages

so this is only if a device drops of the network,

in theory devices should just be stable

also be aware that a reconnect will also be handled

[elder cove]

And how fast do batteries really drain? My eve door sensors have been running for probably 7 months now on matter

and over a year on homekit before that.

[severe flint]

so if a device drops off and reconnects it will be immediately recognized by a controller

[severe flint]

I don't have any data, I only got reports of draining batteries. Eve door sensors are probably a wrong example as their battery level seems to report a wrong value?

[elder cove]

Yeah, but i have heard that its a big challange with some batteries to estimate the charge

[severe flint]

In my opinion 3 hours to detect a device is dead is soon enough for battery powered devices

[elder cove]

They have a consistent voltage until they just have 10% left

[severe flint]

yeah, that is always the case

[elder cove]

I think its really problematic to base automations in my home on wrong sensor inputs.

[severe flint]

You're really looking for edge cases here

why wrong sensor inputs ?

a sensor reports as soon as there is someything to report

just when you take out the battery so it dies it will take a few hours to detect that

[elder cove]

Yeah, but if a device just drops and doesn't reconnect, it takes some time

[severe flint]

I believe most z-wave sleepy devices even have 24 or 48 hours wake up intervals

[severe flint]

why wouldnt it reconnect ?

[elder cove]

And i have expirimented with using matter door and window sensors to detect if all 50 windows in a building are closed.

[elder cove]

Wireless interference and a lot of other unforseen tings

[severe flint]

Yes that would work perfectly fine, I still don not get your point

[elder cove]

I mean im describing the edge case here

[severe flint]

that doe snot count. reconnect will be instant

[elder cove]

Its more about leaving the building. I need to know if a window is really closed

[severe flint]

a permanent failure will take as much time as the ceiling

[severe flint]

if the thread network is stable you will know

otherwise I'd suggest to NOT using a wireless protocol

[elder cove]

Thats expensive and hard to retrofit

And i agree with you that its not a problem in most usecases.

But i just see it as a nice additional "feature"

[severe flint]

the base of all wireless communication is making shure that the foundation is solid right

[elder cove]

And with the current state of things, where you have a lot of devices that arent 100% stable useful.

BUt yes, it should not be nessecary in the future to need to detect offline devices.

Unless you have a girfriend or wife that unplugs your devices 😄

[severe flint]

If a sleepy device has to wake up every few seconds or minutes to send a keep alive it will just drain the battery, period.
If you do not want that, use bluetooth low energy sensors

[elder cove]

Or takes out the batteries of some sensors

[severe flint]

from all sensors I've tested in the past couple of years those sensors are the most battery friendly while still sending regular reports

[severe flint]

haha, yeah, that is also possible. But yet again, I think for net powered devices we can actually have a shorter keep alive

[elder cove]

But bluetooth is a bad technology

with their flooding mesh

[severe flint]

haha, well, it depends on the usecase. Bluetooth low energy sensors with a couple of bluetooth proxies works pretty well

[elder cove]

But thats homeassistant only

[severe flint]

Just like we do not have zigbee or z-wave battery powered devices sending a report every few minutes we are also not going to get that for Thread, unless someone invents a cool low-cost keep alive signal on top of the thread radio or something. But as far as I'm aware such a thing does not exist and the keepalive is done using the regular subscription logic

[elder cove]

But i think there are a lot of aspects to this and there is no "right" opinion. But when i read 20 to 60 minutes, all i see is an excuse for chip vendors to get away with an badly optimised stack.

[elder cove]

But aren't thread devices waking up every few seconds anyway to check for new queued messages?

[severe flint]

yes, that is the wakeup interval. Some people now say that our sub interval should be just aligned with that interval

it seems that when you send a subscription for 2 seconds, most devices then also go wakeup every 2 seconds

so they trust the controller a bit too much

[elder cove]

thats a bit too much

[severe flint]

so the whole problem could be solved if you just can not set a ceiling of less than wakeup interval

so the device itself (or its vendor) can decide what works best

[elder cove]

But you also have to take into account that most "controller" producers just want as stupid devices as possible, so everything can run through their controller

[severe flint]

or maybe even make it user configurable

[elder cove]

instead of a switch directly talking to a bulb using bindings

Especially the vendor that has a 5 second subscription ceiling 😉

[elder cove]

or combine it with thread, where the closest ftd tells the controller that the device just asked for any queued messages and therefore still is alive.

[severe flint]

yeah but subscriptions are push based

[elder cove]

Would reduce the load on sleepy end devices, while being able to have a "low ceilling"

[severe flint]

so setting a certain subscription interval means the device itself is going to push it

[elder cove]

I know that its not possible in the current implementation

[severe flint]

I get your point, detach the keepalive from the report

and make the thread device asking for queued messages the keepalive

[elder cove]

Yeah exactly, that would make everyone happy

[severe flint]

yes but than still we need to adjust the logic so trying to subscribe with a ceiling lower than the keepalive should fail

[elder cove]

yeah, that sounds more like a bug to me

[severe flint]

I'll take your ideas with me when I talk to people

[elder cove]

Great 😄

But i think there are other people in the csa that have a far more qualified opinion than me.

[severe flint]

well, at least somebody can tell us why its currently NOT working this way

[sudden jetty]

Yes, the special 3.6V battery in the EVE D&W reports 100% battery level for a long time and and then does not decrease linearly.

Source: https://www.reddit.com/r/EveHome/s/Mm4xuwfO9s

That person was in contact with EVE.

[sudden jetty]

Hi Marcel, I paired 4 more Nanoleaf E27 bulbs to my Thread network. This time I did what you mentioned.

1. connect the bulb to the Nanoleaf iOS app
2. update the bulb to the latest firmware 3.5.41
3. remove the bulb from the Nanoleaf app
4. reset the bulb by physically switching on/off 5 times
5. pair the bulb directly to Home Assistant
6. pair the bulb to the Nanoleaf iOS app

This procedure worked as expected for all 4 bulbs. But the procedure didn’t work without step 3 and 4. In that case HA showed me the message, that there was an issue while pairing the bulb.

These bulbs still loose their connectivity and reconnect some seconds later.

[severe flint]

You can also just pair to HA first and then to Nanoleaf after that. Note that you need to check the Nanoleaf app if the device is connected over thread before pairing to HA. Often its using bluetooth