devs_matter-archived

[tender oxide]

Report an issue on our matter server project. If it commissions but doesn’t show up, include diagnostics dump

[quick spade]

It shows up but as an on/off light

[modest prawn]

So I should get a job there?

[tender oxide]

We can relay the issues.

[modest prawn]

Great, I have to recheck they're still there in 1.1

I have collected almost 30 editorial issues that I'm sure would be easier to fix in the document source code (I think it's asciidoc?) than to write bug reports for each with my guess of what markup error caused that effect in the PDF... I'm spoiled with the open source world where I can just send a pull request 😅

[kind sedge]

Are you on the latest version of both the add-on and HA integration? As we fixed an issue with the device types just a few days ago. The device type is the mapping on which HA decidec whether to show your device as switch or light

Is it detected as on/off light or plain switch btw ? (you can see that on the icon, if its a light bulb, its a light)

[quick spade]

I'm always testing on latest versions of HA, matter server was pulled two days ago. It's showing as a light with s bulb icon

[solid mountain]

@quick spade can you download the diagnostics (using the download diagnostics button in the device page?)

[quick spade]

i cannot there is no such option

[quick spade]

ok got the option back after an update

[kind sedge]

Well, looking at your dump the devicetype is a humidity sensor ?

The deviceType should be set to the correct type for the endpoint, e.g. ColorLight or ExtendedColorLight

[solid mountain]

Hm, Device Type Classification is mandatory according to Matter Spec 7.15.3. Device Type Classification. Kinda surprised that this device works with Apple and Alexa 🤔

Well I guess the device type specification is there just the wrong one.

So it seems that Apple and Alexa do not rely on Device Type specification to build the device's control features 🤔

[kind sedge]

That would be very strange but cant say I'm that surprised 🙂

[zenith forum]

Hello

Do you know if it is possible to display the Matter device S/N in the 'Device info' section in HA?

We have that information in the matter diagnostic file.

[opal linden]

No

HA doesn’t have an API for showing serial numbers there for any integration

[quick spade]

yes, very strange that it works on Apple, Alexa and Google

[solid mountain]

In the end I guess that is a design decision on Controller side: Do you rely on device types or do you just parse the clusters and rely on that.

It seems they decided to do the latter, which makes their controllers less picky...

However, I am pretty sure that such a device would not pass certification.

[quick spade]

thankfully that won't be an issue 😁

[solid mountain]

Btw, did you update device type? Did that make the device properly appear on HA?

[quick spade]

there is no device type, tasmota only configures endpoint types

[solid mountain]

Wdym? I am guessing you generate the endpoints somehow, can't you add some logic which derrives the correct device type?

[quick spade]

i'm not that deep into matter spec, in tasmota each endpoint has its type, there is no overall device type

[solid mountain]

So how do you come up with the device type number now (775 / 0x0307)? Is that just a static number?

[kind sedge]

No that is correct, each endpoint has a devicetype (yeah the name isa bit confusing). We map the correct platform in HA depending on the device type. So if the devicetype is set to e.g. ColorLight, it wil get color controls.
Without the correct deviceType mapped, we dont know what platform to map and the device would fallback to switch if the on/off cluster is present.

Also, for colorcontrol the "ColorCapabilities" attribute of the ColorControl cluster is queried to determine the supported color modes

In case of lights

[ember summit]

FYI got an email from Silicon Labs about an upcoming virtual IoT conference. Attendance is free and it looks like they have a dedicated track on Matter

https://workswith.silabs.com/agenda?filters=404496

[modest prawn]

neat

[kind sedge]

@jaunty saffron we can talk here

[jaunty saffron]

ok, well does anyone have ideas about https://github.com/home-assistant-libs/python-matter-server/issues/313 - I'm seeing this on the latest main branch, and looks like at least 2 others are

[kind sedge]

Yes, a few new dependencies need to be added to the (now unmaintained) dockerfile

[jaunty saffron]

I see - and iiuc, the Add-On is not running docker?

I had assumed it was because the home assistant docs indicate that add ons are typically containers

[kind sedge]

Yeah, the add-on is based on docker but built around the HA supervisor, hence its images can not work directly on a bare docker box

[jaunty saffron]

Under the hood, add-ons are container images published to a container registry like GitHub container registry and Docker Hub. Developers can create GitHub repositories that contain multiple add-ons for easy sharing with the community.

so is there a dockerfile maintained for that?

[dreamy veldt]

https://github.com/home-assistant/addons/commit/49b412b9be1e6a2944e3b69e0cd4e08ba6c8aaaf

[kind sedge]

You can use this as a base but it will not work out of the box with regular docker due to the hardcoded paths and variables:
https://github.com/home-assistant/addons/blob/master/matter_server/Dockerfile

but 90% of that dockerfile can be re-used

[solid mountain]

It is trivial 2 liner essentially, I can do it in ~1h but busy right now

[jaunty saffron]

well it's pretty obvious from that commit what the issue is

is there any kind of test framework for matter? I can setup a ci hook that will run it in the dockerfile in the repo

[kind sedge]

For the SDK itself there is and also for HA core but we do not have any integration tests for the matter server + client lib + core stack as a whole

Easy enough would be to just buuild the container and start the server, that would catch 95% of these issues already

and the rest is caught in the HA unit tests

[cobalt belfry]

@earnest remnant I converted your message into a file since it's above 15 lines :+1:

[kind sedge]

temporary bandaid to unblock people:
https://github.com/home-assistant-libs/python-matter-server/pull/316/files

[kind sedge]

I guess we could adjust the example script a bit:
https://github.com/home-assistant-libs/python-matter-server/blob/main/scripts/example.py
Run it for 2 minutes, just to spot if it starts correctly.

Unit tests are already handled by a separate gh action but a real smoke test could be nice

[jaunty saffron]

i'll look into it

also, is there a link to threads/prs where the changes for networkmanager are made to haos?

i saw the issue opened on network manager and a suggested patch, but it looked like it wasn't actually merged upstream

and some other change conflicted

[kind sedge]

@solid mountain is aware of all ins and outs regarding that. I know that it has something to do with the networkmanager but also configuration. For thread, IPv6 neighbour discovery needs to work properly and these packets seem to get blocked sometimes, but for all fine details I'm pretty sure @solid mountain will jump in and share his experience from which together we can make a nice writeup to share with the community for people that want to run the whole stack theirselves

Also on network configuration side. For instance I had a lot of weird troubles with my Unifi set-up

[jaunty saffron]

i have the matter server running in the same subnet as my border routers - so i don't think there's any issue with my routers themselves. i can see that my route tables are properly configured, but after reading the thread it looks like i have to kind of monitor them over time to make sure they don't become stale as well

this is the change that I assume was applied to HAOS: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1232#note_1841630 - however that change does not seem to be applied to the main branch

[kind sedge]

That is indeed the change

[jaunty saffron]

conflicts with this change: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/02c3f7c946f467dc4240e595fd8fba47c7608a99

but it's not clear to me if that also fixes the issue

probably need both changes

[kind sedge]

Let's wait for @solid mountain to jump in and give some pointers as I'm not involved with the os side of things and just a messenger here. Your issues sound familiar btw what we experienced with HAOS too in the beginning

As for networking, I have disabled "multicast filtering" since a few weeks and did not spot a single unavailable matter device since but my setup is still a bit "too happy" with all devices and BR in my office. I'm planning on doing more extensive tests by spreading devices around the house (and thus multiple BR's)

[jaunty saffron]

this is for your wifi aps?

you have everything on one vlan though?

[kind sedge]

I had two vlans but the mdns reflector of unifi did not play nice so I flattened it

all matter devices were hindered by the multicast filtering "features", for example I could not pair a Hue bridge with that feature enabled

[jaunty saffron]

i have multiple vlans and mdns is fine - but the ipv6 RA based routing is definitely not working cross vlans for me

[kind sedge]

Exactly

[jaunty saffron]

i'm using avahi to do mdns reflection

[kind sedge]

yeah avahi does a much better job, at least for chromecasts, airplay etc

I just decided to flatten it for now, until it all matures a bit

Do have a seperate WiFi SSID for all IoT related though

[jaunty saffron]

i think my plan might be to keep multiple vlans, but move all iot stuff, including the HA server's into the iot vlan

currently i have a separate server vlan where i put HA

[kind sedge]

Yeah, that is actually the best option imo

so no routing needed

and just access HA securely with https

My advice now to people would be to either go that route or flatten it

[jaunty saffron]

what's the deal with logs like this: matter-server | 2023-06-07 14:43:56 sdk4 chip.clusters.Attribute[1] ERROR 'No Schema found for Attribute 1/319486977/319422471' matter-server | Traceback (most recent call last): matter-server | File "/usr/local/lib/python3.11/site-packages/chip/clusters/Attribute.py", line 754, in _handleReportEnd matter-server | attribute_path = TypedAttributePath(Path=change) matter-server | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ matter-server | File "/usr/local/lib/python3.11/site-packages/chip/clusters/Attribute.py", line 149, in __init__ matter-server | raise KeyError(f"No Schema found for Attribute {Path}") matter-server | KeyError: 'No Schema found for Attribute 1/319486977/319422471'

[kind sedge]

You can ignore those

[jaunty saffron]

is that like some custom thing eve is doing?

[kind sedge]

That are the vendor specific clusters

or attributes that we can not parse

but we do log them, altough a bit harsh perhaps

[jaunty saffron]

does eve publish their custom cluster schemas? or would we have to reverse these to figure out what it is?

[kind sedge]

that would be reversing them

also we need to adjust the upstream code to not try to parse these clusters

so basically just send us the raw tlv data

so at this point we dont even get the data as the code has crashed parsing it (which is the log you see)

[dreamy veldt]

Has someone tried to do that?

[kind sedge]

Nope, not a lot of fun in that 🙂

Energy metering is going to be added to the spec

so next gen matter devices (such as meross) will have it by default

now question is wheter eve is going to adjust their implementation

[dreamy veldt]

Yeah, i have seen the draft. Its a way bigger cluster than i thought it would be

[dreamy veldt]

I think anything else would just be stupid, since it reduces the compability with most ecosystems

[kind sedge]

Yeah, I think they want to be complete also with eyes to a complete energy metering solution and green houses where its all connected it makes sense.

[kind sedge]

Yeah, I feel so too but wouldnt be the first time a manufacturer messes up 🙂

[dreamy veldt]

Yeah, lets hope not, i have 6 of their plugs 😂

[kind sedge]

haha oops, you heavily invested in such a new technology 🙂

I must admit that it is pretty stable now since last weeks

The Eve stuff

in the beginning I ws suprised that a 50 bucks motion sensor could be so unstable

[dreamy veldt]

I have worked with them at my previous job and they seemed invested in matter. So I went all in

[solid mountain]

Right, still on my todo 😰

[solid mountain]

🤨 interesting, I have to check that

[solid mountain]

Yeah that is by design per network/collisision domain. They are also sent to the ffx2::/16 multicast address, which is by spec link-local. If you want routing, than route... This "briding protocols accross VLANs which are not meant to be run accross VLAN" is stupid IMHO, it kinda defeats the purpose of VLANs in first place as well...

[jaunty saffron]

idk - bridging a protocol that's just about visibility isn't the same as falttening the network completely

(for mdns)

[solid mountain]

mDNS uses ff02::fb which is link local as well. It is definitly not intended to be used accross networks 🤷‍♂️

Can you hack around it? Sure. But don't be surprised by surprises 😅

[kind sedge]

I think the best way -if you want to keep iot devices seperate- is to also have HA and the matter server in that seperate vlan

[solid mountain]

I think one of the main issues are IPv6 link-local addresses: In IPv6 world link-local addresses are quite common. And they are, well, link-local. Which isn't a problem for mDNS, since it is by design multicasting... link-local...

So, if you send an annoucment with fe80:..., via mDNS things just work fine in your network.

However, if you start to forward mDNS packets, these link-local addresses stop working.

[jaunty saffron]

and then proxy everything through that - yeah i think that would work fine

[solid mountain]

I guess that is the reason why some DNS reflectors "mangle" the mDNS packets.

Which may or may not work 😰

[jaunty saffron]

i guess that or start routing link local at your router

[dreamy veldt]

So one matter controller pr vlan is maybe a better solution?

[jaunty saffron]

but i get that - i'm not planning to do thread/matter cross vlans - but i do intend to keep vlans in my network

[solid mountain]

No, please don't! They are not meant for that! They are only guaranteed to be uniquie in the same network

Use ULAs if you want to route IPv6 traffic

[solid mountain]

Don't get me wrong, I have VLANs too, I am not "against" VLANs. But I stay away from mDNS reflectors and such for above reasons. The protocols are not designed for that.

[jaunty saffron]

i think once i move my HA server whenever i get a chance i can probably disable the mdns reflector

[solid mountain]

I'd say yes

We have VLAN support in HA from Supervisor/OS side, so you can add tagged VLAN interfaces on your Ethernet network interface. But that is about it. Our Matter Server runs in Host network namespace, so it will have access to all the interfaces. So I think it should work/talk to Matter devices on all interfaces (including VLANs). But haven't tested that.

[dreamy veldt]

The problem with that might be Apple or Google devices though, where you want to cast to them with your phone while still being able to control matter devices on an “iot” network

[solid mountain]

IMHO, it is not worth the trouble to have Chromecasts etc. and my Phone in different networks. I have them on the same main network.

Just go zero trust: Treat your LAN as Internet 😄

[dreamy veldt]

I also kind of think this is out of scope for the HAOS. If you really want to control your stuff and really care about security, you probably dont want an OS where you have “limited” control and stuff is installed in the background.

[solid mountain]

You need updates when you really care about security..

[dreamy veldt]

Yeah, but my point is then your Google Nest hub also needs to be on the same network to cast Music to it. Then you cant put it on the same vlan as your IOT devices

[solid mountain]

Reverse it: Define your main lan as IoT. Split out everything else (e.g. your NAS) into a secure VLAN. Or just shut its port an Wireguard/VPN into it only.

[dreamy veldt]

Yeah, i think you just never have perfect security. There is always some drawback

[dreamy veldt]

Thats maybe a better way to approach it. But i could imagine this would break Plex direct play

Also i have seen the note about changing the matter subscription logic. Have you looked at how much trafic normally is generated with wildcard subscriptions?

Aren’t data only send when clusters are updated?

[solid mountain]

Not really. The TLV format is quite effective, but there are a lot of clusters

[solid mountain]

Yeah, that is the case. However, some might get updates quite often... And I think that is the problem.

[kind sedge]

Yes, but for example endpoint 0 sends a lot of "useless" info like tx rates and such

In the beginning everyone just did a wildcard subscription to all endpoints, so did we

[dreamy veldt]

Ahh, I thought would be something excluded from subscriptions

[kind sedge]

but now people are buying dozens of devices you can see impact on traffic

I have now completely reworked it

and made a smart subscription model

so the client automatically subscribes if its interested

[solid mountain]

True, but I recommend looking into Zero Trust Network architectures if you care a lot about security. I think that is the best we can do somehow. Tailscale really promotes it (and obviously they have a product to solve many issues around it). But you can do the same with Wireguard natively yourself: https://tailscale.com/kb/1123/zero-trust/

[dreamy veldt]

Yeah, I think the biggest problem is that it completely congests unstable networks where a lot of resubscribtions occur

[kind sedge]

Yeah, when e.g. routing is bad due to a far away device etc and then a lot of devices

[dreamy veldt]

Thanks, I’ll look into that.

[solid mountain]

Just to be clear: That is required if you want to talk to Thread Border Routers which are not the device you run the Matter stack on

If your device is a IPv6 router, it also requires another patch for route reachability probing (see https://github.com/home-assistant/operating-system/discussions/2333#discussioncomment-5446082 for details)

Otherwise you end up with stale IPv6 routes, which can be very annoying.

[solid mountain]

I don't think that this fix is relevant to the Thread routing issue. It is about on-link prefixes, but the routing information from a Thread Border Router is a prefix which is not on-link (it is its own IPv6 ULA, from what I remember based on the PAN extended address). So I think that fix is not needed, but also shouldn't hurt.

@jaunty saffron do you use NetworkManager?

[jaunty saffron]

i'm currently using systemd-networkd for this iirc

but i'm not sure if i'm getting stale routes. i am not running the border router on the same device, however i do have several border routers on the same network

i see all the routes in the route table currently - but not sure if i unplug a border router and plug it back in, will i then have a stale route to the old ip?

[solid mountain]

So, IPv6 Neighbor Discovery is kinda wild: The kernel has some support for it, and so have userspace network managers. From what I understand, NM disables kernel level support, do it on its own (presumably "better", then the kernel does it). OpenThread (which probably most Thread BRs are based on today) uses IPv6 ND Route Information Option (RIO) to announce routes to the Thread devices behind the BR. Any device which wants to communicate to (Matter based) Thread devices needs to support those RIOs.

We've mapped out the world of NetworkManager and pure Linux kernel implementation a bit (OpenThread prefers the kernel implementation of IPv6 ND, they advice to configure that see: https://openthread.io/codelabs/openthread-border-router#6). The kernel implementation needs some sysctl set properly, but then seem to work nicely.

The NetworkManager side needs the patches we've talked about.

Now, I am not sure about systemd-networkd. It seems to have it's own implementation of IPv6 ND, and also processes routing information from a quick glance (see https://github.com/systemd/systemd/blob/v253/src/libsystemd-network/ndisc-router.c#L448). What now the question is if two BRs our out there annoucing the same route, does systemd-networkd processes both of them? Does it make sure that it/the kernel does some kind of router availability probing?

It might be that it "just works" with systemd-networkd. But honestly no clue 😰

In the end this is the challenge we are facing with supporting other systems... In the end, that is why we kinda have the policy of:
a) Use HAOS + Add-on if you want the stack best tested and known to be working.
b) If you want to play with other systems: Sure, but don't expect us to solve the problems on your setup. Too many variables we can't handle/don't have the bandwidth. If you systematically engineer/figure out/document their quirks: YES PLEASE! 😍

(side note: There is also a requirement of Andorid 8.1+ for Thread Network communication. It seems that only from that Android version onwards RIOs got supported properly. See https://developers.home.google.com/thread#border_agent_discovery).

[solid mountain]

From what I remember, when removing on BR form the network it stayed in the routing table (as in, it still was in ip -6 route). However, when pinging a device behind it, it switched to the other router quite quickly (as in, there was no noticable package drop using ping). There is ip -6 neigh which I think should show the route(r) availabilty.

In the case where reachability probing wasn't active, it would drop many ICMPv6 echo requests before switching to the second route.

(from ping you can't really tell which route is picked, so you need to Wireshark this... Then just kill the route it takes and see what happens)

[jaunty saffron]

this is all very helpful, thanks. i’ll try to do some more systematic testing of my setup to see if it’s working on these edge cases

[opal linden]

Fwiw, On my systemd-networkd box I had HA in a container so i turned host network off and gave it a macvlan network and let the HA container do its own ipv6 setup. That avoided systemd weirdness

[kind sedge]

good tip!
I'm going to gather all above info and tips in a special readme on the matter server repo as a start for our docs to run the matter server yourself. Then people can add to that document if they find out a working config

[modest prawn]

I saw it in Apple API docs 😉 but yeah no spec yet

I wonder if it will allow simple use cases too

The current spec for HVAC in Matter seemed pretty complex and aimed at massive central HVAC systems, it was hard to tell if ignoring all the optional complexity I could make it fit the use case of a split air conditioner

Would an air conditioner have a Thermostat cluster? I assume yes

Ohhh my problem was fixed in Matter 1.1 spec! LocalTemperature is no longer mandatory in Thermostat

I can tell the air conditioner to heat until it reaches a certain temperature, and it obviously has a temperature sensor to know when to stop, but I can't get that temperature

I only have one-way communication https://cdn.imgpaste.net/2023/06/09/KghBf6.jpg

Aaand I found another spec error, that was fast

[dreamy veldt]

whats the error?

[bronze bough]

I was able to add my google nest thermostat to HA matter today, I just get an on/off switch as I see climate is in development do you have a need for the diagnostics file from this device?

[wispy nacelle]

In Apple Home I see Home Assistant listed as "4939" under Connected Services. Any road map to change that to "Home Assistant" or something more descriptive?

[true crown]

Read from here: #matter-archived message (HA is not certified currently)

[kind sedge]

Yes please! You may send it to me in a PM if you want. We're working on climate support and it would be good to double check with the Nest.

[brazen geode]

hey @kind sedge I am getting some issues with today's released version of the matter server. Can I send the log to you or how should I report it ?

looks like an issue with the configuration: after resetting the config in the UI to default values and restarting the matter server it started to work with no issues

[kind sedge]

Yes, you may send the logs but could you describe what issues you experienced ?

[brazen geode]

when upgrading the matter server it got stuck starting the server or some similar error

[cobalt belfry]

@brazen geode I converted your message into a file since it's above 15 lines :+1:

[brazen geode]

showing the above errors in a loop

then I reseted the configuration (the in matter server ui) to default values and it asked me to restart the server

so I did....and no issues after

so I guess an issue related to the upgrade process not reading the previous settings values correctly ?

[kind sedge]

not sure, I guess if you would have changed the log level it would also been fine. Maybe that issue was in there already but just popped up now. Anyways the new release did not change anything in that regard

[solid mountain]

@brazen geode I've seen this before, unfortunately I have no clue how it happens. Stopping and starting (not restarting) the addon resolved it for me.

[brazen geode]

ok, yeah it was weird. It kinda got stuck trying to initialize the server in that loop. It was immidiately after upgrading the version. Didnt restarted HA or anything. Again after resetting the config and restarting the matter server it worked again (probably by your comments a simple restart would had worked too). No big deal though, thanks for the replies

[ebon olive]

I have definitely seen this a few times now too.

’Restart Home Assistant‘ (not ‘Reboot system‘) solves it for me. It happened after system reboots and after Matter updates, but I think not always.

[solid mountain]

Did you had a change to check how/if systemd-networkd processes the IPv6 ND RIOs?

[jaunty saffron]

i’ve been running in the same state since i posted that - haven’t had a chance to stress test things, but nothing has seemed to fail either

[rare mesa]

Are locks through matter supported yet with HomeKit and HA?

specifically *home key

[odd blade]

You really wanted the #matter-archived channel - this one is for developing (see the topic)

[rare mesa]

My b

[wispy nacelle]

@rare mesa iOS 17 will support matter + Homekey locks. However, I was emailed today with the Tou president (https://findtuo.com) about his new Homekey lock. He said the Matter spec to support home keys won't go to developers until late this calendar year. Possibly in the Spring 24 spec will that go public. So Matter locks that support Home key are probably a year away.

[summer basin]

I mean, there can be ways to work within the current spec, aquara already has a matter lock, which goes through their hub. So devices are out there, it’s just getting them and testing

[wispy nacelle]

I don't think there's a thread/matter lock with homekey support that doesn't use a hub? Only in iOS 17 will that even be supported.

[lean depot]

Hey, i'm struggling to get a border router setup for testing custom matter/openthread devices.
I was able to get it setup no problem on a raspberry pi using the 32 bit image of raspberry pi os. but the NRF docs claim Before you can use the CHIP Tool, you must compile it from source on Linux (amd64/aarch64) or macOS. If you want to run it on Raspberry Pi, it must use a 64-bit OS. so i swapped to the 64 bit one and i absolutely cannot get it to function. I tried bare metal and the docker container. all the system services seem to be running but i am unable to sudo ot-ctl ping whatever_the_prefix_is::8.8.8.8 or sudo ot-ctl ping whatever_the_prefix_is::192.168.1.whatever_local_ip
100% of packets drop every time

the same steps for 32 bit os seem to work fine though on bare metal at least

[solid mountain]

What target are you trying to reach there? Looks ab it you try using NAT64?

Not done that so far. I just ping local things usually

[lean depot]

yeah i can't ping local either though

192.168.1.1 for example is just my local router

and i can ping both 192.168.1.1 and 8.8.8.8 from the 32 bit image that i saved, but not from the 64 bit one

i tried following like 4 different setups too, and for example on this one https://openthread.io/guides/border-router/docker/test-connectivity

otbr-agent[224]: [INFO]-CLI-----: execute command: prefix add fd11:22::/64 pasor
This output is required for internet connectivty for the Thread network.```

i never get that ``otbr-agent[224]: [INFO]-CLI-----: execute command: prefix add fd11:22::/64 pasor`` message

[solid mountain]

Did you setup a proper dataset for your OT instance?

I've realized that OTBR sometimes behaves weird if that is not the case

[lean depot]

yeah, and i tried through ot-ctl as well as just through the webui

on two different clean installs

the 32 bit one i used ot-ctl and that one works fine

[lean depot]

i can get whatever logs you might want later tonight

i appreciate the help

[lean depot]

im home now so if you need any logs or anything that might be useful i can provide

[lean depot]

is there an easy way to just, use the border router running on my HA install for testing custom devices or is that a bad idea

[quick stump]

Hey, any ZHA devs here? I’m looking at the zha cover integration, and can’t figure out how the state is supposed to be updated (to STATE_OPEN/STATE_CLOSED) when set position is used, after the cover reaches it final position. The only places I see that's being done is when async_stop_cover() is called (I assume this is an explicit stop call from HA) or when the position is updated to 0 or 100 (in async_set_position()). Anything in between there seems like it should be handled by async_update() / async_get_state() but it doesn't seem to be called (besides manually using homeassistant.update_entity). Shouldn't there be anything in async_set_position() checking the position against the desired position?

[pulsar sleet]

hi

ive stopped the multiprotocol addon

starting OTBR

there we go, looks like its flashing now

Cross-flashing from FirmwareImageType.RCP_UART_802154 to FirmwareImageType.OT_RCP
NabuCasa_Yellow_OpenThread_RCP_v2.3.0.0_PA32_ot-rcp_hw_460800.gbl

[kind sedge]

ah that is good, its all still a bit early works

[pulsar sleet]

boom

[long nimbus]

Yep, that all looks right

[pulsar sleet]

ok great, but i would've assumed a new thread network created on home assistant would be home assistants preferred

[kind sedge]

well, that is also a bit unclear to me but from what I know is that its pushed by the (andoird) app atm

I have never seen it working because well, I'm on iOS 🙂

[true crown]

The Android app only pushes credentials, making it preferred happens somewhere else

[violet gulch]

This is an issue I’ve run into. In a clean install, it does make it preferred. But after you start messing around with different multi/single protocol and create new networks, you have to manually make it preferred

[pulsar sleet]

lol just set log level on OTBR to debug which triggered a restart and now...

ok its back to just one after a few more seconds and a refresh, phew

[kind sedge]

@solid mountain can you chime in here ?

[solid mountain]

Yeah sorry, at dinner right now

[pulsar sleet]

anyway, the UI will let me add to preferred network via the 3 dot

[solid mountain]

What do you try to achive?

[pulsar sleet]

i'm trying to prove the most valuable company in the world wrong

[solid mountain]

😅

[pulsar sleet]

i just was hoping to have a second thread network set up (in addition to the one that apple created) so that i can tell if my iphone can see it and manage credentials for it or not

i believe i have now succeeded, or at least i will have once i hit add to preferred network

i will then have MyHome47 (Apple created, migrated from HomeKit v1) and home-assistant

oh.

[solid mountain]

So Thread Network credentials and a Thread Border Router are two distinct things.

[pulsar sleet]

i think i know that but go on

[solid mountain]

You had Thread Network credentails already before, with a Thread network name home-assistant

[pulsar sleet]

sure

[solid mountain]

Now you assigned the OTBR to use that THread network.

[pulsar sleet]

well i tried and it failed, but uh huh

[solid mountain]

What version of the OpenThread Border Router are you using?

Anything in the logs?

[pulsar sleet]

just installed OTBR moments ago

nothing relevant in the OTBR logs

[solid mountain]

We have two options:

• SiLabs Multiprotocol
• OpenThread Border Router (the pure upstream version)

Both install the firmware automatically on Skyconnect and Yellow

[pulsar sleet]

the unknown error above is socket.gaierror: [Errno -5] Name has no usable address

[solid mountain]

Ok, did you make sure to stop and ideally remove the SiLabs Multiprotocol?

[pulsar sleet]

i hit remove multiprotocol

then i stopped the multiprotocol addon

then i started OTBR and saw in logs it was installing its firmware

then it appeared as an OTBR in the thread device config entry listing

then i tried to add the OTBR to the preferred network and it failed with socket.gaierror: [Errno -5] Name has no usable address

thats everything ive done so far

[solid mountain]

And OTBR is still up and running?

[pulsar sleet]

yes

[solid mountain]

Sounds like your instance has some DNS issue? 😕

You are running on a Yellow right?

[pulsar sleet]

yeah

and i usually know what i'm doing with dns

[solid mountain]

Which Core version?

[pulsar sleet]

latest

Home Assistant 2023.7.1
Supervisor 2023.07.1
Operating System 10.3

[solid mountain]

I have one with Multiprotocol active, let me try to reproduce here.

[pulsar sleet]

i feel like something fucky is happening with hardware overall because my bluetooth has been failing to come up

im rebooting just to check

[solid mountain]

What do they claim/what do you try to prove actually?

So in theory, the Core web socket endpoints should already return the preferred Thread credentials, which ideally the App should share with Apple.

[pulsar sleet]

i'm trying to get TLV sync built into iOS

they have a very simple API

i've properly implemented the API to grab credentials from them

they don't return credentials for their own created network(s)

restart has fucked things up good. OTBR and ZHA devices are failing to start

[solid mountain]

Well ZHA is epxected

Actually, you should have removed it

[pulsar sleet]

when setting ZHA back up from scratch do i want /dev/ttyAMA1 or 2

[solid mountain]

I mean you run pure Thread.

[pulsar sleet]

ok

[solid mountain]

You can't run Zigbee then.

[pulsar sleet]

got it

[solid mountain]

So what could have happened now is that ZHA and OTBR tried to talk to ttyAMA1, and disturbing each other

(unfortuantely, we don't have a way to give the two things exclusive access.. there are ideas around to ensure this, then only one or the other would fail... But not there yet)

[pulsar sleet]

well i removed OTBR integration just now and can't get it back because i dont know the OTBR REST URL, any idea what it is

[solid mountain]

It depends on the add-on in your case its http://core-openthread-border-router:8081

A restart of the add-on should automatically rediscover

[pulsar sleet]

failed to connect

[solid mountain]

What is the add-on logs saying?

[pulsar sleet]

i did restart the addon, no auto rediscover yet

lets seee

[solid mountain]

Yeah I've noticed that in the past. I think when you deleted, then it doesn't rediscovered, somehow. Not sure if that is a protection of some sort

[pulsar sleet]

logs look fine, there's an (i think) unrelated error

[11:29:07] INFO: Successfully sent discovery information to Home Assistant.
s6-rc: info: service otbr-agent-rest-discovery successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
otbr-agent[180]: 00:00:06.177 [W] Platform------: Unexpected Spinel transaction message: 3
otbr-agent[180]: 00:00:06.177 [W] Platform------: Error processing response: Drop

i can just reinstall the OTBR addon, maybe thatll work

[solid mountain]

@iron basin , and what is also super annoying is that the URL is nowhere shown in the OTBR integration. I think Robbie had the OTBR from the SiLabs Multiprotocol still lingering around, and there is no way to distinguish the two 😅

[pulsar sleet]

also would be good to add the url to the docs https://www.home-assistant.io/integrations/otbr

since the URL question popup links to there

[solid mountain]

yeah OTBR docs are on the 2do

[pulsar sleet]

ok reinstalled OTBR addon, started it up after selecting /dev/ttyAMA1

lets see

oh boy a lot more logs now

still no device re-created though

nevermind just appeared

okay, now we are cooking

this is what i was expecting to see

[solid mountain]

Reinstalling the add-on "reset" the discovery stuff (as in, a new UUID is generated, which treats the config entry as a new one). I guess that helped here.

[pulsar sleet]

sure

[solid mountain]

Yeah now you have Thread credentials and a OTBR assigned to it 🎉

[pulsar sleet]

now lets see if iOS can see it

or register it in the credentials store at least

[solid mountain]

So the mDNS/DNS-SD information on the network are not complete dataset.

You'll have to fetch that through the WS API.

And there is one cavate which could play a role: The Border Agent ID might not match up

They might actually check that.

Actually, I think the Border Agent ID is exposed via mDNS already

But afaik you can't get it through our API.

But then I also don't know if its relevant 🤷‍♂️

[pulsar sleet]

i dont think mDNS matters at all for this actually

i bet they dont validate the credentials at all

i mean reach out to validate the credentials

okay

im just gonna submit this ticket as is

its light on details but the problem isnt that complex either.

[kind sedge]

OK, so you're still able to reproduce the issue and not able to send any preferred dataset towards apple ?

[pulsar sleet]

i havent even tried sending a dataset because i dont know what format they want

right now i'm just trying to get credentials that the system already knows about

expecting to get credentials for the homekit network (mine is named MyHome47)

Might as well try to store some credentials too though ig

feeling around in the dark though just hoping that the format they are shown by Home Assistant as is what Apple wants

[kind sedge]

maybe they do not allow sharing their dataset but allow us to set one

[kind sedge]

well I think so yes as that is a common format

Its the dataset TLV right

[pulsar sleet]

Active dataset TLVs: 0e080000000000010000000300001435060004001fffe00208cc847ca3db9a66e60708fd91e65d6aeda6c505101e3b2570e7623698e329b3cde03fb5de030e686f6d652d617373697374616e7401026c1e0410bee2a0529c5b75fa9867c854326c19180c0402a0f7f8

thats the string im putting in

[kind sedge]

yeah, thats the same format as required for matter

[pulsar sleet]

any idea where i can find a "border agent id / XA" in HA?

[iron basin]

Hmm, so what do you suggest? 🤔

[pulsar sleet]

okay i successfully stored credentials and it detected that it was for HA

i had to dig to find the border agent id / xa which i found via the OTBR web UI

named ExtAddress

value is AA5DF41D967F1A43

[kind sedge]

Well that is already a a great start because it will allow people to commission devices without an Apple BR

[pulsar sleet]

so given the two values i've put in here already (the extaddress and the active dataset TLV) the credential gets stored and comes back out later as:

activeOperationalDataSet: 0e080000000000010000000300001435060004001fffe00208cc847ca3db9a66e60708fd91e65d6aeda6c505101e3b2570e7623698e329b3cde03fb5de030e686f6d652d617373697374616e7401026c1e0410bee2a0529c5b75fa9867c854326c19180c0402a0f7f8
borderAgentID: aa5df41d967f1a43
channel: 20
extendedPANID: cc847ca3db9a66e6
networkKey: 1e3b2570e7623698e329b3cde03fb5de
networkName: home-assistant
panID: 6c1e
pskc: bee2a0529c5b75fa9867c854326c1918

which based on what i see in the HA web UI... matches

so yeah, storing is fine. its just that we cant get out apple's network

[pulsar sleet]

ok so

@iron basin if we want to be able to store HA credentials into iOS we are gonna need to expose that Border Agent ID/XA to the list_datasets websocket call

cc @kind sedge

[true crown]

For Android the border agent ID not only needs to be exposed, it also needs to be included in mDNS (for OTBR)

I think there was ongoing work for that

[pulsar sleet]

ok cool

[kind sedge]

nice progress @pulsar sleet !
Now let's wait and see if we're also allowed to read the existing dataset so we can join the otbr to that

You're going to write up a support ticket for that right ?

[pulsar sleet]

i already submitted the ticket

but i think we already have our answer

their cred store is working as expected, they dont want to give us the TLV of the apple network

[true crown]

https://community.silabs.com/s/question/0D58Y0000AONjvgSQD/border-agent-id-support-from-upstream-openthread?language=en_US was blocking it I believe

[kind sedge]

I'm afraid so yes. But I think this step is already great progress

[pulsar sleet]

i also just confirmed their code does actually validate

on a lark i tried replacing the Border Agent ID with the extended PAN ID

got Thread network credentials does not match with any of the active thread networks around

[pulsar sleet]

oh the xa is published in the mDNS record, so we (the iOS and Android apps) could get it that way too

[true crown]

Google looks for id for validation 😦

[pulsar sleet]

then both are in the mDNS record

as a workaround, both apps could just use mDNS to get the record and then confirm its the same record provided by HA by comparing the extended PAN ID that HA gives us and the extended PAN ID in the record (the xp key in the above screenshot)

if iOS wanted to do it that way we'd need a special entitlement from apple btw (i think)

[true crown]

Could work but still wouldn't pass validation on Android, docs:

Note: The Border Agent ID MUST be the id TXT value of the Border Router mDNS MeshCoP service.

[pulsar sleet]

thats in the screenshot of the record, no?

oh wait i see what you mean

the id= entry doesn't match the xp entry

[kind sedge]

even more ? 🙃

[true crown]

The extended PAN ID (xp) is supposed to be different from the border agent ID (id), right?

One refers to the network, the other refers to the border agent

[pulsar sleet]

no actually i dont think we do as long as we specifically request the _meshcop._udp bonjour service. we only need the entitlement if we want to be able to scan for any bonjour records

[pulsar sleet]

dont look at me, i barely understand all this!

[true crown]

I get confused every time 😄

[pulsar sleet]

i'm only doing this work on iOS because i'm annoyed i have multiple networks and apple's API for the functionality is so simple anyone could do it

[kind sedge]

you';re not alone 🙂

[kind sedge]

Well, not everybody has the development setup in place for the iOS app. I was planning at setting it up, just to learn and maybe look at this but I'm very glad you beat me

Now I'm wondering if you commission a thread device if it starts using the otbr creds

[pulsar sleet]

@tender oxide and i are discussing that in the background right now. that's what i'm betting, because the docs say once credentials are stored they become the preferred network

[tender oxide]

I'm here

[pulsar sleet]

oh hai

i dont have any thread devices around to test commissioning with, only homekit devices

[tender oxide]

For Android we're setting our border agent ID to 1 … works great 😬

[pulsar sleet]

i wonder what happens if i set border agent id to 1 on ios

[tender oxide]

https://www.amazon.com/Aqara-Requires-Contact-Automation-Supports/dp/B0BTL8B72D are these the cheapest Thread devices right now ?

[kind sedge]

yep

and there's a lot more coming. In the US you already have some cheap light bulbs too

[pulsar sleet]

1 is rejected when attempting to add Invalid parameter sent to server...

[tender oxide]

lol looks legit https://www.amazon.com/Linkind-Matter-SmartThings-Changing-Automation/dp/B0C58MCNXL?crid=1FCTOZBE9FN5S&keywords=matter+thread+light+bulb&qid=1689105098&sprefix=matterthread+light+bulb,aps,82&sr=8-2-spons&sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY

[pulsar sleet]

can i just scan a homekit device i have and see what happens

or does it have to be a matter device

[kind sedge]

if its homekit over thread I suppose it should work but @opal linden knows better

[pulsar sleet]

oh it is a thread device (eve door and window sensor)

so lets go try

[kind sedge]

cool, that one is even ugradeable to matter if you want

[true crown]

It complains in the logs but if there's no other Thread BRs it'll accept it, wouldn't call that great

[pulsar sleet]

heh

so i have this eve device, supposedly thread compatible

i have a homekit pairing sticker for it

the ha ios app wont scan it though

[tender oxide]

Right, as that's Matter

different QR code format

[pulsar sleet]

how do i scan it in then

[kind sedge]

well, the apple home app scans both formats

[tender oxide]

Apple Home app

[pulsar sleet]

but that will put it onto the apple network right

[kind sedge]

yes

you can also comission from the HA app

within the Integrations page --> Click Add integartion --> Add matter device

[true crown]

If the preferred Thread network changed, maybe not?

[kind sedge]

hmm, yeah, that would make sense indeed

[pulsar sleet]

pretty sure its going to the apple network, when i scanned in home app the button said Add to "Apple Home"

[kind sedge]

yeah, thread network we mean

apple fabric, ha thread network

[pulsar sleet]

ah

how would i know if its on our thread network

i'd see it in the thread integration?

[kind sedge]

well, if you have the eve app installed you could see it in there

they have like this thread network explorer in there

[pulsar sleet]

oh goodie, the eve app has a "upgrade to matter" button that gives me a new code

lets do that

[kind sedge]

make sure to store that code very well 🙂

self desctruct in 1... 2...

[pulsar sleet]

saved to icloud drive

Just in case lmao

10 minutes to upgrade?!

ok upgrade done

added some logging to the HA app before commissioning just to see what the Matter framework gets from the TLV i added, if anything

right now its set to add to the default thread network for the system

whatever that means

The Matter device provides information about the visible Thread networks that it sees. This allows the ecosystem to choose the correct network for the device to use.

The system may provide the selected Thread network in the completion handler. The device commissions onto the specified network. The device must be able to send the device IP traffic after it associates to the given network. Otherwise, if the system returns defaultSystemNetwork, the device commissions onto the Preferred Thread network known to the device.

Store credentials for the provided Thread network by calling storeCredentials(forBorderAgent:activeOperationalDataSet:completion:) before completing this callback.

ok well

bad news

it didnt add to our thread network

and the logging never got fired which means apple never asked us what thread network we'd like to use

[opal linden]

Too late now, but to complete the circle: homekit over thread uses Bluetooth for commissioning. If you have working Ha Bluetooth and the device is in pairing mode it should already be listed in your discoveries without scanning any qr codes. When you have paired it, there is a button entity that you can press to write the TLV (in a different endianness to matter) over homekit for Bluetooth and the device migrates to Bluetooth. So if the preferred network is set this already works without any assistance from the iOS app. But we can’t use this method for apple border routers.

There is a crappy working for apple border routers and homekit over thread - pair and unpair with iOS, that puts it on the apple network and leaves it in a pairable state. Then pair with Ha, as it was now discovered like any other zeroconf device.

[pulsar sleet]

welp

i deleted my device from the matter integration

and now i can't re-pair it

the apple UI is just saying pairing failed

[kind sedge]

No then you need to factory reset it

and thats kind of a pain with those sensors

[pulsar sleet]

ugh

ok

[kind sedge]

the device is only in commissioning mode after reset or when you enable comisisoning mode from another fabric

so you can join it to the apple fabric and then from apple share it to HA (it opens the device for comisisoning then)

we do not yet have a way to share it from HA to apple but that will follow soon

[pulsar sleet]

ah ok

i found why its not scanning

The app receives the network scan results, or an empty list if no scan occurred. Even if the property is set to true, the scan may not occur if the accessory doesn’t support scanning.

[true crown]

Ping me once that gets closer to being ready, Android has a nice built-in dialog we can show for Matter open multi-admin/sharing

[pulsar sleet]

i imagine that this tiny battery powered door sensor doesnt do scanning

[kind sedge]

no it cant scan on its own no. its meant to be discovered over bluetooth and then the thread creds will be commissioned and then it joins the thread network

[pulsar sleet]

how can i tell from the HA side what thread network it got onto

diagnostic json on the device is useless

i guess i already know the answer since it would appear in the OTBR UI right

sigh

so we can't get apple's network TLV and we can't get devices to add to our thread network from iOS

so this is all useless

[kind sedge]

well, that is where the info is, its just a tiny bit cryptic 😉

[kind sedge]

uhh yes I guess so

[true crown]

If you don't have Apple border routers it might work and not be useless?

[pulsar sleet]

i have apple border routes and home assistant/openthread border router

[solid mountain]

Extended Address and Border Agent ID is not the same

[kind sedge]

I was just about to say that

[pulsar sleet]

ok but i guess apple will accept either as a key and it works for them. or it just so happens that my xa and border agent ID are the same

[kind sedge]

Doesnt Google have this same issue ? If there already is a Google BR I mean ? That Google will always join that

[true crown]

Sort of, yes

The first network added to the device is the preferred network, with no way to change it

[solid mountain]

Border ID is per OTBR. Not sure how this will work with multiple OTBRs...

[true crown]

No function for the app to say which network should be used like on iOS so even less control

[solid mountain]

It is anyway confusing, since why would they even care about an individal BR... 🤔

[pulsar sleet]

good point, the framework they provide appears to assume only one BR per network

well i guess you could just store multiple of the same credentials with different BRs

[true crown]

Because apps control border routers, while a network could contain different border routers

If we get Apple/Google credentials we can join their Thread network

[solid mountain]

In what way?

[true crown]

But the Apple/Google apps won't add our border router probably

[solid mountain]

Ah you mean Google Apps control Google BRs

[true crown]

Contributing the credentials to the system

[solid mountain]

Yeah ok, but for third party this is kinda moot.

Yeah but how do individual BRs matter here? it is like you would provide the BSSID of your acecss point along with your WiFI credentials... Nobody is doing this 😅

[true crown]

If you add network credentials only, who 'owns' those credentials?

HA and Apple could use the same, one decides to delete them - can HA delete credentials originally added by Apple?

[pulsar sleet]

here's everything apple says in the framework docs about multiple brs

Multiple Border Routers on the preferred network strengthen communication between devices and improve resiliency in the home.
and
You can store the credentials for any number of Border Routers this way.
and
The Border Agent is the software component running in the Border Router responsible for advertising itself in the Wi-Fi or Ethernet network.

The framework only stores credentials if it can find an mDNS record for the Border Agent that contains the specified Border Agent identifier.

[solid mountain]

Hm, so they want to validate that the BR you claim is indeed in that network.

But devices will roam between the BRs no matter if you only share one or all the BRs... So that is kinda weird.

[pulsar sleet]

welp

i dug into the headers and found the answer to the TSI

/// Some methods in THClient use the team ID, a string that you store in
/// your application’s Info.plist. The ThreadNetwork framework uses the team
/// ID to preserve the privacy of the Thread network credentials across
/// different clients. For example, credentials stored by one client can’t be
/// deleted or modified by another client.

The framework identifies the developer by the team ID. When calling this method, you receive credentials for your team ID only.

[solid mountain]

So how did iOS behave now? Did it accept the Thread credentials with xa as Border Agent ID?

[pulsar sleet]

its always been accepting them

[pulsar sleet]

this means we will never be able to get apple's TLV via iOS, the credential store only allows us to get at credentials that we set ourselves, because its segmented by team ID

[solid mountain]

Yeah so we really need to add the Border Agent ID to our thread credentials store. We have an API on OTBR side already

[solid mountain]

Yeah so that is the same with Google

[pulsar sleet]

so now that we know for sure, @tender oxide was right: we should fake a matter device that gets added to the apple or google networks so we could get the TLV

[solid mountain]

Can we add ours still and force the Matter onboarding to use ours instead of theirs?

[true crown]

Can't you get them using retrievePreferredCredentials(_:)? https://developer.apple.com/documentation/threadnetwork/thclient/3801787-retrievepreferredcredentials

[pulsar sleet]

as per the header notes that i just pasted above, the credential store is segmented by team ID. we can only get credentials that we insert. even if a million other apps on the device store credentials, we will only ever be able to get credentials out of our segment of the store

[solid mountain]

Hehe evil plan! 😈 We could make an add-on which would flash a custom firmware to the Yellow/SkyConnect which pretends to be a Thread device...

[pulsar sleet]

we can force matter onboarding to use our thread network right now but only if the device is mains powered it seems, because the device has to do the scanning for networks, then gives us the results, then we can tell it what network to use

[solid mountain]

I don't think we could get such a device certified, so you'd need to enable developer mode or whatever is required to get an uncertified Matter device on-board.

[true crown]

Why would it show a permission alert then?
(comparing to Android: we can get anything added by our app in the background, other apps via permission dialog)

[pulsar sleet]

i havent seen any permission alerts on iOS

[true crown]

When you call this method, an alert appears asking for user permission to access credentials.

[pulsar sleet]

oh lol

well i havent gotten an alert

[solid mountain]

Oh I see. That is interesting.

[true crown]

Odd, the description also makes it sound like the function is for credentials from any app

While others do mention the team ID limitation from the header file

[pulsar sleet]

another half baked SDK from apple

ok i give up on all of this

try again next year with ios 18, maybe they'll improve

[solid mountain]

So it seems that the idea behind Border Agent ID is for future expansion to talk to BRs directly.

@pulsar sleet but for networks with not Apple BR, do you think things would work? As in sharing the creds and then it would get picked up by the Appl Matter flow?

[kind sedge]

Well, you did your best. Maybe we get an answer from them someday how to do this.

[pulsar sleet]

they will reply to the ticket within 2 biz days or whatever

but i am 99.9999999999999% sure what their answer will be

https://tenor.com/view/bugs-bunny-no-ño-bugs-bunny-no-bugs-bunny-bugs-bunnyño-no-gif-16818895

[pulsar sleet]

thats what i tried earlier today and the answer is

https://tenor.com/view/cat-no-nonono-noooo-cat-no-gif-25423213

[kind sedge]

haha

[pulsar sleet]

the device decides what thread network to join on its own, or apple tells it what network to join. we do have some control over it but only for mains powered thread enabled devices that support scanning

which i imagine is few

[kind sedge]

that is kind of meh

why did they do that

[solid mountain]

Well, but if there are no other BRs than ours...

Then it would need to join ours 😅

[kind sedge]

ah yeah that makes sense

and is also something needed for people without apple br's right

[pulsar sleet]

i would assume so yeah

[tender oxide]

Even better use SL web flasher and web Bluetooth to do it all in the browser

[solid mountain]

So the Matter Network Commissioning Cluster supports "AddOrUpdateThreadNetwork". I wonder if we can migrate a device which joined the Apple Thread network 😈

[solid mountain]

Not for Yellow 😰

Ah

You mean emulate a device... 🤔 Uh yeah that sounds nice!

[pulsar sleet]

knowing apple (and to a lesser extent, google) i would be somewhat very scared about fucking around with their network at all, whether adding our own BRs to it or adding/removing/migrating devices to it outside of their control

especially on the iOS side because everything is so tightly integrated

[solid mountain]

🤷‍♂️ it defintily works the other way around

[pulsar sleet]

sure

they want devices in their networks under their control

[solid mountain]

As in, if you add a device to our BR, you can share that device with them

[solid mountain]

They want. But they don't get mine 😅

[kind sedge]

@pulsar sleet I got this response back:

Right, reading existing creds may be restricted.  That's what the feedback issue might help answer.
Using a non-Apple BR that forms its own Thread network should work via THClient, is my understanding.  If that's not working, please do file a feedback issue and let me know the issue number.

Can you share me the issue number ?

[quiet grotto]

Sorry to butt in, but from the above discussion, specifically the difficulty Robbie had with getting everything all hunky dory, on HAOS on Yellow, or with a SkyConnect, is there any chance at all for one of you with things working nicely to write up a spec, even bare as anything, for the bare minimum setup, to add a thing (matter, or thread device) with a thing (android or ios phone) with again the bare minimum?

[kind sedge]

Yes, that is on the TODO list.

[quiet grotto]

I'm trying to do much of the same, with more vanilla docker containers, but seem to be hitting some of the very same issues. while the docs give something, for matter, and something, for thread, there's not a lot of "all together"

[quiet grotto]

that's great to hear, I'd love to help out if I can at all

[kind sedge]

No, and I've told you this before, its not there yet. You can experiment but you are a bit on your own still 😉

[quiet grotto]

I know I know! 😄

the thing is, if it were good on HAOS, I'd probably jump there

reading the above, it seems like much of the same is happening on there, atm, can we do everything with just HA and SkyConnect/Yellow

[kind sedge]

Well, at this point that is really the trouble free setup, yes.

[quiet grotto]

Robbie wasn't trouble free!

[kind sedge]

No, that is only for using own own OTBR

That is not working very well yet

Using Matter with Apple or Google BR's works very well

[quiet grotto]

so, that's the part that I'm okay with handling myself, if I know it's possible, I'm okay with (hopefully) figuring out what's wrong where

[kind sedge]

Note that HA will us ethe existing BR's from Google and Apple

[quiet grotto]

okay, so even that's different to my current understanding of how it should be put together, even on HAOS

[kind sedge]

at this point? policies of the big tech, nothing technical

[quiet grotto]

yeah, I've read A LOT of git issues over the last week

what I'd love now is a bare minimum of integrations that HA needs to have, to work well with x, or y

[kind sedge]

You just make sure you have some border routers in place from either apple or google and you can commission devices to HA just fine.

[quiet grotto]

as in, not multiprotocol, but needs OTBR, and needs Thread, and needs Matter, and then needs to commision with x, via y

[kind sedge]

Boot up a VM with HAOS

[quiet grotto]

lol, I've done that twice already

I'll find my feet in a little bit, but I'm still experimenting with exactly the way I want to run my setup - I'd also prefer to be able to run just an OTBR and matter server on a RPi, without anything else really

[kind sedge]

We do provide a way to run the Matter server in your own docker container since a few weeks but we do not yet have any support nor instructions how to run the multi-pan/otbr stuff without HAOS. I'm not so sure if there are actual plans for that

[quiet grotto]

but, that's the thing about having a working OTBR standalone

if I just give that access to the Sky Connect, HA should see that as a standalone BR

[kind sedge]

Yes, but at this point that is only possible with WiFi based Matter devices and bridges, not Thread based matter devices.

[quiet grotto]

okay, so then that's my question about the HA addons

the Thread addon asks you for a OTBR API URL

the OTBR addon shows you TBR networks and routes

my questions here are wheres and whys

if I've a perfectly working OTBR running, what's stopping all things thread running through that, from HA

[kind sedge]

Better ask those questions in the thread channel but at this time it is what it is now. We're already swamped in work to cover the basic stuff so more advanced setups will come later unless someone helps out and contributes if you understand what I mean.

[quiet grotto]

nah, I do totally understand what you mean

[kind sedge]

Like, when we have the basic setup working well and documented with HAOS, we can also focus on more DIY/advanced setups

[quiet grotto]

my desire here is to be able contribute

[kind sedge]

but we;re not even there yet with the first steps

[quiet grotto]

I really do understand, fr fr

just, for my head, I'll always need to have a very good understand of the lower levels, first, and for me, the first step of that is to have a good OTBR running right

[kind sedge]

I'm not the Thread expert, I do the Matter development within HA which relies on Thread but is its own protocol.
If you monitor the thread channel (we dont have a devs_thread as it seems) you probably have beter luck for answers, for example Stefan is active there who works more low level on thread

[quiet grotto]

but yeah, just to confirm this point, even on Yellow, or HAOS with SkyConnect, you do need another thread border router, and cannot just use the OTBR that's on board? that's correct?

[kind sedge]

Yes you can, if you have an Android device

It does not work yet on iOS (that was the conversation above)

So if you have a google account with NO Google Border router active and an Android Phone, you can sync the thread credentials from OTBR

[quiet grotto]

okay. and, the setup of that would be - needed integrations would be OTBR, with the OTBR API URL in, and Matter

[kind sedge]

This is the part we are going to document soon

[kind sedge]

yes, just the otbr add-on and the matter integration (which will auto install matter addon)

[quiet grotto]

yes

[kind sedge]

no need for the url as that will be handled automatically

[quiet grotto]

and then, in the Thread integration, HA's own network, home-assistant should have that border router in? or it "should" be in other networks?

as in, something like this https://i.imgur.com/E7zg6mX.png

the alternative to that is a border router "should" be in other networks, and then the choice is to make that network our prefered network, or add that border router to our network

sorry for all the q's, but I do actually think I'm very close to having my setup working "nicely", but just have a small bit of ... misbehaviour from my running OTBR

[swift void]

@kind sedge Based on the GitHub response operational discovery via mDNS wasn’t implement yet? But if that chip PR is merged it will be?

[kind sedge]

Well I’m theory yes and when a new sdk version is released including that change. Do maybe 1.2

For now I’ve added some additional polling. I’ll release a new server version tonight so you could test it

[dreamy veldt]

Be careful, my 40+ thread matter devices take a long time before becoming stable right now

It might make big networks more unstable if the traffic is increased

[kind sedge]

No worries, polling only happens for unavailable devices

I'm not going to undo all my work in decreasing the amount of traffic 🙂

[dreamy veldt]

Yeah, but that’s the problem. Once a lot of devices are unavailable available devices start dropping again because of all the traffic.

HA actually handles it better than apple

So well done 😊

[kind sedge]

why would they be unavailable ? Its not a very good idea to just drop power from your routing devices anyway

[dreamy veldt]

I’m not sure, everything in the building drops sometimes.

But I mean my point was just to keep this side effect in mind.

[kind sedge]

I think that is unavoidable to have a sporadic drop (and recover within a second or so) as this is IP based. If it is like continuously dropping unavailable I'd seriously look at the network/mesh (both thread and ethernet)

I have my testsetup here with various thread devices spread across the house (well, I only have a few, not 40+ like you haha) and thise are commissioned to my prod HA instance, test HA instance, Dev instance and Apple Home. That way I can track availability and any issues. I get a notification if one of those devices drop unavailable. So far only happened once for a motion sensor in the past 4 days

[dreamy veldt]

Yeah, I totally agree that it’s unavoidable. With fewer devices It also runs very smoth for me. But what I see is that when a large number of devices drop it takes multiple retries before all devices are back online. So decreasing the polling interval might increase the traffic even more, especially when both HA and apple try to reestablish connections at the same time.

[kind sedge]

Again, no worries those devices will only be polled after the subscription failed for 10 minutes.

Out of curiosity: how many devices do you have per BR ?

[dreamy veldt]

Great, I also wasn’t worried 😁 the improvements until now have been great

[dreamy veldt]

Right now only 2 apple TV’s with Ethernet

[kind sedge]

The poll will eventually be eliminated completely btw as soon as operational discovery gets in

[dreamy veldt]

And today I pulled 2 20 meter lab cables to connect them around some UniFi switches I don’t have access to.

[kind sedge]

I wonder if there may be a limitation per BR or routing device ? Do you have a lot of powered devices ?

[dreamy veldt]

And the mdns multicast is blocked

[dreamy veldt]

Around 10

But I think right now most traffic is routed through 1 otbr.

And it’s getting kind of slow.

[kind sedge]

That seems like a fair coverage then. Interesting to hear such large setups already and how they perform

[dreamy veldt]

The worst is opening the apple home app.

It seems like it sometimes tries to reconnect to all devices

And then all go offline.

[kind sedge]

hmm, that sounds like a bug

[dreamy veldt]

Another problem is that the building is an old factory converted to an open office. It has large metal pillars and some parts of the floor is pure metal. Sometimes I can get Wi-Fi signals from building that are over 200 meters straight line of sight away. So there is a ton of noise.

[dreamy veldt]

Yeah I think so too. I have reset the whole network yesterday and added the second otbr today. So for now it actually seems better.

[kind sedge]

I assume you mean Apple BR and not otbr right ?

[dreamy veldt]

But I can export the complete matter logs if it gives you some insights

[dreamy veldt]

Yea

[kind sedge]

ah yes that sounds like things can go wrong indeed

[dreamy veldt]

Not using the otbr right now because some devices didn’t get assigned IP’s anymore.

First when I restarted the otbr

That’s probably a bug in the otbr in combination with all the noise

[ebon olive]

I can’t wait for it… 😉

[kind sedge]

thanks for the reminder, almost forgot 🙂

[ebon olive]

I can only confirm that. Even though some devices are unavailable in HA since the latest changes, it’s really fast to reestablish the connection for available devices now.

[kind sedge]

New matter server is released so please give it a good test drive 🙂

[ebon olive]

When will we see the update?

[kind sedge]

now

[ebon olive]

Thank you so much for all the work you guys investigate to make this work.

[kind sedge]

just update the addon store

[ebon olive]

Yes, it’s there. From the first view it seems to work. At the moment I have 11 EVE Matter devices shared with HA. All 11 devices came up within one minute. Didn’t ever see it that fast ever before. 🔥🔥🔥

But it’s to early to say something about the reliability/stability. I also have 11 other EVE Matter devices paired to Apple Home, that I want to share with Home Assistant. I will do some further tests, when my wife is in bed. 😇 Thank you

Ok, stopped my Matter server and restarted it. Again all devices were there within one minute. If it really works that well, it is fantastic. ❤️😃👍

I am going to share all my EVE Matter devices from Apple Home to HA tonight.

After that I share my results here.

Ok, third try. Again, all devices are there within one minute. This is fu**ing amazing crazy! 😂😂😂

In the past I sometimes had to wait more than an hour…

[swift void]

Looks good cuts down the wait time for devices to reconnect after power off to 10 minutes as expected a welcome improvement over an hour

[ebon olive]

@kind sedge Congratulations! Well done! 😃👍 Never had such a great performance with my HA Matter devices. I shared the other 11 EVE Matter devices with HA, I already had paired to Apple Home. So in sum 22 EVE Matter over Thread devices (13 EVE Energies, 2 EVE Motion and 7 EVE D&W) now.

Stopped/started and rebooted my HA. In both cases all 22 devices came back within under 3 mins. Great! Thank you so much. 🥳

[kind sedge]

very nice to hear!

[ebon olive]

The issue where devices change its state between available and unavailable seems also to be gone. 😃

I had this issue always after I shared a new Matter over Thread device from Apple Home to HA. In the last hour I shared 11 devices from Apple Home to HA and the devices are all available without issues.

[kind sedge]

oh wow in that case my hunch was right about an interlocking issue

I stumbled on that while fixing issue 1

and just to be safe I guarded simultane access to the node

so if that fixed issue 2 that is even more great

[ebon olive]

Let’s wait some days and see, but at the moment, I am absolutely happy. I am also interested in the feedback of others. 😉

By the way… I have Unifi network and my HAOS runs in a QEMU VM on Openmediavault 6 (Debian).

[kind sedge]

Yeah, this so far sounds good, lets wait and see if others experience the same.

Yeah I know you're also running Unifi gear. For me it works stable since i removed the mdns forwarder

Enabling that and it goes haywire within minutes

[ebon olive]

Yes, after all your comments, I also disabled it all. mDNS, IGMP Snopping and Multicast Enhancements…

[kind sedge]

Running a VM is perfectly fine on a NAS OS 😉

[kind sedge]

I actually turned IGMP snooping back on on the UDM

And I commission a lot of devices every day (well the same ones over and over haha)

and it keeps working

[ebon olive]

Ok, I wasn’t sure about it. The last days I read somewhere that it’s better to run HAOS bare metal. Thanks for clarification.

[kind sedge]

I have now a seperate 2,4ghz ssid for the iot gear

[kind sedge]

No, that was to try out if the hypervisor was to blame for that specific issue

I think a lot people run haos as vm, especially proxmox is very popular

[ebon olive]

Ok, good to know. But at the moment I don’t see any need for IGMP Snooping. Everything works very well.

[kind sedge]

I considered it once but I just run haos on my NUC, all I need is in the addons nowadays anyways

[kind sedge]

in that case leave it. I had to enable it because some google cast devices were acting up

[ebon olive]

I also have a separate IoT VLAN and since I disabled mDNS completely I miss some entries in the mDNS browser for my Philips Hue Bridge, my Xiaomi Fans/Vacuum/Heater and my Bosch Home Connect Refrigerator for example. But all devices still work without issues.

[kind sedge]

yes, just give them a static ip and be done with it

[ebon olive]

Ok, I don’t use google cast. Apple and Sonos only at the moment.

[ebon olive]

Yes, that seems to be enough.

[kind sedge]

sonos is also pretty sensitive for multicast traffic so may also help to enable igmp snooping just to not flood the network with multicast traffic where it doesnt belong

[ebon olive]

Ah no… I do not complicate my life where it’s not necessary. All my Apple devices, my Sonos devices and my HAOS are in my Main VLAN. There are special VLANs for:

• network equipment
• untrustable devices
• trustable devices
• protect devices
  At the moment I do not see an issue with all the settings disabled.

When I see some issues, I will report them here or better in the normal #matter-archived channel. 😉

Thanks again, peace! ✌️

[swift void]

While doing the packet capture I noticed that when mDNS was on at least Wi-Fi looked fine but on Ethernet the ubiquiti was duplicating them like crazy setting the port to block other vlans resulted in that calming down to just one copy per vlan

[ebon olive]

Sorry to write it again. Did one last test, stopped/started my Matter server again. All 22 devices where back up in under 1 min and 20 seconds. 😱 What did you do @kind sedge ? This is so amazing 🥳🥳🥳

[kind sedge]

yeah something is funky about it and it just shouldnt be enabled if you see what it does.

[ebon olive]

Which ports do you mean exactly? 5353 and 5540?

[swift void]

Sorry I mean the Ethernet ports on my UDM you can set a traffic restriction. Since all wired devices are on main network.

[ebon olive]

Ok, so you used the port profile ’All’ for your end devices? All my switches are configured to use 802.1x, all my end devices use Radius to auth and thus every port allows the traffic for exactly one vlan.

[swift void]

Yeah my setup is simple. Like Marc said mdns reflector in UniFi seems borked. I just did a packet capture on my devices for mDNS and saw the issues. Maybe in your setup it might not duplicate messages as much

[ebon olive]

Ok, good to know. As already said after all the comments here I disabled all mDNS related functions…

A lot of Unifi users here. That’s good. We are all in the same boat. 😉 Have a nice evening

[swift void]

2023-07-12 18:00:48 core-matter-server chip.IN[126] DEBUG SecureSession[0x7f45c00b06b0]: MarkAsDefunct Type:2 LSID:6166
Exception ignored on calling ctypes callback function: <function _OnResubscriptionAttemptedCallback at 0x7f45d45a45e0>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/chip/clusters/Attribute.py", line 881, in _OnResubscriptionAttemptedCallback
    closure.handleResubscriptionAttempted(terminationCause, nextResubscribeIntervalMsec)
  File "/usr/local/lib/python3.11/site-packages/chip/clusters/Attribute.py", line 738, in handleResubscriptionAttempted
    if (self._subscription_handler._onResubscriptionAttemptedCb_isAsync):
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I saw this error in debug mode. Not sure if its related but after testing it seems that this specific device took longer than 10 minutes but not much longer to connect the rest were fine and looking through the logs this is the only error I saw

[kind sedge]

Yes that is an actual bug in the sdk's python wrapper. Not related to your issue though.
So all devices came back online then ?

[swift void]

Yes within 10-15 min of startup

[kind sedge]

yes, that is expected when you completely power them off. It will improve once we have operational node discovery in place in a few months or so.

[quiet grotto]

I'm real sorry for being a pain Marcel, but I just got your latest code and running it here, but getting

2023-07-13 11:42:04 homeassistant matter_server.server.helpers.paa_certificates[1] INFO Fetching the latest PAA root certificates from DCL.
2023-07-13 11:42:04 homeassistant matter_server.server.helpers.paa_certificates[1] INFO Fetched 0 PAA root certificates from DCL.
2023-07-13 11:42:04 homeassistant matter_server.server.helpers.paa_certificates[1] INFO Fetching the latest PAA root certificates from Git.
2023-07-13 11:42:04 homeassistant matter_server.server.helpers.paa_certificates[1] INFO Fetched 0 PAA root certificates from Git.
2023-07-13 11:42:36 homeassistant chip.DIS[1] ERROR Timeout waiting for mDNS resolution.
2023-07-13 11:42:50 homeassistant chip.DIS[1] ERROR OperationalSessionSetup[1:0000000000000003]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout
2023-07-13 11:43:21 homeassistant chip.DIS[1] ERROR Timeout waiting for mDNS resolution.
2023-07-13 11:43:35 homeassistant chip.DIS[1] ERROR OperationalSessionSetup[1:0000000000000003]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout
2023-07-13 11:44:06 homeassistant chip.DIS[1] ERROR Timeout waiting for mDNS resolution.
2023-07-13 11:44:20 homeassistant chip.DIS[1] ERROR OperationalSessionSetup[1:0000000000000003]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:114: CHIP Error 0x00000032: Timeout

all of that looks wrong, obviously, but I'm trying to figure out if it's the matter server, my rpi, or thread router

[kind sedge]

Yes so you still have mdns resolving issues

[quiet grotto]

yep

but also those cert issues? can they be ignored too?

[kind sedge]

Well, those are not issues

[quiet grotto]

I've made progress from yesterday, but still trying to nail down some of these errors, I think if I can figure the true cause of the mdns stuff, most of the rest will start to resolve

cool

[kind sedge]

Did you do what I said and install HAOS on a host as a start ?

If that works you can start toying with other stuff but you are overcomplicated it all, there are now 10000 possible causes for your issues

start simple and work from there

does HAOS on a pi work ? great, keep that or move on part by part

if even HAOS on a pi does not work, its network related

[quiet grotto]

okay yeah, I've the second SD Card on this one, so I can just install it onto there to vanilla test

you're totally right, if this is network related, I've been digging into all the wrong places

[kind sedge]

yeah exactly my point

just pick that extra sd card, load HAOS and start testing with that

if that works, you have a valid data point

and you can compare difference between HAOS and your self brew setup

if that does not work you'll have to start digging in your network config

[quiet grotto]

that's one of my main gripe points about HAOS, I found it very cumbersome to get into the nitty gritty of it, when I had it running

but, I'm on my way to installing now

[kind sedge]

it's a appliance, no need to get into the nitty gritty of that 🙂

[quiet grotto]

I know, I know, but that's exactly why I can't run iOS either... I like to take the lid off of everything I play with

[kind sedge]

Been there, done that. Now happily using iOS and HAOS 😎 and picking my battles elsewhere haha

But I get your point

Lots of people run proxmox for that reason btw

so they can have a VM for HAOS and other vm(s) for toying around

[quiet grotto]

but, this is the thing about having things properly running on RPis, or lesser. I would personally like to have a fully running, open sourced multiple just thread border routers around the place running all their own stuff

I specifically want to start prototyping on different bits of tiny hardware

[kind sedge]

Get the espessif dev board in that case

they released a board that can be used as otbr

[quiet grotto]

yep, it's as much "in my cart" as something can be

[kind sedge]

or maybe pi zero devices can be used too but they're still hard to get

[quiet grotto]

but this is thing about otbr now, I think what I have "should" be working now, already

yeah, though my Dad has about 10 of 'em haha. started getting the "SUPER" pis from alibaba and the like a year or so ago, they were just all pi zeros

[quiet grotto]

Installed HAOS, loads of things auto discovered, Thread installed, but not OTBR, no Thread network nor router shown inside the configure section of Thread. Re-read the HA OTBR docs and it states that it's only used for the multiprotocol stuff, so enabled and started that, waited a bit and then tried to add - same old, same old

I then tried sharing already Threaded devices with HA, and that, for the first time, has actually worked

My Thread networks are a mess... https://i.imgur.com/pvmTxNa.png

So yeah... kind of working, but realistically, not really

[opal linden]

In that scenario your thread integration is out of sync with your OTBR instance. It used to happen when people changed settings via the OTBR web interface, which is notorious for being problematic to the point that upstream were talking about removing it AIUI. Anyway, regardless of how, the state got corrupted. There was supposed to be Ui to help recover from states like this, but the Pr I was aware of got dropped (around consensus of where the UI for it should go).

There is a manual process you can follow

Unless someone can point out where that feature has ended up

Basically, remove (backup) the thread.datasets file from /config/.storage and restart HA. That will make HA forget it’s current (wrong) preferred network.

Hopefully it will automatically sync with the OTBR instance and start up and pull the new network, but if it doesn’t (it used to only do this sync once ever), then you need to delete the OTBR integration (not addon) and re-add it

If you need the OTBR url, last time I did this it was #thread-archived message

[swift void]

Good to hear hopefully they merge operational dns discovery soon!

[pulsar sleet]

@kind sedge @solid mountain @tender oxide @true crown shocking response from apple

Hello Robert,

Thank you for contacting Apple Developer Technical Support (DTS).

We believe this issue is a bug. Please file a bug report via Feedback Assistant (https://feedbackassistant.apple.com). For more information on Feedback Assistant, visit https://developer.apple.com/bug-reporting.

Be sure to test on the most recent version of the OS available, including beta/seed releases if available. Include the results of your testing in the bug report. Before upgrading a device to a beta, be sure to make a backup first because restoring it to the current release requires erasing the device first.

Important: For feedback related to a specific framework or API, select "Developer Technologies & SDKs" as your Topic, then select the specific technology and relevant OS.
For feedback related to Xcode, App Store Connect, or other developer tools and resources, select "Developer Tools & Resources" as your Topic, then select the specific tool or resource.

While your bug is being investigated by the relevant engineering team, if you would like us to use the TSI to search for a possible workaround, please send us the Feedback ID number in a reply to this email.
Discovering a workaround can take a significant amount of time and a positive result is not guaranteed.

While a Technical Support Incident (TSI) was initially debited from your Apple Developer Program account for this request, we have assigned a replacement incident back to your account.
If you choose to request a search for a workaround, we will then re-debit a TSI from your developer account and assign your request to a DTS Engineer at that time.

Best Regards,

Developer Technical Support
Worldwide Developer Relations
Apple, Inc.

we should be able to get the credentials for apple's thread network from that API

[fast wigeon]

Hi,

I setup a raspberry pi to act like otbr (installed openthread border router) I can see it discovered in Home assistant under "Other networks". I want to combine that otbr into my home assistant's (skyconnect) thread network so that I have multiple tbr in the same network. However, in order to add otbr to HA network it needs Dataset in TLV format for otbr. From where can I get Dataset in TLV format from otbr?

[opal linden]

User support is in #matter-archived @fast wigeon

[pulsar sleet]

i'm not going to be able to submit a feedback for another week, currently on vacation in a place without a thread network

[kind sedge]

enjoy your vacation!

[barren flame]

Great news. And this makes sense -- how else could the Nanoleaf app make their BR join the HomeKit network, which they did for some of their products (in my case an Elements) with their latest firmware.

[solid mountain]

FYI, the Wireshark dissector for Matter from Martin is now public https://github.com/project-chip/matter-dissector

[modest prawn]

so they did beat me to making the dissector useful 😛

I got distracted with a dozen other things and didn't really make any progress...

[modest prawn]

...otoh it doesn't look like it's suitable for upstreaming

[versed hazel]

Not sure where to ask, but is there a way to build the matter-addon server on armv7 myself? it seems not to be officially supported, but i couldnt find any reason in the source code why it would only work on 64 bit architectures.

[kind sedge]

No the sdk itself does not run on 32 bits... It's 2023, time for 64 bits

[versed hazel]

i have multiple Odroid XU4 deployed in various homes of parents and extended family 🙈🙈🙈

but i am right that i need the matter server addon to use thread-matter devices with the XU4 as OTBR?

its a little bit sad, that matter wouldnt work while thread itself does...

[kind sedge]

Yes that is correct

[unique peak]

does this support matter as a controller https://www.home-assistant.io/yellow/

[opal linden]

It has a thread radio and can run OTBR, much the same experience as with SkyConnect.

Anything running HAOS and the matter addon AND integration can control matter devices, when everything is correctly configured

There is a #yellow-archived channel that might be a better place to ask about that hardware

Or #matter-archived or #thread-archived for support questions with those protocols

[lapis hinge]

Hi,
I then tried using the environment set up as specified in the chip-wheels repo (chip-wheels/.github/workflows/build.yaml) and tried to run the python-matter-server again. But now I am getting an initialization error at startup.

'
chip.exceptions.ChipStackError: src/system/SystemLayerImplSelect.cpp:45: CHIP Error 0x00000003: Incorrect state
Segmentation fault (core dumped)
'

Many thanks.

[lapis hinge]

Solved: I found that I needed to create a data folder with the same permission as /tmp in the root directory of my ubuntu install. I will create a PR to update the python-matter-server repo's README file and the admins can include this info if they feel its needed.

[kind sedge]

@lapis hinge as clearly stated, the matter server is not intended to be run by you manually, too many moving parts involved.
You can run the matter server by using the HAOS add-on or the docker container. Running it directly is at your own risk and will get you in trouble due to all the dependencies.

[lapis hinge]

Hi, yes. I understand. However I am aiming to use the python matter server for another project outside of HAOS. I am familiar with all the dependencies. As per the readme this appears to be "allowed" - 'The goal of this project is primary to have Matter support in Home Assistant but its universal approach makes it suitable to be used in other projects too.' I was writing my own python controller for a personal project until I came across this project. I am very happy to contribute where I can.

[kind sedge]

Ah all clear and very much appreciated of course!

[lapis hinge]

Great stuff! While I have you.. I think I found a code bug in the matter server when the open commissioning window command is called. I will raise an issue and PR on the GitHub repo when I get a chance. Just wondering if any matter users are reporting issues with opening a commissioning window on a device post initial commissioning onto the fabric?

[kind sedge]

We're not using open commission window yet in HA - we're first designing the whole flow to make the commission process more user friendly in relation to wifi vs thread devices, utilize the bluetooth adapter or the iOS/Android SDK etc. In that process we're also going to implement the "share feature" which will ofc call the "open commissioning window".

Last time I checked it worked but if you find a bug, please let me know on the github issue tracker for the python-matter-server project, thanks!

[ebon olive]

Is Homey Open Source?

I watched the following YouTube video from Shane Whatley and at 9:30 minutes he showed how it’s possible to pair a Matter device straight to Homey:

https://youtu.be/Y6jGIVRv-7Y?si=PK47F5OdencK6Ctv

When it’s completely open source, maybe one of the HA devs can look at their code to get it solved for HA. What do you think?

[true crown]

No, Homey is closed source/for profit

Pairing a Matter device like that is already supported in Home Assistant

[ebon olive]

Ok, what a bummer.

[ebon olive]

Ok, I thought this was not the case. All my Matter devices are paired to Apple Home and then shared with HA. Ok, thanks for the enlightenment. Have a nice evening 😉

[summer basin]

hey guys looking for a way to put my eve energy outlets into homeassistant. They are all on matter and I can see them in apple home... but no sign of it within homeassistant. Anyone knows how to get them into HA? I can scan the code via the phone but it doesnt find the devices

[true crown]

For support, ask in #matter-archived

[kind sedge]

That is just the way you commissioned them. You can safely remove them from Apple Home and they keep working. We are working on ways to improve the commissioning

[rapid niche]

What's the current state of aligning different vendor credentials on thread networks? It would be awesome if I could view my setup in more details such that I know if some TBR device is not being used in my ha network because the credentials are different. And the alignment to form a single network is done transparently (for vendors that provide the operational dataset in TLV format)?

Also, regarding multiple Matter Controllers from different vendors -- if I initially pair a matter enabled device on HomeKit, and then I want to add it to Home Assistant, I know that I have to use HomeKit's pairing functionality, but the thing is, after I add it to Home Assistant, is there a "main" controller or each controller will work individually and if one is deleted or stops working it won't affect the other?

[opal linden]

Merging thread networks is a PITA and might not be a win. For example, right Apple BR's have something called TREL which should pretty much eliminate "mesh splits". But a Nanoleaf BR does not. So adding one to your Apple mesh can actually be a net negative. In terms of seeing the current state of play though, you can see the current TBR's and the networks they are on in the HA UI. Find the thread integration and "configure" it. You should see a UI with the information.

[dreamy veldt]

It looked like TREL might get mandatory in thread 1.3.1 though.

So in the future it will probably be a win

[dreamy veldt]

If you delete one controller it doesn’t affect the others

[kind sedge]

The nice thing about Matter is that it is actually "multi fabric" which means multiple controllers can manage it independently. so if you paired a device to Apple Home first, then shared it to HA, you can safely remove it from Apple Home. HA will be controling it completely on its own.

[kind sedge]

As already explained by @opal linden our original idea of trying to merge the thread networks is probably not going to hold up. We better wait for the members of the Thread Group to resolve this someday where we can finally have the benefit of all thread border routers forming one big network (or at least are capable of routing to eachother).

[rapid niche]

@kind sedge so that means that all that magic advertised about any device you have that acts as a TBR can be used in your home setup is bullshit? In simpler terms, Apple TV / HomePod can act as TBR for some devices but not for others? That's weird because my HA shows all my TBR devices (from Apple, Google, Home Assistant Yellow) on the thread integration page

Or... did you mean forming one big network in the sense that multiple thread networks can be seen as "disjoint" graphs except for the TBRs nodes, so that the only common node they are linked to are TBR nodes? And merging the thread networks would be essentially having a single connected graph without redundancy

[kind sedge]

Yeah, the promise of Thread of forming one big ecosystem indepdendent network has not yet delivered. So you will have a Thread network with Google Thread border routers, used by Google Home, you have the same for Apple and so on. So better extend devices on one of those ecosystems to form a more strong network as in the near future they wont work together

[rapid niche]

@kind sedge can you take a look here? It's the view of my thread networks from HA https://i.imgur.com/aHRyHK3.png

You can see that I set "Silicon Labs Multiprotocol" and "Google Nest Hub Max" as my options for android+ios credentials

But now I realized that they are under "Other networks", so each one of those groups do not interact with the other, and my devices on Apple Home that rely on thread will only use my Apple TBRs, and for Google the same thing? If that's the case, I will rethink relying on Home Assistant sillicon labs chip which is still in beta for being the TBR as you're basically saying that all my thread devices on Home Assistant will only use my home assistant yellow as TBR?

[opal linden]

With that setup, any devices you added to matter with iOS will be on your HomePods. Even if you added them to ha. Any that you add with android may be on nest or may be on yellow. But if they are on yellow, yes they will only be using that one TBR.

[rapid niche]

I installed Leviton Decora switches on all my apartment which recently integrated with matter, and for some reason, when adding to HomeKit first and turning the pairing on for them, they fail to connect to Home Assistant. Tried for hours. So my workaround was: add to Google Home, then pair from Google Home to HomeKit and to Home Assistant. Everything else is added to HomeKit first and Home Assistant second. I never add to Home Assistant first because I'd have to generate the pairing code which involves going into the matter controller and I didn't want to go into that trouble for something I would do for a lot of devices

I'm glad I didn't add them to Home Assistant first then... The location of my home assistant is kind of in the corner of the farthest room in the apartment and I really thought that wouldn't matter because, unless I'm fucking stupid, what we are talking about here is rarely disclosed publicly, the hype I read everywhere is that it shouldn't "matter" because the device will use the most optimal path to any TBR in your network

[opal linden]

Before iOS 17 that wouldn’t even have been true if you’d just had 3 home pods 🙂

[covert saffron]

My question is probably better suited for this channel: #matter-archived message

feel free to berate me for my naive question

[dreamy veldt]

What was the bug exactly?

[covert saffron]

I've opened a bug for the Matter integration and I think I see where we can implement a fix - the bug I opened is here: https://github.com/home-assistant/core/issues/102491
The place we should probably check if the heat/cool range is valid: https://github.com/home-assistant/core/blob/dev/homeassistant/components/matter/climate.py#L125-L156
I don't have much experience with thermostats, but I would assume thermostats can have a different range for HEAT/COOL mode.
Is there a practical way to query for that value from the Matter device?

It could be hardcoded/mapped on a device-specific setting if we don't have a way to query for that info.

[covert saffron]

I added some info on the bug report and I think it will be possible to query for that info. I was able to connect via chip-tool but wasn’t able to connect via the client Python module since my credentials weren’t working.

[dreamy veldt]

Did you try to use the bypass certificate option in the chip tool? I think pr default it only has developer certificates. You could copy the production ones into the right folder or just choose to ignore option.

[covert saffron]

On chip tool I used the root-paa-certs in the development folder

Wasn’t sure how to translate using that command line flag to the set_thread_dataset command

[covert saffron]

I was wanting to test the Python matter client to see if I could send a bad request to the device

[covert saffron]

I could probably achieve that using chip tool but I figured it would be easier to use the Python library

[dreamy veldt]

You could just use multiadmin

Then you dont need to set the thread dataset

[covert saffron]

Well right, but the Google Home app already has some logic to ensure it doesn’t send a bad request

Assuming that’s what you mean by using multiadmin

[kind sedge]

I'll have a look at your issue report within a few days, probably just a matter of adding a guard. Maybe the device even reports its limits within some attributes, I did not check

[covert saffron]

As far as I know the Nest Thermostat (the cheaper one from Google themselves) has a range of 3F. I would share a screen capture but I don’t have permissions

I thought it would be the MinSetpointDeadband but that value reports back “16” from matter when querying cluster attributes

Which I guess could have some sort of scale conversion…

I’m happy to help with research and even with the solution

By not matching, I mean 16 doesn’t really translate to a 3F degree range. 4 would make sense

But you can go down to 3F difference on the thermostat, i.e. 70F-73F range

Maybe it is some sort of scale value… the code around climate suggests that thermostats have a scaling of 100

But I think that’s returned in Celsius

If the MinSetpointDeadband is in Fahrenheit that would give it a 5.3333 scale ratio… but that seems messy with rounding. A lot of assumptions in that take

We could probably derive what that value means if we had another thermostat of a different breed, or the other Nest thermostat that nest themselves makes

[kind sedge]

Did you look at the cluster specification if there is an attribute that reports the device limitations ?
If you download the diagnostics of your device in HA, you will get the raw data keyed by AttributePath: EndPointId\ClusterId\AttributeId

[covert saffron]

I glanced at it over lunch

I’m off in a bit and can look into the diagnostic data more, but I couldn’t find where the attribute IDs mapped to their values in the diagnostics

[covert saffron]

I'm looking at this attribute, however, when I try to use the chip-tool, I'm getting a suppressed response. Maybe something in my configuration is wrong

Diagnostics: "0/40/3": "Nest Thermostat"
Command I ran: ./matter-build/chip-tool any read-by-id 0x28 0xFFFFFFFF 1 1
Response:

[1698192108.539219][102781:102783] CHIP:DMG: ReportDataMessage =
[1698192108.539223][102781:102783] CHIP:DMG: {
[1698192108.539226][102781:102783] CHIP:DMG:     SuppressResponse = true,
[1698192108.539230][102781:102783] CHIP:DMG:     InteractionModelRevision = 1
[1698192108.539236][102781:102783] CHIP:DMG: }

I guess that may not be what I thought it was, but I don't seem to be getting a response when I try to read the value

[cobalt belfry]

@covert saffron I converted your message into a file since it's above 15 lines :+1:

[covert saffron]

aha, I'm just not good with using the chip-tool. Figured it out, I was hitting the wrong endpoint

These endpoints in the Spec don't seem to work though... may be me failing to use the data correctly.

[kind sedge]

No, it will just mean your device doesnt support the those attributes

[covert saffron]

I was playing with the humidity ones last night. I don’t remember if I tried to find the endpoint for the range

I did find in Google’s thermostat Matter docs that everything from the cloud service should be mapped? One sec and I’ll link

https://developers.home.google.com/matter/clusters

https://developers.home.google.com/cloud-to-cloud/traits/temperaturesetting

If that truly is the case, maybe whatever bufferRangeCelcius is mapped to in Matter would be what we’re looking for

[kind sedge]

isn't that the other way around? how a (lcoal) matter device is mapped to the google home cloud ?

It does not say anything about your thermostat which is also google but that is just coincidence

your themostat just implements a few clusters and attributes and a lot of attributes are optional

you will need the matter cluster specification, not the google specs

From what I know the google nest thermostat matter mappings are very minimal

so chances are big they only mapped the very bare minimum

[covert saffron]

That would make sense considering Matter support came in a software update

[kind sedge]

did you check your device dump what attributes it reports ?

Hoem Assistant --> Settings --> Devices & Integrations --> Matter --> devices --> Your thermostat --> downlaod diagnostics

[covert saffron]

Yeah, last night I was deciphering the attributes #devs_matter-archived message

[kind sedge]

the resulting json file includes the full raw dump

[covert saffron]

I was able to figure out my issue there though, but I think the endpoint that was supposed to be for MinSetpointDeadband weren’t returning any results; not even an UNSUPPORTED_CLUSTER response

[kind sedge]

if its not returning, the device is not returning the results. Hence I ask to check the device dump as that is from a full interview

so your device is not returning that data at all, not really a surprise there but we should then assume a default in our integration

my suggestion for now is to use the 4 as minimum which you discovered

[covert saffron]

I think it’s actually 3. I’ll try and query for it later after I’m off, but if it doesn’t work that sounds like a good plan. Would it be good to query the device information to make sure it is indeed a Nest Thermostat? Since this value is probably specific to this Brand/Model

And by query later, I’m going to try and confirm I’m not using the tool wrong

But yeah

I’ll share my diagnostics in here shortly, but I was converting the decimal values to hex (I believe for cluster and endpoint) and trying to find what they mapped to in the spec. The endpoints I was thinking were what we needed didn’t show up in the diagnostic data

[dreamy veldt]

have you tried to read the spec on it and how its supposed to work?

(the thermostat is still running on the 1.0 spec)

[covert saffron]

Yeah the 1.0 spec is what I’ve been following

[dreamy veldt]

Yeah okay, i would rather use this to understand the clusters exactly than querying the device. They need to use the clusters according to the spec to get certified.

[covert saffron]

I'm following the spec of the thermostat cluster. MinSetpointDeadband seems to be the closest possible attribute, but the value coming back doesn't make sense

It's returning 16, as a signed temperature

a signed temperature is (temperature in °C) x 10

meaning it would be 1.6C

which is 34.88F

So yeah, idk. If it was 3.488F (rounded down) it would kind of make sense but something doesn't seem right about that.

This field has to have a value from 0 to 2.5C.

well, since i'm here to muddy the waters more, 3F is actually -16.1111C

😬

If that's truly the attribute we want, then that means it's formula is -(MinSetpointDeadband) degrees C, but this could all be coincidental at best.

And based on Googling 'Thermostat deadband', this really does seem like the exact value we want... it's just that it's in such a strange format

https://feds.pnnl.gov/faq/what-thermostat-dead-band

And here's a video describing deadband too: https://youtu.be/1A7aln4urOQ

[kind sedge]

Nice findings!

[faint idol]

Hey there! 👋🏻
I'm a newbie on this server, so I posted this message on another channel, but I guess here is the right place to ask this 😅
#matter-archived message

[kind sedge]

I have experience with the Matter codebase but already swamped with work so unfortunately no time left to support you for now. You could have a look at this discord server where more DIY matter devs are active: https://discord.gg/4BXunJyH

[faint idol]

Perfect, thanks! 🙂
The most important question for me right now, before I start writing my thesis, is just about the amount of effort and if there are any show-stoppers. But I will ask on the other Server 🙂

If there are any other tasks related to HomeAssistant or Matter, which is suitable for a Bachelor Thesis, let me know!

[covert saffron]

Thanks, I guess now that I have this, I'll make the changes and open a PR. I'll add a code comment mentioning that this may not be the correct way to do it and should probably be the point to reevaluate if users start opening bugs

I'll make sure that if MinSetpointDeadband doesn't return a value, a default will be used (perhaps 3F)

[dreamy veldt]

Depends on what you are trying to accomplish. There are still a lot of showstoppers and bugs, so don’t try anything that is not used a lot yet. The implementation for for example bindings is horrible at the moment. So stick to what is used by devices on the market.

[covert saffron]

I've got a PR up for the thermostat deadband logic... and it's got some ideas going

[kind sedge]

If there are any other tasks related to

[lunar flame]

hi have cloned this python matter server repo : https://github.com/home-assistant-libs/python-matter-server
after that i need to do some development from the readme i followed the below steps.

1)Download/clone the repo to your local machine.
2)Create a Python virtual environment.
3)Install the correct SDK wheels for both the cluster and core package, see instructions above if there is no wheel for your setup prebuilt.

for the third step i did the
pip install home-assistant-chip-clusters
pip install home-assistant-chip-core.
is my appraoch is correct ?and i would like to know after code modification how to build wheels out of it

[kind sedge]

Wheels are being built by our CI task, you can check the details. So if you made any improvements you can PR your changes and the wheels will be built.

[lunar flame]

(python-matter-server) rntbci@inmh100170:~/home_assistant_addon_devlopment/python-matter-server$ python -m matter_server.server
im getting segmentation error
chip.exceptions.ChipStackError: src/system/SystemLayerImplSelect.cpp:56: CHIP Error 0x00000003: Incorrect state
Segmentation fault (core dumped)

[kind sedge]

What OS ?

[lunar flame]

ubuntu

i dono why cant able to sent the log in chat

[kind sedge]

I assume you have installed the chip sdk wheels ?
Are you planning on developing for the matter server ?

[kind sedge]

OK, Ubuntu on 64 bits Intel machine ?

[lunar flame]

these are the steps i followed so the chip wheels are installed i guess

[lunar flame]

yes

[kind sedge]

Maybe you need sudo

lots of access rights needed

I'm developing on MacOS and there it works great

Linux we only use for the docker container

Also make sure that config directory is writeable

[lunar flame]

okay ,i need to read electrical measurement cluster on my all-clusters-app example , i checked the code and in client/model/device type.py ,i found that all cluster app does not include electrical measurement cluster, so i need some ways to read the eclectrical measurement clusters and its attribute and to show it in overview of home assistant

[kind sedge]

No, that is matter 1.3

That does not yet exist

So you will have to wait a bit

[lunar flame]

so will come under custom cluster ?

[kind sedge]

Our Matter server is based on 1.2 atm

[kind sedge]

We do support custom clusters yes

But only if it makes sense

You will have to wait until Matter 1.3 is released, which includes electrical measurement

It does not exist now

[lunar flame]

ok @kind sedge , now next doubt is the docker file in python server , when i build it will take the source codes from local repo ?

btw thanks for your reply

[kind sedge]

Why would you want to build a docker container as I just told you its not possible to do what you want.

Electrical measurement cluster is matter 1.3

[lunar flame]

just for understanding the docker file

[kind sedge]

It's built by a CI action in the same go as the py wheels. Just see the github actions

[lunar flame]

ok

i think this would be the solution ryt?: https://github.com/home-assistant-libs/python-matter-server/issues/235

[kind sedge]

Yes, either the data directory or the ssl. Data dir needs to be writeable (as I said above)

[modest summit]

@kind sedge have you tested out motionblinds ?

[kind sedge]

Unfortunately not, I do not have access to a test device.

[modest summit]

Would you be interested in one ?

[kind sedge]

yeah, I'm always interested in test devices as actual devices are really sparse here in NL

[modest summit]

Can i send you a DM ?

[jaunty saffron]

while I was looking through some docs/code yesterday I noticed that this endpoint is paginated: https://github.com/home-assistant-libs/python-matter-server/blob/main/matter_server/server/vendor_info.py#L52 - so it seems that any devices with vendors on the second page wouldn't have their vendor info available? unless I'm missing something.

i still haven't dug in enough to figure out where this is used. the node info i get back from the server has the vendor info already, but I'm not sure if that comes from the devices themselves or is populated based on this data (all my devices happen to be on the first page). if anyone has a google device, can you confirm if the vendor info is properly populated?

api docs are available here: https://on.dcl.csa-iot.org/#/Query/VendorInfoAll

[kind sedge]

the vendor info is basically just retrieved from DCL - its not use dyet but the whole idea is that you can show which fabrics are joined to a device (so match vendor id of a farbic to that list)

I don't know if the API is also paginated and if it does, I think this was implemented when there were only a few record in that list 🙂 So, if records are misisng now, let me know and I'll adjust the code to get all results

[jaunty saffron]

I can make a PR to fix it - was just confirming that it's actually an issue. I guess if it's not really used yet it might not be noticed. Google is the only vendor I even recognized on the second page

[kind sedge]

Yeah, its not in use yet so I guess nobody noticed 🙂

[jaunty saffron]

if I call discover api on the server i get a response like details : "Type is not JSON serializable: CommissionableNode" error_code : 0

pretty sure this is coming from the server and not my code

[kind sedge]

discover api call is still broken

see https://github.com/home-assistant-libs/python-matter-server/pull/448

it's not in use yet but will be useful at some point

that PR is open for a while, waiting for the comments to be addressed btw

[jaunty saffron]

ah ok - you had mentioned a UI would want to call it - so i was testing it out

[kind sedge]

I see you already fixed it, nice !

[jaunty saffron]

something about my setup isn't working right, possibly network related - I know the "approved" way to run the containers is via HAOS, but since I'm trying to run code still in development, is there a way to get the HAOS in a dev mode that I can access the container directly to run code i'm working on? or is there some other base image I should be using? I have a raspberry pi dedicated to this, so I can reconfigure it - just not sure the best dev setup

matter-server  | 2024-01-05 15:47:06 sdk4 chip.EM[1] ERROR Failed to send Solitary ack for MessageCounter:187589243 on exchange 37610i:src/messaging/ExchangeContext.cpp:103: CHIP Error 0x00000002: Connection aborted```

[jaunty saffron]

actually - i just implemented doing this from the ui and it worked

[kind sedge]

I'm not sure anymore what you mean.
On my dev machine (macos) I just run the whole stack in a venv - works well because macos is optimized for ipv6 and mdns.
On a very modern and updated host OS you can also run it in a container on on a linux host in an venv just like my mac machine.

The images are built by our CI job on release of a new version. Those images we test on a dedicated test HAOS machine but daily environment testing is doen locally on my machine.

In your log above I see some communication issues. Was that resolved ? Not sure what you mean with "i just implemented doing this from the ui and it worked" ?

[jaunty saffron]

basically when I try to commission from my iphone, I was getting those messages above. when I do commission_with_code from the websocket api, it works. I don't understand why there would be communication issues with one mechanism but not the other since they should be doing the same thing apparently

i assumed when you said there were communication issues that i may need to figure out how to get closer to the "approved" haos setup. I setup my system to work correctly, but it's kind of custom - so I wasn't sure if that may have drifted. but since it does work via the api, now I'm not sure if it is a networking issue afterall

separately - where would I find good documentation on defined clusters? for example, I can see my motion sensor updates this attribute: 1/1030/0 . If I understand correctly, 1 is the endpoint, 1030 is a "cluster", which should map to the function/type of device, and 0 is an attribute within that cluster. Is there a list of all these predefined values somewhere?

[kind sedge]

hmmm interesting... did you also use it with "network_only" set to true ?

[kind sedge]

Yes, I told you a few days ago the key is in AttributePath format:
<EndpointId>/<ClusterId>/<AttributeId>

You can lookup the specification to know what cluster and attribute id.

https://cdn.discordapp.com/attachments/986700592145645579/1110521390345486406/matter-1-1-application-cluster-specification.pdf?ex=65a660a3&is=6593eba3&hm=ed0b4071c00c1b72e6e5c4573f432e17e174a6952bc1c259f690dbe369d5c368&

[jaunty saffron]

yes, network_only: true

[kind sedge]

Well, we tweaked HAOS with all kind of (kernel) patches to ensure the whole neighbour discovery works right, as well as some stuff for thread border routers. The details are on the server's github repo.
We can ensure that everything works on HAOS as we maintain both the host OS and the application layer. You;d be suprised how many people try to run the docker container (or Ha container) on some NAS with an ancient kernel, guarantee for failure.

[kind sedge]

hmmm, strange. And you only have one matter server on the network ? The phone on the same vlan etc. ? As there is literally no difference between the phone issuing the command and you doing it over websocket when receiving a shared device. For a completely new device its another story because then the first step (PASE connection) is handled by the phone first over bluetooth

[jaunty saffron]

the process i use is go to homekit, pair the device from scratch. then i go to the device in homekit and turn on pairing mode.

1. for the case where I do it from phone, i copy the pairing code, go to ha ios app, go to matter, add device, it brings up the homekit overlay, I select the device, input the code, and then it takes forever and eventually gives that step 18 error code.

2. for the case where i do it on websocket, I get the code from homekit just like before, but then just enter it directly to commission_with_code command network_only: true

for case 1 i can see in the error log that the same api is called

super strange - i had the issue on two devices and was able to pair commission successfully with case 2 and not case 1 on both

[kind sedge]

Ah I think its a race condition, I actually added some retries in the code yesterday

So basically the app is a bit too fast

and the device not yet ready

[jaunty saffron]

i'm running the version with your retries though

[kind sedge]

are you sure ? that is not yet published

You should notice it on the logging that its mentioning attempts

[jaunty saffron]

matter-server  | 2024-01-05 15:46:58 sdk4 matter_server.server.device_controller[1] INFO Starting Matter commissioning with code using Node ID 23 (attempt 3/3).
matter-server  | 2024-01-05 15:47:05 sdk4 chip.CTL[1] ERROR Failed to perform commissioning step 18```

[kind sedge]

ah so maybe we need even more attempts then or a delay.

[jaunty saffron]

i also got some crashes like: matter-server | 2024-01-05 15:48:45 sdk4 chip.CTL[1] ERROR Commissioning already in progress - not restarting matter-server | 2024-01-05 15:48:45 sdk4 chip.-[1] ERROR src/controller/CHIPDeviceController.cpp:838: CHIP Error 0x00000003: Incorrect state at src/controller/SetUpCodePairer.cpp:280 - not sure if that's related to the retries

[kind sedge]

I do not see this issue on my testsetups btw

I did see it on a meross plug but that one was resetting all the time

Incorrect state is interesting

We need to verify this on a real HAOS setup

to rule out its your docker setup

[jaunty saffron]

i think that happened when i tried to commission again too quick

[kind sedge]

My feeling is that its all timing sensitive and some devices misbehaving in multi admin mode

[jaunty saffron]

multi admin is when i setup homekit, and then pair to a second server?

[kind sedge]

correct, that takes a HUGE hit on the device so some devices get unstable

especially the first batch of matter devices

Now everybody is switching to more powerful chips with more memory