Some mmWave chips always detecting objects regardless of placement

[wide mulch]

Original comment: #ep1-general message

All of the EP1s across the ground floor of my house are stuck in detection state due to the mmWave chip. 7 units in 7 rooms. Tried a bunch of tuning steps and the only thing that stops it is reducing the mmWave distance to less than 80cm. Sometimes less than 50cm is required. But that makes the mmwave sensor essentially unusable so I need another solution.

The 3 in my basement are fine and the one in my garage is fine, so this seems isolated to the ground floor of my house. But across all 7 of them it's the same issue, permanently stuck in detected state. Not sure what it could be.

mmWave always detected on ground floor of my house

[spice nymph]

The fact your saying it's on one floor makes me wonder if vibration is being transmitted through the walls from something, that would also tie in with why setting the distance has no real effect and why even the on latency being so high has no effect

Have you tried them not on the wall?

Which country are you in btw?

[wide mulch]

Midwestern USA.
I'm going to turn off the master breaker to the house and see if that makes it clear. I have 50 shellies in my walls, I hope that's not the cause.
The network, home assistant, and the power to the EP1s is on a 10 hour UPS so they'll stay going through the blackout.

Killed the power for a couple minutes and didn't see any change in the detection. So it doesn't seem to be caused by something that's powered.
(Or at least not powered by something inside my home, unless it's coming from my own network equipment.)

All of them remained mmwave detected.

@spice nymph can you post the code for the sensors that exposes the SNR and distance? I'll add that to my yaml tomorrow and see what's up. I bet it's interference from something but i have no idea what.

[spice nymph]

I'll send you a binary for it

[wide mulch]

Another thing, do you know why even when the "on latency" is set to 60 seconds it still shows detected after a couple seconds? I would expect it to wait the full duration before marking as detected.

[spice nymph]

And yeah there has got to be for all 7 on the same floor with the same behaviour

[spice nymph]

It's not documented but I suspect there is an upper limit that the sensor just ignores anything above

[wide mulch]

How would that work though? My network info is different from yours.

[wide mulch]

Ah okay, i just increased it to the max your setting slider allows to see what would happen.

[spice nymph]

Use captive portal or USB to join network.

[wide mulch]

Alright I'll do that tomorrow. It's very late, and even later for you! Lol. Night.

And thanks for the support

[spice nymph]

It's morning for me 😅

I'm just compiling it I'll send it in a sec for you to use whenever

[wide mulch]

Ahh. Cresting 0200 here.

[spice nymph]

No problems, sorry your having issues but hopefully we can get it sorted

[wide mulch]

Any specific reason for keeping that sensor code secret?

[spice nymph]

It's not a secret, I just don't want people using it and coming to me for support when they are trying to do some weird stuff with it is all

Then trying to ask me how to add it to the main code etc etc

[wide mulch]

Fair to a point, but I'm sure some industrious user/patron will find a way to add it to the main code if it's reliable. Maybe i will, lol. Distance is really valuable.

I'm a software engineer by trade, just don't do much hardware level stuff until recently.

One of the more headache inducing things in the hobby space recently for me was debugging zwave issues. That system is an absolute nightmare from a reliability and speed perspective, which basically encompasses its whole reason for existing. And the amount of technical knowledge required to debug it when things go wrong is far too high for a mainstream system.

[bitter crane]

Congratulations @wide mulch! You just advanced to level 3!

[spice nymph]

Haha yeah indeed!!

Here we go

Let me know once you have it

[wide mulch]

Thanks. It will be tomorrow late in the evening most likely. Date night with the Mrs.

[spice nymph]

No problem, do you have it dowmloaded?

[wide mulch]

Yep i have it

[spice nymph]

Just ping back when you are free to troubleshoot and we can try figure it out

[wide mulch]

Have had a ton of things taking my time the last few days. Will hopefully loop back to this tomorrow evening.

[wide mulch]

@spice nymph I should have asked this before. How would this binary work to upload to the EP1 when I can't modify the network it's connecting to?
Even over USB via powershell i supply a yaml for configuration, not a binary file.
How do I use this file?

HA is running in a vm on a mini pc so to use the captive portal option i would need to passthrough the ep1 to the vm?

[spice nymph]

It has improv enabled or captive portal so you can connect to the AP that will be created

[wide mulch]

Ahhh i see

[spice nymph]

Huh? Captive portal just means you plug it into power then use your phone or any computer to connect to the AP on it

[wide mulch]

Oh, i thought you were referring to the esphome portal in HA. Must have my terms mixed up.
Okay, I'll look up how to upload a binary, then boot it, connect to its AP, supply my own wifi creds there, then it connects to the wifi and exposes the sensors to HA? What do I do about my hidden network? I had to use the fast connect config so my current EP1s could see the network.

[spice nymph]

Captive portal might allow you to do that, not actually sure. Give it a shot maybe and see

[wide mulch]

Alright, on it. Gotta look up how to upload a binary lol, have only built thusfar

[spice nymph]

Use ESPHome web, super easy takes 2 secs

https://web.esphome.io/

[wide mulch]

Thanks.
Here are the results for the first room. Nobody in the room at the time.

Definitely nothing within 1m of the front of the sensor. No idea why it's having detection there.

[spice nymph]

Where is it mounted now?

[wide mulch]

Here's one from the bathroom. Same sensor just moved to the bathroom mount.
The weirdest part is walking in front of the sensor causes no change in the readings.

[spice nymph]

The distance won't actually change till you click away, just the way HA works

[wide mulch]

These mounts are the two places i tried the debug device

[wide mulch]

But i would expect the SNR to change right? When I walk in front of the sensor the SNR should rise a lot right?

[spice nymph]

Depends if it's already locked onto a target or not I guess

[wide mulch]

Also the distances are updating for me in real time

No need to navigate away

[spice nymph]

That's not what I meant

I meant changing the setting for distance (the 1.95) I can see you still have it selected or in focus

Changing the distance setting won't actually send the new distance to the sensor till you click away

[wide mulch]

Oh you mean the setting. It was already set i just had my cursor back in there

[spice nymph]

I'm still waiting for you to try the thing I said way back in the beginning and try one off the wall and just on the counter 😄

[wide mulch]

Sitting on the sink now. Still detecting movement. Sensitivity is at 4 btw, not sure that matters for these readings though

Moved to the floor on the carpet aimed up and still getting the same readings.
The only time the snr climbs high is when i wave my hand around directly in front of it. Is that normal?

[spice nymph]

If you set distance down to 15cm does it clear?

[wide mulch]

Results!
I tried kitchen counters, on the floors on the sinks nothing changed....until....

I placed it in the exact same mount as one of the sensors in the basement that was working, and it still detected things where the other ep1 did not.
So i swapped the mmwave chips between the two boards, and now the debug device clears detection in my bathroom

Looks like bad mmwave chip?

[spice nymph]

Didnt you say you were getting it on all the first floor ones?

[wide mulch]

And yes i know the settings are saved on the chip so i updated the distance to match

Yes they are, however i installed one of the spares in the garage and it is also stuck detecting. That one i planned to install on the first floor but never did, put in the garage instead and it's stuck detecting.
I only have 3 in the basement, they were fine. The majority of them were on the first floor

[bitter crane]

Congratulations @wide mulch! You just advanced to level 4!

[wide mulch]

Yeah the test ep1 is working great after swapping the sensor. I'm gonna put it back on the wall in the same spot and see what happens

Yep! Works great on the wall mount. Clears right away when i leave the room

[spice nymph]

So how many do you think aren't working properly? 1 I could see but that many would be highly strange

[wide mulch]

I'll swap out all the sensors and see.
I just tried mounting the test one in the kitchen and works great there too

Can they be swapped with the ep1 running? I've currently been shutting it down to swap them

I'll also check the chips for damage, it would suck if this was my fault when putting them in the cases. Hope that isn't what happened here.

And yes such a high malfunction rate is alarming

[spice nymph]

Yeah it would be very very unlikely I guess, we've had to send 1 replacement I think out of 8000 sent

[wide mulch]

It doesn't look damaged. Working one is on the right. Calibration issue maybe? I already tried factory resetting some of the weird ones and that didn't help.

Some of the text is slightly different on the chips but that's not too alarming, i know different bins are a thing sometimes.

I'll test all of them tonight.

I'll test in the basement with the same control board strapped to a pole so it's repeatable. I'll factory reset each chip and post pics of the debug entities in HA

[wide mulch]

Setup

• Used the basement for measurements since this area was wide open and still.
• EP1 from my kitchen installed on a pole with two zip ties connected to the mounting holes. This board was flashed with the debug firmware and was used for all tests below.
• Stood behind the pole for all readings.
• For still readings, just did nothing, for motion readings i walked out and spun the chair and took the reading while it was still spinning.

Steps

• Unplugged EP1
• Installed mmwave sensor
• plugged in EP1
• factory reset mmwave sensor
• rebooted EP1 from HA
• applied mmwave settings from HA
  • 3.9m distance (13 feet from the pole to the wall)
  • sensitivity 5
  • latency 5 seconds
• rebooted EP1 from HA
• while behind the sensor, took a still screenshot of the readings
• spun the chair in front of the sensor then moved behind the sensor, took a motion screenshot of the readings
• repeat the above process for each sensor

Findings

• The wonky ones are always detecting something. They flail around between finding 1-6 objects at once, with the 2-4 object range being the most common.
• There seems to be a pattern where targets 1 2 3 and 4 are usually around 1 meter, 2 meters, 3 meters, and 4 meters out, respectively. Although that's not always the case. Also, with this firmware after factory resetting the mmwave chips they have 9.45m max distance by default. With that distance the weird ones detect anywhere from 5-8 targets all the time.
• I thought it was just the ones on my first floor that were having issues but one in the garage was also having problems.
• There doesn't appear to be any damage on the sensors, they all look identical bar some different lettering on some of the chips due to different suppliers i assume.
• Factory resetting these sensors doesn't seem to help, and changing all of the settings around makes no difference, except for the max distance. Reducing the max distance to less than the first ghost it's seeing makes it clear the detection. With this reduced distance, moving my hand in front of the sensor does prompt detection and the SNR climbs to match, but allowing anything beyond 1m will always be locked in detection mode.
• Everything else on the boards are working fine. Controller is great, PIR is snappy and accurate, humidity and luminance are fine. The temp sensor reads a little high but that's to be expected, need to tune the offsets once they are on the walls for a while. I haven't confirmed with a dedicated thermocouple, just guessing based on the numbers, although it could also be because they are at the top of my rooms.

Final layout with results above the sensors. 5 good. 8 with bad output.

What should we do? The wonky ones are technically functional, but their output is garbage for anything beyond ~50-100cm. Seeing lots of ghosts.
Thought i was going nuts when every setting and position of the sensors were still resulting in detection.

[wide mulch]

This person mentioned a similar thing happening to them, though they might just have real motion going on. https://github.com/EverythingSmartHome/everything-presence-one/issues/55#issuecomment-1519368412

[wide mulch]

Do you know anyone at DFRobot that could help us with this? Should I send them back for analysis? They don't appear to be broken, just wildly uncalibrated and seeing ghosts.

[wide mulch]

Some mmWave chips always detecting objects regardless of placement

[spice nymph]

Sorry been away all day.

Based on all the data we have of 8k units in the wild, it just seems so unlikely that you'd have 8 bad sensors in total, that's a really high rate. That said obviously those sensors aren't usable for you so will need to get them sorted with replacements.

The only other thing I can think of is vibration in the socket since the sensor floats there? Maybe some sensors are tighter/less worn down than others?

[wide mulch]

I definitely want to send the sensors back to you so we can find out what is actually wrong with them. If you'd prefer I send them directly to DFRobot I can do that too, just let me know, I'm sure you have a contact there given you do so much business with them.

It's true the sensors do kinda float in the socket, but the testing i did last night had the exact same mounting for every sensor. And they don't look any different from each other, i checked for missing components or black spots indicating something shorted and nothing stood out, they all look identical to the working ones.

[spice nymph]

It's not really worth it the shipping costs unfortunately so probably just get you to destroy them.

Sure but it's not the socket on the EP1 that can become loose, it's the pins on the sensor that go into the socket

Also regarding what you said about the distance resetting to 9m, that can actually cause issues when the distance is set to large for the room, can cause it to bounce giving unwanted readings

[wide mulch]

Yep understood that, it was just a finding in itself. Had the tape measure on the ground and set 3.9m as the distance for all tests. 9.45m was also greater than the advertised 8m max distance so it stood out to me.

[wide mulch]

You sure? I can probably snail-mail the chips to you for relatively cheap in a bubble mailer. Already dropped 1300ish on these including import fees so spending another 20-30 to get closure on this issue is fine from my side. Up to you though.

Thanks for offering to send replacements . Don't worry about the speed, slower delivery method is fine.

[spice nymph]

I'd just have concerns about sending another 8 known good working sensors and you have the same issue is all, that would be a bit of a waste of sensors

Like I say, only had 1 returned sensor out of 8k so for you to have 8 seems fishy (not towards you but that something else is at play!)

[wide mulch]

Yeah I understand. I'm not sure what is making these malfunction and that's part of why I'd be fine paying to send them back to you for analysis.
If they just stopped working or had blown circuits I'd immediately assume it was something I did wrong, but they are still functional just seeing ghosts beyond 1m and IDK how to debug that.

[bitter crane]

Congratulations @wide mulch! You just advanced to level 5!

[spice nymph]

One way might be to place them inside a metal container, since mmWave cannot detect inside

[wide mulch]

If the replacements end up having the same or similar problem after I install them I would take responsibility at that point, even though we're not sure the cause. From my perspective it's weird that only the mmwave chips are affected. If this was a handling/power surge/ESD issue I would expect other things on that board to be affected but everything is functional, even the mmwave chips themselves in part

[spice nymph]

Used that for troubleshooting people who assured me nothing was moving only for them to later discover something was 😆

Using a metal pan for example is sometimes a good way

[wide mulch]

Sure, i have a USB power supply, i can put it in a metal box and test. Does the max distance matter for that?
I'll probably use two bread pans on top of each other

I'll see if i have anything thicker too

Inside an unpowered microwave?

[spice nymph]

Well remember you still need wifi to be able to read the results

[wide mulch]

didn't think the mesh in a microwave was a full Faraday for everything, but that would make sense. Time to test

Just stuck my phone in an unplugged microwave for a minute and it stayed connected to the WiFi so that's promising

So either wifi can get out and mmwave/microwave can't, or I have a very badly shielded microwave lol

[wide mulch]

Reiterating, does max distance matter for this metal box test?

[spice nymph]

I'd start with 15cm and work up

Though I don't know if reflections will come into play at higher distances

[wide mulch]

Yeah that's more what i was asking about.

So for this test i took a known working one and put it in the microwave and it stays connected to the WiFi with the door closed. Set to 2m distance it does not detect movement around the microwave, but does detect movement when i shake the microwave.
All seems good with this one.
Now testing one of the wonky chips.

[wide mulch]

Okay, even more wacky results. I was having a bear of a time getting readings from the wonky sensor because the esp32 wouldn't stay connected to the WiFi once the door on the microwave was closed. But that only happened with the wonky chip installed. If i took the wonky chip out then plugged in the board it stayed connected just fine. And it was also fine with the working chip installed.
These wonky ones are perhaps putting out so much interference that the esp32 can't stay connected from within that box.

I'm gonna plug them into a watt meter and check the power draw differences between the two

[spice nymph]

They are completely different frequencies though, 24ghz vs 2.4ghz 🤔

[wide mulch]

Idk the physics, all i know is the esp32 won't connect to the wifi when inside the microwave with the door closed and the malfunctioning chip installed, but it will connect without a mmwave chip or with a working one

Maybe a harmonics thing? 2.4 vs 24? Either way, getting the power draw results now

Working: 0.88w, 0.90 power factor
Not working: 1.31w, 0.64 power factor
No mmwave: 0.35w, too low to get a pf reading

So the not working one is pulling twice the power that the working one is? Strange

And that aligns with the interference theory, if that's actually a thing

[spice nymph]

Ah did you test the whole board rather than just the mmWave?

Because esp does use more power when trying to connect to WiFi

It would be interesting to check the serial logs while it was doing that, could you shut the door on the cable?

[wide mulch]

Yeah i was testing the whole board, but that power draw test was outside the microwave, just laying on the ground.
I can't do any more testing for a bit. Will need to loop back on this if we want more testing. What is the current plan for these sensors?

[wide mulch]

@spice nymph when should the replacement sensors be sent? I can continue testing for correlations but that doesn't fix the malfunctioning sensors.

[spice nymph]

Sorry completely missed this, send an email to support through the website so we can get your ones back and get new ones sent

[wide mulch]

Alright, message sent via the site. I'll wait for a response there 👍 . The sensors are ready to be sent. Still have the anti-static bags from the original shipment.

[wide mulch]

DHL says the driver stopped to pick up the package and nobody was home. My security cameras say otherwise . Rescheduled the pickup for tomorrow.

IDK why they lie when so many people have cameras now.

[spice nymph]

That's really annoying. Hopefully tomorrow will be the day! 🤞

[wide mulch]

Lol the guy just arrived and picked it up. Sensors are on their way.

[spice nymph]

Excellent thanks!

[wide mulch]

Of course! I really hope we can find out what's wrong with the current ones. Maybe some kind of 'harder reset' is required, or something else completely.

[spice nymph]

My guess is they will work normally when I test them unfortunately 😔

[eager zenith]

Did u tried one that is working for example the one in the garage, in a place of another that is not working and vice versa?

[wide mulch]

Yes, the working sensors operated as expected when in the same location as the malfunctioning sensors. The malfunctioning ones detected all the time in places the working ones correctly cleared. Have detailed pictures of the test results and some findings above.

I really hope the problem continues once Lewis gets the sensors. Otherwise my house is actually haunted and accounting for inflation i probably can't afford to call in the Ghostbusters.

[eager zenith]

Ohh probably the threads I skipped. Sorry then the stupid question 🙂

[wide mulch]

@spice nymph Looks like the package was delivered! Fingers crossed for the results.

[spice nymph]

Yes received earlier and have arranged another to be sent, should be collected tomorrow!

[wide mulch]

Sweet! When do you think you'll be testing the current ones?

I'm really curious what you end up seeing

[spice nymph]

Probably not till the weekend maybe later but will let you know

[spice nymph]

Testing your sensors now

[wide mulch]

Sweeeeeeet. Excited to hear the results.

[spice nymph]

Yeah I'm pretty astounded, set up a rig with a known good one and it cleared fine then switched your ones in and had issues with each one like you say above 1m. Also noticed they ran a bit hotter than a good one.

I really don't know how you could have got so many bad ones in a row, like I say I've only seen one bad one previously that we've had to replace out of more than 8K units now so it's crazy luck!

Anyways didn't manage to post them yesterday sorry but will be sent on Monday, but while I've got this setup I'm just testing 8 new ones for you and verifying they all clear correctly so you definitely have all new good ones

[wide mulch]

Wonderful! Thank you very much for doing the testing and confirming my worries.

May I ask for something since you haven't posted the package yet? Can i buy three additional EP1 boards without the sensors and include them in this package?
I know you don't normally carry the sku anymore but one of mine just died and I think it was my fault, bad power supply. It won't power on anymore from either the USB or 5v headers, no lights or detection on the computer either.

Let me know if I'm able to purchase those either now or separately later, no real rush

[spice nymph]

Damn what did you plug into it? 🥲

Uhhhh I'm not sure if the system will let us, I'll get back to you

8 tried and tested sensors on their way back to you 😅

[wide mulch]

Just a USB power supply, but it's the same one that the EP1 that started boot looping was plugged into. Flashed a spare with the same configs and put it in the same spot and it died permanently. So now I'm questioning the supply rather than my configs or the board. So ordering 3 boards if I can, just in case i can't revive that boot looping one.

[wide mulch]

Wonderful. You the man!

[spice nymph]

Do you think you'd be able to do a bank transfer? Might be able to do it that way

[wide mulch]

Yes i can. You have a secure way of doing that? Let's discuss in DMs.

[spice nymph]

Send you an invoice for it, you can try to pay via the link or if you cant then send an email again and I can get you the details

[wide mulch]

Unfortunately this is what i get

It removes the items

[spice nymph]

Yeah that's what I thought. Send an email and can give you alternate details

Sorry I'm just sending you one now

[wide mulch]

Confused. You want me to send another email through the site or you're going to send me one?

[spice nymph]

I'm just emailing you now

[wide mulch]

Sent the wire, bank says it's scheduled for tomorrow. 95GBP.

[spice nymph]

Yeah got the email thanks! Got the box all ready

[wide mulch]

Yeeee the replacements appear to be arriving Friday. Perfect.
I hope they don't hit me with more import fees on the replacement sensors, understand for the new boards but I hope the RMA portion of that doesn't get dinged.

[spice nymph]

Noice 🙌🏻

You are in the US right? Import fees only apply to orders over 800usd as far as I'm aware (though could be wrong)

[wide mulch]

Lol if that's the case it was a massive mistake to order 13 of them at once the first time. Should have made two smaller orders.

[wide mulch]

@spice nymph replacement sensors arrived along with the new ep1 boards, I'll be setting up test rigs tonight

[spice nymph]

Noiiiiiice! 🙌🏻

[wide mulch]

@spice nymph found out what fried those 2 EP1 boards. The dc-dc converter I was using caught fire .
Melted the heatshrink tubing and thank God it didn't cause a house fire in the attic.

Swapping all of them out for different brand and formfactor, and adding fuses significantly below their rated current. Nuts something like this happened with essentially 1 watt of load on them.

[spice nymph]

Wow that could have been really bad, glad everything is OK! Wonder if it had any influence on the mmWave sensors?

[wide mulch]

Possibly? Electrical noise even while they were working? Perhaps the rest of the ESP32 and the attached sensors were less sensitive to electrical noise than the mmWave chip? If so then it was my choice of power supply that caused those chip's demise, which would sadden me greatly.

I don't have an oscilloscope to measure electrical noise at the moment. But i did have a USB voltage/current tester that I was using to validate the original concept of this setup and it was reporting fine.

[wide mulch]

Main reason for going this route was avoiding voltage drop. A 50ft usb cable will drop down below 3v under load, and they are greatly influenced by line voltage running nearby them, lots of noise. So started with a 24v lighting controller going to a bunch of dc-dc breakouts to USB. I had a test bench set up with all of them before the install and it was working great then, but if the QC was so bad they had a ton of noise that could have been the cause of the mmwave issues.
Would certainly be an explanation for 8 of them failing in short order if that was the reason. Some of the sensors connected here still worked fine though, the remaining 5. Could definitely have been massive QC issues on the part of the dc-dc converters.

Already have a replacement power supply and new fully-encased dc-dc converters with over/under voltage and current protection running in my basement to validate.
Anything further that comes from this I'll be assuming is my fault just because it seems a lot more likely something with these power supplies caused the mmwave sensors to go wonky vs them all being wrong from the factory at the same time.

[spice nymph]

Yeah cause I was wondering how you managed to get 8 in a row with that exact issue that I've never seen before haha

[wide mulch]

All of the replacement sensors and the new boards work great. Thanks Lewis 🙂

Gonna leave them going on the test bench with these new power supplies for a few days before installing them back in the ceiling.

[spice nymph]

Nice one, glad they are working

[wide mulch]

Such a beautiful sight 🥲

[wide mulch]

Gahhhhhhh one of the EP1s has started eating mmwave chips. They enter the same malfunctioning state of erroneous detection after about 2 days of being connected to the EP1.
It's specifically one board so I'm not sure what's causing it this time. The rest of the EP1s are fine, no failures outside of this one board.
I've contacted DFRobot about what to do, and I have some spare mmwave chips already but it sucks this started happening again. Thankfully it's just one board that seems to be doing it, but I have tried 3 different known good mmwave chips in this board now and they start malfunctioning a couple days after being installed.
Hopefully DFRobot comes back with some answers.

[spice nymph]

Bad PSU again?