How should the outlier filter work? | Home Assistant | Page 1

static sparrow Mar 29, 2025, 11:53 PM

#

I have about 20 evergy monitoring devices. Every so often it will jump in readings because the value of the meter went form 32kwh to 0 then back up and to home assistant in a utility meter it thinks that you just used 32 kWh! I was told to use a filter between my sensor and the utility meter. I created an Outlier filter that looks at 1 sample (which would be the last one) and the radius is set to 0.1 and precision is set to 4. My expectation is that it will take the last x reading and average it, which because there is only 1 reading will be the same number as the last reading. Then it will determine if the new reading is more than 0.1 off that last reading if so it will dispose of the new reading and replace it with the old reading. Problem is I am still having big spikes in reading way above 0.1 How does the filter allow this if this is the setting? How can I check to see what is going on to narrow down the issue

topaz reef Mar 30, 2025, 12:00 AM

#

If you only look at one sample, and you get a bad datapoint, doesn't the next datapoint after that then use the mean centered on that bad datapoint as the new filter?

static sparrow Mar 30, 2025, 12:06 AM

#

I see what you are saying but if from the beginning its been filtered then all reading would have to be 0.1 off from the last one no? how could it get a bad reading?

topaz reef Mar 30, 2025, 12:08 AM

#

You might be right. I read the filter doc but it's not clear to me if it's using the median of the previous values of the input sensor or the output sensor.

static sparrow Mar 30, 2025, 12:09 AM

#

that was my first thought also.. the problem with getting lest say 10 last reading it averages them.. then does the radius on the average.. so get the average then compare it against the new reading? Doe snot make sense really and to be honest if a big spike happens it will replace it with an average number meaning it could be lower then the last reading

#

its very vague i agree

topaz reef Mar 30, 2025, 12:09 AM

#

You could just make a triggered template sensor that does not allow the value to decrease?

static sparrow Mar 30, 2025, 12:09 AM

#

i have issues with some sensors that it read 3000wKh for some reason.. LOL

#

No the errors are not as much as they used to be but they still happen

#

I have been having this issue when doing 10 samples and doing 100 and now 1

#

so either I dont understand (very possible) it or this filter is not working as expected 🙂

#

ChatGpt seems to agree with me after i told it my settings:

#

maybe its a bug.. but I have no way to prove it

shy spade Mar 30, 2025, 12:23 AM

#

Don't trust chatgpt for this kind of stuff, it has no clue

#

If you do use the outlier filter it can only have 1 historical value at the most

topaz reef Mar 30, 2025, 12:31 AM

#

If you think it's not working, download a CSV of the input and output sensor, identify the datapoints you think are incorrect, and open a bug with that information.

static sparrow Mar 30, 2025, 12:32 AM

#

yeah best thing I can do problem is that its random when it happens.. Ill have to look to see how many times it does work vs does not work.. I almost wna tto log just this one filter to say what it did each time it filtered

shy spade Mar 30, 2025, 1:09 AM

#

If your filter has a length of one and it catches an outlier, then it just keeps that old value

static sparrow Mar 31, 2025, 10:18 PM

#

seems like it's not... https://community.home-assistant.io/t/outlier-filter-does-not-feel-like-its-working-properly/871810

Home Assistant Community

Outlier Filter Does Not Feel Like It's Working Properly

Sorry everyone, I decided to make my own thread on this instead of jumping on that old thread here. Here is the old post talking about this: I just did a test and created a template sensor called “123 Test” that is set to {{ 0 }}. I then created a filter that used this as input and set it to the same settings window side: 1 radius 0.5, p...

#

I did a test today

#

it does not seem to filter it out

shy spade Mar 31, 2025, 10:53 PM

#

"The included Outlier filter will discard any value beyond a band centered on the median of the previous values, replacing it with the median value of the previous values. If inside the band, the current state is returned."

#

that's what it's doing

#

although I'm not sure that's desired or not

static sparrow Mar 31, 2025, 11:03 PM

#

so if the previous value is 1.0 then it will take the median of the window size.. there is only one value so the median would be the previous value no?

#

so the previous value is 1.0 the new value is 5.0 so replace it with the median of the previous values which because the window size is 1 should use the previous value..

#

what I think it's doing and I could be wrong but I believe the very first reading of 5.0 is corrected and that's what it looks like in the logs.. but the next reading seems to be taking the previous reading from the original Source not the filter source.. so even though I've replaced it in the filtered device. I believe the filter is pulling it's window size values from the original sensor

#

and then what ends up happening because the new reading is 1.0 but the previous reading is 5.0 it takes the 5.0 value.. and then the same thing happens again with the next reading

#

I hope any of that made sense

shy spade Mar 31, 2025, 11:35 PM

#

Yes it's taking the sensor input as source

#

So with a filter length 1 if you get rejected value 1, then rejected value 2. Then the update is the median of the previous value... Which is rejected 1

static sparrow Apr 1, 2025, 12:09 AM

#

yeah it jsut moves the bad value one read up

#

but does not eliminate it..

#

dohh

#

I thought this was a cleaver use of this wither using a window size of 1

#How should the outlier filter work?