Integral (Riemann sum) gives unexpected results | Home Assistant | Page 1

jovial mirage Dec 13, 2024, 5:59 AM

#

I am using an integral helper (Riemann sum) to calculate the energy produced (kWh) based on the power production (W) of my Panasonic A2W heat pump.

In the morning when the heat pump starts up after a night without any production (0 W), the power production is negative for a few minutes. The result of this start is a really large and faulty negative energy production (kWh) value.

I tried integral methods "trapezoidal" and "left". Only the trapezoidal method gives the unexpected results. I started with the example integral yaml code on the HA website and only adjusted the source, name & unique ID, see code below.

Anything I am doing wrong here?

# Use Riemann Sum to track total Aquarea Total Energy Produced Calculated (kWh)
# Source is template sensor Aquarea Power Produced Calculated (W)
- platform: integration
  source: sensor.aquarea_power_produced_calculated
  name: "Aquarea Total Energy Produced Calculated"
  unique_id: "panasonic_heat_pump_total_energy_produced_calculated"
  unit_prefix: k
  round: 2
  max_sub_interval:
    minutes: 5

Schermafbeelding_2024-12-13_om_06.31.03.png

sharp wyvern Dec 13, 2024, 6:14 AM

#

That's really weird... Why is it showing negative after starting up?

jovial mirage Dec 13, 2024, 6:38 AM

#

It is! Because the power production is calculated based on water temp outlet/inlet + flow. And when the heat pump starts up the outlet temp is temporary colder than the inlet temp, since the compressor isn't running yet.

Formula: heat produced (W) = flowrate (l/sec) * 4.153 (specific heat of water) * temperature rise (delta temperature) * 1000

This is the template sensor for the power production calculation and the source for the integral helper.

    # Sensor for calculating the actual Power production
    # Using Panasonic flow rate, outlet temp, inlet temp
    - unique_id: "panasonic_heat_pump_power_produced_calculated"
      name: "Aquarea Power Produced Calculated"
      unit_of_measurement: "W"
      state_class: "measurement"
      state: >
        {% set flowRate = states('sensor.panasonic_heat_pump_main_pump_flow') | float(default=0) %}
        {% set flowRateSec = flowRate / 60 %}
        {% set outletTemp = states('sensor.panasonic_heat_pump_main_main_outlet_temp') | float(default=0) %}
        {% set inletTemp = states('sensor.panasonic_heat_pump_main_main_inlet_temp') | float(default=0) %}
        {% set deltaTemp = outletTemp - inletTemp %}
        {% set value = flowRateSec * 4.153 * deltaTemp * 1000 %}
        {% if flowRate > 1 %}
          {{ '%0.0f' % value }}
        {% else %}
          0.0
        {% endif %}

But the scope of the issue is the integral which gives the stange result, not that the power production is negative for a few minutes.

sharp wyvern Dec 13, 2024, 8:40 AM

#

So the integral you shared looks sensible, however the graph was for "Daily" not "Calculated" as in the yaml - do you have a utility meter in there as well

jovial mirage Dec 13, 2024, 9:27 AM

#

Yes I do have an utility meter based on "Aquarea Total Energy Produced Calculated" integral sensor.
I shared the daily utility meter graph screenshot so it easier to visualize what is happening.

See attached image for screenshot with power production (W) + energy produced (kWh) directly.

If the integral I shared looks sensible and this is the result - can it be a bug in the integral implementation? Maybe something related to the negative power after being zero for ~8 hours?

sharp wyvern Dec 13, 2024, 10:25 AM

#

The reason I asked about the utility meter is that it has an automatic "reset" function under the default settings which can cause weirdness but looks like that's real.
The max sub interval should take care of the "zero for 8 hours then negative" issue, but it does look of the right order of magnitude.. Is that addition new? Or was it present when the issues were occurring?

#

You might want to limit your deltaTemp value to >= 0, as that being negative isn't actually your heater outputting negative power. If you did that and it then created a big positive spike then it would definitely seem to be related to the integral incorrectly integrating from a long period at constant value

jovial mirage Dec 13, 2024, 12:22 PM

#

Do you mean the addition of the sub interval? No, it isn’t new.

What is new that before I didn’t allow negative power production and now I do. During a defrost a heat pump will extract heat from the house to defrost the evaporator outside. For accurate COP calculation this energy “subtraction” needs to be accounted for.

I checked another date when this negative power production wasn't allowed, and it happens there as well but less aggressive because the production increase is less rapid. But it is clearly not related to the negative value because it happens with positive values as well.

#

screenshot of possitive spike

sharp wyvern Dec 13, 2024, 12:47 PM

#

Yeah I'm very suspicious that the max_sub_interval isn't working. I'll have a look at it after work for you

sharp wyvern Dec 13, 2024, 1:11 PM

#

(before I spend an age on this, what home assistant version are you on?)

#

Because max sub interval was only added in may iirc

jovial mirage Dec 13, 2024, 1:17 PM

#

My HA is provisioned somewhere in October this year. Pretty standard with mostly standard integrations and this logic around my heat pump is my first own technical solution / integration.

sharp wyvern Dec 13, 2024, 1:18 PM

#

OK cool

jovial mirage Dec 13, 2024, 1:19 PM

#

Thank you for helping me! Curious to find out what the core issue is. If I can do anything to help you help me, please let me know

sharp wyvern Dec 13, 2024, 1:20 PM

#

I got bored in my meeting and had a quick look - it seems that if your template sensor state is not resolving as a number (more specifically a Decimal) then the max_sub_interval won't work

#

Not sure why that would be the case.. But I think it's likely that causing the issue

jovial mirage Dec 13, 2024, 2:58 PM

#

Haha. So I have to | float(0) the power production (W) template sensor value?

sharp wyvern Dec 13, 2024, 8:11 PM

#

No that shouldn't be an issue. I'm wondering if it's the indentation - your 0.0 is a straight print with leading spaces rather than a template out ({{... }}) , wonder if that could mess things up..

#

Will need to do some testing

jovial mirage Dec 14, 2024, 8:45 AM

#

@sharp wyvern I have added "device_class: power" to my power production template sensor. Just to see if this makes any difference. I will post an update here later.

sharp wyvern Dec 14, 2024, 8:59 AM

#

Failing that you could replacing the end of your template with

{{'%0.0f' % value if flowRate>1 else '0.0'}}

Or alternatively you could set flowRateSec = 0 if flowRate <= 1 else flowRate/60 and that would propagate through just fine (could remove the final if statement and just have the template output)

jovial mirage Dec 14, 2024, 10:56 AM

#

@sharp wyvern Done! Thanks!

At another HA instance I help "administrating" at my neighbours home I've added the "device_class: power" without the template code change. He is running the same heat pump and HA version etc etc. I will keep you updated on the result of both HA instances 😄

My power produced template sensor:

    - unique_id: "panasonic_heat_pump_power_produced_calculated"
      name: "Aquarea Power Produced Calculated"
      device_class: power
      unit_of_measurement: "W"
      state_class: "measurement"
      state: >
        {% set flowRate = states('sensor.panasonic_heat_pump_main_pump_flow') | float(default=0) %}
        {% set flowRateSec = flowRate / 60 %}
        {% set outletTemp = states('sensor.panasonic_heat_pump_main_main_outlet_temp') | float(default=0) %}
        {% set inletTemp = states('sensor.panasonic_heat_pump_main_main_inlet_temp') | float(default=0) %}
        {% set deltaTemp = outletTemp - inletTemp %}
        {% set value = flowRateSec * 4.153 * deltaTemp * 1000 %}
        {{'%0.0f' % value if flowRate > 1 else '0.0'}}

jovial mirage Dec 14, 2024, 12:09 PM

#

@sharp wyvern

The result at my neighbors home with just the addition of “device_class: power” resulted in the expected behavior! No weird start-up spike anymore.

Do you want me to test your template sensor code without the “device_class: power” addition? So that you are certain what the solution is?

If so, I can have the result tomorrow morning.

sharp wyvern Dec 14, 2024, 12:19 PM

#

It'd be interesting but don't worry about it, I'm sure I can set up something similar enough to test it without needing to do weird stuff with your live install

jovial mirage Dec 14, 2024, 12:24 PM

#

Ok. in that case I will keep both your code change and the device class.

What is your goal, to update HA Integral documentation to warn that the source sensor should have a "supported" device class?
Or to have the implementation of the "max_sub_interval" changed so it is less strict regarding the source sensor? Or..?

This is result at my neighbors (with only device_class: power)

sharp wyvern Dec 14, 2024, 2:55 PM

#

Yeah that doesn't looked fixed - the blip resulted in a small but significant jump

#

I'll do a documentation update and it's useful to know if someone else runs into the issue

jovial mirage Dec 14, 2024, 4:47 PM

#

You're right, I've overlooked that. I'll post an update tomorrow on our situation with your suggested code change

jovial mirage Dec 15, 2024, 5:11 AM

#

@sharp wyvern I can confirm your code change suggestion works!! Woohoo 😁 I’m super happy. Thank you so much for your help. Virtual hug 🤗

-0,07 instead of -7 kWh. My calculated COP will increase a lot haha.

sharp wyvern Dec 15, 2024, 5:23 AM

#

Awesome

jovial mirage Dec 16, 2024, 5:07 AM

#

@sharp wyvern I celebrated too soon. This morning the same issue occurred. I haven’t changed anything to my HA setup/configuration since yesterday.

sharp wyvern Dec 16, 2024, 10:00 AM

#

Dangit

#

I should have time this evening to give it a further poke

sharp wyvern Dec 16, 2024, 11:00 AM

#

Can you check your base sensor is actually outputting 0 overnight and isn't unknown/unavailable (might be able to see in logs if it's failing to calculate for whatever reason)

jovial mirage Dec 16, 2024, 12:42 PM

#

I can't find any log entry related to this sensor
Question, we are now doing this:

{{'%0.0f' % value if flowRate > 1 else '0.0'}}
I am not a real coder, but doesn't output this as a string instead of a number?
Shouldn't it be this, without the quotes around the 0.0?
{{'%0.0f' % value if flowRate > 1 else 0.0}}

sharp wyvern Dec 16, 2024, 12:43 PM

#

States are always strings

#

Doesn't matter what you output them as

#

I thought sensible to keep it the same as your other output which is a formatted string

#

It would be under template I think

sharp wyvern Dec 16, 2024, 1:55 PM

#

But you should be able to check the history of the sensor too

jovial mirage Dec 16, 2024, 2:53 PM

#

sorry, can you point me in the right direction. In the graph of the power production sensor I can see a reading every 5 min. But not sure if that is the sensor or just the "graph resolution"

sharp wyvern Dec 16, 2024, 3:36 PM

#

Hmmm then it's probably behaving sensibly then...

jovial mirage Dec 16, 2024, 3:37 PM

#

I would be suprised if it wasn't. All the heat pump sensors which are used in the power production sensor are available all the time as well

regal wedge Dec 16, 2024, 3:58 PM

#

Have you confirmed you're actually looking at the right sensors.

The one time you give yaml you have an integral named "Aquarea Total Energy Produced Calculated", but your charts keep showing something called "Aquarea energy produced daily".

#

Well I guess some have one sensor and some have another.

#

If you think max sub is not working please open a core GH issue for it.

jovial mirage Dec 16, 2024, 8:22 PM

#

@regal wedge Yes I am sure. The "Aquarea energy produced daily" sensor is an utility meter based on "Aquarea Total Energy Produced Calculated"

#

I use that one for easier visualisation of the problem. But I can conform the Energy Produced Calculated shows a dip of multiple kWh's

#

I'll open a GH isue for it. Thanks

#

Integral (Riemann sum) gives unexpected results

jovial mirage Dec 16, 2024, 8:44 PM

#

Done: https://github.com/home-assistant/core/issues/133382

GitHub

Integral (Riemann sum) gives unexpected results · Issue #133382 · h...

The problem I am using an integral helper (Riemann sum) to calculate the energy produced (kWh) based on the power production (W) of my Panasonic A2W heat pump. The power production (W) is calculate...

sharp wyvern Dec 16, 2024, 9:33 PM

#

Question - have you restarted home assistant recently?

#

Or are you just reloading the configuration from the developer tools

jovial mirage Dec 17, 2024, 7:57 AM

#

Well, I experienced that a yaml reload isn't always enough to apply certain changes. I restart HA after bigger logic changes, also after making above changes

sharp wyvern Dec 17, 2024, 8:59 AM

#

Hmmm OK. Have you tried making an integral helper via the ui and checking if it has the same issue, would help narrow it down if its an integration issue or a yaml issue

jovial mirage Dec 17, 2024, 3:41 PM

#

I haven’t. Will do.

Apparently I’m not alone and someone found out what could be the problem: https://github.com/home-assistant/core/issues/133375#issuecomment-2547650948

GitHub

Integral integration does not work correctly · Issue #133375 · home...

The problem The maximum sub interval is not used correctly when the input value is 0 and then changes to a different value. This is very common for solar energy returned to the grid, as there is no...

sharp wyvern Dec 17, 2024, 3:42 PM

#

Ohhh sneaky, I completely missed that edge case in that condition...

#

Yeah that will be a bugger... In the meantime you could set it to have an "off" power of 0.0001W or something rather than 0

jovial mirage Dec 17, 2024, 3:49 PM

#

sharp wyvern Yeah that will be a bugger... In the meantime you could set it to have an "off" ...

Good idea!! Will do

jovial mirage Dec 17, 2024, 9:08 PM

#

#

gotta love workarounds 😄

sharp wyvern Dec 17, 2024, 9:09 PM

#

Aye

jovial mirage Dec 17, 2024, 9:11 PM

#

If tomorrow morning everything is fine with this workaround then we know for certain. I am pretty convinced 🙂

regal wedge Dec 17, 2024, 9:32 PM

#

Or just use left mode like you should have from the beginning 😂

sharp wyvern Dec 17, 2024, 9:35 PM

#

Left underestimates quite significantly on data like this that rises quickly but trails off slowly

jovial mirage Dec 18, 2024, 5:06 AM

#

regal wedge Or just use `left` mode like you should have from the beginning 😂

HA website disagrees:

“TRAPEZOIDAL

The trapezoidal method follows the Trapezoidal rule. This method is the most accurate of the currently implemented methods, if the source updates often, since it better fits the curve of the intrinsic source”

jovial mirage Dec 18, 2024, 5:08 AM

#

jovial mirage

This works as a workaround!

#Integral (Riemann sum) gives unexpected results