Yellow out of memory, HA dies. | Home Assistant | Page 1

spice wigeon May 22, 2024, 6:49 AM

#

Will post some logs here.

#

Right now it's unresponsive, but I did capture this: ```
b6f5759f205c ghcr.io/home-assistant/yellow-homeassistant:2024.5.1 "/init" 26 hours ago Up 3 minutes homeassistant
cb880c1cc874 ghcr.io/home-assistant/aarch64-hassio-supervisor:latest "/init" 30 hours ago Up 4 minutes hassio_supervisor

#

The "up 3 minutes" is about as much as I get.

jolly prawn May 22, 2024, 6:52 AM

#

That's just a partial docker ps. Not that useful.

spice wigeon May 22, 2024, 6:52 AM

#

I get about a 5 minute window to grab any data, then have to wait for the restart.

#

CONTAINER ID   NAME                   CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
5d6740ff7cd0   addon_a0d7b954_ssh     1.52%     53.17MiB / 1.844GiB   2.82%     0B / 0B           58.1MB / 26.1MB   20
f0e4bd7f1ce3   addon_core_mosquitto   0.17%     65.36MiB / 1.844GiB   3.46%     84.2kB / 3.67kB   55.9MB / 348kB    20
0d79dd283b9b   hassio_multicast       0.00%     4.887MiB / 1.844GiB   0.26%     0B / 0B           4.83MB / 307kB    8
7d8aefab3797   hassio_audio           0.03%     27.39MiB / 1.844GiB   1.45%     90.6kB / 0B       22.2MB / 389kB    12
ec2b26b8276f   hassio_dns             0.04%     23.38MiB / 1.844GiB   1.24%     102kB / 9.1kB     19.2MB / 315kB    17
c8c23e0e5a9a   hassio_cli             0.00%     10.41MiB / 1.844GiB   0.55%     99.3kB / 3.58kB   8.7MB / 303kB     9
5f1d97327cd8   hassio_observer        0.00%     9.02MiB / 1.844GiB    0.48%     103kB / 0B        6.26MB / 0B       5
b6f5759f205c   homeassistant          102.88%   658MiB / 1.844GiB     34.84%    0B / 0B           249MB / 565kB     32
cb880c1cc874   hassio_supervisor      0.11%     117.4MiB / 1.844GiB   6.22%     185kB / 101kB     70.7MB / 492kB    26

jolly prawn May 22, 2024, 6:53 AM

#

My current guess is that one of the integrations goes bananas. You can test with with ha core restart --safe-mode.
--safe-mode is relatively recent so hopefully you're up to date.

spice wigeon May 22, 2024, 6:54 AM

#

yes, up to date. The memory usage in my working RPI4 is stable, and about the same (1.3G for the ha container) so nothing is leaking memory.

#

I'll leave a top running and hopefully capture what it gets to before it dies.

jolly prawn May 22, 2024, 6:56 AM

#

You can try docker top homeassistant -faux.

#

The current memory usage is fine though. The CPU usage is not.

spice wigeon May 22, 2024, 6:59 AM

#

root                3045                97.3                62.0                1898152             1200380             ?                   Ssl                 06:52               5:01                \_ python3 -m homeassistant --config /config

#

top - 16:59:28 up 8 min,  0 user,  load average: 1.14, 0.93, 0.49
Tasks:  13 total,   1 running,  12 sleeping,   0 stopped,   0 zombie
%Cpu(s): 24.7 us,  5.3 sy,  0.0 ni, 69.8 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1888.7 total,     50.4 free,   1800.1 used,    100.5 buff/cache
MiB Swap:    623.3 total,    120.3 free,    503.0 used.     88.7 avail Mem

jolly prawn May 22, 2024, 7:00 AM

#

Please share the whole output. Also try the safe mode as mentioned.

spice wigeon May 22, 2024, 7:00 AM

#

Discord barfs on more than 2000 chars.

devout cypressBOT May 22, 2024, 7:00 AM

#

Please use a code share site to share code or logs, for example:

https://dpaste.org/ (select YAML for the language, and consider picking a longer expiry)
http://pastie.org/ (select YAML for the language)
https://paste.debian.net/ (you guessed it, select YAML as the language)

Please don't use Pastebin, since it can randomly add spaces to the main view. Please also don't share text as images since it makes it harder for people to help you. Remember that others may have colour blindness, impaired vision, etc.

spice wigeon May 22, 2024, 7:02 AM

#

Right now I think I'll just monitor my working system. If it stays up, which it has so far, I would consider the problem solved. Running in safe mode will prevent loading some integrations, so memory usage will reduce anyway, so it won't really prove much.

jolly prawn May 22, 2024, 7:03 AM

#

🤷

spice wigeon May 22, 2024, 7:04 AM

#

It's just died again (triggered by trying to access the UI). This was the last output of top: ```
top - 17:03:45 up 12 min, 0 user, load average: 50.91, 24.84, 9.88
Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.9 us, 68.6 sy, 0.0 ni, 0.0 id, 28.7 wa, 0.0 hi, 1.9 si, 0.0 st
MiB Mem : 1888.7 total, 24.3 free, 1869.9 used, 44.7 buff/cache
MiB Swap: 623.3 total, 0.2 free, 623.1 used. 18.8 avail Mem

#


    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    573 root      20   0    3984    640    384 R   3.1   0.0   0:03.04 top
    463 root      20   0    6588      0      0 S   0.3   0.0   0:01.24 sshd
     18 root      20   0     208      0      0 S   0.0   0.0   0:00.00 s6-supervise
     27 root      20   0     208      0      0 S   0.0   0.0   0:00.00 s6-supervise
      1 root      20   0     424      0      0 S   0.0   0.0   0:00.05 s6-svscan
     29 root      20   0     208      0      0 S   0.0   0.0   0:00.00 s6-supervise
     30 root      20   0     208      0      0 S   0.0   0.0   0:00.00 s6-supervise
     38 root      20   0     196      0      0 S   0.0   0.0   0:00.00 s6-ipcserverd
     20 root      20   0     192      0      0 S   0.0   0.0   0:00.00 s6-linux-init-s
     28 root      20   0     208      0      0 S   0.0   0.0   0:00.00 s6-supervise
    310 root      20   0    7068      0      0 S   0.0   0.0   0:00.18 ttyd
    313 root      20   0    6252    384    256 S   0.0   0.0   0:00.09 sshd
    465 root      20   0    6448    172    128 S   0.0   0.0   0:01.74 zsh
client_loop: send disconnect: Broken pipe

#

Notice it's out of swap, and has only 24+44MB of RAM left.

#

I'm running exactly the same configs on the Yellow and the RPI4.

#

The large swap usage suggests thrashing.

spice wigeon May 22, 2024, 8:01 AM

#

Just harking back to your suggestion that the RAM use is abnormal, how would you know that without more information? I did say I have few add-ons, but I do have quite a few integrations. Can't provide a list, since the only place to see those appears to be in the Repairs/overflow menu/Integrations startup times, which does not provide a copy-able format. I've seen several comments to the effect that 2GB is a bare minimum for HA, which isn't unreasonable, just that shipping the Yellow with only the bare minimum seems a bit silly, given that to upgrade requires a complete new CM4 module (which I now have on order - 8GB)

#Yellow out of memory, HA dies.