Have noticed the last weeks worth of nightly builds I occasionally have an issue where /dev/shm runs out of space, nothing else has changed on the system...
#Latest frigate nightlies keep running out of /dev/shm space..
mossy kettle
silent musk
we need to see logs, config, and preferably lsof and du -h outputs of /dev/shm otherwise there is nothing we can help with
mossy kettle
Hi nick, thanks and yes, apologies I should have got more info rather than just panicking and restarting the docker instance, but will next time depending on the time of day, prefer not to do it at night like now.
mossy kettle
The only logs I can get are from the Portainer log view:
s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device
I can connect, looking now
Output from du -h /dev/shm:
lsof apparently doesn't exist in the image?
lsof of the server running docker is HUGE
It's only running docker with the portainer image and frigate image
I didn't go and grab frigate logs before restarting, apologies
silent musk
lsof apparently doesn't exist in the image?
Just need to install it
silent musk
Output from du -h /dev/shm:
This is before restarting?
mossy kettle
Just need to install it
Oh, how do I do that? I'm not that familiar with what funcationality I have within a docker shell. Thanks
mossy kettle
This is before restarting?
Before restarting, hope that's helpful
mossy kettle
Ah cool
mossy kettle
apt-get update && apt-get install -y lsof
Sorry, forgot to tag yoy
mossy kettle
This is from df, just before I did the lsof that I've uploaded:
shm 1048576 925408 123168 89% /dev/shm
And this is just before writing this message:
shm 1048576 931516 117060 89% /dev/shm
silent musk
and it crashed with shm error correct?
silent musk
we found another issue and fixed it, so that will hopefully imporove what you are seeing as well