FFmpeg crashing in Frigate | Home Assistant | Page 1

real hamlet Dec 26, 2024, 1:11 PM

#

Every few minutes when watching birdseye view, the logs say "Unable to keep up with recording segments in cache for [camera_name]. Keeping the 6 most recent segments out of 11 and discarding the rest..." for every one of my cameras.

This coincides with FFmpeg crashing and then restarting and going back to normal.

I am currently writing over the network to a NFS share, though the NFS share only has a 100mbps uplink, which I am working on rectifying.

Is it because of the slow network/write speed that FFmpeg in Frigate is crashing?

upper marlin Dec 26, 2024, 1:31 PM

#

Maybe: https://github.com/blakeblackshear/frigate/discussions/5392

GitHub

Recordings copy speed from cache to NAS - Unable to keep up with re...

Hi all, I am looking for a bit of help. Thanks to the recent Frigate BETAs I have identified what seems like a very intermittent issue writing to my NAS. The NAS is dedicated to Frigate with a sing...

real hamlet Dec 26, 2024, 2:02 PM

#

Do you think this is linked to FFmpeg crashing?

upper marlin Dec 26, 2024, 2:02 PM

#

Like I said, maybe.

real hamlet Dec 26, 2024, 2:23 PM

#

I bumped the size of shm a bit and it seems to have stabilised, but we'll have to wait and see

#

nevermind

#

Got some error logs though

#

2024-12-26 14:24:40.614829529  14:24:40.614 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://[user]:[pass]@[ip]:554/h265Preview_01_main#video=copy#audio=copy#audio=opus

2024-12-26 14:24:40.631422616  14:24:40.631 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://[user]:[pass]@[ip]:554/h265Preview_01_main#video=copy#audio=copy#audio=opus

2024-12-26 14:24:40.642194705  14:24:40.642 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://[user]:[pass]@[ip]:554/h265Preview_01_main#video=copy#audio=copy#audio=opus

2024-12-26 14:24:40.649816200  14:24:40.649 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error=EOF url=ffmpeg:rtsp://[user]:[pass]@[ip]:554/h265Preview_01_main#video=copy#audio=copy#audio=opus

#

followed by

2024-12-26 14:24:45.732981117  [2024-12-26 14:24:45] ffmpeg.camera2.record          ERROR   : [segment @ 0x563de706ec80] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly

2024-12-26 14:24:45.733039952  [2024-12-26 14:24:45] ffmpeg.camera2.record          ERROR   : rtsp://127.0.0.1:8554/camera2: Connection timed out

2024-12-26 14:24:45.733199135  [2024-12-26 14:24:45] watchdog.camera2               INFO    : Terminating the existing ffmpeg process...

2024-12-26 14:24:45.738159656  [2024-12-26 14:24:45] watchdog.camera2               INFO    : Waiting for ffmpeg to exit gracefully...

real hamlet Dec 26, 2024, 2:54 PM

#

another instance of this happening:

2024-12-26 14:53:13.258785914  [2024-12-26 14:53:13] watchdog.camera4               INFO    : No frames received from camera4 in 20 seconds. Exiting ffmpeg...

2024-12-26 14:53:13.259053388  [2024-12-26 14:53:13] watchdog.camera4               INFO    : Waiting for ffmpeg to exit gracefully...

2024-12-26 14:53:13.260215232  [2024-12-26 14:53:13] ffmpeg.camera2.record          ERROR   : rtsp://127.0.0.1:8554/camera2: Invalid data found when processing input

2024-12-26 14:53:13.260404858  [2024-12-26 14:53:13] watchdog.camera2               INFO    : Terminating the existing ffmpeg process...

2024-12-26 14:53:13.260655582  [2024-12-26 14:53:13] watchdog.camera2               INFO    : Waiting for ffmpeg to exit gracefully...

2024-12-26 14:53:13.266819716  [2024-12-26 14:53:13] frigate.video                  ERROR   : camera4: Unable to read frames from ffmpeg process.

2024-12-26 14:53:13.267639477  [2024-12-26 14:53:13] frigate.video                  ERROR   : camera4: ffmpeg process is not running. exiting capture thread...

2024-12-26 14:53:13.304459115  [2024-12-26 14:53:13] watchdog.camera5               INFO    : No frames received from camera5 in 20 seconds. Exiting ffmpeg...

2024-12-26 14:53:13.305975386  [2024-12-26 14:53:13] watchdog.camera5               INFO    : Waiting for ffmpeg to exit gracefully...

2024-12-26 14:53:13.342908438  [2024-12-26 14:53:13] frigate.video                  ERROR   : camera5: Unable to read frames from ffmpeg process.

2024-12-26 14:53:13.343329868  [2024-12-26 14:53:13] frigate.video                  ERROR   : camera5: ffmpeg process is not running. exiting capture thread...

real hamlet Dec 27, 2024, 8:31 AM

#

this is really frustrating, as ffmpeg keeps crashing

#

hmm, I just disabled recording, and so far, ffmpeg hasn't crashed yet

#

I think it is the slow NAS connection

#

what can I do about this (other than obviously upgrading the 100mbps link on the NAS)?

#

that also makes sense why I couldn't access the other containers if ffmpeg was crashed

#

I saturated the connection

#

what can I do so that under no circumstances does ffmpeg crash, including dropping parts of recordings

#

my HP Elitedesk 800G4 mini (which I'm running frigate on) doesn't support NAS drives (3.5" drives).

sharp forge Dec 27, 2024, 8:43 AM

#

Do you have a way to get stats on bandwidth used? E.g. running one of these in a shell: https://superuser.com/questions/778051/how-to-get-current-bandwidth-usage-from-command-line-using-built-in-linux-tools

Super User

How to get current bandwidth usage from command line using built-in...

I'm looking for a good way to get the current total bandwidth usage on a linux machine. I’ve tried iftop, nethogs, but they don’t show real bandwidth usage compared to system monitor; they don’t su...

sharp forge Dec 27, 2024, 8:46 AM

#

sharp forge Do you have a way to get stats on bandwidth used? E.g. running one of these in a...

That way you could confirm that it is the network

real hamlet Dec 27, 2024, 8:52 AM

#

also after disabling recording my CPU usage dropped a good 20%-30%

#

which is weird

#

yes I just checked, the total network requirements top out at around 150mbps, and so, the 100mbps link is not good enough.

#

thank you for that advice @sharp forge

sharp forge Dec 27, 2024, 8:56 AM

#

👍 figured it might save you upgrading to GbE if your bottleneck is elsewhere

real hamlet Dec 27, 2024, 8:56 AM

#

interesting thing is CPU usage.

#

My hypothesis is that because of the saturated connection, the CPU has to keep resending the dropped recording packets

#

causing a spike in usage

upper marlin Dec 27, 2024, 12:31 PM

#

Kinda curious why your NFS share doesn't have Gigabit.

real hamlet Dec 27, 2024, 12:56 PM

#

upper marlin Kinda curious why your NFS share doesn't have Gigabit.

so I kinda stupidly didn't realise that the SATA port in the HP elitedesk 800 G4 mini didn't support 3.5" drives, due to the missing 12v rail, and so, as a stop gap solution, I'm using a computer from 2008 as a small NAS with my cctv drive in it.

upper marlin Dec 27, 2024, 1:26 PM

#

Always good to have a USB to Ethernet adapter on hand. And/or a docking station/SATA adapter. If you work with computers that's just often very useful.
2008 is old but usually not Megabit old.

real hamlet Dec 27, 2024, 1:45 PM

#

core 2 quad 🫣

#

you would be surprised how much hardware only support 100mbps

#

I have a laptop from 2014 that doesn't support gigabit

real hamlet Dec 28, 2024, 7:27 AM

#

I still have the crashing, even on gigabit connection

#

on the NAS

#

it syas connection timed out

#

rtsp://127.0.0.1:8554/camera2: Connection timed out

#

e.g.

#

this is really annoying

#

there are no more of the discarding frame errors though, because the NAS NIC is able to handle the connections

real hamlet Dec 30, 2024, 1:39 AM

#

I think I may have figured it out (?)

#

I think one of my cameras is faulty

#

which sends corrupt frames

#

and totally just doesn't work at night

#

and it crashes occasionally throughout the day

#

and might be causing a cascade

#FFmpeg crashing in Frigate