High FFMPEG CPU usage in frigate. | Home Assistant | Page 1

rapid socket Oct 24, 2024, 9:32 PM

#

I keep getting a message in frigate that my camera has high FFMPEG CPU usage (63%).

I'm using a coral, using proxmox for Home Assistant. I only have one stream. My processor is an i5-4570T(2C/4T), but will soon upgrade to a I7-4785T (4C/8T).

My stream is a MJPG stream, VLC says media information on it is video resolution 1920x1080, buffer dimensions 1920x1088.

Wondering if my config args are bad or not optimal;

detectors:
  coral:
    type: edgetpu
    device: usb
mqtt:
  host: 192.168.1.30
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqtt
  password: mypass
  stats_interval: 60
cameras:
  lorexrelay:
    ffmpeg:
      input_args: preset-http-reolink
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps
          1 -strftime 1 -c:v libx264 -an
      inputs:
        - path: http://192.168.1.214:3030
          roles:
            - detect
            - record
            - rtmp
    objects:
      track:
        - person
        - car
        - truck
        - cat
    detect:
      width: 1920
      height: 1080
      fps: 15
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 10
    record:
      enabled: true
      retain:
        days: 3
        mode: all
      events:
        retain:  
          default: 30
          mode: motion

shell cedar Oct 24, 2024, 9:45 PM

#

you shouldn't be running detect at 15 fps

#

at most 10 but in most cases the default of 5 is sufficient for detect

#

in general your CPU usage is going to be high because you are encoding the recordings to h264 entirely on the CPU

#

only way to fix that is to use the iGPU for encoding

rapid socket Oct 24, 2024, 9:48 PM

#

shell cedar only way to fix that is to use the iGPU for encoding

is this something i should/can do? it's Intel HD Graphics 4400/4600.

shell cedar Oct 24, 2024, 9:49 PM

#

no reason not to try

rapid socket Oct 24, 2024, 10:41 PM

#

shell cedar no reason not to try

i enabled my igpu in my home assistant vm and installed my drivers for it, it seems to detect in frigate now (intel GPU % in status bar).

frigate no longer sees my camera feed, though. my cameras section now looks like this, i think i butchered something;

cameras:
  lorexrelay:
    ffmpeg:
      hwaccel_args: preset-vaapi
      input_args: preset-http-reolink
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps
          1 -strftime 1 -c:v h264_vaapi
      inputs:
        - path: http://192.168.1.214:3030
          roles:
            - detect
            - record
            - rtmp
    objects:
      track:
        - person
        - car
        - truck
        - cat
    detect:
      width: 1920
      height: 1080
      fps: 5
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 30
    record:
      enabled: true
      retain:
        days: 3
        mode: all
      events:
        retain:  
          default: 30
          mode: motion

shell cedar Oct 24, 2024, 10:44 PM

#

need to see logs

#

oh

#

you can't use preset-vaapi for the hwaccel_args because your input is not h264/h265

#

set hwaccel_args: []

rapid socket Oct 24, 2024, 11:38 PM

#

2024-10-24 23:34:30.263649431 [INFO] Preparing Frigate...
2024-10-24 23:34:30.300484659 [INFO] Starting Frigate...
2024-10-24 23:34:32.562415821 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:32.562540830 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:32.562666625 [2024-10-24 19:34:32] frigate.util.config 2024-10-24 23:34:32.583949968 [2024-10-24 19:34:32] frigate.util.config 2024-10-24 23:34:32.796120190 [2024-10-24 19:34:32] frigate.util.services 2024-10-24 23:34:32.825941390 [2024-10-24 19:34:32] peewee_migrate.logs 2024-10-24 23:34:32.826588512 [2024-10-24 19:34:32] peewee_migrate.logs 2024-10-24 23:34:32.838717382 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:32.845557052 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:32.851647356 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:32.891145309 [2024-10-24 19:34:32] detector.coral 2024-10-24 23:34:32.892984622 [2024-10-24 19:34:32] frigate.app 2024-10-24 23:34:33.068269455 [2024-10-24 19:34:33] frigate.app 2024-10-24 23:34:33.068323454 [2024-10-24 19:34:33] frigate.app 2024-10-24 23:34:35.654151367 [2024-10-24 19:34:32] frigate.detectors.p 2024-10-24 23:34:35.659035869 [2024-10-24 19:34:35] frigate.detectors.p 2024-10-24 23:35:36.952602707 [2024-10-24 19:35:36] frigate.video 2024-10-24 23:35:36.952797422 [2024-10-24 19:35:36] frigate.video 2024-10-24 23:35:43.104052427 [2024-10-24 19:35:43] watchdog.lorexrelay 2024-10-24 23:35:43.104327729 [2024-10-24 19:35:43] watchdog.lorexrelay 2024-10-24 23:35:43.104441127 [2024-10-24 19:35:43] ffmpeg.lorexrelay.detect 2024-10-24 23:35:43.104626420 [2024-10-24 19:35:43] ffmpeg.lorexrelay.detect 2024-10-24 23:35:43.104742967 [2024-10-24 19:35:43] ffmpeg.lorexrelay.detect 2024-10-24 23:35:43.104893340 [2024-10-24 19:35:43] ffmpeg.lorexrelay.detect INFO : Starting Frigate (0.14.1-f4f3cfa)
INFO : Creating directory: /tmp/cache
INFO : Checking if frigate config needs migration...
INFO : frigate config does not need migration...
INFO : Automatically detected vaapi hwaccel for video decoding
INFO : Starting migrations
INFO : There is nothing to migrate
INFO : Recording process started: 362
INFO : Recording process started: 364
INFO : go2rtc process pid: 114
INFO : Starting detection process: 391
INFO : Output process started: 393
INFO : Camera processor started for lorexrelay: 443
INFO : Capture process started for lorexrelay: 444
lugins.edgetpu_tfl INFO : Attempting to load TPU as usb
lugins.edgetpu_tfl INFO : TPU found
ERROR : lorexrelay: Unable to read frames from ffmpeg process.
ERROR : lorexrelay: ffmpeg process is not running. exiting capture thread...
ERROR : Ffmpeg process crashed unexpectedly for lorexrelay.
ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
ERROR : [AVHWDeviceContext @ 0x55de64d67ec0] libva: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
ERROR : [h264_vaapi @ 0x55de64d659c0] No quality level set; using default (20).
ERROR : [mpjpeg @ 0x55de64d5c400] Packet corrupt (stream = 0, dts = NOPTS), dropping it.
ERROR : http://192.168.1.214:3030: Connection timed out

#

cameras:
  lorexrelay:
    ffmpeg:
      hwaccel_args: 
        - -hwaccel
        - vaapi
        - -hwaccel_device
        - /dev/dri/renderD128 # Adjust this path based on your setup
      input_args: preset-http-reolink
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_vaapi -an -vf "format=nv12,hwupload"
      inputs:
        - path: http://192.168.1.214:3030
          roles:
            - detect
            - record
            - rtmp
    objects:
      track:
        - person
        - car
        - truck
        - cat
    detect:
      width: 1920
      height: 1080
      fps: 5
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 30
    record:
      enabled: true
      retain:
        days: 3
        mode: all
      events:
        retain:  
          default: 30
          mode: motion

#

tried a bunch of different record options but it just crashes when i try to introduce vaapi args.

shell cedar Oct 24, 2024, 11:56 PM

#

You need to set the driver to i965

rapid socket Oct 25, 2024, 5:45 AM

#

environment_vars:
  LIBVA_DRIVER_NAME: i965
detectors:
  coral:
    type: edgetpu
    device: usb
mqtt:
  host: 192.168.1.30
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqtt
  password: PASSWORD
  stats_interval: 60
cameras:
  lorexrelay:
    ffmpeg:
      hwaccel_args: []
      input_args: preset-http-reolink
      output_args:
        record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps
          1 -strftime 1 -c:v libx264 -an
      inputs:
        - path: http://192.168.1.214:3030
          roles:
            - detect
            - record
    objects:
      track:
        - person
        - car
        - truck
        - cat
    detect:
      width: 1920
      height: 1080
      fps: 5
    snapshots:
      enabled: true
      bounding_box: true
      retain:
        default: 30
    record:
      enabled: true
      events:
        retain:  
          default: 30
          mode: motion
    zones:
      backyard:
        coordinates: 721,707,1205,714,1208,377,713,376
      backyard_2:
        coordinates: 1359,716,1847,710,1849,374,1356,374
      driveway:
        coordinates: 1351,353,1845,353,1843,22,1349,0
      front_door:
        coordinates: 81,1080,570,1080,572,734,75,737
      garage:
        coordinates: 719,359,1207,356,1205,0,711,0
      patio:
        coordinates: 78,714,572,713,566,373,73,373
      side_door:
        coordinates: 81,357,572,357,572,23,78,0
    motion:
      mask:
        - 0.108,0.057,0.244,0.068,0.24,0.054,0.24,0.036,0.109,0.037
        - 0.44,0.063,0.572,0.057,0.57,0.03,0.434,0.03
        - 0.776,0.063,0.907,0.056,0.904,0.032,0.772,0.033
        - 0.102,0.392,0.242,0.389,0.24,0.367,0.104,0.366
        - 0.438,0.394,0.579,0.392,0.576,0.363,0.437,0.367
        - 0.764,0.394,0.907,0.392,0.904,0.363,0.764,0.364
        - 0.106,0.722,0.239,0.721,0.239,0.694,0.102,0.696
        - 0.371,0.017,0.629,0.021,0.628,0.152,0.372,0.142
version: 0.14

here's what i got now. i don't see the gpu in the status bar anymore. cpu 15%, but camera has high ffmpeg cpu usage (48-60%).

ffmpeg process crashing unexpectedly fairly frequently, dunno if that's because the driver is old and buggy. i don't even know if i'm utilizing the driver, it doesn't seem like i'm leveraging the igpu. maybe my record argument is wrong?

shell cedar Oct 25, 2024, 2:39 PM

#

need to see the logs

#

oh wait you changed back to libx264

#

you should still be using vaapi there

#

just not preset-vaapi since that is used for decoding

rapid socket Oct 25, 2024, 6:27 PM

#

when i switch to vaapi args it doesn't detect the camera anymore. only when i use libx264.

record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
this works with high cpu usage, log; https://dpaste.org/AeKhk

full config ( LIBVA_DRIVER_NAME: i965 and hwaccel_args: [] );
https://dpaste.org/BRvEG

failed vaapi arg attempts;

record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_vaapi -an
doesn't work. log; https://dpaste.org/WZYzr

record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps 1 -strftime 1 -vf "scale=1920:1080" -c:v h264_vaapi -an
https://dpaste.org/zAwXW

record: -f segment -segment_time 30 -segment_format mp4 -reset_timestamps 1 -strftime 1 -vf "scale=1920:1080" -c:v h264_vaapi -pix_fmt yuv420p -an
https://dpaste.org/yNx4K

i''m running haos inside a vm in proxmox, passing my igpu as seen in attached image.

rapid socket Oct 25, 2024, 6:46 PM

#

ffmpeg -i on my input stream gives me this info;

  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 96:96 DAR 16:9], 25 tbr, 25 tbn```

shell cedar Oct 25, 2024, 7:34 PM

#

you need to let the logs run longer so we can see the full errors,

rapid socket Oct 25, 2024, 8:24 PM

#

think i have two issues going on. i'm crashing a lot and restarting because of watchdog, even on libx264 or -c:v mjpeg..

i get "Detection appears to be stuck. Restarting detection process...", google says this is a coral issue. i've tried from a powered hub and i still have the same issue. i know the coral isn't defective because i wasn't getting this issue when running haos on a pi4.

with -c:v mjpeg it does mention that it detects vaapi hwaccel, which i think is a good sign, but the crashing is driving me crazy.

2024-10-25 19:08:38.219373979  [INFO] Preparing Frigate...
2024-10-25 19:08:38.238004374  [INFO] Starting Frigate...
2024-10-25 19:08:41.060881530  [2024-10-25 15:08:41] frigate.app                    INFO    : Starting Frigate (0.14.1-f4f3cfa)
2024-10-25 19:08:41.061262536  [2024-10-25 15:08:41] frigate.app                    INFO    : Creating directory: /tmp/cache
2024-10-25 19:08:41.061266377  [2024-10-25 15:08:41] frigate.util.config            INFO    : Checking if frigate config needs migration...
2024-10-25 19:08:41.081820948  [2024-10-25 15:08:41] frigate.util.config            INFO    : frigate config does not need migration...
**2024-10-25 19:08:41.284352875  [2024-10-25 15:08:41] frigate.util.services          INFO    : Automatically detected vaapi hwaccel for video decoding**
2024-10-25 19:08:41.313715410  [2024-10-25 15:08:41] peewee_migrate.logs            INFO    : Starting migrations
2024-10-25 19:08:41.314699700  [2024-10-25 15:08:41] peewee_migrate.logs            INFO    : There is nothing to migrate
2024-10-25 19:08:41.321441636  [2024-10-25 15:08:41] frigate.app                    INFO    : Recording process started: 391
2024-10-25 19:08:41.329050568  [2024-10-25 15:08:41] frigate.app                    INFO    : Recording process started: 394
2024-10-25 19:08:41.334227596  [2024-10-25 15:08:41] frigate.app                    INFO    : go2rtc process pid: 114
2024-10-25 19:08:41.369557413  [2024-10-25 15:08:41] detector.coral                 INFO    : Starting detection process: 420
**2024-10-25 19:08:41.377657847  [2024-10-25 15:08:41] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb**
2024-10-25 19:08:41.384742540  [2024-10-25 15:08:41] frigate.app                    INFO    : Output process started: 422
2024-10-25 19:08:41.479830809  [2024-10-25 15:08:41] frigate.app                    INFO    : Camera processor started for lorexrelay: 446
2024-10-25 19:08:41.480171812  [2024-10-25 15:08:41] frigate.app                    INFO    : Capture process started for lorexrelay: 447
2024-10-25 19:08:44.234538139  [2024-10-25 15:08:44] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
**2024-10-25 19:17:21.709517391  [2024-10-25 15:17:21] frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...**
2024-10-25 19:17:21.709664043  [2024-10-25 15:17:21] root                           INFO    : Waiting for detection process to exit gracefully...
2024-10-25 19:17:51.739759579  [2024-10-25 15:17:51] root                           INFO    : Detection process didn't exit. Force killing...
2024-10-25 19:17:51.745552285  [2024-10-25 15:17:51] root                           INFO    : Detection process has exited...
2024-10-25 19:17:51.772628834  [2024-10-25 15:17:51] detector.coral                 INFO    : Starting detection process: 2029
2024-10-25 19:17:54.621161736  [2024-10-25 15:17:51] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-10-25 19:17:54.628512552  [2024-10-25 15:17:54] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found```

#

my coral device in the haos vm that i'm attaching also looks like this, doesn't have a name. dunno if this matters, i saw in another post that you mentioned haos vm would take care of all the drivers.

shell cedar Oct 25, 2024, 8:36 PM

#

yeah, you may want to try a different USB cable

rapid socket Oct 25, 2024, 8:43 PM

#

really? i'm using the stock cable that came with the coral, usb-c to usb-a, pretty short cable.

#

i'm starting to think this is some kind of issue with the device not being passed properly or with full permission from the proxmox system to the haos vm that i'm running frigate in. do you think that's unlikely?

#

https://github.com/home-assistant/operating-system/issues/2552

For me the solution was to move over to an LXC instead of VM. They both handle USB interactions differently, the VM via USB emulation (unless you pass through the whole PCI bus) and the LXC directly via the host.

I can finally report I no longer see any TPU/USB restarts and as an added bonus my inference speed is down from near 30ms in the VM to 9ms on the LXC. Seems like the better choice unless for some reason you must use a VM, passing through the whole PCI bus can have it's own issues so for me would be a last resort.

Video solution I followed in my post here, was all very easy and took around 15 mins in total: blakeblackshear/frigate#8193

i'm getting these stupid errors in my console, too.

shell cedar Oct 25, 2024, 8:48 PM

#

the stock cable is known to be problematic. If the coral works but gets stuck it likely is not a passthrough issue

unique pivot Oct 26, 2024, 7:33 PM

#

Not all USB ports provide enough power for the Coral .

rapid socket Oct 27, 2024, 8:19 PM

#

unique pivot Not all USB ports provide enough power for the Coral .

i'm using an externally powered usb 3.0 hub.

#

i just bought a mini pcie version of the coral, will that give better results? my board has mini pcie.

#High FFMPEG CPU usage in frigate.