#camera-streamer and webrtc without cropping a Raspberry Pi Camera v2

49 messages · Page 1 of 1 (latest)

rocky tulip
#

I have a Raspberry Pi Camera 2 and a Raspberry Pi 4 8GB. I was hoping to use the onboard encoder for the webcam to avoid the bandwidth consumption of MJPEG. This is my first trip down this particular rabbit hole. With the guidance of previous threads, I've already recompiled camera-streamer to get it to work at all. Now it appears to work completely when using the Adaptive MJPEG-Streamer service.

Switching the URL Stream to /webcam/webrtc and the service to WebRTC (camera-streamer) gives me a blank box. Some advice I found from searching other threads suggested reducing the selected mode. So I did that.

Available cameras
-----------------
0 : imx219 [3280x2464] (/base/soc/i2c0mux/i2c@1/imx219@10)
    Modes: 'SRGGB10_CSI2P' : 640x480 [30.00 fps - (0, 0)/0x0 crop]
                             1640x1232 [30.00 fps - (0, 0)/0x0 crop]
                             1920x1080 [30.00 fps - (0, 0)/0x0 crop]
                             3280x2464 [30.00 fps - (0, 0)/0x0 crop]
           'SRGGB8' : 640x480 [30.00 fps - (0, 0)/0x0 crop]
                      1640x1232 [30.00 fps - (0, 0)/0x0 crop]
                      1920x1080 [30.00 fps - (0, 0)/0x0 crop]
                      3280x2464 [30.00 fps - (0, 0)/0x0 crop]```

And the relevant snippet from `crowsnest.conf`:
```device: /base/soc/i2c0mux/i2c@1/imx219@10                     # See Log for available ...
resolution: 1920x1080                     # widthxheight format
custom_flags: --camera-video.height=1080```

This works, but it crops to the center of the frame. This is unfortunate because it makes for a too-narrow field-of-view. Is there a way to use a the 2x2 binning mode to do this downsizing rather than a crop? I'd like to preserve the FOV.
fiery spear
#

Basically the v2 is a 4:3 sensor, so a 16:9 aspect ratio will ofc cut something. This (0, 0)/0x0 crop is totally misleading

rocky tulip
#

OK fair enough, but picking the center 1920x1080 pixels out of a 3280x2464 is kind of undesirable. I don't see anything in that doc about enabling binning so that the incoming stream is 1640x1232. I'm going to see if there are some other options to set somewhere.

fiery spear
#

But the overall problem is that everything goes through the hardware encoding, so no possibility to get over 1080p

rocky tulip
#

Unfortunately those all seem to crop the incoming stream at the sensor, at least with the v2 camera.

#

Oh I see. Maybe the 1640x1232 mode is better. Let's try that.

fiery spear
#

You have to use native resolutions (4:3 aspect ratio), like 3280x2464 and 1640x1232

fiery spear
rocky tulip
#

And it looks like there's no software downscaler prior to sending video in to the hardware encoder.

fiery spear
#

Nope, that's not efficiently possible.

#

The picams don't have hardware encoding, so the pi has to do the encoding. So you would have to first encode the image as mjpg, then downscale and then use hardware encoding for webrtc. This doesn't make sense, as this would just end up using a lot of CPU resources.

#

So overall this would be possible ofc, but you would have to do your own solution, as camera-streamer won't support something like that

rocky tulip
fiery spear
#

Ok, but you still miss the important part. The GPU can only interact with images up to 1080p

rocky tulip
#

The video encoder can only ingest images that large. I'm sure the GPU can do larger, given that it can output 4K video.

#

(it would need to be able to resize its framebuffer)

#

So the thought would be to receive 1640x1232 camera images, downscale them to 1438x1080 and run them through the hardware encoder.

fiery spear
#

Ok, to get to an end. You can read yourself into this if you want. It's not possible with camera-streamer, I already talked about that with the dev.
It doesn't matter what you do. The Pi4 will hit a lot of CPU load if you software encode an image of 3280x2464. So yes, you can do it ofc, but not efficient

rocky tulip
#

Understood; thanks.

fiery spear
#

So if you want to do something yourself, have a look at picamera2 and go2rtc/ffmpeg

#

Oh, one point I forgot to say....That was a bit dumb sry
The actual size is 1920x1920@30fps iirc. So the 1640x1232 shouldn't a problem.

rocky tulip
#

Interesting. I wonder why webrtc doesn't seem to work at that size.

#

I'm able to get an MJPG stream but not a WebRTC one at that size.

fiery spear
#

Did you set --camera-video.height=1080 explicitely?

rocky tulip
#

I've tried with and without that flag without change in result at 1640x1232 input.

#

I guess that's a lie. With that flag I get a green background and without that flag I get nothing.

fiery spear
#

Someone in the team reported something similar, but I didn't yet investigate.

#

Just fyi, my settings with a picam v3:

[cam 1]
mode: camera-streamer
enable_rtsp: false
rtsp_port: 8554
port: 8080
device: /base/soc/i2c0mux/i2c@1/imx708@1a
resolution: 4608x2592
max_fps: 15
v4l2ctl: afmode=0, lensposition=4.5
#

That's working without any problems

rocky tulip
#

Not using webrtc then? Doesn't seem like you're scaling down to <=1920

fiery spear
#

I'm using webrtc and it's working

#

The default parameter for --camera-snapshot.height=1080 is always set and stream and video are scaling to that too then

#

And I just confirmed with the snapshot endpoint, the image is 1920x1080

rocky tulip
# 
mode: camera-streamer
enable_rtsp: true
rtsp_port: 8554
port: 8080
device: /base/soc/i2c0mux/i2c@1/imx219@10
resolution: 1640x1232
custom_flags: --camera-video.height=1080
max_fps: 15```
fiery spear
#

It might be a problem with the v2, I will check real quick

#

I will talk to the camera-streamer dev. There is definitely something wrong

#

Or maybe it's because of the way camera-streamer is requesting the sensor resolution and that isn't available. I will get back to you

rocky tulip
#

OK, thanks! And let me know if there's anything I can do to help get more information.

fiery spear
#

Try out --camera-snapshot.height=1232

rocky tulip
#

I just went out for lunch. I'll let you know how it goes when I'm back!

rocky tulip
#

Should I set both snapshot and video to the same height?

#

e.g., this command resolution: 1640x1232 custom_flags: --camera-video.height=1232 --camera-snapshot.height=1232

#

I just tried that and it appears to work.

#

Wow the video is so much smoother now that it's not a bazillion bits per second.