stream keeps dying periodically

[davidus-sk]

Stream works for a few minutes then dies and this is what the binary outputs:

=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(61) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(61) at index 0
[v4l2_utils] Error: fd(61) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(61) at index 1
[v4l2_utils] Error: fd(62) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(62) at index 0
[v4l2_utils] Error: fd(62) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(62) at index 1
[v4l2_utils] Error: fd(68) turn on stream: No such file or directory
[v4l2_utils] Error: fd(60) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(60) at index 0
[v4l2_utils] Error: fd(68) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(68) at index 0
[v4l2_utils] Error: fd(60) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(60) at index 1
[v4l2_utils] Error: fd(68) queue buffer(10): Bad address

[v4l2_codec] Error: QueueBuffer V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE. fd(68) at index 1
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(145) turn on stream: No such file or directory
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(158) turn on stream: No such file or directory
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(137) turn on stream: No such file or directory
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(137) turn on stream: No such file or directory
=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(160) turn on stream: No such file or directory
[v4l2_utils] Error: fd(69) turn on stream: No such file or directory
[v4l2_utils] Error: fd(60) turn on stream: No such file or directory
[v4l2_utils] Error: fd(69) turn on stream: No such file or directory
[v4l2_utils] Error: fd(70) turn on stream: No such file or directory
[v4l2_utils] Error: fd(62) turn on stream: Timer expired
[v4l2_utils] Error: fd(70) set ext ctrl(10029541): Invalid argument
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(62) turn on stream: Timer expired

Restarting the binary fixes things temporarily until it happens again.

Also seeing this in dmesg:

[ 3723.226847] ------------[ cut here ]------------
[ 3723.226852] WARNING: CPU: 2 PID: 10508 at drivers/media/common/videobuf2/videobuf2-core.c:2024 __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[ 3723.226901] Modules linked in: tun rfkill vc4 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper binfmt_misc drm_kms_helper imx708 snd_soc_core dw9807_vcm bcm2835_unicam bcm2835_codec(C) v4l2_dv_timings bcm2835_v4l2(C) bcm2835_isp(C) v4l2_fwnode v4l2_mem2mem snd_compress videobuf2_vmalloc v4l2_async bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_memops snd_pcm_dmaengine videobuf2_v4l2 videodev snd_bcm2835(C) snd_pcm raspberrypi_hwmon videobuf2_common snd_timer mc snd vc_sm_cma(C) raspberrypi_gpiomem uio_pdrv_genirq uio i2c_dev usb_f_ecm g_ether drm usb_f_rndis u_ether libcomposite fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6 dwc2 i2c_mux_pinctrl i2c_mux i2c_bcm2835
[ 3723.227084] CPU: 2 PID: 10508 Comm: CaptureThread Tainted: G WC 6.6.51+rpt-rpi-v8 #1 Debian 1:6.6.51-1+rpt3
[ 3723.227093] Hardware name: Raspberry Pi Compute Module 3 Plus Rev 1.0 (DT)
[ 3723.227099] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 3723.227107] pc : __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[ 3723.227136] lr : __vb2_queue_cancel+0x38/0x2a0 [videobuf2_common]
[ 3723.227164] sp : ffffffc084943a70
[ 3723.227168] x29: ffffffc084943a70 x28: ffffff800316e438 x27: 0000000000000009
[ 3723.227182] x26: 0000000000000001 x25: 0000007fa119070c x24: ffffff800316e240
[ 3723.227195] x23: ffffff8002aa0020 x22: ffffff801234b828 x21: ffffff8005cf8530
[ 3723.227208] x20: ffffff801234b8d0 x19: ffffff801234b828 x18: ffffffffffffffff
[ 3723.227221] x17: 756f656d6954203a x16: 737265666675625f x15: 6873756c665f6365
[ 3723.227235] x14: 646f635f35333832 x13: 676e69646e617473 x12: 74756f2032202d20
[ 3723.227248] x11: 64656e7275746572 x10: ffffffe20aeb3710 x9 : ffffffe2099ff8a0
[ 3723.227261] x8 : 00000000ffffefff x7 : ffffffe20aeb3710 x6 : 80000000fffff000
[ 3723.227274] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 3723.227286] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000002
[ 3723.227299] Call trace:
[ 3723.227304] __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[ 3723.227333] vb2_core_queue_release+0x2c/0x60 [videobuf2_common]
[ 3723.227361] vb2_queue_release+0x18/0x30 [videobuf2_v4l2]
[ 3723.227387] v4l2_m2m_ctx_release+0x28/0x50 [v4l2_mem2mem]
[ 3723.227419] bcm2835_codec_release+0x64/0x110 [bcm2835_codec]
[ 3723.227439] v4l2_release+0xec/0x100 [videodev]
[ 3723.227539] __fput+0xc4/0x290
[ 3723.227550] ____fput+0x18/0x30
[ 3723.227558] task_work_run+0x80/0xe0
[ 3723.227568] do_exit+0x30c/0x988
[ 3723.227576] do_group_exit+0x3c/0xa0
[ 3723.227583] get_signal+0x9e4/0x9f0
[ 3723.227591] do_notify_resume+0x318/0x1370
[ 3723.227599] el0_svc+0xd8/0xe8
[ 3723.227611] el0t_64_sync_handler+0x100/0x130
[ 3723.227618] el0t_64_sync+0x190/0x198
[ 3723.227627] ---[ end trace 0000000000000000 ]---
[ 3723.227634] videobuf2_common: driver bug: stop_streaming operation is leaving buf 00000000e1bfe418 in active state
[ 3723.227644] videobuf2_common: driver bug: stop_streaming operation is leaving buf 0000000074a362bb in active state
[ 3726.330759] bcm2835_mmal_vchiq: timed out waiting for sync completion
[ 3726.330783] bcm2835-codec bcm2835-codec: bcm2835_codec_stop_streaming: Failed disabling i/p port, ret -62
[ 3729.402774] bcm2835_mmal_vchiq: timed out waiting for sync completion
[ 3729.402797] bcm2835-codec bcm2835-codec: bcm2835_codec_stop_streaming: Failed enabling component, ret -62

[TzuHuanTai]

What is your full command for pi_webrtc? Are you using the --libcamera flag?

I see Raspberry Pi Compute Module 3 and kernel 6.6.51 in your dmsg. Raspberry Pi officials had mentioned that Broadcom camera stack will not support latest camera module, include module 3. ref

[davidus-sk]

Thanks for your response:

Full command: pi_webrtc --use_libcamera --fps=10 --width=1280 --height=720 --hw_accel --no_audio --mqtt_host=<server> --mqtt_port=8883 --mqtt_username=<user> --mqtt_password=<pass> --uid=<uuid>

I am using an Arducam camera, not any of the official pi cam module(s).

I ran rpi-update on the CM3 which I think fixed something, but now I am getting bunch of these:

[ 2864.924229] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 2893.945228] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 2941.055586] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 2941.185915] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 2962.704167] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 2992.734680] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 3019.842865] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 3041.367150] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2
[ 3097.282571] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -2

and still getting bunch of these:

[v4l2_utils] Error: fd(62) turn on stream: No such file or directory
[v4l2_utils] Error: fd(61) turn on stream: No such file or directory
[v4l2_utils] Error: fd(64) turn on stream: No such file or directory
=> Set sdp success!
=> Set sdp success!
[v4l2_utils] Error: fd(192) turn on stream: No such file or directory
[v4l2_utils] Error: fd(64) turn on stream: No such file or directory
[v4l2_utils] Error: fd(192) turn on stream: No such file or directory
[v4l2_utils] Error: fd(64) turn on stream: No such file or directory
[v4l2_utils] Error: fd(64) turn on stream: No such file or directory

[davidus-sk]

After about 2 hours it the binary outputs this and streaming no longer works:

[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(57) dequeue buffer: Invalid argument
[v4l2_utils] Error: fd(65) dequeue buffer: Invalid argument

[TzuHuanTai]

Could you please remove --hw_accel and take a try under software encoding mode?

The error log from [v4l2_utils] is kind of weird. There're only two function use v4l2_utils, one is reading camera via legacy v4l2 stack, another is hardware encoder. Your command already apply --use_libcamera, so the problem must come from hw encoder.

[davidus-sk]

Thanks for the suggestion. I took --hw_accel out but now it won't do anything at all. It wont stream or even connect to MQTT.

The executable hangs with low CPU utilization (at like 5%) and the log file ends here:

[1:26:00.118424808] [26907] INFO Camera camera_manager.cpp:325 libcamera v0.3.2+99-1230f78d
[1:26:00.284158309] [26915] WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[1:26:00.289021286] [26915] INFO RPI vc4.cpp:447 Registered camera /base/soc/i2c0mux/i2c@0/imx708@1a to Unicam device /dev/media1 and ISP device /dev/media0
[libcamera_capturer] camera id: /base/soc/i2c0mux/i2c@0/imx708@1a
[libcamera_capturer] Fps: 10
[libcamera_capturer] AutoFocus: 2
[libcamera_capturer] Rotation: 0
[libcamera_capturer] camera original format: 1920x1080-YUV420
[libcamera_capturer] camera validated format: 1280x720-YUV420.
[libcamera_capturer] width: 1280, height: 720, stride: 1280
[1:26:00.290127839] [26907] INFO Camera camera.cpp:1197 configuring streams: (0) 1280x720-YUV420
[1:26:00.290717625] [26915] INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@0/imx708@1a - Selected sensor format: 1536x864-SBGGR10_1X10 - Selected unicam format: 1536x864-pBAA

^ there are no more lines in the log even if I let it sit for an hour. When I kill the process and restart it, it does the same thing.

[davidus-sk]

Could you please remove --hw_accel and take a try under software encoding mode?

The error log from [v4l2_utils] is kind of weird. There're only two function use v4l2_utils, one is reading camera via legacy v4l2 stack, another is hardware encoder. Your command already apply --use_libcamera, so the problem must come from hw encoder.

I have tried it without --hw_accel on the CM3 and it is way more stable. Still you get error messages, but at some point I got itt to stream for 7 hours non-stop.

[TzuHuanTai]

I've never tried to watch a live stream for more than 3 hours without a break.
I don't have a CM3(BCM2837) so I'm testing on my pi zero 2w (BCM2710A1) and --hw_accel.
I'll let the client play the stream for 24hr and see what happens.
The cpu usage is steady between 15 and 20% with the help of hardware acceleration, hopefully it won't overheat.

pi@home-pi-zero2w:~ $ uname -a
Linux home-pi-zero2w 6.6.51+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.51-1+rpt3 (2024-10-08) aarch64 GNU/Linux
pi@home-pi-zero2w:~ $ /home/pi/pi_webrtc --fps=30 --width=1280 --height=720 --mqtt_host=xxxx --mqtt_port=8883 --mqtt_username=xxxx --mqtt_password=xxxx --uid=home-pi-zero2w --use_libcamera --no_audio --hw_accel

[TzuHuanTai]

My client-side application has been streaming from the Pi Zero 2W for over 24 hours uninterrupted without any issues.
I believe it can continue working for another 24 hours. I suspect your issue might be related to the hardware or driver level.