using IR LED and mono cameras for skeleton detection

[stephanschulz]

I am able to display the right mono camera and can see the IR LED is illuminating the scene.
But the blazepose seems to be run on the RGB camera.
I would like to use the camera in a dark environment and would like to learn how to let blazepose use the mono camera that can see in the dark thanks to the IR LED.

[geaxgx]

You have to modify the pipeline in order to replace the ColorCamera node with a MonoCamera node and slightly modify the ImageManip nodes to convert the mono frame to a color frame (because the NeuralNetwork nodes after the ImageManip nodes take color frame as input). I think you should need to add something like this:
manipRight.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p)

IBut it is not guaranted that the models work well on mono frames. I know for instance that hand palm detection does not work well on mono frames.

[stephanschulz]

Thanks for the hint.
This sounds pretty involved to make these changes.

I guess i can pick for MonoCamera the left or right camera.
They already get set in line 249.

right = pipeline.createMonoCamera() right.setBoardSocket(dai.CameraBoardSocket.RIGHT) right.setResolution(mono_resolution) right.setFps(self.internal_fps)

Would i need to comment out the right camera pipeline since i am going to use it as my main camera already?
But it is used in the right.out.link(stereo.right) stereo camera output link.

I can't just simple change cam = pipeline.create(dai.node.ColorCamera) to cam = pipeline.create(dai.nodeMonoCamera) because the MonoCamera does not have setInterleaved, setIspScale, setVideoSize, setPreviewSize, etc

As you already mentioning i need to convert one of the MonoCamera images in to a RGB to hand to balzepose.
I just do not know how to setup a manipRight image properly.

i get AttributeError: 'depthai.node.MonoCamera' object has no attribute 'initialConfig'

IBut it is not guaranted that the models work well on mono frames. I know for instance that hand palm detection does not work well on mono frames.
I have tested blazepose on greyscale video before and it works pretty well.

I wonder if your "DepthAI Pipeline Graph" can help me set this up easier?

[geaxgx]

"DepthAI Pipeline Graph" will just help you to visualize the nodes and the connections between the nodes.
An easy way to check if blazepose works well on mono images is the following:

• make a color video (with any camera);
• convert the video to grayscale (for instance by using this method: https://davidwalsh.name/convert-video-grayscale);
• run blazepose in host mode with the grayscale video as input : ./demo.py -i video.mp4

I think the error you get (AttributeError: 'depthai.node.MonoCamera' object has no attribute 'initialConfig') because you are trying to apply .initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p) to a MonoCamera node instead of ImageManip node, no ?

[shamus333]

@geaxgx Hello, I also used the grayscale camera to load the algorithm of blazepose, updated the pipeline, and added color_ The cam node is changed to mono_ Cam, and added the operation of gray2bgr, but the output detection frame and posture point are not on my body, but on my upper left, and the posture point is too jittery. I changed the resolution of the internal frame to 720p, but the effect has not changed. May I ask if my changes are not complete enough? I hope to get an early reply. Thank you

[geaxgx]

@shamus333 I can't say without having a look at your code.
Have you an output detection output to demonstrate the behaviour ?

[shamus333]

@geaxgx OK, I have modified part of the code, and now it is uploaded to you in the form of an attachment. The running instruction is Python 3 demo py -e --lm_ M Lite, depthai version is 2.17.0.0. The phenomenon is shown in the picture, （The source file is too large. I only uploaded the modified file to you. You can directly overwrite the modified code on the basis of the source code）

modify.zip

[geaxgx]

It seems to be a problem with the padding. I am busy and can't spend time on your problem now but I had a quick glance to your code.
It is maybe related to this line:
mono_manip.initialConfig.setResize(300, 300) which crops the frame. Can you try without it ?

Also can you give me the result printed for this line:
print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}") ?
The call to mpu.find_isp_scale_params() does not make sense in the context of a monochrome camera.

[stephanschulz]

Thanks @shamus333 for posting your code. It is very helpful to see how ti use mono_manip.
here is my repo: https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono/BlazeposeDepthaiEdge.py#L270

print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}")
prints out
Internal camera image size: 1152 x 648 - pad_h: 252

I did remove mono_manip.initialConfig.setResize(300, 300)

FYI i am working on the OAK-D Pro POE not the USB version.

I commented everything related to cam = pipeline.create(dai.node.ColorCamera) since pose and depth detection is only done on the mono camera feeds.

 mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_400_P
            left = pipeline.createMonoCamera()
            left.setBoardSocket(dai.CameraBoardSocket.LEFT)
            left.setResolution(mono_resolution)
            left.setFps(self.internal_fps)

            right = pipeline.createMonoCamera()
            right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
            right.setResolution(mono_resolution)
            right.setFps(self.internal_fps)

            # Define mono camera Gray2BGR
            print("Creating Gray2BGR Camera...")
            mono_manip = pipeline.createImageManip()
#            mono_manip.initialConfig.setResize(300, 300)
            mono_manip.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
            right.out.link(mono_manip.inputImage)
            
            mono_manip_out = pipeline.create(dai.node.XLinkOut)
            mono_manip_out.setStreamName("mono_manip_out")
            mono_manip_out.input.setQueueSize(1)
            mono_manip_out.input.setBlocking(False)
            mono_manip.out.link(mono_manip_out.input)
            
            print(f"Internal camera image size: {self.img_w} x {self.img_h} - pad_h: {self.pad_h}")
        
            stereo = pipeline.createStereoDepth()
            stereo.setConfidenceThreshold(230)
            # LR-check is required for depth alignment
            stereo.setLeftRightCheck(True)
#            stereo.setDepthAlign(dai.CameraBoardSocket.RGB)
            stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT)
            stereo.setSubpixel(False)  # subpixel True -> latency

I too see the offset even though i am doing on right.out.link(mono_manip.inputImage) and output mono_manip.out.link(mono_manip_out.input)

I also see this SpatialLocationCalculator(6) error
`
Here as a graph

[stephanschulz]

i have tested the monoCamera + tracking in the dark outside and it works OKish. It works best when using the heavy model.'
But i am still not certain if the stereo calibration (know that i took out all the RGB camera stuff) is accurate. I see the Z values jump around a lot. And maybe the fact that the mono_manip image is a different size (which size?) than the left+right stereo images could also mean wrong depth data?

https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L323

[geaxgx]

@stephanschulz I think you should align the stereo depth frame with the mono camera instead of the color camera:
https://github.com/stephanschulz/depthai_blazepose/blob/81f17065cc9c051ba0e69db16340396691083e77/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L342

[stephanschulz]

@geaxgx thanks for the tip.
Visually this still does not fix the offset between IR image and skeleton.
But i guess it results in better depth data?

The options seem to be AUTO, RGB, LEFT, RIGHT
https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.CameraBoardSocket

Since i do right.out.link(mono_manip.inputImage) to create the image blazepose gets handed i assume i should also do stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT).

I did notice that detection and tracking is much better on the RGB image, probably due to the fact that the model was trained on RGB data. But i need to use it outside in the dark.
Since the resolution of the right image is much lower than the RGB image
Do you think lower the threshold and setting subPIxels to True helps to improve the chance to find skeletons?
setConfidenceThreshold(50)
stereo.setSubpixel(False)

[geaxgx]

@stephanschulz

Since i do right.out.link(mono_manip.inputImage) to create the image blazepose gets handed i assume i should also do stereo.setDepthAlign(dai.CameraBoardSocket.RIGHT).

That's what I would have tried too.

Do you think lower the threshold and setting subPIxels to True helps to improve the chance to find skeletons?
I don't think so. Modifying these values will impact only the stereo processing result but has no impact on the body detection or landmarks regression.

If you use the depthai demo, you can have a feeling about the influence of these values:
https://docs.luxonis.com/en/latest/pages/tutorials/first_steps/#depth-properties

[stephanschulz]

I am now trying to increase the monocamera resolution, which i think is done with mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_400_P. THE_480_P also works.
But when i try the other options
mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_720_P
mono_resolution = dai.MonoCameraProperties.SensorResolution.THE_800_P
it get this error message:

[18443010C121C61200] [10.100.0.21] [147.942] [ImageManip(3)] [error] Output image is bigger (2764800B) than maximum frame size specified in properties (1048576B) - skipping frame.
Please use the setMaxOutputFrameSize API to explicitly config the [maximum] output size.

Maybe the error does not come from the left or right camera but rather from mono_manip?
https://github.com/stephanschulz/depthai_blazepose/blob/monocamera/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py#L318
But if i try to set the image size with mono_manip.initialConfig.setResize(self.frame_size)# setResize(1280, 800)
i get this error

Traceback (most recent call last):
  File "demo_osc.py", line 60, in <module>
    tracker = BlazeposeDepthai(input_src=args.input, 
  File "/Users/stephanschulz/Documents/depthai-python-2/depthai_blazepose/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py", line 226, in __init__
    self.device.startPipeline(self.create_pipeline())
  File "/Users/stephanschulz/Documents/depthai-python-2/depthai_blazepose/depthai_mono_osc/BlazeposeDepthaiEdge_osc.py", line 320, in create_pipeline
    mono_manip.initialConfig.setResize(self.frame_size)# setResize(1280, 800)
TypeError: setResize(): incompatible function arguments. The following argument types are supported:
    1. (self: depthai.ImageManipConfig, w: int, h: int) -> depthai.ImageManipConfig
    2. (self: depthai.ImageManipConfig, size: Tuple[int, int]) -> depthai.ImageManipConfig

[geaxgx]

@stephanschulz When you use an ImageManip node, you must specify the maximum output size, like it is done there:

depthai_blazepose/BlazeposeDepthaiEdge.py

Line 280 in a3ce15a

pre_pd_manip.setMaxOutputFrameSize(self.pd_input_length*self.pd_input_length*3)

So in your case, depending on the resolution width x height you use, you should have:
mono_manip.setMaxOutputFrameSize(width*height*3)

[stephanschulz]

that works.
obvisouly it slows down everything. with the heavy model i am now getting 3.4 fps.
thanks again.