[Feature-Request] Use ImageTransport to Handle Publishing Images

[sjgiewont]

why:

I have been playing with the relatively new Compressed Image Publisher feature (#580). This feature seems to not take advantage of the ImageTransport class that automatically handles encoding, publishing and tweaking parameters. For example you are not taking advantage of the compressedDepth image transport that is included in image_transport_plugin package that automatically encodes depth images and creates ROS parameters to tweak the publisher on the fly. Currently there are no default parameters created for the compressed images that you would typically expect (ros-perception/image_transport_plugins#140)

what:

Publishing should be handled by the ImageTransport, see examples here: (https://github.com/ros-perception/image_transport_tutorials/tree/humble).

[Serafadam]

Hi, image publishing actually uses image_transport's CameraPublisher (which uses IT plugins) by default. So in case of MJPEG encoding on DAI side the messages are automatically converted (this is not possible unfortunately with H264 frames), separate publishers are created in two different scenarios

1. i_publish_compressed :true -> this one mitigates message conversion for MJPEG frames to limit overhead and allows publishing H264 frames directly via ffmpeg_msgs (same format image_transport uses)
2. IPC is enabled for the node. Before Jazzy image_transport didn't support IPC.

[sjgiewont]

Thanks for the quick reply. To clarify though, are you saying that enabling the i_publish_compressed flag just forwards the already compressed images from the camera to a compressed publisher?

depthai-ros/depthai_ros_driver/src/dai_nodes/sensors/img_pub.cpp

Line 67 in 28b70a3

node->create_publisher<sensor_msgs::msg::CompressedImage>(pubConfig.topicName + pubConfig.compressedTopicSuffix, rclcpp::QoS(10), pubOptions);

)

Otherwise, for the default ImageTransport::CameraPublisher case, this will perform compression/encoding via the ROS node.

So I then have a question regarding the depth compression. It seems like with the i_publish_compressed flag enabled you are only adding /compressed to the topic name, so on an image_transport subscriber, this is going to be using the /compressed transport instead of the /compressedDepth transport to handle decompression. Is this valid?

[sjgiewont]

So looking into this more I think I see an "issue" with the depth publishing due to what I was alluding to above.

With i_publish_compressed disabled and using the ImageTransport to subscribe to the raw image, the encoding that is resolved by the final ROS ImageMsg is 16UC1 (what i would typically expect for a depth image).

With i_publish_compressed enabled and using the ImageTransport to subscribe to the compressed image, the encoding that is resolved by the final ROS ImageMsg is bgr8.

I think if the compressedDepth transport was used this would resolve to the expected 16UC1 encoding and would not result in loss of depth precision.

[Serafadam]

To clarify though, are you saying that enabling the i_publish_compressed flag just forwards the already compressed images from the camera to a compressed publisher?

Yes

Otherwise, for the default ImageTransport::CameraPublisher case, this will perform compression/encoding via the ROS node.

Yes, based on which IT plugins you have installed.

So I then have a question regarding the depth compression. It seems like with the i_publish_compressed flag enabled you are only adding /compressed to the topic name, so on an image_transport subscriber, this is going to be using the /compressed transport instead of the /compressedDepth transport to handle decompression. Is this valid?

Yes, AFAIK compressedDepth works only with 16bit/float depth values and in case of using low_bandwidth mode for stereo we can only use 8bit (this is a limitation on the device side, might get resolved in some future firmware release). Low bandwidth mode also prevents using subpixel enhancement which improves depth precision.

[sjgiewont]

Yes, AFAIK compressedDepth works only with 16bit/float depth values and in case of using low_bandwidth mode for stereo we can only use 8bit (this is a limitation on the device side, might get resolved in some future firmware release). Low bandwidth mode also prevents using subpixel enhancement which improves depth precision.

Ah ok, so this is a known limitation. Is there an existing feature request for 16bit depth with i_low_bandwidth=True mode on the device side?

Also, if that's the case, if we had i_low_bandwidth=False and i_publish_compressed=True for stereo, shouldn't we ideally be publishing out compressedDepth since the depth data from the camera should be 16bit?

[Serafadam]

Ah ok, so this is a known limitation. Is there an existing feature request for 16bit depth with i_low_bandwidth=True mode on the device side?

We are currently working on V3 version of DepthAI firmware which might include such features (and many more!) but at this moment it's hard to give ETA due to large amount of work still needed. Also, it might be possible that this feature will be RVC4 (OAK4) exclusive.

Also, if that's the case, if we had i_low_bandwidth=False and i_publish_compressed=True for stereo, shouldn't we ideally be publishing out compressedDepth since the depth data from the camera should be 16bit?

publish_compressed works in tandem with low_bandwidth, it won't really do anything on it's own since it transforms DAI compressed messages to ROS compressed messages. If low bandwidth is disabled then image_transport's publishers/plugins kick in, in that case we convert DAI raw message to ROS Image message which in turn gets converted to compressed messages via IT's internals, this way compressedDepth topic is included automatically.
In case of using just low_bandwidth, we convert DAI compressed message to ROS Image message, this introduces additional overhead on host PC but improves bandwidth performance on camera->host communication.

[sjgiewont]

If low bandwidth is disabled then image_transport's publishers/plugins kick in, in that case we convert DAI raw message to ROS Image message which in turn gets converted to compressed messages via IT's internals, this way compressedDepth topic is included automatically.

Is that the case though? I'm using the latest release (v2.10.5-humble) and when I set the following:

stereo:
   i_publish_compressed: true
   i_low_bandwidth: false

I do not see any data being published on /oak/stereo/image_raw/compressedDepth topic, and I see an empty buffer being published on the /oak/stereo/image_raw/compressed topic. /oak/stereo/image_raw looks fine