V3 RGBD Node #1198
V3 RGBD Node #1198
Conversation
Serafadam
commented
Dec 18, 2024
•
Serafadam
commented
- Host RGBD Node
- Basalt fixes
- Using stock TBB for Basalt
- Added optional Kompute support for performing calculations on GPU devices with Vulkan support
cmake/ports/kompute/portfile.cmake
Outdated
There was a problem hiding this comment.
There's no official package?
There was a problem hiding this comment.
Not yet, though I can make PR to the official repo
examples/cpp/HostNodes/rgbd.cpp
Outdated
There was a problem hiding this comment.
Maybe we can move the example to a dedicated folder
There was a problem hiding this comment.
Let's wait on the needed Visualizer changes so this examples start working.
So either that blocks that PR or we add the example later.
| auto pc = std::make_shared<PointCloudData>(); | ||
| pc->setTimestamp(colorFrame->getTimestamp()); | ||
| pc->setTimestampDevice(colorFrame->getTimestampDevice()); | ||
| pc->setSequenceNum(colorFrame->getSequenceNum()); | ||
| pc->setInstanceNum(colorFrame->getInstanceNum()); | ||
| auto width = colorFrame->getWidth(); | ||
| auto height = colorFrame->getHeight(); | ||
| pc->setSize(width, height); | ||
|
|
||
| std::vector<Point3fRGB> points; | ||
| // Fill the point cloud | ||
| auto* depthData = depthFrame->getData().data(); | ||
| auto* colorData = colorFrame->getData().data(); | ||
| // Use GPU to compute point cloud | ||
| pimpl->computePointCloud(depthData, colorData, points); | ||
|
|
||
| pc->setPointsRGB(points); | ||
| pcl.send(pc); |
There was a problem hiding this comment.
We should make sure that at width&height of the RGB frame and the Depth frame match.
Ideally also that they are aligned between each other.
There was a problem hiding this comment.
Width & height are checked in initialize method. For aligment checking, we should just check frame id, right?
| auto calibHandler = getParentPipeline().getDefaultDevice()->readCalibration(); | ||
| auto camID = static_cast<CameraBoardSocket>(colorFrame->getInstanceNum()); | ||
| auto width = colorFrame->getWidth(); | ||
| auto height = colorFrame->getHeight(); | ||
| auto intrinsics = calibHandler.getCameraIntrinsics(camID, width, height); | ||
| float fx = intrinsics[0][0]; | ||
| float fy = intrinsics[1][1]; | ||
| float cx = intrinsics[0][2]; | ||
| float cy = intrinsics[1][2]; | ||
| pimpl->setIntrinsics(fx, fy, cx, cy, width, height); |
There was a problem hiding this comment.
Let's get the intrinsics from the first frame instead (after rvc2_img_transformations_v2 is merged which will likely be before this PR)
| std::shared_ptr<RGBD> RGBD::build(bool autocreate, std::pair<int, int> size) { | ||
| if(!autocreate) { | ||
| return std::static_pointer_cast<RGBD>(shared_from_this()); | ||
| } | ||
| auto pipeline = getParentPipeline(); | ||
| auto colorCam = pipeline.create<node::Camera>()->build(); | ||
| auto depth = pipeline.create<node::StereoDepth>()->build(true); | ||
| auto* out = colorCam->requestOutput(size, dai::ImgFrame::Type::RGB888i); | ||
| out->link(inColor); | ||
| out->link(depth->inputAlignTo); | ||
| depth->depth.link(inDepth); | ||
| return build(); | ||
| } |
There was a problem hiding this comment.
Let's add a warning that this API is "global" and that if used the cameras can't be used for other things.
And we can add the same warning in StereoDepth too.
