Standalone Tool

[kevinmore]

Hi my awesome community,

I would like to have some sort of CLI tool so that I can deploy it to AWS.
My workflow will be something like:

The user uploads a 2D photo, and then a corresponding SBS 3D stereo photo is generated. And the 3D stereo photo will be consumed by a VR headset.

It would be great to have this feature, and I'm aware of the discussion here. However, that thread doesn't quite fit my workflow.

What's the best practice for displaying 3D photos in a VR headset, using SBS 3D stereo photo or RGB + depth in a plane geometry?

Also, could we generate 3D video using this method?

Would you please list the detailed steps for making a CLI tool please?

Thank you!

[semjon00]

Hopefully CLI stand-alone will be possible in the future. Right now the script is tied to WebUI so it is not possible. Maybe it is possible to deploy the whole WebUI with the plugin?.. but yes, this sounds sketchy.

Not sure what you mean by "RGB + depth in a plane geometry", but Generate stereoscopic image(s) option is what I use. Processing video is possible by extracting individual frames from the video and running a batch process on them, but the results would be lacking - the output might not be temporally consistent (will look jarring in animation). There is research to make it much better, though.

[kevinmore]

Thanks for your answer. "RGB + depth in a plane geometry" is the way the blender tutorial does in the readme. Basically it creates a plane geometry and set the depth texture as the displacement, and then assign the original RGB image as the main texture.
Yep, I guess 3D Stereo image is the correct way for VR headsets consuming. And I agree with you, 3D video will be buggy.

Would you provide more information on the steps for making a standalone tool?

[semjon00]

That would require making the code more modular by splitting it into multiple files. The run_depthmap function that is directly responsible for the generation should be in a file where stuff that relies on WebUI is not imported. Such as modules and gradio. This way, the run_depthmap could be imported separately, without the direct need in WebUI. It you decide to embark on this adventure, please also help us by submitting a MR (but there may be merge conflicts... 😬).

[semjon00]

This would be a very welcomed contribution, but I think that this task is better done in small steps.

[thygate]

I processed a video using a ZoeDepth model, no boost, stereo image generation, and I found the output to already be very temporally coherent. I was expecting somewhat ok results since those models are supposed to output in metric units, and indeed, there is only a slight shimmer around the foreground subjects.
This is promising! Batch processing is not very optimized at the moment though, and I can't use more than a few thousand files before i get a 'too many open files' error ..

[disk02]

What's best practice for displaying 3D photos in a VR headset

I find the best way is to use Virtual Desktop to enable SBS mode to view photos loaded into StereoPhoto Maker. StereoPhoto Maker is a free tool with a ton of functionality I really could not give it higher praise. Among many features the biggest benefit I find is the ability to zoom while maintaining the correct alignment for stereo SBS. Unfortunately Virtual Desktop on PC does cost $15 but it's essential in my opinion.

[kevinmore]

Hello, OP here.

Just a quick updated:
I've managed to create a standalone tool, using LeRes model with the BOOST merge model for estimating a depth map, and using it further for generating stereo images. It's a totally standalone repository, and mostly just copy/paste stuff, so I decide not to make a PR here.

And for the best way(cost $0) to view the stereo images, I found this tutorial to be very helpful:
https://youtu.be/_HdoPnvChe0

I'm still investigating the possibilies to add depth to a video, but not urgent at the moment.

Thank you all for your seggestions.