Questions about reproducing the demo

[Srameo]

Hi there, I am very interested in the demo showed on the project page, particularly the parts related to repositioning, refocusing, and relighting. I was wondering if it would be possible to open-source the code for the demo?

[Ilya-Muromets]

Hi! Glad you liked the renders in the video, I made all of them in Blender. Unfortunately, as I was just manually rendering things right before the CVPR deadline, my .blend files are not in a working state.

However, I should have some helpful code snippets for turning the depth maps into .glb files that you can input into Blender. I'll look for those and post them here when I find them.

As for repositioning/refocusing, this is all done through manipulating the camera object in Blender. There's a ton of really good tutorials online (e.g. https://youtu.be/Y8-6X0m5hrM for simulating depth-of-field effects) so I recommend going through those if you're interested in rendering these kinds of scenes. It also has a Python API that you can use to automate rendering, but I am not very familiar with it.

[Srameo]

Thank you for your response! However, I have another question. Can these amazing effects be created with just depth data? Can I create these effects with monocular data + depth data?

[Ilya-Muromets]

Yeah, you can turn any image + depth into a mesh to port to blender. Here's the code I use to meshify a depth map, basically turn each square of pixels into a pair of connected triangles. You'll need to have open3d installed.

import open3d as od
import numpy as np
import matplotlib.pyplot as plt
from skimage.transform import resize
from tqdm import tqdm

depth = np.load('depth.npy') # H x W
# make sure the image is float32 between 0-1 or else open3d will not save it properly
img = plt.imread('img.png') # H x W x 3

# resize (12 megapixel is a bit large for Blender)
height, width = 960, 720
depth = resize(depth, (height, width))
img = resize(img, depth.shape)

x, y = np.meshgrid(np.linspace(0, width/height, width), np.linspace(0, 1, height))
x = x.flatten()
y = y.flatten()
z = depth.flatten()

# if you have intrinsics you can properly re-project, e.g.
# x = (x - cx) * (z/fx)
# y = (y - cy) * (z/fy)

xyz = np.stack((x,y,z), -1)

r,g,b = img[:,:,0].flatten(), img[:,:,1].flatten(), img[:,:,2].flatten()
rgb = np.stack((r,g,b), -1)

def make_faces(outliers):
    faces = []
    H,W = depth.shape
    for x in tqdm(range(W-1)):
        for y in range(H-1):
            A,B,C,D = W*y + x, W*y + x + 1, W*(y+1) + x, W*(y+1) + x + 1
            A_out, B_out, C_out, D_out = outliers[A], outliers[B], outliers[C], outliers[D]
            A_in, B_in, C_in, D_in = not outliers[A], not outliers[B], not outliers[C], not outliers[D]
            # A B
            # C D
            if A_in and B_in and C_in and D_in: # fill square
                faces.append(np.array([A,B,C]))
                faces.append(np.array([C,B,D]))
            elif A_in and B_in and D_in and C_out:
                faces.append(np.array([A,B,D]))
            elif B_in and D_in and C_in and A_out:
                faces.append(np.array([B,D,C]))
            elif D_in and C_in and A_in and B_out:
                faces.append(np.array([D,C,A]))
            elif C_in and A_in and B_in and D_out:
                faces.append(np.array([C,A,B]))
                    
    faces = np.array(faces)
    return faces

def write_mesh(name):
    mesh = od.geometry.TriangleMesh()
    mesh.vertices = od.utility.Vector3dVector(xyz)
    mesh.triangles = od.utility.Vector3iVector(faces)
    mesh.vertex_colors = od.utility.Vector3dVector(rgb)
    mesh = mesh.compute_triangle_normals()
    od.io.write_triangle_mesh(name, mesh, write_vertex_colors=True, write_vertex_normals=True, write_triangle_uvs=True)

# remove outliers (e.g. if you want to threshold depth)
outliers = (z >= 100) # or whatever threshold you want

faces = make_faces(outliers)
write_mesh("mesh.glb")

You can then import this mesh directly into Blender. To render the colors correctly you should use the Vertex Color shader node and add a Gamma correction of ~2, see:

The render will then look something like this:

If the depth map looks bumpy or squished, try playing with median filter and thresholding/rescaling.

Best of luck!

[Srameo]

Thank you for your detailed explanation!