Feature: Adding Surface Normals Sensor

[mathfac]

Author @danielgordon10:
I have code to turn depth into surface normals, but ideally it should be something provided by habitat-api directly.

Here is my surface normals code. One nice thing about it is I can batch it and run on GPU, though you should at least be able to do the second part on habitat-sim side if necessary.

import torch
import torch.nn.functional as F

surfnorm_kernel = None
def depth_to_surface_normals(depth, surfnorm_scalar=256):
# depth is torch tensor in N x C x H x W order.
global surfnorm_kernel
if surfnorm_kernel is None:
surfnorm_kernel = torch.from_numpy(
np.array([[[1, 2, 1],
[0, 0, 0],
[-1, -2, -1]],
[[1, 0, -1],
[2, 0, -2],
[1, 0, -1]],
[[0, 0, 0],
[0, 0, 0],
[0, 0, 0]]])
)[:, np.newaxis, ...].to(
dtype=torch.float32, device=depth.device)
with torch.no_grad():
surface_normals = F.conv2d(depth, surfnorm_scalar * surfnorm_kernel, padding=1)
surface_normals[:, 2, ...] = 1
surface_normals = surface_normals / surface_normals.norm(dim=1, keepdim=True)
return surface_normals

[danielgordon10]

Formatting:
I have code to turn depth into surface normals, but ideally it should be something provided by habitat-api directly.

Here is my surface normals code. One nice thing about it is I can batch it and run on GPU, though you should at least be able to do the second part on habitat-sim side if necessary.

import torch
import torch.nn.functional as F

surfnorm_kernel = None
def depth_to_surface_normals(depth, surfnorm_scalar=256):
    # depth is torch tensor in N x C x H x W order.
    global surfnorm_kernel
    if surfnorm_kernel is None:
    surfnorm_kernel = torch.from_numpy(
        np.array([[
            [1, 2, 1],
            [0, 0, 0],
            [-1, -2, -1]],
           [[1, 0, -1],
            [2, 0, -2],
            [1, 0, -1]],
           [[0, 0, 0],
            [0, 0, 0],
            [0, 0, 0]]])
             )[:, np.newaxis, ...].to(dtype=torch.float32, device=depth.device)
    with torch.no_grad():
        surface_normals = F.conv2d(depth, surfnorm_scalar * surfnorm_kernel, padding=1)
        surface_normals[:, 2, ...] = 1
        surface_normals = surface_normals / surface_normals.norm(dim=1, keepdim=True)
        return surface_normals

[Yash621]

@mathfac can I work on this ?

[mathfac]

Yes, sure.

…

On Fri, Nov 5, 2021 at 10:13 AM Yash Goel ***@***.***> wrote: @mathfac <https://github.com/mathfac> can I work on this ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGWHAE3SPJK2G72EZTNDFDUKQGELANCNFSM4HDONC2Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.