Points not lying on the surface

[zgojcic]

Hi,

thanks for open-sourcing this project. I have used it to render some of the shapenet models but have observed that the 3D point do not lie exactly on the surface once back-projected to 3D. The cloud2mesh distance lie in the range of 0.002.

Have you maybe observed the same? Do you know if this is an inherent precision of Blender depth rendering or if there is maybe some precision loss occurring at some point?

Zan

[yinyunie]

Hi,

which mesh was used in your cloud2mesh? If you used the fused mesh, there could be two reasons: 1. we erode the depth map with a kernel size of 2. It enlarges the shape a little bit to avoid missing small structures (like bars on the chair back), see dist_to_dep in tools/utils.py; 2. discretization error with marching cubes algorithm.

[zgojcic]

Hi,

thanks for the answer, maybe I was not clear in the description. I take the original meshes from the ShapnetV2 and render them using this code with Blender Cycles rendering engine. I then take the depth maps and project them to 3D based on the intrinsic/extrinsic camera parameters. If one then loads the mesh model and point cloud into meshlab or CC the points obtained from the depth image do not lie exactly on the surface of the mesh that was used to render them in the first place. If one then computes the cloud2mesh distance the distances are in the range up to 0.002.

So tldr. there are no marching cubes or TSDF fusion involved and the models are the original ones from Shapenet

[yinyunie]

Hi,

I did not come across this issue in your description. Note that, the *.exr file from blender is 'distance map', not 'depth map', where each pixel on the 2D plane stores the distance value from a point on surface to the camera center. You can check dist_to_dep in tools/utils.py to check the conversion.

[zgojcic]

Hi,

it actually does happen with the provided configuration (I know that cycles returns the distance map and that one has to transform it to the depth map). To reproduce what I am describing you can do the following steps:

1. run python create_viewpoints.py
2. run python run_render.py
3. change python visualization/pc_painter.py such that "point_canonical" is saved to a ply or txt file.
4. Load the ".obj" file that was used as input to the run_render.py script and the saved point clouds from step 3 to Meshlab or CloudCompare.
5. Check that the points of the point cloud do not lie exactly on the mesh faces of the "*.obj" file.

[skymanaditya1]

@zgojcic were you able to resolve this issue? I am facing the exact same issue. This is how the pointcloud (rendered from a fixed camera viewpoint) registers with obj file -- notice the slight offset.

[skymanaditya1]

I finally got this to work. I was using custom code to read the exr files and I think that was the reason for the imprecision between the back-projected pointclouds. Here is the code for generating the pointcloud for anyone facing the same issue as me --

import imageio
import sys
sys.path.append('../')
from data_config import *
from tools.read_and_write import *
import random

def read_exr(exr_file_list):
    print(f'The exr file list is : {exr_file_list}')
    '''
    read exr files and output a matrix.
    :param exr_file_list:
    :return:
    '''
    if isinstance(exr_file_list, str):
        exr_file_list = [exr_file_list]

    im_list = []

    for exr_file in exr_file_list:
        if not exr_file.endswith('exr'):
            raise TypeError('file is not with the format of .exr.')

        im_list.append(imageio.imread(exr_file, format='exr')[:,:,0])

    return np.array(im_list)

# get the pointcloud given the rgb image and the depth map 
def get_point_cloud(depth_maps, cam_Ks, cam_RTs, rgb_imgs=None):
    '''
    get point cloud from depth maps
    :param depth_maps: depth map list
    :param cam_Ks: corresponding camera intrinsics
    :param cam_RTs: corresponding camera rotations and translations
    :param rgb_imgs: corresponding rgb images
    :return: aligned point clouds in the canonical system with color intensities.
    '''
    point_list_canonical = []
    camera_positions = []
    color_intensities = []

    if not isinstance(rgb_imgs, np.ndarray):
        rgb_imgs = 32*np.ones([depth_maps.shape[0], depth_maps.shape[1], depth_maps.shape[2], 3], dtype=np.uint8)

    for depth_map, rgb_img, cam_K, cam_RT in zip(depth_maps, rgb_imgs, cam_Ks, cam_RTs):
        u, v = np.meshgrid(range(depth_map.shape[1]), range(depth_map.shape[0]))
        u = u.reshape([1, -1])[0]
        v = v.reshape([1, -1])[0]

        z = depth_map[v, u]

        # remove infinitive pixels
        non_inf_indices = np.argwhere(z < np.inf).T[0]

        color_indices = rgb_img[v, u][non_inf_indices]
        z = z[non_inf_indices]
        u = u[non_inf_indices]
        v = v[non_inf_indices]

        # calculate coordinates
        x = (u - cam_K[0][2]) * z / cam_K[0][0]
        y = (v - cam_K[1][2]) * z / cam_K[1][1]

        point_cam = np.vstack([x, y, z]).T

        point_canonical = (point_cam - cam_RT[:, -1]).dot(cam_RT[:,:-1])
        cam_pos = - cam_RT[:, -1].dot(cam_RT[:,:-1])
        focal_point = ([0, 0, 1] - cam_RT[:, -1]).dot(cam_RT[:,:-1])
        up = np.array([0,-1,0]).dot(cam_RT[:,:-1])

        cam_pos = {'pos':cam_pos, 'fp':focal_point, 'up':up}

        # point_clouds.append(np.expand_dims(point_canonical, axis=0))

        point_list_canonical.append(point_canonical)
        camera_positions.append(cam_pos)
        color_intensities.append(color_indices)

    return point_list_canonical, camera_positions, color_intensities

depth_sample_dir = '02818832/e91c2df09de0d4b1ed4d676215f46734'
n_views = 20
assert n_views <= total_view_nums # there are total 20 views surrounding an object.
view_ids = range(1, n_views+1)

shapenet_rendering_path = '../datasets/shapenet_sample_renderings'
metadata_dir = os.path.join(shapenet_rendering_path, depth_sample_dir)
print(f'metadata_dir : {metadata_dir}')
dist_map_dir = [os.path.join(metadata_dir, 'depth_{0:03d}.exr'.format(view_id)) for view_id in view_ids]
print(f'dist map dir : {dist_map_dir}')
camera_path = '../datasets/camera_settings'
cam_RT_dir = [os.path.join(camera_path, 'cam_RT','cam_RT_{0:03d}.txt'.format(view_id)) for view_id in view_ids]

dist_maps = read_exr(dist_map_dir)
cam_K = np.loadtxt(os.path.join(camera_path, 'cam_K/cam_K.txt'))
cam_RTs = read_txt(cam_RT_dir)

depth_maps = np.float32(dist_to_dep(dist_maps, [cam_K]*len(view_ids)))
rgb_images = [os.path.join(metadata_dir, 'color_{0:03d}.png'.format(view_id)) for view_id in view_ids]
point_list_canonical, camera_positions, color_intensities = get_point_cloud(depth_maps, [cam_K]*len(view_ids), cam_RTs, rgb_imgs=read_image(rgb_images))

# visualize the pointcloud 
pcd_index = random.randint(0, len(point_list_canonical)-1)
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(np.vstack(point_list_canonical))
pcd.colors = o3d.utility.Vector3dVector(np.vstack(color_intensities)/255.0)

# pcd_combined = o3d.geometry.PointCloud()
# pcd_combined.points = o3d.utility.Vector3dVector(np.vstack(point_list_canonical))

o3d.visualization.draw_geometries([pcd])