Convert colored mesh to volume

[JeffreyWardman]

Hi, given a mesh with colors appointed to each vertex, how can I convert it into a volume (voxel format) with the colors of the voxels relating to these points. Is there a simple way to achieve this? My purpose requires taking either the nearest point as the color, or the most frequent color within the space of the voxel.
Thanks.

Note that I currently use mesh.binarize(spacing, fg_val=1, bg_val=0) but this (obviously) doesn't convert the colors into voxel metadata.

[marcomusy]

Hi - I'm not sure what visualization are you trying to obtain, I guess you want to hide voxels far from the surface?
is the color give by a scalar (colormapping)?

E.g.

om vedo import *

mesh = Mesh(dataurl+"beethoven.ply").subdivide()

step = 255/len(mesh.points())
red   = np.arange(0, 255, step).astype(np.uint8)
green = np.arange(0, 255, step).astype(np.uint8)
blue  = np.arange(0, 255, step).astype(np.uint8)
mesh.pointdata["red"] = red
mesh.pointdata["green"] = green
mesh.pointdata["blue"] = blue
mesh.pointdata["color"] = np.c_[red, green, blue]

vol = mesh.tovolume(radius=1, dims=(30,30,20), null_value=0)
# vol = mesh.tovolume(n=1, dims=(30,30,20))

# this returns the numpy array of colors
print(vol.pointdata['color'])

lego = vol.legosurface(vmin=1)

show(mesh, lego, N=2, axes=1).close()

[JeffreyWardman]

Thanks @marcomusy, this is fantastic! I'm trying to replicate OpenCV's INTER_NEAREST, so I use n=1 instead of radius=1. However, the computational cost is significantly more (dims=(256,256,256)).
PERFORMANCE: Can this be improved upon somehow? E.g. by selecting a subset of points based on the radius and calculating the n nearest neighbors from the subset?

FEATURE: In your example for instance, I want the face to be the color green, and the hair to be blue. At the border of the face and hair, I want the voxel value to either be green/brown, but not anything else. I believe this is currently not possible in the API due to it forcing a kernel rather than allowing for no kernel/the identity matrix.

[JeffreyWardman]

It also looks like when using n=1 the volume is heavily distorted (zoomed in).

[marcomusy]

Hi Jeffrey, you should explain the context of what are you trying to achieve because it's not very clear to me... is it just a matter of visualization? why would you do something like that? Maybe there is better way doing it... not sure if the Volume is the right data structure...

About performance it's probably not something that can be improved. Under the hood there is already a KDtree search which speeds up the computation...

[JeffreyWardman]

Hi Marco, sure thing. I want to train a deep learning model with the labels for each vertex. However, I want to do this by converting the mesh (and labels) into voxel space. At this stage if I want to try to predict if a voxel is related to either: a dog (1), cat (2) or background (0), then I need the labels to accurately map to voxel space as well.

In terms of the K-NN alternative algorithm, does VTK offer a way to parallelize the compute or (even better) offload to GPU?

[marcomusy]

Hi, sorry for the late reply... still it's not very clear to me how you would deal with - or the meaning of - the voxels which are internal (or external) to the surface mesh.
When you say "related" you mean "inside" of a surface?

I think VTK has some mechanism for parallelization (not gpu afaik) - but this is not implemented in vedo.

[JeffreyWardman]

No issues at all! So let's consider an image of a cat next to a dog. The labels may be a 2D binary mask, where each pixel is either foreground (1; either cat or dog) or background (0). This contains two classes (foreground and background) and the problem can be extended to n classes (e.g. background (0), dog (1), cat (2)). So every pixel in this labels image aligns with the original image. If you do image[np.where(label_mask == 1)] then all pixels will belong to a cat.

Now, in our previous discussion, this is in 3D space. So I want a label mask (integers) where each voxel is a label for the raw volume (in lego format according to vedo docs). To retain this information from a mesh, we need to convert the mesh to lego format where the voxel value is determined by the labels assigned to the mesh vertices. Note that it cannot be determined by a sum or rounded mean, but by KNN. The sum would not work because 3 might represent bird and is unrelated to 1 (dog) and 2 (cat). The mean would not work because it is not descriptive of the problem, e.g. 1.5 means either dog or cat, but the mean of 1 (dog) and 3 (bird) results in 2 (cat), which is not representative of the data.

raw mesh
--> change colors of mesh to labels (at pointcloud level)
--> convert mesh to lego
     --> colours determined by most frequent label in KNN of points
--> plot mesh with voxels having integer labels representing various classes