When RGB is higher res, interpolate by rasterizing depth triangles onto the registered image
Combine all operations into one matrix multiply on (u,v,d)