Abstract

This work proposes a novel representation of injective deformations of 3D space, which overcomes existing limitations of injective methods, namely inaccuracy, lack of robustness, and incompatibility with general learning and optimization frameworks. Our core idea is to reduce the problem to a “deep” composition of multiple 2D mesh-based piecewise-linear maps. Namely, we build differentiable layers that produce mesh deformations through Tutte’s embedding (guaranteed to be injective in 2D), and compose these layers over different planes to create complex 3D injective deformations of the 3D volume. We show our method provides the ability to efficiently and accurately optimize and learn complex deformations, outperforming other injective approaches. As a main application, we produce complex and artifact-free NeRF and SDF deformations.

Representation of injective 3D deformations

Detailed NeRF Deformation Process

We show detailed deformation on one NeRF shape with the above mentioned representation. Here we use 6 layers and alternate with x-y-z deformation planes.

Elastic NeRF Deformation

We show the orignal and deformed NeRFs with Instant-NGP method. The selected constraints illustrated with green arrows on the left column.

We show the deformation results of phone-captured NeRFs with Nerfacto method.

Human Reposing

We learn a deformation model on SMPL human models and apply the trained network to deform neural fields. The neural network from the learning experiment (Section 4.2 in the paper), trained to predict deformations on a dataset of human SMPL meshes (top row, demonstrating the desired target pose), is seamlessly applied to deform synthetic and real NeRFs (middle two rows), and SDFs (bottom row).

Citation

Acknowledgements

The website template was borrowed from Michaël Gharbi.