Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video


Agudo, A; Martinez-Montiel, JM; Agapito, L; Calvo, B



Año: 2017 Volumen: 57 (1) Páginas: 75-98






This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The nonrigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the time-varying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-to-fine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

Palabras clave

Sequential ; nonrigid ; structure ; from ; motion ; Dense ; reconstruction ; Modal ; analysis ; Finite ; elements


Agudo, A (Reprint Author), Inst Robot & Informat Ind CSIC UPC, Barcelona 08028, Spain.
Agudo, Antonio, Inst Robot & Informat Ind CSIC UPC, Barcelona 08028, Spain.
Martinez Montiel, Jose M.; Calvo, Begona, I3A Univ Zaragoza, Mariano Esquillor S-N, Zaragoza 50018, Spain.
Agapito, Lourdes, UCL, Gower St, London WC1E 6BT, England.