Tal Hassner: "Viewing Expressive Real-World Faces in 3D"

We present a data-driven method for estimating the 3D shapes of faces viewed in single, unconstrained photos. Our method was designed with an emphasis on robustness and efficiency—with the explicit goal of deployment in real-world applications which reconstruct and display faces in 3D. Our key observation is that for many practical applications, warping the shape of a reference face to match the appearance of a query is enough to produce realistic impressions of the query’s 3D shape. Doing so, however, requires matching visual features between the (possibly very different) query and reference images, while ensuring that a plausible face shape is produced. To this end, we propose using 3D computer graphics models for on-the-fly generation of example references, similar to the query in both pose and expression. This allows us to naturally handle faces in extreme poses and facial expressions. A robust optimization process is described which seeks to maximize the similarity of appearances and depths, jointly, to those of the generated reference, resulting in a face shape tailored to the appearance of the query face. We describe our system for monocular face shape reconstruction and present both qualitative and quantitative experiments, comparing our method against alternative systems, and demonstrating its capabilities.