Effective Face Frontalization in Unconstrained Images

Tal Hassner

Shai Harel*

Eran Paz*

Roee Enbar

The Open University of Israel Adience

Frontalization examples
Figure 1. Frontalized faces. Top: Input photos; bottom: our frontalizations, obtained without estimating 3D facial shapes.
Abstract: Frontalization is the process of synthesizing frontal facing views of faces appearing in single unconstrained photos. Recent reports have suggested that this process may substantially boost the performance of face recognition systems. This, by transforming the challenging problem of recognizing faces viewed from unconstrained viewpoints to the easier problem of recognizing faces in constrained, forward facing poses. Previous frontalization methods did this by attempting to approximate 3D facial shapes for each query image. We observe that 3D face shape estimation from unconstrained photos may be a harder problem than frontalization and can potentially introduce facial misalignments. Instead, we explore the simpler approach of using a single, unmodified, 3D surface as an approximation to the shape of all input faces. We show that this leads to a straightforward, efficient and easy to implement method for frontalization. More importantly, it produces aesthetic new frontal views and is surprisingly effective when used for face recognition and gender estimation.


Reference: Tal Hassner, Shai Harel*, Eran Paz* and Roee Enbar, Effective Face Frontalization in Unconstrained Images, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Boston, June 2015
* These author names are in alphabetical order due to equal contribution.

Camera ready (PDF)
Extended abstract (PDF)
CVPR poster (PDF)
arXiv preprint PDF (early arXiv preprint arXiv:1411.7964, 28 Nov. 2014)
BibTex


Downloads

1. Frontalization code: Our MATLAB implementation of the method described in the paper is available on the FTP in the file frontalization.0.1.3.zip. Please see the README.txt for details on how to install and run the code. Though our code is written entirely in MATLAB, it has a few dependencies, including the calib function and a facial feature detector. See  the README.txt for further details.

March, 21st 2016, New! The entire frontalization project was independently ported to Python by Douglas Souza. This useful port is available directly from his github page. This port uses the DLIB facial feature detector. Thank you Douglas!

April, 18st 2016, New! Yuval Nirkin has provided a MATLAB wrapper for the DLIB face and facial landmark detector used with the frontalization code. Thank you, Yuval!



2. LFW3D: The entire LFW collection, frontalized and stored in the same directory structure as the original LFW, is available on the FTP in the file LFW3D.0.1.1.zip. 



3. Adience3D: The entire Adience image collection, frontalized and stored in the same directory structure as the original Adience collection, is available on the FTP in the file Adience3D.0.1.1.zip.

Images in the above sets were frontalized using the SDM facial feature detector (Intraface). Though we have found SDM to provide the best results, alternative detectors can easily be used with our code in its stead and we offer a number of other options. Please see documentation in the README.txt and the MATLAB functions themselves.

Note that we are continuing to improve the quality of the frontalized images, and may post updates to both these collections (hence the version numbers on both files).


What's new

March, 21st 2016
To help run frontalization on MATLAB, Yuval Nirkin has provided a MATLAB MEX for detecting faces and facial landmarks using the DLIB library.


March, 21st 2016
Added link to Python port of the frontalization project, contributed by Douglas Souza.


Dec., 8th 2015
Demo now allows for frontalization with any standard sparse (five-point) landmark detection method such as the one described in our recent paper here. Choosing this option in demo.m currently allows for manual five-point localization.


May, 18th 2015
frontalization.0.1.2 now includes example usage with the DLIB facial feature detector (based on Kazemi and Sullivan, CVPR'14). We have found it to provide excellent performance, and so offer it as an alternative to the commercial SDM detector and the older detector by Zhu and Ramanan.




Download instructions:
These resources are available on our FTP server: agas.openu.ac.il. Direct access to the download directory is available through http://www.cslab.openu.ac.il/download/.

For the FTP user name and password, please provide the details below. We ask for these details so we can keep in touch in case we find any need for a critical update or in case we would organize a dedicated workshop, etc. When done, you will immediately be able to see the FTP user name. 


    * Required field
If you have problems accessing the files through the direct link above please use instead a dedicated FTP client, and connect to the server agas.openu.ac.il. Two free clients are FileZilla and WinSCP.




Other related papers / projects / code:

  • Yue Wu*, Tal Hassner*, KangGeon Kim, Gerard Medioni and Prem Natarajan, Facial Landmark Detection with Tweaked Convolutional Neural Networks, arXiv preprint arXiv:1511.04031, 21 Mar. 2016 (PDF, project and code)
    * Denotes joint first authorship / equal contribution
  • E. Eidinger, R. Enbar, and T. Hassner, Age and Gender Estimation of Unfiltered Faces, Transactions on Information Forensics and Security (IEEE-TIFS), special issue on Facial Biometrics in the Wild, Volume 9, Issue 12, pages 2170 - 2179, Dec. 2014 (PDF, project, data, code)
  • T. Hassner, L. Assif, and L. Wolf, When Standard RANSAC is Not Enough: Cross-Media Visual Matching with Hypothesis Relevancy, Machine Vision and Applications (MVAP), Volume 25, Issue 4, Page 971-983, 2014 (Code, PDF, Springer)
  • T. Hassner, Viewing Real-World Faces in 3D, International Conference on Computer Vision (ICCV), Sydney, Austraila, Dec. 2013 (Code, project, PDF)

Copyright 2014, Tal Hassner

TThe SOFTWARE ("frontalization" and all included files) is provided "as is", without any guarantee made as to its suitability or fitness for any particular use. It may contain bugs, so use of this tool is at your own risk. We take no responsibility for any damage that may unintentionally be caused through its use.

Last update 18th of April, 2016