New advances in facial recognition improve the accuracy and bias of algorithms in regard to gender and skin colour

Facial recognition has been routinely utilized by both private and governmental organizations around the world. Automatic facial recognition can be used for legitimate and beneficial purposes like improving security. However, its power and ubiquity can also give way to the potentially negative impact unfair methods can have for society, such as discrimination against ethnic minorities. A necessary, but not standalone, condition for legitimate use of facial recognition algorithms is that accuracy levels be the same for all demographic groups. 

With this purpose in mind, Sergio Escalera, from the Human Pose Recovery and Behavior Analysis group at the Computer Vision Center (CVC) - University of Barcelona, led his team of researchers in organizing a challenge within the European Conference of Computer Vision (ECCV) 2020. As part of the challenge, participants submitted algorithms that would carry out facial recognition tasks in the presence of confounding attributes. The results, recently published in Computer Vision – ECCV 2020 Workshops, evaluated the algorithms' accuracy and bias regarding gender and skin colour in light of the attributes present.

For Julio C. S. Jacques Jr., researcher with the CVC and the Universitat Oberta de Catalunya's (UOC) Scene Understanding and Artificial Intelligence (SUNAI) Lab and at the Faculty of Computer Science, Multimedia and Telecommunications, the challenge was a success. "It attracted 151 participants, who made more than 1,800 submissions in total, exceeding our expectations regarding the number of participants and submissions," he said. 

The participants used an unbalanced image dataset, which simulates a real-world scenario where AI-based models are to be trained and evaluated based on imbalanced data (considerably more white males than dark females). In total, participants worked with 152,917 images depicting 6,139 identities.

The images were annotated for two protected attributes – gender and skin colour – and five legitimate attributes: age group (0-34, 35-64, 65+), head pose (frontal, other), image source (still image, video frame), wearing glasses and a bounding box size.

Promising results

The results obtained were very promising. The winning solutions achieved 0.999  accuracy along with very low scores in the proposed bias metrics, an outcome which could be considered a step forward in the development of fairer facial recognition methods. An analysis of the top ten teams showed a higher rate of false positives for females with dark skin tones and for samples where both individuals wore glasses. Contrarily, there were higher false negative rates for males with light skin tones and for samples where both individuals were under 35. "This was not a surprise, as the adopted dataset was not balanced with respect to different demographic attributes. However, it shows that overall accuracy is not enough when the goal is to build fair facial recognition methods, and that future work on the topic must take into account both accuracy and bias mitigation," concluded Jacques.

Reference paper

Sixta T., Jacques Junior J.C.S., Buch-Cardona P., Vazquez E., Escalera S. (2020) FairFace Challenge at ECCV 2020: Analyzing Bias in Face Recognition. Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science, vol. 12540. Springer, Cham. DOI: 10.1007/978-3-030-65414-6_32

UOC R&I

The UOC's research and innovation (R&I) are helping 21st-century global societies to overcome pressing challenges by studying the interactions between ICT and human activity, with a specific focus on e-learning and e-health. Over 400 researchers and 50 research groups work among the University's seven faculties and two research centres: the Internet Interdisciplinary Institute (IN3) and the eHealth Center (eHC).

The United Nations' 2030 Agenda for Sustainable Development and open knowledge serve as strategic pillars for the UOC's teaching, research and innovation. More information:research.uoc.edu. #UOC25years