Facial Expression Recognition

Computers in these days try to interpret certain human characteristics so as to react better. These characteristics include facial expressions, eyes gaze, body gait, speech etc. . Many applications such as virtual reality, videoconferencing, user profiling, customer satisfaction studies for broadcast and web services and interfaces for people with special needs require efficient facial expression recognition in order to achieve the desired results.

The basic facial expressions are defined as six anger, disgust, fear, happiness, sadness and surprise. A set of muscle movements (Facial Action Units-FAUs) was created to produce those facial expressions, forming the Facial Action Coding System ( FACS ).

An example of each facial expression for a poser from the Cohn-Kanade database

Facial expressions are generally hard to recognize as:

  • Every person expresses in a different way, no international patterns are available.
  • The conditions must be ideal, meaning that a full frontal pose of the poser has to be available.
  • The neutral state has to be found in videos in order to be able to define the fully expressive video frame and thus perform facial expression recognition.
  • No proper databases available and difficult to create a new one, as supervision from psychologists is required.


Our Method

A novel method that performs facial expression recognition has been developed:

  • Introduces a new class of Support Vector Machines (SVMs) .
  • Introduces a subset of Candide grid used for facial expression recognition.
  • Introduces a new set of simplified rules for facial expression synthesis.
  • Takes into account the geometrical information of the Candide grid nodes between the first and the last video frame representing the neutral state and the expressive facial expression, respectively.
  • Uses SVMs as a classifier for the geometrical information extracted.
  • Classifies deformed grids into facial expressions and detects the FAUs that are activated in the grid under examination. It also performs later facial expression recognition using the detected FAUs.

Examples of grids depicting the 6 basic facial expressions

System architecture for facial expression recognition in facial videos

The accuracy achieved is equal to 99.7% when using multi-class SVMs for facial expression recognition and to 95.1% when using two-class SVMs for FAUs detection and afterwards facial expression recognition.



Example of grid tracking for the 6 basic facial expressions


Relevant Publications

I. Kotsia and I. Pitas, "Real time facial expression recognition from video sequences using Support Vector Machines", in Proc. of Visual Communications and Image Processing (VCIP 2005), Beijing, China, 12-15 July, 2005.

I. Kotsia and I. Pitas, "Real time facial expression recognition from image sequences using Support Vector Machines", in Proc. of IEEE Int. Conf. on Image Processing (ICIP 2005), Genova, Italy, 11-14 September, 2005.

I. Kotsia and I. Pitas, "Facial Expression Recognition in Image Sequences using Geometric Deformation Features and Support Vector Machines", IEEE Transactions on Image Processing, January, 2007.


Research Projects

SIMILAR - The European research taskforce creating human-machine interfaces SIMILAR to human-human communication, IST, FP6

PENED 01 - Virtual Reality tools for education on natural disasters


© 2006