The Computer Expression Recognition Toolbox

Littlewort, Wu, Whitehill, Fasel, Movellan, Bartlett

Computer Expression Recognition Toolbox (CERT)is an end-to-end system for fully automated facial expression recognition that operates in real-time. CERT was developed at the University of California, San Diego, originating from a collaboration between Ekman and Sejnowski (Bartlett et al., 1996, 1999, 2006; Donato et al., 1999; Littlewort et al., 2006). The current system automatically detects frontal faces in the video stream and codes each frame with respect to 40 continuous dimensions, including basic expressions of anger, disgust, fear, joy, sadness, surprise, contempt, a continuous measure of head pose (yaw, pitch, and roll), as well as 30 facial action units (AU’s) from the Facial Action Coding System. The technical approach to CERT is an appearance-based, discriminative approach. Current research focuses on machine learning and computer vision approaches to improve facial expression detection performance, analysis of dynamics, and temporal segmentation. This system is being employed in pioneering experiments on spontaneous behavior, including automated discrimination of posed from genuine expressions of pain, automated detection of driver drowsiness, adaptive tutoring systems, and an intervention for children with autism.

NSF CISE/CRI: Automated Facial Expression Measurement: Toolbox and Database 7/15/05-6/30/10. PI: Bartlett

NSF IIS ITR: Automatic Analysis of Spontaneous Facial Expressions 9/1/02-8/31/04 PI: Movellan

Obtain a copy of the Computer Expression Recognition Toolbox (CERT)

Sikka K, Dykstra, Sathyanarayana, Littlewort, Bartlett (2013). Multiple Kernel Learning for Emotion Recognition in the Wild. Proc. 15th ACM International conference on multimodal interaction. Pages 517-524. Best paper award. Download pdf

Sikka K, Wu T, Susskind J, and Bartlett M (2012). Exploring Bag of Words Architectures In the facial expression domain. In A Fitzgibbon, S Lazebnik, P Perona, Y Sato, and C Schmid (Eds), Lecture Notes in Computer Science, Computer Vision – ECCV 2012, Vol. 7584, Springer, p. 250-259. Download pdf

Wu T, Butko NJ, Ruvolo P, Whitehill J, Bartlett MS, Movellan JR (2012). Multi-Layer Architectures for Facial Action Unit Recognition. IEEE Transactions on Systems, Man and Cybernetics -- Part B (TSMC-B). Download pdf

Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, and Bartlett M (2011) The Computer Expression Recognition Toolbox (CERT). Proc. IEEE International Conference on Automatic Face and Gesture Recognition, p. 298-305. Download pdf

Wu, T., Bartlett, M.S., and Movellan, J. (2010). Facial expression recognition using gabor motion energy filters. IEEE CVPR workshop on Computer Vision and Pattern Recognition for Human Communicative Behavior Analysis, p. 42-47. Download pdf

Bartlett, M., Littlewort, G., Whitehill, J., Vural, E., Wu, T., Lee, K., Ercil, A., Cetin, M. Movellan, J. (2010). Insights on spontaneous facial expressions from automatic expression measurement.” In Giese,M. Curio, C., Bulthoff, H. (Eds.) Dynamic Faces: Insights from Experiments and Computation. MIT Press.Download pdf

Bartlett, M., Whitehill, J. (2010). Automated facial expression measurement: Recent applications to basic research in human behavior, learning, and education.” In Handbook of Face Perception, Andrew Calder, Gillian Rhodes, James V. Haxby, and Mark H. Johnson (Eds). Oxford University Press. Download pdf

Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., & Movellan, J. (2009). Towards practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 99(2) p. 2106-2111. Download pdf

Bartlett, M.S., Littlewort-Ford, G.C., and Movellan, J.R. (2008). Computer Expression Recognition Toolbox. Demo: 8th Int’l IEEE Conference on Automatic Face and Gesture Recognition. Amsterdam. Download pdf

Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of Multimedia 1(6) p. 22-35. 
Download pdf

Littlewort, G., Bartlett, M., Fasel, I., Susskind, J., and Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. Image and Vision Computing 24(6), p. 615-625. 
Download pdf

Bartlett, M.S., Littlewort, G., Frank, M.G., Lainscsek, C., Fasel, I., and Movellan, J. (2005). Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior. IEEE International Conference on Computer Vision and Pattern Recognition. p. 568-573. 
Download pdf

Bartlett, M.S., Littlewort, G., Braathen, B., Sejnowski, T.J., & Movellan, J.R. (2003). A prototype for automatic recognition of spontaneous facial actions. In S. Becker & S. Thrun & K. Obermayer, (Eds.) Advances in Neural Information Processing Systems, Vol 15, p. 1271-1278, MIT Press. 
Download pdf

Bartlett, M.S., Littlewort, G., Fasel, I., Movellan, J.R. (2003). Real time face detection and expression recognition: Development and application to human-computer interaction. CVPR Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction. 
Download pdf

Bartlett, M.S., Donato, G.L., Movellan, J.R., Hager, J.C., Ekman, P., and Sejnowski, T.J. (2000). Image representations for facial expression coding. In S. Solla, T. Leen, & K. Mueller, Eds. Advances in Neural Information Processing Systems 12, Cambridge, MA: MIT Press, p. 886-892.

Donato, G.L., Bartlett, M.S., Hager, J.C., Ekman, P., and Sejnowski, T.J. (1999). Classifying Facial Actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10) p. 974-989.
Abstract, Download pdf

Bartlett, M.S., Hager, J.C., Ekman, P., and Sejnowski, T.J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology 36, p. 253-263.
Abstract, Download ps

Bartlett, M. Stewart, Viola, P.A., Sejnowski, T.J., Golomb, B.A., Larsen, J., Hager, J.C., and Ekman, P. (1996). Classifying facial action, Advances in Neural Information Processing Systems 8, MIT Press, Cambridge, MA. p. 823-829.