Computational Analysis of Nonverbal Behavior in Adaptive Tutoring

Whitehill, Littlewort, Reilly, Phan, Movellan, Bartlett

There has been growing recognition of the importance of adaptive tutoring systems that respond to the student’s emotional and cognitive state. However little is known about children’s facial expressions during a problem solving task. What are the actual signals of boredom, interest, confusion, or uncertainty in real, spontaneous behavior of students? The field also is in need of spontaneous datasets to drive automated recognition of these states. To date, there is a paucity of empirical data to support understanding nonverbal behavior in teaching at a computational level. The goal of this project is to develop computational methods to measure and model the nonverbal behavior and interactive strategies observed during face-to-face teaching. To support these models, we are collecting datasets of student-teacher interactions as well as nonverbal behavior during problem solving. The computational models will serve as a foundation for a new generation of embodied teaching agents that approximate the benefits of face-to-face human tutoring. This research builds the foundation for automated tutoring systems that sense the state of the student and adapt accordingly. The project will help advance the science of learning and teaching by improving our understanding of the dynamics of nonverbal behavior in teaching at a computational level, across multiple time scales: From low-level micro-expressions in the timescale of tens of milliseconds, to cognitive and affective processes with time scales of seconds, to higher level strategic behaviors operating at longer time scales.

NSF / IIS / HCC Grant: Computational Analysis of Nonverbal Behavior in Adaptive Tutoring. PI: Bartlett. Co-I: Reilly, Movellan. 8/1/2009-7/31/2013.

NSF Science of Learning: Temporal Dynamics of Learning Center, NSF SBE 0542013

Workshops:

NIPS Workshop: Personalizing Education with Machine Learning. Organizers: Mike Mozer, Javier Movellan, Rob Lindsey, Jake Whitehill. Neural Information Processing Systsems, Lake Tahoe CA, Dec 8, 2012.

TDLC Workshop: Optimal Teaching. Organizers: Javier Movellan and Jacob Whitehill. University of California, San Diego, May 4, 2012.

Papers and Presenatations

Sathyanarayana S, Satzoda, Carini, Lee, Salamanca, Reilly, Forster, Bartlett, Littlewort (2014) Towards Automated Understanding of Student-Tutor Interactions using Visual Deictic Gestures. Proc. Computer Vision and Pattern Recognition Workshops (CVPRW), p. 480-487.

Sathyanarayana S, Littlewort G, Bartlett M (2013). Hand Gestures for Intelligent Tutoring Systems: Dataset, Techniques & Evaluation. Proc. IEEE International conference on Computer Vision, Workshop: Decoding Subtle Cues from Social Interactions. pg 769 – 776. Paper on IEEE Xplore

Dykstra K, Whitehill J, Salamanca L, Lee M, Carini A, Reilly J, and Bartlett, M (2012). Modeling One-on-one Tutoring Sessions. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf

Salamanca L, Carini A, Lee M, Dykstra K, Whitehill J, Angus D, Wiles J, Reilly J, and Bartlett, M (2012). Characterizing the Temporal Dynamics of Student-Teacher Discourse. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf

Brian S, Salamanca L, Whitehill J, Reilly J, Bartlett M, Angus D, and Wiles J (2012). Using Recurrence Plots to Visualize the Temporal Dynamics of Tutor/Student Interactions. Proc. International Conference on Development and Learning / Epigenetic Robotics, San Diego, CA. Download pdf

Salamanca L, Lee M, Carini A, Dykstra K, Whitehill J, Bartlett M, Reilly, J. (2012) Profiling Student-Teacher Eye Gaze Behaviors. Poster presentation, Southern California Cognitive Neuroscience Meeting at San Diego State, March 2, 2012.

Whitehill, J., Serpell, Z., Foster, A., Lin, Y.C., Pearson, B., Barlett, M., et al. (2011). Toward an Optimal Affect-Sensitive Instructional System of Cognitive Skills. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop on Human Communicative Behavior, (pp. 20 - 25). Download pdf

Littlewort GC, Salamanca LP, Reilly JS, and Bartlett MS (2011). Automated measurement of children’s facial expressions during problem solving tasks. Proc. IEEE International Conference on Automatic Face and Gesture Recognition, p. 30-35. Download pdf

Butko, N.J., Theocharous, G., Philipose, M., Movellan, J.R., (2011). Automated facial affect analysis for one-on-one tutoring applications.” Proceedings of the 9th IEEE Conference on Automatic Face and Gesture Recognition, p.382-387.
Download

Serpell Z, Whitehill Z, Movellan J, Foster A, Lin Y, Pearson B, and Bartlett M (2011). Sensitivity to Non-verbal Behavior Influences Cognitive Training of Minority Students. Presentation at American Psychological Association conference, May 2011, Washington, DC.

Phan, L., Meza, R., Filizardo, J., Littlewort-Ford, G., Bartlett, M., Movellan, J., Reilly, J. (2009). Behavioral indices of cognitive processing in children.” Oral presentation, Multimode Conference, Toulouse, France. Abstract pdf

Whitehill, J., Bartlett, M., and Movellan, J. (2008). Automatic facial expression recognition for intelligent tutoring systems. Workshop on CVPR for Human Communicative Behavior Analysis, IEEE Conference on Computer Vision and Pattern Recognition, pg. 1-6. Download pdf

Whitehill, J., Bartlett, M., and Movellan, J. (2008). Measuring the perceived difficulty of a lecture using automatic facial expression recognition.  Intelligent tutoring systems. Montreal, Canada June 23-27, 2008.