The development of real-time affect detection models often depends upon obtaining annotated data ... more The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in annotating affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally similar to the students being labeled, as this impacts the cost feasibility of obtaining the labels. In this study, we investigate the following research questions: For affective state annotation, how does the socio-cultural background of human expert labelers, compared to the subjects, impact the degree of consensus and distribution of affective states obtained? Secondly, how do differences in labeler background impact the performance of affect detection models that are trained using these labels?
Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In... more Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In this paper, we focus on the problem of emotional engagement, and propose to detect important affective states (i.e., ‘Satisfied’, ‘Bored’, and ‘Confused’) of a learner in real time. We collected 210 hours of data from 20 students through authentic classroom pilots. The data included information from two modalities: (1) appearance which is collected from the camera, and (2) context-performance that is derived from the content platform. In this paper, data from nine students who attended the learning sessions twice a week are analyzed. We trained separate classifiers for different modalities (appearance and context-performance), and for different types of learning sections (instructional and assessment). The results show that different sources of information are generically better representatives of engagement at different sections: For instructional sections, generic appearance classifie...
To investigate the detection of students' behavioral engagement (On-Task vs. Off-Task), we pr... more To investigate the detection of students' behavioral engagement (On-Task vs. Off-Task), we propose a two-phase approach in this study. In Phase 1, contextual logs (URLs) are utilized to assess active usage of the content platform. If there is active use, the appearance information is utilized in Phase 2 to infer behavioral engagement. Incorporating the contextual information improved the overall F1-scores from 0.77 to 0.82. Our cross-classroom and cross-platform experiments showed the proposed generic and multi-modal behavioral engagement models' applicability to a different set of students or different subject areas.
2011 IEEE 19th Signal Processing and Communications Applications Conference (SIU), 2011
ABSTRACT Occlusions complicate the process of identifying individuals us- ing their 3D facial sca... more ABSTRACT Occlusions complicate the process of identifying individuals us- ing their 3D facial scans. We propose a D face recognition system that automatically removes occlusion artifacts and iden- tifies the facial image using regional classifiers. Automatic lo- calization of occluded areas is handled by using a generic face model. Restoration of missing information after occlusion re- moval is performed by the application of an improved version of Gappy Principal Component Analysis (GPCA), which we call partial Gappy PCA (pGPCA). After the removal of noisy data introduced by realistic occlusions, occlusion-free faces are rep- resented by local regions. Local classifiers operating on these local regions are then fused to achieve occlusion-robust identi- fication performance. Our experimental results obtained on re- alistically occluded facial images from the Bosphorus 3D face database illustrate that our occlusion compensation scheme drastically improves the recognition accuracy from 78.05% to 94.20%.
Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016
Affective states play a crucial role in learning. Existing Intelligent Tutoring Systems (ITSs) fa... more Affective states play a crucial role in learning. Existing Intelligent Tutoring Systems (ITSs) fail to track affective states of learners accurately. Without an accurate detection of such states, ITSs are limited in providing truly personalized learning experience. In our longitudinal research, we have been working towards developing an empathic autonomous 'tutor' closely monitoring students in real-time using multiple sources of data to understand their affective states corresponding to emotional engagement. We focus on detecting learning related states (i.e., 'Satisfied', 'Bored', and 'Confused'). We have collected 210 hours of data through authentic classroom pilots of 17 sessions. We collected information from two modalities: (1) appearance, which is collected from the camera, and (2) context-performance, that is derived from the content platform. The learning content of the content platform consists of two section types: (1) instructional where students watch instructional videos and (2) assessment where students solve exercise questions. Since there are individual differences in expressing affective states, the detection of emotional engagement needs to be customized for each individual. In this paper, we propose a hierarchical semi-supervised model adaptation method to achieve highly accurate emotional engagement detectors. In the initial calibration phase, a personalized context-performance classifier is obtained. In the online usage phase, the appearance classifier is automatically personalized using the labels generated by the context-performance model. The experimental results show that personalization enables performance improvement of our generic emotional engagement detectors. The proposed semi-supervised hierarchical personalization method result in 89.23% and 75.20% F1 measures for the instructional and assessment sections respectively.
We address the question of 3D face recognition and expression understanding under adverse conditi... more We address the question of 3D face recognition and expression understanding under adverse conditions like illumination, pose, and accessories. We therefore conduct a campaign to build a 3D face database including systematic variation of poses, different types of occlusions, and a rich set of expressions. The expressions consist of a judiciously selected subset of Action Units as well as the six basic emotions. The database is designed to enable various research paths from face recognition to facial landmarking and to expression estimation. Preliminary results are presented on the outcome of three different landmarking methods as well as one registration method. As expected, observed non-neutral and non-frontal faces demand new robust algorithms to achieve an acceptable performance.
The development of real-time affect detection models often depends upon obtaining annotated data ... more The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in labeling affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally similar to the students being labeled, as this impacts the cost and feasibility of obtaining the labels. In this study, we investigate the following research questions: For affective state labeling, how does the socio-cultural background of human expert labelers, compared to the subjects (i.e., students), impact the degree of consensus and distribution of affective states obtained? Secondly, how do differences in labeler background impact the performance of affect detection models that are trained using these labels? To address these questions, we employed experts from Turkey and the United States to label the same data collected through authentic classroom pilots with stud...
There is still considerable disagreement on key aspects of affective computing - including even h... more There is still considerable disagreement on key aspects of affective computing - including even how affect itself is conceptualized. Using a multi-modal student dataset collected while students were watching instructional videos and answering questions on a learning platform, we investigated the two key paradigms of how affect is represented through a comparative approach: (1) Affect as a set of discrete states and (2) Affect as a combination of a two-dimensional space of attributes. We specifically examined a set of discrete learning-related affects (Satisfied, Confused, and Bored) that are hypothesized to map to specific locations within the Valence-Arousal dimensions of Circumplex Model of Emotion. For each of the key paradigms, we had five human experts label student affect on the dataset. We investigated two major research questions using their labels: (1) Whether the hypothesized mappings between discrete affects and Valence-Arousal are valid and (2) whether affect labeling is...
We address the question of 3D face recognition and expression understanding under adverse conditi... more We address the question of 3D face recognition and expression understanding under adverse conditions like illumination, pose, and accessories. We therefore conduct a campaign to build a 3D face database including systematic variation of poses, different types of occlusions, and a rich set of expressions. The expressions consist of a judiciously selected subset of Action Units as well as the six basic emotions. The database is designed to enable various research paths from face recognition to facial landmarking and to expression estimation. Preliminary results are presented on the outcome of three different landmarking methods as well as one registration method. As expected, observed non-neutral and non-frontal faces demand new robust algorithms to achieve an acceptable performance.
Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems, 2016
With the advances in computing technologies, we have been undergoing a shift towards a digital wo... more With the advances in computing technologies, we have been undergoing a shift towards a digital world. As an inevitable result of this shift, the technology penetrates into education in myriad forms. Intelligent tutoring systems (ITS) are essential outcomes of this penetration, emerging to satisfy the needs of learners and instructors. Their working principle is based on collecting and processing data of all students through various modalities to understand the strengths and needs of learners. Yet, more important is that ITSs untangle the overlooked problem of traditional education: One size does not fit all, and there is a need for personalized tutoring for each individual. It is well known that that learning is emotional as well as intellectual. To truly meet the needs of education, we need empathic companions, ones that are affectively aware and thus can accompany the learner for an enhanced learning experience.
Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In... more Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In this paper, we focus on the problem of emotional engagement, and propose to detect important affective states (i.e., 'Satisfied', 'Bored', and 'Confused') of a learner in real time. We collected 210 hours of data from 20 students through authentic classroom pilots. The data included information from two modalities: (1) appearance which is collected from the camera, and (2) context-performance that is derived from the content platform. In this paper, data from nine students who attended the learning sessions twice a week are analyzed. We trained separate classifiers for different modalities (appearance and context-performance), and for different types of learning sections (instructional and assessment). The results show that different sources of information are generically better representatives of engagement at different sections: For instructional sections, gene...
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can... more We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can provide just-in-time personalized support to students who risk disengagement. To investigate the impact of the technology, we ran an exploratory semester-long study with a teacher in two classrooms. We used a multi-method approach consisting of a quasi-experimental design to evaluate the impact of the technology and a case study design to understand the environmental and social factors surrounding the classroom setting. The results show that the technology had a significant impact on the teacher's classroom practices (i.e., increased scaffolding to the students) and student engagement (i.e., less boredom). These results suggest that the technology has the potential to support teachers' role of being a coach in technology-mediated learning environments. CCS CONCEPTS • Applied computing-> Education-> Learning management systems • Human-centered computing-> Human computer interaction (HCI)-> Empirical studies in HCI
The development of real-time affect detection models often depends upon obtaining annotated data ... more The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in annotating affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally similar to the students being labeled, as this impacts the cost feasibility of obtaining the labels. In this study, we investigate the following research questions: For affective state annotation, how does the socio-cultural background of human expert labelers, compared to the subjects, impact the degree of consensus and distribution of affective states obtained? Secondly, how do differences in labeler background impact the performance of affect detection models that are trained using these labels?
Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In... more Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In this paper, we focus on the problem of emotional engagement, and propose to detect important affective states (i.e., ‘Satisfied’, ‘Bored’, and ‘Confused’) of a learner in real time. We collected 210 hours of data from 20 students through authentic classroom pilots. The data included information from two modalities: (1) appearance which is collected from the camera, and (2) context-performance that is derived from the content platform. In this paper, data from nine students who attended the learning sessions twice a week are analyzed. We trained separate classifiers for different modalities (appearance and context-performance), and for different types of learning sections (instructional and assessment). The results show that different sources of information are generically better representatives of engagement at different sections: For instructional sections, generic appearance classifie...
To investigate the detection of students' behavioral engagement (On-Task vs. Off-Task), we pr... more To investigate the detection of students' behavioral engagement (On-Task vs. Off-Task), we propose a two-phase approach in this study. In Phase 1, contextual logs (URLs) are utilized to assess active usage of the content platform. If there is active use, the appearance information is utilized in Phase 2 to infer behavioral engagement. Incorporating the contextual information improved the overall F1-scores from 0.77 to 0.82. Our cross-classroom and cross-platform experiments showed the proposed generic and multi-modal behavioral engagement models' applicability to a different set of students or different subject areas.
2011 IEEE 19th Signal Processing and Communications Applications Conference (SIU), 2011
ABSTRACT Occlusions complicate the process of identifying individuals us- ing their 3D facial sca... more ABSTRACT Occlusions complicate the process of identifying individuals us- ing their 3D facial scans. We propose a D face recognition system that automatically removes occlusion artifacts and iden- tifies the facial image using regional classifiers. Automatic lo- calization of occluded areas is handled by using a generic face model. Restoration of missing information after occlusion re- moval is performed by the application of an improved version of Gappy Principal Component Analysis (GPCA), which we call partial Gappy PCA (pGPCA). After the removal of noisy data introduced by realistic occlusions, occlusion-free faces are rep- resented by local regions. Local classifiers operating on these local regions are then fused to achieve occlusion-robust identi- fication performance. Our experimental results obtained on re- alistically occluded facial images from the Bosphorus 3D face database illustrate that our occlusion compensation scheme drastically improves the recognition accuracy from 78.05% to 94.20%.
Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016
Affective states play a crucial role in learning. Existing Intelligent Tutoring Systems (ITSs) fa... more Affective states play a crucial role in learning. Existing Intelligent Tutoring Systems (ITSs) fail to track affective states of learners accurately. Without an accurate detection of such states, ITSs are limited in providing truly personalized learning experience. In our longitudinal research, we have been working towards developing an empathic autonomous 'tutor' closely monitoring students in real-time using multiple sources of data to understand their affective states corresponding to emotional engagement. We focus on detecting learning related states (i.e., 'Satisfied', 'Bored', and 'Confused'). We have collected 210 hours of data through authentic classroom pilots of 17 sessions. We collected information from two modalities: (1) appearance, which is collected from the camera, and (2) context-performance, that is derived from the content platform. The learning content of the content platform consists of two section types: (1) instructional where students watch instructional videos and (2) assessment where students solve exercise questions. Since there are individual differences in expressing affective states, the detection of emotional engagement needs to be customized for each individual. In this paper, we propose a hierarchical semi-supervised model adaptation method to achieve highly accurate emotional engagement detectors. In the initial calibration phase, a personalized context-performance classifier is obtained. In the online usage phase, the appearance classifier is automatically personalized using the labels generated by the context-performance model. The experimental results show that personalization enables performance improvement of our generic emotional engagement detectors. The proposed semi-supervised hierarchical personalization method result in 89.23% and 75.20% F1 measures for the instructional and assessment sections respectively.
We address the question of 3D face recognition and expression understanding under adverse conditi... more We address the question of 3D face recognition and expression understanding under adverse conditions like illumination, pose, and accessories. We therefore conduct a campaign to build a 3D face database including systematic variation of poses, different types of occlusions, and a rich set of expressions. The expressions consist of a judiciously selected subset of Action Units as well as the six basic emotions. The database is designed to enable various research paths from face recognition to facial landmarking and to expression estimation. Preliminary results are presented on the outcome of three different landmarking methods as well as one registration method. As expected, observed non-neutral and non-frontal faces demand new robust algorithms to achieve an acceptable performance.
The development of real-time affect detection models often depends upon obtaining annotated data ... more The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in labeling affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally similar to the students being labeled, as this impacts the cost and feasibility of obtaining the labels. In this study, we investigate the following research questions: For affective state labeling, how does the socio-cultural background of human expert labelers, compared to the subjects (i.e., students), impact the degree of consensus and distribution of affective states obtained? Secondly, how do differences in labeler background impact the performance of affect detection models that are trained using these labels? To address these questions, we employed experts from Turkey and the United States to label the same data collected through authentic classroom pilots with stud...
There is still considerable disagreement on key aspects of affective computing - including even h... more There is still considerable disagreement on key aspects of affective computing - including even how affect itself is conceptualized. Using a multi-modal student dataset collected while students were watching instructional videos and answering questions on a learning platform, we investigated the two key paradigms of how affect is represented through a comparative approach: (1) Affect as a set of discrete states and (2) Affect as a combination of a two-dimensional space of attributes. We specifically examined a set of discrete learning-related affects (Satisfied, Confused, and Bored) that are hypothesized to map to specific locations within the Valence-Arousal dimensions of Circumplex Model of Emotion. For each of the key paradigms, we had five human experts label student affect on the dataset. We investigated two major research questions using their labels: (1) Whether the hypothesized mappings between discrete affects and Valence-Arousal are valid and (2) whether affect labeling is...
We address the question of 3D face recognition and expression understanding under adverse conditi... more We address the question of 3D face recognition and expression understanding under adverse conditions like illumination, pose, and accessories. We therefore conduct a campaign to build a 3D face database including systematic variation of poses, different types of occlusions, and a rich set of expressions. The expressions consist of a judiciously selected subset of Action Units as well as the six basic emotions. The database is designed to enable various research paths from face recognition to facial landmarking and to expression estimation. Preliminary results are presented on the outcome of three different landmarking methods as well as one registration method. As expected, observed non-neutral and non-frontal faces demand new robust algorithms to achieve an acceptable performance.
Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems, 2016
With the advances in computing technologies, we have been undergoing a shift towards a digital wo... more With the advances in computing technologies, we have been undergoing a shift towards a digital world. As an inevitable result of this shift, the technology penetrates into education in myriad forms. Intelligent tutoring systems (ITS) are essential outcomes of this penetration, emerging to satisfy the needs of learners and instructors. Their working principle is based on collecting and processing data of all students through various modalities to understand the strengths and needs of learners. Yet, more important is that ITSs untangle the overlooked problem of traditional education: One size does not fit all, and there is a need for personalized tutoring for each individual. It is well known that that learning is emotional as well as intellectual. To truly meet the needs of education, we need empathic companions, ones that are affectively aware and thus can accompany the learner for an enhanced learning experience.
Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In... more Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In this paper, we focus on the problem of emotional engagement, and propose to detect important affective states (i.e., 'Satisfied', 'Bored', and 'Confused') of a learner in real time. We collected 210 hours of data from 20 students through authentic classroom pilots. The data included information from two modalities: (1) appearance which is collected from the camera, and (2) context-performance that is derived from the content platform. In this paper, data from nine students who attended the learning sessions twice a week are analyzed. We trained separate classifiers for different modalities (appearance and context-performance), and for different types of learning sections (instructional and assessment). The results show that different sources of information are generically better representatives of engagement at different sections: For instructional sections, gene...
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can... more We developed a real-time, multimodal Student Engagement Analytics Technology so that teachers can provide just-in-time personalized support to students who risk disengagement. To investigate the impact of the technology, we ran an exploratory semester-long study with a teacher in two classrooms. We used a multi-method approach consisting of a quasi-experimental design to evaluate the impact of the technology and a case study design to understand the environmental and social factors surrounding the classroom setting. The results show that the technology had a significant impact on the teacher's classroom practices (i.e., increased scaffolding to the students) and student engagement (i.e., less boredom). These results suggest that the technology has the potential to support teachers' role of being a coach in technology-mediated learning environments. CCS CONCEPTS • Applied computing-> Education-> Learning management systems • Human-centered computing-> Human computer interaction (HCI)-> Empirical studies in HCI
In our longitudinal research, we have been working towards an adaptive learning system automatica... more In our longitudinal research, we have been working towards an adaptive learning system automatically detecting student engagement as a higher-order user state in real-time. The labeled data necessary for supervised learning can be obtained through labeling conducted by human experts. Using multiple labelers to label collected data and obtaining agreement among different labelers on same samples of data is critical to train final engagement model accurately. Addressing these challenges, we developed a rigorous labeling process (HELP) specific to educational context with multi-faceted labels and multiple expert labelers. HELP has three distinct stages: (1) Pre-Labeling, including planning, labeler recruitment, training, and evaluation steps; (2) Labeling, involving actual labeling conducted by multiple labelers, and related steps for formative assessment of their performance; and (3) Post-Labeling, generating final labels and agreement measures through processing multiple decisions. In this paper, we outline proposed methods in HELP and describe the developed labeling tool.
Uploads
Papers