CN114663910A - Multi-mode learning state analysis system - Google Patents

Abstract

The invention relates to the field of digital data processing, in particular to a multi-mode-based learning state analysis system, which comprises an image acquisition module, a pre-processing module and a learning module, wherein the image acquisition module is used for collecting image data in a classroom and carrying out pre-processing on the image data to obtain a processed image; the teacher position information acquisition module is used for acquiring the position information of the teacher based on the processed image and the Faster R-CNN target detection model; the teacher posture information acquisition module is used for acquiring the posture information of the teacher based on the processed image and the Faster R-CNN target detection model; the student face information acquisition module is used for acquiring student face information based on the processed image and an ERT person face characteristic point detection method; and the analysis module is used for analyzing the learning state based on the position information, the posture information and the facial information of the student. Therefore, the student can be better supervised the class condition, and the learning efficiency of the student is improved.

Description

State Analysis System Based on Multimodal Learning

technical field

The invention relates to the field of digital data processing, in particular to a state analysis system based on multimodal learning.

Background technique

In traditional classroom education, the teacher judges the students' learning status through the facial expressions and head posture of the students. However, due to the limited energy of the teacher, it is impossible to observe the classroom learning status of each student in time, and thus cannot analyze the learning status of each student in time. Adjust teaching strategies according to the learning situation.

Smarter teaching is a hot word in current educational informatization research in my country. Some scholars call it a new form, new realm, and new stage of educational informatization development, raising the research on smarter teaching to a very high level. Nowadays, regarding the detection of students' classroom learning situation analysis, the detection can be divided into methods based on face recognition, recognition methods based on micro-expressions and detection methods based on brain waves.

Since there is only one variable in the above method, the detection accuracy is low.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a multi-modal learning state analysis system, which aims to analyze the data of the students' facial expressions, the teacher's position and the teacher's speech in class, and provide a visual analysis result. The analysis of students' concentration can be given, which is helpful for teachers to formulate appropriate teaching plans and promote classroom interaction between teachers and students.

In order to achieve the above purpose, the present invention provides a multi-modal learning state analysis system, including an image acquisition module, a teacher position information acquisition module, a teacher posture information acquisition module, a student facial information acquisition module and an analysis module,

The image acquisition module is used for collecting the image data inside the classroom and preprocessing to obtain the processed image;

The teacher position information acquisition module is used to obtain the teacher's position information based on the processed image and the Faster R-CNN target detection model;

The teacher's attitude information acquisition module is used to obtain the teacher's attitude information based on the processed image and the Faster R-CNN target detection model;

Described student facial information acquisition module, is used for acquiring student facial information based on processing image and ERT facial feature point detection method;

The analysis module is used to analyze the learning state based on the position information, the posture information and the student's face information.

Wherein, the image acquisition module includes an acquisition unit and a processing unit, and the acquisition unit is used to collect the image data of the camera;

The processing unit is used for normalizing the size of the image data and performing normalization processing to obtain a processed image.

Wherein, the specific method of collecting the image data of the camera is to acquire an image for one frame every 5 seconds.

Wherein, the teacher position information acquisition module includes a feature map extraction unit, a candidate unit, a region feature map generation unit and a teacher position acquisition unit,

The feature map extraction unit is used to extract the original feature map based on the processed image;

The candidate unit is used to input the original feature map into the candidate frame extraction network to generate a regional candidate frame;

The regional feature map generation unit is used to map the regional candidate frame to the original feature map, and pool it into a regional feature map;

The teacher position obtaining unit is used for inputting the regional feature map into the Faster R-CNN target detection model to obtain teacher position information.

Wherein, the teacher attitude information acquisition module includes a key point acquisition unit and a normalization unit, and the key point acquisition unit is used for acquiring human body attitude key point information based on the Faster R-CNN target detection model;

The normalization unit is used to perform attitude normalization module processing based on the human body attitude key point information.

Wherein, the specific steps of analyzing the learning state based on position information, posture information and student facial information are:

Build and train a multimodal feature fusion network structure based on a fully connected layer;

The location information, pose information and student face information are mapped to the feature fusion space for learning state analysis.

Wherein, the specific steps of analyzing the learning state based on position information, posture information and student facial information are:

A weighted fusion method is used to fuse position information, attitude information and student face information to obtain weighted fusion features;

Input the weighted fusion features into the fully connected layer;

Obtain the classification probability distribution of weighted fused features.

According to the multi-modal learning state analysis system of the present invention, cameras can be placed in the front and rear of the classroom respectively, and then the image data is acquired through the image acquisition module, and then the images are processed respectively through the Faster R-CNN target detection model to obtain The location information and facial information of the teacher, and then the facial information of the students in the image data is extracted by the ERT facial feature point detection method, and then the students' learning status can be evaluated comprehensively, so as to better supervise the students' class situation, to improve students' learning efficiency.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of the pooling of the present invention.

FIG. 2 is a flow chart of the teacher attitude information calculation according to the present invention.

Fig. 3 is a state diagram of a student's classroom learning by multimodal fusion analysis of the present invention.

FIG. 4 is a flow chart of the three-dimensional model features based on different modalities that can be obtained through joint learning of a three-modal convolutional neural network according to the present invention.

FIG. 5 is a structural diagram of a state analysis system based on multimodal learning of the present invention.

FIG. 6 is a structural diagram of an image acquisition module of the present invention.

FIG. 7 is a structural diagram of a teacher position information acquisition module of the present invention.

FIG. 8 is a structural diagram of a teacher attitude information acquisition module of the present invention.

1-Image acquisition module, 2-Teacher position information acquisition module, 3-Teacher attitude information acquisition module, 4-Student face information acquisition module, 5-Analysis module, 11-Acquisition unit, 12-Processing unit, 21-Feature map extraction Unit, 22-candidate unit, 23-region feature map generation unit, 24-teacher position acquisition unit, 31-key point acquisition unit, 32-normalization unit.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

Example 1

Please refer to FIG. 1 to FIG. 8 , the present invention provides a state analysis system based on multimodal learning:

Including image acquisition module 1, teacher position information acquisition module 2, teacher posture information acquisition module 3, student facial information acquisition module 4 and analysis module 5,

The image acquisition module 1 is used to collect image data inside the classroom and perform preprocessing to obtain a processed image;

The teacher position information acquisition module 2 is used to obtain the teacher's position information based on the processed image and the Faster R-CNN target detection model;

The teacher attitude information acquisition module 3 is used to obtain the teacher's attitude information based on the processed image and the Faster R-CNN target detection model;

Described student facial information acquisition module 4, is used for obtaining student facial information based on processing image and ERT facial feature point detection method;

The analysis module 5 is used to analyze the learning state based on the position information, the posture information and the student's face information.

In this embodiment, cameras can be placed in the front and back of the classroom, and then the image data is acquired by the image acquisition module 1, and then the images are processed by the Faster R-CNN target detection model to obtain the teacher's position information and facial information. , and then extract the student's face information in the image data through the ERT face feature point detection method, and then comprehensively evaluate the student's learning status, so as to better supervise the student's class situation and improve the student's learning efficiency. The formula of the ERT face feature point detection method is as follows:

ξ ^(t+1) = ξ ^(t) +r _t (I,ξ ^(t) )

Among them: t represents the serial number of the cascade, and r _t (,) represents the regressor of the current stage. The regressor inputs the face image I and the position coordinates of the facial feature points updated by the previous regressor, and the adopted feature can be gray value or other features. When the image is obtained, the algorithm generates the location of the accident, and the initial location is the location of the estimated facial feature points. The gradient boosting algorithm is used to minimize the error, and each cascaded regression silver is obtained.

Further, the image acquisition module 1 includes an acquisition unit 11 and a processing unit 12, and the acquisition unit 11 is used to collect the image data of the camera;

The processing unit 12 is configured to standardize the size of the image data and perform normalization processing to obtain a processed image.

The specific manner of collecting the image data of the camera is to acquire an image for one frame every 5 seconds. The sampling frequency is more suitable, and the image processing efficiency is improved while satisfying the real-time performance.

Further, the teacher position information acquisition module 2 includes a feature map extraction unit 21, a candidate unit 22, a region feature map generation unit 23 and a teacher position acquisition unit 24,

The feature map extraction unit 21 is used to extract the original feature map based on the processed image;

The candidate unit 22 is used to input the original feature map into the candidate frame extraction network to generate a regional candidate frame;

The regional feature map generation unit 23 is used to map the regional candidate frame to the original feature map, and pool it into a regional feature map;

The teacher position obtaining unit 24 is configured to input the regional feature map into the Faster R-CNN target detection model to obtain teacher position information.

In this embodiment, the specific method of extracting the original feature map is to first downscale the teacher behavior image in the class scene of any size to a fixed size of 256*256, and then input the 256*256 image into the CNN (convolutional layer), etc. The basic network, so as to take the feature map of the original image, the subsequent RPN layer and the fully connected layer can share the feature map.

Step 2: Input the feature map to the RPN to generate a region candidate frame. The East China window is applied to the feature map to determine and classify the target region, thereby generating a region candidate frame. RPN first performs a convolution operation on the feature map input by the shared convolution layer to obtain the feature vector, and then inputs the feature vector into a fully connected layer, namely the bounding box regression layer and the bounding box classification layer.

Step 3: Perform POI Pooling on each region feature map. Input the feature map and candidate region, map the candidate region to the feature map, and pool it into a uniform size region feature map, and then input the region feature map into the full connection to obtain the classification vector and position coordinates

Step 4: Finally, input the regional feature map into Faster R-CNN. According to the candidate frame proposed by RPN, further refine the position of the candidate frame and perform category regression.

For each selected area _i in the teacher's class behavior picture, fi is defined as the average pooled convolution feature from the modified area, so the dimension of the image feature vector is _2048. The fully connected layer converts fi into an h-dimensional vector .

v _i =W _v f _i +b _v

Therefore, the complete representation of the teacher's behavioral images during class is a set of embedding vectors.

V={v ₁ ,...,v _k },vi∈R

Among them, vi encodes a salient region _i , and k is the number of regions.

Further, the teacher attitude information acquisition module 3 includes a key point acquisition unit 31 and a normalization unit 32, and the key point acquisition unit 31 is used to acquire human body attitude key point information based on the Faster R-CNN target detection model;

The normalization unit 32 is configured to perform posture normalization module processing based on the human body posture key point information.

In this embodiment, the Faster R-CNN target detection model obtains the key point information of the human body posture, and performs the posture normalization module processing.

During pose normalization, the estimation of the coordinates of the key points, thereby calculating the transformation matrix.

The process is shown in Figure 2. In the figure, the coordinates of the key points are first extracted based on the key points obtained by the previous key point detection network. Then select the maximum point as the estimation of the key point, and have the coordinates of the key point.

Further, the concrete steps of analyzing the learning state based on position information, posture information and student face information are:

Build and train a multimodal feature fusion network structure based on a fully connected layer;

The location information, pose information and student face information are mapped to the feature fusion space for learning state analysis.

In order to solve the evaluation of students' classroom learning status by single modal data, this paper adopts the method of multi-modal fusion to analyze students' classroom learning status from three modalities. The overall flow chart is shown in Figure 3.

By building and training a multi-modal feature fusion network structure based on a fully connected layer, the multi-dimensional and multi-scale features are mapped to the feature fusion space. The advantage of this feature fusion method is that the model can learn two parallel networks in the training phase. The characteristic parameters of the model are completed independently, and the coordination feedback is completed, which realizes the end-to-end training of the model.

This paper selects the Softmax function to map the feature vector into a probability sequence, and retains the original information of more features. The process of Softmax calculating the output category y ⁽ⁱ⁾ is shown in the formula.

In the formula: η ⁱ is the feature value after fusion; k is the number of categories; P represents y ⁽ⁱ⁾ represents the probability value of category k.

The first stage: Pay attention to the teacher's position information feature and the student's eye offset in each teacher's picture, and get whether the student is looking at the teacher

The second stage: analyze the characteristic area of the teacher's behavior image, whether to look at the blackboard or the teacher at this time; then compare with the student's eye offset to get where the student should look at this time.

The third stage: analyze the student's facial feature map and the teacher's behavior and position, and obtain a similar embodiment 2

The difference between Embodiment 2 and Embodiment 1 is only that the specific steps of analyzing the learning state based on position information, posture information and student facial information are:

A weighted fusion method is used to fuse position information, attitude information and student face information to obtain weighted fusion features;

Input the weighted fusion features into the fully connected layer;

Obtain the classification probability distribution of weighted fused features.

Through the joint learning of the three-modal convolutional neural network, three 3D model features based on different modalities can be obtained. Different from the traditional feature fusion method using pooling operation, this paper uses the weighted fusion method to fuse three feature vectors based on the statistical method. The framework of the method is shown in Figure 4.

Specific formula:

Among them: f represents the feature vector extracted from different modal table features; α _i represents the weight of different modalities, the feature vector of the weighted fusion feature is input to the fully connected layer (FC layer), the dimension of the fully connected layer The sequence is 512, 256, C, and C represents the number of dataset categories. Finally, the classification probability distribution of the 3D model is obtained through a softmax layer. Divide the learning state into calm, normal, distracted, and difficult

In this paper, the correlation loss function is used to ensure that multiple modalities can guide each other during the training process, improve the learning speed of network training, and improve the robustness of the final feature vector.

The specific formula is as follows:

L _C (M _i ,M _j )=|ξ(f _Mi )-ξ(f _Mj )| ₂

where: 2 represents the correlation of two different eigenvectors. f represents the feature vector extracted from different modal tables; the subscript of M represents the data of the 1st, 2nd, and 3rd modalities; ξ=sigmoid(log(abs())) represents a normalized excitation function . During the training process, the value of the correlation loss gradually decreases, indicating that different modal features guide each other during the training process, which will speed up the training convergence speed and obtain more robust feature vectors. Taking modal M as an example, based on the design of this correlation loss function, the final loss functions of different modal networks are as follows:

表示基于单模态的交叉嫡损失；L_C(M₁,M2)和L_C(M₁,M₃)分别表示模态M,与模态M₂和M₃的相关性损失。最后通过随机梯度下降的反向传播优化这三个单模态网络。in:

Represents the single-modal-based cross-inheritance loss; L _C (M ₁ , M 2 ) and L _C (M ₁ , M ₃ ) represent the mode M, the correlation loss with modes M ₂ and M ₃ , respectively. Finally, the three unimodal networks are optimized by backpropagation with stochastic gradient descent.

The above disclosure is only a preferred embodiment of the present invention, and of course, it cannot limit the scope of rights of the present invention. Those of ordinary skill in the art can understand that all or part of the process for realizing the above-mentioned embodiment can be realized according to the rights of the present invention. The equivalent changes required to be made still belong to the scope covered by the invention.

Claims (7)

1. A multi-modal learning-based state analysis system is characterized in that,

comprises an image acquisition module, a teacher position information acquisition module, a teacher posture information acquisition module, a student face information acquisition module and an analysis module,

the image acquisition module is used for collecting image data in a classroom and carrying out preprocessing to obtain a processed image;

the teacher position information acquisition module is used for acquiring the position information of the teacher based on the processed image and the Faster R-CNN target detection model;

the teacher posture information acquisition module is used for acquiring the posture information of the teacher based on the processed image and the Faster R-CNN target detection model;

the student face information acquisition module is used for acquiring student face information based on a processed image and an ERT person face characteristic point detection method;

the analysis module is used for analyzing the learning state based on the position information, the posture information and the facial information of the student.

2. The multi-modal learning state analysis-based system of claim 1,

the image acquisition module comprises an acquisition unit and a processing unit, wherein the acquisition unit is used for collecting image data of the camera;

and the processing unit is used for standardizing the size of the image data and carrying out normalization processing to obtain a processed image.

3. The multi-modal-based learning state analysis system of claim 2, wherein the image data of the camera is collected by capturing an image for a frame every 5 seconds.

4. The multi-modal learning state analysis-based system of claim 1,

the teacher position information acquisition module comprises a feature map extraction unit, a candidate unit, a region feature map generation unit and a teacher position acquisition unit,

the feature map extraction unit is used for extracting an original feature map based on the processed image;

the candidate unit is used for inputting the original feature map into a candidate frame extraction network to generate a region candidate frame;

the regional characteristic diagram generating unit is used for mapping the regional candidate frame into the original characteristic diagram and pooling the regional candidate frame into a regional characteristic diagram;

and the teacher position acquisition unit is used for inputting the regional characteristic diagram into a Faster R-CNN target detection model to acquire teacher position information.

5. The multi-modal learning state analysis-based system of claim 4,

the teacher posture information acquisition module comprises a key point acquisition unit and a normalization unit, wherein the key point acquisition unit is used for acquiring key point information of the posture of the human body based on a Faster R-CNN target detection model;

and the normalization unit is used for carrying out posture normalization module processing based on the human posture key point information.

6. A multi-modal based learning state analysis system as defined in claim 1,

the specific steps of analyzing the learning state based on the position information, the posture information and the student face information are as follows:

constructing and training a multi-modal feature fusion network structure based on a full connection layer;

and mapping the position information, the posture information and the student face information to a feature fusion space for learning state analysis.

7. The multi-modal learning state analysis-based system of claim 1,

the specific steps of analyzing the learning state based on the position information, the posture information and the student face information are as follows:

fusing position information, posture information and student face information by adopting a weighted fusion method to obtain weighted fusion characteristics;

inputting the weighted fusion features into the full connection layer;

and obtaining the classification probability distribution of the weighted fusion characteristics.

Images