CN112131938A - Class attendance supervision method, system and storage medium based on TensorFlow framework - Google Patents

Abstract

The invention proposes a class attendance supervision method, system and storage medium based on the TensorFlow framework, which automatically crawls student information from the existing educational administration system and student information system of the school, and provides information on student identity information and daily class attendance. The information is collected, and the camera is used to obtain the student's face information in real time. Through the built-in training model, each face information is automatically matched with the student's identity information, which effectively solves a series of problems such as omission of attendance data, sign-in on behalf of, and escaping after sign-in. Effectively improve the school's basic teaching management, and the school can automatically complete the classroom roll call and the quality assessment of students' lectures.

Description

Class attendance supervision method, system and storage based on TensorFlow framework medium

technical field

The invention relates to the technical field of intelligent identification, in particular to a class attendance supervision method, system and storage medium based on a TensorFlow framework.

Background technique

Nowadays, there are many "head bowers" and "skimmers" in college classrooms. Students are late in class, leaving early, truancy, skipping classes, playing with mobile phones and other bad behaviors, which will not only affect their learning, but also affect the mood of teachers in class, and they are also disrespectful to teachers' labor. . The current roll call system in colleges and universities adopts two technical methods: one uses fingerprint attendance machine combined with terminal software to realize student attendance statistics. The design uses special photoelectric conversion equipment and computer image processing technology to collect, analyze and compare students' fingerprints, which can automatically, quickly and accurately identify students' identities. The other uses the mobile phone attendance application to achieve student punching statistics. Mainstream attendance applications mainly realize student attendance authentication through GPS and face recognition. Students need to arrive at the designated classroom location and perform face punching in order to successfully achieve attendance punching.

However, the traditional attendance methods generally have the following disadvantages: fingerprint attendance recognition requires queuing, low efficiency, suitable for scenarios with small numbers of people, large environmental impact, requiring fingerprint cleaning, dry fingers, too cold, water or peeling will affect the punching effect , it is easy to leak fingerprints; there is a deviation in the positioning of students in the mobile phone attendance application. It may happen that students arrive in the classroom but cannot punch in, which is greatly affected by the network speed. If the classroom information is poor, it will affect punching.

The traditional attendance punch-in technology is only suitable for regular punch-in before, during or after class, and cannot monitor the student's course attendance status in real time; the traditional attendance punch-in technology can only realize punch-in and check-in, and cannot analyze and supervise the students' class status.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve at least one of the aforementioned technical defects.

To this end, one purpose of the present invention is to propose a class attendance supervision system and method based on the TensorFlow framework, which adopts deep learning combined with intelligent biometric identification technology to collect student identity information and daily class information, and effectively solve the problem of attendance data. A series of problems such as omission, sign-in on behalf of, and escaping after sign-in have effectively improved the basic teaching management of the school. With its cooperation, the school can automatically complete the class roll call and the quality assessment of students' listening to the class.

In order to achieve the above purpose, an embodiment of the present invention provides a method for classroom attendance supervision based on a TensorFlow framework, including acquiring real-time face images of students in class through a terminal device, and inputting the acquired face images into trained students The convolutional neural network of the TensorFlow framework is used to calculate the input face image under the TensorFlow framework, and the prediction result is obtained; the attendance record is generated according to the prediction result to realize classroom supervision.

Preferably, the convolutional neural network includes:

The first layer is a convolution layer, the convolution layer is used to perform feature extraction on the preprocessed image information; form a feature vector matrix;

The second layer, the pooling layer, is used to extract features from the feature vector matrix again to form face feature data;

The third layer, the fully connected layer, uses the triple loss function to perform metric learning on the facial feature data, and uses the softmax loss function to perform binary classification on the facial feature data after metric learning, and form a prediction result according to the classification result.

Preferably in any of the above embodiments, the constructed convolutional neural network is trained, including

Obtain the data set of students from the existing educational administration system and student information system of the school; divide the obtained data set into training set and verification set, input the training set data into the constructed convolutional neural network, and perform the convolutional neural network analysis. Traverse the training, and input the validation set data into the trained convolutional neural network to verify whether the prediction result obtained by the convolutional neural network is correct.

Preferably in any of the above embodiments, the triple loss function is used to perform metric learning on the facial feature data; each facial feature data is classified into a triple array Xi, and each triple array Xi, The following conditions need to be met:

where x _a is the anchor image, x _p is the image of the same subject, X _n is the image of another different subject, f is the mapping learned by the model, and α is the residual applied between the distances between positive and negative pairs The positive ratio is the image of the same subject, and the negative ratio is the image of different subjects.

Preferably in any of the above embodiments, the softmax loss function is used for binary classification; the decision boundary between each class can be given by the following formula:

||x||(||w ₁ ||cosθ ₁ -||W ₂ ||cosθ ₂ )=0

where χ is the feature vector, W ₁ and W ₂ are the weights corresponding to each class, and θ ₁ and θ ₂ are the angles between χ and W ₁ and W ₂ , respectively.

Preferably in any of the above embodiments, dividing the acquired data set into a training set and a validation set also includes preprocessing the data set, forming feature engineering, and performing similarity measurement, exploratory analysis, data analysis on the data set Normalization, outlier handling, and missing value handling operations.

Preferably in any of the above embodiments, the data set includes at least three types of data sets: CK+ facial expression data set, AFLW face data set and CMU Panoptic Dataset human body key point data set and student identity information.

In any of the above embodiments, preferably, in the traversal training process, the aggregated scalar of tensorboard is used to measure the overall loss and accuracy, and a migration algorithm is also used to migrate the trained convolutional neural network to the terminal device.

The present invention also provides a classroom attendance supervision system based on the TensorFlow framework, which is used to implement the above-mentioned supervision method, including a collection terminal and a background face recognition server arranged in front of the classroom;

The acquisition terminal includes a camera and a built-in processor; the camera is used to obtain real-time face images of students in class, and input the obtained face images into the built-in processor;

The built-in processor is provided with a convolutional neural network migrated from the background face recognition server, and the inputted face image is calculated by using the convolutional neural network under the framework of TensorFlow, a prediction result is obtained, and attendance is generated according to the prediction result. Record;

The background face recognition server is used to construct and train a convolutional neural network, migrate the trained convolutional neural network to the built-in processor, communicate with the built-in processor through an API interface, and form the generated attendance records into supervision data .

The present invention also provides a computer storage medium, where a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the steps of the above method are implemented.

The classroom attendance supervision system and method based on the TensorFlow framework provided according to the embodiment of the present invention has at least the following advantages compared with the prior art:

1. The acquisition terminal is installed in front of the classroom, the mechanical structure is hidden, the camera is used to obtain the student's face information in real time, and the convolutional neural network is used to automatically match the student's identity information to each face information, so as to realize the student's attendance record. The back-end high-performance back-end face recognition server serves for campus classroom attendance supervision.

2. It can be connected to the existing educational administration system and student information system of the school, so as to automatically obtain student information and class scheduling information and reduce human input. At the same time, the system conducts multi-dimensional information analysis by collecting the information and data of the whole course of the school, including student attendance, teacher attendance, students' concentration, and students' location analysis, to provide data support and statistics for the school's basic teaching management. analyze. In addition, the system supports data sharing, which can provide classroom teaching data support for the school's big data platform;

3. A business platform is built in the background face recognition server, and in the process of visual analysis of the supervision data, the students' facial expressions can be further studied, and the facial expressions can be extracted by extracting fine facial movements such as frowning and smiling. , the use of deep learning is helpful for teachers to adjust the rhythm of the class and to know the students' mastery of the important and difficult points of the course in time.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a structural block diagram of a classroom attendance supervision system based on a TensorFlow framework provided by an embodiment of the present invention;

2 is a flowchart of a method for classroom attendance supervision based on a TensorFlow framework provided by an embodiment of the present invention;

Fig. 3 is a kind of TensorBoard visual neural network model hierarchy diagram in the classroom attendance supervision system based on TensorFlow framework provided by the embodiment of the present invention;

Fig. 4 is a kind of classroom attendance supervision system TensorBoard based on the TensorFlow framework provided by the embodiment of the present invention. Visual training model error and accuracy change diagram;

5 is a platform interface of a classroom attendance supervision system based on a TensorFlow framework provided by an embodiment of the present invention;

6 is another interface diagram of a classroom attendance supervision system based on a TensorFlow framework provided by an embodiment of the present invention;

7 is a structural diagram of a convolutional neural network based on a class attendance supervision method under the TensorFlow framework provided by an embodiment of the present invention;

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

As shown in FIG. 2 , a method for classroom attendance supervision based on the TensorFlow framework according to an embodiment of the present invention includes:

S1. Obtain real-time face images of students in class through a terminal device, and input the obtained face images into the trained convolutional neural network;

It should be noted that the convolutional neural network in the terminal device is a mature neural network migrated from the background face recognition server. On the background face recognition server, it is necessary to build a convolutional neural network in advance, and use the existing network from the school. The educational administration system and student information system of the system obtain the data sets of students; train the built convolutional neural network, and use the migration algorithm to migrate to the terminal device after training.

S2. Use the convolutional neural network to calculate the input face image under the TensorFlow framework to obtain the prediction result;

In one embodiment of the present invention, training a neural network includes acquiring a data set of students from the existing educational administration system and student information system of the school; dividing the acquired data set into a training set and a verification set, and inputting the training set data to construct In a good convolutional neural network, the convolutional neural network is traversed and trained, and the validation set data is input into the trained convolutional neural network to verify whether the prediction result obtained by the convolutional neural network is correct.

Dividing the acquired data set into training set and validation set also includes preprocessing the data set, forming feature engineering, and performing similarity measurement, exploratory analysis, data normalization, outlier processing and missing value processing on the data set. operate.

It should be noted that the dataset includes at least three types of datasets: CK+ facial expression dataset, AFLW face dataset and CMU Panoptic Dataset human key point dataset and student identity information.

As shown in Figure 7, the convolutional neural network includes:

The first layer is a convolution layer, the convolution layer is used to perform feature extraction on the preprocessed image information; form a feature vector matrix;

The second layer, the pooling layer, is used to extract features from the feature vector matrix again to form face feature data;

The third layer, the fully connected layer, uses the triple loss function to perform metric learning on the facial feature data, and uses the softmax loss function to perform binary classification on the facial feature data after metric learning, and form a prediction result according to the classification result.

Use the TensorFlow framework to build a multi-layer convolutional neural network architecture with 3×3 convolution kernels, and these convolutions are connected at the end. The first layer predicts the PAFs Lt and the last layer predicts the evidence map St, connecting the predictions of each stage and their corresponding image features to each subsequent stage. The convolutional neural network is continuously adjusted and optimized in the training process to achieve the optimal recognition effect, and finally the successfully trained model is used in the collection terminal using the migration algorithm. The neural network architecture looks like this:

Use triplet loss function to perform metric learning on face feature data; classify each face feature data into a triplet array Xi, and each triplet array Xi needs to meet the following conditions:

where x _a is the anchor image, x _p is the image of the same subject, X _n is the image of another different subject, f is the mapping learned by the model, and α is the residual applied between the distances between positive and negative pairs The positive ratio is the image of the same subject, and the negative ratio is the image of different subjects.

The softmax loss function is used for binary classification, and the decision boundary between each class can be given by the following formula:

||x||(||w ₁ ||cosθ ₁ -||W ₂ ||cosθ ₂ )=0

where χ is the eigenvector, W ₁ and W ₂ are the weights corresponding to each class, and θ ₁ and θ ₂ are the angles between χ and W ₁ and W ₂ , respectively

S3. Generate attendance records according to the prediction results to realize classroom supervision.

As shown in Figure 3, using the TensorFlow framework, combined with the convolutional neural network and the open source OpenPose framework, the training model of the neural network is obtained, and the training model is used to train the data set; The fitting process affects the effect of class parameters and sub-models on the effect of class parameters on the training effect, and adjusts the parameters; the successfully trained model is applied in the collection terminal using the migration algorithm; as shown in Figure 4, in the traversal training process, the tensorboard summary scalar is used to Measure overall loss and accuracy.

As shown in Figure 1, a classroom attendance supervision system based on the TensorFlow framework includes a collection terminal and a background face recognition server arranged in front of the classroom;

The acquisition terminal includes a camera and a built-in processor; the camera is used to obtain real-time face images of students in class, and input the obtained face images into the built-in processor;

The built-in processor is provided with a convolutional neural network migrated from the background face recognition server, and the inputted face image is calculated by using the convolutional neural network under the framework of TensorFlow, a prediction result is obtained, and attendance is generated according to the prediction result. Record;

The background face recognition server is used to construct and train a convolutional neural network, migrate the trained convolutional neural network to the built-in processor, communicate with the built-in processor through an API interface, and form the generated attendance records into supervision data .

The background face recognition server also has a data storage module, a data preprocessing module,

The data storage module is used to store the data sets of students obtained from the existing educational administration system and student information system of the school; in one embodiment of the present invention, the data sets include at least three types of data sets: CK+ face expression data set, AFLW Face dataset and CMU Panoptic Dataset human keypoint dataset.

The data preprocessing module is used to preprocess the data set, form feature engineering, and perform similarity measurement, exploratory analysis, data normalization, outlier processing and missing value processing operations on the data set;

A convolutional neural network building module for constructing a convolutional neural network, the convolutional neural network including:

The first layer is a convolution layer, the convolution layer is used to perform feature extraction on the preprocessed image information; form a feature vector matrix;

The second layer, the pooling layer, is used to extract features from the feature vector matrix again to form face feature data;

The third layer, the fully connected layer, uses the triple loss function to perform metric learning on the facial feature data, and uses the softmax loss function to perform binary classification on the facial feature data after metric learning, and form a prediction result according to the classification result.

It should be noted that the triple loss function is used to perform metric learning on the facial feature data; each dataset is classified into a triple array Xi

Each ternary array Xi needs to meet the following conditions:

where x _a is the anchor image, x _p is the image of the same subject, X _n is the image of another different subject, f is the mapping learned by the model, and α is the margin applied between the distances between positive and negative pairs .

CNNs trained with triplet loss converge slower than those with sofimax because a large number of triples (or pairings in contrastive loss) are required to cover the entire training set. In this case, we can adopt another embodiment, using the softmax loss function for binary classification; the decision boundary between each class can be given by:

||x||(||W ₁ ||cosθ ₁ -||W ₂ ||cosθ ₂ )=0

where χ is the feature vector, W ₁ and W ₂ are the weights corresponding to each class, and θ ₁ and θ ₂ are the angles between χ and W ₁ and W ₂ , respectively.

As shown in Figure 3, using the TensorFlow framework, combined with the convolutional neural network and the open source OpenPose framework, the training model of the neural network is obtained, and the training model is used to train the data set; The fitting process affects the effect of class parameters and sub-models on the effect of class parameters on the training effect, and adjusts the parameters; the successfully trained model is applied in the collection terminal using the migration algorithm; as shown in Figure 4, in the traversal training process, the tensorboard summary scalar is used to Measure overall loss and accuracy.

The acquisition terminal includes a camera and a microprocessor. It should be noted that the microprocessor can be a Raspberry Pi, namely Raspberry Pi, abbreviated as RPi, (or RasPi/RPI) is designed for learning computer programming education, only a credit card. Small-sized microcomputer whose system is based on Linux. With the release of Windows10 IoT, we will also be able to use the Raspberry Pi running Windows. The camera is used to collect real-time face images of students in class, and send the collected face images to the Raspberry Pi. According to the Raspberry Pi Face images and built-in migration algorithms are used to evaluate students' classroom status, and send the evaluation and analysis results to the background face recognition server through the API interface, and the face recognition server forms the supervision data.

TensorFlow: TensorFlow is the second-generation artificial intelligence learning system developed by Google based on DistBelief. Its name comes from its own operating principle. Tensor (tensor) means N-dimensional array, Flow (flow) means calculation based on data flow graph, TensorFlow is the calculation process for tensors flowing from one end of the flow graph to the other end. TensorFlow is a system that transmits complex data structures to artificial intelligence neural networks for analysis and processing.

TensorBoard: TensorBoard is a suite of visualization tools provided by TensorFlow, which can help developers easily understand, debug, and optimize TensorFlow programs.

Neural Network: Artificial Neural Networks (ANNs for short), also referred to as Neural Networks (NNs) or Connection Models, is a method that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. The mathematical model of the algorithm. This kind of network depends on the complexity of the system, and achieves the purpose of processing information by adjusting the interconnected relationship between a large number of internal nodes.

As shown in Figure 5-6, the camera is used to obtain the student's face information in real time. Through the built-in training model, the Raspberry Pi automatically matches the student's identity information to each face information, and realizes the student's attendance record. The information data of the whole process, including students' attendance, teachers' attendance, students' concentration, students' location analysis, carry out multi-dimensional information analysis, and provide data support and statistical analysis for the school's basic teaching management. In addition, the system supports data sharing, which can provide classroom teaching data support for the school's big data platform;

In the process of building a business platform in the background face recognition server, in the process of visual analysis of the supervision data, the students' facial expressions can be further studied, and the facial expressions such as frowning and smiling can be extracted by extracting fine movements, and the changes in facial expressions can be used to form lecture feedback. Deep learning is helpful for teachers to adjust the rhythm of the class, and to know the students' mastery of the major and difficult points of the course in time.

According to the system and method for classroom attendance supervision based on the TensorFlow framework provided by the embodiments of the present invention, the collection terminal is installed in front of the classroom, the mechanical structure is concealed, and the back-end high-performance background face recognition server is used to serve the campus classroom attendance supervision. It can be connected to the existing educational administration system and student information system of the school, so as to automatically obtain student information and class scheduling information and reduce manpower investment. At the same time, the system conducts multi-dimensional information analysis by collecting the information and data of the whole course of the school, including student attendance, teacher attendance, students' concentration, and students' location analysis, to provide data support and statistics for the school's basic teaching management. analyze. In addition, the system supports data sharing, which can provide classroom teaching data support for the school's big data platform; a business platform is built in the back-end face recognition server, and students' facial expressions can be further studied in the process of visual analysis of the supervision data. By extracting fine facial movements such as frowning and smiling, and telling the changes in facial expressions to form lecture feedback, the use of deep learning is helpful for teachers to adjust the rhythm of the class and timely understand the students' mastery of the important and difficult points of the course.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Variations, modifications, substitutions, and alterations to the above-described embodiments are possible within the scope of the present invention without departing from the scope of the present invention. The scope of the invention is defined by the appended claims and their equivalents.

Claims (10)

1. a kind of classroom attendance supervision method based on TensorFlow framework, is characterized in that, comprises Obtain real-time face images of students in class through the terminal device, and input the obtained face images into the trained convolutional neural network; Using the convolutional neural network under the TensorFlow framework to calculate the input face image to obtain the prediction result; Generate attendance records according to the prediction results to realize classroom supervision. 2. The supervision method according to claim 1, wherein the convolutional neural network comprises: The first layer is a convolution layer, the convolution layer is used to perform feature extraction on the preprocessed image information; form a feature vector matrix; The second layer, the pooling layer, is used to extract features from the feature vector matrix again to form face feature data; The third layer, the fully connected layer, uses the triple loss function to perform metric learning on the facial feature data, and uses the softmax loss function to perform binary classification on the facial feature data after metric learning, and form a prediction result according to the classification result. 3. The supervision method according to claim 1, wherein the training of the convolutional neural network comprises: Obtain the data set of students from the existing educational administration system and student information system of the school; divide the obtained data set into training set and verification set, input the training set data into the constructed convolutional neural network, and perform the convolutional neural network analysis. Traverse the training and input the validation set data into the trained convolutional neural network to verify whether the prediction result obtained by the convolutional neural network is correct. 4. supervising method according to claim 2, is characterized in that, described adopting triple loss function to carry out metric learning to facial feature data; Each facial feature data is classified into a triple array Xi, each The ternary array Xi needs to meet the following conditions:

where x _a is the anchor image, x _p is the image of the same subject, X _n is the image of another different subject, f is the mapping learned by the model, and α is the residual applied between the distances between positive and negative pairs The positive ratio is the image of the same subject, and the negative ratio is the image of different subjects. 5. The supervision method according to claim 2, wherein the softmax loss function is used for binary classification, and the decision boundary between each class can be given by the following formula: ||x||(||W ₁ ||cosθ ₁ -||W ₂ ||cosθ ₂ )=0 where χ is the feature vector, W ₁ and W ₂ are the weights corresponding to each class, and θ ₁ and θ ₂ are the angles between χ and W ₁ and W ₂ , respectively. 6. The supervising method according to claim 3, characterized in that, when dividing the acquired data set into a training set and a verification set, it also includes preprocessing the data set, forming feature engineering, and performing similarity measurement on the data set, Exploratory analysis, data normalization, outlier handling, and missing value handling operations. 7. The method for supervision according to claim 3, wherein the data set comprises at least three types of data sets: CK+ facial expression data set, AFLW face data set and CMU Panoptic Dataset human key point data set and Student identification information. 8. the class attendance supervision system based on TensorFlow framework according to claim 3, it is characterized in that, adopt tensorboard summary scalar to measure overall loss and accuracy in the traversal training process, also comprise utilizing migration algorithm to train the good volume. Integrating neural networks to end devices. 9. a class attendance supervision system based on the TensorFlow framework, is characterized in that, for implementing the supervision method described in any one of the above-mentioned claims 1-8, comprises the collection terminal that is arranged in front of the classroom, the backstage face recognition server; The acquisition terminal includes a camera and a built-in processor; the camera is used to obtain real-time face images of students in class, and input the obtained face images into the built-in processor; The built-in processor is provided with a convolutional neural network migrated from the background face recognition server, and the inputted face image is calculated by using the convolutional neural network under the framework of TensorFlow, a prediction result is obtained, and attendance is generated according to the prediction result. Record; The background face recognition server is used to construct and train a convolutional neural network, migrate the trained convolutional neural network to the built-in processor, communicate with the built-in processor through an API interface, and form the generated attendance records into supervision data . 10 . A computer storage medium, wherein a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images