CN111580649B - Deep learning-based aerial handwriting interaction method and system - Google Patents

Abstract

The invention discloses an aerial handwriting interaction method and system based on deep learning, wherein action data are acquired through a sensor; preprocessing the action data to obtain an action data set; constructing a neural network and training the neural network through the action data set after manual labeling to serve as an action recognition model; identifying action data acquired by the sensor through an action identification model; the invention has the main functions of facilitating the text input of equipment such as virtual reality, augmented reality and the like, and has better naturalness and high efficiency compared with the most commonly used information input interaction equipment such as a keyboard and a tablet at present; compared with other interaction modes, such as voice interaction, the method has stronger robustness; the neural network algorithm model is adopted for carrying out attitude estimation analysis, so that the accuracy is higher.

Description

A method and system for in-air handwriting interaction based on deep learning

technical field

The present disclosure relates to a technology combining deep learning and data communication, and in particular to a method and system for in-air handwriting interaction based on deep learning.

Background technique

With the popularity and implementation of wearable devices, smart home, Internet of Things and other fields in the technology circle, building an intelligent life in an all-round way has become the next focus, and multi-channel, multimedia intelligent human-computer interaction methods will gradually become A key part of achieving this kind of life. The new human-computer interaction environment represented by virtual reality and the mobile interactive platform represented by handheld computers and smart phones are two important development trends of current computers. The human-computer interaction technology represented by mouse and keyboard is the main bottleneck affecting their development. Interacting with the (visible or invisible) computer environment in a parallel and non-precise manner by using multiple perceptual interaction channels of humans (such as voice, handwriting, gesture, sight, expression, etc.) can improve the naturalness of human-computer interaction and efficiency. Multi-channel, multimedia intelligent human-computer interaction is both a challenge and an excellent opportunity for us.

In-air handwriting recognition is an important branch of gesture recognition, and its main function is to facilitate text input in virtual reality, augmented reality and other devices. These devices focus on immersive experience, so it is impossible to use a mouse and keyboard for human-computer interaction like a computer. At this time, the research and development of handwriting in the air is of great significance.

Contents of the invention

The purpose of the present invention is to propose a method and system for in-air handwriting interaction based on deep learning, so as to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

In order to solve the above problems, the present disclosure provides a deep learning-based air handwriting interaction method and a technical solution of the system, which collects motion data through sensors; preprocesses the motion data to obtain motion data sets; constructs a neural network and passes manual annotation The action data set trains the neural network as an action recognition model; the action data collected by the sensor is recognized through the action recognition model.

In order to achieve the above object, according to one aspect of the present disclosure, a method for in-air handwriting interaction based on deep learning is provided, the method includes the following steps:

Wear the motion capture glove on your hand; touch the touch sensor with your thumb, and wave your index finger to complete the action of writing in the air; wherein, the motion capture glove is equipped with multiple sensors, and the sensors are gyroscopes.

S100, collecting action data through sensors;

S200, preprocessing the action data to obtain an action data set;

S300, constructing a neural network and training the neural network as an action recognition model through the manually marked action data set;

S400. Recognize the motion data collected by the sensor by using the motion recognition model.

Further, in S100, the sensor is a gyroscope.

Further, in S200, the method for preprocessing the action data includes but not limited to any one or more of Kalman filtering, mean centering, and principal component analysis.

Further, in S300, the so-called neural network is any neural network among LSTM and BP.

Further, in S300, the method of constructing a neural network and training the neural network as an action recognition model through the manually marked action data set is as follows:

S301: Construct any neural network among LSTM and BP, and set the input of the neural network as the action data collected by the sensor, and set the output as the judgment text;

S302: The method for manually labeling the data set of action data is: by sampling, the length of N action data is used as a data group (for example, N=60, 60 action data for each data group), from the database Read the letters, characters or symbols corresponding to the top three similarities of each data group in the pre-stored data group (that is, after the similarity is sorted, the data group represented by the top three similarities), (that is, , obtain the three largest similarity values, output the letters, characters or symbols in the pre-stored data groups corresponding to the three similarity values at the same time, manually mark the output letters, characters or symbols, and mark the correct Letters, characters or symbols;), there are multiple pre-stored data groups stored in the database, and each pre-stored data group corresponds to a letter, character or symbol, and then manually from the pre-stored data corresponding to the top three high schools of similarity The correct letters, characters or symbols are marked in the letters, characters or symbols corresponding to the group; the default setting of N is 60 action data, which can be manually set and adjusted; among them, the prestored data groups in the database The calculation method for reading the similarity with each data group is as follows: Let each data group be Act, the pre-stored data group in the database is Act _j , i=1...m, m is the number of the pre-stored data group in the database The number of data, the similarity between Act and Act _j is the number of the same action data in the data group Act and the pre-stored data group Act _j in each database, that is, if there are 10 identical action data in Act and Act _j , they are similar The degree is 10; if there are pre-stored data groups with the same similarity, then output the letters, characters or symbols corresponding to the pre-stored data groups with the same similarity at the same time for manual selection and labeling, and the action data after the labeling is completed The data set is used as the labeled data set.

S302: Obtain an action recognition model by training the neural network through the labeled data set;

Among them, the training neural network adopts the backpropagation algorithm, the calculation of the gradient of the network parameters adopts the stochastic gradient descent method, and the cross entropy (cross entropy) is used to calculate the classification error, and this process is iteratively carried out until the average error is satisfied; the loss function L is:

L is the loss function, y _j is the true value label, and s _j is the jth value of the output vector s of the softmax logistic regression model, indicating the probability that this sample belongs to the jth category.

Further, in S400, the motion data collected by the motion recognition model recognition sensor is the letter, character or symbol corresponding to the motion data collected by the motion recognition model recognition sensor in the database.

The present invention also provides a deep learning-based air handwriting interaction system, the system comprising: a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor executing Said computer program runs in units of the following systems:

an action acquisition unit, configured to acquire action data through sensors;

A preprocessing unit, configured to preprocess the action data to obtain an action data set;

The recognition model construction unit is used to construct a neural network and train the neural network as an action recognition model through the manually marked action data set;

The motion recognition model unit is used to recognize the motion data collected by the sensor through the motion recognition model.

The beneficial effects of the present disclosure are as follows: the present invention provides a method and system for in-air handwriting interaction based on deep learning, and its main function is to facilitate the text input of devices such as virtual reality and augmented reality. keyboard, writing board, the present invention has better naturalness and high efficiency; compared with other interactive modes, such as voice interaction, the present invention has stronger robustness; adopting the neural network algorithm model for attitude estimation analysis has higher Accuracy.

Description of drawings

The above and other features of the present disclosure will be more apparent through a detailed description of the embodiments shown in the drawings. The same reference numerals in the drawings of the present disclosure represent the same or similar elements. Obviously, the appended The drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative work. In the drawings:

Fig. 1 shows the flow chart of a kind of aerial handwriting interaction method based on deep learning;

Figure 2 shows a system diagram of an air handwriting interaction system based on deep learning.

Detailed ways

The concept, specific structure and technical effects of the present disclosure will be clearly and completely described below in conjunction with the embodiments and drawings, so as to fully understand the purpose, scheme and effect of the present disclosure. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

FIG. 1 is a flowchart of a method for handwriting in-air interaction based on deep learning according to the present disclosure. The method for handwriting in-air interaction based on deep learning according to an embodiment of the present disclosure will be described below in conjunction with FIG. 1 .

The present disclosure proposes a method for in-air handwriting interaction based on deep learning, which specifically includes the following steps:

S100, collecting action data through sensors;

S200, preprocessing the action data to obtain an action data set;

S300, constructing a neural network and training the neural network as an action recognition model through the manually marked action data set;

S400. Recognize the motion data collected by the sensor by using the motion recognition model.

Further, in S100, the sensor is a gyroscope.

Further, in S200, the method for preprocessing the action data includes but not limited to any one or more of Kalman filtering, mean centering, and principal component analysis.

Further, in S300, the so-called neural network is any neural network among LSTM and BP.

Further, in S300, the method of constructing a neural network and training the neural network as an action recognition model through the manually marked action data set is as follows:

S301: Construct any neural network among LSTM and BP, and set the input of the neural network as the action data collected by the sensor, and set the output as the judgment text;

S302: The method for manually labeling the data set of action data is: by sampling, the length of N action data is used as a data group (for example, N=60, 60 action data for each data group), from the database Read the letters, characters or symbols corresponding to the top three similarities of each data group in the pre-stored data groups (that is, after the similarity is sorted, the data groups represented by the top three similarities), the database There are multiple pre-stored data groups stored in it, and each pre-stored data group corresponds to a letter, text or symbol, and then the letters, text or symbols corresponding to the pre-stored data groups corresponding to the top three high schools are manually selected. The correct letters, words or symbols are marked in the output letters, words or symbols; the default setting of N is 60 action data, which can be manually set and adjusted; the similarity between the pre-stored data groups in the database and each data group is read The calculation method of is, let each data group be Act, the pre-stored data group of the database is Act _j , i=1...m, m is the data quantity of the pre-stored data group in the database, then the distance between Act and Act _j The similarity is the number of identical action data between the data group Act and the pre-stored data group Act _j of each database, that is, if there are 10 identical action data in Act and Act _j , the similarity is 10; if there are pre-stored data with the same similarity If there is a data group with the same similarity, the letters, characters or symbols corresponding to the pre-stored data groups with the same similarity are output at the same time for manual selection and labeling, and the data set of the action data after the labeling is completed is used as the labeling data set.

S302: Obtain an action recognition model by training the neural network through the labeled data set;

Among them, the training neural network adopts the backpropagation algorithm, the calculation of the gradient of the network parameters adopts the stochastic gradient descent method, and the cross entropy (cross entropy) is used to calculate the classification error, and this process is iteratively carried out until the average error is satisfied; the loss function L is:

L is the loss function, y _j is the true value label, and s _j is the jth value of the output vector s of the softmax logistic regression model, indicating the probability that this sample belongs to the jth category.

Further, in S400, the motion data collected by the motion recognition model recognition sensor is the letter, character or symbol corresponding to the motion data collected by the motion recognition model recognition sensor in the database.

An in-air handwriting interaction system based on deep learning provided by an embodiment of the present disclosure, as shown in FIG. 2 is a diagram of an in-air handwriting interaction system based on deep learning in the present disclosure. The handwriting interaction system includes: a processor, a memory, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, the above-mentioned air handwriting interaction based on deep learning is realized. Steps in the system embodiment.

The system includes: a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor executes the computer program to run in the following system units:

an action acquisition unit, configured to acquire action data through sensors;

A preprocessing unit, configured to preprocess the action data to obtain an action data set;

The recognition model construction unit is used to construct a neural network and train the neural network as an action recognition model through the manually marked action data set;

The motion recognition model unit is used to recognize the motion data collected by the sensor through the motion recognition model.

The air handwriting interaction system based on deep learning can run on computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers. The operable system of the in-air handwriting interaction system based on deep learning may include, but not limited to, a processor and a memory. Those skilled in the art will understand that the example is only an example of a deep learning-based air handwriting interaction system, and does not constitute a limitation to a deep learning-based air handwriting interaction system, and may include more or more few components, or combine certain components, or different components, for example, the air handwriting interaction system based on deep learning may also include input and output devices, network access devices, buses and so on.

The so-called processor can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., the processor is the control center of the operating system of the air handwriting interactive system based on deep learning, and utilizes various interfaces and The line connects the various parts of the entire operating system.

The memory can be used to store the computer programs and/or modules, and the processor realizes the one by running or executing the computer programs and/or modules stored in the memory and calling the data stored in the memory. Various functions of a deep learning-based aerial handwriting interaction system. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) and the like; the storage data area may store Data created based on the use of the mobile phone (such as audio data, phonebook, etc.), etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, internal memory, plug-in hard disk, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card , flash card (Flash Card), at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

While the description of the present disclosure has been presented with considerable detail and in particular has described a few described embodiments, it is not intended to be limited to any such details or embodiments or to any particular embodiment, but rather should be read by reference The appended claims provide the broadest possible interpretation of these claims in view of the prior art, effectively encompassing the intended scope of the present disclosure. Furthermore, the disclosure has been described above in terms of embodiments foreseeable by the inventors for the purpose of providing a useful description, and insubstantial modifications of the disclosure which are not presently foreseeable may still represent equivalent modifications of the disclosure.

Claims (5)

1. An air handwriting interaction method based on deep learning, which is characterized by comprising the following steps:

s100, collecting action data through a sensor;

s200, preprocessing action data to obtain an action data set;

s300, constructing a neural network and training the neural network through the manually marked action data set to serve as an action recognition model;

s400, identifying action data acquired by a sensor through an action identification model;

in S300, the method for constructing the neural network and training the neural network as the motion recognition model through the manually marked motion data set is as follows

S301: constructing any one of LSTM and BP, setting the input of the neural network as action data acquired by a sensor, and outputting characters set as judgment;

s302: the method for manually labeling the data set of the action data comprises the following steps: reading letters, characters or symbols corresponding to the first three high similarity of each data group from the prestored data groups of a database by taking the length of N pieces of action data as one data group through sampling, wherein a plurality of prestored data groups are stored in the database, each prestored data group correspondingly represents one letter, character or symbol, and then correct letters, characters or symbols are marked from letters, characters or symbols which are simultaneously output from the letters, characters or symbols corresponding to the prestored data groups corresponding to the first three high similarity manually; wherein the database is pre-storedThe method for calculating the similarity between the reading of the data sets and each data set is that each data set is made to be Act, and the pre-stored data sets of the database are made to be Act

I= … m, m is the number of data of the pre-stored data set in the database, then Act and +.>

The similarity between the data set Act and the pre-stored data set +/for each database>

The same action data number; if the prestored data sets with the same similarity exist, letters, characters or symbols corresponding to the prestored data sets with the same similarity are output at the same time for manual selection marking, and a data set of action data after marking is used as a marking data set;

s302: training the neural network through the labeling data set to obtain an action recognition model;

the training neural network adopts a back propagation algorithm, the calculation of network parameter gradients adopts a random gradient descent method, cross entropy (cross entropy) is adopted to calculate classification errors, and the process is iterated until the average errors are not descended any more; the loss function L is:

l is the loss function of the optical fiber,

for a true value tag, ++>

Is the j-th value of the output vector s of the softmax logistic regression model, representing the probability that this sample belongs to the j-th class;

in S400, the motion data collected by the motion recognition model recognition sensor is a letter, a word or a symbol corresponding to the motion data collected by the motion recognition model recognition sensor in the database.

2. The method of air handwriting interaction based on deep learning according to claim 1 and wherein in S100, said sensor is a gyroscope.

3. The method of air handwriting interaction based on deep learning according to claim 1 and wherein in S200, the method of preprocessing the motion data includes any one or more of kalman filtering, mean centering and principal component analysis.

4. The method of air handwriting interaction based on deep learning according to claim 1, wherein in S300, the neural network is any one of LSTM and BP.

5. An air handwriting interaction system based on deep learning, which implements the air handwriting interaction method based on deep learning as claimed in claim 1, and is characterized in that the system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to run in units of the following system:

the action acquisition unit is used for acquiring action data through the sensor;

the preprocessing unit is used for preprocessing the action data to obtain an action data set;

the recognition model construction unit is used for constructing a neural network and training the neural network through the action data set after manual labeling to serve as an action recognition model;

and the motion recognition model unit is used for recognizing the motion data acquired by the sensor through the motion recognition model.

Images